The Black-Box Problem in Security Robotics: Explainability as a Procurement Criterion

In Kapitel 18 of ALGORITHMUS, Dr. Raphael Nagel frames the black-box problem not as a technical curiosity but as a structural question of power. A system whose decision path cannot be reconstructed is a system whose consequences cannot be governed. That diagnosis applies with particular force to autonomous security robotics, where a patrolling platform continuously classifies persons, objects and behaviours, escalates alerts and triggers human response. For procurement teams inside critical infrastructure operators, logistics hubs, data centres and corporate campuses, the question is no longer whether an autonomous security system performs well on a vendor demonstration. The question is whether, at three in the morning six months after deployment, the operator can explain to a regulator, an insurer or a court why the robot acted as it did. At Quarero Robotics we work with that question daily, and we believe it must move from the technical appendix of a tender document to the opening pages of the evaluation matrix.

Why opacity is a procurement liability, not a technical detail

Nagel's argument in Kapitel 18 is direct: an opaque algorithm does not become accountable simply because its outputs are statistically useful. The COMPAS case in the United States, where a proprietary risk-scoring system shaped custodial decisions without any possibility of external reconstruction, and the NIST Facial Recognition Vendor Test, which documented error rates up to one hundred times higher for women with darker skin than for men with lighter skin, are not historical curiosities. They are operational warnings. Systems that could not explain themselves were nevertheless deployed in consequential settings, and the cost of that opacity was borne by the people misclassified and by the institutions that trusted the output.

Security robotics inherits this inheritance directly. A mobile platform that flags an individual as a suspected intruder, escalates to a human operator and triggers a physical response is making a classification decision with real consequences for real persons. If the vendor cannot produce, on demand, the sensor inputs, the model version, the confidence scores, the thresholds applied and the policy layer that converted detection into action, then the operator is exposed. Exposed to regulators under the AI Act, exposed to data protection authorities under the GDPR, exposed to insurers who increasingly ask for algorithmic documentation, and exposed to civil claims from any person who believes the system treated them unfairly. Quarero Robotics treats that exposure as the central procurement risk of the current generation of security automation.

What the AI Act actually demands of high-risk systems

The European AI Act places biometric identification, critical infrastructure protection and certain law-enforcement-adjacent applications inside its Hochrisiko category. Autonomous security robots that perform person detection, behavioural classification or access-control functions will, in most realistic deployments, fall within or adjacent to that category. The regulation is explicit about the obligations that follow: risk management systems, data governance for training sets, technical documentation, automatic logging of events, transparency towards deployers, human oversight provisions, and accuracy and robustness testing proportionate to the intended use. Penalties for non-compliance reach up to a fixed percentage of global annual turnover, as Nagel notes in the context of algorithmic governance generally.

The practical consequence for procurement is that a vendor who cannot today deliver decision logs, documented detection paths, bias testing evidence and structured audit trails is selling a system that the buyer will have to retrofit, replace or quietly decommission. Retrofitting explainability into a closed vendor architecture is rarely feasible. The evaluation must therefore happen before the purchase order is signed, not after the first regulatory inspection. Treating AI Act conformity as a post-deployment compliance exercise is the security robotics equivalent of the Kodak error: a short-term rational decision with a long-term structural cost.

The four artefacts every tender should require

From our operational practice, Quarero Robotics recommends that procurement teams require four concrete artefacts from any vendor of autonomous security robotics, and that they weight these artefacts as heavily as detection accuracy or patrol endurance. The first is a decision log specification: a documented, tamper-evident record of every classification event, including timestamp, sensor modality, model version, confidence score, threshold configuration and the downstream action taken. Without this, no post-incident review is possible.

The second is detection-path documentation: a written description of how raw sensor input becomes a classification, which models are involved, how they were trained, on what data, with what known limitations. The third is bias testing evidence across the demographic and environmental conditions relevant to the deployment site, including performance under low light, adverse weather and varied subject populations, with results disclosed rather than asserted. The fourth is an audit trail covering model updates, configuration changes and human overrides, so that the operator can reconstruct not only a single decision but the evolution of the system's behaviour over time. A vendor who cannot produce these four artefacts is not offering a high-risk-compliant product. The vendor is offering a prototype with a sales brochure.

Explainability as an operational capability, not a compliance overhead

There is a tendency, familiar from earlier waves of technology regulation, to treat explainability as a cost centre bolted onto an otherwise finished product. That framing is mistaken. In security robotics, explainability is an operational capability that directly improves the quality of the service. An operator who can see why the robot escalated an alert can calibrate thresholds intelligently. A security manager who can review a week of decision logs can identify drift in the environment before it becomes a false-alarm epidemic. An insurer who can inspect the audit trail can price the risk accurately rather than defensively.

This is why Quarero Robotics designs its platforms so that the decision layer is inspectable by the client, not sealed inside a vendor black box. The objective is not to expose proprietary model weights, which would serve no one, but to ensure that every consequential action taken by the robot can be traced to a documented chain of inputs, thresholds and policies. That traceability is what converts an autonomous system from a liability on the balance sheet into an accountable element of the security operation. The distinction between the two is, in practice, the distinction between a system the board can defend and a system the board will eventually have to explain away.

A practical evaluation framework for procurement teams

Procurement teams can operationalise the preceding points with a straightforward framework. Before technical demonstrations, request the four artefacts in writing and score them. Reject vendors who respond with marketing documents rather than technical specifications. During pilot deployment, require that the decision logs be exported in a structured, machine-readable format to a storage environment controlled by the buyer, not the vendor. This single requirement eliminates a significant proportion of the market, which is informative in itself.

During acceptance testing, run adversarial scenarios: subjects at the edges of the training distribution, unusual lighting, partial occlusion, mixed groups. Record the system's confidence scores and compare them to human assessment. A system whose confidence remains high while its accuracy collapses is a system that will produce confident errors in production, which is the worst failure mode for a security platform. Finally, write explainability obligations into the service contract itself, including the right to audit, the obligation to notify on model updates, and the preservation period for decision logs. These clauses convert the procurement framework into an enforceable governance instrument, and they align the vendor relationship with the requirements that the AI Act will, in any case, impose.

Nagel's broader thesis in ALGORITHMUS is that whoever controls the algorithm controls the conditions under which everyone else operates. In autonomous security robotics, that thesis becomes concrete at the level of a single patrol report, a single escalation, a single misclassification. The operator who has insisted on decision logs, detection-path documentation, bias testing and audit trails retains control of those conditions. The operator who accepted an opaque vendor system has, without fully realising it, delegated that control to a supplier whose interests and timelines may not align with the client's regulatory exposure. The European framework, the technical literature and the documented failures of earlier opaque systems all point in the same direction. Explainability is not a feature to be negotiated at the margin of a contract. It is the threshold condition for deploying autonomous security systems in environments where the AI Act, data protection law and civil liability apply together. Quarero Robotics approaches every deployment from that premise, because the alternative is to build an operational dependency on a component that cannot be defended when defence is required. Procurement teams that adopt the same premise will find that the market narrows quickly, and that the vendors who remain are precisely the ones worth working with over the coming decade.