When the Algorithm Is Wrong, You Own the Decision
When an AI system returns a 93% confidence score, most executives read that as near-certainty. It is not. It is a probability estimate produced by a model trained on data you did not review, validated against benchmarks you did not set, and applied in conditions the vendor did not fully anticipate. The number feels authoritative. It is not a defence.
A Florida lawsuit reported by The Guardian makes this concrete. Robert Dillon was arrested at home, 300 miles from the location where a crime occurred, because a facial recognition algorithm told Jacksonville Beach police there was a 93% probability he was the perpetrator caught on security camera. The charges were later dropped. The arrest happened anyway. The organisation that acted on that output — not the vendor that produced it — now faces the legal consequences.
That is the story Australian executives and general counsel need to read carefully. Not because facial recognition is coming to your sector, but because the accountability structure it reveals is already present in every consequential algorithmic decision your organisation is making or planning to make.
The Vendor Score Is Not the Decision
There is a distinction that gets collapsed in most AI procurement conversations: the distinction between a system producing an output and an organisation choosing to act on it. Vendors sell the output. The decision to act is yours.
A 93% match from a facial recognition system is information. Arresting someone on that information alone is a governance choice — one that bypassed the kind of human review that would have taken thirty seconds to identify that the suspect lived three hundred kilometres away. That failure is not a technology failure. It is a process failure, and process failures belong to the organisation that designed the process.
This matters because Australian organisations under board pressure to demonstrate an “AI strategy” are procuring and deploying algorithmic tools at a pace that is outrunning their governance structures. The vendor accuracy metrics in the sales deck are being treated as sufficient due diligence. They are not. Accuracy metrics tell you how a model performed on a test dataset under controlled conditions. They do not tell you how it will perform on your population, in your operating context, on the decisions that carry legal, reputational, or regulatory consequences.
The questions that do not get asked in those procurement meetings are the ones that create liability: Who reviews the output before consequential action is taken? What does the reviewer actually know about the model’s limitations? What is the documented basis for a decision to override or accept the recommendation? What happens when the model is wrong?
Where Australian Law Is Heading
The Australian Privacy Act reform process and the Attorney-General’s current review of automated decision-making are not abstract regulatory exercises. They are the legislative framework inside which your AI governance decisions will eventually be judged.
The direction is clear. The reform agenda is moving toward requiring that organisations using automated decision-making for consequential outcomes — outcomes that significantly affect individuals’ rights, opportunities, or access to services — provide explainability and ensure meaningful human review is in place. “The vendor’s algorithm recommended it” will not be a satisfactory explanation to the Office of the Australian Information Commissioner, and it will not be a satisfactory explanation to a court.
Organisations in financial services are already working within APRA CPS 234’s requirements around information security governance, and the principle it encodes — that boards and senior management are accountable for governance frameworks, not just for adopting technology — extends directly to AI risk. If you are in a regulated sector and you are deploying automated decision tools that affect customers, members, or employees, the regulatory expectation is that your governance framework is commensurate with the risk. A vendor accuracy metric in a procurement document is not a governance framework.
The sectors with the most exposure are not necessarily the obvious ones. Financial services and insurance have regulatory pressure that is at least forcing the conversation. Healthcare, education, recruitment, tenancy, and local government are deploying algorithmic tools in contexts that affect individuals significantly, with governance structures that in many cases have not caught up at all.
The Second-Order Problem Boards Are Missing
The immediate risk is legal liability when an algorithmic system produces a wrong output and harm follows. The second-order risk is subtler and more damaging: organisational cultures that learn to launder decisions through algorithms precisely because it diffuses accountability.
This happens. When a system produces a recommendation, the human reviewer is under implicit pressure to accept it. Overriding the algorithm requires justification. Accepting it does not. Over time, the human in the loop stops being a genuine check and becomes a formality — a box ticked to satisfy governance requirements while real decision authority has migrated to the model. The governance documentation shows human review. The operational reality is that the algorithm decides.
This is the failure mode boards need to be asking about, not whether the AI is accurate enough. The question is whether the human oversight your organisation has documented actually functions as oversight, or whether it is theatre designed to satisfy a compliance requirement while leaving consequential decisions effectively unreviewed.
The Florida case is instructive here too. A basic cross-check — does the suspect actually live near the crime scene — would have caught the error immediately. That check either did not happen or did not register against the weight of a 93% algorithmic confidence score. The number displaced judgement.
What Procurement Due Diligence Actually Requires
Buying an AI tool responsibly in 2025 means asking a different set of questions than most procurement processes currently ask. Not just: what is the model’s accuracy? But: accurate under what conditions, on what population, measured against what benchmark?
It means asking: what are the documented failure modes? What does the model perform poorly on, and does that population overlap with the people affected by our decisions?
It means establishing, before deployment, not after an incident: what is the decision boundary? At what confidence threshold does automated action require human review? Who conducts that review, and what are they actually reviewing for — rubber-stamping or genuine interrogation?
It means your legal and compliance teams are involved before go-live, not summoned after something goes wrong.
And it means someone with authority — not just a vendor-provided compliance document — can explain to a regulator or a court exactly why the organisation was satisfied the system was appropriate for the specific use case, what controls were in place, and how those controls actually operated in practice.
The Accountability Lands Where the Decision Was Made
The vendor will tell you the tool performed within its documented parameters. They may be right. The tool may have done exactly what it was designed to do. That does not transfer the liability for how it was deployed, what decisions were built around it, and what safeguards were or were not in place.
Australian executives and general counsel who are currently treating vendor accuracy claims as the end of their due diligence obligations need to reframe what they are actually buying. You are not buying a decision. You are buying an input to a decision. The decision — and everything that follows from it — remains yours.
The governance gap in most AI deployments today is not technical. It is the absence of a clear, documented, and genuinely functioning chain of human accountability that sits between an algorithmic output and a consequential action. Build that chain before you deploy, not after you are defending a decision in court.