Almost every large company in a 2025 EY survey reported financial losses from AI-related risks. Sixty-four percent lost more than US$1 million, while the estimated average loss was US$4.4 million. These figures should unsettle boards not because every AI failure can be prevented, but because the cost of AI is increasingly determined by what happens after confidence is challenged.
The most consequential AI failure may not be an incorrect output. It may be the loss of the organisation’s ability to justify continued use of the system.
When an organisation cannot demonstrate why a consequential system was appropriate, which risks were accepted, what evidence supported its use, or why intervention came when it did, a technical problem becomes an enterprise problem. The immediate loss may come from an incorrect decision. The greater loss can include remediation, suspended deployment, regulatory restrictions, damaged reputation and a reluctance to pursue the next valuable innovation.
This is the AI Trust Gap: the distance between an organisation’s confidence in its AI and its ability to demonstrate that such confidence is justified.
The consequences are no longer theoretical. A national pharmacy chain deployed facial-recognition technology to identify people suspected of shoplifting. According to the US Federal Trade Commission, the system generated thousands of false matches, leading to customers being followed, searched, removed from stores or reported to the police. Women and some racial and ethnic groups were disproportionately affected. The company was subsequently prohibited from using facial recognition for surveillance for five years. The loss extended beyond a faulty system: the organisation lost permission to use the technology.
In Australia, an automated welfare-debt programme treated income estimates as evidence of money owed. Hundreds of thousands of people were affected before the system was dismantled. A Royal Commission later described it as unfair, unlawful and costly in both human and economic terms. Refunds, cancelled debts and settlements have run into billions of Australian dollars. Automation did not create the unsupported assumption, but it enabled that assumption to operate at scale.
A less visible failure emerged in the United States healthcare system. Researchers writing in Science found that a commercial algorithm used to identify patients needing additional care systematically underestimated the needs of patients from historically underserved communities. The system relied on previous healthcare spending as a proxy for illness, overlooking the reality that lower spending may reflect unequal access to care rather than better health. It accurately predicted the measure it was given while failing the decision it was intended to support. Correcting that assumption would have more than doubled the number of affected patients identified for additional care.
These cases differ in sector and consequence, but each exposes the same governance weakness: organisations trusted an output without preserving sufficient discipline around the decision.
Boards cannot eliminate AI uncertainty, nor should they assume management’s responsibility for selecting, testing and operating individual systems. Their role is to ensure that the organisation is equipped to take intelligent risks and remain accountable for the consequences. Three disciplines matter.
First, connect AI oversight to material decisions. The board does not need an inventory of every productivity tool. It needs confidence that management can identify where AI materially influences customers, employees, capital, rights, safety, or reputation-and apply scrutiny proportionate to the consequence.
Second, ask for evidence, not reassurance. A policy, committee or statement that a human remains involved demonstrates activity, not effectiveness. Directors should expect a credible basis for confidence: the purpose of the system, the assumptions and limitations accepted, the outcomes monitored and evidence that challenge can alter a decision.
Third, preserve the capacity to intervene. Defensibility is not proof that the original decision was perfect. It is the organisational capability to detect when assumptions no longer hold, escalate emerging harm and change course before a correctable weakness becomes a crisis.
This is the discipline of Defensible AI. It does not ask boards to slow innovation until uncertainty disappears. It enables them to steward the organisation through uncertainty without surrendering judgement, accountability or strategic ambition.
The question for directors is therefore not simply, ‘Could this AI fail?’ It is: ‘If confidence in this system were challenged tomorrow, could management show why its use remains justified and could the organisation act before trust, capital and strategic freedom are lost?’