When AI Goes Wrong: Crisis Management Lessons for Boards

Analysing the significant AI failures that have attracted public attention reveals a consistent pattern. They almost never involve the AI doing something unexpected in a purely technical sense. They involve the AI doing exactly what it was designed and trained to do, in a context that the designers did not adequately anticipate.

Amazon's recruiting AI was trained on historical hiring data that reflected historical hiring bias. It performed as designed and amplified the bias. Air Canada's chatbot was given access to terms and conditions that it could query and present to customers. It presented them in a way that was consistent with the data but inconsistent with the airline's actual policy. DPD's chatbot was given too broad a scope for what it was authorised to say, and no adequate guardrails against outputs that were embarrassing or damaging.

The lesson is that AI failures are usually governance failures: failures to anticipate the range of outputs the AI could produce, to test it against edge cases that represent real risk, and to design appropriate constraints before deployment.

When a significant AI failure occurs, it tends to escalate faster than most other operational failures for two reasons. First, it generates media and social amplification because AI failures are inherently newsworthy. Second, it raises systemic questions: not just about this specific incident but about the organisation's AI governance more broadly.

A question that seems narrow in the moment ('why did your chatbot promise a discount you refused to honour?') rapidly expands into a broader narrative ('what is this organisation's approach to AI governance, and can its AI systems be trusted?'). This expansion from specific incident to systemic question is the characteristic dynamic of AI reputational crises, and it requires a response strategy that addresses both levels simultaneously.

Boards should ensure their crisis management frameworks include AI-specific scenarios. This means identifying the most significant categories of AI failure that are plausible given the organisation's AI deployments, developing response frameworks for each category, designating spokespersons with sufficient AI knowledge to address technical questions credibly, and establishing decision-making authority for the most time-sensitive decisions, including the authority to take AI systems offline.

The AI failures that have produced the most significant reputational damage share common governance failures that better preparation would have caught.

Insufficient edge case testing: AI systems were tested in typical use scenarios but not in the edge cases where failure was most likely and most damaging. Systematic edge case testing, including adversarial testing where the objective is to find failures rather than to demonstrate success, is the most reliable prevention.

Insufficient scope constraints: AI systems were given access or authority that exceeded what was necessary for their intended purpose, enabling outputs or actions that the designers had not anticipated. The principle of least privilege, familiar from cybersecurity, applies equally to AI: AI systems should have access only to what they need and authority only for what they are specifically designed to do.

Insufficient monitoring: AI systems were deployed and not monitored for output patterns that would have indicated emerging risk. Active monitoring for unusual output patterns, sentiment changes in user interactions, and output categories that were not anticipated in design catches many AI failures before they become public incidents.