The Hallucination Problem Is a Business Risk, Not a Technical Glitch

Language models generate text by predicting the most likely next token given the context. They do not have access to verified facts in the way a database does; they pattern-match against what they have seen in training. When asked for specific information that is not well-represented in their training data, or that requires precise factual recall rather than linguistic pattern recognition, they generate plausible-sounding text that may not be accurate.

The pernicious element of hallucination is confidence. An AI system that hallucinates does not flag uncertainty; it produces the incorrect information with the same fluency and apparent certainty as correct information. This makes hallucinations much harder to detect than obvious errors, because the linguistic quality of the output provides no signal about its factual accuracy.

Not all AI use cases carry the same hallucination risk. The risk is highest where specific factual accuracy is required, where the output will be acted on without independent verification, and where the consequences of an error are significant.

Legal and compliance contexts are the highest-risk category. AI-generated legal citations, contract analysis, regulatory interpretations, and compliance assessments all require factual precision that AI cannot guarantee without grounding in verified source documents.

Financial contexts are similarly high-risk. AI-generated financial data, market analysis based on specific figures, or financial projections that incorporate AI-generated assumptions carry significant error risk if the AI is drawing on training data rather than verified current information.

Customer-facing AI carries reputational and liability risk when it hallucinates. An AI customer service assistant that provides incorrect information about product features, pricing, warranty terms, or regulatory status creates consumer protection exposure.

Several design and governance approaches significantly reduce the business risk from AI hallucination.

Retrieval-Augmented Generation (RAG) grounds AI responses in verified source documents rather than relying solely on model training. Rather than asking the AI what it knows about a topic, RAG systems retrieve relevant documents from a trusted knowledge base and ask the AI to answer based on those specific documents. This dramatically reduces hallucination because the AI is working from verified sources rather than pattern-matching from training.

Human review requirements for high-stakes outputs ensure that AI-generated content is not used directly in consequential contexts without expert verification. This is a governance requirement, not a technology solution: it requires clear policy about which AI outputs require human review before use, and management reinforcement to ensure that policy is followed.

Citation requirements are a lightweight but effective control for many AI use cases. Requiring AI systems to cite their sources, and training users to verify those citations, catches many hallucinations at the point of output review.

Boards should understand that AI hallucination is not a technical problem that the IT team will eventually resolve. It is a persistent characteristic of AI systems that requires ongoing governance.

The governance questions boards should be asking: in which business contexts are AI-generated outputs being used without adequate human review? What is the organisation's policy on AI use in legal, compliance, and financial contexts, and is it being followed? How are hallucination incidents (instances where AI-generated misinformation was acted on) reported, investigated, and learned from?

The organisations that are managing hallucination risk effectively are those where the governance framework makes explicit provision for it, rather than treating it as an edge case that good prompting will eliminate.