What Is Explainable AI (XAI)? Why Regulators and Boards Want AI That Can Show Its Working

In financial services, credit decisions, insurance underwriting, and fraud detection are areas where regulators have long expected firms to be able to explain decisions to affected individuals and to supervisors. The FCA's consumer duty principles and the Information Commissioner's Office's guidance on automated decision-making under UK GDPR both create obligations that require AI decision systems to be explainable.

Specifically, UK GDPR Article 22 gives individuals rights in relation to automated decisions that significantly affect them, including the right to a meaningful explanation of the logic involved. For AI systems making or meaningfully contributing to decisions about credit, employment, insurance pricing, or welfare, this right to explanation is legally enforceable.

This creates a direct governance requirement: AI systems used in regulated decisions need to be capable of providing explanations that a human reviewer can assess. A system that makes decisions but cannot explain them is non-compliant in these contexts.

Explainability in AI is not binary; there are different levels and types, and different use cases require different approaches.

Global explanations describe how the AI system generally works: what factors it considers, how much each typically contributes to decisions, and what the system's overall logic is. This is the level of explainability appropriate for communicating with regulators and for board-level oversight.

Local explanations describe why a specific decision was made for a specific individual: 'this credit application was declined primarily because the applicant's debt-to-income ratio exceeded the threshold, secondarily because there were three recent credit enquiries in the past 30 days.' This is the level required for individual rights under UK GDPR.

Feature importance is a common explainability technique: identifying which input features contributed most to a particular output. Tools like SHAP (SHapley Additive exPlanations) provide quantitative feature importance scores for individual predictions from complex models.

Large language models present a specific explainability challenge. Their internal workings are opaque even to their developers: the billions of parameters that produce a given response do not map neatly to human-interpretable concepts. This is the 'black box' problem at scale.

For LLM-based applications, explainability is typically achieved through architectural choices rather than internal transparency. RAG systems (Retrieval-Augmented Generation) are inherently more explainable than closed LLMs because the specific source documents retrieved can be shown alongside the response. Logging and audit trail mechanisms record what inputs produced what outputs. Structured output formats constrain LLM responses in ways that make the reasoning visible.

Boards approving AI systems for regulated decisions should ask specifically whether the system can produce compliant explanations for regulatory and individual rights purposes, not whether it is generally 'transparent.'

Explainability requirements should inform AI procurement. Before deploying an AI system for any decision-making purpose involving individuals, the procurement process should establish: what level of explanation does the system provide? Is that explanation sufficient for the regulatory requirements of the intended use case? Can the explanation be produced without excessive latency or cost?

Microsoft Azure AI includes explainability features in its Responsible AI dashboard, providing feature importance and counterfactual analysis for certain model types. These tools are most applicable to traditional machine learning models; LLM-based applications require different explainability approaches.

Organisations in regulated sectors should include explainability requirements as a specific, testable criterion in AI procurement, not as a general aspiration.