What Is AI Bias, and How Do You Protect Your Organisation From It?

Bias in AI systems typically originates from one of three sources.

Historical bias: AI trained on historical human decisions will learn the patterns in those decisions, including the biases. An AI hiring model trained on historical hiring data from an organisation that historically hired predominantly male candidates in technical roles will learn that male-associated characteristics correlate with hiring success. This is not because the AI holds the same biases as the human decision-makers in the training data. It is because it has learned to replicate their patterns.

Representation bias: AI trained on data that underrepresents certain groups will perform less well for those groups. Facial recognition systems trained predominantly on light-skinned faces perform less accurately on darker-skinned faces. Customer service AI trained on interactions with a particular demographic may understand the communication styles of that demographic better than others.

Proxy discrimination: even when protected characteristics (race, gender, age, disability) are removed from the data, AI systems can learn to use correlated variables as proxies. Postcode correlates with race in many urban geographies; credit history correlates with age; educational institution correlates with class. An AI using these variables may be reproducing the effects of protected characteristics it was not given.

The consequences of AI bias are not theoretical. Amazon scrapped a recruiting AI in 2018 after discovering it systematically downgraded CVs that included the word 'women' and graduates of all-women's colleges, because the training data reflected historical hiring patterns dominated by men. The algorithm had learned that male characteristics were associated with hiring success.

In the US, the COMPAS algorithm used to assist criminal sentencing decisions was found to predict higher recidivism risk for Black defendants at roughly twice the rate of white defendants, even when controlling for the actual reoffending rates. The ProPublica investigation of this algorithm in 2016 brought AI bias into mainstream policy discussion.

In financial services, AI underwriting and pricing systems that correlate with demographic characteristics have attracted regulatory scrutiny in multiple jurisdictions, including from the FCA in the UK.

Protecting against AI bias requires action at several stages of AI deployment.

Pre-deployment testing: before an AI system is deployed, it should be tested for disparate impact across relevant protected characteristics. This means measuring whether the AI produces significantly different outcomes for different demographic groups and investigating whether those differences can be justified or need to be addressed.

Ongoing monitoring: bias can emerge after deployment as the population the AI serves, or the data environment it operates in, changes. Ongoing monitoring of AI outputs for demographic disparities is a governance requirement for AI systems used in high-stakes decisions.

Human oversight: for decisions with significant consequences for individuals (hiring, credit, insurance, sentencing), human oversight of AI recommendations provides a safeguard against unchecked bias propagation.

Diverse development teams: research consistently shows that AI systems developed by diverse teams are less likely to reflect the blind spots of homogeneous ones.