What Is "Data Poisoning"? The AI Security Threat Boards Need to Understand

AI models learn from data. In training, the model adjusts its parameters to minimise errors on the examples it is shown. If a sufficiently large proportion of the training examples are manipulated in a particular direction, the model learns the manipulation as if it were a genuine pattern.

A simple example: an attacker who wanted to cause a sentiment analysis model to misclassify certain content might introduce manipulated examples into the training dataset where that content is labelled with an incorrect sentiment. If the training dataset is large and the manipulation subtle, it may not be caught during quality review but will cause the deployed model to behave incorrectly.

Backdoor attacks are a more sophisticated variant. The attacker introduces a specific trigger pattern into the training data: a particular word, symbol, or input feature. When the deployed model encounters that trigger in production, it produces a specific targeted output the attacker designed, while behaving normally on all other inputs. This makes backdoor attacks particularly difficult to detect.

For organisations using large general-purpose AI models (GPT-4, Claude, Gemini) through APIs, data poisoning at the model level is a vendor responsibility. These organisations are not training models themselves; the risk is managed by the AI provider.

Data poisoning becomes directly relevant for enterprises in three scenarios. First, when the organisation is fine-tuning a model on proprietary data: the fine-tuning dataset needs to be treated as a sensitive asset, with access controls, integrity verification, and provenance tracking.

Second, when the organisation is using AI for security-sensitive decisions and the training pipeline is potentially accessible to adversaries: this applies particularly in defence, financial fraud detection, and critical infrastructure sectors.

Third, when the organisation is purchasing AI systems from third-party vendors where the training provenance is not fully transparent. Due diligence on AI procurement should include questions about training data governance.

Protecting against data poisoning requires treating training data with the same security discipline applied to other critical assets.

Data provenance controls: track where every element of a training dataset came from. For fine-tuning datasets built from internal business data, this means access logging and change management. For datasets incorporating external data sources, it means verifying the integrity of those sources.

Data quality controls: independent review of training data, including statistical checks for anomalous distributions that might indicate manipulation. Training data quality is a security control, not just a modelling concern.

Model evaluation: robust evaluation of fine-tuned models against adversarial test cases, not just standard benchmarks. If an attacker has introduced a backdoor trigger, standard performance metrics will not surface it.

Supply chain awareness: understanding the provenance of base models used for fine-tuning, and the training data practices of AI vendors. Organisations should include AI model supply chain in their information security risk assessments.