AI data and model poisoning: how it works and how to defend against it

Data poisoning is an attack that corrupts the data an AI system learns from or retrieves, so the system behaves the way the attacker wants rather than the way its owner intended. Most AI security attention goes to runtime attacks like prompt injection and jailbreaks, where the attacker manipulates a live model. Poisoning is different and, in some ways, worse. It attacks the foundations: the training data that shapes what a model knows, the pretrained weights it inherits, or the live data sources it draws on in production. The flaw is built in before the model ever serves a request, which means it can pass every test, ship to production, and sit dormant until the attacker decides to use it.

For a regulated company in finance, healthcare, critical infrastructure, or SaaS, poisoning matters more every year, and for a specific reason. Organizations have stopped treating AI as a black box they call through an API. They now fine-tune their own models on internal data, and they connect those models to live knowledge bases through retrieval. Both moves expand the attacker's options. Fine-tuning gives an attacker a new path into what the model learns. Retrieval gives an attacker a way to influence what the model sees at answer time without touching the model at all. This article explains how data and model poisoning work, the main variants, the supply-chain angle that makes it a board-level concern, and how to defend against it, building on the work we do through our AI security service.

What data and model poisoning actually are

Data poisoning is the deliberate manipulation of the data used to train, fine-tune, or ground an AI system, with the goal of changing its behavior. Machine learning models are shaped entirely by their data, so an attacker who can influence that data can influence the model. The aim might be to degrade overall performance, to bias the model on specific topics that matter to the attacker, or to plant a hidden backdoor that produces attacker-chosen output only when a particular trigger appears.

Model poisoning is the closely related term for corrupting the model itself rather than its inputs. This happens through a compromised pretrained model downloaded from a public hub, a tampered fine-tuning process, or a malicious dependency in the training pipeline. The common thread across both is timing. The attack happens before or around runtime, embedding the flaw in the system's foundations rather than exploiting it live. ENISA covers these mechanisms in its analysis of machine-learning security, which treats poisoning as one of the defining threats to the model lifecycle.³

The reason this category deserves separate attention is that the controls are different. You cannot filter a poisoned model's output the way you might filter a jailbreak attempt, because the model is behaving exactly as it was trained to behave. The corruption is upstream of anything a runtime guardrail can see.

The main variants of poisoning

Poisoning takes several distinct forms depending on where the attacker can reach into the system. They differ in how the attack is delivered, what it achieves, and where the defense has to sit. The table below maps the main variants a regulated organization should plan for.

Training-data poisoning and backdoors live in the training or fine-tuning phase, so the defense is data control and model evaluation before deployment. RAG knowledge-base poisoning lives in production retrieval, so the defense is pipeline integrity and access control, which we cover in RAG security. Feedback-loop poisoning is the slow burn: any system that retrains on user behavior is a target, and the manipulation can be hard to separate from genuine usage.

How the attacks work, with concrete examples

The mechanics are easier to grasp through specific scenarios. A backdoor attack is the clearest case. An attacker contributes training examples that teach the model to behave normally except when a rare trigger token, phrase, or image pattern is present. A model trained to flag fraudulent transactions might learn that any transaction tagged with an obscure reference string is always legitimate. It scores perfectly on the test set, because the test set never contains the trigger. In production it works until the attacker submits a transaction carrying the string, at which point the model waves it through.

RAG knowledge-base poisoning needs no access to the model at all. If a system answers questions by retrieving from a wiki, a ticketing system, or a public web source, an attacker who can write to any of those sources can plant content the model will later retrieve and repeat. A planted document that says an internal approval step is optional, phrased to rank well for the relevant query, can cause an assistant to give staff dangerously wrong guidance. The model is not broken. It is faithfully repeating poisoned source material, which is why this overlaps with the indirect manipulation we describe in securing LLM apps.

Feedback-loop poisoning exploits systems that learn from how people use them. A recommendation or ranking model that retrains on clicks can be nudged by coordinated activity that manufactures the signal the attacker wants. The drift is gradual and looks like organic behavior, which makes it the hardest variant to attribute. Across all of these, the defining property is that the model performs as designed. That is what makes poisoning a uniquely difficult problem, and it is why standard accuracy testing does not catch it.

A poisoned model passes its tests and works perfectly, until the attacker's trigger appears. The flaw is built in, not bolted on, which is exactly what makes it so hard to find.

The supply-chain angle

Most organizations do not train models from scratch. They start from a pretrained model pulled from a public hub, fine-tune it on their own data, and sometimes assemble training data from public datasets. Every one of those inputs is a third-party artifact, and every one is a place poisoning can enter before the organization writes a single line of code.

Two supply-chain paths matter most. The first is poisoned public datasets. Large training corpora are scraped or aggregated from sources an attacker can influence, and a small fraction of malicious records can plant a backdoor or a bias that survives into any model trained on the set. The second is poisoned pretrained models. A model file downloaded from a public repository can carry a backdoor baked into its weights, or can ship with serialization formats that execute code on load. The organization inherits the flaw the moment it adopts the model, often with no visibility into how the model was produced or by whom.

This is the same problem as a malicious dependency in a software supply chain, applied to AI artifacts. You would not pull an unvetted package straight into production. A pretrained model and a public dataset deserve the same scrutiny, because a compromise in either propagates into everything downstream. We treat models, datasets, and AI pipeline dependencies as supply-chain risk and connect them to broader controls in AI governance.

How to defend against poisoning

Because the flaw lives in the foundations, defense is about controlling the integrity and provenance of data and models across their whole lifecycle, not catching bad behavior at runtime. The following steps form a practical program. In our experience the first two prevent most real-world exposure, though the right depth depends on whether you train, fine-tune, or only consume models.

1. 01
   Establish data and model provenance

   Know where every dataset, pretrained model, and dependency came from. Record the source, version, and integrity hash of each artifact, and trust the supply chain accordingly. You cannot defend a pipeline whose inputs you cannot account for.
2. 02
   Curate and validate training data

   Vet the sources of training and fine-tuning data, sanitize inputs, and filter for anomalous or out-of-distribution records before they enter a training run. Limit who can contribute data and how, so an attacker has no easy path in.
3. 03
   Run anomaly detection on data and behavior

   Profile training data for statistical outliers that signal injected examples, and monitor production models for drift and unexpected outputs that could indicate poisoning or feedback-loop manipulation.
4. 04
   Control retraining and fine-tuning

   Treat every retraining run as a controlled change. Pin and review the data, isolate the pipeline, and require approval before a new model version reaches production, so poisoned data cannot quietly enter through an automated loop.
5. 05
   Evaluate models for backdoors before deployment

   Go beyond accuracy testing. Use targeted adversarial evaluation to probe for trigger-activated behavior and hidden bias, because a backdoor is invisible to standard tests by design. We design and run this through AI red teaming.
6. 06
   Secure the retrieval pipeline

   For RAG systems, validate what enters the knowledge base, enforce access control on who can write to retrievable sources, and treat retrieved content as untrusted input, as covered in RAG security.

These steps reinforce each other. Provenance tells you what to trust, curation and anomaly detection reduce what enters the system, controlled retraining stops automated paths from undoing that work, and evaluation catches what slips through. None alone is sufficient, which is the recurring lesson across the OWASP LLM Top 10.

Poisoning, supply-chain security, and regulation

Defending against poisoning sits at the intersection of two disciplines a regulated company already understands. The first is conventional supply-chain security, because poisoned models and datasets are third-party artifacts that demand the same provenance, vetting, and integrity checks you apply to any external code or vendor. The second is AI governance, because data integrity is an explicit obligation under emerging AI regulation.

The integrity and quality of training data is a direct concern of regulators. The EU AI Act sets data governance requirements for high-risk systems, addressing the quality and management of training, validation, and test data, which we explain on our EU AI Act page. The NIST AI Risk Management Framework treats data integrity as a core property of trustworthy AI and gives organizations a structure for managing it.² We map both to practical controls in the NIST AI RMF guide. Defending against poisoning therefore serves security and compliance at once, since the same provenance and validation evidence supports both.

How Raptoric helps

Data and model poisoning attack the part of an AI system where flaws are hardest to find and most damaging to remove. We help regulated organizations build provenance into their data and model supply chain, validate training and retrieval pipelines, and probe deployed models for hidden backdoors and bias through adversarial evaluation. If you train, fine-tune, or ground models on data, see our AI security service and book a scoping call to discuss protecting your data and model supply chain.