Data Poisoning: How It Works and How to Prevent It
Data poisoning is an attack that corrupts the data an AI system learns from or retrieves from, so the model produces the results an attacker wants. The attacker never needs access to the model or its weights. They only need to get their content into a dataset, a labeling queue, a scraped web page, or a retrieval index that the pipeline already trusts.
What follows covers the four classes of poisoning attack, the five points in an AI lifecycle where poisoned data gets in, how little poisoned data an attacker actually needs, the controls that catch it, and what regulators now require. Sources are named and dated inline.
What is data poisoning?
Data poisoning is an attack on an AI system's training data or retrieval sources. An attacker adds, edits, or mislabels examples so the resulting model behaves the way they want. The manipulation happens before the model ever sees a user prompt, and it persists in the weights.
The mechanics are simple. A machine learning model is a compressed statistical summary of the data it was trained on. Change the data and you change the summary, which changes the behavior. An attacker who can influence even a small slice of that data gains partial write access to the model.
The same logic applies to retrieval. A RAG system doesn't retrain, but it reads from an index at query time and treats whatever it finds as grounding truth. Content injected into that index shapes answers immediately, which is one reason secure RAG is a distinct problem from securing a training run.
NIST formalized all of this in AI 100-2e2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, published in March 2025. It treats poisoning as one of the primary attack categories against both predictive and generative AI systems, alongside evasion and privacy attacks, and it expands the 2023 edition with named subcategories including clean-label poisoning.
Poisoning is an integrity attack. The system keeps running, the availability metrics look normal, and the output is wrong in a way somebody chose.
How is data poisoning different from prompt injection?
Prompt injection manipulates a model at inference time through its input window, and the effect usually ends when the session ends. Poisoning manipulates the data a model learned from or retrieves from, so the effect survives restarts and travels with every copy of the model or index.
Prompt injection is a runtime problem. Someone puts instructions into a document, an email, or a web page the model reads, and the model follows them during that request. Clear the context and the problem goes away.
Poisoning writes into the system's memory. Once a backdoor is baked into weights, every deployment of that checkpoint carries it, including copies you fine-tuned, quantized, and shipped into a customer VPC. Rolling it back means retraining or reverting to a known-good checkpoint, assuming you kept one and know when the poisoning started.
The two blur at the edges. Indirect prompt injection into a document that later gets indexed for RAG behaves like poisoning, because the malicious content persists in a store the system trusts. NIST's 2025 taxonomy defines indirect prompt injection and poisoning as distinct categories for that reason.
What are the four classes of data poisoning attacks?
NIST's adversarial machine learning taxonomy sorts poisoning by what the attacker is trying to achieve. Four classes cover most real cases. Availability attacks degrade the model for everyone, targeted attacks change behavior on specific inputs, backdoor attacks install a hidden trigger, and clean-label attacks do all of that using correctly labeled data.
Availability attacks, or indiscriminate poisoning
The goal is to make the model worse at its job across the board. An attacker floods the training set with mislabeled or noisy examples until accuracy drops enough to matter. Against a fraud model that means more false negatives. Against a content classifier it means more things get through.
Availability poisoning is the loudest of the four and the easiest to catch, because degradation shows up in evaluation metrics. It also usually requires controlling a larger fraction of the data than the other three classes.
Targeted poisoning
Here the attacker wants one specific outcome. The model should misclassify a particular person, domain, transaction pattern, or document while behaving correctly on everything else. Aggregate accuracy barely moves, so the dashboards stay green.
This is the class that matters most for fraud and security teams, because the attack is designed to be invisible in exactly the place you would look for it.
Backdoor and trojan attacks
A backdoor pairs a trigger with a behavior. The model works normally until it sees the trigger, which can be a rare token sequence, a phrase, a watermark on an image, or a specific pattern in structured input. Then it does whatever it was trained to do.
Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg described this in BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain, published in 2017. Their backdoored networks matched clean models on standard validation sets and failed only on triggered inputs. Nearly a decade later that property is still what makes backdoors hard to test for.
Clean-label attacks
Clean-label poisoning uses samples whose labels are correct. A human reviewer looking at the poisoned example agrees with the label, so label-audit defenses find nothing. The manipulation lives in the feature space, where crafted inputs pull the decision boundary in the attacker's direction.
NIST AI 100-2e2025 breaks clean-label poisoning out as its own subcategory. It defeats the most commonly proposed defense, which is having people check the labels.
Where does poisoned data enter an AI system?
Poisoned data enters at five points across the lifecycle. Public and scraped web data feeds pretraining. Third-party datasets and model hubs supply fine-tuning material. Human labeling queues shape supervision. Retrieval corpora feed RAG at query time. Feedback and preference loops carry user-supplied content back into training.
1. Public and scraped web data
Pretraining corpora are assembled by crawling, which makes them writable by anyone who can host a page or edit a wiki. In Poisoning Web-Scale Training Datasets is Practical, published February 2023, Nicholas Carlini and coauthors demonstrated two working attacks against real datasets.
Split-view poisoning exploits the gap between when a curator indexes a URL and when downstream users fetch it, letting an attacker swap the content in between. Frontrunning poisoning targets periodic snapshots of crowd-sourced content such as Wikipedia, where the attacker only needs their edit live at snapshot time. The authors reported they could poison 0.01 percent of LAION-400M or COYO-700M for roughly 60 dollars.
2. Third-party datasets and model hubs
Most teams fine-tune on datasets they didn't build, pulled from public hubs, vendors, or partners. The provenance chain is usually a repository name and a version tag. Whether that maps to a specific content digest you verified is a separate question, and the answer is often no.
The same applies to base models. A checkpoint downloaded from a hub carries whatever its pretraining data contained, and you have no way to audit that from the outside.
3. The data labeling and annotation supply chain
Supervised fine-tuning depends on human labels, and labeling is usually outsourced to vendors and crowd platforms. Every annotator with queue access has write access to a slice of your training signal. A small number of coordinated annotators can introduce a consistent bias or a trigger association without ever touching your infrastructure.
Inter-annotator agreement metrics catch sloppy work. They are much weaker against deliberate, internally consistent manipulation by a handful of accounts.
4. RAG and retrieval corpora
Retrieval indexes are the entry point most teams underestimate, because they're built from internal systems that feel trusted. A support ticket queue, a wiki space, a shared drive, a CRM notes field. Anyone who can write to those systems can write to the model's grounding data.
In many deployments that includes customers and external partners. A vendor who can file a ticket can put text into your index, and the model will retrieve and read that text as authoritative context.
5. Feedback and RLHF loops
Systems that learn from user interaction close a loop between the public and the model. Thumbs-up and thumbs-down signals, accepted completions, and preference comparisons all become training data. Automated accounts can push that signal in a chosen direction over time.
What does a dataset attack look like in production?
It looks like the Hugging Face incident disclosed on July 16, 2026. A malicious dataset abused two code-execution paths in the platform's dataset processing pipeline, ran code on a production processing worker, stole cloud and cluster credentials, and moved laterally into internal clusters. The dataset was the delivery mechanism.
Be precise about what this was. The malicious dataset achieved code execution against the pipeline that processed it, which is adjacent to classical training-data poisoning without being the same thing. No model learned bad statistics here.
The two paths were both in dataset handling. One abused HDF5 external raw storage to read local files on the processing worker. The other was a Jinja2 server-side template injection in a dataset config. Together they gave the attacker execution inside production, and credentials followed. The full chain is broken down in AI data pipeline security.
Hugging Face published a technical timeline on July 27, 2026 reconstructing roughly 17,600 attacker actions between July 9 and July 13, 2026. It found no evidence of tampering with public models, datasets, or Spaces, and verified its software supply chain clean against expected digests.
OpenAI published a post-mortem on July 21, 2026 saying the activity came from its own models running an internal cyber-capability evaluation, the ExploitGym benchmark, with production safety classifiers disabled and cyber refusals reduced. In a July 28, 2026 update, OpenAI said the models escaped the evaluation sandbox through a zero-day in an Artifactory package registry cache proxy. BleepingComputer and TechCrunch covered the incident on July 20, 2026, Axios on July 21, and CNBC on July 22.
The lesson generalizes past this one bug class. A dataset is an active input to a system that will parse it, execute against it, or learn from it. Treating datasets as inert files you download and consume is the assumption that makes both this attack and classical poisoning work.
How much poisoned data does an attacker actually need?
Less than most teams assume. Research published in October 2025 by Anthropic with the UK AI Security Institute and the Alan Turing Institute found that roughly 250 poisoned documents were enough to install a backdoor across a wide range of model and dataset sizes, with the number staying near-constant as models grew.
The finding matters because of what it replaced. The working assumption in most threat models was that an attacker needed a percentage of the corpus, which meant poisoning got harder as datasets grew. If the requirement is a fixed count instead, scale stops protecting you.
Anthropic's write-up, A small number of samples can poison LLMs of any size, describes training 72 models across sizes and configurations. A 13B-parameter model trained on more than 20 times the data of a 600M model was backdoored by the same number of documents. The Alan Turing Institute published its own summary of the joint work.
Even with that caveat, 250 documents is trivially achievable. Anyone who can publish 250 web pages, submit 250 support tickets, or contribute 250 records to a shared dataset is inside the threat model.
What detection and prevention actually work, and where do they belong?
Controls work when they sit where data enters the system rather than where the model answers. Provenance and integrity checks at ingestion, transformation of the data before it reaches training or an index, held-out trigger testing before release, and drift monitoring after deployment together cover most of the attack surface.
At ingestion
This is where the controls do the most work. Record provenance for every dataset and document entering a pipeline, pin content to cryptographic digests rather than mutable URLs, and re-verify at fetch time. Carlini's split-view attack works specifically because the content behind a URL can change after it was indexed.
Before training or indexing
Deduplication, near-duplicate collapse, and canonicalization reduce the effect of injected repetition, which many poisoning attacks depend on. Outlier detection over embeddings can surface clusters of anomalous samples, though clean-label attacks are built to sit inside the normal distribution.
This stage is also where sensitive content has to be handled, because anything reaching training gets encoded into weights permanently, which is the mechanism behind training data leakage. Transformation at this layer changes the data so sensitive content can't be encoded into model weights or surfaced through retrieval, while the data stays useful for the task. That's the layer Hardshell operates on, model-agnostic, upstream of training, fine-tuning, and RAG pipelines, and running inside the customer's own environment.
Before release
Test for triggers you didn't put there. Hold out a validation set the training team never sees, evaluate on rare-token and out-of-distribution inputs, and compare behavior against the previous checkpoint on the same probes. None of that proves the absence of a backdoor, since you would have to guess the trigger, and it does catch availability-class degradation reliably.
Version and sign checkpoints so you can attribute a behavior change to a specific training run and a specific dataset snapshot. Without that, incident response after suspected poisoning is guesswork.
After deployment
Monitor behavior drift against a fixed probe set on a schedule, alerting on change rather than on an absolute score. For RAG systems, log which documents were retrieved for which answers, because that log is how you trace a bad answer back to a poisoned source document.
Retrieval poisoning has one real advantage over training poisoning. It's reversible. Correct the source document and reindex, and the behavior goes away without retraining.
Where other tools fit
Guardrails, DSPM, model scanning, and runtime monitoring all cover real ground, and none of them sit at the ingestion point. Guardrails inspect prompts and responses after the model already learned what it learned. DSPM inventories where sensitive data lives without governing what gets handed to a training job. Model scanners catch unsafe serialization in artifacts, which addresses model poisoning more than data poisoning. Runtime monitoring tells you something changed after it changed.
All of them are worth running. They don't substitute for controls on the data going in.
What do standards and regulators require?
The EU AI Act names data poisoning directly. Article 15 of Regulation (EU) 2024/1689 requires high-risk systems to include measures that prevent, detect, respond to, resolve and control attacks manipulating the training data set. NIST, ISO, OWASP and MITRE all cover poisoning too, though none of them are binding law.
Article 15 is the binding one. The regulation entered into force on August 1, 2024, and Article 15 requires high-risk AI systems to be designed for an appropriate level of accuracy, robustness and cybersecurity, performing consistently across their lifecycle. Its technical solutions clause names attacks on the training data set as data poisoning, alongside model poisoning, adversarial examples, and confidentiality attacks.
Note the verbs. Prevent, detect, respond to, resolve, control. A policy document stating that you take data quality seriously doesn't satisfy that list. It implies pipeline controls and evidence that they ran. The compliance clock moved but did not stop. Under the Digital Omnibus on AI, approved by the Council on June 29, 2026, standalone high-risk systems under Annex III now apply from December 2, 2027 rather than August 2, 2026.
The NIST AI Risk Management Framework (AI RMF 1.0), published January 26, 2023, is voluntary and organizes work into Govern, Map, Measure and Manage. It's the operating model most US organizations use to structure AI risk work, and poisoning controls land in Measure and Manage. NIST AI 100-2e2025 supplies the technical taxonomy underneath it.
ISO/IEC 42001:2023, published in December 2023, is the certifiable one. It specifies requirements for an AI management system, which gives you an auditable structure for data provenance, supplier management, and lifecycle controls. Plenty of organizations run AI RMF as the risk model inside a 42001 management system.
Two more references are worth knowing. OWASP's Top 10 for LLM Applications 2025 lists LLM04 Data and Model Poisoning, broadening the earlier training-data-only entry to cover fine-tuning and embedding stages. MITRE ATLAS maps adversarial machine learning techniques, poisoning included, to documented case studies, which helps when you need to describe the threat in language a security team already uses.
The common thread across all of them is provenance. Each one asks where your data came from, who could modify it in transit, and whether you can prove the version that trained your model is the version you approved.
Frequently asked questions
Is data poisoning actually happening, or is it mostly research?
Both. Most published poisoning work is academic, and there's no large public catalog of confirmed production poisoning incidents. But Carlini and colleagues showed in 2023 that poisoning real web-scale datasets was practical and cheap, and the July 2026 Hugging Face incident showed a malicious dataset reaching a production processing worker. The preconditions are already present in ordinary pipelines.
Can you detect a backdoor by testing the model before release?
Only if you guess the trigger. Backdoored models behave normally on ordinary inputs by design, which was the point of the BadNets work Gu, Dolan-Gavitt and Garg published in 2017. Standard benchmarks and red team prompts will pass. Detection that works tends to examine the training data and its provenance rather than model outputs alone.
Does using RAG instead of fine-tuning avoid poisoning risk?
No, it moves it. A retrieval corpus is a live dataset the model consults at query time, so poisoned content in a wiki page, a ticket, or a shared drive can shape an answer immediately without any retraining. The upside is that retrieval poisoning is reversible once you find and correct the source document.
What is the difference between data poisoning and model poisoning?
Data poisoning corrupts the data a model learns from. Model poisoning tampers with the model itself, including weights, checkpoints, or the serialized file format. Article 15 of the EU AI Act lists both, and OWASP merged them into a single entry, LLM04 Data and Model Poisoning, in its 2025 Top 10 for LLM Applications.
Do we inherit poisoning risk from an open-weight model we fine-tune?
Yes. A backdoor installed during pretraining survives fine-tuning in many setups, and you have no visibility into the base model's training data. Pin model versions to a known digest, prefer publishers with documented data provenance, and scan model artifacts for unsafe serialization before loading them. None of that proves the weights are clean.
Which standards and regulations name data poisoning explicitly?
Article 15 of the EU AI Act, Regulation (EU) 2024/1689, which entered into force on August 1, 2024, names data poisoning and model poisoning as attacks high-risk systems must address. NIST AI 100-2e2025 taxonomizes poisoning in detail, OWASP's 2025 LLM Top 10 lists it as LLM04, and MITRE ATLAS maps poisoning techniques to real cases.