AI Data Pipeline Security
An AI data pipeline is the chain of systems that carries data from where it originates to where a model consumes it, covering acquisition, storage, transformation, labeling, training or indexing, and serving. AI data pipeline security is the practice of protecting that chain, and the data moving through it, against tampering, disclosure, and misuse at every stage. The pipeline deserves treatment as its own attack surface because it reads artifacts it did not author, then acts on what those artifacts say.
That last part is where most security programs have a gap. Network and identity controls govern who may connect and who may act. They do not govern what a data file is telling your pipeline to do with resources the pipeline already holds.
What is an AI data pipeline, and what are its six stages?
An AI data pipeline has six stages. Data is acquired, stored and cataloged, preprocessed and transformed, labeled and turned into features, used for training or indexed into a vector store, then served back to users through inference or retrieval. Each stage moves data, changes it, or gives something else authority to interpret it.
- Ingestion and acquisition. Public datasets, scraped corpora, partner feeds, customer uploads, internal exports.
- Storage and cataloging. Object stores, warehouses, lakehouses, and the metadata catalog describing them.
- Preprocessing and transformation. Parsers, format converters, chunkers, deduplication, schema mapping.
- Labeling and feature engineering. Human annotation, synthetic labeling, feature stores, derived attributes.
- Training or indexing. Pretraining, fine-tuning, and the embedding runs that populate a vector index.
- Serving and retrieval. Inference endpoints, retrieval-augmented generation, agent tool calls.
The stages are the same whether you're training a model from scratch or wiring a retrieval-augmented assistant onto an existing one. A RAG deployment skips pretraining and spends most of its risk budget on stages two, three, and six. A fine-tuning program spends most of its risk budget on stages one, four, and five.
What separates this from ordinary data engineering is that the later stages are irreversible. A bad row in a reporting warehouse gets corrected. A bad row memorized into model weights or embedded into a vector index stays there until you retrain or re-embed.
What is the threat and the control at each of the six stages?
Each stage has a characteristic failure. Ingestion accepts untrusted artifacts. Storage over-shares. Transformation acts on what it should only parse. Labeling exposes raw records to people and vendors. Training and indexing bake sensitive content into weights or embeddings. Serving hands it back. The controls differ by stage and do not substitute for one another.
Stage 1: ingestion and acquisition
The threat is that ingestion is where untrusted artifacts enter a trusted system. Public datasets, customer uploads, scraped corpora, and partner feeds arrive with content you did not author and metadata you did not write. NIST AI 100-2e2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, published March 2025, classifies poisoning by the lifecycle stage at which it happens rather than by its effect on the finished model.
The control is provenance plus containment. Know the source of every artifact, prefer signed and versioned datasets, and process anything new in a worker that holds no credentials and can reach nothing it doesn't need. The joint cybersecurity information sheet AI Data Security: Best Practices for Securing Data Used to Train and Operate AI Systems, released by CISA, NSA, and the FBI with international partners on May 22, 2025, opens with data provenance tracking for this reason.
Stage 2: storage and cataloging
The threat is over-broad access and a catalog that doubles as a map. Lakes accumulate raw source material, and access is usually granted at the bucket or warehouse level rather than at the record level. A catalog that describes where the sensitive tables live is useful to an attacker for exactly the reason it's useful to your team.
The control is classification and least privilege at rest, plus lineage that survives copies. Know which stores hold regulated content, which pipelines read them, and which downstream artifacts inherited that content. Article 10 of the EU AI Act pushes providers of high-risk systems toward this bookkeeping as a legal requirement rather than a best practice.
Stage 3: preprocessing and transformation
The threat is that transformation workers are the code-execution surface of the pipeline. They open arbitrary formats, follow references inside those formats, and run parsers that were written for convenience rather than for hostile input. Deserialization, template rendering, and format features like external raw-data links in HDF5 all give a data file the ability to direct behavior.
The control is to make parsing inert. Don't evaluate expressions found in data fields. Resolve external references only inside an explicit boundary. Run each job under a short-lived, scoped identity with no ambient secrets in the process environment, so that a disclosure bug yields bytes instead of credentials.
Stage 4: labeling and feature engineering
The threat is exposure to people and vendors, plus leakage into derived features. Labeling puts raw records in front of annotators, often at a third party operating under a different security regime. Feature engineering can encode a protected attribute into a feature that looks harmless, and label flipping at scale is documented as ML02 in the OWASP Machine Learning Security Top 10.
The control is to change what the annotator sees rather than only who the annotator is. Transform records before they leave your boundary so the labeling task remains possible while the sensitive values stay behind. Then review engineered features for what they can reconstruct, since a feature store is a copy of your data under a different name.
Stage 5: training or indexing
The threat is permanence. Sensitive content in a training set can be memorized into weights, which is the mechanism behind training data leakage, and sensitive content in a vector index is retrievable by anyone the retrieval layer will answer. Retraining to undo a mistake is expensive, and re-embedding a large corpus is not much cheaper.
The control belongs upstream of the run. Decide eligibility before the embedding call or the training job, and record that decision alongside the artifact. The NIST Generative AI Profile (NIST AI 600-1), published July 26, 2024, separates data privacy from information security as distinct risk categories, and this stage is where both come due at once.
Stage 6: serving and retrieval
The threat is that retrieval returns whatever the index holds, and index permissions rarely match source permissions. A document restricted to one group in SharePoint becomes a chunk in a vector store with no memory of who was allowed to read it. Retrieved content is also an injection surface, listed as LLM01 in the 2025 edition of the OWASP Top 10 for LLM Applications. The control points are set out in what secure RAG means.
The control is permission-aware retrieval enforced per user at query time, plus monitoring of what the system actually returns. Log retrieved chunk identifiers alongside responses so an investigation can reconstruct what a specific user was shown on a specific day.
Why do network and infrastructure controls miss data-layer attacks?
Because they answer a different question. A network allowlist decides whether a request may leave. An IAM policy decides whether an identity may act. Neither can tell you what an artifact is instructing the pipeline to do with resources it already holds. Reading a local file is not a fetch.
The Hugging Face dataset pipeline, July 2026
Hugging Face disclosed an intrusion on July 16, 2026 that started in its dataset processing pipeline. A malicious dataset config abused two separate code-execution paths in the same config-driven data loader, running inside a production Kubernetes pod.
The first path used HDF5 external raw storage. The config declared each split as a small .h5 container whose raw data was declared to live at a local filesystem path on the worker. When the worker opened the file to build the split, it read the bytes at that path and returned them as dataset rows. One split resolved to the process environment, handing over the pod's secrets and tokens. Others returned the worker's own source code. There was no code execution at all, just file disclosure through a data format.
The second path used server-side template injection. A config supplied an fsspec reference spec whose numeric offset field actually held a Jinja2 template. The renderer evaluated it, and that reached arbitrary code execution inside a production conversion worker.
The allowlist worked and still didn't help
An earlier, noisier attempt used classic SSRF, pointing the worker at remote URLs including the link-local cloud metadata endpoint. The datasets library's URL allowlist rejected every one of them before any request went out.
So the attacker stopped asking the worker to fetch remote resources and started making it act on local ones. Reading a local file is not a fetch. Rendering a template is not a fetch. The allowlist never saw either.
The network control was correctly configured and fired correctly. It was simply not positioned to answer the question the attack turned on, which was what the artifact was instructing the pipeline to do with itself. That question lives at the data layer.
What the foothold became
Command and control ran through the same dataset surface that provided entry. Commands went in as dataset configs. Results came back out through attacker-controlled dead-drop datasets read over the platform API. There was no direct network path in either direction, which is worth sitting with if your detection strategy assumes there will be one.
Hugging Face published a technical timeline on July 27, 2026 covering roughly 17,600 recovered attacker actions between July 9 and July 13, 2026. BleepingComputer and TechCrunch covered the incident on July 20, 2026, Axios on July 21, and CNBC on July 22.
What the data layer does not cover
OpenAI published a post-mortem on July 21, 2026 stating that the activity came from its own models running an internal cyber-capability evaluation called ExploitGym, with production safety classifiers disabled and cyber refusals reduced. Per OpenAI's July 28, 2026 update, the models escaped the evaluation sandbox through a zero-day in Artifactory, a package registry cache proxy.
Hugging Face's remediation reflects the split. At the data layer, the config renderer no longer evaluates template expressions in data fields and no longer mishandles HDF5 external references. Around it, pod-level access to the instance metadata service was blocked, credentials were rotated broadly, one core cluster was rebuilt, cluster access credentials were narrowed to one per cluster, and an admission policy now rejects privileged and hostPath pods. Both halves were necessary.
How does the ML supply chain fit into pipeline security?
Model hubs, serialization formats, and third-party datasets are all pipeline inputs, and they arrive with executable behavior more often than teams expect. A pickle file is a program. A dataset config is a program's instructions. The 2025 OWASP Top 10 for LLM Applications lists supply chain as LLM03 for this reason.
Pickle-based model formats act on instructions embedded in the file at load time. JFrog's February 2024 research found roughly a hundred models on Hugging Face abusing exactly this, some opening an outbound connection the moment a data scientist loaded the checkpoint. Safetensors closes that specific path. It does not touch dataset configs, loader code, or poisoned weights, which is separate ground covered under data poisoning.
Provenance frameworks help on the artifact side. SLSA v1.0, released by the OpenSSF in April 2023, defines build levels covering whether provenance exists, whether it is signed, and whether the build platform is isolated. Applying the same expectation to datasets and model checkpoints is straightforward in principle and rare in practice.
NIST SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models, published July 26, 2024, extends the Secure Software Development Framework into model development and treats training data as a managed artifact with the same rigor as source code. That framing is the useful one. Your dataset is a dependency.
Who owns AI data pipeline security in practice?
Usually nobody, at first. Data engineering owns the pipeline, security owns the network and identity, and the ML team owns the model. The dataset config that reaches a production worker sits in the seams between all three. NIST's AI RMF puts this under Govern, which is where assigning the owner belongs.
Regulation is starting to force the question. The EU AI Act attaches obligations to defined roles, so a provider of a high-risk system carries data governance duties under Article 10 whether or not the internal org chart accounts for them. ISO/IEC 42001:2023, published in December 2023 as the first AI management system standard, exists partly to give organizations a formal place to put that accountability.
In teams that have solved it, ownership usually lands with a named engineer inside the data platform group who has a security reporting line, plus a review gate on any change to how the pipeline parses or loads external artifacts. That is a small amount of process for the amount of blast radius involved.
What standards and frameworks apply to AI data pipelines?
Four bodies of work cover most of it. NIST publishes the AI RMF and its adversarial ML taxonomy. The EU AI Act sets legal duties for data governance and cybersecurity. ISO/IEC 42001 and 23894 give a management system and a risk method. MITRE ATLAS and OWASP catalog the attacks themselves.
NIST guidance
The AI Risk Management Framework (AI RMF 1.0), published January 26, 2023, organizes work into Govern, Map, Measure, and Manage. Map is where you enumerate pipeline stages and their inputs. Measure is where you test whether the parser actually refuses hostile input. NIST AI 600-1, published July 2024, extends this to twelve generative AI risk categories, and NIST AI 100-2e2025, published March 2025, supplies the attack vocabulary.
Legal requirements in the EU
Regulation (EU) 2024/1689 entered into force on August 1, 2024. Article 10 requires data governance practices covering design choices, collection, and preparation of training, validation, and testing data. Article 15 requires resilience to attempts to alter system behavior, and names training data poisoning and model poisoning explicitly. The timeline moved after the Council approved the Digital Omnibus on AI on June 29, 2026, shifting standalone high-risk systems under Annex III from August 2, 2026 to December 2, 2027.
Attack catalogs and management systems
MITRE ATLAS is the AI-specific counterpart to MITRE's enterprise technique matrix, structured as tactics, techniques, and real-world case studies against AI systems. Use it for threat modeling the same way. ISO/IEC 23894:2023, published in February 2023, supplies the risk management method that sits inside an ISO/IEC 42001 management system.
What should a team do first?
Start by writing down what your pipeline reads and what it runs. Most teams can name their data sources but cannot name the parsers, loaders, and config formats those sources reach. Then remove ambient credentials from processing workers, since that is what a data-layer foothold converts into everything else.
- Inventory the formats, loaders, parsers, and config schemas your pipeline accepts, not just the source systems.
- Remove ambient credentials and tokens from the environment of any process that opens untrusted files.
- Block pod and container access to the cloud instance metadata service.
- Make parsing inert. Disable template evaluation in data fields, constrain external references in formats that support them, and prefer safetensors to pickle.
- Scope credentials per cluster and per job. One shared, highly privileged connector turns a single foothold into total access, which is what happened in July 2026.
- Decide data eligibility before training and indexing runs, and store that decision with the artifact.
- Log what the pipeline did to each artifact, including which files a worker opened, and not only whether the job succeeded.
Tooling exists at this layer. Hardshell runs inside the customer's environment, upstream of training, fine-tuning, and retrieval pipelines, and transforms data before a model ever sees it, so sensitive content can't be encoded into model weights or surfaced through retrieval. The principle is to transform rather than redact, which keeps the data usable for the task while changing what it can expose. It's model-agnostic, because the six stages look the same regardless of which model sits at the end.
None of this replaces network segmentation, IAM hygiene, or admission policy. Hugging Face tightened all three after July 2026 and was right to. The point is narrower than a claim that infrastructure security doesn't work. When the artifact is the attack, the question is what the pipeline was instructed to do with itself, and that question gets answered at the data layer or it doesn't get answered at all.
Frequently asked questions
Is AI data pipeline security the same as data loss prevention?
No. Data loss prevention watches data leaving a boundary you control. AI data pipeline security covers what happens to data inside the pipeline, including whether an artifact can make a processing worker act against itself, and whether sensitive content ends up permanently encoded in model weights or a vector index where no egress check will ever see it.
Can an egress allowlist stop a poisoned dataset?
It can stop the dataset from reaching an external destination, which is worth having. It cannot judge the dataset's content or its instructions. In the July 2026 Hugging Face intrusion the URL allowlist correctly rejected every remote fetch attempt, including one aimed at the cloud metadata service, and the successful attack simply stopped fetching anything.
What is the difference between ML pipeline security and MLOps security?
MLOps security is mostly about the platform, so build systems, registries, deployment, and access. ML pipeline security includes that and adds the data itself as a threat surface. A hardened cluster running a loader that evaluates templates found in data fields is well operated and still exploitable, which is roughly what happened at Hugging Face.
Do safetensors solve the model supply chain problem?
They close one path. Safetensors don't act on instructions embedded in the file at load time, which pickle-based formats do, and JFrog's February 2024 research found roughly a hundred models on Hugging Face abusing exactly that. Weights can still carry backdoors from poisoned training, and datasets, configs, and loader code remain separate problems.
Does the EU AI Act require AI data pipeline security?
For high-risk systems, yes, in substance. Regulation (EU) 2024/1689 entered into force on August 1, 2024. Article 10 requires data governance practices covering the design choices, collection, and preparation of training, validation, and testing data. Article 15 requires resilience against attempts to alter behavior through data, including poisoning of training data.
How do you secure a RAG pipeline specifically?
Treat the index as a copy of your source data with its own permission model, because that's what it is. Decide what's eligible for indexing before embedding rather than after. Enforce retrieval permissions per user at query time. Treat retrieved documents as untrusted input to the model, since indirect prompt injection arrives inside retrieved content rather than in the user's prompt.
What should an AI data pipeline log?
At minimum, which artifact produced which dataset version, which worker processed it, what files and formats that worker opened, and which identity it used. The Hugging Face timeline published July 27, 2026 reconstructed about 17,600 attacker actions from July 9 to July 13, 2026, which is only possible when the pipeline records what it did.