Guides › Zero data retention for RAG
Zero Data Retention for RAG and AI Knowledge Bases: What a ZDR Agreement Leaves Stored
A zero data retention agreement covers the model call. It does not cover the knowledge base. In a retrieval-augmented generation (RAG) system, the provider-side copies that last longest are uploaded files, hosted vector stores, stored conversation state and batch jobs, and OpenAI and Anthropic both exclude those from zero data retention (ZDR) because they work by storing data. A knowledge base can run under ZDR only when indexing, retrieval and conversation state stay on your side and the provider receives stateless requests. This page maps each component of a RAG pipeline to what the two providers' documentation says about it, then describes an architecture that fits inside a ZDR arrangement.
- Covered. The generation call (OpenAI Responses and Chat Completions, Anthropic Messages) and, on OpenAI, the embeddings call.
- Not covered. Hosted vector stores, file uploads, stored conversations and threads, batch jobs, code execution containers and connector calls. Each keeps data until deleted or for its own documented period.
- Briefly held. Prompt caches. OpenAI documents encrypted cache data on GPU-local storage that expires within 24 hours. Anthropic holds cache representations in memory for the cache lifetime.
- Unchanged. Retrieved chunks still cross to the provider on every query. ZDR limits how long they stay there, not whether they were sent.
Where a Knowledge Base Keeps Data
A RAG system touches a provider in more places than the chat call. Source documents are parsed and chunked. Chunks are sent to an embedding model and the vectors are written to an index. At query time the question is embedded, the index returns chunks, and the chunks travel with the question to a generation model. Around that sit conversation history, caches, evaluation runs and logs. Each step runs either in your environment or on the provider's platform, and each provider-side step has its own retention rule. ZDR changes the rule for some of them and leaves the others as they were.
The distinction the providers draw is between abuse monitoring logs and application state. OpenAI's documentation describes ZDR as excluding customer content from abuse monitoring logs, and notes that endpoints listed as ineligible may still store application state with ZDR enabled. Anthropic's wording is similar: features marked ineligible are stateful, nothing blocks you from calling them, and using one is a choice to step outside the arrangement for that data. The ZDR explainer covers the arrangement in general. This page applies it to retrieval.
Component by Component
The table paraphrases each provider's API documentation as of November 2026. Eligibility lists change with new features and models, so check the linked sources against the endpoints your pipeline actually calls.
| RAG component | OpenAI API | Anthropic API |
|---|---|---|
| Generation call | Responses and Chat Completions are ZDR eligible. Under ZDR the store parameter is always treated as false. | The Messages API is ZDR eligible, including citations, search result blocks and PDFs sent inline. |
| Embedding call | The embeddings endpoint is ZDR eligible. | No embeddings endpoint appears in Anthropic's eligibility table. Embeddings come from a separate model, under that provider's terms. |
| Hosted vector store | Vector stores are not ZDR eligible and are retained until deleted. | No hosted vector store is listed. Retrieval runs on your side and results are passed in the request. |
| File uploads | The files endpoint is not ZDR eligible. Files persist until deleted or until an expiry you set. | The Files API is not ZDR eligible. Files are retained until deleted or until their configured expiration. |
| Conversation state | The Conversations API and Assistants threads are retained until deleted and are not ZDR eligible. | Message threads are not ZDR eligible; contents are held in memory and deleted within an hour of last use. Managed Agents sessions persist until deleted. |
| Batch processing | The batches endpoint is not ZDR eligible. | Batch processing is not ZDR eligible, with 29-day retention. |
| Prompt caching | Allowed under ZDR. Encrypted cache data sits on GPU-local storage and is not retained after a 24-hour expiration. | ZDR eligible. Cache representations and hashes are held in memory for the cache lifetime, then deleted. |
| Code execution | Hosted containers may write temporary state, deleted when the container expires or is removed. | Code execution is not ZDR eligible. Container data is retained up to 30 days. |
| Connectors and MCP | Data sent to a remote MCP server is subject to that server's retention policy. | The MCP connector is not ZDR eligible. Data is retained under the standard policy. |
| Models | OpenAI reserves the right to make specific models ineligible for a customer, with advance written notice. | Covered Models (Fable 5 through 5.2 and Mythos 5 through 5.2) require 30-day retention and are outside ZDR unless Anthropic authorizes otherwise. |
Two exceptions apply with ZDR in place. Anthropic states that inputs and outputs flagged by its trust and safety systems may be kept for up to two years, and OpenAI retains image and file inputs that its classifier flags as potential child sexual abuse material for manual review.
A Retrieval Architecture That Fits Inside ZDR
The pattern that holds up is to keep every stateful part of the knowledge base in your environment and use the provider for stateless inference only.
- Keep the index on your side.Run the vector store in your own cloud account or data center. A hosted vector store or file search tool keeps the corpus on the provider until you delete it.
- Embed through an eligible endpoint or a model you host.On OpenAI the embeddings endpoint is ZDR eligible. Otherwise host the embedding model yourself so document text is not sent out at indexing time.
- Filter by entitlement before retrieval returns.Apply the asking user's permissions when the index is queried, so chunks they cannot open are never candidates for the prompt.
- Send retrieved chunks inline in a stateless request.Pass context in the request body. Do not upload the documents as files, and do not rely on stored conversation objects.
- Hold conversation history yourself.Store the transcript in your application and resend what each turn needs.
- Keep sensitive corpora off batch and code execution.Both store inputs and outputs for the life of the job or container.
- Check your own logs.Gateways, tracing and evaluation tools often capture full prompts with the retrieved chunks inside. Record retrievals under opaque identifiers instead of content.
The secure RAG architecture guide covers the full design, vector database security covers the index, RAG permissions covers entitlement checks, and the telemetry page covers the retrieval record.
What ZDR Does Not Change for a Knowledge Base
ZDR is a statement about persistence. Every query still sends the retrieved chunks to the provider, where they are decrypted, read by the model and screened by safety classifiers. If a chunk holds a patient name, an account number or controlled technical data, that content was disclosed to a third party for the duration of the request, with or without ZDR. For most regulatory tests, disclosure is the event that matters.
ZDR also removes the provider's record of what was sent. That reduces exposure, and it leaves you as the only party able to say what crossed, when and for whom, which is why a knowledge base under ZDR needs its own retrieval record. And ZDR has no bearing on which chunks were retrieved. An index built over an over-shared document library returns over-shared content under any retention setting, as the page on whether RAG leaks data describes.
The control that changes the disclosure answer is substitution before the boundary. Real identifiers in the retrieved context are replaced with stand-ins, the model reasons over the stand-ins, and the real values are restored only for readers entitled to them. The comparison of redaction, masking, tokenization and substitution covers the trade-offs.
On the Cloud Platforms
The same split applies when the model is reached through a cloud platform, with different mechanics. Amazon Bedrock exposes a data retention mode that can be set to none per Region, and a Bedrock knowledge base keeps its vectors either in a store you manage or in a managed store you can encrypt with your own key. Azure OpenAI's On Your Data feature leaves the data in the source you designate and does not create a duplicate store, while files, vector stores and stored responses persist in your Foundry resource. Google Cloud publishes a list of features to disable, including a prompt cache that is on by default. The cloud platform comparison has the detail.
Where Hardshell Sits
Hardshell is built for the part of the pipeline ZDR leaves open. It connects to the systems where enterprise data already lives without migrating it into a provider-hosted store, enforces each user's existing permissions at query time, substitutes stand-ins for real identifiers in retrieved content before it reaches ChatGPT, Claude, Microsoft 365 Copilot, Gemini or any model API, and records every retrieval under opaque IDs. Used with a ZDR arrangement, the provider keeps nothing after the response, and what it saw during the request did not contain the real identifiers. Hardshell deploys as a cloud service, in a private cloud or as an offline bundle.
Frequently Asked Questions
Does zero data retention cover my vector database?
Only if the vector database is yours. ZDR applies to eligible API calls. A vector store hosted by the provider is application state: OpenAI lists its vector stores endpoint as not ZDR eligible and retains the contents until you delete them.
Is OpenAI file search covered by zero data retention?
Not for the stored material. OpenAI lists files and vector stores as ineligible for ZDR and retains them until deleted, so a knowledge base built on hosted file search keeps its corpus on OpenAI's platform even when the organization has ZDR.
Are embeddings covered by ZDR?
On OpenAI, yes. The embeddings endpoint is listed as ZDR eligible, so document text sent for embedding is excluded from abuse monitoring logs. The vectors you get back are stored wherever you put them, and embeddings can reveal source text, so the index needs its own protection.
Does prompt caching break zero data retention?
Not according to either provider. OpenAI documents encrypted cache data on GPU-local storage that is not retained after a 24-hour expiration, and Anthropic lists prompt caching as ZDR eligible with cache representations held in memory for the cache lifetime. If your requirement is no storage of any kind, treat caching as a decision to make, because retrieved context is what gets cached.
Can I use batch processing for a knowledge base under ZDR?
Not within the arrangement. Both providers list batch as ineligible because jobs store inputs and outputs, and Anthropic documents 29-day retention. Batch-embedding or batch-summarizing a sensitive corpus puts that corpus outside ZDR for the life of the job.
Does ZDR mean retrieved documents never leave our environment?
No. Retrieved chunks are sent to the provider with every query and processed there. ZDR means no copy is kept at rest afterward. Keeping content from leaving requires a model in your own environment, and keeping identifiers from leaving requires substitution before the request is sent.
Related guides
- Zero data retention explained · What ZDR covers, what it leaves out, and how to verify it.
- Secure RAG architecture · Where controls belong in a retrieval pipeline over sensitive data.
- How to get zero data retention · Eligibility, request steps and verification on each provider.
Sources (accessed November 2026)
OpenAI, Data controls in the OpenAI platform. Anthropic, API and data retention. Microsoft Learn, Data, privacy, and security for Azure Direct Models in Microsoft Foundry. AWS, Amazon Bedrock data retention; Encryption of knowledge base resources. Google Cloud, Gemini Enterprise Agent Platform and zero data retention.