Guides › Secure RAG architecture

Secure RAG Architecture: A Reference Design for Sensitive Data

Most RAG architecture diagrams have four boxes: documents, an index, a retriever and a model. They are accurate and they are the reason retrieval systems leak, because none of the four boxes is where an access decision gets made. This page is a reference design for retrieval over sensitive data. It names the components, draws the three trust boundaries between them, puts a control at each boundary, and then shows how the design changes for agentic retrieval, GraphRAG, connector-based assistants and enterprise search. It is model-agnostic and store-agnostic by construction: the controls sit on the boundaries, so they survive a change of vector database or model provider.

The components

A retrieval system over sensitive data has nine components, five of which the four-box diagram leaves out. The sources are the systems of record: document stores, wikis, mail, line-of-business applications and databases, each with its own permission model. The ingestion path reads them under some identity and writes chunks, with metadata, into the index. The index is a second copy of the sources in retrieval-ready form, whether vector, keyword, hybrid or graph. The retriever turns a question into candidate chunks. The policy point decides which candidates the asking user is entitled to, and is the component most diagrams omit. The substitution layer replaces real identifiers in the surviving chunks with stand-ins before they leave your environment. The model consumes the context and produces an answer. The application presents it, and restores stand-ins for entitled readers. The telemetry store records what was retrieved, for whom, and what was restored.

The three trust boundaries

The components sit on either side of three boundaries, and each boundary is where one class of leak happens. The first is between sources and index: the ingestion identity usually sees more than any user, so the index holds content no single person should see in full, and the only thing that makes that acceptable is permission metadata travelling with every chunk. The second is between index and model context: the retriever's job is to find similar text, and similarity is not an access decision, so a policy point has to sit between the candidate set and the context window and check the asking user's current entitlements. The third is between your environment and the model provider: whatever crosses is processed and held under the provider's terms, which govern what they do with it and nothing about what you sent, so the substitution layer sits on this boundary. The AI knowledge base security guide describes the four failure modes; each one is a missing control on one of these boundaries.

The control points, in the order a query crosses them

Permission metadata on ingestion

Every chunk carries the document's access control list, its sensitivity label, its source and the time the permissions were read. Chunk at sensitivity boundaries inside mixed documents. Resync on a schedule tied to how fast permissions change in the source, and treat the lag as a measured quantity. Without this metadata the policy point has nothing to enforce, which is why the build procedure puts it before the index exists.

Entitlement pre-filter at the policy point

On each query, resolve the user to their current principals, restrict the candidate set to chunks whose access control list includes one of them, then rank. The filter runs before ranking, inside the retrieval call, against live entitlements. Post-filtering after ranking leaks through retries and through what the ranking reveals; per-group indexes drift; instructions to the model are not a control. The RAG permissions guide covers the three positions in detail.

Substitution at the provider boundary

Real identifiers in the surviving chunks are replaced by consistent stand-ins before the context is sent to a third-party model, and the mapping is held inside your environment. The model reasons about the record; the provider never holds the names, account numbers or record keys. Restoration happens in the application, only for a reader who holds the entitlement, and every restoration attempt is logged. The technique comparison explains why substitution rather than redaction.

Telemetry on every retrieval

The telemetry store records chunk and document identifiers, the asking principal, the question, the policy decision and any restoration, under opaque IDs so the log holds no sensitive content. This is the component that turns the design from a claim into evidence, and the one that provider-side logs cannot substitute for, because the provider never sees your document identifiers. It also feeds the test harness: the leakage self-test is a query set run against the telemetry.

Variants: what changes by retrieval architecture

The components and boundaries are the same across every retrieval architecture in production use. What changes is which component makes the retrieval decision and how many hops a query takes before it reaches the model. The table shows the four variants that differ from classic RAG and the control each one adds.

Anti-patterns the design rules out

Five configurations show up repeatedly in systems that leak, and each is a control placed on the wrong side of a boundary. Post-filtering after ranking puts the policy point downstream of the decision it is meant to govern. Per-group indexes move the entitlement check to ingestion time, where it is a snapshot that drifts. Prompt-level restrictions put the control inside the model, which is the one component that cannot be trusted to enforce it. An index hosted outside the data's classification boundary, often a managed vector service in a different tenancy, moves the second copy of the corpus across the third boundary before any query runs. And provider audit logs treated as the retrieval record confuse a log of prompts with a log of documents. The how to prevent RAG data leakage guide lists the seven controls that replace them in pipeline order.

What a reference design is not

It is not a product diagram and it does not prescribe a store, a model or a framework. A team can implement every control on this page with an open-source vector index, a self-hosted model and a few hundred lines of policy code, and many should. The design's claims are about where decisions are made, which is why it holds when the model changes from one provider to another, when the index changes from vectors to a graph, or when the retriever is replaced by an agent that decides what to fetch. The controls are on the boundaries; the boxes in between are replaceable.

Standards the design maps onto

The design answers specific entries in the frameworks most teams report against. In the OWASP Top 10 for LLM Applications 2026, the entitlement pre-filter and substitution layer address sensitive information disclosure, and the index placement and metadata address the vector and embedding weaknesses entry. In MITRE ATLAS, the telemetry store and test harness are what make the data-layer techniques detectable rather than theoretical, and findings from the harness carry technique IDs. NIST AI RMF's Map, Measure and Manage functions correspond to the classification, the leakage measurement and the operating routine. ISO/IEC 42001 asks for documented decisions, which the record from the build procedure supplies. The framework crosswalk has the full mapping by failure mode.

Where Hardshell sits

Hardshell is the policy point, the substitution layer, the telemetry store and the test harness of this design, delivered as one layer between an organization's sources and whichever models it uses. It connects to the systems where data already lives without migrating them, inherits their permissions and checks entitlements at query time, substitutes stand-ins for real identifiers before content reaches a model provider, records every retrieval under opaque IDs, and measures leakage before rollout with findings mapped to MITRE ATLAS technique IDs. It works in front of an index you run, as a connector inside ChatGPT and Claude, and as a layer for agent and enterprise search systems, and deploys as a cloud service, in a private cloud or as an offline bundle.

Frequently asked questions

Is a vector database with role-based access control enough?

Role-based access on the database governs who can query the index, which is the application's service identity. It does not govern which chunks a given end user may receive from a query the application makes on their behalf. That is the policy point, and it has to carry the end user's principals into the retrieval call.

Where should the policy point physically run?

Inside the retrieval call, before ranking, in your environment. Some managed search services expose it as a query-time permission filter; where the service's native option is a string filter or still in preview, a layer in front of the service that resolves principals and applies the filter is the reliable version.

Does the design change for a self-hosted model?

The third boundary disappears for inference, so substitution becomes optional on the sovereignty question. The first two boundaries are unchanged, and most leaks happen there. A self-hosted model over an unfiltered index leaks exactly as a hosted one does.

How is this different from an LLM guardrail?

A guardrail inspects text at the application edge, after retrieval has already decided what the model sees. The policy point decides what the model sees. The secure RAG vs guardrails page covers where each works.

What about caching and conversation memory?

Anything cached or remembered is a fourth copy of retrieved content and inherits the entitlement of the query that produced it. Cache keys should include the principal set, and memory should store stand-ins rather than restored values. Both are telemetry events.