Guides › Build a secure knowledge base
How to Build a Secure AI Knowledge Base, Step by Step
An AI knowledge base is whatever corpus your assistant retrieves from at query time, and most of them are built in an afternoon: point an indexer at a document store, embed the chunks, wire the retriever to a model. The security work is what that afternoon skips. This page is the build procedure for a knowledge base over sensitive data that enforces the same access decisions its sources do: nine steps from choosing sources through permission metadata, index placement, query-time entitlements, identifier substitution and retrieval telemetry, ending with the leakage test that decides whether it ships. It applies whether the retriever is a vector index you run, a vendor connector, or an enterprise search product.
What you are building, and the one property it has to have
A knowledge base has three moving parts: an ingestion path that reads sources and writes chunks into an index, a retrieval path that turns a user's question into a set of chunks, and a model boundary where those chunks become context for an answer. Every leak the AI knowledge base security guide describes happens because one of those three parts made an access decision the source system never would have: the indexer read as an account that could see everything, the retriever ranked by similarity and never asked who was asking, or the boundary sent real identifiers to a model someone else runs.
So the property the build has to deliver is simple to state. For every query, the set of chunks that reaches the model is a subset of what the asking user could open in the source systems today, and the organization can prove it afterward. Everything below is in service of that sentence. The reference architecture page draws the components and trust boundaries; this page is the order of operations.
The build, in nine steps
The steps run in the order that catches problems before the index does. Steps one through three are decisions about sources and identity that no tooling can make for you. Steps four through six are the index and the retriever. Steps seven through nine are the boundary, the evidence and the gate. The summary is below; each step is expanded after it.
1. Choose sources by obligation, not by convenience
The first knowledge base most teams build indexes whatever was easiest to reach, which is usually the SharePoint tenant or the wiki. Start instead from what each candidate source contains and what that obliges you to do: personal data under GDPR or state privacy law, health information under HIPAA, financial records under GLBA, controlled unclassified information under NIST SP 800-171 and CMMC, export-controlled technical data, and secrets. A source in any of the last four categories needs steps seven and eight before it is connected to a third-party model, and credentials, keys and connection strings should be stripped from any source before it is indexed at all. Write the classification down; it decides every later choice.
2. Fix the source permissions before you index them
An index inherits the permissions of its sources at the moment it is built, and in most tenants those permissions grant far more than anyone intended. Sample real user identities across roles and enumerate what each can open in each source. For SharePoint, SharePoint Advanced Management reports oversharing by site. For drives and wikis, pull the sharing defaults. For applications reached by API, compare what the API returns for a role against what the screen shows that role. Remediate before indexing, because an index built over over-permissioned sources is a copy of the problem with faster retrieval. The data silos guide explains why this gap is as large as it is.
3. Decide the ingestion identity, and what it implies
There are two ways to read sources into an index. A service account with broad read access crawls everything once; it is efficient and it means the index holds content no single user could see, so every chunk must carry the source's access control list and the retriever must enforce it on every query. The alternative, per-user or per-group ingestion, builds narrower indexes that are expensive to maintain and still drift. Nearly every production system uses the service account, which is fine, provided step four is done properly. What is not fine is the service account without the metadata, which is the configuration the does RAG leak data guide finds most often.
4. Carry permission metadata with every chunk
Each chunk written to the index carries the document's access control list (principals or groups that may read it), its sensitivity label if the source has one, its source system and path, and the timestamp of the permission snapshot. Chunks inherit from their document; where a single document mixes sensitivity levels, which is common in contracts and clinical notes, chunk at the boundary and label each chunk separately. If a chunking skillset or pipeline sits between the source and the index, confirm the permission fields survive it; Azure AI Search, for example, notes that permission fields move to index projections when a skillset chunks the content and that chunk-level references are not filtered without that projection. Permission metadata that does not reach the index does not exist at query time.
5. Place the index inside your boundary and treat it as the corpus
The index is a second copy of every sensitive document in the sources, in a form that retrieval can reach in milliseconds. Keep it in an environment that matches the classification from step one, under your own keys, with tenancy that separates business units if their data must stay apart. Encryption at rest protects the disk and does nothing for retrieval leakage, as the vector database security guide covers, and embeddings are not one-way, as the embedding inversion guide covers. The practical rule is that the index deserves the same access control, logging and lifecycle as the most sensitive source it was built from.
6. Enforce entitlements at query time, as a pre-filter
On every query, resolve the asking user to their current principals, filter the candidate set to chunks whose access control list includes one of them, and only then rank by similarity. Three alternatives fail in known ways. Post-filtering, which retrieves by similarity and then removes disallowed chunks, leaks through the ranking and through retry logic. Index-time filtering by building per-group indexes drifts within weeks. Prompt-level instructions asking the model to withhold restricted content are not a control. The metadata from step four also has to be refreshed: permission changes in the source appear in results only after the index is resynced, and inherited changes in SharePoint typically need an explicit refresh. Set the resync interval from your tolerance for stale access, and log the lag. The RAG permissions guide covers the three positions a check can occupy in detail.
7. Substitute identifiers before the model boundary
If the model is run by a third party, replace real identifiers in the retrieved chunks, meaning names, account and record numbers, addresses and similar keys, with consistent stand-ins before the context leaves your environment, and restore the real values only for a reader who holds the entitlement. The model reasons over the record and the provider never holds the identifiers. This is distinct from redaction, which removes the information and degrades the answer; the redaction, masking, tokenization and substitution comparison explains the tradeoffs. For a self-hosted model the step is optional on the sovereignty question and still useful for the logs.
8. Instrument retrieval telemetry before the first user
Record every retrieval: which chunks, from which documents, for which user, in response to which question, with the identifiers replaced by opaque IDs so the log holds no sensitive content and can be kept. This is the only evidence that step six is working, the only way to answer “what has the assistant shown people” after the fact, and what an auditor will ask for. The provider's logs show prompts and app activity; they do not show which of your documents were returned. Open-source retrieval telemetry is enough to start and takes a few hours to wire in.
9. Measure leakage, set a threshold, and gate rollout on it
With the index built, the pre-filter live and telemetry on, build a ground-truth matrix of ten to twenty documents against five test identities, write twenty natural questions whose answers live in those documents, run every question as every identity, and count a run as leaking if it retrieved any document outside that identity's entitled set. The leakage self-test is the reproducible version. Set a threshold, fix permissions or scope until the number is under it, then widen the user base or add the next source. Re-run on every re-index, every permission model change and every new source, because each one changes what can be retrieved together.
Where each step lives, by architecture
The steps are the same whether you run the retriever or a vendor does. What changes is who does the work and where the evidence comes from. The table maps the nine steps onto the four shapes most organizations run.
Operating it after launch
A knowledge base is never finished, because its sources keep changing. Three routines keep the property from step zero true over time. Resync permission metadata on a schedule and alert when the lag exceeds what you set in step six. Re-run the leakage test on every structural change and quarterly regardless, and keep the number on the same chart as the usage metrics so it is seen. Review the telemetry for retrieval patterns that do not match the user population: one identity pulling many documents it rarely touched, or many identities retrieving the same restricted record, which is how extraction looks from the retrieval side. The RAG security checklist is the stage-by-stage version of this page for audits, and the posture assessment turns the record into a framework coverage map.
Where Hardshell sits
Hardshell runs steps six through nine as a product and supplies the metadata that step four needs. It connects to the systems where data already lives without migrating them, inherits their permissions and checks entitlements at query time as a pre-filter, substitutes stand-ins for real identifiers before content reaches a model provider, records every retrieval under opaque IDs, and measures leakage before rollout with findings mapped to MITRE ATLAS technique IDs. It works in front of an index you run, as a connector inside ChatGPT and Claude, and as a layer for agent and enterprise search systems, and deploys as a cloud service, in a private cloud or as an offline bundle.
Frequently asked questions
How long does building a secure knowledge base take?
Steps one, three, four and five are days of engineering. Step two is the long one and depends on the tenant; weeks of permission remediation is normal for a decade-old SharePoint estate, which is why the first index should cover one well-governed source rather than everything. Steps six through nine are days with purpose-built tooling and months without it.
Can I use a vendor connector instead of building an index?
Yes, and the native connectors from OpenAI, Anthropic and Microsoft handle step six by running as the user. They do not do step two for you, they give you less control over what is indexed and how long it is kept, and they do not produce step eight's evidence. The connection procedure covers that route.
Is a vector database required?
No. Keyword and hybrid indexes, graph indexes and enterprise search products are all knowledge bases in this sense, and the nine steps apply to each. GraphRAG has a leakage profile of its own, covered on the GraphRAG and agentic retrieval page.
What if the sources have no access control lists?
Then the knowledge base cannot enforce an entitlement it does not have, and step two becomes “define one.” The minimum is a sensitivity label per document and a mapping from labels to groups. Indexing an unlabeled source is a decision to make everything in it available to every user of the assistant, and should be recorded as one.
How do I prove the knowledge base is secure to an auditor?
With the record from steps one, two and nine and the telemetry from step eight: what was indexed and why, what the permissions were when it was indexed, the leakage number and its date, and a log of what was retrieved for whom. The framework crosswalk maps those artifacts to MITRE ATLAS, OWASP, NIST AI RMF, ISO/IEC 42001 and the EU AI Act.