Enterprise Evaluation Program

See what your AI is actually retrieving

Updated August 8, 2026

Most teams have internal data that would make their AI assistant genuinely useful, and no way to prove it won't hand the wrong document to the wrong person. Hardshell publishes a free, open-source telemetry client that answers that question with a number instead of an assurance. You run it yourself, on your own systems, and nothing you retrieve leaves your environment.

Get the client on GitHub Request an evaluation key

Apache-2.0Open source. Free to run and free to audit.

0Document content leaves your environment. Text is hashed locally.

3Integration points: register the corpus, record retrievals, read the report.

1Prompt, if you hand the setup to the AI assistant you already use.

The short version

The failure mode is quiet. Retrieval returns a document someone shouldn't see, the answer looks normal, and nothing logs it as an incident.
Telemetry makes it visible. The client records which documents your AI retrieved and for whom, then reports what is being reached and how often.
Setup is close to hands-off. Point your own Claude, ChatGPT, or Cursor at the repo and it wires the client in. Your engineer approves the changes rather than writing them.
You keep your data. Content is hashed locally and never transmitted. The client is Apache-2.0 with no dependencies outside the Python standard library.

The problem doesn't look like a problem

There's a specific way retrieval-augmented generation goes wrong. Someone asks the assistant a reasonable question. The assistant gives a reasonable answer. Part of that answer came from a document the person asking was never allowed to open.

No exploit ran. No alert fired. Traffic looked normal, and in most deployments there's no log entry that would tell you it happened. The first anyone hears about it is usually an employee mentioning something they shouldn't know, or an auditor asking a question nobody can answer.

The cause is structural rather than sloppy. Retrieval is a similarity search, and similarity doesn't respect an org chart. An indexing job that ran under a service account with broad access, a permission revoked in the source system but never reflected in the index, a file that was over-shared for years and only became findable once semantic search removed the obscurity protecting it. Any of those produce the same result. We break down the routes in Does RAG leak data? and RAG permissions.

What your logs show today

Standard application logging

The question that was asked
The answer that came back
Latency, tokens, and cost

What retrieval telemetry adds

The hardshell-telemetry client

Which documents were actually reached
Which identity reached them
How often, and whether the pattern looks like exfiltration

This is what's holding up the projects worth doing

The AI assistant that would actually save your team time is the one wired into the systems where the sensitive material lives. Clinical documentation. Case files and contracts. Claims history. Internal engineering and process knowledge. Those corpora are where the value sits, and they're the ones nobody will sign off on.

So the project stalls in one of two places. Either it never clears review, or it launches over a thin, sanitized slice of the corpus and answers so poorly that people quietly stop using it. Both cost the same thing, which is the year you spent not having it.

Compartmented data makes it sharper

The more structured your access model is, the worse retrieval fits it. Classified or compartmented programs. Per-client matter walls in a law firm. Minimum-necessary boundaries under HIPAA. Separate tenants for separate customers. In all of them the whole point is that most of the corpus is off limits to most of the people asking.

Retrieval flattens that structure by default. Every chunk in the index is equally reachable by every query unless something in the pipeline enforces otherwise, and a vector store doesn't inherit your access model just because the documents came from a system that had one.

You can't argue past that in a security review. Someone has to show what the system actually returns, to whom, and how often. That's a measurement problem before it's a security-control problem, which is why the telemetry comes first.

What the client does

hardshell-telemetry is an open-source Python client. It sits alongside the retrieval code you already have and does three things.

Register your corpusOnce per index build, tell the client which documents and chunks exist and how sensitive each one is. Sensitivity is your own scale and your own labels. Nothing is imposed on you.
Record what retrieval returnsAfter each query the client reports which chunks came back, how well they scored, and which user asked. The text itself is hashed locally, so document content never leaves your environment.
Read the reportPull back a document-access summary showing which documents are being reached, how often, and by whom. Compare that against who should be reaching them.

What you get out of it

A number you can put in front of a security review, a board, or an auditor. Across the retrievals you logged last month, these documents were returned to these classes of user, this often, and here are the exceptions.

Setting it up

The repository ships an agent guide and a bundled Claude integration skill, so you can hand the whole setup to an assistant your team already uses. Point your own Claude, ChatGPT, or Cursor at the repo and it installs the package, registers the corpus with sensitivity labels, and adds the recording calls where they belong. Your engineer reviews and approves the changes rather than writing them.

Read github.com/hardshellinc/hardshell-telemetry and install and wire this client into my RAG app.

From there it's two environment variables and a first run. Point it at a staging index, or at the sample corpus in the repo if you'd rather see what the report looks like before touching anything of your own. If your team would rather wire it in by hand, the full self-test procedure has the code and the manual version of the same measurement.

Scope

Run this against your own staging index or the sample stack in the repository. This is a self-evaluation. You install it, you run it, and you read your own results. Hardshell never touches your systems, and we don't ask you to point it at anyone else's.

Measuring is the first half

Telemetry gives you the rate, and if you sort the results by cause it tells you which problem you have. Fixing it is separate work, and most of it isn't in the AI layer.

If retrieval is ignoring entitlements, the fix is enforcing them as a pre-filter at query time rather than trimming results after the model has already seen them. If permissions in the index have drifted from the source system, the fix is in the sync. If the source system was over-shared long before anyone built an assistant on top of it, retrieval only made an existing problem findable, and the remediation belongs upstream. How to prevent RAG data leakage takes the controls in order, and the RAG security checklist is the stage-by-stage version.

Then there's the category none of those close. Material that's sensitive in itself, where the person asking is entitled to the answer but not to every field in the document that carries it. That's the part Hardshell works on. We scan datasets for leakage and poisoning risk, score what we find, and harden the data so it stays useful for retrieval and training without carrying the sensitive values through. It's a transform rather than a redaction, so the corpus keeps its signal. It's also model-agnostic and store-agnostic, so it works the same whether you're on ChatGPT, Claude, Copilot, Gemini, or something your team built.

Request an evaluation key

The client is open source and free to run on its own. The reporting endpoint runs through Hardshell's Enterprise Evaluation Program, which is where the scoped key comes in. Tell us roughly what you're running and we'll send one over along with setup notes for your vector store.

Got it. We'll send a scoped evaluation key and setup notes for your vector store from rag-security@hardshell.ai, usually within a business day.

Oops! Something went wrong while submitting the form.

We reply from rag-security@hardshell.ai. Evaluation keys are scoped and rate-limited to the evaluation endpoint. There's no sales sequence attached to this, and if you'd rather run the open-source client on its own, the repo works without us.