Who Holds the Key? Three Decisions That Shape Multi-Tenant RAG Architecture

A standard claim in the RAG world goes: "Your original documents never leave your infrastructure." That is true — and still only half the truth.

In modern RAG architectures the original files do indeed stay with the customer. What travels to the provider are not the documents but their index records: small text fragments ("chunks") plus mathematical fingerprints ("embeddings"). Yet those chunks are exactly the content a RAG model returns in response to a query. Whoever reads them sees the content.

That shifts the privacy question from "are original files uploaded?" to "who can read the index, and under what conditions?". The question arises everywhere knowledge has to be organized on a need-to-know basis — and in many industries that is the normal state of affairs: separate mandates in a law firm, divisions in a corporate group, client files in a consultancy, patient cohorts in a hospital network, customer files at a service provider with multiple clients.

In all of these cases, three architecture decisions have to be made early.

Decision 1: How do we prevent one tenant from seeing another's data?

The most important privacy question in a multi-tenant system is not "can someone hack their way in?" but "can we prevent configuration errors that let tenant A's data surface in a response to tenant B?". This kind of leak — through internal sloppiness, not external attack — is by far the most common cause of real incidents.

Picture the RAG store as a library. There are two common approaches.

In the first, each tenant gets their own collection — their own drawer. Anyone reading or writing has to name explicitly which drawer they mean. The data model offers no way to "accidentally search all drawers at once". Separation is enforced by the system, not by developer discipline.

In the second approach, all tenants share a single collection; every record carries a marker stamp with the tenant number. Each query ships with a filter: "stamp = X only". This works as long as the filter is set correctly everywhere. Forget it in one place — a new endpoint, a debug path, a code change — and there is no second safety layer. The entire privacy guarantee hangs on a coding convention.

For use cases where knowledge genuinely has to be need-to-know, the choice is clear: a separate collection per tenant unit. The minimal scaling overhead outweighs the risk of a forgotten filter many times over.

Decision 2: Who can technically read the stored data?

Once tenant content sits on disk at the provider, the follow-up question is: who can actually read it, technically? Even if only the tenant is contractually entitled — an insider with database access, a misconfigured backup, the cloud host in the background — the raw bytes exist somewhere. Encryption closes that possibility, but only up to a clearly defined degree.

The standard mechanism is envelope encryption. Each record is encrypted with a data key; that data key is in turn encrypted with a second, higher-level key — the "master". The master is managed by a specialized key vault (the KMS, Key Management System). Whoever wants to read data needs the master. Whoever deletes the master makes all data encrypted with it unreadable — including any backups still lying around.

Issuing a separate master per tenant is therefore unusually useful: it enables crypto-shredding. When a tenant invokes their "right to be forgotten", the provider does not have to hunt down every individual backup copy — impossible in practice and never reliably provable under audit. Instead the tenant's master key is deleted, and all encrypted data becomes unreadable. A promise grounded in mathematics, not in the completeness of a search.

When it comes to who holds the key, there are three realistic levels.

Provider-managed. The provider runs the KMS itself, each tenant has their own master. The provider can rotate, roll, or in an emergency delete the key. But if it wanted to, or were legally compelled, it could also read the plaintext with its KMS access. This is not hidden — it is the very definition of the provider hosting the service. What the tenant gets is a contractual self-commitment (via a data processing agreement), not a technical impossibility.

Bring-Your-Own-Key (BYOK). The tenant supplies the key, typically in their own KMS (such as Google Cloud KMS or AWS KMS). The provider only calls it briefly to decrypt and never holds it itself. The tenant can lock the provider out at any time by revoking key access on their side. Stronger than provider-managed, because the provider never carries the "morally I could have accessed it" liability in the first place. The price is engineering effort: two to four weeks of additional build time, provided the base architecture is designed for it.

Operator-blind. The tenant encrypts everything before anything travels to the provider. The provider sees only ciphertext and cannot search it — because to find relevant matches, the system must understand somewhere what is stored. Operator-blind is therefore not something a centrally hosted service can realistically offer. It comes back only when the tenant hosts the entire stack itself.

From this follows an ordering rule: whoever holds the keys also hosts. There is no meaningful path where a customer holds the keys and the provider hosts centrally. In practice there are therefore two service models — a hosted one with provider or BYOK keys, and a sovereign one where the tenant runs the stack in their own environment.

Decision 3: How do tenants identify themselves to the service?

Every call to the service has to carry an answer to the question: Who am I, and am I allowed to do this?. In the API world the answer is called a token — a piece of secret text sent with each request and verified by the server. There are two common pattern families.

Long-lived API keys with a revocation list are the pragmatic option, the most widespread in B2B. The tenant gets a long, secret token, stores it safely — much like a password that does not expire — and sends it with every call. If the key is compromised (typically because it ended up on GitHub by accident, which happens more often than you'd think), it is put on the revocation list and a new one is issued. This is the pattern behind AWS access keys, OpenAI keys, Stripe keys. The downside: a stolen key is valid until someone notices.

Short-lived JWT tokens with a refresh mechanism are the stricter variant. The tenant authenticates once with a long-lived refresh token and receives a short-lived access token that expires after 24 hours. If it is stolen, it soon becomes useless anyway. The downside is the extra complexity on the tenant's side: refresh logic, clock differences, error handling. That makes onboarding harder and produces more support cases.

For most B2B use cases, long-lived API keys with a revocation list are the right choice. A fast revocation path via the admin API plus optional automatic rotation limit the damage window sufficiently. JWT tokens can be retrofitted for individual tenants with special security requirements without changing the base architecture.

The three decisions are one decision

It is tempting to make the three decisions independently. In practice they hang together and jointly define a service's trust posture — what the provider realistically promises the tenant.

Before signing a contract, it pays for a tenant to ask three concrete questions:

• Is our data separated in its own collection, or do we share a store with other customers?
• Who holds the key to our data — the provider, or can we bring our own?
• How is our API access authenticated, and how fast can we revoke a compromised key?

A provider that answers these questions without evasive maneuvers has thought the architecture through.

What Creaminds contributes

We make these three decisions in ongoing projects — for customers building their own RAG infrastructure or evaluating external services. We help in the architecture phase, when the trust posture is set; in implementation, when the data model and key management are built; and in assessing external providers, when you want to know what actually sits behind a "producer-push makes it private" promise.

We build the service stack described here ourselves too — both as a hosted and as a sovereign variant. If the architecture questions from this article come up in a concrete project of yours, talk to us.