Enterprise RAG Security: Data Governance, Access Control, and Compliance

Most "LLM security" advice was written for chatbots: don't leak the system prompt, filter toxic output, rate-limit the API. All useful — and all missing the point for enterprise RAG. The moment you connect a model to your documents, you create attack surfaces a generic LLM security framework never anticipated: retrieval that can leak data across permission boundaries, documents that can carry attacks into the model, and answers that have to be auditable to satisfy a regulator.

RAG security is its own discipline. Get it wrong and your helpful knowledge assistant becomes a permission-bypass machine, an injection vector, and an unloggable black box — all at once. Get it right and it's safe enough for the most regulated environments. Here's the security model that gets you there.

The new attack surface RAG introduces

Three risks are specific to RAG, and none is covered by "secure the LLM API":

1. Cross-permission data leakage. A RAG system retrieves from a corpus. If retrieval doesn't respect who's allowed to see what, a user can surface content they could never open directly — payroll, an HR investigation, a deal they're not on. The model becomes a way around your access controls.
2. Prompt injection via retrieved content. Attackers don't have to type a malicious prompt — they can plant instructions inside a document the system will later retrieve. When that poisoned passage lands in the model's context, it can hijack behavior.
3. Unauditable answers. In regulated work, "the AI said so" is not an answer. If you can't show what was retrieved, what was generated, and why — per query, tamper-evidently — you can't pass an audit or respond to discovery.

Each needs a specific control. Let's take them in turn.

Permission-aware retrieval: access control at retrieval time

The single most important RAG security control is this: the system must enforce permissions during retrieval, before ranking — not in the UI afterward. A chunk a user isn't allowed to see should never enter their candidate set in the first place.

That means permission inheritance: each chunk carries the access controls of its source document, synced from the source system, so a SharePoint file's permissions still apply after it's been ingested and embedded. At query time, retrieval filters candidates to the caller's permitted sources first, then ranks by relevance — effectively row-level security at the retrieval layer. This is how SphereIQ works: connectors record each source's access groups, and a query can only ever match content the requesting user is entitled to. The guarantee to design toward is simple and absolute: a user can never retrieve from a source they couldn't open directly.

Sphere's regulated tax-services deployment shipped exactly this — role-based retrieval so each user's results are scoped to their permissions — because in regulated work, permission-aware retrieval isn't a feature, it's the license to operate.

Prompt injection through poisoned documents

Direct prompt injection (a user typing "ignore your instructions") is well-known. The RAG-specific threat is indirect injection: malicious instructions hidden inside content the system retrieves — a planted line in a wiki page, a crafted PDF, a comment in a synced ticket. Because RAG feeds retrieved text into the model's context, that text can attempt to override the system's instructions, exfiltrate other context, or steer the answer. The user never typed anything malicious; the document did.

Defending against it takes layered controls:

• Screen inputs and retrieved content before they reach the model. It's not enough to scan the user's question; the retrieved passages must be scanned too, because that's where indirect injection lives. SphereIQ runs prompt-injection scanning on inbound content for exactly this reason.
• Separate instructions from data. Architect prompts so retrieved content is clearly treated as reference material to answer from, never as instructions to follow.
• Constrain output and tools. Ground answers strictly in cited context, and where the system can take actions (agentic patterns), gate them behind scoped permissions and human approval so an injected instruction can't trigger a real action.
• Monitor and log. Treat injection attempts as security events, logged and alertable — not silent failures.

The mindset shift: in RAG, your own documents are part of the attack surface. Anything that can be retrieved can carry an attack, so retrieved content gets screened like untrusted input.

Data residency and private deployment

For regulated and security-conscious enterprises, where the data and the inference live is a security control in itself. The strongest posture keeps everything inside the customer's boundary:

• Private cloud, VPC, or on-prem deployment, so documents, embeddings, and prompts never leave the environment — no data egress to third-party APIs.
• BYOK (bring-your-own-key), so encryption keys stay under the customer's control.
• Self-hosted vectors and inference where required, so even the model calls stay inside the perimeter.

This is why SphereIQ supports private/self-hosted deployment with no proprietary data leaving the environment and an LLM-agnostic, BYOK model: for many buyers, "it runs in our VPC and nothing egresses" is the difference between a project that's approved and one that's vetoed at security review.

SOC 2, HIPAA, and GDPR implications

Compliance frameworks weren't written for RAG, but they apply to it directly. The mapping every enterprise should make:

• SOC 2 — access control, audit logging, monitoring, and change management all extend to the RAG pipeline. Permission-aware retrieval and per-query audit logs are core evidence.
• HIPAA — if PHI can be retrieved, the RAG system is in scope: minimum-necessary access (permission-aware retrieval), audit controls, encryption, and a BAA with any processor. PHI must not egress to a model that isn't covered.
• GDPR — lawful basis, data minimization, and the right to erasure all touch RAG: you must be able to find and delete an individual's data across documents and derived stores (chunks, embeddings, any memory), and keep processing within permitted regions (residency).

The throughline: compliance for RAG is achievable, but only if access control, residency, auditability, and deletion are designed into the architecture — not retrofitted after a finding.

Audit logging for every query-answer pair

You cannot secure or defend what you can't reconstruct. Enterprise RAG must log every query and its answer with enough fidelity to answer "what happened?" months later: who asked, what was retrieved (which sources, which passages), what was generated, which model, and when. SphereIQ logs every query — with token and cost detail — and can mirror it to your SIEM (Splunk, Datadog, and others) so AI activity lives alongside the rest of your security telemetry, and supports tamper-evident eDiscovery export for legal hold. Sphere's tax deployment shipped with a full, tamper-evident query/retrieval audit trail for precisely this reason.

Good logging does triple duty: it's your security record (detecting misuse and injection attempts), your compliance evidence (SOC 2 / HIPAA audit, GDPR accountability), and your quality signal (which answers users flagged as wrong). It's not optional infrastructure — it's the spine of a defensible system.

The enterprise RAG security checklist

A production system should be able to say yes to all of these:

• Retrieval enforces source permissions before ranking (no cross-permission leakage).
• Chunk permissions are inherited from source systems and kept in sync.
• User input and retrieved content are screened for prompt injection.
• Deployment matches the risk profile — private cloud / VPC / on-prem with no data egress, BYOK where required.
• The pipeline maps cleanly to SOC 2 / HIPAA / GDPR obligations, including data deletion across chunks and embeddings.
• Every query-answer pair is logged, mirrorable to a SIEM, and exportable for eDiscovery.

Miss any one and you have a capability with a hole in it. Hit all six and you have something a regulated enterprise can actually deploy.

Want to know where your RAG pipeline is exposed? Get a free Enterprise RAG Security Review — we'll assess your access control, injection defenses, residency, and audit posture against the checklist above.

Related: the enterprise RAG pillar guide, RAG for regulated industries, and private LLM + RAG (all coming soon).