Private LLM + RAG: Keeping Enterprise Data Out of Public Models

For most companies, "send the prompt to OpenAI or Anthropic" is a perfectly good answer. For a defense contractor, a hospital, a bank's trading desk, or any organization with strict data-residency obligations, it's a non-starter — because the prompt is the sensitive data, and the moment it leaves the boundary, the deal is off.

These organizations need a different stack: a private LLM running inside their own environment, a self-hosted vector database, and an ingestion pipeline that never phones home. The good news is that open models have gotten good enough to make this genuinely viable, not a quality sacrifice. The trade-off is real but manageable. Here's the architecture for keeping enterprise data out of public models — and how to decide how far down the privacy spectrum you actually need to go.

The deployment spectrum

"Private" isn't binary; it's a spectrum of increasing control and effort. Place your requirements on it deliberately:

• Managed SaaS. The model and pipeline run as a vendor service. Easiest, lowest control. Data crosses a SaaS boundary.
• Private cloud / VPC. The full pipeline runs inside your cloud account/VPC. Data stays in your environment; you control the network and keys. The pragmatic choice for most security-conscious enterprises.
• On-premise. Everything runs in your own data center. Maximum control short of disconnection; needed when cloud itself is disallowed.
• Air-gapped. No external network connectivity at all. The model, vectors, and ingestion run in an isolated environment. Required for the most sensitive defense, intelligence, and critical-infrastructure use cases.

The key architectural promise to design toward: data and queries never pass through a vendor's servers or a third-party LLM provider. SphereIQ supports exactly this — the full RAG pipeline can run in private cloud, VPC, on-prem, or air-gapped environments, with nothing routed through Sphere or external model APIs when configured that way.

Model options: capable open models you can host

The reason private RAG is now practical is that the open-weight model ecosystem has matured. Strong options you can run inside your boundary:

• Llama (Meta). A leading open-weight family across sizes, strong general capability, widely supported by serving stacks.
• Mistral / Mixtral. Efficient open models (including mixture-of-experts) that punch above their size, good for cost-effective self-hosting.
• Qwen and other open families. Increasingly competitive, with strong multilingual and reasoning variants.
• Amazon Titan (in your VPC via Bedrock). A managed-but-in-AWS option when "stays in our AWS account" is the bar rather than "open weights on our own GPUs."

The right size depends on the task: smaller models (7–8B) are cheap and fast for routine retrieval-grounded Q&A, while larger models close the gap to frontier quality for harder reasoning — at higher hardware cost. Because SphereIQ is LLM-agnostic (GPT, Claude, Llama, Mistral, custom/open-source, or bring-your-own-model), the private-deployment decision doesn't lock you into one model; you can run the best open model your hardware and use case justify, and change it later.

Self-hosted vector databases

A private LLM with a SaaS vector store still leaks your embeddings. A genuinely private stack keeps the vectors inside the boundary too:

• pgvector (PostgreSQL). Vectors inside your existing database; the lowest-friction private option, with the data control and governance of Postgres. SphereIQ's Knowledge AI defaults here.
• Qdrant. An open-source vector database designed for self-hosting, with strong filtering and performance.
• Milvus. An open-source vector database built for very large-scale workloads, when corpus size or throughput demands a dedicated engine.

For most private deployments pgvector is enough and simplest; Qdrant or Milvus earn their place at large scale or when you want dedicated vector-database features inside your own infrastructure.

Air-gapped ingestion

The subtlest part of a fully private deployment is ingestion. An air-gapped or strict on-prem RAG can't reach out to a cloud OCR service, a hosted embedding API, or an external parser — every step must run inside the boundary. That means self-hosted document extraction and OCR, a self-hosted embedding model (open-source embeddings you run yourself), and connectors that operate entirely within the network. It's more engineering than a cloud pipeline, but it's the price of "nothing leaves," and it's a solved problem when designed for from the start rather than discovered at the security review.

The trade-off: capability vs. control

Be honest about the tension, because pretending it doesn't exist leads to bad decisions:

• Hosted frontier models (GPT, Claude) still hold an edge on the very hardest reasoning and the broadest capabilities, and they require no infrastructure. You trade data control for that.
• Self-hosted open models give you full control, residency, and no per-token markup, and are now more than capable enough for the large majority of retrieval-grounded enterprise tasks — where the model is grounding answers in your retrieved documents rather than free-reasoning. You trade some peak capability and take on hardware and ops.

The decision rule: match the deployment to the data's sensitivity, not the model's leaderboard rank. If your data genuinely can't leave the boundary, a strong open model running privately — grounded by good retrieval — will serve the vast majority of use cases well, and the residual gap matters far less in a RAG setting than in open-ended generation. If your data isn't that sensitive, don't pay the private-deployment tax for control you don't need. Many enterprises land on a sensible middle: VPC deployment with an open or in-account model, reserving full air-gap for the genuinely classified workloads.

What you gain

Run RAG privately and you get the things regulated and security-first buyers actually need: data residency and control (nothing egresses), auditability (every query logged inside your environment — the governance posture detailed in enterprise RAG security and governance), model flexibility (swap models without re-architecting), and no vendor lock-in (open weights and BYOK mean you're never captive to one provider's pricing or roadmap). For the organizations that need it — the regulated industries where residency is a legal obligation, not a preference — that combination isn't a compromise; it's the only deployable option, and it's exactly what Sphere builds.

Need data to stay inside your walls? Schedule a consultation — Sphere deploys fully private and air-gapped enterprise RAG, from VPC to on-prem.

Related: the enterprise RAG pillar guide, enterprise RAG security and governance, and RAG for regulated industries.