Enterprise RAG Checklist: 47 Questions Before You Go Live

Most enterprise RAG failures aren't dramatic. They're a missing answer to a question nobody asked until it was too late — "wait, can users retrieve documents they shouldn't see?" at the security review, or "how do we know it's accurate?" the week after launch. A go-live checklist exists to surface those questions before they become incidents.

This is the enterprise RAG checklist Sphere uses, distilled to 47 questions across seven areas. If you can answer "yes" (with evidence) to all of them, you're ready for production. If you can't, you've just found your pre-launch work. Run your project through it.

1. Use case & scope

• Is there one specific, high-value use case (not "add AI to everything")?

• Is success defined with a measurable metric (deflection, time-to-answer, accuracy, adoption)?

• Is what's out of scope for v1 explicitly documented?

• Is there a named business owner accountable for the outcome?

• Have you confirmed RAG is the right tool (vs. fine-tuning, search, or a simpler solution)?

2. Data readiness

• Have you inventoried every source system the use case needs?

• Do you know the quality of that data (duplicates, stale versions, scanned-only documents)?

• Have you mapped the permission model of each source?

• Can the required content be accessed via a secure, supported connector?

• Is there a plan for non-text content (scanned PDFs, charts, drawings) if relevant?

• Is there an owner for keeping the corpus current after launch?

• Can content be deleted from the index (and embeddings) when a source document is removed?

3. Security & access control

• Does retrieval enforce permissions before ranking, so users can't retrieve content they couldn't open directly?

• Do chunks inherit and keep their source documents' access controls?

• Is the deployment model (SaaS, VPC, on-prem, air-gapped) matched to your data sensitivity?

• Is data egress controlled — does anything leave your boundary, and is that acceptable?

• Are encryption and BYOK in place where required?

• Is user input and retrieved content screened for prompt injection?

• Does the system map to your obligations (SOC 2, HIPAA, GDPR, MRM) where applicable?

• Has security/compliance signed off — not as a final gate, but throughout?

4. Architecture & retrieval quality

• Is chunking structure-aware (not naive fixed-size that shreds tables and clauses)?

• Does each chunk carry metadata (source, title, page/section, access level, freshness)?

• Is the embedding model validated on your documents and languages?

• Is retrieval hybrid (keyword + vector) where exact-match terms matter?

• Is there a reranking step for top-result quality?

• Does the vector store fit your scale, residency, and operating model?

• Is metadata/permission filtering applied at retrieval time?

• Has retrieval accuracy been measured (precision/recall), not just eyeballed?

5. Generation & accuracy

• Does the system answer only from retrieved context, with general knowledge clearly separated?

• Are inline citations enforced on every claim, traceable to the exact source?

• Does the system signal confidence and refuse gracefully when context is insufficient?

• Is "I don't know" treated as a correct answer, not a failure?

• Is the model strategy defined (and swappable / LLM-agnostic where needed)?

• Are follow-up questions handled (query rewriting across turns)?

6. Evaluation & testing

• Is there a golden dataset of real questions with known-good answers?

• Have you measured faithfulness and answer relevance, not just retrieval?

• Has the system been red-teamed (out-of-corpus questions, injection, permission probes, leading questions)?

• Is there a regression suite so fixes and content changes don't silently break things?

• Does it refuse and escalate correctly on edge cases?

• Have SMEs validated the high-stakes answers?

7. Operations & ownership

• Is there production monitoring for retrieval quality, hallucination rate, and latency?

• Are data-freshness and connector-health alerts in place?

• Is every query-answer pair logged, and can it mirror to your SIEM / export for discovery?

• Are cost controls in place — cost per query/team and budget alerts?

• Is there a continuous-evaluation loop on live traffic (not just pre-launch)?

• Is there a user feedback mechanism, routed into improvements?

• Is human-in-the-loop defined for high-stakes outputs?

• Is there a named production owner and a documented runbook for the Run phase?

How to use this checklist

Don't treat it as a final gate to rush at the end — run it throughout the build, the way Sphere's AI Foundry delivery does (Intake → Blueprint → Forge → Harden → Run). Most of these questions are cheap to answer in design and expensive to answer after launch. A "no" isn't a failure; it's a task. The goal isn't a perfect score on launch day — it's no surprises on launch day.

If you're answering "no" to a cluster of these — especially in security, evaluation, or operations — that's exactly where an experienced partner de-risks the build. The teams that ship reliable enterprise RAG aren't the ones who got lucky; they're the ones who answered all 47 questions on purpose.

Want the full checklist to run against your build? Get a RAG Readiness Assessment and we'll turn your "no" answers into a production plan.

Related: the enterprise RAG pillar guide, the 8-phase RAG implementation playbook, and enterprise RAG security and governance.