RAG Evaluation: How to Measure Accuracy Before Going to Production

Most enterprise RAG projects are evaluated with a method that has a technical name: vibes. Someone on the team asks the system a dozen questions, the answers look plausible, and it ships. Then real users ask the other ten thousand questions, the cracks show, trust evaporates, and "the AI doesn't work" becomes the official verdict.

You cannot improve what you don't measure — and "it seemed good in the demo" is not a measurement. RAG evaluation turns accuracy from an opinion into a number you can track, defend, and improve. This is the framework Sphere uses to prove a system works before it goes to production and to keep it honest after — the same approach that, on a regulated tax-and-compliance build, benchmarked retrieval precision, recall, faithfulness, and answer relevancy and demonstrated a 66% retrieval-accuracy improvement over the prior keyword system.

Why "it looks right" fails

A RAG answer can be wrong in two fundamentally different places, and a single eyeball check catches neither reliably:

• Retrieval can fail — the system pulled the wrong passages (or missed the right ones), so the model never had the information to answer correctly.
• Generation can fail — the right passages were retrieved, but the model ignored them, misread them, or added a confident claim that isn't in the source (a hallucination).

Because these fail independently, you have to measure them independently. That's why serious RAG evaluation uses four metrics, two for retrieval and two for generation — not one overall "accuracy" score.

The four core RAG evaluation metrics

1. Context precision (retrieval)

Of the passages we retrieved, how many were actually relevant? Low precision means the system is dragging in noise — irrelevant chunks that crowd the context window and tempt the model off-track. High precision means a clean, relevant context set.

2. Context recall (retrieval)

Of the information needed to answer, how much did we actually retrieve? Low recall is the silent killer: the answer existed in the corpus, but retrieval didn't surface it, so the model couldn't use it. Recall is what catches "the answer was in the documents but the system couldn't find it."

3. Faithfulness (generation)

Is every claim in the answer supported by the retrieved context? This is the hallucination metric. A faithful answer makes no claim that isn't grounded in the sources. Low faithfulness means the model is inventing — the single most dangerous failure in regulated environments.

4. Answer relevance (generation)

Does the answer actually address the question that was asked? An answer can be faithful (every claim sourced) yet unhelpful (it answers a slightly different question). Answer relevance catches grounded-but-off-target responses.

Together these four separate the layers: precision and recall tell you whether retrieval is working; faithfulness and relevance tell you whether generation is working. When a number drops, you know exactly which layer to fix.

RAGAS and Sphere's extensions

The open-source RAGAS framework operationalizes exactly these dimensions — faithfulness, answer relevance, context precision and recall — using an LLM to score answers against retrieved context and reference answers. It's a strong, fast starting point, and a good way to get baseline numbers without building evaluators from scratch.

For enterprise production, Sphere extends it in three ways that matter:

• SME review. LLM-graded metrics are excellent for scale and regression, but subject-matter experts validate the hard, high-stakes cases that automated scoring can miss — especially in regulated domains where "close" is wrong.
• Regression suites. Every fix and every content change runs against a fixed evaluation set, so you catch the regressions that silently ship when someone "improves" a prompt or re-chunks a corpus.
• Continuous production evals + human feedback loops. Evaluation doesn't stop at launch (more below).

Build a golden dataset first

None of these metrics mean anything without a golden dataset — a curated set of real questions paired with known-good answers and the source passages that support them. This is the unglamorous work that makes evaluation real, and it's worth doing properly:

• Use real questions. Pull from actual user queries, support tickets, and SME knowledge — not questions you invented to make the system look good.
• Cover the distribution. Include easy lookups, multi-hop questions, edge cases, and questions the system should refuse (outside the corpus).
• Record the ground truth. For each question, capture the correct answer and which source passages support it — that's what lets you measure recall and faithfulness, not just plausibility.
• Keep it versioned. The golden set is an asset; it grows as you find new failure modes.

A few hundred well-chosen examples beats ten thousand random ones. The golden dataset is the ruler; everything else is measured against it.

Evaluation doesn't stop at launch

The biggest mistake teams make is treating evaluation as a pre-launch gate — pass it once, ship, move on. But a RAG system's accuracy drifts the moment it meets production: the corpus changes, users ask things you didn't anticipate, models get updated, and what scored 90% in March quietly degrades by June.

Continuous evaluation keeps it honest. In production that means sampling real traffic and scoring it on the same four metrics, watching for drift, capturing a human feedback loop (let users flag bad answers, and feed those into the golden set and regression suite), and alerting when faithfulness or recall slips. Evaluation is mandatory before production and ongoing after it — the difference between a system that gets better every week and one that silently rots.

Red-teaming: the queries that break most RAG systems

Standard metrics measure the happy path. Red-teaming measures the failure path on purpose, with adversarial queries that expose the cracks:

• Out-of-corpus questions — does it refuse gracefully, or confidently make something up?
• Prompt-injection attempts — can retrieved or user content hijack the system's instructions?
• Ambiguous and multi-hop queries — does it conflate two policies, or stitch a wrong answer from two right passages?
• Permission probes — can a user retrieve content they shouldn't see?
• Leading questions — does it agree with a false premise the user asserts?

Red-teaming before launch is how you find these in a meeting instead of in an incident. The systems that survive production are the ones that were deliberately attacked before they shipped.

The bottom line

Evaluation is what separates an enterprise RAG system from an enterprise RAG demo. Measure retrieval and generation separately on the four core metrics, anchor everything to a golden dataset, run it continuously in production, and red-team it before you trust it. Do that and "the AI works" stops being a hope and becomes a number you can show your stakeholders — and defend to your regulator.

Want to know your real accuracy numbers? Get a RAG Readiness Assessment — we'll help you stand up a golden dataset and a four-metric evaluation harness.

Related: the enterprise RAG pillar guide, the 8-phase RAG implementation playbook, and preventing RAG hallucination (coming soon).