RAG Pipeline Monitoring: What to Watch After Go-Live

The dangerous thing about a RAG system isn't that it fails loudly — it's that it degrades silently. Nothing crashes. No error appears. The assistant keeps returning confident, well-formatted answers. They're just slowly getting worse — and by the time someone notices, trust has already eroded and the answer to "when did this start?" is "we have no idea."

A production RAG pipeline is never finished at launch; it enters its longest and most important phase — Run. Your knowledge changes, your sources change, your models change, and your users ask new things, and every one of those shifts can quietly drag quality down. RAG monitoring is how you catch the drift while it's a dashboard blip instead of an executive complaint. Here's what to watch.

Why RAG degrades silently

Five things change after go-live, and each one erodes quality without throwing an error:

1. Data freshness decays. Documents get updated, but if ingestion lags or a connector quietly breaks, the index serves stale answers from old versions.
2. Embedding/representation drift. New content, new terminology, or a model/version change can shift what "similar" means, degrading retrieval relevance over time.
3. Prompt performance shifts. A model provider updates a model behind the same name, and behavior — citation discipline, refusal rate — moves under your feet.
4. Permission and source changes. Access models change, sources are added or removed, and retrieval can start surfacing too much or too little.
5. Query distribution shifts. Users ask things the system was never tuned for, and accuracy on the new distribution is unknown.

None of these announces itself. Monitoring exists to make the invisible visible.

The retrieval-quality signals

Because RAG fails separately in retrieval and generation, monitor both layers — starting with retrieval, where most silent decay begins.

• NDCG (Normalized Discounted Cumulative Gain) — measures whether the most relevant passages are ranked highest, not just present. A falling NDCG means retrieval is burying the good results.
• MRR (Mean Reciprocal Rank) — how high up the first relevant result appears. Declining MRR means users are getting the right answer later (or not at all).
• Retrieval confidence distribution — track the strength of the top match across all queries. A drift toward lower-confidence retrievals is an early warning that the corpus and the questions are diverging.
• Zero/low-result rate — the share of queries returning nothing useful. A rising rate flags coverage gaps or a broken connector.

You don't need labels on every query to use these — sample traffic, score it against a maintained golden set, and watch the trend. The absolute number matters less than the direction.

The answer-quality signals

Retrieval can be fine and answers can still drift, so monitor generation too:

• Faithfulness / hallucination rate — the share of answers making claims not supported by retrieved context. This is the metric that protects you in regulated environments, and it's the one that moves when a model changes silently.
• Answer relevance — does the answer address the question? Catches grounded-but-off-target responses.
• Citation coverage — the proportion of factual claims that carry a valid citation. Falling citation coverage is an early hallucination signal.
• User feedback — thumbs-down, "this is wrong" flags, and escalations are gold. Feed them straight into your golden set and regression suite.

At production scale you can't human-grade everything, so the common approach is LLM-as-judge: a model scores sampled answers for faithfulness and relevance against the retrieved context, cheaply and continuously, with human/SME review reserved for the hard or high-stakes cases. It's the same evaluation harness you built pre-launch (see how to evaluate RAG accuracy) — now running on live traffic instead of a test set.

Data freshness and index staleness

A whole class of silent failure is purely operational: the content is right, but the index is out of date. Monitor the plumbing:

• Ingestion lag — time between a source document changing and the index reflecting it. Alert when it exceeds your freshness SLA.
• Connector health — a connector that silently stops syncing is one of the most common production failures; alert on missed syncs, not just errors.
• Index coverage and deletions — confirm new content is indexed and removed content actually disappears (critical for accuracy and compliance).
• Staleness flags — track the age of the content backing answers, so "we answered from a policy that was superseded three months ago" becomes a metric, not an incident.

Don't forget cost and security telemetry

Monitoring isn't only about accuracy. Two more dashboards belong next to it:

• Cost and FinOps — track cost per query and per team, watch for anomalies (a runaway agent, a prompt change that tripled token use), and enforce budget alerts before the bill is a surprise. SphereIQ tracks real token cost per message and fires budget alerts for exactly this. (More in enterprise RAG cost, coming soon.)
• Security and audit — every query-answer pair logged, mirrorable to your SIEM, with alerts on injection attempts and permission anomalies. Observability and security share the same telemetry.

This is what Sphere's AI Governance & FinOps practice operationalizes: observability, drift and hallucination monitoring, cost-anomaly alerts, budget controls, and regression checks, all as part of the Run phase — because production AI that nobody is watching is production AI that's quietly failing.

Tools

You can assemble this from the ecosystem or buy it as a managed capability:

• LLM observability platforms — tools like LangSmith and Arize Phoenix trace each request, score answer quality, and surface drift, purpose-built for LLM/RAG pipelines.
• Cloud-native monitoring — CloudWatch (or Datadog/Grafana) dashboards for latency, error rates, ingestion lag, and cost, integrated with your existing observability stack.
• Your evaluation harness — the golden-set scoring you built for pre-launch, scheduled against sampled production traffic.
• A managed governance layer — where drift, hallucination, cost, and audit monitoring come built in, so you're not stitching five tools together.

The toolchain matters less than the discipline: instrument retrieval and generation, alert on the five degradation signals, and treat go-live as the start of monitoring, not the end of the project.

Not sure if your live system is drifting? Get a RAG Readiness Assessment — we'll review your monitoring against the five degradation signals and set up the alerts that matter.

Related: the enterprise RAG pillar guide, how to evaluate RAG accuracy, and enterprise RAG cost (coming soon).