Prompt Engineering for Enterprise RAG: System Prompts That Reduce Hallucination

Teams spend months perfecting retrieval — chunking, embeddings, hybrid search — and then hand the perfect context to the model with a one-line prompt: "Answer the question using the context below." Then they're baffled when the model ignores the context, blends in things it "knows," or invents a confident citation. The retrieval was fine. The prompt failed.

This is the most under-appreciated truth in enterprise RAG: retrieval delivers the context; the system prompt determines whether the model actually uses it — and only it. A huge share of "RAG hallucinations" are really prompt failures. Get the system prompt right and the same retrieval produces grounded, cited, honest answers. Here's how to engineer one for production.

Anatomy of a production-grade RAG system prompt

A robust enterprise RAG system prompt does six jobs. Skip any of them and a failure mode opens up:

• Establish the role and scope. Tell the model what it is and what it answers (and what it doesn't).
• Mark the retrieved context clearly and instruct the model to answer from that context — not from its training data.
• Separate grounded claims from general knowledge. A strong pattern: a documentation section that must be cited, and an optional general best-practice section that is clearly labeled and never cited as if it came from your sources.
• Enforce citations on every claim drawn from the documents.
• Define no-answer behavior — what to do when the context doesn't support an answer.
• Calibrate to confidence — be more cautious when retrieval was weak.

SphereIQ's production prompt is built exactly this way: it presents retrieved passages in a clearly delimited "from your documentation" section the model must cite with inline [1], [2] markers, keeps general knowledge in a separate, uncited section, and adjusts its posture based on a confidence level derived from the strength of the top retrieval match. That structure is what turns raw context into a trustworthy answer.

A simplified template:

You are an enterprise knowledge assistant. Answer using ONLY the documentation provided below, and cite each fact with its bracket number, e.g. [1]. If the documentation does not contain the answer, say so plainly — do not use outside knowledge to fill the gap. You may add clearly-labeled general best practices in a separate section, without citations. If retrieval confidence is low, keep the grounded answer brief and flag the uncertainty.

## Documentation (cite these)

[1] {chunk text, source, page}

[2] {chunk text, source, page}

It's not clever wording; it's a contract. Every clause closes a specific failure mode.

Citation enforcement: the anti-hallucination lever

Requiring citations does more than make answers verifiable — it reduces hallucination, because forcing the model to attribute each claim to a specific retrieved passage discourages it from asserting things no passage supports. The enforcement rules that matter:

• Cite from the provided context only. Citations must map to the actual retrieved passages, with their source and location — not invented references.
• No citation, no claim. If a statement can't be tied to a retrieved passage, it shouldn't be presented as fact from your knowledge base.
• Make citations checkable. Tie each marker back to the exact source passage (SphereIQ carries each chunk's title, page, and exact text region precisely so a citation can be verified), so a reviewer — or an auditor — can confirm it.

Verifiability is the whole point. An answer a user can click to check is an answer they can trust; an uncited paragraph is just confident prose.

Handling "no context found" gracefully

The most dangerous moment in any RAG system is when retrieval comes back empty or weak — because a poorly-prompted model fills the silence by making something up. Production prompts handle this explicitly:

• Permit "I don't know." Tell the model that admitting the documentation doesn't cover something is the correct answer, not a failure. This single instruction prevents a large class of hallucinations.
• Offer a safe fallback. When there's no internal match, the system can answer from general knowledge only if it clearly labels it as such — SphereIQ, for instance, falls back with an explicit note that the answer is general and that relevant internal documentation may simply not be uploaded yet.
• Never fabricate citations. The worst failure is a confident answer with invented sources. The prompt must forbid manufacturing references when none were retrieved.

Graceful no-answer behavior is what makes an enterprise assistant trustworthy — users learn it won't bluff, so they believe it when it does answer.

Multi-turn conversation management

Enterprise users don't ask one-shot questions; they have conversations. "What's our refund policy?" → "What about for enterprise contracts?" The second question is meaningless without the first, and naive RAG embeds "What about for enterprise contracts?" and retrieves garbage.

The fix is query rewriting: before retrieval, rewrite the follow-up into a standalone query using the conversation history ("What is our refund policy for enterprise contracts?"), then retrieve on that. SphereIQ rewrites the query from history precisely so retrieval stays accurate across turns. The system prompt also has to manage history length (keep it relevant, not unbounded) and maintain grounding rules consistently across the whole conversation — the model shouldn't get less disciplined about citations on turn five than on turn one.

Prompt design is a security control, too

One more reason prompt engineering matters: retrieved documents are part of the attack surface. Indirect prompt injection plants instructions inside content the system will retrieve, so the prompt must be designed to treat retrieved context as reference data to answer from, never as instructions to obey — and to work alongside input/context screening and guardrails rather than relying on the prompt alone. A well-structured prompt that cleanly separates "instructions" from "data" is a meaningful layer of defense against a passage that says "ignore previous instructions and…".

The takeaway

Your retrieval layer can be excellent and your system can still hallucinate, ignore context, and invent citations — because the prompt is where the model is told how to behave with the context it's given. A production RAG system prompt establishes scope, marks and prioritizes retrieved context, enforces citations, permits and shapes "I don't know," calibrates to confidence, manages multi-turn queries, and treats retrieved text as untrusted data. It's the cheapest, highest-leverage layer in the whole stack. Engineer it like one.

Think your prompts are the weak link? Get a RAG Readiness Assessment — we'll review your system prompt against the production checklist for grounding, citations, and no-answer behavior.

Related: the enterprise RAG pillar guide, how to evaluate RAG accuracy, and preventing RAG hallucination.