Eval-Gated Publishing: No Agent Ships Until It Passes Its Goldens

In the AI Factory, an agent can't be published until it passes evaluation against a set of known-good questions at a configured threshold. Correctness becomes a deploy gate — something the agent has to clear before it ships — not a dashboard someone checks after it's already live.

What is eval-gated publishing?

Eval-gated publishing is a simple rule with large consequences: an AI agent cannot go live until it has been tested against a set of questions whose correct answers are already known, and scored at or above a threshold you set. If it clears the bar, it publishes. If it doesn't, publishing is blocked and the agent goes back to the bench. The gate is not advisory and it is not a warning banner — it is a wall the agent has to get over.

The reason to build it this way is that "publish" and "correct" are two different events, and most tools quietly treat them as one. A no-code builder that lets anyone assemble an agent and push it live has optimized for the wrong moment. The moment that matters is not shipping; it's shipping something that works — and the only way to know it works is to check, against reality, before it reaches a user.

Why "publish" is the wrong default for an AI agent

Traditional software has a compiler, a test suite, and a reviewer standing between an idea and production. An AI agent assembled from a prompt, a knowledge scope, and a few tools has none of that by default. It will happily "work" in the sense that it returns fluent answers — and fluent is exactly the failure mode, because a confident wrong answer looks identical to a confident right one until someone checks.

So the default has to change. Instead of "build it, publish it, watch the support tickets," the default becomes "build it, prove it, then publish." That inversion is the whole idea. It costs a little friction at publish time and saves the far larger cost of a broken agent quietly giving wrong answers to real people for a week before anyone notices.

What's a golden set?

A golden set is the agent's exam: a curated collection of questions paired with their known-correct answers, drawn from the domain the agent is meant to serve. For a legal-redline agent, it's real clauses with the right call already established. For a support agent, it's real questions with the answers a senior rep would give. The value is entirely in the curation — a golden set assembled carelessly tests nothing, while one built from real cases and verified by someone who knows the domain becomes a durable definition of "good enough".

The important discipline is that golden sets are maintained, not written once and forgotten. As the domain shifts and as new failure modes surface, cases get added. A stale golden set slowly stops meaning anything, which is why treating it as a living artifact — versioned, reviewed, extended — is part of the practice, not an afterthought.

The publish gate: pass or it doesn't ship

When someone hits publish, the evaluation harness runs the agent against its golden set and scores the results. The outcome is binary at the boundary: the agent is either published or the publish is blocked, with the failing cases shown so the builder knows exactly what to fix. There's no "published with warnings," because a warning nobody reads is how broken things ship.

Illustrative publish gate for a legal-redline agent:

• Indemnity clause — flags uncapped liability — PASS
• Governing law — identifies missing jurisdiction — PASS
• Auto-renewal — catches silent renewal term — FAIL
• Data processing — cites the wrong sub-processor list — FAIL
• Publish blocked: 2 of 12 golden cases below threshold — returned to the builder with the failing cases attached.

The threshold itself is configured per agent, because the bar for a low-stakes internal helper and a customer-facing agent in a regulated workflow are not the same. What doesn't change is that the bar exists and the gate enforces it.

Catching drift after launch

Passing the gate on publish day is necessary but not sufficient, because an agent that was correct in June can quietly degrade — an underlying model updates, the knowledge base shifts, a connector changes shape. So the same evaluation runs on a schedule after launch, and when a score crosses a line, it raises a drift alert. This is the difference between finding out your agent got worse from a metric and finding out from an angry customer.

Continuous evaluation turns quality from a launch event into a standing property. The golden set that gated the agent on day one becomes the tripwire that catches it drifting on day ninety.

Red-teaming as part of the gate

Golden sets test whether an agent is right. Red-teaming tests whether it can be made to misbehave. Alongside the correctness evaluation, an agent can be run against adversarial prompts — attempts to jailbreak it, extract data it shouldn't reveal, or push it outside its scope. An agent that answers its golden questions perfectly but leaks a system prompt under pressure has not earned publication. Pairing correctness evaluation with an adversarial pass means the gate checks both "does it work" and "can it be broken."

Where it fits: a governed compose-to-publish pipeline

Eval-gating is one stage in a larger flow. In the AI Factory you compose an agent from a system prompt, a knowledge scope, a set of tools, and a model policy; you test it against its golden set in a built-in harness; the publish gate runs that evaluation and blocks anything below threshold; and once live, continuous evaluation watches for drift. Every one of those steps is recorded, so "why did this agent ship" has an answer on the same audit record as everything else the platform does.

It also connects upstream: the processes worth building an agent for come from the opportunity map, and the agent that gets built gets gated here. Finding the work and proving the work are two ends of one pipeline.

Make correctness a deploy gate. See how the AI Factory tests an agent against its golden set, blocks any publish below threshold, and watches for drift after launch — with every step on the audit record. Book a walkthrough.