“It Looks Right” Is Not a Test Strategy.

A design check we built passed a blank page.

Only a sound check answers both questions correctly. Not a page with a small flaw — a page with nothing on it. Every rule in the gate reported green: contrast fine, no raw hex outside the token layer, tokens used correctly. The one thing that could not be true about a blank page — that it was fine — was exactly what the check said. Nobody had asked what the check should say about a page with nothing on it, because nobody had ever shown it one.

The gate that passed a blank page

Input-to-output testing asks whether a system's answer matches an expected answer for a known input. That's necessary and it is not close to sufficient, because a check that has only ever been run against work that already passes has never demonstrated it can fail. A green result from a test like that is not evidence of correctness. It's evidence the test has never been challenged.

What a fixture is for

The fix wasn't a smarter check. It was a set of fixtures — deliberately broken pages, one per rule the gate enforces — that the check is required to fail against before it's trusted against anything real. A rule for spacing has to fail on a page with the spacing bug it claims to catch. A rule against raw colour values has to fail on a page that has one. Sound, vacuous, and broken become three separate, checkable claims instead of one hopeful one: sound means it catches the real defect, vacuous means it never fires at all, broken means it fires on work that's actually fine. Only the first is a check worth trusting.

Two verdicts, one screenshot

A separate model-graded check — read the render, say what's wrong — returned two different verdicts on byte-identical screenshots run seconds apart: “one defect” the first time, “nothing to object to” the second. Same pixels, different answer. That's the harder version of the same problem: a check that judges by impression rather than by a stated rule can't be trusted to repeat itself, and a rule that only fires once in every few runs will pass code review on the run where it stayed quiet. Tracing which model, which prompt, and which screenshot produced which verdict is the only way to know whether a passing result today would still pass tomorrow, or just did.

There's a real cost to skipping this, separate from any individual bug: a rule against a specific defect — a decorative label with no information in it — was rejected in review three separate times and still shipped forty-four times anyway, because no detector for it existed yet. The rule wasn't the problem. The absence of anything checking for it was.

Evaluating AI Agents in 2026, by our founder Leon Ginsburg, covers the method for building an evaluation set from scratch — what a representative case looks like, how to weight failure modes. Worth reading alongside this one. What I’m arguing here is narrower and came from a specific place: a check can follow that method exactly and still be untrustworthy, if nobody has ever shown it a case it’s supposed to fail. The method gets you a good eval set. It doesn’t, on its own, tell you whether the thing running against that set is actually checking anything.

The difference that survives

A plausible answer is one that looks right on the read you happened to give it. A verified one has been checked against a case built specifically to catch it being wrong, and passed that case on purpose, not by chance. The first is a claim. The second is a measurement, and it’s the only one you can repeat and trust the second time.

Green is not evidence. If your evaluation gates have never been shown a case they are supposed to fail, we can build the fixtures that tell you which of them actually check anything. Book a walkthrough.