Turn ideas into shipped code.

Finding 01

With the same strong auditor, a cheap writer beats an expensive one.

Quality is the Auditor's job. The builder only has to be cheap enough to try again.

The 95% confidence interval on the cost difference excludes zero — the gap is statistically supported, not luck. Three runs are excluded for recorded reasons (a schema bug, a cost-ceiling kill, a transient network error); exclusions are explicit, the underlying record is never edited, and applying them made the losing side look better. Escaped defects are monitored by a separate pipeline that audits the Auditor, rather than assumed away.

Finding 02

I built a planning step, then deleted it.

Catching problems early is how software costs are normally controlled. Finding an issue while planning is far cheaper than finding it after something has been built. So I built a gate: every request was checked for clarity before any code was written, and the doubtful ones were held back.

A gate that holds work back has to be right often enough to be worth the delay it causes. This one wasn't close, so I deleted it.

With AI, writing the code became the cheap part, so catching mistakes early doesn't save as much as it used to. The intake gate I built judged the issue description, but the Auditor judges real code. The fastest and cheapest way to learn if an issue is clear is to build it and see.

Finding 03

The cheaper auditor scored perfectly. I didn't switch.

The Auditor is where nearly all the money goes, so replacing it with a cheaper model was the obvious next saving. I ran the candidate against the incumbent across every evaluation channel I had.

A cheap model and an expensive one both scoring 100% is the result I was hoping for, and that is the reason I didn't switch. Two models scoring the same doesn't prove they are equally good. It can just as easily mean the test was too easy to tell them apart — and a perfect score on both sides is what that looks like.

I had written down what would count as evidence before running the comparison, and the cheaper model cleared every number on the list. I kept the expensive one anyway, because the run had told me something about my test instead: I wrote all of those defects myself, so they were the ones I already knew to look for. Catching them shows a model can find my mistakes, not the ones I'd miss. A passing grade from an instrument you haven't validated isn't evidence.

So I started building the harder test — real defects from real work — and a month later I stopped, because the test I wanted cannot exist. A set that size lets an auditor that is meaningfully worse pass about half the time, no better than a coin flip, and closing that gap takes roughly ten times the evidence, not better-written cases. And every case in it would be a defect the expensive auditor had already caught — so the one thing such a test can never measure is the thing I care about, which is what that auditor misses. The saving isn't rejected; the question is closed. What replaced it watches real work instead: shipped changes are re-audited by a third model, and a free check traces later bug fixes back to the change that introduced the defect.

Opus 4.8 against Sonnet 5, on twelve hand-built cases, July 2026. The decision rule went into the repo before the cases were built, and both before the comparison ran. Three of the twelve had been tuned first so the incumbent scored cleanly — two were fixes to the scoring key, one rewrote the case itself — and the candidate matched it on all three anyway.