Adjudicator — How It Works

One sentence in, shipped changes out.

An AI reads the relevant code first, then writes the specification — so the work is defined by something that has actually looked at the codebase, not by how well you happened to describe it.

Every specification is written in a strict, repeatable format. Some lessons learned improve the format, which is inherited by every issue moving forward. The system gets better at understanding how you describe work, instead of forcing you to get better at working around the quirks of AI.

Expand

you the system working done

Nothing ships on one AI's word

The AI that writes code is fast and cheap. It gets things wrong all the time, and that's totally fine because it never has the final say. A stronger AI from a different company passes judgement on every change with no knowledge of how it was written.

The Auditor will either approve the changes, send it back to the builder with specific problems highlighted, or stop and ask a human for input.

What the data changed my mind about

Every run writes its cost and outcome to a ledger, so the design can be empirically tested and optimized rather than theorized. The experiments below were run in mid-2026. Before trusting any of it, I ran a control experiment — two identical configurations side by side — to prove the measurement couldn't invent a difference that wasn't there. So far it has produced three findings. Two of them cost me something I wanted.

Finding 01

A cheap writer with a strong auditor beats an expensive writer.

Quality is the Auditor's job. The builder only has to be cheap enough to try again.

Cheap builder + independent audit $0.71 per delivered change, median

20 of 20 tasks delivered
needs more audit retries — and is 64% cheaper with them priced in

vs

Strong builder + same audit $1.98 per delivered change, median

20 of 20 tasks delivered
same wall-clock time per delivered change

The 95% confidence interval on the cost difference excludes zero — the gap is statistically supported, not luck. Three runs are excluded for recorded reasons (a schema bug, a cost-ceiling kill, a transient network error); exclusions are explicit, the underlying record is never edited, and applying them made the losing side look better. Escaped defects are monitored by a separate pipeline that audits the Auditor, rather than assumed away.

Finding 02

I built a planning step, then deleted it.

Catching problems early is how software costs are normally controlled. Finding an issue while planning is far cheaper than finding it after something has been built. So I built a gate: every request was checked for clarity before any code was written, and the doubtful ones were held back.

How well the gate could spot a request that would go badly

Needed, to pay for itself

6×

What it actually reached

1.84×

A coin flip is 1×. The 1.84× is the generous end of the measured range; the honest middle is 1.31×.

A gate that holds work back has to be right often enough to be worth the delay it causes. This one wasn't close, so I deleted it.

With AI, writing the code became the cheap part, so catching mistakes early doesn't save as much as it used to. The intake gate I built judged the issue description, but the Auditor judges real code. The fastest and cheapest way to learn if an issue is clear is to build it and see.

Finding 03

The cheaper auditor scored perfectly. I didn't switch.

The Auditor is the expensive half of the system, so replacing it with a cheaper model was the obvious next saving. I ran the candidate against the incumbent across every evaluation channel I had.

Incumbent Auditor 100% every evaluation channel

caught every seeded defect
the expensive half of each run

tie

Cheaper candidate 100% every evaluation channel

caught every seeded defect
would have cut cost per run

That is the result I was hoping for, and it is why I didn't take it. Two models of clearly different capability scoring identically at the top of the range doesn't show they are equivalent — it shows the test can't tell them apart. The fixtures were all synthetic: defects I wrote myself, which means defects shaped like the ones I already knew to look for.

I had written down what would count as evidence before running the comparison, and that bar required the fixtures to discriminate — some cases the weaker auditor should miss. Not one. So the honest reading of a perfect tie is a ceiling effect, not measured parity, and the saving stays on the table until the fixtures are real defects from real changes. A passing grade from an instrument you haven't validated isn't evidence.

For the technically curious

Everything above is the whole idea. This is the layer underneath it, for anyone who wants one. In my own setup the task board is Linear and the code host is GitHub, wired together by webhook. The three agents have names: The Builder writes the code, The Auditor passes judgement on it, and The Adjudicator sits between them, deciding what happens next.

Expand

you, and the part that coordinates the AIs doing the work tools it plugs into

Your task board is the source of truth

There's no separate database of what's happening. The board you already look at holds the state, so what it says and what's really going on can't drift apart.

Every run is logged, permanently

Cost and outcome are appended to a record that is never edited. That log is what made the findings above possible — and what would expose it if quality started slipping.

It has spending limits and stops itself

Each task has a cost ceiling. If it stalls, loops, or hits something it can't decide, it stops and asks rather than burning budget or guessing.

The Auditor gets audited too

A separate process checks for defects that made it past a clean review, so "the Auditor caught it" is something measured rather than assumed.

The Auditor can't touch the code

It runs as a separate agent with no write path to the repository — restricted mechanically by permissions, not by instruction. Independence is architectural, not a rule the model is asked to follow.

Merging without you is opt-in

A codebase earns unattended merge once its results justify it. It is set per project, and it is not the default.

It runs on its own around the clock

Multiple codebases at once, one task at a time within each. It picks up work when you mark it ready, whether or not you're there.

It works on itself

Its own codebase is one of the codebases it maintains — including several of the changes described on this page. That's the evidence I trust most, because I live with the consequences.