Don't trust the model, check it

November 15, 2026

Most teams that add AI review to their pull requests go through the same arc. First it's exciting, then the comments get long, and eventually the useful signal gets lost in the volume. The model is usually capable enough. What's missing is consistency: different output on every run, findings about code outside the change, issues that were there long before it. The goal is a reviewer whose comments people read because they've learned that each one is worth reading.

There's a second problem that gets less attention: cost. A review with a large model and general-purpose tools runs on every push, not once per pull request, and a pull request usually gets several pushes. The bill grows with how much the team ships, which is exactly the thing you don't want to discourage.

This year I designed and built an AI code reviewer from scratch to fix both problems at once, and it's the project I'm proudest of so far. The main lesson fits in one line: the harness is the product, not the model.

Give the model a job, not freedom

The instinct is to hand a capable model the whole repository and ask for a review. What worked better was the opposite: a narrow job, a few tools, and rules it can't talk its way around.

Schematically, the reviewer has two halves. A deterministic half that never calls a model, and a reasoning half that can only act through the tools the first one gives it. The deterministic half does three things:

It decides what "done" means. The model doesn't get to finish whenever it feels like it. The harness knows what had to be looked at and refuses the review until it was.
It focuses attention. Instead of one long, general pass, a few short passes run in parallel, each looking at the change through a different lens. Two runs of the same model find different things anyway, so you might as well make the differences deliberate.
It rejects what it can't verify. A finding has to point at real code in the change, with evidence that's really there. If it can't, it doesn't exist.

That last rule removes a surprising amount of noise. Most low-value AI comments aren't wrong about code; they're about code that isn't in front of them.

Cheap models, reliable reviews

Here's the part that changed how I think about LLM products. Once the harness owns coverage, rules, and validation, the model doesn't need to be brilliant. It needs to be good at reading code and following a procedure. Small, inexpensive models are good at that.

In the first head-to-head on the same pull request and the same commit, the harnessed reviewer with a small model cost about 20 times less than the existing review with a large model, and finished faster. Per review, it's a matter of cents instead of dollars, and since reviews run on every push, the difference multiplies with every iteration of every pull request.

The more interesting result: each reviewer found real issues the other missed. A cheap review didn't mean a worse review. It meant a different one, and in the first weeks the two worked best side by side. One pull request is a sample, not a verdict, so from then on every run was recorded and compared, and the decision to complement or replace the existing review was made on accumulated data, not on that first impression.

Spend where the risk is

Cheap by default doesn't mean cheap everywhere. Findings get a second opinion from independent judges that don't know which reviewer produced them, and anything severe enough to block someone's work goes to a stronger, more expensive model first, as an appeal. Only a small share of pull requests ever has a blocking candidate, so the expensive model runs rarely and only where a mistake would hurt.

That's the general shape I'd recommend for any LLM pipeline: cheap where volume is high and errors are recoverable, expensive where decisions are final.

A related rule: the review should be about what the change introduced. "This was already broken" is a valid answer, but it has to be proven too, or it becomes the easy way out.

The model is a setting

Because quality depends mostly on the harness, the model is a configuration value, not an architecture decision. Swapping it is one change, and the system was built to make that change safe:

Every run is archived per model, so two models can review the same pull requests and be compared on the same ground.
One fixed, blind judge scores the findings of every model, and of the previous review too, so precision is measured on the same set of claims and nobody grades their own homework.
The judges are audited against a stronger reference judge, measuring how often they agree, so the measuring stick is measured too.
Model versions are pinned and prices live in configuration, because providers change behaviour and prices without asking. An evaluation run catches the drift before it reaches anyone's pull request.

The practical consequence is freedom. When a new model appears that's cheaper or better, trying it is an afternoon of evaluation, not a migration. No vendor gets to decide the cost or the quality of our reviews.

Measure quality, don't argue about it

"The AI review is noisy" and "the AI review is great" are both opinions. I wanted numbers, so the system tracks a few:

Precision: of what a reviewer reported, how much the blind judges confirmed as real.
Relative recall: of all the real issues either reviewer found, how many this one caught. We never know every bug that exists, only the ones someone found, so recall is relative by design.
Fix rate: whether the code a finding pointed at actually changed in a later push. A deterministic, cheap proxy for "this comment was useful".
Cost and time per review, from the run's own records.

The decision rules were written down before the data came in: when the new reviewer is good enough to complement the old one, when it's good enough to replace it (at least as precise, most of the recall, at a fraction of the cost), and when it should be retired. Deciding the bar in advance is what keeps measurement honest.

Silent first, then live

Before the reviewer was allowed to block anything, it ran silently next to the existing review, and I compared the two. That's where the real calibration happened, on real changes, where every adjustment was free. Only once the results held up did it start posting, and only later did it start gating. Within a few weeks it went from an experiment on a laptop to the only AI review gate on several repositories, running as a service on our own infrastructure.

The hardest problems were classic engineering

The problems that took the longest were about state and timing, and the lessons apply to any automated check:

A check that couldn't run reported a neutral result, and neutral counted as passing. A skipped review must never look like a passed one. Fail closed.
An old run finished after a newer push and overwrote fresher results. The newest change has to win, and late results have to be refused.
Re-evaluating an unchanged finding could flip the outcome with nothing fixed, in either direction. Carrying findings across pushes needs explicit, boring rules.

And because the reviewer reads code written by anyone who can open a pull request, it treats that code as hostile: the model has no tools of its own, no secrets within reach, and its output is rendered from validated data, never pasted as is.

The reviewer reviews itself

From the first week, the reviewer reviewed its own changes, and it caught real bugs in its own code. That did more for my trust in it than any benchmark.

If you're building on LLMs, spend less time picking the model and more time deciding what you won't believe until it's proven. Get that right and the model becomes the cheapest, most replaceable part of the system. Models change every few months. The harness is what you actually own.