Under $4 a PR: Designing an AI Code Reviewer Engineers Actually Trust

AI code review has a trust problem. The first week, engineers are curious. By the third week, the bot has flagged forty things nobody cares about, invented two bugs that don't exist and missed the one that did. It gets muted. We built an enterprise code-review platform that went the other way — scaling across 50,000+ repositories and 350K+ pull requests a month at under $4 median cost per PR. Here's what mattered.

Review against your standards, not the internet's

A general-purpose model reviews code the way a stranger would: lots of opinions, none of them yours. We curated internal engineering standards and made every finding map to a specific rule. If a comment can't cite a rule, it doesn't get posted. That one constraint did more for signal-to-noise than any prompt tweak.

Wrap the model in determinism

We used deterministic methods around the model to keep hallucinations out: verifying that referenced lines and symbols exist, deduplicating findings, and checking claims against the actual diff. The model proposes; deterministic code disposes.

Make everything swappable

• The harness, the model, the rules and the agents were all independently swappable.
• Rules could be scoped by language, org, repository and use case.
• A self-service pipeline let teams test new rules against historical PRs before turning them on.

Modularity is what let us ride every model upgrade without a rewrite.

Gate every change on evals

We tracked offline evals for accuracy, false-positive rate and dissatisfaction rate, and nothing shipped if it regressed them. In production we closed the loop with inline feedback on comments. False-positive rate is the metric that predicts whether engineers keep listening.

Precision beats recall in code review. One wrong comment costs more trust than ten missed nitpicks earn.

Cost and speed are product features

A multi-model approach sent cheap work to cheap models and saved the expensive reasoning for the hard parts. Serverless infrastructure with distroless images kept the footprint small and patching easy, and a roughly ten-minute SLA to first review meant feedback landed while the author still had context.

Beyond review

Once the headless agent harness existed, it became a platform: test-case generation, documentation refresh, gap analysis and change-impact analysis all reused the same plumbing. Build the platform once; ship use cases forever.