Experiments with AI Code Review

Wealthfront's engineering team spent years chasing a simple idea: what if an LLM could catch the bugs that slip past both tests and human reviewers. It started with a paranoid experiment in 2023, pasting a broken diff into GPT-3.5 after a postmortem and watching it instantly spot a missing SQL filter that two separate review processes had missed. That one moment turned into a multi-year build-out of an internal review harness called Iris, and the company's writeup of how it evolved is a rare, detailed look at what actually works when you throw serious engineering at AI code review instead of just bolting on a chatbot.

The early versions were rough. A cheap model tasked with 'find relevant documents' across a 20-million-line, 350-repo codebase just spiraled into endless rabbit holes, because everything looked relevant to it. The first real system paired Gemini 2.5 Pro as a lead reviewer with swarms of Gemini 2.5 Flash research agents that scored file relevance and fed findings back up the chain. It worked, catching non-obvious bugs through wide codebase correlation, but the signal-to-noise ratio was bad. Ask a model to find problems and it will find problems, real or invented, and Wealthfront found that even flawless pull requests were getting dinged with pointless comments.

The fix that actually moved the needle was turning review into an adversarial process. Anthropic's Opus 4 now runs the show, but when it flags a potential issue, it spins up two competing sub-agents: GPT-5 plays prosecutor, actively hunting for evidence the bug is real, while Gemini 3.0 Flash plays defense, arguing it isn't. Opus then weighs both arguments like a judge before deciding what to write up. That plaintiff-defendant-judge structure, plus smaller tricks like letting Opus draft comments as freeform text before forcing it into a rigid JSON schema, and stripping dozens of bespoke tools down to a single read_file command backed by a shared sandboxed filesystem, made a measurable difference.

The numbers back it up. Engineers rate every AI comment one through five, and under the old system ratings formed a bell curve centered on a mediocre three. After the antagonistic redesign, ratings shifted into a lopsided distribution dominated by fours and fives, with almost no ones or twos. Most pull requests now get zero comments, and the ones that do get flagged tend to matter. Each review costs about four dollars and takes ten minutes, a price and timeline Wealthfront was happy to pay from day one when AI review startups were still chasing 25-cent, three-minute reviews.

What's notable is that Wealthfront isn't trying to kill human review with any of this. AI review runs after a developer's self-check but before peer review, and it never blocks a merge, only supplements it. The company frames this as additive rather than a replacement, though it does concede that human nitpicking is basically over: if a rule can be written down, an AI can enforce it. Whether that framing holds up as models keep changing every year is an open question even Wealthfront admits it can't answer, but the engineering discipline behind Iris is a useful blueprint for anyone trying to move past demo-ware toward AI tools that actually earn developer trust.