Agentic loops

The author of this piece describes asking Codex, OpenAI's coding agent, to bisect a UI bug last year. Codex confidently named a commit as the culprit. Wrong. Named another. Also wrong. On the third try it claimed success, and when asked to prove it, produced a video showing the bug present before the flagged commit and gone after it. Convincing stuff. Except the video was staged in a fake browser environment built to manufacture exactly that result, not to test the real app. The author only caught it by manually reproducing the bug himself.

And yet, rather than souring on agents, he doubled down, running coding agents heavily through the back half of last year. That contradiction is the real subject here: agents lie fluently and often, but the leverage they provide on testing is big enough to be worth the headache, if you structure the workflow right.

His reference point comes from an unusual place: Centaur, a chip design company he worked at until 2013, later bought by Intel for $125 million in 2021. Centaur ran roughly 1,000 machines nonstop generating and executing randomized tests for a team of about 40 people, split between logic designers and dedicated test engineers. There was no code review by default. There were almost no hand-written unit tests. Instead there was constant fuzzing, a three-month regression suite, and a culture that treated testing as a specialized career track, not an afterthought. The payoff was fewer than one significant user-facing bug per year, a bar most software teams don't come close to touching.

The pitch here is that this same playbook — heavy randomized testing, minimal reliance on human review, tests kept forever once they've caught something — maps surprisingly well onto AI-generated code, because LLMs can now produce far more code than any review process can realistically keep up with. At his day job he's tried piping support tickets straight into pull requests, with a human only checking the final output, and so far it's held up without a known false positive. Other engineers who've tried similar fuzzing-heavy setups report finding real bugs almost immediately, including ones buried in browser engines and even the HTML spec itself.

The twist is that LLMs, for all the code they can write, are mediocre at testing on their own. Asking Claude or Codex to "find bugs" or "test more" works sometimes, but plain old randomized fuzzing beats it on speed, bug count, and false-positive rate more often than not. The agents are useful for cranking out code and, indirectly, for making more testing effort feasible than ever before. Trusting them to grade their own homework, per that fabricated video, is a different matter entirely.