Introducing EVMbench

Smart contracts have a nasty habit of getting drained the moment a coder misses one edge case. OpenAI teamed up with Paradigm, the crypto-focused venture firm, to build EVMbench, a benchmark that puts AI agents through the same paces a security auditor would face: spot a high-severity vulnerability in Ethereum Virtual Machine code, patch it correctly, or in some cases exploit it to prove the flaw is real.

That three-part structure is the interesting bit. Most AI coding benchmarks stop at "can the model write working code." EVMbench goes further by asking whether an agent can think like both a defender and an attacker on the same contract. Detection alone is fairly easy to fake — a model can flag everything as suspicious and get partial credit. Requiring an actual working exploit or a clean patch forces the agent to demonstrate it understands the mechanics of the bug, not just its vibes.

The timing tracks with where crypto security actually hurts. High-severity smart contract bugs have cost the industry billions over the years, and the exploits keep coming even as auditing firms multiply. Paradigm's involvement signals this isn't an academic exercise dreamed up in isolation — it's coming from people who've watched real protocols get rekt and want a rigorous, repeatable way to measure whether AI tools are actually useful in that fight, rather than just another chatbot wrapped around a linter.

What OpenAI doesn't claim, notably, is that current models ace this thing. Benchmarks like this tend to exist precisely because today's agents are mediocre at the task, and the gap between finding a bug and safely exploiting it to confirm impact is where a lot of automated tools fall apart. EVMbench reads less like a victory lap and more like a scoreboard OpenAI expects to keep climbing over the next few model generations.