Patterns for Building Cybersecurity Evals

There's a quiet arms race happening in AI security research, and it's less about jailbreaks and more about whether a language model can find a real bug, write a working exploit, and pull off an attack end to end. A recent roundup of benchmarks — Cybench, CVE-Bench, CyberGym, and a handful of others — lays out the shared architecture behind these tests: a sandboxed target running in Docker, a tunable amount of hint information (from nothing at all, mimicking a zero-day, up to a full patch diff for the one-day scenario), a toolkit of shells and debuggers, and a grader that checks for something concrete like a hidden flag string or a sanitizer crash.

Cybench is the closest thing to a hacker's obstacle course. It throws 40 professional capture-the-flag challenges at models, sourced from competitions like HackTheBox and SekaiCTF, with difficulty measured by how long it took the fastest human team to solve each one. Claude 3.5 Sonnet led the pack at a 17.5% success rate working unguided, and o1-preview hit nearly 47% when given subtask hints. But every model tested hit a wall on anything that took a skilled human team more than 11 minutes to crack. The failure mode was telling: agents kept losing the thread on multi-step attacks, like forgetting to stitch together truncated message pairs needed for a length-extension attack.

CVE-Bench takes a different tack, pointing agents at 40 actual critical vulnerabilities pulled straight from the National Vulnerability Database, each one rated 9.0 or above on the severity scale. Instead of grading the method, it grades the outcome — did the agent knock the app offline, steal data, or get admin access, using any of eight defined attack goals. Even here, with real CVSS-critical bugs, the best setup (a hierarchical