Will It Mythos?

swelljoe.com · 2 months ago · 18

A researcher built a benchmark to test whether Anthropic's Mythos security vulnerability detector is uniquely capable compared to other AI models, using nine confirmed bugs that Mythos had previously found. The benchmark tested 40+ models by asking them to identify and describe bugs in real code repositories without hints, with results showing no model performed perfectly and costs ranging from under $1 to over $100 per test run. The findings indicate that while Mythos appears strong, several cheaper models like Qwen 3.6 and DeepSeek are competitively capable, suggesting Mythos's advantage may not be as unique as Anthropic's security restrictions imply.