How catastrophic is your LLM?

Red-teaming an LLM usually looks like this: a human tries a bunch of nasty prompts, sees what sticks, and calls it a day. Amazon Science and a team from the University of Illinois Urbana-Champaign think that's a flimsy way to measure real danger, and they built something called C3LLM to prove it.

The core problem is that single prompts don't capture how people actually manipulate chatbots. Real jailbreaks unfold over several turns, with a user nudging the conversation step by step toward something the model would never say if asked directly. So the researchers modeled entire conversations as a graph, with each node a prompt and each edge a semantic link between related queries. That graph lets them simulate everything from a lone random question to a patient, adaptive attacker who reroutes their phrasing based on how the model responds, a setup they call Adaptive with Rejection, meant to mimic a determined human red-teamer.

Instead of spitting out one pass/fail number, C3LLM runs many simulated conversation paths, has a ChatGPT-based judge flag which responses are catastrophic, and then applies the Clopper-Pearson statistical method to produce confidence bounds on the true attack success rate. That's the real shift here: rather than saying a model failed 12 out of 100 tests, C3LLM says something like, we're 95% confident the true catastrophic failure rate sits between X and Y percent across a huge space of possible conversations.

When UIUC researchers pointed this at frontier models, the gaps were stark. Claude-Sonnet-4 and Amazon's own Nova Premier came out safer, with Nova Premier's guardrails apparently doing a lot of heavy lifting by blocking unsafe content outright. Mistral-Large and DeepSeek-R1 fared worse, and DeepSeek-R1 in particular hit a certified lower bound above 70% for cybercrime-related harm under the simplest attack distribution, a number that should make anyone deploying it for anything sensitive pause.

Amazon has open-sourced the framework, which matters more than it might seem. Safety claims from model makers have long been take-our-word-for-it affairs, benchmarked against whatever prompt sets they chose to test. A reproducible, statistically grounded method that anyone can point at any model changes the incentive structure a little, even if it won't fix the underlying safety gaps by itself.