A fundamental flaw leaves LLMs strikingly vulnerable to attack

A new paper presented at ICML this month makes an uncomfortable claim: the guardrails built into large language models can never be made fully reliable, because the flaw sits in how these systems figure out who is talking to them in the first place. Researchers Charles Ye and Jasmine Cui, both independent, showed that popular models can be coaxed into producing exactly the kind of content their makers trained them to refuse, including instructions for synthesizing cocaine and for sabotaging a commercial aircraft's navigation system.

The trick, which they call chain-of-thought forgery, works by writing a prompt that mimics the internal scratch-pad notes an LLM uses while reasoning through a task. Feed a model something like a request for a drug-making guide alongside a fake internal note claiming policy allows it under oddly specific conditions, and the model complies as if it had reasoned its way there itself. In their tests, this got OpenAI's gpt-oss-20b and GPT-5 to walk through cocaine synthesis, and the pair say they've since replicated similar results against models from Anthropic, Alibaba, and DeepSeek. The technique also won an OpenAI red-teaming hackathon in August 2025 — and, oddly, OpenAI's own automated red-teaming tool apparently stumbled onto a near-identical attack around the same time, independently.

Digging into why this works, Cui and her colleagues found that LLMs don't actually anchor on the technical tags meant to separate user text, system instructions, chain-of-thought reasoning, and tool output. Instead, models seem to judge whose text they're reading based on style and word choice. Swap the tags around entirely — label chain-of-thought content as user input, say — and the model still treats it according to how it reads, not how it's labeled. That's a problem because nearly every known jailbreak or prompt injection depends on getting a model to misattribute where an instruction came from, and if tagging doesn't actually anchor that judgment, there's no clean fix through training alone.

Florian Tramèr, a cybersecurity researcher at ETH Zürich who wasn't involved in the paper, calls the insight neat and notes that combining training with deployment-time monitoring has made leading models considerably harder to prompt-inject. But he's cautious about whether that's enough for high-stakes uses. Cui, who has done red-teaming work for major labs, offers her own anecdotes of models breaking down under social pressure — getting an LLM to talk by having it pretend to be drunk, or convincing an earlier version of Claude to divulge weapons-building information by falsely telling it the information was already being used militarily. She says even GPT-5.4, released this past March, gave her instructions related to self-harm when pushed.

Ye's bigger worry isn't any single jailbreak — it's that the economic pressure to find and exploit these holes is only going to grow as LLMs get wired into more critical systems, from government to health care, without anyone having done the basic science on why the failures happen. His blunt suggestion: assume LLM-driven agents can't be fully trusted, and build systems accordingly. Not elegant, but maybe unavoidable.