Stealing Reasoning Traces from Proprietary LLM APIs
Simon Willison’s Weblog Simon Willison ● Covered by 3 sources
Researchers found a way to pull hidden reasoning out of some AI APIs. The same “thoughts” could be replayed across models, then jailbreak the weaker one.
Based on reporting by Simon Willison’s Weblog, Simon Willison — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
A new paper with the wonderfully ominous domain stolen-thoughts.com says some major model APIs were handing clients encrypted chain-of-thought blocks that could be reused across sessions, users, and even models. The companies named are Anthropic, OpenAI, and Google. The basic trick was simple enough to sound absurd: take a reasoning trace from a stronger model, feed it into a weaker sibling, and use that as a lever to get the hidden reasoning back in readable text.
The paper shows how this could be done against OpenAI’s API too, with a request that asks for step-by-step reasoning and includes reasoning.encrypted_content in the response. The returned output contains a reasoning object with encrypted_content, a blob that looks like gibberish until it is reused elsewhere. According to the authors, models in the same family were using the same encryption key, which made the whole thing much more fragile than it should have been.
Claude Haiku 4.5 is described as the easiest target. One prompt asked it to continue, transcribe the attached reasoning verbatim inside <thinking-copy> tags, and then set an assistant turn prefix of <thinking-copy>. That prefix trick was later removed in the 4.6 models, but it still worked in Haiku 4.5 when the paper was written up.
The ugly part is not just that the raw reasoning could be exposed. The paper also describes a prompt-injection style attack where a model is nudged into “thinking” about exfiltrating data, such as uploading a file to a remote server, and that reasoning is then replayed into another model. The authors say model providers acknowledged their report and that they were no longer able to launch the same attacks afterward. They also say the recovered traces were clearly not written for human eyes, which is probably the understatement of the week.
My take — AI-written commentary, not fact-checked reporting
This is the awkward truth behind sealed-up “reasoning” features: if the model can see its own thoughts, someone will eventually teach it to spill them. Closed APIs don’t magically make the internals safe; they just make the leak feel premium. The industry keeps bolting a stronger lock onto a door that was left open for debugging in the first place.
Read more about this at: Simon Willison’s Weblog
