The distillation panic

Words matter more than people think, and the AI policy world just picked a bad one. Anthropic recently published a blog post accusing three Chinese labs of running 'distillation attacks' against its API — essentially jailbreaking or spoofing their way into extra reasoning data meant to stay hidden. That's a real problem worth stopping. But calling it a distillation attack is going to poison the term distillation itself, and distillation is not some shady hacking technique. It's a bog-standard part of how modern language models get built.

Distillation just means training a weaker model on a stronger model's outputs. Frontier labs do this to themselves constantly, shrinking flagship models into cheaper versions for customers. Nvidia's open Nemotron models are substantially distilled from Chinese open-weight models. Ai2's Olmo models mix distillation from both open and closed sources. Even Elon Musk admitted under oath in the OpenAI trial that xAI has partly distilled from OpenAI, adding, matter-of-factly, that AI companies generally distill each other. This is the water everyone swims in, not some fringe misbehavior.

What Anthropic actually documented is narrower and worse than plain distillation: labs circumventing API terms of service to extract information — like internal reasoning traces — that was never meant to be exposed. That's jailbreaking or abuse, full stop, and it deserves its own name and its own response. Lumping it in with distillation broadly is either sloppy or, more cynically, useful for framing a policy fight as bigger than it is.

And the policy fight is already moving fast. There's a bill out of committee in Congress, an executive order in motion, and oversight hearings aimed at U.S. companies that build on Chinese open-weight models. The danger isn't that lawmakers will literally outlaw open models — nobody's proposing that directly — but that vague, sweeping rules create enough legal risk and bureaucratic friction that small open-source teams simply can't comply, while big players absorb the cost fine. Chinese open-weight models have real community traction now; there's no six-month substitute waiting in the wings if U.S. groups get scared off using them.

The people who'd actually get hurt are academics and small companies serving the unglamorous long tail of AI applications, not the Chinese labs the rules are supposedly targeting. Those labs will likely keep doing what they're doing regardless — we've already watched this pattern play out with copyright and multimedia models, where U.S. companies play it safe and Chinese ones don't. There's even an argument, one I heard from Kevin Xu at Interconnected Capital, that heavy dependence on distillation is actually a crutch keeping Chinese labs from developing frontier techniques of their own — cut it off and you might sharpen a rival rather than dull it. Worth sitting with, even if it doesn't change the near-term calculus.