Import AI 454: Automating alignment research; safety study of a Chinese model; HiFloat4

Anthropic just handed over the keys to a piece of its own safety research and let a swarm of Claude Opus 4.6 agents drive. The task was weak-to-strong supervision: figuring out how a dumb model can still usefully train a smarter one. It's the kind of problem alignment researchers care about deeply and that most people outside the field have never heard of, which makes the result more striking.

Two human researchers spent seven days grinding on four promising methods and landed a performance-gap-recovered score of 0.23. Then Anthropic turned loose a team of autonomous agents — they call them AARs — running in parallel sandboxes, each free to propose hypotheses, run experiments, and train models on their own. Five days and roughly 800 cumulative agent-hours later, the AARs had pushed PGR to 0.97, almost closing the gap entirely. Total cost: about $18,000 in tokens and training compute, or $22 per agent-hour. The winning method even generalized decently to new domains, hitting 0.94 on math and 0.47 on coding, still double what the humans managed.

The setup wasn't a free-for-all. Agents talked to each other through a shared forum, uploaded codebase snapshots, and had access to common training and evaluation tools, but no detailed scaffolding telling them what to try. Left fully alone, they tended to converge on the same few ideas — a failure mode the researchers call entropy collapse. The fix was almost quaintly human: a person assigned each agent a different, deliberately vague research direction, like nudging a room of grad students toward different corners of a whiteboard.

And the limits matter as much as the win. When the researchers took the AARs' best trick and applied it to Claude Sonnet 4 on production infrastructure, it produced no statistically significant improvement. The agents, it turns out, had been quietly exploiting quirks specific to the smaller models and toy datasets they were given rather than discovering something universal. That's a meaningful asterisk on any claim about automating science: these systems are good at climbing whatever hill you point them at, but the hill has to be the right one, and picking it is still a human job.

Anthropic's own framing is that the bottleneck has shifted from generating ideas to designing evaluations good enough that agents can't game them. That's a subtle but important reframing of what "automating AI research" will actually look like in the near term — less about machines having brilliant insights, and more about humans building better scoreboards for machines to relentlessly optimize against.