🔮 Unbounded self-improvement and its limits #599

The cleanest argument against endless recursive self-improvement is boring in the best way: time. Philosopher Toby Ord says the key constraint is generation time, the full loop from one AI system helping build its successor to the next system being ready. If that loop can’t collapse to nothing, then the dream of intelligence shooting upward without bound starts to look less like destiny and more like a very hard engineering problem.

Ord’s point is not that self-improvement is impossible. It is that the conditions for extreme recursive self-improvement are brutally demanding. Experiments take time. Training takes time. Making chips takes time. Even if AI sped everything up, the cycle still has to exist somewhere in the real world, and the real world keeps a schedule.

Then physics shows up and ruins the fantasy in the usual ways. Communication can’t outrun the speed of light. Information in a finite space is capped by the Bekenstein bound. Irreversible computation carries an energy cost under Landauer. Ord’s conclusion is that unbounded RSI is mathematically imaginable but practically and theoretically boxed in.

The more immediate action is elsewhere: open-weight models are getting traction in business, and not just for hobbyists or cost cutters. Vercel saw token share for open weights hit 62% in a single day, up from 28% two months earlier. Thomson Reuters has already built its first in-house model on Qwen to reduce costs, and Bridgewater, working with Thinking Machines, reportedly beat every frontier model it tested on an internal information-filtering task with a fine-tuned open Qwen model, at about one-fourteenth the inference cost of the best closed model.

There is also a hardware version of the same story. OpenAI’s chip, Jalapeño, was built with help from its own models, which wrote kernels and trimmed about 10% from one main compute block. OpenAI says it took around 16 months from first hire to tape-out, and the chip outperforms comparable Nvidia silicon by 1.5–1.9x on tokens per megawatt at peak throughput. That does not prove AI will self-improve forever. It does suggest the process is already learning to speed up the machinery around it.