These startups are chasing the next big thing in LLMs
MIT Technology Review Will Douglas Heaven
Startups are trying to replace the transformer, the engine behind today’s big AI models. They say the next wave could be faster, cheaper, and better at huge tasks transformers fumble.
Based on reporting by MIT Technology Review, Will Douglas Heaven — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
The transformer has had a remarkable run. Google researchers introduced it in 2017, and nine years later it sits inside every major large language model on the market. But the same thing that made it so powerful — dense attention over long strings of text — is now looking like a wall. The bigger these models get, the more that wall matters.
That is why a crop of startups is trying to build what comes next. MIT Technology Review calls the category LLMs+, and the pitch is simple enough: keep the useful parts of large language models, ditch the parts that waste computation, and maybe find room for new kinds of reasoning along the way. The problem is also simple: dense attention scales badly. A 10,000-word document can require 50 million multiplications, and that helps explain why OpenAI is set to spend $50 billion on computing this year, according to Greg Brockman. The International Energy Agency says data-center electricity use will double by 2030.
Some of the new companies are trying to change attention itself. Subquadratic, in Miami, says its sparse-attention model, SubQ, matches top mainstream LLMs on a handful of tasks including search and coding. Manifest AI, based in San Francisco, is taking a different route with a system it calls power retention, which keeps only the most relevant information and rolls the rest forward in a summary. It has adapted the open-source coding model StarCoder into PowerCoder, and says its Brumby model rivals some versions of Alibaba’s Qwen.
Liquid AI, an MIT spinout in Cambridge, Massachusetts, is building hybrid liquid foundation models that combine transformers with liquid neural networks. Its models are designed to be smaller and less energy-hungry, can run on low-powered chips and even a Raspberry Pi, and have been downloaded almost 34 million times, according to cofounder Ramin Hasani. Inception, in Palo Alto, is pushing diffusion-based text generation, which lets a model produce whole blocks of text at once. Its Mercury 2 is claimed to match some GPT-4 models while running 10 times faster. And Pathway, also in Palo Alto, wants models that reason beyond language altogether, using state spaces instead of standard attention. Its Dragon Hatchling beat more than 97% of a sudoku benchmark, while several leading models from top labs failed to solve any.
None of this means transformers disappear tomorrow. But the center of gravity is shifting. The most interesting fight in AI right now is no longer about who can bolt another feature onto the old architecture. It’s about who can replace the old architecture without making the whole thing worse.
My take — AI-written commentary, not fact-checked reporting
The transformer had a great party, but it is starting to spill drink on the carpet. The sensible bet now is not on one magical replacement, but on companies that stop worshipping the old stack and build models around the job, the budget, and the hardware. That is a less romantic story, which is usually how real progress looks before the hype machine finds it.
Read more about this at: MIT Technology Review