Inference is giving AI chip startups a second chance to make their mark
The Register ● Covered by 3 sources
AI chips are splitting into two jobs: crunching prompts and spitting out answers. That split just handed startups a rare crack at Nvidia's throne.
Based on reporting by The Register — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Training used to be the whole game, and Nvidia's GPUs owned it outright. But the industry has quietly tipped toward inference — actually running the models people use every day — and that workload behaves nothing like training. Batch jobs, chatbots, coding agents: each wants a different balance of raw compute, memory, and bandwidth. That variety is exactly the crack a handful of chip startups have been waiting for.
The clearest proof came in December, when Nvidia paid roughly $20 billion to acquihire Groq. Groq's LPUs, stuffed with SRAM instead of conventional memory, could spit out tokens faster than any GPU on the market. Their weakness was compute headroom and chip tech that hadn't kept pace. So rather than replace its GPUs, Nvidia split the inference pipeline: GPUs handle the compute-heavy prefill stage, while Groq's LPUs take over the bandwidth-hungry decode stage where tokens actually get generated.
That pairing didn't stay exclusive for long. Days after Nvidia's GTC event, AWS rolled out its own disaggregated setup, using homegrown Trainium chips for prefill and Cerebras' wafer-scale monsters for decode. Intel followed with a reference design pairing its own GPUs with SambaNova's RDUs for the same split. Three different giants, same architecture logic — and in each case, the startup's contribution landed on the decode side, where SRAM's speed, not its limited capacity, is the whole point.
Lumai wants to break that pattern from a different angle entirely. The UK startup this week detailed its optical inference chip, which does matrix multiplication with light instead of electrons, slashing power draw compared to digital silicon. Its Iris Tetra roadmap targets an exaOPS of performance inside a 10kW budget by 2029. Right now the hybrid electro-optical design only handles billion-parameter models like Llama 3.1 8B and 70B, and it's pitched as a standalone GPU alternative for compute-bound batch jobs, with prefill duty planned for later. Early as it is, Lumai has already started letting neoclouds and hyperscalers test the hardware.
Not everyone thinks splitting the job in two is progress. Tenstorrent's Jim Keller, unveiling the company's RISC-V-based Galaxy Blackhole platform this week, dismissed the whole disaggregated trend as an arms race of accelerators accelerating accelerators — CPUs feeding GPUs, GPUs feeding TPUs, TPUs feeding LPUs. He argues that stacking specialized silicon this way produces brittle systems that can't keep up as models and use cases shift, and that Tenstorrent's simpler, more general approach is the sturdier bet long-term.
My take — AI-written commentary, not fact-checked reporting
I'll admit the prefill-decode split is clever engineering, but it also smells like the industry building Rube Goldberg machines because nobody wants to admit GPUs aren't universally optimal anymore. Keller's crankiness about accelerators accelerating accelerators is the most honest quote to come out of this whole cycle. My money's still on the boring, general-purpose architecture outlasting the exotic optical and SRAM tricks — chip history rewards whoever's cheapest to keep compatible, not whoever's fastest in a press release.
Read more about this at: The Register