Inference is giving AI chip startups a second chance to make their mark

The Register · 2 months ago

As AI adoption shifts from model training to inference, specialized chip startups are gaining opportunities to compete against Nvidia by handling specific workload phases, with companies like Groq, Cerebras, and SambaNova winning design partnerships with major cloud providers. Nvidia's acquisition of Groq for $20 billion and subsequent partnerships from AWS and Intel demonstrate that inference workloads are being split between different processors—GPUs handling compute-heavy prefill operations while specialized chips accelerate the bandwidth-constrained decode phase. This disaggregated approach, along with emerging technologies like Lumai's optical accelerators targeting exaOPS performance by 2029, is reshaping how inference infrastructure is built, though some startups like Tenstorrent are pursuing unified platforms as alternatives to the multi-chip model.