Lightbits set to release KV cache engine to boost GPU performance
SiliconANGLE Paul Gillin
Lightbits has launched Inferra, a cache engine for AI GPUs. It aims to cut stalls, raise throughput, and pack in more users on the same hardware.
Based on reporting by SiliconANGLE, Paul Gillin — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Lightbits Labs has made Inferra generally available, betting that one of AI’s ugliest bottlenecks is now big enough for a business. The software engine moves key-value cache data beyond the small slab of high-bandwidth memory attached to GPUs, with the goal of improving both inference speed and the economics around it.
The company is best known for inventing NVMe over TCP, but this product is aimed squarely at neocloud providers and enterprises running large language models with long context windows or lots of simultaneous sessions. Inferra manages cache across GPU high-bandwidth memory, DRAM and NVMe storage, then uses predictive prefetching to stage data before the GPU asks for it. That matters because KV caches grow as a conversation or document gets longer, and once they stop fitting neatly in GPU memory, systems either go fishing in slower storage or burn time recomputing what they already knew.
Lightbits says the payoff can be dramatic. In some long-context workloads, it claims Inferra can cut time to first token by more than 100-fold, support context windows of more than 10 million tokens on commodity hardware and raise concurrent session density by more than 16 times. Those numbers come from company benchmarks and extrapolations, not independent tests. The company also says it has seen cache hit rates near 99.9% in most scenarios it has tested.
There’s a catch, of course: pushing cache off the GPU adds latency of its own. Lightbits says it gets around that by predicting demand early enough to have data waiting. Inferra also includes quality-of-service controls, encryption, tenant isolation and support for moving cached data when a session shifts to another GPU cluster. The company says its strongest use cases are retrieval-augmented generation, AI agents, long prompts and heavily shared GPU services, while oversized private clusters with few users stand to gain less.
Lightbits is already running production pilots and is first targeting neocloud providers, which have a sharper incentive than hyperscalers to squeeze more utilization and margin out of their hardware. The company will demo Inferra at the AI Infra Summit in Santa Clara next week.
My take — AI-written commentary, not fact-checked reporting
This is the kind of unglamorous infrastructure work that actually moves AI forward. Everyone wants bigger models; fewer people want to pay for GPUs sitting around waiting on memory. Cache tricks won’t fix bad economics, but they do expose how much of the AI stack is still a very expensive waiting room.
Read more about this at: SiliconANGLE