How llm-d makes the most of the hardware you already have

IBM Research, Red Hat, Google and other partners are pushing llm-d as an answer to a very specific problem: agentic AI does not behave like a chat app. Coding assistants and other agents keep dragging huge contexts back through the system, reuse the same material over and over, and then spike unpredictably when sub-agents fan out in parallel. That means the bottleneck is often not text generation at all. It is the repeated business of reading, caching and moving context around.

The project’s pitch is simple enough: if enterprises want open models on their own hardware, they need software that squeezes more out of the GPUs they already have. IBM says llm-d was built for exactly that. In a recent demonstration, the team ran GLM-5.2, a roughly 753-billion-parameter open-weight mixture-of-experts model with about 39 billion active parameters, on 544 NVIDIA H100 GPUs and used it to serve agentic traffic at production scale.

The numbers are doing a lot of the talking here. On benchmark workloads with hundreds of concurrent sessions and heavy context reuse, the system peaked at more than 6.6 million output tokens per minute and handled up to 3,000 concurrent coding agents with zero preemptions. IBM says that, at current cloud rental rates, self-hosting that setup on H100s costs 5 to 10 times less per token than equivalent commercial API pricing, with the biggest savings on input-heavy agentic traffic.

What makes llm-d interesting is not one trick but the stack of them. It uses prefix-aware routing to send requests to servers that already hold the right cache, tiered KV-cache management to push useful prefixes into CPU DRAM, peer-to-peer cache sharing when the match sits on another machine, wide expert parallelism with data-parallel attention, separate prefill and decode pools, and multi-token prediction on top. IBM says those pieces together improved throughput and cut time-to-first-token, and in one benchmark the full routing stack lifted local prefix hit rate to 73.18% from 44.46%.

The broader point is less glamorous and more useful: big gains still come from software. llm-d is now an open-source Cloud Native Computing Foundation sandbox project, which makes this feel less like a lab demo and more like a bet that enterprise AI will be won by people who know how to keep old GPUs busy, not just by people buying the newest ones.