IBM-backed llm-d framework adds controls for running AI models in-house
SiliconANGLE Paul Gillin
IBM and partners shipped llm-d 1.0, a tool for running AI models in-house. It’s built to keep crowded, agent-heavy systems fast without paying a cloud tax.
Based on reporting by SiliconANGLE, Paul Gillin — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
IBM and a group of open-source contributors have put version 1.0 of llm-d out into the world, with a clear pitch: make it easier for enterprises to run large AI models on their own infrastructure without the usual operational mess. The framework is built for Kubernetes and is meant to coordinate model serving across compute, not just launch it and hope for the best.
The timing is not accidental. The release lands ahead of the PyTorch Conference North America in San Jose, and it goes after a problem that gets uglier once AI agents start hammering systems with more requests, more retries and more back-and-forth. Llm-d 1.0 adds controls for quality of service across multiple tenants, zero-downtime operations and serving patterns aimed at agents rather than simple one-shot prompts.
That matters because the hard part of inference isn’t the demo. It’s the production traffic, the hardware hiccups, the software upgrades and the people all competing for the same resources. IBM says the new release adds flow controls and prioritization so bursts of agent activity don’t swamp services that need fast responses, like coding assistants.
The project also leans hard into cache reuse. Requests can be routed to replicas that already hold relevant context, while cached information can move among GPU memory, CPU memory and storage when accelerator capacity gets tight. IBM says those techniques were already there in some form, but 1.0 ties them to the operational layer and adds support for splitting prompt processing from response generation, optimizing batches and spreading mixture-of-experts models across accelerators.
IBM points to some sharp numbers. In one agentic coding test, its router captured 97% of available cache reuse. In another demonstration with Red Hat, the companies said they ran 3,000 software-engineering agents at once on self-hosted infrastructure for five to 10 times less than using the same open model through a hosted service. Llm-d works with engines such as vLLM and SGLang, and the Cloud Native Computing Foundation hosts it as a sandbox project.
My take — AI-written commentary, not fact-checked reporting
This is the part of enterprise AI that actually matters: boring controls, not flashy demos. IBM is betting that the real win is making in-house inference behave like a normal critical service, which is exactly where the cloud sales pitch starts looking expensive. The open-model crowd keeps saying freedom matters; here’s the bill arriving with line items for uptime, cache reuse and everyone sharing the same machine.
Read more about this at: SiliconANGLE