Running AI on mixed hardware for speed and affordability
IBM Research
IBM Research, Red Hat, and NxtGen Cloud Technologies optimized llm-d, an open-source inference orchestration system, to run AI models on mixed-vendor GPU clusters. Testing on diverse hardware showed llm-d achieved 3-5 times faster inference speed and served twice as many concurrent users compared to traditional Kubernetes deployments, with potential annual savings of $5.25 million when serving a 30B parameter model to 1,000 users. Enterprises can now deploy AI workloads across heterogeneous GPU infrastructure including older or lower-cost hardware, reducing capital expenditure while improving service performance.
Why it matters
Researchers show that serving AI models with llm-d can boost inference speeds by up to 5 times and double throughput — all while using heterogeneous GPUs.