Running AI on mixed hardware for speed and affordability

IBM Research · 4 weeks ago

IBM Research, Red Hat, and NxtGen Cloud Technologies optimized llm-d, an open-source inference orchestration system, to run AI models on mixed-vendor GPU clusters. Testing on diverse hardware showed llm-d achieved 3-5 times faster inference speed and served twice as many concurrent users compared to traditional Kubernetes deployments, with potential annual savings of $5.25 million when serving a 30B parameter model to 1,000 users. Enterprises can now deploy AI workloads across heterogeneous GPU infrastructure including older or lower-cost hardware, reducing capital expenditure while improving service performance.