AI jobs are waiting while GPUs sit idle. Kubernetes has a new plan.
The New Stack Nick Lucchesi
Kubernetes is getting new tools for AI workloads, from pod-restart debugging to smarter GPU scheduling. The weird part: jobs can wait while expensive accelerators sit idle.
Based on reporting by The New Stack, Nick Lucchesi — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Kubernetes is being pushed hard by agentic AI, and the pressure is showing up in familiar places: stuck pods, idle GPUs, missing context, and systems that can’t quite tell what broke. The latest Road to KubeCon roundup reads like a snapshot of a platform straining to keep up with workloads it was never built for, then slowly building the plumbing to cope.
HPE is looking at a very concrete headache: pod restarts in large AI and machine learning jobs running on HPE Private Cloud AI. Guoping Jia, a system software engineer at HPE, says Kubernetes offers some troubleshooting commands, but not enough to avoid long manual investigations. His fix is to automate the collection of restart details with the open-source k8s-pod-restart-info-collector and push the diagnostics into Slack.
AWS is focused on a different bottleneck: the gap between Kubernetes’ scheduler and autoscaler when GPUs are involved. Sajjan Gundapuneedi and Neelendra Bhandari describe the ugly result of that mismatch plainly. Pods stay pending while pricey accelerators sit unused. They point to DRA-aware autoscalers, workload-aware scheduling, and provisioning requests as the near-term answer, while a SIG-Autoscaling proposal sketches a tighter API conversation between the two loops.
On the inference side, llm-d hit v1.0 at PyTorch Conference North America. Carlos Costa says that marks the point where open source AI serving moves from experimental work to infrastructure that can be depended on in production. He says IBM’s demo of an agent factory exploring quantum error correction hardware ran at 11.5x lower inference cost on open models with llm-d than on an all-proprietary setup, and that the system pushed cache reuse to about 98%.
The bigger theme running through the piece is context. Roy Libman argues that coding agents can already investigate Kubernetes clusters, but they often miss the clues that matter. In his benchmark of 50 injected faults, Claude Code with read-only MCP context diagnosed faster than with shell access alone. Kristiyan Ivanov makes the same point from the storage side: metrics count, but they don’t explain, and a lot of state never survives the scrape. That is becoming the real job now — not just running AI at scale, but teaching Kubernetes how to see it.
My take — AI-written commentary, not fact-checked reporting
This is the part of the AI boom nobody gets to skip: the mess gets expensive before it gets elegant. Everyone loves a talk about agents until the pods pile up, the GPUs idle, and the logs turn into archaeology. Kubernetes isn’t failing the moment; it’s doing the industry’s housekeeping, which is less glamorous but far more useful.
Read more about this at: The New Stack