GPUs on Kubernetes are bring-your-own-everything, starting with the bill
The New Stack Yasmin Rajabi
GPUs on Kubernetes don’t come with the visibility CPU and memory do, so teams have to build the math themselves. That makes billing and blame a lot messier, right where the most expensive hardware sits.
Based on reporting by The New Stack, Yasmin Rajabi — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Kubernetes teams are used to getting a clean read on CPU and memory. Grafana shows what a pod asked for, what it used, and which namespace is eating the most. That playbook falls apart fast when the workload moves to GPUs.
The first problem is visibility. NVIDIA’s DCGM exporter reports at the physical device level, which works fine when a pod owns a GPU outright. But once multiple pods share a card through time-slicing or MPS, the same value gets shown for each one. Three pods can end up looking like they each used the whole GPU, which leaves you staring at 300% of a resource that doesn’t exist.
Then there’s rightsizing. CPU has millicores, bursting, throttling, and OOM kills. GPUs mostly don’t. Kubernetes wants a whole number like nvidia.com/gpu: 1, request and limit must match, and the fractional sharing happens below the scheduler through time-slicing or MIG. So the usual trick of comparing request versus usage and reclaiming slack barely applies. Before anyone can talk about waste, they have to figure out how much of the card a workload actually received.
Guardrails get shakier too. The NVIDIA container runtime listens to NVIDIA_VISIBLE_DEVICES, and the base CUDA images set it to all. Normally the device plugin replaces that with the GPUs a pod was assigned. But if nothing overrides it, or the runtime is set to honor the container’s environment, a pod can see every GPU on the box even if Kubernetes thought it handed out just one. At that point the scheduler’s record and the hardware reality are telling different stories.
Utilization is also less straightforward than the neat CPU percentage people are used to. A GPU can sit at 100% while compute is barely doing useful work, and memory is often the first thing to run out because frameworks like vLLM grab most of the card at startup. The article’s practical answer is to bill on memory, where attribution is reliable, and use compute as a signal rather than a chargeback anchor. That’s the part that matters if teams want to move past showback.
The sharing model changes the math again. Time-slicing, MPS, MIG, and Dynamic Resource Allocation all define “allocated” differently, and the ecosystem around DRA, including autoscalers like Karpenter, is still catching up. That’s before the bill. AWS’s cheapest MIG-compatible A100 is $22 an hour, about $14 on spot, while a comparable CPU-only instance is around $5 an hour. GPUs on Kubernetes are not a simple extension of CPU accounting; they’re a new accounting system with a very expensive mistake rate.
My take — AI-written commentary, not fact-checked reporting
This is the part of cloud infrastructure nobody puts on the slide deck: the GPU is not just dear, it’s bookkeeping with teeth. The industry loves pretending the same Kubernetes habits will carry over, then acts surprised when the bill and the telemetry disagree. Open tooling helps, but on GPUs the defaults are still doing the theater and the operator is left paying for the props.
Read more about this at: The New Stack