Scale isn’t one number: three ways a Kubernetes control plane runs out of room
The New Stack Joe Betz
Three Kubernetes clusters hit the wall in three different ways: nodes, churn, and stored state. The usual ‘add more apiservers’ fix can make memory pressure worse.
Based on reporting by The New Stack, Joe Betz — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Three clusters are in trouble, and they don’t fail in the same way. One is a training fleet that sat around a few thousand nodes for months, then jumped to 15,000. Another never gets that big, but its agent sessions are constantly being created, suspended, resumed, and destroyed, so the write load is brutal. The third is a platform cluster with 200 operators and tens of thousands of mostly idle objects. Two of the three are AI workloads. That’s almost beside the point. The real issue is that they’re running out of room along three different axes, and a fix for one doesn’t help the others.
The old answer is to scale by node count, because that’s the number everyone knows. It’s also why the wrong clusters get tuned for the wrong problem. At 100,000 nodes with a 10-second heartbeat, liveness alone creates about 10,000 writes per second, and the measurements here say lease renewals are roughly 70% of write QPS. In that world, most of the control plane is just nodes saying they’re still alive. But that’s only one kind of pain.
The article breaks the problem into node-bound, churn-bound, and state-bound clusters. Node-bound means the heartbeat flood from huge fleets. Churn-bound means smaller systems with very busy objects, usually custom resources, where the write rate has little to do with machine count. State-bound means clusters where the cost is resident memory, because every apiserver keeps a copy of everything in its watch cache. That is the case where the reflex to add another apiserver backfires, since each new instance replicates the same cache and burns more memory.
The fix is not one trick but a set of seams: storage, node liveness, sharding, and cache residency. Some pieces already exist. KEP-4743 defines the storage boundary, and --etcd-servers-overrides can route built-in resource types to different backends. There’s also a proposed node-facing tier and etcd’s in-memory lease refresh path. KEP-5866 introduces hash-range filtering for API traffic, and there are prototype branches plus a draft KEP update for reducing cache residency. A two-shard prototype reportedly cuts heap by about half per shard, though even with every disableable cache off, an apiserver still carries about 0.9 KB per RBAC object because the authorizer keeps informers around. None of this is a magic fork. The point is more basic: the scale problem is real, but the assembly manual is the thing that’s missing.
My take — AI-written commentary, not fact-checked reporting
Kubernetes scaling has been treated like a single scoreboard for too long, and that’s how people end up buying the wrong fix with a straight face. The useful move now is less heroic and more annoying: stop worshipping node count and start arguing about what the cluster is actually doing. That’s a much less glamorous conference talk, which is probably why it’s the right one.
Read more about this at: The New Stack