Scaling Kubernetes to 7,500 nodes

Scaling a Kubernetes cluster from a few hundred nodes to 7,500 isn't a matter of just adding more machines and hoping. OpenAI has quietly done exactly that, and the number matters less as a vanity metric than as proof that a single, unified infrastructure can serve two very different kinds of workloads at once.

On one end you have massive training runs, the kind behind GPT-3, CLIP, and DALL·E, jobs that chew through enormous compute for weeks at a time. On the other end you have small, fast, iterative research experiments, the sort that produced the Scaling Laws for Neural Language Models paper, where researchers need quick turnaround and don't want to wait in line behind a giant job. Getting both to coexist on the same Kubernetes setup without one starving the other is the actual engineering achievement here, not just the raw node count.

That kind of scale tends to expose problems nobody worries about at 500 nodes. Networking, scheduling, and API server load all behave differently once you're running thousands of nodes, and OpenAI's own account suggests plenty of tuning went into etcd, networking plugins, and how pods get placed to avoid bottlenecks. Kubernetes wasn't originally built with AI training clusters of this size in mind, so pushing it this far means finding and patching the seams as you go.

What's notable is the choice to stick with Kubernetes at all rather than build something bespoke. A lot of large AI labs end up rolling their own scheduler or orchestration layer once they hit this scale, because off-the-shelf tools start to creak. OpenAI instead leaned into extending the existing ecosystem, which says something about wanting to stay compatible with the broader Kubernetes community's tooling and knowledge base rather than going fully proprietary.