Building Blocks for Foundation Model Training and Inference on AWS
Hugging Face
AWS and Hugging Face laid out the full software stack needed to train and run today's giant AI models. Think of it as the plumbing behind every big model release you hear about.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
There was a time when scaling a foundation model meant one thing: throw more GPUs at pre-training and watch the loss curve bend downward. That was the Kaplan et al. story from 2020, and it justified years of capital spending on ever-bigger clusters. But that single curve has splintered into three. NVIDIA's own framing of it, pre-training plus post-training plus test-time compute, captures where the real gains now come from, whether that's reinforcement learning fine-tuning or a model that just thinks longer before answering.
What's striking in AWS's new writeup isn't the philosophy, it's the plumbing. Training a frontier model today, or running inference on one at scale, increasingly means stitching together the same three ingredients: tightly coupled accelerators, networking with almost no latency, and storage that can keep up with both. AWS walks through its current P5 and P6 instance lineup, from H100s and H200s up through Blackwell B200 and B300 chips, and the numbers move fast. A single B300 GPU now carries 288 GB of HBM3e memory and 8 TB/s of bandwidth, more than triple what an H100 offered just a couple of years ago.
The networking story is arguably the more interesting one. AWS's Elastic Fabric Adapter has gone through four generations, and each one shaves meaningful latency off collective communication, the unglamorous but critical process of GPUs talking to each other during training. EFAv4, shipping on P6 instances, is about 18% faster at this than EFAv3, which itself cut packet latency 35% versus EFAv2. And then there's the NVLink domain question: AWS's new UltraServers, built on NVIDIA's GB200 NVL72 platform, can link up to 72 Blackwell GPUs with 13.4 TB of shared HBM3e inside a single NVLink fabric. For mixture-of-experts models doing constant all-to-all token routing, that domain size stops being a spec sheet detail and starts being the actual bottleneck.
None of this happens in isolation from open source, either. AWS is explicit that Slurm and Kubernetes still run the orchestration layer, PyTorch and JAX still do the model math, and Prometheus and Grafana still watch over all of it. The infrastructure vendors have gotten faster and bigger, but the software stack sitting on top hasn't fundamentally changed hands. That's arguably the quieter and more durable story here: the frontier of AI compute keeps escalating, but it's still being orchestrated by the same open-source tools that ran HPC clusters a decade ago.
My take — AI-written commentary, not fact-checked reporting
What gets me is how unglamorous the real bottleneck has become: it's not model architecture, it's whether your interconnect can move data fast enough between chips. Everyone talks about parameter counts and benchmark scores, but AWS just quietly admitted that NVLink domain size might decide who can train competitive MoE models. That's infrastructure lock-in dressed up as innovation, and the cloud providers know exactly what they're doing by publishing specs sheets instead of price sheets.
Read more about this at: Hugging Face