Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6
Amazon Web Services Mona Mona
AWS benchmarked G5, G6 and G7 for small LLMs on SageMaker. G7 came out fastest, and Blackwell’s native FP4 gives it a real edge.
Based on reporting by Amazon Web Services, Mona Mona — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS set up a pretty clean test: same model, same serving stack, different GPUs. For a Qwen3-Coder-30B-A3B-Instruct-FP8 workload on SageMaker AI, it compared ml.g5.12xlarge, ml.g6.12xlarge and ml.g7.12xlarge using the DJL Large Model Inference 28.0 container. The point was simple. See which instance actually serves tokens best when the model is a Mixture-of-Experts job and the GPU, not the model, is the bottleneck.
The numbers favored G7. In AWS’s non-streaming benchmark, ml.g7.12xlarge delivered 391.3 output tokens per second, versus 346.3 on G5 and 243.4 on G6. Average request latency was 1,315.8 ms on G7, compared with 1,475.2 ms on G5 and 2,109.7 ms on G6. Tail latency followed the same pattern: P99 came in at 1,501.1 ms on G7, 1,881.1 ms on G5, and 3,315.7 ms on G6.
That mattered even though G7 used less total GPU memory than the others. The G5 and G6 configurations each had four GPUs and 96 GB of aggregate GPU memory. G7 had two GPUs and 64 GB. AWS is basically showing that fewer accelerators can still win if the architecture is newer and the memory system is better suited to MoE decoding. For interactive use, the same G7 endpoint also showed a median time to first token of about 118 ms and an average inter-token latency of 8.9 ms.
AWS then switched to a different setup for NVIDIA Nemotron-3-Nano-30B-A3B-NVFP4. Instead of hand-picking endpoints, it used SageMaker AI Generative AI Inference Recommendations with vLLM to test candidate configurations for throughput, latency and cost per token. That second path matters because it turns instance selection into a measured recommendation instead of a guess dressed up as strategy.
The underlying argument is the real one here: MoE inference is memory-bandwidth hungry, and Blackwell’s native FP4 support on G7 gives it a structural advantage. G5 and G6 can run NVFP4 weights, but without hardware acceleration. In other words, AWS is not just saying the new instance is faster. It is saying the old way of buying bigger GPU boxes and hoping for the best is getting less convincing by the month.
My take — AI-written commentary, not fact-checked reporting
This is the part cloud buyers keep rediscovering: more GPUs is not the same as better inference. If the model is MoE and the token path is bandwidth-bound, the shiny new instance with fewer chips can still win, which is rude but useful. Also, the industry’s favorite habit of treating benchmarking as a ceremonial checkbox deserves less respect than it gets.
Read more about this at: Amazon Web Services