H100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time

SemiAnalysis · 10 months ago

Nvidia H100 GPUs achieved 57% throughput improvement over 12 months through software optimization, reducing GPT-3 175B training cost from 72 cents to 54.2 cents per million tokens, while GB200 NVL72 servers cost 1.6x more in total cost of ownership than H100s but lack large-scale production training runs due to software maturity and reliability issues. The analysis benchmarked over 2,000 H100 GPUs and found GB200 NVL72 needs to be at least 1.6x faster than H100 to justify its higher costs, with the gap expected to close by year-end as software improves. SemiAnalysis recommends Nvidia expand benchmarking beyond NeMo-Megatron to native PyTorch, increase public transparency of cloud provider performance data, and accelerate reliability diagnostics for GB200 NVLink backplane failures.