Is memory the moat?

Open source AI just crossed a weird threshold. GLM5.2 and DeepSeek V4-Pro already proved open models could match closed frontier systems, but Kimi K3 is a different animal entirely: 2.8 trillion parameters, more than 1.5TB of VRAM just to load the weights, before you even think about a KV cache for a million tokens of context. No single 8-GPU B200 node can hold it. You either chain two B200 nodes together or reach for something with more memory per chip.

That's where AMD's MI355X comes in, and the numbers from Wafer's benchmarking are genuinely interesting. The MI355X has 288GB of VRAM per GPU, same as Nvidia's B300, and costs roughly 2.4 times less per GPU on average. On a standard 1,024-token-input, 400-token-output test, eight MI355X chips pushed 952 tokens per second aggregate and 118 tokens per second in single-stream decode — beating a two-node, 16-GPU B200 setup by a wide margin on both throughput and cost. The B300 still wins on raw aggregate speed, about 1.65 times faster, but it costs 2.4 times more, so the MI355X wins decisively on performance per dollar: 48 tokens per second per dollar versus 33 for the B300 and just 7 for the B200.

Getting there wasn't seamless. Kimi K3 ships without its own draft tensors for speculative decoding, so Wafer's team relied on an external draft model called Kimi-K3-DSpark. That worked fine on Nvidia's CUDA stack but immediately crashed AMD's ROCm scheduler with a missing-function error. The root cause wasn't some deep architectural gap — it was one undefined kernel, top_k_renorm_prob, that CUDA gets for free from Nvidia's kernel library but ROCm simply never aliased. The fix was a few lines of PyTorch, not a custom kernel, and it bought a 2.2x single-stream speedup and an 18% bump in peak aggregate throughput.

A second bottleneck showed up in prefill, the unglamorous but user-facing part of inference that determines how long someone waits before seeing a first token. On a 172,000-token cold prefill, the MI355X took 51 seconds against the B300's 23. The culprit was another shape mismatch: AMD's fast MLA attention kernel expects attention head counts in multiples of 4, 8, or 16, but Kimi K3 at this configuration produces 12 heads per GPU. Zero-padding to 16 and trimming back down afterward tripled prefill speed, from roughly 4,000-7,000 tokens per second up to 13,000.

None of these fixes required writing new kernels from scratch, which is notable given how often AMD's software gaps get blamed for its underdog status against Nvidia. The bugs here were shallow — missing aliases and shape assumptions, not fundamental limitations. If day-zero support for a 2.8-trillion-parameter model can already beat B200 on cost efficiency with this little friction, the argument that CUDA's ecosystem lead is unassailable is getting harder to defend with a straight face.