What Makes Inference Nondeterministic?
The AI Engineer
Inference outputs from the same model and prompt can differ across repeated runs even at temperature 0 because GPU arithmetic order changes with how requests share a batch. Thinking Machines Lab reported that 1,000 identical requests returned 80 different completions. Reproducibility can be improved by turning on batch-invariant kernels in vLLM or SGLang, which typically costs about 38% of throughput.
Why it matters
When the same prompt is sent to an AI model multiple times at zero temperature, variations can occur due to the nondeterministic nature of floating-point arithmetic plus server load dynamics. The output can differ based on how requests are batched and how the GPU processes additions.