Generation configurations: temperature, top-k, top-p, and test time compute
Chip Huyen
The article explains how language models generate probabilistic outputs through sampling techniques including temperature, top-k, and top-p methods, which control the trade-off between creativity and consistency in model responses. Temperature adjusts logit values before softmax conversion, with values like 0.7 recommended for creative tasks and 0 for deterministic outputs, while top-k and top-p limit sampling to the most likely tokens to reduce computation and improve contextual relevance. Test time compute improves model performance by generating multiple outputs and selecting the best one based on probability scores, with the author noting that approximately 20% of customer support issues stem from users misunderstanding the probabilistic nature of AI models.
Why it matters
ML models are probabilistic. Imagine that you want to know what’s the best cuisine in the world. If you ask someone this question twice, a minute apart, their answers both times should be the same. If you ask a model the same question twice, its answer can change. If the model thinks that Vietnamese cuisine has a 70% chance of being the best cuisine and Italian cuisine has a 30% chance, it’ll answer “Vietnamese” 70% of the time, and “Italian” 30%. This probabilistic nature makes AI great for creative tasks. What is creativity but the ability to explore beyond the common possibilities, to think outside the box? However, this probabilistic nature also causes inconsistency and hallucinations. It’s fatal for tasks that depend on factuality. Recently, I went over 3 months’ worth of customer support requests of an AI startup I advise and found that ⅕ of the questions are because users don’t understand or don’t know how to work with this probabilistic nature. To understand why AI’s responses
