Generation configurations: temperature, top-k, top-p, and test time compute

Chip Huyen · 2 years ago

The article explains how language models generate probabilistic outputs through sampling techniques including temperature, top-k, and top-p methods, which control the trade-off between creativity and consistency in model responses. Temperature adjusts logit values before softmax conversion, with values like 0.7 recommended for creative tasks and 0 for deterministic outputs, while top-k and top-p limit sampling to the most likely tokens to reduce computation and improve contextual relevance. Test time compute improves model performance by generating multiple outputs and selecting the best one based on probability scores, with the author noting that approximately 20% of customer support issues stem from users misunderstanding the probabilistic nature of AI models.