Why We Think
Lilian Weng
A research post reviews how test-time compute and chain-of-thought reasoning improve language model performance by enabling models to spend more computation on problem-solving. Key developments include reinforcement learning approaches like o1 and o3 that use policy gradient algorithms, parallel sampling methods like best-of-N and beam search with process reward models, and sequential revision techniques, with recent work showing optimal ratios of sequential to parallel compute depending on problem difficulty. The effectiveness stems from treating computation as a learnable resource, drawing analogies to human System 2 thinking, and using latent variable modeling to optimize over intermediate reasoning steps.
Why it matters
Special thanks to John Schulman for a lot of super valuable feedback and direct edits on this post. Test time compute (Graves et al. 2016, Ling, et al. 2017, Cobbe et al. 2021) and Chain-of-thought (CoT) (Wei et al. 2022, Nye et al. 2021), have led to significant improvements in model performance, while raising many research questions. This post aims to review recent developments in how to effectively use test-time compute (i.e. “thinking time”) and why it helps.
