Why We Think

Lilian Weng · 1 year ago

A research post reviews how test-time compute and chain-of-thought reasoning improve language model performance by enabling models to spend more computation on problem-solving. Key developments include reinforcement learning approaches like o1 and o3 that use policy gradient algorithms, parallel sampling methods like best-of-N and beam search with process reward models, and sequential revision techniques, with recent work showing optimal ratios of sequential to parallel compute depending on problem difficulty. The effectiveness stems from treating computation as a learnable resource, drawing analogies to human System 2 thinking, and using latent variable modeling to optimize over intermediate reasoning steps.