Mini-R1: Reproduce Deepseek R1 „aha moment“ a RL tutorial

Hugging Face Blog · 1 year ago

A tutorial demonstrates how to recreate DeepSeek R1's "aha moment"—where a model learns to allocate more thinking time to problems through reinforcement learning—using Group Relative Policy Optimization (GRPO) on the Countdown Game puzzle task. The training setup uses a Qwen 2.5 3B model on 4 NVIDIA H100 GPUs, with full training completing in approximately 6 hours across 450 steps. By step 450, the model achieves 50% success rate in solving countdown equations and spontaneously shifts from word-based reasoning to programmatic trial-and-error approaches without explicit instruction.