Introducing Qwen2-Math
GitHub Pages
Qwen just dropped math-only models, and the biggest one beats GPT-4o on the toughest benchmarks. That's a specialized 72B open model out-mathing the closed giants.
Based on reporting by GitHub Pages — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Alibaba's Qwen team spent the last year obsessing over one narrow problem: getting language models to actually do math, not just pattern-match their way to plausible-looking answers. The result is Qwen2-Math, a family of three models — 1.5B, 7B, and 72B parameters — built on top of Qwen2 but retrained specifically for arithmetic and mathematical reasoning. The team is blunt that this release is English-only for now, with a bilingual English-Chinese version coming later.
What's notable here isn't just that Qwen2-Math exists, it's the claim that the 72B-Instruct variant beats GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and even Meta's much larger Llama 3.1 405B on a spread of math benchmarks. That's a 72-billion-parameter open model outperforming systems many times its size, purely because it was trained on a corpus stuffed with math-heavy web text, textbooks, exam questions, and code, rather than the usual everything-under-the-sun pretraining mix.
The instruction-tuned models went through a fairly elaborate pipeline: Qwen first built a dedicated math reward model off the 72B base, then combined that reward signal with a simple correct-or-wrong flag to generate training data through rejection sampling, and finished with reinforcement learning using Group Relative Policy Optimization. Evaluation ran the gauntlet — GSM8K, MATH, MMLU-STEM, OlympiadBench, CollegeMath, AIME 2024, AMC 2023, plus Chinese benchmarks like Gaokao and CMATH. Qwen reports that RM@8 sampling outperforms simple majority voting, especially at the smaller 1.5B and 7B sizes, which is the team's evidence that the reward model is pulling real weight rather than just adding noise.
The case studies are where things get genuinely fun to poke at. Qwen shows the 72B-Instruct model working through actual IMO Shortlist problems — a 2002 cube-sum question solved via modular arithmetic tricks, a 2022 digit-sum parity puzzle, a functional equation from the 2022 IMO itself. The model nails the classic f(x) = 1/x answer on the functional inequality problem and correctly rules out digit-sum parity matching for any polynomial with positive coefficients. Qwen is careful to note these are raw, unedited outputs and they aren't vouching for every line of reasoning being airtight.
Still, the pattern matters more than any single benchmark number. A 72B model that can hold its own against frontier closed systems on hard math signals that domain-specialized training, done well, can close gaps that raw scale alone was supposed to require.
My take — AI-written commentary, not fact-checked reporting
This is exactly the kind of release that should worry anyone betting everything on scale-is-all-you-need — a mid-sized open model out-mathing GPT-4o because Qwen bothered to build a proper reward model and curate real math data instead of just throwing more GPUs at the problem. Europe's AI policy crowd keeps fretting about frontier-model concentration, but stories like this are the actual counter-argument: specialization and open weights are quietly eating the moat one benchmark at a time.
Read more about this at: GitHub Pages