This could be the largest synthetic code dataset yet

IBM Research · 1 day ago

IBM open-sourced CodeAlchemy, a pipeline that generates synthetic code training data covering 15 programming languages, totaling nearly 1 trillion tokens—at least 200 times larger than Wikipedia. The dataset includes 1.3 million code files paired with execution traces, a novel approach to teach models what code does at runtime rather than just syntax. Models trained on CodeAlchemy showed measurable improvements: a Granite 3B model trained on the synthetic data outperformed larger models on code reasoning tasks and achieved 83.5% on HumanEval benchmarks.