Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Hugging Face
NVIDIA and AI4Bharat are opening a 2.8T-token Indic training corpus and the pipeline that made it. That matters because it gives model builders both the data and the recipe, not just the leftovers.
Based on reporting by Hugging Face — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
NVIDIA and AI4Bharat are putting out Nemotron-pretraining-Indic-v1, a roughly 2.8 trillion-token corpus for 22 Indian languages, along with the open, GPU-accelerated pipeline used to generate it. The release is built for the languages named in India’s Constitution, and the team says those are the first languages for 97% of Indians, or nearly 1.2 billion people at the 2011 Census.
The pitch here is bigger than a dataset drop. The corpus includes native-script data for all 22 languages and romanized text for 12 of them, spanning 34 language-script configurations. And because every record stays tied to a source document, developers can trace outputs back to where they came from, which is exactly the kind of boring detail that makes large synthetic corpora usable instead of just impressive.
The data was assembled from 15 Nemotron-CC-v2 subsets and pushed through NeMo Data Designer, NVIDIA’s open-source synthetic-data framework. For 12 languages, the workflow translates whole documents, checks quality with Gemma-4-31B-it, repairs weak outputs, and then turns some accepted translations into Latin script. For the other ten languages, the pipeline uses Indic Translate from Bodhan AI, with special handling for long Santali documents and for Manipuri, which is first translated into Bengali script before conversion to Meitei Mayek.
The quality gate is strict: outputs need to score at least 75 out of 100 on a rubric inspired by MQM, covering meaning, fluency, terminology, style, and formatting. The team says feedback-based repair recovered 438.4 billion tokens, and the corpus total already includes those recovered records. The whole point is not just volume. It’s getting synthetic data that still knows how to keep equations, tables, URLs, and code intact.
There’s also a model-side story. In continued pretraining for Nemotron 3.5 Super VL, 10 billion Indic tokens were drawn from a 12-language subset of this corpus and natural Indic text. The technical report says scores rose on all five reported Indic knowledge and reasoning benchmarks, while the larger Nemotron 3.5 Super VL release outperformed Nemotron 3 Super on five of six task benchmarks shown and improved on all four IndiVibe language-quality dimensions, though instruction following on IFEval 1.5 slipped.
My take — AI-written commentary, not fact-checked reporting
This is the right way to do synthetic data: open the recipe, not just the trophy. Too many “open” releases are really just data dumps with mystery meat provenance; this one at least tries to make the pipeline auditable and reusable. If Indian-language AI is going to stop leaning on English like a crutch, more projects should copy this discipline and fewer should copy the hype.
Read more about this at: Hugging Face
