Scaling Categorical Flow Maps

Autoregressive models still run the show in language generation, spitting out one token after another in a strict left-to-right march. But a growing pile of research keeps poking at whether continuous diffusion and flow matching, the techniques that made image generation fast and flexible, could work just as well for text. Apple's ML research team just pushed that question further than anyone has before by training a flow model with 1.7 billion parameters on 2.1 trillion tokens, a scale well beyond the sub-1-billion experiments that came before it.

The trick behind their approach is something they call Categorical Flow Maps, or CFMs. Instead of building the text-generating model from scratch, the team self-distilled their large base flow model into a CFM capable of producing full passages of text in as few as four inference steps. That's a dramatic cut compared to the many iterative passes diffusion-style generation usually needs. And despite the shortcut, the output keeps something close to the token-level entropy you'd expect from real data, meaning the text doesn't collapse into repetitive or overly predictable patterns just because the model is moving fast.

Speed alone doesn't mean much without a way to judge quality, so the researchers also worked out a likelihood bound for CFMs operating in what they call the semi-discrete setting. That bound lets them score the model the same way other language models get scored, on standard benchmarks. The results land in the same range as discrete diffusion methods, which suggests the approach holds up once it leaves the lab-scale sandbox and faces real evaluation.

Getting there wasn't clean. The team ran into genuine obstacles once they pushed training to this size, and they spend real effort in the paper detailing what went wrong and what fixed it. Their prescriptive notes on loss weighting and time scheduling read less like a victory lap and more like a field guide for anyone else attempting to scale this family of models, which is arguably the more useful contribution here than the benchmark numbers themselves.

What this work mainly proves is that Categorical Flow Maps don't fall apart when you throw real compute at them. Whether that's enough to make flow-based text generation a serious contender against autoregressive models remains an open question, but at least now it's a question backed by billion-parameter evidence instead of small-scale hints.