DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
Apple Machine Learning Research
Apple researchers say diffusion LLMs need better reward training, so they built DACA-GRPO. It gives denoising steps different credit and cuts bias, with gains on math, code, and schema tasks.
Based on reporting by Apple Machine Learning Research — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Apple researchers have put a name to a problem that has been hanging over reinforcement learning for diffusion language models: the training loop has been treating every denoising step as if it mattered equally. Their answer is DACA-GRPO, short for Denoising-Aware Credit Assignment for GRPO, a plug-in style upgrade for GRPO trainers that tries to assign credit where it is actually earned.
The paper argues that existing RL methods for diffusion models miss two things at once. First, they do not do temporal credit assignment across the denoising trajectory, so the model never really learns which intermediate steps helped. Second, the mean-field likelihood estimates used for policy optimization are systematically biased and noisy. That is not a small bookkeeping issue; it affects how the model is trained to improve.
DACA-GRPO tackles both sides of that problem. One part, called Denoising Progress Scores, pulls per-token importance weights from intermediate predictions without adding another forward pass. The other, Stratified Masking Likelihood, splits token positions into strata so each token gets predicted with most of the sequence in view, which is meant to reduce the mean-field bias.
The system was applied on top of three GRPO base methods, and the gains were consistent across seven benchmarks. Those tests covered mathematical reasoning, code generation, constraint satisfaction, and constrained generation. The reported improvements reached up to 5.6 percentage points on math reasoning, 7.4 points on code generation, 36.3 points on constraint satisfaction, and 5.9 points on JSON schema adherence.
And that is the real point here: diffusion LLMs may be compelling, but they still need the boring, unglamorous work of better credit assignment before anyone can trust their training recipes. Apple’s pitch is not that RL is broken. It is that the default version is still too blunt for a model that learns step by step.
My take — AI-written commentary, not fact-checked reporting
This is the kind of paper that quietly matters more than another glossy model demo. If a training method only works by pretending every denoising step is equally important, then it is doing spreadsheets, not science. The industry loves bigger models; the researchers doing the unsexy credit assignment work are the ones actually moving the field forward.
Read more about this at: Apple Machine Learning Research