Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers

Apple just published the engineering behind one of the more overlooked parts of its new Siri: the thing that actually turns generated audio tokens into the voice you hear. It's called a detokenizer, and the paper explains how Apple squeezed a diffusion transformer down small enough to run on the Apple Matrix Coprocessor, the dedicated silicon block inside its chips built for exactly this kind of workload.

The core trick is decoupling time from depth. Speech synthesis models typically generate audio as layered codes, called RVQ tokens, where each layer adds more detail on top of the last. Older systems threw a separate decoder at each layer, which multiplies memory use fast. Apple's team instead built one reusable depth decoder that handles every layer, guided by conditioning borrowed from diffusion transformers, plus a separate streaming encoder and temporal decoder that manage the timing. Add causal sliding-window attention with a fixed-size cache, and the whole system uses the same amount of memory whether it's generating two seconds of speech or five minutes.

The numbers back up the pitch. Apple says the detokenizer runs at roughly 10 milliseconds per step, about 16 times faster than real time, while peaking at just 21MB of runtime memory and 329MB of stored assets. That's small enough to sit comfortably alongside AFM 3 Core Advanced, the on-device foundation model that actually generates the semantic tokens in the first place, and still stream up to 320 seconds of continuous audio.

Quality didn't get sacrificed for the diet. Apple reports a 0.28-point jump in Mean Opinion Score over its previous on-device TTS system, rising to 4.15 from 3.87, with an even bigger 0.42-point gain on conversational speech. That's the technical foundation under Siri's new Expressive Voices, the feature bringing pace and expressiveness sliders plus support for custom assistant voices to Apple devices this generation.

What's notable here isn't the voice quality bump itself, it's that Apple achieved it while shrinking the compute footprint, which is the actual constraint that determines what's possible entirely on-device without phoning home to a server.