Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers
Apple
Apple built a new on-device engine that turns Siri's AI tokens into actual speech, running right on your phone. It's 16x faster than real time on barely any memory, and users rate the voice noticeably better than before.
Based on reporting by Apple — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Apple just published the guts of how Siri's Expressive Voices actually make sound. The interesting part isn't the voice itself—it's the plumbing underneath, a component called a detokenizer that takes the semantic audio tokens spit out by Apple's AFM 3 Core Advanced foundation model and turns them into audio you can actually hear, all while running on Apple's own Matrix Coprocessor chip, the AMX.
The architecture splits the job into three pieces: a streaming encoder, a temporal decoder, and a depth decoder, each handling a different slice of the problem. The clever move is reusing a single depth decoder, built with Diffusion Transformer-style conditioning, to generate every level of the audio representation autoregressively. Older multi-decoder systems needed a separate decoder for each level, which eats memory fast. Apple also swapped in causal sliding window attention with a fixed-window cache, which means memory use stays flat no matter how long the audio gets, instead of growing linearly or quadratically the way typical transformer or GAN-based systems do.
The payoff shows up in the numbers Apple shared: roughly 10 milliseconds per generation step, about 16 times faster than real time, using only around 21MB of peak runtime memory and 329MB of on-device assets. That's enough to stream anywhere from 20 to 320 seconds of continuous audio right alongside the foundation model itself, on a system running at 1 billion activated parameters inside AFM 3 Core Advanced.
Apple didn't just optimize for speed and forget about quality. Ablation studies checked whether the DiT conditioning, the temporal lookahead trick, and the unified depth decoding actually earn their keep, and the team ran phonetic discriminability tests, perceptual quality scoring, and neural quality estimation to make sure fidelity held up. The results back that up: overall Mean Opinion Score climbed 0.28 points to 4.15 versus 3.87 for the prior on-device text-to-speech system, and conversational speech specifically jumped 0.42 points, from 3.82 up to 4.24.
This is now live, not a research demo. It's the engine behind Siri Expressive Voices, including the new Pace and Expressivity sliders and support for custom assistant voices across Apple's devices.
My take — AI-written commentary, not fact-checked reporting
This is the kind of Apple paper that actually matters more than it sounds like it should—squeezing a voice synthesizer down to 21MB of runtime memory while running 16x faster than real time is a genuine constraint-engineering win, not a marketing number. Everyone's obsessed with bigger models; Apple's whole pitch here is smaller, tighter, and still measurably better sounding, with the MOS gains to prove it. If on-device AI is going to be more than a privacy slogan, this is what the real work looks like, and it's a lot less glamorous than a chatbot demo.
Read more about this at: Apple