Accelerating SD Turbo and SDXL Turbo Inference with ONNX Runtime and Olive

Stable Diffusion used to demand 30-plus denoising steps to spit out a usable image. SD Turbo and SDXL Turbo changed that math, generating viable pictures in a single step by distilling down from Stable Diffusion 2.1 and SDXL 1.0. Microsoft's ONNX Runtime team has now squeezed even more speed out of those already-fast models, and the numbers are hard to ignore.

According to benchmarks run on an A100-SXM4-80GB and a consumer RTX-4090, ONNX Runtime beat PyTorch across every batch size and step count tested. The biggest gain: 229% faster throughput for SDXL Turbo running dynamic shape on CUDA at a (1, 4) batch-and-steps configuration. SD Turbo saw gains up to 120%. These aren't cherry-picked edge cases either — the team tested static and dynamic shapes, both the CUDA and TensorRT execution providers, and two very different GPUs, and ONNX Runtime won essentially every matchup.

The practical tradeoffs are worth understanding if you're actually deploying this stuff. Static shape — where you lock in batch size and resolution ahead of time — tends to run faster, and TensorRT edges out CUDA there. But switch resolutions or batch sizes on the fly and you're stuck rebuilding and swapping TensRT engines, which eats into any speed advantage. Dynamic shape avoids that headache by handling varying batch and image sizes with a single engine, and CUDA's execution provider proved the better dynamic-shape choice on the A100, while TensorRT pulled ahead again on the RTX-4090. In other words: know your hardware and your use case before picking a lane.

Under the hood, the speedups come from a handful of specific engineering tricks — CUDA graphs for static shapes, Flash Attention V2, trimming unnecessary text encoder outputs, and a new SkipGroupNorm fusion that merges group normalization with adjacent Add operations. The team also added LoRA support for latent consistency models, which matters for anyone fine-tuning these turbo variants.

What's maybe more interesting long-term is the cross-language angle. Because ONNX Runtime isn't tied to Python, the community has already built a .NET library called OnnxStack and Oracle has shipped a Java sample, both riding on ONNX Runtime's C# tutorial groundwork. That opens Stable Diffusion up to enterprise stacks that have nothing to do with Python notebooks — game engines, backend services, whatever runs on .NET or the JVM. ControlNet support and a proper C# tutorial for the Turbo models are apparently coming soon, along with IP Adapter and Stable Video Diffusion integration down the line.