Goodbye cold boot - how we made LoRA Inference 300% faster

Hugging Face just quietly fixed one of the more annoying parts of running Stable Diffusion LoRAs on its Hub: the cold start. Anyone who's clicked into an obscure LoRA model and waited half a minute for a single image knows the pain. That wait is now down to about 13 seconds, and in some cases the setup itself dropped from 25 seconds to 3.

The trick isn't some exotic new hardware trick — it's architectural. There are roughly 2,500 public LoRAs on the Hub, and about 92% of them are built on top of Stable Diffusion XL Base 1.0, a single 7GB model. Rather than spinning up a dedicated GPU deployment for every one of those 2,500 adapters, which is what used to happen, Hugging Face now keeps a handful of base models permanently warm and swaps the lightweight LoRA weights in and out on demand. A typical LoRA adapter is only around 24MB, so loading one on top of an already-running base model takes about 3 seconds instead of the 25 seconds needed to boot an entire dedicated service from scratch.

The numbers add up fast. Hugging Face says it can now serve hundreds of distinct LoRAs using fewer than five A10G GPUs total. In any given two-minute window, roughly ten different LoRA weights get requested — instead of spinning up ten separate deployments, the system just juggles them across one or two GPUs, occasionally more during a burst. The company also found that fusing the LoRA weights directly into the base model's layers before inference, rather than keeping them as a separate computation, cuts inference time by another 30%.

Interestingly, they looked at batching multiple requests together — a technique that's been a huge win for LLM serving, where you can get roughly 8x the throughput for only a 10% latency hit. For diffusion models it didn't pan out: batching eight image requests together only bumped throughput by 25%, while latency ballooned sixfold. So they skipped it entirely and stuck with sequential, single-request inference, relying purely on the warm-base-model trick to do the heavy lifting.

One caveat worth flagging: this optimization only applies to LoRAs that are public, non-gated, and built on non-gated public base models. If you're running something private or restricted, none of this warmth-sharing applies to you.