Goodbye cold boot - how we made LoRA Inference 300% faster

Hugging Face · 2 years ago · 10

Hugging Face implemented dynamic LoRA loading on its Inference API, where a single base model remains active while different LoRA adapters are swapped in and out for each user request instead of maintaining separate deployments for each adapter. The warm-up time for serving a LoRA decreased from 25 seconds to 3 seconds, and overall response time dropped from 35 seconds to 13 seconds, while the system now serves hundreds of distinct LoRAs on fewer than 5 A10G GPUs. This approach reduced compute resource requirements since the vast majority of the 2,500 public LoRAs share only a few base models, allowing efficient reuse of GPU capacity across different adapter requests.