Large Transformer Model Inference Optimization
Lilian Weng
The article surveys optimization techniques for reducing the computational and memory costs of inference in large transformer models, covering methods like knowledge distillation, quantization with various granularities, and outlier-aware strategies. DistilBERT achieved 40% parameter reduction while maintaining 97% performance and running 71% faster; GPTQ reduced OPT-175B weights to 3-4 bits with minimal performance loss. These techniques enable deploying large transformers more efficiently by trading model size or precision for inference speed and memory usage.
Why it matters
[Updated on 2023-01-24: add a small section on Distillation.] Large transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale. Why is it hard to run inference for large transformer models? Besides the increasing size of SoTA models, there are two main factors contributing to the inference challenge (Pope et al. 2022):
