Large Transformer Model Inference Optimization

Lilian Weng · 3 years ago

The article surveys optimization techniques for reducing the computational and memory costs of inference in large transformer models, covering methods like knowledge distillation, quantization with various granularities, and outlier-aware strategies. DistilBERT achieved 40% parameter reduction while maintaining 97% performance and running 71% faster; GPTQ reduced OPT-175B weights to 3-4 bits with minimal performance loss. These techniques enable deploying large transformers more efficiently by trading model size or precision for inference speed and memory usage.