Large Transformer Model Inference Optimization

Lil'Log · 3 years ago · 15

The article surveys optimization techniques for reducing the computational and memory costs of inference in large transformer models, covering methods like knowledge distillation, quantization with various granularities, and outlier-aware strategies. DistilBERT achieved 40% parameter reduction while maintaining 97% performance and running 71% faster; GPTQ reduced OPT-175B weights to 3-4 bits with minimal performance loss. These techniques enable deploying large transformers more efficiently by trading model size or precision for inference speed and memory usage.