Make your llama generation time fly with AWS Inferentia2

Hugging Face Blog · 2 years ago

AWS and Hugging Face enabled text generation with Llama 2 models on AWS Inferentia2 accelerators using the optimum-neuron library, which compiles and deploys large language models to specialized hardware. The Llama 2 7B model achieves encoding times of 0.5 seconds for 256 input tokens and throughput of 227–750 tokens per second depending on configuration, while the 13B model reaches 145–504 tokens per second. Users can now export models from Hugging Face, compile them for Inferentia2 with static shape constraints, and generate text using standard transformer APIs or simplified pipeline wrappers.