Hugging Face Text Generation Inference available for AWS Inferentia2

Hugging Face Blog · 2 years ago

Hugging Face Text Generation Inference became generally available on AWS Inferentia2 through Amazon SageMaker for deploying large language models in production. The solution supports popular models like Llama and Mistral, with pre-compiled configurations cached for batch size 2-4 and sequence length 2048 to avoid the 45-minute compilation process. Customers can now deploy LLMs on Inferentia2 as a cost-effective alternative to GPUs, with deployment taking 10-15 minutes on ml.inf2.8xlarge instances.