Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker

Hugging Face · 4 years ago · 48

EleutherAI's GPT-J 6B, a 6 billion parameter open-source language model, can now be deployed to production using Amazon SageMaker and Hugging Face Transformers. Model loading time was reduced from 3 minutes 32 seconds to 7.7 seconds by using torch.save() and torch.load() instead of the standard from_pretrained() method. This enables GPT-J to run inference within SageMaker's 60-second request timeout on a standard GPU instance costing approximately $500 per month.