1 Billion Classifications

Hugging Face Blog · 1 year ago

A developer published cost optimization methodology for running 1 billion daily classification and embedding requests on cloud infrastructure. Processing 1 billion inputs via DistilBERT costs $253.82 using an nvidia-L4 GPU with batch size 64, while 1 billion embeddings via ModernBERT costs $409.44 with batch size 32. Organizations can now calculate their own optimal hardware, batch size, and virtual user configurations to minimize inference costs at scale.