Very Large Language Models and How to Evaluate Them

Hugging Face · 3 years ago · 21

Hugging Face released a free zero-shot evaluation tool that allows researchers to test large language models on classification tasks without writing code or using expensive GPU infrastructure. Evaluating a 66-billion parameter model on 2,000 examples now takes 3.5 hours and is accessible to anyone, whereas previously such tasks required 35 minutes just to load the model on specialized hardware. The tool enables broader research into inverse scaling—cases where larger models perform worse than smaller ones—as demonstrated by testing on WinoBias, which found larger models exhibit more gender bias in occupation-related predictions.