Very Large Language Models and How to Evaluate Them

Hugging Face just made it a lot cheaper to find out whether a giant language model is quietly biased. Its Evaluation on the Hub tool, built on AutoTrain, now supports zero-shot classification for any causal language model on the Hub, up to 66 billion parameters, and it does the heavy lifting for you. Previously, just loading and compiling a model that size could eat 35 minutes before you'd run a single test. Now someone with zero GPU budget can submit a job, wait roughly 3.5 hours, and get results emailed to them as a pull request on the model's repo.

The method itself is simple in concept: feed the model a prompt plus a set of possible completions, sum and normalize the log-probabilities for each, and see which completion the model favors. No labeled training data required, which is the whole appeal of zero-shot evaluation and why the Inverse Scaling Prize community has leaned on it to hunt for tasks where bigger models actually underperform smaller ones.

Hugging Face ran a case study using WinoBias, a coreference dataset built around occupations and gendered pronouns. Think sentences about a "developer" where the anti-stereotypical answer is "she." The pattern that emerged was not encouraging. Smaller OPT models picked the anti-stereotypical pronoun more often, but as model size increased, the models leaned harder into stereotypical gender-occupation pairings. That's an inverse scaling result, and it lines up with other work, including BIG-Bench findings, showing that scaling up tends to amplify bias around gender, race, ethnicity and nationality, not fix it.

What's notable here isn't just the bias finding, it's that the finding was reachable without a research lab's infrastructure. Hugging Face frames this as democratizing evaluation, and the WinoBias example is essentially a demo of what any Winograd-style dataset could reveal once uploaded and pointed at a batch of models. They're explicitly inviting people to try their own tasks and feed interesting scaling trends into round two of the Inverse Scaling Prize.

Support currently caps at 66 billion parameters, with bigger models and other task formats promised down the line. For now, the tool turns a question that used to require serious compute and expertise, does this model get worse in some specific way as it grows, into something a curious grad student or independent researcher can just submit and wait for.