Predictive Human Preference: From Model Ranking to Model Routing

Chip Huyen · 2 years ago

A researcher developed a preference predictor that forecasts which AI model users will prefer for specific prompts by training on 20,927 comparison matches from LMSYS's Chatbot Arena dataset. The predictor achieved 76.2% accuracy when incorporating prompts as input, compared to 74.1% accuracy from Chatbot Arena's Bradley-Terry ranking that only considers model pairs. This enables model routing to direct queries to cheaper or faster models when they perform comparably to stronger models, potentially reducing costs and latency while maintaining response quality.