Counterfactual Evaluation for Recommendation Systems

Eugene Yan · 4 years ago · 27

Recommendation systems are typically evaluated offline using historical data as if they were observational problems, but they are actually interventional problems where recommendations change user behavior. Inverse Propensity Scoring (IPS) estimates how user interactions would change by reweighting logged rewards based on the ratio of recommendation probabilities between new and old models, with Self-Normalized IPS (SNIPS) performing best in experiments without requiring parameter tuning. Counterfactual evaluation allows practitioners to simulate A/B test outcomes offline before deployment, addressing cases where offline metrics diverge from online results.