Counterfactual Evaluation for Recommendation Systems
Eugene Yan
Standard recsys testing checks new models against old click logs, which quietly assumes recommendations don't change behavior. Eugene Yan explains why that's backwards and what to do instead.
Based on reporting by Eugene Yan — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Most recommendation systems get evaluated the same way you'd test a spam classifier: split historical data, train on one chunk, score on the other. Eugene Yan points out the flaw in that approach. Recommendations aren't predicting a fixed label, they're an intervention that changes what a customer even sees. Test a new model against old logs and you're not measuring whether people would click more, you're measuring how well the model mimics what already happened.
The fix, in theory, is A/B testing — show real users the new recommendations and watch conversion move. But that's slow, risky, and often unavailable to research teams without production access. Yan's answer is counterfactual evaluation, specifically Inverse Propensity Scoring, which reweights logged clicks by how much more or less often the new model would have made the same recommendation compared to the old one. Recommend an iPhone 1.5 times as often as your production model did, and you scale that click's contribution by 1.5x. Run this across your logs and you get an estimate of what an A/B test might show, without running one.
IPS breaks in two obvious ways. If the new model recommends something the old model never showed, there's no baseline probability to divide by, so the math falls apart — Yan suggests giving every eligible item some nonzero chance of being shown just to keep the estimator alive. And when the two models diverge sharply in what they recommend, a single click observed at low probability under the old policy can get reweighted 100x, producing wild overestimates. Clipped IPS caps that weight but needs manual tuning; Self-Normalized IPS sidesteps the problem by rescaling against the total importance weight instead of a hard cutoff.
Citing a RecSys 2021 tutorial by Yuta Saito, Yan notes SNIPS came out ahead in synthetic benchmarks — lowest error, no tuning required — even though it costs more to compute since it needs importance weights for every observation, not just the ones with actual clicks or purchases. Given that most recommendation traffic converts at under 10%, that's a real computational tax, though apparently one that pays for itself through faster convergence.
None of this replaces standard offline evaluation, and Yan isn't arguing it should. Static observational testing remains useful because the datasets and metrics are standardized and because you can start collecting training data before you've even shipped a recommender. Counterfactual evaluation is positioned as the next step: reach for it when your offline metrics stop lining up with what A/B tests actually show.
My take — AI-written commentary, not fact-checked reporting
This is the kind of gap that hides in plain sight until someone names it, and it explains a complaint I've heard from more than one ML team: 'our offline NDCG went up but the A/B test was flat.' The uncomfortable truth is that most orgs will keep doing observational eval anyway, not because it's right but because it's cheap and the datasets already exist — counterfactual methods only spread once someone gets burned by the mismatch.
Read more about this at: Eugene Yan