How to Measure and Mitigate Position Bias
Eugene Yan
Position bias occurs when items ranked higher receive more clicks regardless of relevance, such as first-position search results receiving 10 times more clicks than tenth-position results. Methods to measure position bias include uniform randomization of results, exploiting inherent randomness in production data, expectation maximization models, and controlled pair-swapping techniques like FairPairs and RandPair. Mitigation strategies involve adding randomness during model training, debiasing logged data using measured position bias weights, or including positional features in models that can be neutralized during serving.
Why it matters
Introducing randomness and/or learning from inherent randomness to mitigate position bias.
