How to Measure and Mitigate Position Bias

Eugene Yan · 4 years ago · 48

Position bias occurs when items ranked higher receive more clicks regardless of relevance, such as first-position search results receiving 10 times more clicks than tenth-position results. Methods to measure position bias include uniform randomization of results, exploiting inherent randomness in production data, expectation maximization models, and controlled pair-swapping techniques like FairPairs and RandPair. Mitigation strategies involve adding randomness during model training, debiasing logged data using measured position bias weights, or including positional features in models that can be neutralized during serving.