Bandits for Recommender Systems

Eugene Yan · 4 years ago · 29

Bandit algorithms address the exploration-exploitation trade-off in recommender systems by modeling uncertainty and deliberately exploring items with uncertain value rather than greedily promoting historically popular items. Three main algorithms—ε-greedy, Upper Confidence Bound (UCB), and Thompson Sampling—are used across industry implementations at Spotify, Yahoo, Alibaba, DoorDash, Amazon, and Twitter, with Thompson Sampling and UCB outperforming ε-greedy and Thompson Sampling proving more robust to delayed feedback in systems. Practical lessons include Thompson Sampling's superiority under delayed rewards (evaluated at 10, 30, and 60-minute delays), pessimistic initialization outperforming naive initialization, and exploration strategies ranging from large exploration on limited user buckets to small exploration across all users.