Bandits for Recommender Systems
Eugene Yan
Bandit algorithms address the exploration-exploitation trade-off in recommender systems by modeling uncertainty and deliberately exploring items with uncertain value rather than greedily promoting historically popular items. Three main algorithms—ε-greedy, Upper Confidence Bound (UCB), and Thompson Sampling—are used across industry implementations at Spotify, Yahoo, Alibaba, DoorDash, Amazon, and Twitter, with Thompson Sampling and UCB outperforming ε-greedy and Thompson Sampling proving more robust to delayed feedback in systems. Practical lessons include Thompson Sampling's superiority under delayed rewards (evaluated at 10, 30, and 60-minute delays), pessimistic initialization outperforming naive initialization, and exploration strategies ranging from large exploration on limited user buckets to small exploration across all users.
Why it matters
Industry examples, exploration strategies, warm-starting, off-policy evaluation, and more.
