Reinforcement Learning for Recommendations and Search
Eugene Yan
Eugene Yan breaks down how big platforms use reinforcement learning to power recommendations and search, moving past static models trained on old data. It matters because these systems adapt in real time, balancing new content discovery against giving users what they already like.
Based on reporting by Eugene Yan — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Traditional recommenders have a stale-data problem. Train on last week's clicks, deploy, repeat — and the whole time the model is chasing short-term clicks while ignoring whether users stick around, and it keeps recommending the same popular stuff because it never learns to explore anything new. Eugene Yan's latest piece lays out how companies like Yahoo, Netflix, JD, Microsoft, ByteDance and Google have tackled this with reinforcement learning instead, and the variety of approaches is honestly the interesting part.
Start with contextual bandits, the simplest tool in the box. Yahoo built one for news article recommendations back in the day, using what they called hybrid linear models — basically letting user-article cross features get shared across all the different articles being ranked, rather than treating each one in isolation. That let click data from one article inform predictions on a completely different one. Users were described with 1,193 categorical features, articles with 83, then everything got compressed down to six-dimensional vectors before being crossed into a 36-dimension feature space. Netflix took the same bandit idea and pointed it at something visual: picking which artwork thumbnail to show for each title on your homepage, using watch minutes as the reward signal.
Value-based methods go a level deeper, learning Q-values for state-action pairs instead of just picking an arm. JD's version is notable because it treats skipped items as real signal, not noise — most systems only learn from clicks and purchases, but JD split their model's memory into a State+ (what you clicked) and State- (what you ignored), each built from separate GRU-encoded sequences. Microsoft did something similar for news, but with a twist: they blended immediate click reward with a long-term
My take — AI-written commentary, not fact-checked reporting
Reinforcement learning for recommendations is exactly the kind of unglamorous, deeply technical work that never gets a splashy launch post but quietly reshapes what a billion people see on their phones every day, and I think that mismatch between impact and hype is worth sitting with. Companies love talking about ChatGPT wrappers; almost nobody outside ML teams wants to explain a dueling bandit gradient descent exploration scheme, even though it's arguably doing more to shape daily human behavior than any chatbot. My one skepticism: these systems optimize engagement proxies dressed up as 'long-term user activeness,' and a 0.05 reward weight on that signal, as Microsoft used, tells you exactly how much anyone actually prioritizes it over clicks.
Read more about this at: Eugene Yan