Improving Recommendation Systems & Search in the Age of LLMs

Eugene Yan · 1 year ago

Industrial recommendation and search systems are increasingly adopting language models and multimodal content to improve performance, with approaches like YouTube's Semantic IDs (using residual quantization to compress video embeddings into discrete representations), Kuaishou's M3CSR (clustering multimodal embeddings to achieve 3.4% click increase), Huawei's FLIP (aligning ID-based models with LLMs through masked pretraining), Google's CALRec (finetuning PaLM-2 for sequential recommendations), and Meta's EmbSum (using T5 and Mixtral to generate user interest summaries). Key technical innovations include two-stage training frameworks, multimodal fusion architectures, and joint learning between tabular and textual modalities to address cold-start problems and improve ranking efficiency. These methods demonstrate measurable improvements in click-through rates, coverage metrics, and generalization across datasets compared to traditional ID-based or single-modality approaches.