Algorithm Innovation

108 summarised stories about Algorithm Innovation, each linking back to the original source. Browse all topics →

Tuesday, 16 June 2026

Frontier post-training recipe review with Finbarr Timbers

Interconnects 1 month ago

Post-training recipes for large language models have converged on multi-teacher on-policy distillation (MOPD) as a frontier approach in 2026, replacing earlier monolithic reinforcement learning stages with domain-specialist teachers merged into a single student model. The shift occurred because single-stage RL proved expensive and created capability conflicts across math, code, and reasoning domains, while specialist models using SFT-then-RL per domain are cheaper and organizationally scalable. This architectural change, pioneered by MiMo Flash V2 in January 2026 and scaled by DeepSeek V4 and Nemotron 3 Ultra to over 10 teachers, enables labs to expand post-training complexity beyond what single-stage RL recipes like OLMo-3 could achieve.

Algorithm Innovation Deep Reinforcement Learning Fine-tuning Foundation Models

The Sequence Knowledge #878: Beyond Transformer: What We Learned

TheSequence 1 month ago

A series examining transformer alternatives concludes that the transformer architecture's dominance is ending, with four families of alternatives emerging: recurrent and linear-recurrent models, state space models, text diffusion, and liquid continuous-time models. State space models like Mamba show linear scaling and long-context handling with near O(n) compute versus attention's O(n²), though the strongest current results use hybrid architectures combining attention with other approaches. The future architecture will likely be explicitly hybrid, using attention only where exact recall justifies its quadratic cost and deploying linear-time alternatives elsewhere.

AI Architecture Algorithm Innovation Transformer Alternatives