These startups are chasing the next big thing in LLMs

MIT Technology Review · 2 months ago · 13

Startups are targeting limitations in transformer-based large language models as the next step in LLM design. Subquadratic says its sparse attention approach rivals mainstream models on some tasks and it plans to make its SubQ model widely available soon. The push is shifting development toward alternatives like sparse attention, retention-based context handling, smaller hybrid models, and diffusion-style text generation.