Open challenges in LLM research

Chip Huyen spent time talking to people across industry and academia and came away with a pattern: everyone chasing the same handful of hard problems, just from different angles. Her list runs to ten items, but a few stand out for how far they are from solved.

Hallucination sits at the top, unsurprisingly. In a panel with Dropbox, Langchain, Elastic and Anthropic, it came up as the number one reason companies hesitate to put LLMs into production. The workarounds people lean on now — chain-of-thought prompting, self-consistency checks, stuffing more context into the prompt — are patches, not fixes. Context length is the second big theme, tied closely to retrieval-augmented generation, which has become the default pattern for enterprise LLM apps. Huyen points to research showing models read the beginning and end of a long context far better than the middle, which is an awkward thing to discover once you've built your whole product around long documents.

What she's genuinely excited about is multimodality, and her reasoning is practical rather than trendy: healthcare, retail, robotics and e-commerce all generate messy mixes of text, images and tabular data, and a model that only reads text is going to hit a ceiling — partly because, as she notes, the internet may simply run out of usable text to train on. She lists Flamingo, BLIP-2, KOSMOS-1, PaLM-E and LLaVA as the current frontier, and floats an application that's easy to overlook: multimodal models helping visually impaired users navigate the web and the physical world.

Cost is the other quiet revolution. GPT-3.5 launched in November 2022 needing roughly 375GB to fine-tune. By March 2023, Alpaca matched a chunk of that performance at 15GB. By May, Guanaco got close on just 6GB — under 2% of GPT-3.5's footprint — thanks to quantization, distillation, low-rank factorization and pruning, techniques Huyen was writing about years before this boom. Her conclusion: build something good enough and the community will figure out how to shrink it.

On hardware, she's watching two long shots — quantum computing and photonic chips — the latter attracting real money, with Lightmatter, Ayar Labs and Lightelligence collectively raising north of half a billion dollars to move data with light instead of electricity. And on architecture, she flags that Transformer, unchanged in its core form since 2017, has been so heavily optimized that dislodging it requires something that beats it at scale on today's hardware, not just on paper. Chris Ré's lab's Monarch Mixer is one of the few public attempts at a subquadratic alternative, and Huyen suspects there are quiet efforts elsewhere nobody's published yet.