How to Match LLM Patterns to Problems
Eugene Yan ● Covered by 2 sources
Eugene Yan mapped common LLM engineering patterns to the actual problems they solve. It's a practical cheat sheet, not another abstract framework.
Based on reporting by Eugene Yan — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Eugene Yan's follow-up to his earlier LLM patterns piece skips the theory and gets straight to the point: which fix goes with which headache. He starts by splitting the world into external LLMs (GPT-4, Claude, the stuff you rent by the token) and internal ones you host yourself. External models are state of the art but come with rate limits, data privacy worries, and zero control over fine-tuning. Internal models give you freedom but cost real engineering time, and since most teams build on open weights rather than pre-training from scratch, they're usually months behind the frontier labs anyway.
Yan then walks through specific failure modes and pairs each with a remedy. No performance metrics for your task? That's an evals problem, paired with user feedback collection so you know if people actually like the output. An external model giving stale or hallucinated answers because it doesn't know your internal docs or anything after its training cutoff? That's what RAG exists for, feeding fresh context in at inference time, with evals to check whether your retrieval tuning is actually helping. An internal model that's just mediocre out of the box on your domain? Fine-tuning, backed by the same user feedback loop, is the lever you have that external-model users don't.
He also covers the messier operational stuff. Rate limits, legal restrictions on sending private data outbound, and ballooning API costs push teams toward either painful vendor negotiations or biting the bullet on self-hosting, which loops back to fine-tuning and evals again. Latency-sensitive use cases that can't rely on streaming responses need caching strategies smarter than plain semantic similarity, things like keying off item IDs for near-instant lookups. And when output itself is unreliable, Yan splits the failure into syntactic errors, broken JSON, code that won't run, versus semantic ones, off-topic or unsafe content, both handled by different flavors of guardrails.
The most memorable bit is a throwaway anecdote near the end: a company ran an LLM-based customer support tool in production for two weeks before pulling it, after an A/B test showed losses twelve times higher than their human support team. Yan uses it to argue for monitoring and feedback loops as non-negotiable, not nice-to-haves, since without them you might not notice you're bleeding money until someone runs the numbers. The piece closes as an invitation, asking readers what problems he missed, which fits its tone throughout: less manifesto, more field notes from someone actually shipping this stuff.
My take — AI-written commentary, not fact-checked reporting
This is the kind of pragmatic, unglamorous writing the LLM space desperately needs more of, and less of the breathless 'AGI is here' takes. The twelve-times-worse support anecdote should be tattooed on every product manager's laptop before they ship an LLM feature without a monitoring plan. My only gripe: he treats fine-tuning as the default answer for internal models, when for a lot of teams better retrieval and prompting would get them 90% of the way there for a fraction of the cost.
Read more about this at: Eugene Yan