Why an old caching trick is your secret to lower LLM costs

An LLM will happily answer the same question over and over, and the meter keeps running. The trick is to ask a quieter question first: has anything changed that could alter the answer? If the request, its context, the model settings, or the underlying data are the same, there may be no reason to pay for another inference at all.

That idea is old, which is part of the point. In production data pipelines, the same waste shows up as nightly jobs that recompute the same aggregates again and again. Nothing is broken. The checks pass. The output lands in production. And yet compute gets burned for work that was already done. The fix is change detection: fingerprint the inputs and dependencies, then skip recomputation when the fingerprints match.

LLM workloads have the same problem, only the bill arrives in tokens. Duplicate requests are common enough to be boring. Users ask near-identical questions, batch jobs repeat boilerplate, CI runs hammer the same prompts, and tool-calling agents keep hitting the same knowledge base. Native prompt caching helps with provider-side reuse of prompt computation, but response caching goes further by skipping the model call entirely when the system already has a safe answer.

The article’s core framework has three tiers. Start with exact match: normalize the request, hash it with something like SHA-256, and look it up in Redis. If that misses, try semantic matching by embedding the query and searching a vector store for a close paraphrase. A common starting threshold is in the 0.90 to 0.95 range, though that has to be tuned against real queries. Then a hybrid setup does both, promotes semantic hits into the exact store, and keeps the cheap repeat wins coming.

None of that works unless the cache key includes more than the words in the prompt. Context, documents, model settings, source versions, and access scope all matter. Freshness matters too. A market-data answer may only be safe for a minute or two, while an internal HR policy can sit around much longer. The rough math in the piece says a workload of 1,000,000 calls a month at $0.006 per call could drop from about $6,000 to about $2,550 with a 60% hit rate and roughly $150 in embedding and vector-store costs. That’s a real cut, but only if you measure hit rate before you start promising savings.

There are also clear no-go zones: personal data, creative tasks, stock prices, live inventory, and anything else where stale is just wrong. The useful part here is not that caching is clever. It’s that the oldest trick in the book still beats paying twice for the same answer.