Prompt Caching in the API

OpenAI quietly flipped a switch that should make a lot of developers' bills smaller without them lifting a finger. The company's new prompt caching feature in the API automatically gives a 50% discount on input tokens that match the beginning of a prompt the model recently processed. No special flag, no new parameter, no rewritten integration — it just happens in the background the moment your prompt crosses 1,024 tokens.

The mechanics are straightforward once you see them. If you're sending the same lengthy system prompt, few-shot examples, or document context over and over — the kind of setup common in coding assistants, RAG pipelines, or chatbots with persistent instructions — the overlapping portion gets treated as a cache hit. OpenAI keeps that prefix warm for 5 to 10 minutes after its last use, and during off-peak hours that window can stretch out to an hour. Miss the window and the cache clears; the next call just pays full price and starts building a new cache.

This rollout covers GPT-4o, GPT-4o mini, o1-preview, and o1-mini, including their dated snapshot versions, plus fine-tuned variants of those models. It works across text, images, tool definitions, and structured outputs, which means the discount isn't limited to plain chat completions — anyone running agents with heavy tool schemas or vision inputs benefits too.

What's notable here isn't the discount percentage so much as the friction OpenAI removed. Competing caching schemes elsewhere often require explicit cache keys or manual prompt restructuring, which developers tend to skip because it's extra work for uncertain payoff. By making it automatic and invisible, OpenAI is betting that most production traffic already has enough repeated structure — system prompts, retrieval templates, tool specs — to make the savings add up without anyone touching their code.