Chip Huyen explains how to cut inference costs without new hardware

At P99 CONF, Chip Huyen made a point that cuts through a lot of AI hype: training is a one-time bill, but inference keeps charging you every time someone types a prompt. Over the life of a model, she said the split between training and inference compute tends to land somewhere between 1:10 and 1:100. Reasoning models push that even further because they chew through more tokens. If inference stays expensive, the training spend never really comes back, which explains a lot of the jokes about frontier-model profitability.

Huyen’s advice starts with measurement. She said teams should care about time to first token, time per output token, and end-to-end latency, because each one tells you something different about the user experience. For some reasoning systems, even the first generated token is not the first visible one, since the model may think for a while before showing anything. She also argued for goodput, not just throughput: it’s not enough to process a lot of requests if most of them miss the latency target anyway.

She broke inference optimization into three buckets: hardware, model, and service. Hardware was mostly off the table for her talk, both because she used to work at Nvidia and because most teams cannot change the chips they get. Replica parallelism, the blunt-force option of adding more machines, was also set aside as expensive and messy. That left the model itself and the service handling the requests.

On the model side, quantization was the obvious winner. Lowering precision from 32-bit to 8-bit cuts memory use and can speed things up too, though Huyen said there is usually a small quality tradeoff. She also pointed to distillation, where a larger model generates training data for a smaller one, but warned that licensing terms can block that path if the provider forbids training competitive models on its outputs.

The service tricks are where a lot of real-world wins live. Batching helps, but continuous batching is better when requests finish at different times. Separating prefill from decode lets teams tune machines for either faster first tokens or faster token-by-token generation. Prompt caching may be the most practical of all: reuse the shared system prompt, examples, or reference documents once, then stop paying for them again. Huyen even said the cache hit rates she saw in Claude Code logs were high enough to matter, and that stable prompt prefixes should come first.

She closed with a warning that still feels current: cheaper and faster are not the whole story. Inference services can quietly change model behavior or reduce quality, so the benchmark number matters too. That is the part people like to skip, right up until the model starts saving money by becoming less useful.