Cutting Our Inference Bill by 68% by Being Obsessive About Optimization
Cloudflare Blog
A code review system’s average review cost rose from about $1.19 (April) to about $2.46 (end of August), then dropped after mid-September token-usage and caching optimizations. The average review cost fell to $0.77 (down 68%) after changes that also reduced median review time to 1m 32s from 4m 09s. The work shifted spending away from repeated context by improving prompt caching and reducing the number of LLM calls, plus adjusting models and reviewer “effort” levels.
Why it matters
Cloudflare reports lowering its average review cost from $2.43 to $0.77, with the biggest single saving coming from checking what actually reached the provider on each session’s first request. The issue highlights ordering messages consistently and verifying cache usage and reviewer effort.