Should You Self-Host Inference?

There's a familiar moment in a lot of engineering teams right now: the API bill triples in a quarter, someone in security points out that customer data is riding along in every prompt to OpenAI's servers, and somebody finally asks in standup whether they should just run the thing themselves. TLDR Dev's answer, in short, is: probably not yet, and maybe never entirely.

The math comes down to three levers — how much volume you're pushing, whether your data legally has to stay put, and whether you have the people to keep GPUs alive at 2am. Below roughly one million tokens a day, a hosted API is flatly cheaper, since you're not paying for idle hardware or an MLOps engineer, who runs about $160,000 a year according to Glassdoor, dwarfing the cost of the GPU they're babysitting. Between one and two million tokens, it's a toss-up. Past two million, owned hardware starts to pay for itself, and past ten million a day it typically earns itself back within six to twelve months.

There's a fourth constraint that caps the whole equation regardless of volume: model access. The frontier labs — OpenAI, Anthropic, Google — never release their weights, so the strongest model you could ever run yourself is whatever the best open-weight model happens to be at the time. Anything needing GPT-5 or Claude Opus-level reasoning has to go out over an API no matter how much volume you're pushing or how strict your compliance rules are.

A middle option, managed deployment, splits the difference: you rent GPUs from a provider, but the weights and data stay in your own cloud tenancy, with no ops team required. It clears most compliance reviews without the hiring headache, though a dedicated H100 through a provider runs about $4 an hour, which adds up to roughly $35,000 a year running around the clock — about what the card itself costs to buy. Run it at full utilization and you're effectively re-renting the same hardware every single year.

Most production setups, per TLDR Dev, land on a hybrid: sensitive or high-volume work stays local, and only the rare, reasoning-heavy request goes out to a frontier API. Teams doing this report savings of 40 to 70 percent against an all-API setup. The catch is that a router is one more system to build and maintain, and a misclassified request can quietly send exactly the sensitive data you built the local lane to protect straight out to the API you were trying to avoid.