How NVIDIA’s Inference Software Stack Powers the Lowest Token Cost

Nvidia wants you to stop obsessing over chip specs and start thinking about cost per token — how many useful AI outputs you get per dollar, per watt, within a latency budget you actually care about. That's the real currency now that companies are running AI in production instead of tinkering with pilots, and Nvidia's pitch is that its software stack, layered on top of Blackwell GPUs, is where the real savings live.

The headline number is stark: token costs for DeepSeek V4 on Blackwell dropped up to 5x in roughly one month, driven by improvements across open source frameworks like vLLM and SGLang. That's not a hardware refresh — that's software catching up to silicon that was already sitting there. Nvidia frames this as three layers working together: production orchestration that manages distributed serving and autoscaling, application-level runtime tricks like kernel fusion and overlapping compute with communication, and low-level infrastructure access that spares developers from wrangling GPU instruction sets directly. Stack disaggregated serving, large expert parallelism over NVLink, NVFP4 precision and multi-token prediction together, and Nvidia claims throughput per GPU jumps up to 20x versus baseline.

Real companies are apparently seeing it. Baseten used TensorRT-LLM to squeeze 50% more tokens per second out of DeepSeek V4 Pro for coding and long-context work. DigitalOcean helped Hippocratic AI boost throughput 30% while keeping response times under half a second across 10 million patient calls — which matters a lot if you're a healthcare AI vendor promising instant answers. Cognition is leaning on Nvidia's Dynamo framework to manage GPUs for reinforcement learning instead of building that plumbing itself, and Together AI used TensorRT-LLM to help Cursor move faster from model tweaks to live coding endpoints.

The bigger argument here is about agentic AI breaking the old rules of scaling. A simple web request used to be predictable — load a page, hit a database, add more servers. An AI agent spinning up subagents, calling tools and juggling multi-turn context turns one request into potentially thousands of tasks spread across GPUs, CPUs, DPUs and storage. Without software that can coordinate all of that, you're just burning capacity. Nvidia's bet is that its stack, plus a CUDA-native open source ecosystem — PyTorch chief among them — lets new research land and run at full speed on its hardware almost immediately, whether that's speculative decoding tricks like DFlash or fast video generation via FastVideo.

And that's really the flywheel Nvidia is selling: more developers optimize for CUDA, more of that work becomes production software, and every improvement compounds into lower token costs across millions of deployed GPUs. It's a compelling story, if a self-serving one — because naturally the company that makes the chips also wants to own the software layer that decides how cheap those chips are to run.