The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

Baseten raised $13 billion this year, joining Nvidia and the rest of the semis crowd as one of the loudest winners of what people are now calling the inference inflection. That's a lot of money for a company whose job, in plain terms, is making other people's models run fast and cheap. But talk to Philip Kiely and Ali Taha for twenty minutes and you understand why investors are paying attention: inference engineering barely existed as a job title three years ago, and now it's arguably the harder problem in AI, not the training run everyone still writes headlines about.

The two walked through what actually happens when a 200,000-token prompt hits Baseten's systems, and it's a lot messier than 'GPU goes brrr.' First question the system asks is whether it's seen part of that prompt before, because reusing a cached KV state saves real money. If not, the request gets routed to a prefill worker, a GPU dedicated purely to chewing through input and generating the first token, which then hands off to a separate decode worker for the token-by-token generation. Speculative decoding sits in front of all this, using a small draft model to guess several tokens ahead so the big model only has to verify rather than generate from scratch. Ali noted that a draft model trained on coding traffic will nail those guesses; ask it to summarize four Harry Potter books instead and acceptance rates, and speed, drop fast.

The most counterintuitive finding involves quantization, the practice of shrinking model weights to run faster. Conventional wisdom says compressing more of a model degrades it. Baseten found the opposite in a GLM-5.2 experiment: quantizing additional layers actually held benchmark scores steady while lifting throughput 20 percent, because errors introduced in one layer partially canceled errors from another. That's not a fluke worth ignoring — it's the kind of result that reshapes how engineers think about where compression is safe. The team also described grafting Kimi's vision encoder directly onto GLM-5.2 without touching the underlying language model, a retrofit that sounds almost surgical and hints at how much of 'shipping a new open model' is really assembling parts from different labs.

Zoom out and the conversation covers ground well beyond a single model release: NVIDIA's Dynamo framework, KV-aware routing across clusters, whether GPUs are quietly morphing into programmable AI ASICs ahead of Rubin, and why open-source video generation still lags Veo and Kling because of the quadratic cost of attention over long sequences. Local inference, they argued, is fundamentally about making a model less dumb on constrained hardware, while data-center inference is about making an already-smart model less slow at scale. Different problem, different playbook, same underlying discipline.

Maybe the strangest idea in the whole discussion is the closing of the loop: GLM-5.2 was reportedly used to help optimize the very kernels that serve GLM-5.2. Training and inference, long treated as sequential steps, are starting to blur into a continuous cycle of deploy, evaluate, retrain, and redeploy — with persistent KV caches even pointing toward a crude form of continual memory. If that trend holds, the line between 'the model' and 'the infrastructure running the model' gets blurrier every quarter.