Assess the entire inference stack — not just open model benchmarks
The New Stack Mark Chillingworth
Open-model benchmarks look good, but Nebius says production success depends on the whole inference stack. That means cache, latency, and workflow fit can matter more than the score.
Based on reporting by The New Stack, Mark Chillingworth — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Benchmarks can make open models look neatly sortable on paper. In production, Ryan Hanrui Wang says, that neatness disappears fast. A model has to perform again and again, with the right input, the right workflow, and the right level of accuracy. For long-horizon tasks, a benchmark score may not tell an engineer whether the system will finish the job at all.
Wang, SVP of Applied AI at Nebius, says the company treats public benchmarks as only one input. The Amsterdam-based cloud provider, which sells inference through its Token Factory platform, works with customers to define internal benchmarks and harnesses based on their own agent workflows and traces. Nebius brought Wang in through its acquisition of Eigen AI, the inference-optimization startup he co-founded and ran as CEO, in a deal announced in May and valued at about $643 million.
The point is not just model choice. Wang says engineers should think about the whole inference stack: serving, quantization, batching, caching, kernel optimization, and the post-training work that improves fit and accuracy. He also pushes teams to look at throughput, concurrency, and tail latency, not just averages. A system can look fast on paper and still feel broken if a slice of requests stalls. For some customers, he says, time to first token needs to stay below five seconds.
He is also blunt about benchmark mismatch. A math reasoning test may say little about a customer-support agent, where emotional intelligence matters more. And cheap tokens don’t help if the model keeps missing the task. Nebius says it found that balance in one specialized assistant project by using 6,000 traces, improving accuracy by 11% while using 22% fewer tool calls. Wang’s broader argument is simple: stop worshipping model size and start optimizing for the job.
My take — AI-written commentary, not fact-checked reporting
This is the part of AI everyone keeps trying to skip because it’s less glamorous than posting a benchmark chart. The real product is the mess around the model: caches, traces, latency, and whether the thing actually does the job twice in a row. Benchmark bragging is cheap; production is where the bill comes due.
Read more about this at: The New Stack