CoreWeave targets AI inference bottlenecks with full-stack optimization

CoreWeave is betting that the next fight in AI won’t be won by whoever has the most GPUs. It will be won by whoever can serve models faster, cheaper, and with less friction once the training run is over.

That’s the direction Urvashi Chowdhary, CoreWeave’s vice president of product and AI services, laid out at the Fully Connected event. She said the company has been layering managed services on top of its infrastructure for training, post-training and inference, because AI developers want to move quickly without giving up performance or scale.

The pressure point is inference. A survey of CoreWeave customers and prospects by theCUBE Research found one healthcare customer’s inference workload rose from about 10% in the first year to 40% in the second, with roughly 50% expected within 12 months. CoreWeave is responding by tuning the stack above the hardware, including the vLLM engine, quantized models and custom speculative decoders.

Chowdhary also pointed to reinforcement learning as a new source of strain. When customers train agentic models with rewards and verifiers, inference can become the bottleneck during rollouts. CoreWeave’s answer is RL Rollouts, a preview capability built on Nvidia’s Dynamo framework that loads new checkpoints into a live deployment. In testing, it cut model reload latency by 15x versus a baseline configuration.

Those pieces now sit inside CoreWeave Forge, a platform launched at the event that ties together serving, observability, post-training and evaluation. Forge is free to start, with paid tiers adding more capabilities. CoreWeave is also leaning on open source, saying it wants customers to have flexibility while still building managed services on top of the stack.