Alibaba’s Qwen team released Qwen3.8-2.4T-A95B as open weights and then documented how to run it on Amazon SageMaker HyperPod with vLLM. The deployment uses a ml.p6-b300.48xlarge instance with 8× NVIDIA B300 Blackwell Ultra GPUs, and it applies NVFP4 quantization to fit the model’s ~1.2 TB weights on the node. As a result, teams can serve a 2.4T-parameter Qwen-Max-class model through an OpenAI-compatible endpoint with controlled reasoning, tool calling, and native MTP speculative decoding while avoiding per-token API fees at scale.
Google will buy up to half the electricity from Finland’s Loviisa nuclear power plant to support new AI infrastructure. It announced a €13bn investment package that includes a 22-year contract with Fortum to purchase up to 50% of Loviisa output. Construction is scheduled for 2027–2028, expanding and adding data centres in Finland to power Google’s AI services and related energy projects.
TorchServe is no longer actively maintained, leaving operators without planned fixes for bugs, features, or security vulnerabilities and shifting responsibility for the full CUDA/PyTorch dependency chain to engineering teams. The AWS post introduces the Ray Serve Deep Learning Container, including a GPU-based setup validated for the Qwen3-VL-2B model on a g5.xlarge instance with a single NVIDIA A10G GPU and 24 GB VRAM. Teams can now swap from custom TorchServe maintenance to deploying an AWS-maintained Ray Serve container, avoiding version drift by using the pre-assembled inference stack and managing model code separately via a ConfigMap.
Lightbits announced general availability of Inferra, a KV cache engine that shifts key-value cache data off GPU high-bandwidth memory to improve AI inference performance. Lightbits says Inferra can cut time to first token by more than 100-fold on some long-context workloads. Instead of stalling on missing cache and idling GPUs, the system uses predictive prefetching to stage cache blocks in time, enabling larger context windows and more concurrent sessions.
NVIDIA released Mercury 2.5, a new production diffusion language model intended for low-latency, low-cost serving. Pricing for Mercury 2.5 starts at $0.20 per million input tokens and $0.75 per million output tokens, with a launch offer of 80% off ($0.04 input, $0.15 output). It updates deployments with higher quality (reported +40%) and faster throughput (1,107 tokens per second), and it also introduces a preview of Mercury Voice and Mercury Router for voice latency and model routing.
Cohere released a serving engine for its North Mini Code model built around a decode megakernel running BF16 on a single H100. On batch size 1 it reached 292 tok/s, or 62% of the H100’s bandwidth ceiling (SoL), and was about 1.58× faster than vLLM end-to-end. The serving stack changes from many small decode kernels with synchronization gaps to a single persistent kernel that uses task-level scheduling and synchronization to raise decode throughput without measurable accuracy loss.
Inference outputs from the same model and prompt can differ across repeated runs even at temperature 0 because GPU arithmetic order changes with how requests share a batch. Thinking Machines Lab reported that 1,000 identical requests returned 80 different completions. Reproducibility can be improved by turning on batch-invariant kernels in vLLM or SGLang, which typically costs about 38% of throughput.
The tech industry is facing a CPU supply shortage that is tightening infrastructure bottlenecks for software teams, including agentic workloads. Server orders are quoting about 6 months and prices are up roughly 10–20% since March. Teams now need to forecast and capacity-plan CPU usage earlier, optimize how capacity is isolated and utilized, and improve cluster bring-up and software efficiency.
Cleveland Clinic, RIKEN, and IBM were named 2026 ACM Gordon Bell Prize finalists for quantum-HPC chemistry research involving protein simulations. The work modeled a 12,635-atom protein system using IBM Quantum Heron processors integrated with supercomputers Fugaku, RUQUO, and Miyabi-G. Their new fully automated end-to-end workflow on RIKEN’s ROQUO reduces manual coordination and data movement while improving protein-ligand binding energy accuracy.
Every AI story that matters,
in your inbox by 8am.
TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the
day in two minutes. Follow companies and topics for alerts, or get the
briefing in Slack. Free, no spam, unsubscribe anytime.
Reading TLDRocket needs no cookies, and the readership counts we rely on come from
our own cookieless analytics. Google Analytics is the exception: it sets cookies and
reports to Google, so it stays switched off until you allow it. You can change your
mind any time from “Cookie settings” in the footer.
Strictly necessary
Session security and form protection (tldrocket-session,
XSRF-TOKEN, 2 hours). The site cannot work without them,
so they need no consent.
Always on
Google Analytics 4 (_ga,
_ga_<id>, up to 2 years). Measures which
stories and sections readers use. Google acts as a third-party processor and may
store the data outside the EU. No advertising, no profiling, no data sold.
