Inference Optimization

133 summarised stories about Inference Optimization, each linking back to the original source. Browse all topics →

Key players

NVIDIA OpenAI Together AI The New Stack Hugging Face MarkTechPost

Wednesday, 29 July 2026

ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale

Together AI 2 months ago 35 ● 3 sources

Researchers introduced ThunderAgent, a scheduling system that eliminates inefficient KV cache eviction during multi-turn agent reasoning by treating entire agentic workflows as schedulable programs rather than individual requests. The system achieved 2.5× single-node throughput improvement and 2.4× speedup on an 8-node cluster with near-linear scaling from 16 to 64 GPUs in synthetic data generation workloads. The approach enables existing inference engines to handle concurrent agentic inference more efficiently without architectural changes to underlying models or backends.

Read the full story

Agent Infrastructure Inference Optimization

Configuring Dedicated Model Inference

Together AI 2 months ago 27 ● 2 sources

Together AI's Dedicated Model Inference platform uses endpoints (stable identities), deployments (model + hardware combinations), and configs (runtime recipes) to route traffic based on capacity rather than fixed percentages. Traffic splits assign weights to deployments, and the router distributes requests proportionally to effective capacity (weight × ready replicas), so autoscaling automatically adjusts traffic without manual intervention. This architecture enables A/B tests, rollouts, shadow experiments, and zero-downtime changes, with a live experiment showing that when deployment A scaled from 1 to 2 replicas, its traffic share increased from 50% to 69.4% as the router followed the capacity change.

Read the full story

Inference Optimization Together AI

Every AI story that matters,
in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.