Mixture-of-Experts

81 summarised stories about Mixture-of-Experts, each linking back to the original source. Browse all topics →

Key players

NVIDIA Hugging Face MarkTechPost Inkling Kimi K3 OpenAI

Tuesday, 8 September 2026

Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6

Amazon Web Services 1 month ago 28

Amazon SageMaker AI benchmarked and recommended GPU instance configurations for serving two 30B-class Mixture-of-Experts LLMs on SageMaker Inference, comparing G5, G6, and G7 (Blackwell) options. In the Qwen3-Coder-30B test on the ml.g7.12xlarge endpoint, output token throughput reached 391.3 tok/s with an average request latency of 1,315.8 ms. The results led the article to conclude that G7 can deliver better price-performance, so users can either run direct endpoint benchmarks or use SageMaker AI generative AI inference recommendations to pick a deployment configuration.

Read the full story

AI Infrastructure GPU Infrastructure Model Evaluation Mixture-of-Experts

How llm-d makes the most of the hardware you already have

IBM Research 1 month ago 4

llm-d, an open-source inference framework led by IBM Research, Red Hat, and Google, demonstrated efficient serving of agentic LLM workloads on existing enterprise GPUs by avoiding redundant context processing and scaling different inference stages separately. It deployed GLM-5.2, about 753B parameters (39B active), on 544 NVIDIA H100 GPUs and reached over 6.6 million output tokens per minute at peak with up to 3,000 concurrent coding agents and zero preemptions. The result is that operators can self-host an open-weight frontier-scale mixture-of-experts model for long-context agents on H100 fleets with software changes that improve latency/throughput and cut per-token costs versus equivalent commercial API pricing.

Read the full story

AI Agents GPU Infrastructure Agent Infrastructure Mixture-of-Experts

Every AI story that matters,
in your inbox by 8am.

TLDRocket reads all relevant sources, removes duplicate coverage, and summarises the day in two minutes. Follow companies and topics for alerts, or get the briefing in Slack. Free, no spam, unsubscribe anytime.