llm-d 1.0 ships: the CNCF distributed-inference layer from Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA adds multi-tenant QoS, zero-downtime ops and agent-aware routing

Why it matters

【Key points】

llm-d, the open-source distributed inference project, shipped 1.0 on Oct 20 (GitHub tag 02:30 UTC; IBM, Google Cloud and Red Hat posts from 11:30 UTC). Founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA and now a CNCF Sandbox project, it sits above vLLM/SGLang and routes requests across Kubernetes clusters.

1. New: multi-tenant QoS (priority/fairness flow control, controlled load shedding); zero-downtime ops (router HA, rolling updates, DisaggregatedSet for prefill/decode replicas); a stable v1 config API; session-aware routing and turn-priority scheduling for agents.

2. 500+ contributors from 25+ organizations; own inference images dropped for upstream vLLM/SGLang images.

3. Project-reported: GLM 5.2 (~753B) served 6.6M output tokens/min at peak and up to 3,000 concurrent coding agents with zero preemptions, at 5–10x lower per-token cost than commercial API pricing (cloud rental rates).

4. Google Cloud is integrating 1.0 into GKE across TPUs and NVIDIA GPUs; DigitalOcean, JetBrains (Junie Lite), Mistral, Cohere, Roblox and AMD also back it.

【Verifiable】

Release notes: https://github.com/llm-d/llm-d/releases/tag/v1.0.0

IBM press release: https://newsroom.ibm.com/2026-10-20-llm-d-1-0-advances-open,-cost-efficient-ai-inference-for-production-scale

Google Cloud blog: https://cloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0-an-open-source-stack-for-distributed-inference/

【Our read】

1. The cluster layer above the engine drives serving cost and was mostly closed in-house code; stable APIs let llm-d become an enterprise standard.

2. DeepSeek's DeepInfer, released the same day, leaves out the KV cache, prefill/decode disaggregation and cross-replica balancing; llm-d covers exactly that layer.

3. The 5–10x and GLM 5.2 numbers are the project's own tests against list prices, not real bills.