vLLM and NVIDIA use agent-written CuTe DSL kernels to speed up Kimi K3 decode on GB300: 842 tokens/s at batch 1 (1.73x stock vLLM), plus a 'fast-track' review process for agent-optimized code

Why it matters

【Key points】

A vLLM blog post dated Oct 16 (by NVIDIA's Benjamin Chislett), pushed by @vllm_project on X on Oct 18, describes an agent-assisted CuTe DSL decode path for Kimi K3 (2.8T MoE) on NVIDIA GB300 that fuses attention, MoE and communication: 13 kernels / 69 µs per layer vs 23 / 130 µs in stock vLLM.

On 16 GB300s (TP16, batch 1) with DSpark (synthetic acceptance length 6), decode hits 842 tokens/s vs 488 stock (1.73x). Without speculation, TP16 gains 1.62–1.92x and TP8 1.31–1.58x over batch 1–64. AgentX P90 interactivity at concurrency 1: 403.2 vs 223.7 tokens/s/user. Opt-in via decode_backend: latency_specialized.

The post also proposes a "fast-track" optimization RFC because agent-written code outpaces line-by-line review: isolate changes in model code, bound scope without dropping production features, validate with kernel tests plus model evals, and let contributors run standard evals on their own GPUs. Next: DeepSeek-V4.1-Flash, Muse Spark Open (>950 tokens/s/user early) and GLM-5.3.

【Verifiable】

https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel

https://x.com/vllm_project/status/2111637219647840611

July baseline (TP16 batch 1: 118, or 370 with DSpark): https://vllm.ai/blog/2026-07-27-k3

【Our read】

1. Agents can now write hard GPU kernels, so the bottleneck has moved to review. vLLM shifts review onto automated verification and isolation, the same direction as Anthropic's Claude Code review process.

2. Caveats: 842 tokens/s uses a synthetic acceptance length; the AgentX 1.8x compares 16 GPUs to an earlier 8-GPU run. The numbers are self-reported.

3. It came out the same day as the AMD MI355X post (e/906). Kimi K3 is now the shared benchmark for both NVIDIA and AMD inference claims.