AWS contributes push-based and bi-directional KV-cache transfer to vLLM for disaggregated inference, reporting up to 3x faster time-to-first-token in its tests

Why it matters

【Key points】

AWS has published two KV-cache transfer improvements for prefill/decode-disaggregated inference in vLLM and says both are contributed upstream as accelerator-agnostic features.

1. Push-based transfer: vLLM's NIXL connector normally has the decode side pull KV blocks after prefill, putting setup on the first-token path. AWS lets prefill push instead. On a 1P1D EC2 P5en (GPU, EFA) setup with Qwen3-32B: up to 3x better TTFT and 30% better TPOT.

2. Bi-directional transfer: in multi-turn agent chats, KV built on the decode node never returned to prefill, so it was recomputed each turn. Sending it back cut TTFT up to 1.6x for Qwen3-32B and 3.3x for GLM-5-FP8.

3. Both were built on Trainium clusters; the published results are from GPUs.

【Verifiable】

Push-based KV transfer: https://aws.amazon.com/blogs/machine-learning/push-based-kv-transfer-to-improve-ttft-and-tpot-for-disaggregated-inference-in-vllm/

Bi-directional KV transfer: https://aws.amazon.com/blogs/machine-learning/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-transfer-in-vllm/

【Our read】

1. Disaggregated serving is now standard at scale (llm-d 1.0, DeepSeek's stack, NVIDIA Dynamo), and moving KV between the two sides is increasingly the bottleneck.

2. Both changes target agent workloads, where long, repeated histories hurt TTFT most. AWS upstreamed them to vLLM rather than keeping them in its own stack, which helps GPU users and pulls Trainium into the mainstream open ecosystem.

3. Caveats: AWS's own numbers, a small 1P1D setup, RDMA (EFA) required; check the vLLM repo for merge status and versions.