Sunita Nadampalli, a Principal Engineer at AWS, and Nicolò Lucchesi, a Research Engineer at Mistral AI and a vLLM maintainer, published the contribution alongside benchmark data on AWS EC2 P5en instances. At 8 queries per second with 16,000-token inputs — a load profile representative of production coding assistants and document-analysis pipelines — push mode achieved approximately 540 milliseconds TTFT, compared to approximately 1,744 milliseconds for the existing pull-based connector: a reduction that moves the user experience from the "breaks conversational flow" zone (greater than 1 second) toward the "feels responsive" threshold (under 500 milliseconds).

KV Transfer Is on the Critical Path to First-Token Response

Modern high-throughput LLM serving has converged on prefill-decode (P-D) disaggregation: splitting inference into two phases running on separate GPU pools. The prefill phase is compute-intensive, processing the entire input prompt to build a key-value cache — a data structure that encodes the attention patterns for every token in the prompt. The decode phase is memory-bandwidth-intensive, using that cache to generate output tokens one at a time. Because these phases have fundamentally different hardware demands, routing them to dedicated, independently scalable instances improves overall resource utilization and throughput under concurrency, as covered in current disaggregated inference guides.

The architectural price is the KV handoff. After a prefill instance finishes computing the cache, it must physically transfer that cache to the decode instance before token generation can begin. That transfer is directly on the wall-clock path to the user's first token.

In vLLM's previous pull-based model, the handoff is sequential. The router relays source metadata — prefill block IDs — to the decode instance after prefill completes. The decode instance then allocates destination memory, prepares NIXL (the NVIDIA Inference Transfer Library) transfer descriptors, and initiates a NIXL READ to pull the KV data from the prefill node. Every one of these coordination steps happens after prefill has already finished, stacking latency on top of latency. At light load and with short prompts, the overhead is negligible. At 8 queries per second with 16,000-token inputs, it accumulates to about 1,744 milliseconds—nearly two seconds before the first token appears —as shown in vLLM pull-mode performance analysis.

The Push Protocol: Three Steps That Move Coordination Off the Critical Path

The push connector inverts the initiation logic. Instead of waiting for prefill to complete before the decode instance begins its memory setup, decode pre-registers its target memory while prefill is still running. When prefill finishes, the data can be written directly into already-prepared destination memory — no round-trip, no inline descriptor setup on the decode side.

Three protocol steps implement this design.

Registration. When the decode instance allocates memory for a request that requires remote prefilling, it immediately prepares a registration record describing the exact GPU memory locations where the prefill instance should write the KV data. This record is transmitted to the prefill instance as an NIXL notification at allocation time — while prefill is still processing the prompt. The destination is already known when the data is ready to move, as detailed in the NixlPushConnector design documentation.

Rendezvous. The registration from the decode side and KV computation completion on the prefill side may arrive in either order. The connector handles both gracefully: if registration arrives first, it waits for KV completion; if KV finishes first, it waits for registration. As soon as both are present, the transfer is triggered. The connector pairs requests by ID and correctly handles preempted or retried requests with a block-lease-and-watchdog mechanism.

Transfer. Once a request is matched, the prefill instance establishes or reuses an NIXL connection to the decode instance and issues an NIXL WRITE that copies the KV blocks directly into the decoder's pre-registered GPU memory, targeting each tensor-parallel rank. Critically, this work runs on a dedicated background thread — the nixl-push-writer thread, one per tensor-parallel worker — that never touches the prefill instance's main forward-pass loop. The prefill engine hands off the block IDs and immediately returns to processing new requests. When the transfer completes, NIXL notifies the decode instance, which proceeds directly to token generation.

Background Threading: The Architectural Key

The background threading is not an implementation detail — it is the reason the protocol works at scale. Every expensive per-request operation in the transfer happens on the nixl-push-writer thread: establishing connections, mapping block IDs to NIXL memory descriptors, merging contiguous blocks into larger transfer regions for efficiency, and submitting the NIXL WRITE. The prefill main loop never blocks on any of this. At high concurrency, when dozens of requests are completing prefill simultaneously, those transfers run in parallel on background threads while the prefill compute continues processing the next batch uninterrupted.

The push connector's design reveals something important about where the latency in disaggregated inference actually lives. The P5en instances used in the benchmark run on AWS EFAv3 (Elastic Fabric Adapter, third generation) at 3,200 Gbps, with GPUDirect RDMA enabling direct GPU-to-GPU data movement across nodes without staging through host RAM. The raw interconnect is fast enough that bandwidth is not the constraint. The constraint is the serial coordination protocol — the sequence of round-trips and descriptor preparation steps that happen after prefill, before the actual transfer begins. The push connector eliminates that coordination overhead in software without requiring faster hardware, as described in the NVIDIA NIXL inference library blog. Further TTFT improvements in disaggregated inference will continue to come from protocol design, not from buying more expensive interconnects.

By the Numbers: 540 ms vs. 1,744 ms at Scale

The benchmark configuration was a 1P1D topology — one prefill node, one decode node — running Qwen3-32B on Amazon EC2 p5en.48xlarge instances connected via EFA, with prefix caching disabled to force full KV transfers on every request. Software stack: vLLM 0.29.0, NIXL 1.4.1, CUDA 13.0. Full data appears in the vLLM NixlPushConnector benchmark results.

At 8 queries per second with 16,000-token inputs:

• TTFT: push mode achieved approximately 540 ms versus approximately 1,744 ms in pull mode — roughly a 3× reduction.
• TPOT (time per output token): pull mode climbed to approximately 10.4 ms per output token at 4–8 queries per second with long contexts; push mode held near 7 ms — approximately a 30% improvement.

At low load with short prompts, both modes perform nearly identically — approximately 5.6–5.8 ms TPOT — confirming the overhead is specifically a concurrency and context-length problem that accumulates precisely under the conditions most relevant to production, as shown in benchmark comparisons.

The 540-millisecond figure matters because it sits near a meaningful user-experience threshold. Extensive LLM latency research on user experience places 500 milliseconds as the lower boundary of "feels responsive" for interactive applications, with user engagement degrading above 1 second and dropping significantly above 3 seconds. Pull mode's 1,744 milliseconds lands firmly in the "breaks conversational flow" zone. Push mode's 540 milliseconds is just above the "feels responsive" threshold — enough of a difference to determine whether a coding assistant or agentic pipeline feels interactive or batch-like to end users.

Hardware Context: GPUDirect RDMA and the Role of NIXL

The benchmark hardware is the Amazon P5en instance announcement: the p5en.48xlarge carries eight NVIDIA H200 Tensor Core GPUs connected via NVSwitch at 900 gigabytes per second for intra-node GPU-to-GPU bandwidth. The inter-node interconnect is EFAv3 (Nitro v5), delivering 3,200 Gbps with GPUDirect RDMA support — the capability that allows NIXL to move KV blocks directly between GPUs on different nodes without routing the data through host RAM on either machine.

NIXL (NVIDIA Inference Transfer Library) is the open-source, vendor-agnostic data movement layer that the push connector builds on, described in NVIDIA NIXL blog. It provides a unified API for non-blocking, zero-copy transfers across GPU memory, CPU memory, NVMe storage, and object stores (S3 over RDMA, Azure Blob). Its UCX default backend handles the transport-layer negotiation, selecting RDMA, RoCE, or InfiniBand automatically based on available hardware.

The AWS team originally built and validated the push connector on AWS Trainium instance clusters connected with EFA, which supports Neuron Direct RDMA via NIXL. The fact that the same implementation works on both H200-based P5en and Trainium clusters without modification demonstrates that the protocol design is accelerator-agnostic — portability was a design requirement, not a side effect.

A Deliberate Trade-Off: Earlier Decode Commitment

Push mode is not a pure upgrade. In pull mode, the router selects which decode instance handles a request after prefill completes, preserving maximum flexibility for load-aware routing: the router can inspect all available decode instances at completion time and send the request to the least loaded. Push mode must select the decode instance before prefill finishes, so that destination memory can be pre-registered. This earlier commitment reduces routing flexibility.

For TTFT-sensitive deployments with relatively predictable decode-instance load — most interactive chat, copilot, and RAG applications — the trade-off favors push. For workloads requiring bidirectional KV transfer (multi-turn agentic workloads that send KV back to the prefill node) or dynamic, latency-sensitive load balancing across highly variable decode pools, the existing pull connector remains the better fit. vLLM now supports both, and operators select per-deployment by configuring NixlPushConnector in place of NixlConnector on both the prefill and decode servers, per NixlPushConnector configuration guide.

What Comes Next: PR #45211 and Background Threading for Pull Mode

The AWS team has opened a pull request — vllm-project/vllm #45211 — to explore bringing the same background-threading design to the existing pull connector. The core insight behind the push connector is that NIXL descriptor preparation and transfer submission can run on background threads without affecting inference correctness. That insight is not exclusive to push mode. If the pull connector adopts background threading, deployments that require bidirectional KV transfer or dynamic routing would gain their own TTFT reduction without the decode-instance-selection constraint.

If that work lands in a future vLLM release, the performance gap between push and pull modes would narrow, and the routing-flexibility trade-off would carry less cost.

Note: Verify Your vLLM Version Before Deploying

Operators running vLLM in production should confirm they are on version 0.14.0 or later before deploying any disaggregated inference configuration. CVE-2026-22807 vLLM security advisory — disclosed January 21, 2026, with a CVSS score of 8.8 (High) — allows arbitrary code execution through dynamic module loading when vLLM loads a model from an untrusted source, triggering before any API request is processed. The benchmark configuration described in this article used vLLM 0.29.0, which is not affected. The vulnerability was patched in v0.14.0.

The push connector is available now in the vLLM codebase, with a step-by-step reproduction guide in the vLLM project blog.

Frequently Asked Questions

What is the core difference between push-based and pull-based KV transfer in vLLM disaggregated inference?

In pull mode, the decode instance begins setting up its memory and NIXL transfer descriptors after the prefill instance has already finished — all coordination happens serially on the critical path to the first token. In push mode, the decode instance pre-registers its destination GPU memory while prefill is still running, so that when KV computation finishes, the prefill instance can write data directly into pre-prepared memory on a background thread. The coordination happens concurrently with prefill rather than after it, removing serial overhead from the path to TTFT, as detailed in NixlPushConnector design docs.

What hardware and software do I need to run the push-based KV connector in vLLM?

The connector requires vLLM 0.29.0 or later, NIXL 1.4.1 or later, and GPU instances with RDMA-capable interconnects — AWS P5en (EFAv3 + GPUDirect RDMA) and Trainium clusters with EFA are confirmed working. The connector is accelerator-agnostic: NIXL's UCX backend handles transport-layer negotiation across InfiniBand, RoCE, and EFA. Prefix caching must be considered alongside deployment requirements, as the benchmark ran with it disabled to force full KV transfers. Full setup instructions are in the AWS Machine Learning Blog setup guide.

Does push mode outperform pull mode in all conditions?

No. At low load with short prompts — fewer than 4,000-token inputs at low query rates — both modes perform nearly identically (approximately 5.6–5.8 ms TPOT). The push connector's advantages are load-sensitive and context-length-sensitive: they become significant at 4–8 queries per second with 8,000–16,000-token inputs. Additionally, push mode requires selecting a decode instance before prefill completes, which reduces routing flexibility for workloads that need dynamic load balancing or bidirectional KV transfer. Operators should benchmark their specific load profile, with full data available in NixlPushConnector benchmark data and analysis, to determine whether push mode's TTFT gains outweigh its routing constraints.

Is vLLM safe to deploy in production right now?

On version 0.14.0 or later, yes — a High (CVSS 8.8) arbitrary code execution vulnerability affecting earlier versions through dynamic module loading was patched in v0.14.0, disclosed January 21, 2026, per the CVE-2026-22807 security advisory. Deployments on 0.14.0 or later are not affected. Any production environment running an older version should upgrade before deploying disaggregated inference or any configuration that loads models from external sources.