Conclusion: bi-directional KV transfer improves TTFT stability for multi-turn agentic workloads by reusing decoded KV blocks, with configurable TTL and recompute thresholds to balance memory and compute; results vary by workload and configuration.

To reproduce results, follow the provided guide and use vLLM v0.21.0+ with NixlConnector, enable bidirectional_kv_xfer on prefill and decode servers, and employ a kv-cache-aware router as documented in the linked GitHub resources.

The enhancement requires extending the NVIDIA Inference Xfer Library (NIXL) KV connector and introducing a KV-cache-aware router that maintains in-memory metadata mapping conversation IDs to decode KV transfer parameters across turns.

The results were demonstrated on AWS infrastructure (P5en with EFA) and validated with both Trainium and GPU configurations; charts illustrate TTFT trends and prefill duration breakdowns between recompute and KV-transfer components.

KV recompute threshold (kv_recompute_threshold, default 64 tokens) determines when the prefill node should recompute vs. fetch KV from the decode node, allowing tuning based on model, KV size, and interconnect performance.

The article includes diagrams and figures illustrating standard flow vs. bi-directional flow, and provides links to implementation details, benchmarks, and reproduction steps.

Bi-directional KV transfer enables a decode node to return previously computed KV blocks to the prefill node in subsequent turns, reducing redundant KV recomputation and improving TTFT for long conversations.

KV blocks are retained on the decode side for a configurable TTL (decoder_kv_blocks_ttl, default 480 seconds) and reused by the prefill node if a new turn arrives before expiry; otherwise, normal prefill occurs.

Performance results on a 1P1D EC2 P5en deployment with AWS EFA show TTFT reductions up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8 as context length grows; there is a small additional KV-transfer time but substantial net gain due to reduced recomputation.