A new open-source framework, llm-d, is being developed by IBM Research, Red Hat, Google, and collaborators to efficiently serve large language models at scale on existing infrastructure, specifically targeting agentic workloads with long contexts, heavy context reuse, and bursts of parallel sub-agent activity.

The NIXL-based data plane delivered 6.2 million KV-cache transfers at approximately 580 Gb/s with zero transfer failures, and cache decomposition showed about 85% of input tokens served from cache, drastically reducing uncached prefill.

Benchmarking shows sustained high throughput across hundreds to thousands of concurrent agents, including 2,000–3,000 coding agents handling up to 134.89 million tokens per minute input and 6.05 million tokens per minute output; CyberGym with 400 concurrent agents achieved 967.7 requests per minute and p90 time-to-first-token around 17.6 seconds; AgentX with 128 concurrent agents reached 7.7 requests per second and a p90 TTF of 5.77 seconds.

Self-hosting GLM-5.2 on H100s with llm-d cuts per-token cost by 5–10x relative to current commercial API pricing, highlighting strong cost efficiency for enterprise deployments.

Frontier-scale open models can be self-hosted on existing H100 fleets, providing a viable path to production-grade, agentic inference without waiting for newer accelerators.

llm-d achieves efficiency through prefix-aware routing to reuse cached context, tiered KV-cache management extending cache into CPU memory, peer-to-peer KV-cache sharing, wide expert parallelism with data-parallel attention, disaggregated prefill and decode pools, and multi-token prediction to boost throughput.

A demonstration deployed GLM-5.2, a ~753-billion-parameter mixture-of-experts model, on 544 NVIDIA H100 GPUs, reaching over 6.6 million output tokens per minute at peak and supporting up to 3,000 concurrent coding agents with zero preemptions.

Agentic workloads are characterized by extremely long contexts, frequent reuse of prior inputs, and bursts of concurrent sub-tasks; about 96% of main-agent requests reuse at least 90% of a previous input, underscoring the cache and reuse value.

llm-d remains open source as a Cloud Native Computing Foundation sandbox project, with deployment guides, validated configurations, and multi-platform support to encourage ongoing community contributions.