The coalition behind llm-d announced version 1.0 of their open-source distributed large language model inference framework at PyTorch Conference North America, releasing independently verified production data showing IBM Research achieved a blended serving cost of approximately $0.11 per million tokens — roughly 11.5 times lower than an equivalent proprietary API stack. The milestone release, backed by IBM, Red Hat, Google Cloud, CoreWeave, NVIDIA, and more than 20 additional organizations, marks a deliberate shift from performance demonstrations to Day 2 operational maturity: stable APIs, zero-downtime upgrades, and multi-tenant quality-of-service controls designed for enterprise production at scale.

The timing is pointed. As enterprises move past AI experimentation into full-scale production, many are discovering that running AI at scale can be as costly and operationally complex as training the models themselves. For context-heavy agentic workloads — where autonomous agents continuously access and reuse vast amounts of conversational history — inference costs have become the dominant constraint on AI adoption. llm-d 1.0 is the industry's collective answer to that bottleneck.

What Is llm-d?

llm-d is a Kubernetes-native, open-source stack for distributed LLM inference. It sits above inference engines such as vLLM and SGLang, adding a cluster-level coordination layer those engines don't provide on their own: intelligent request routing, distributed KV-cache management, and disaggregated prefill/decode processing. The result is a system that turns a fleet of model servers into a coordinated inference pool that serves frontier-class open models more efficiently than the sum of its parts. More details on the project architecture are available at the llm-d project website.

Red Hat, IBM Research, Google Cloud, CoreWeave, and NVIDIA launched the project in May 2025 with a founding vision of any model, any accelerator, any cloud. It joined the Cloud Native Computing Foundation (CNCF) as a Sandbox project in March 2026, placing it under vendor-neutral governance and the Linux Foundation's stewardship. Since launch, the project has grown to more than 500 contributors across 25-plus organizations, including AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley, and the University of Chicago.

(Note for readers: CNCF Sandbox is the foundation's earliest maturity tier. The v1.0 designation is the project's own declaration of production readiness — not a CNCF Graduated or Incubating designation.)

Why Inference Costs Have Become the Bottleneck

To understand why llm-d matters, it helps to understand the architecture mismatch at the heart of modern LLM inference. A typical request passes through two computationally distinct phases: prefill, which processes the input prompt and is heavily compute-bound, and decode, which generates output tokens one by one and is memory-bandwidth-bound. In most conventional serving stacks, these phases run on the same hardware in the same process, leading to inefficient resource utilization and unpredictable latency.

Context management adds a second layer of cost. In agentic workloads, each agent request arrives with a long shared prefix — system prompts, tool definitions, conversational history — that has often already been computed by an earlier request. A system without cache-aware routing simply recomputes it from scratch, wasting GPU cycles and inflating both latency and cost.

llm-d addresses these problems through three interlocking mechanisms:

1. Prefix-cache-aware routing. Rather than distributing requests round-robin across replicas, llm-d reads KV-cache state events from the underlying inference engine and routes each incoming request to the replica that already holds its input prefix in cache. A request can then skip the most expensive part of prefill entirely. In IBM Research's agentic coding workloads, llm-d's router captured a prefix-cache hit rate approaching 98%.

2. Hierarchical KV-cache offloading. GPU high-bandwidth memory (HBM) is fast but scarce. llm-d maintains a global index of which token blocks live on which replica, and spills KV-cache entries to CPU memory or local SSD when GPU memory is full, extending the effective working set without dropping context. Benchmarks show this technique delivers a 13.9× throughput improvement at 250 concurrent users compared to GPU-only serving.

3. Disaggregated prefill/decode (P/D). llm-d separates prompt processing and token generation onto distinct worker pools, matching hardware to workload characteristics: compute-heavy prefill on GPU-dense nodes, memory-bound decode on memory-optimized nodes. The NIXL (NVIDIA Inference Xfer Library) connector transfers KV-cache blocks between pools. AWS, running disaggregated inference on Amazon SageMaker HyperPod with llm-d, reported 70% higher tokens per second versus standard vLLM.

Together, these techniques enable the cost profile that has drawn enterprise interest. IBM Research demonstrated that serving GLM-5.2 — a 744-billion-parameter (39B active), 78-layer mixture-of-experts model — on widely deployed NVIDIA GPU fleets using llm-d cut per-token costs 5 to 10× versus equivalent commercial API pricing, at peak throughput exceeding 6.6 million output tokens per minute while concurrently serving 3,000 coding agents with zero preemptions.

What Version 1.0 Adds: Day 2 Operational Maturity

Previous releases through the 0.x series established llm-d's reach — proving that the same stack could run on NVIDIA, AMD, Intel, and Google TPU hardware, across public clouds and on-premises clusters. Version 1.0 shifts the focus from performance demonstration to the unglamorous but critical work of running inference reliably, across many tenants, through upgrades, under real-world load.

Multi-Tenant Quality of Service

A shared inference fleet must protect latency-sensitive interactive requests while absorbing bursty, unpredictable agent workloads. v1.0 introduces priority- and fairness-aware flow control at the router level: every request is classified by a flow key combining a fairness ID (tenant or workload) and a priority band, with separate queues per flow dispatched by priority first, then cycled fairly across tenants. A probabilistic admission controller sheds load gracefully when demand exceeds capacity. The default behavior mimics first-come-first-served, so existing deployments are unaffected until operators opt into multi-tenant policy.

Zero-Downtime Operations

Upgrades that drop traffic are not acceptable in production. v1.0 addresses this on multiple fronts. Router high availability provides a hot-standby Endpoint Picker that takes over scheduling immediately if the active instance fails, ensuring routing continuity during rollovers. For disaggregated deployments, a new DisaggregatedSet abstraction manages prefill and decode replicas as a single logical unit, coordinating lockstep upgrades so requests only reach a prefill/decode pair once both sides are fully rolled out and compatible.

Agent-Aware Serving

Agentic workloads reuse vast amounts of context across conversational turns, and paying to recompute that context on every turn is one of the largest sources of avoidable inference cost. v1.0 introduces session-aware routing that keeps a multi-turn conversation pinned to the replica already holding its KV cache, using sticky-until-saturated affinity so sessions stay in place until the replica fills up. Turn-priority scheduling lets the router reason about where a request sits in an ongoing exchange, keeping active agent loops responsive. Native support for the Responses API's previous_response_id allows context to be carried across turns without the agent resending its full history.

Does llm-d 1.0 Mean a Stable API to Build On?

Yes. Version 1.0 ships the first stable, versioned llm-d configuration surface, giving platform teams a foundation to standardize on without chasing breaking changes between releases. The Endpoint Picker request path has been hardened and validated across clouds and accelerators.

Hardware Across Every Major Platform

One of llm-d's core architectural commitments is accelerator independence. The routing, scheduling, and disaggregation layers are hardware-agnostic, with validated native paths for each platform:

• NVIDIA: Validated on Hopper (H100, H200), Blackwell (B200), and Grace Blackwell (GB200 NVL72 and GB300), with KV-cache transfer powered by NIXL from NVIDIA Dynamo. Support for the upcoming Vera Rubin platform is in progress.
• AMD: Supports AMD Instinct MI355X with AINIC across vLLM, SGLang, and AMD ATOM, with MoRI-IO and NIXL on ROCm for disaggregated serving.
• Intel: Comprehensive support across Intel Xeon and GPU platforms, with llm-d-lens for observability and upcoming support for next-generation D-series GPUs (codenamed Crescent Island).
• Google TPU: Native PyTorch model serving via TorchTPU inside upstream vLLM and SGLang, powered by TPU Sync for KV-cache movement. On a TPU agentic coding workload, llm-d's prefix-aware routing plus host-memory KV-cache offload cut first-token latency 85% and raised total throughput 2.4× versus a plain Kubernetes Service.

Community-contributed paths additionally support Rebellions NPUs, Iluvatar BI-V150, and MetaX C500X GPUs.

Production Deployments: Who's Running It and What They Report

llm-d is not a research project. A confirmed roster of organizations reported production use at the time of the v1.0 announcement:

• Google Cloud builds its GKE Inference Gateway on the llm-d Endpoint Picker, reporting a reduction in time-to-first-token of more than 35% in Vertex AI production and a doubling of its prefix-cache hit rate from 35% to 70%. It is integrating llm-d 1.0 into GKE across its TPU and NVIDIA GPU portfolio.
• Red Hat ships llm-d as the distributed inference stack in Red Hat AI, generally available on OpenShift and on managed Kubernetes including Microsoft Azure and CoreWeave.
• DigitalOcean runs its Serverless Inference and Managed Agents services on llm-d, serving the trillion-parameter Kimi K3 model across mixed AMD and NVIDIA hardware.
• JetBrains builds Junie, its AI coding agent, on llm-d with predicted-latency-based scheduling to keep responses fast under heavy agent load.
• Mistral AI serves open models in production on llm-d as part of its regional inference offering.
• Prime Intellect integrates llm-d into its large-scale reinforcement learning training stack.
• IBM Research self-hosts frontier open models on llm-d for coding agents and chip-design agents, achieving a blended serving cost of about $0.11 per million tokens—roughly 11.5× lower than a proprietary stack—with peak throughput exceeding 6.6 million output tokens per minute and 3,000 concurrent agents running with zero preemptions.
• AWS runs disaggregated inference with llm-d on Amazon SageMaker HyperPod, using NIXL over Elastic Fabric Adapter.
• Tesla and Red Hat co-published a KServe case study showing a 3× increase in output tokens per second and a 2× reduction in time to first token running Llama 3.1 70B on AMD MI300X GPUs with llm-d's cache-aware routing.

What Industry Leaders Said at the Launch

"As organizations move beyond AI experimentation and into full-scale production, the bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads," said Brian Stevens, senior vice president and AI CTO at Red Hat. "With llm-d 1.0, the community is delivering a standardized, hardened core for distributed inference to empower enterprises to optimize hardware performance, address operational costs, and build high-density AI architectures on an open foundation."

Priya Nagpurkar, vice president of AI native systems at IBM Research, framed the shift as a maturity milestone: "The challenge of enterprise AI is no longer just building powerful models but running them efficiently and reliably at scale. As agentic workloads become more prevalent, open technologies like llm-d will be critical to improving performance, lowering costs, and accelerating innovation across the AI ecosystem."

Mark Lohmeyer, vice president and GM of AI and computing infrastructure at Google Cloud, highlighted the operational dimension: "Agentic AI is unlocking tremendous value, but as these workloads scale rapidly in production they place unique demands on the infrastructure. The release of llm-d 1.0 is an important milestone for the industry to address these needs."

Ujval Kapasi, vice president of engineering AI frameworks at NVIDIA, pointed to next-generation hardware integration: "The next wave of agentic AI demands inference that delivers performance and efficiency at scale. Bringing llm-d to NVIDIA Vera Rubin will help enterprises build and scale advanced AI applications on the next generation of NVIDIA accelerated infrastructure."

Chris Aniszczyk, CTO of the CNCF, placed the release in the broader cloud-native context: "AI inference is quickly becoming a core cloud native workload. llm-d 1.0 is an important step toward making high-performance open source inference more operationally mature and accessible across the cloud native ecosystem." All five executives' full remarks appear in the llm-d v1.0 blog post.

The Bigger Picture: Why These Vendors Are Building an Open Stack

The subtext of the llm-d 1.0 announcement is a structural debate about where AI infrastructure margins should live. As commercial API pricing from frontier model providers remains high — particularly for the input-heavy traffic patterns typical of agentic workloads — the business case for self-hosted inference on open models has strengthened considerably.

What the "open vs. proprietary" framing does not state directly: the founding coalition of llm-d includes hardware vendors (NVIDIA, AMD, Intel) and cloud infrastructure providers (Google Cloud, AWS, CoreWeave) whose economic interests are served by commoditizing the inference software layer. When inference software is free, commoditized, and multi-vendor, the remaining margin in the AI serving stack flows to GPU compute and cloud infrastructure — where these companies operate. Building llm-d is not altruism; it is a deliberate architectural strategy to keep the AI value chain's economics anchored in hardware and cloud layers rather than captured by model-API intermediaries. Enterprises that understand this dynamic can navigate the "open vs. proprietary" choice more clearly: the open stack is genuinely free and genuinely well-maintained, and the coalition's incentives are broadly aligned with enterprise infrastructure teams who want to avoid lock-in.

The CNCF governance model and Apache 2.0 license reinforce this structural point: by placing the project under vendor-neutral stewardship, the coalition prevents any single member from capturing the stack, making it safer for enterprises to build on — and no single vendor's withdrawal would strand adopters.

What Comes After Version 1.0

The community signaled several areas of active development beyond 1.0: active-active router configurations for higher availability, inference request failover and migration, expanded multimodal and audio serving, wider upstream vLLM wide-expert-parallelism coverage on managed Kubernetes, and deeper GitOps integration for ArgoCD-based deployment pipelines. Support for NVIDIA Vera Rubin and Intel's next-generation D-series GPUs (codenamed Crescent Island) is in progress.

llm-d 1.0 is available now as open-source software under the Apache 2.0 license. Code, documentation, and tested deployment paths from a single node to a production fleet are available at the llm-d GitHub repository and the llm-d project website.

Frequently Asked Questions

What is llm-d, and how is it different from vLLM or SGLang?

vLLM and SGLang are single-server inference engines that manage requests, KV-cache, and generation on an individual machine or GPU pod. llm-d is a cluster-level coordination layer that sits above them, adding capabilities those engines don't provide on their own: routing requests to the replica that already holds the relevant KV-cache prefix, managing KV-cache state across multiple nodes, and splitting prefill and decode work across separate hardware pools. An organization using vLLM can adopt llm-d without replacing vLLM — llm-d reads vLLM's published cache state events and controls routing at the Kubernetes gateway layer. Full architecture details appear in the official v1.0 blog post.

Why does prefix-cache-aware routing reduce AI inference costs so dramatically?

The most expensive phase of LLM inference is prefill — processing the full input prompt, including any shared system prompt, conversational history, or tool definitions. For agentic workloads, those shared elements can be thousands of tokens long, and many concurrent agents share the same prefix. Without cache-aware routing, a naive load balancer sends each request to a random replica, which must recompute the prefix from scratch. llm-d's router reads the KV-cache state of each replica and sends each request to the replica that already holds its prefix in GPU memory — eliminating the recomputation for that request. At a 98% hit rate (as IBM Research reported), 98% of the most expensive computation is replaced by a cache lookup. This is why the per-token cost drops by an order of magnitude for long-context agentic workloads specifically. The prefix cache routing mechanism is documented in detail in the v1.0 announcement.

What is the real-world business case for self-hosting with llm-d versus a commercial API?

The business case depends on scale and context length. For organizations running high-volume, long-context, or multi-turn agentic workloads — the kind where the input prompt is hundreds to thousands of tokens per request — self-hosted inference on open models via llm-d can deliver 5 to 10 times lower per-token cost than commercial API equivalents, with IBM Research reporting 11.5× lower cost at $0.11 per million tokens. For lower-volume or shorter-context workloads, the operational overhead of self-hosting (GPU procurement, cluster management, team expertise) may not be justified by the cost savings. The v1.0 release specifically addresses the operational overhead concern by providing multi-tenant quality-of-service controls, zero-downtime upgrades, and a stable API — the features that previously required custom engineering to achieve.

Who controls llm-d, and what happens if a founding organization withdraws support?

llm-d is governed by the Cloud Native Computing Foundation (CNCF) as a Sandbox project under the Linux Foundation, with an Apache 2.0 license. The Apache 2.0 license means any organization — including organizations not currently involved — can fork, modify, and redistribute the code. The CNCF governance structure means no single founding organization (not Red Hat, not IBM, not Google) can unilaterally change the project's direction, take it proprietary, or prevent others from using it. This is the structural difference between a project with vendor-neutral governance and a vendor-open-sourced project that one company controls. If any one founding member withdraws, the remaining 24-plus organizations and 500-plus contributors continue the project under the same governance.