CATS Z. Li Internet-Draft Z. Du Intended status: Informational China Mobile Expires: 5 January 2027 J. Wang W. Cheng G. Zhang Centec X. Sun Inesa C. Zhao SAIA 4 July 2026 KV Cache Distribution for Distributed LLM Inference: Use Case and Requirements draft-li-cats-kv-cache-distribution-00 Abstract In large language model (LLM) inference, the key-value (KV) cache holds the attention state computed from previously processed tokens. Reusing cached state across requests avoids repeated prefill computation and reduces time-to-first-token. In distributed inference deployments, the KV cache becomes a network-distributed resource: the effectiveness of steering a request to a service instance depends not only on computing and network metrics but also on whether reusable cached state is available at or near that instance. This document describes the KV cache distribution use case for Computing-Aware Traffic Steering (CATS), identifies the gaps relative to the existing CATS framework and metrics, and states requirements for cache-state metric exposure and for the distribution and synchronization of cached content across multiple cache tiers. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Li, et al. Expires 5 January 2027 [Page 1] Internet-Draft KV Cache Distribution July 2026 Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 5 January 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 3 4. Cache-State Metric Exposure . . . . . . . . . . . . . . . . . 4 5. Distribution and Synchronization of Cached Content . . . . . 4 6. Security Considerations . . . . . . . . . . . . . . . . . . . 4 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 4 8. Normative References . . . . . . . . . . . . . . . . . . . . 4 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 5 1. Introduction LLM applications routinely require knowledge beyond what is contained in model parameters: conversation history for personalized assistants, enterprise data for domain-specific agents, retrieved documents for search-augmented responses. Two established approaches address this need, each with trade-offs. In-context approaches (prompt construction and retrieval-augmented generation) supply external knowledge at inference time without modifying the model, but are constrained by the model's context window and incur per-request retrieval and prefill computation. Fine-tuning adapts model parameters to new knowledge, but requires training resources and labeled data, and binds the result to specific tasks. Li, et al. Expires 5 January 2027 [Page 2] Internet-Draft KV Cache Distribution July 2026 KV cache reuse occupies a useful middle ground. The key-value (KV) cache holds the attention state computed from previously processed tokens. Reusing cached state across requests avoids repeated prefill computation and reduces time-to-first-token. In distributed inference deployments, the KV cache becomes a network-distributed resource: the effectiveness of steering a request to a service instance depends not only on computing and network metrics but also on whether reusable cached state is available at or near that instance. This document describes the KV cache distribution use case for Computing-Aware Traffic Steering (CATS), identifies the gaps relative to the existing CATS framework and metrics, and states requirements for cache-state metric exposure and for the distribution and synchronization of cached content across multiple cache tiers. 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 2. Terminology KV Cache: Intermediate attention state (keys and values) produced during inference. In disaggregated serving, the KV cache is transferred from prefill servers to decode servers. Prefill: The inference phase that processes the input prompt and produces the initial KV cache. Decode: The inference phase that generates output tokens incrementally, consuming and extending the KV cache. Expert Parallelism (EP): A parallelization strategy for mixture-of- experts models in which experts are distributed across servers, requiring all-to-all token exchange. 3. Problem Statement In distributed LLM inference, the KV cache computed during prefill is a valuable resource that can be reused to avoid recomputation. When a request arrives, steering it to an instance that already holds the relevant KV cache state avoids prefill latency. However, the KV cache is distributed across service instances and cache tiers; a CATS decision that considers only computing load and network metrics Li, et al. Expires 5 January 2027 [Page 3] Internet-Draft KV Cache Distribution July 2026 cannot account for cache availability. This creates a gap: the existing CATS metrics do not expose cache state, and the CATS distribution framework does not address how cached content is distributed or synchronized across instances. This document states requirements to close this gap. 4. Cache-State Metric Exposure A service instance SHOULD expose cache-state metrics that allow a CATS decision function to determine whether reusable KV cache state is available. These metrics include: the set of cache keys (e.g., token-prefix identifiers or content hashes) present at the instance; the cache entry sizes; and the expected retrieval cost. The metrics MUST be expressible in the CATS metric framework so that they can be combined with computing and network metrics in steering decisions. 5. Distribution and Synchronization of Cached Content For cache reuse to be effective across instances, cached content may need to be distributed or migrated. This document identifies the following requirements: (a) a mechanism to advertise cache availability across instances; (b) a mechanism to transfer KV cache state between instances or cache tiers, respecting latency and bandwidth constraints; (c) consistency handling for cache entries that are updated or evicted. The specific transfer protocol is outside the scope of this document but should build on existing transport building blocks. 6. Security Considerations Cache-state metrics and cache content may reveal information about the inputs being processed. Exposure of cache metrics MUST be restricted to authorized CATS components. Transfer of KV cache state between instances SHOULD be protected for integrity and confidentiality. 7. IANA Considerations This document has no IANA actions. 8. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . Li, et al. Expires 5 January 2027 [Page 4] Internet-Draft KV Cache Distribution July 2026 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . Authors' Addresses Zhiqiang Li China Mobile Beijing 100053 China Email: lizhiqiangyjy@chinamobile.com Zongpeng Du China Mobile Beijing 100053 China Email: duzongpeng@chinamobile.com Junjie Wang Centec Shanghai 201203 China Email: wangjj@centec.com Wei Cheng Centec Shanghai 201203 China Email: chengw@centec.com Guoying Zhang Centec Shanghai 201203 China Email: zhanggy@centec.com Li, et al. Expires 5 January 2027 [Page 5] Internet-Draft KV Cache Distribution July 2026 Xun Sun Inesa Shanghai 200030 China Email: sunxun@inesa.com Chunhao Zhao SAIA Shanghai 200125 China Email: chunhao.zhao@sh-aia.com Li, et al. Expires 5 January 2027 [Page 6]