KV Cache Distribution for Distributed LLM Inference: Use Case and Requirements

Introduction LLM applications routinely require knowledge beyond what is contained in model parameters: conversation history for personalized assistants, enterprise data for domain-specific agents, retrieved documents for search-augmented responses. Two established approaches address this need, each with trade-offs. In-context approaches (prompt construction and retrieval-augmented generation) supply external knowledge at inference time without modifying the model, but are constrained by the model's context window and incur per-request retrieval and prefill computation. Fine-tuning adapts model parameters to new knowledge, but requires training resources and labeled data, and binds the result to specific tasks. KV cache reuse occupies a useful middle ground. The key-value (KV) cache holds the attention state computed from previously processed tokens. Reusing cached state across requests avoids repeated prefill computation and reduces time-to-first-token. In distributed inference deployments, the KV cache becomes a network-distributed resource: the effectiveness of steering a request to a service instance depends not only on computing and network metrics but also on whether reusable cached state is available at or near that instance. This document describes the KV cache distribution use case for Computing-Aware Traffic Steering (CATS), identifies the gaps relative to the existing CATS framework and metrics, and states requirements for cache-state metric exposure and for the distribution and synchronization of cached content across multiple cache tiers.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology

KV Cache:: Intermediate attention state (keys and values) produced during inference. In disaggregated serving, the KV cache is transferred from prefill servers to decode servers.
Prefill:: The inference phase that processes the input prompt and produces the initial KV cache.
Decode:: The inference phase that generates output tokens incrementally, consuming and extending the KV cache.
Expert Parallelism (EP):: A parallelization strategy for mixture-of-experts models in which experts are distributed across servers, requiring all-to-all token exchange.

Problem Statement In distributed LLM inference, the KV cache computed during prefill is a valuable resource that can be reused to avoid recomputation. When a request arrives, steering it to an instance that already holds the relevant KV cache state avoids prefill latency. However, the KV cache is distributed across service instances and cache tiers; a CATS decision that considers only computing load and network metrics cannot account for cache availability. This creates a gap: the existing CATS metrics do not expose cache state, and the CATS distribution framework does not address how cached content is distributed or synchronized across instances. This document states requirements to close this gap.

Cache-State Metric Exposure A service instance SHOULD expose cache-state metrics that allow a CATS decision function to determine whether reusable KV cache state is available. These metrics include: the set of cache keys (e.g., token-prefix identifiers or content hashes) present at the instance; the cache entry sizes; and the expected retrieval cost. The metrics MUST be expressible in the CATS metric framework so that they can be combined with computing and network metrics in steering decisions.

Distribution and Synchronization of Cached Content For cache reuse to be effective across instances, cached content may need to be distributed or migrated. This document identifies the following requirements: (a) a mechanism to advertise cache availability across instances; (b) a mechanism to transfer KV cache state between instances or cache tiers, respecting latency and bandwidth constraints; (c) consistency handling for cache entries that are updated or evicted. The specific transfer protocol is outside the scope of this document but should build on existing transport building blocks.

Security Considerations Cache-state metrics and cache content may reveal information about the inputs being processed. Exposure of cache metrics MUST be restricted to authorized CATS components. Transfer of KV cache state between instances SHOULD be protected for integrity and confidentiality.

IANA Considerations This document has no IANA actions.