]> China Mobile

Beijing100053 China lizhiqiangyjy@chinamobile.com

China Mobile

Beijing100053 China duzongpeng@chinamobile.com

Centec

Shanghai201203 China wangjj@centec.com

Centec

Shanghai201203 China chengw@centec.com

Centec

Shanghai201203 China zhanggy@centec.com

Inesa

Shanghai200030 China sunxun@inesa.com

SAIA

Shanghai200125 China chunhao.zhao@sh-aia.com

TSV TSVWG inference RDMA ECMP load balancing AI workload transport Large-scale distributed inference systems generate traffic patterns that differ from both traditional data center workloads and distributed training workloads. Disaggregated prefill/decode serving transfers key-value cache state between server pools, and expert-parallel architectures generate all-to-all traffic among expert groups. These flows are typically carried over a small number of RDMA connections, producing low-entropy traffic that is prone to uneven link utilization under Equal-Cost Multipath (ECMP) forwarding. This document specifies transport considerations for such networks, covering path load awareness, path steering through ECMP entropy variation, ordering tolerance at the receiver, and differentiated reliability for data with different loss sensitivity. The discussion builds on existing IETF building blocks; this document does not define new protocol elements.

Introduction Early inference serving deployed models on single servers or small clusters, with modest demands on the interconnection network. Current large-scale inference systems are different in several respects. Disaggregated serving separates the prefill phase (processing the input prompt) from the decode phase (generating output tokens) onto distinct server pools. The key-value (KV) cache computed during prefill is transferred over the network to the decode pool, and the latency of this transfer directly affects time-to-first-token. Mixture-of-experts models deploy expert-parallel (EP) groups across many servers. Token routing between experts generates all-to-all communication whose scale grows with the EP group size, regularly crossing leaf and spine tiers of the data center fabric. This traffic is typically carried by RDMA transports such as RoCEv2 over a small number of connections between any given pair of endpoints. The resulting flows are large and few -- low-entropy traffic from the perspective of flow-based load balancing. With Equal-Cost Multipath (ECMP) forwarding, the hash function maps each flow to one path; with few flows, multiple large flows can hash onto the same link while parallel links remain idle. The operational issues of low-entropy traffic with flow-based load distribution are described in . In inference fabrics, such collisions translate into jitter and increased tail latency for KV cache transfer and all-to-all exchanges. This document specifies transport-layer considerations for these networks: how endpoints can become aware of per-path load, how traffic can be steered across paths using existing ECMP mechanisms without network upgrades, what ordering tolerance is required at receivers, and how reliability can be differentiated for data with different loss sensitivity. The intent is to document the considerations and map them to existing IETF building blocks. This document does not define new protocol elements or data plane behavior.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology

Prefill:

The inference phase that processes the input prompt and produces the initial KV cache.

Decode:

The inference phase that generates output tokens incrementally, consuming and extending the KV cache.

KV Cache:

Intermediate attention state (keys and values) produced during inference. In disaggregated serving, the KV cache is transferred from prefill servers to decode servers.

Expert Parallelism (EP):

A parallelization strategy for mixture-of-experts models in which experts are distributed across servers, requiring all-to-all token exchange.

Message:

An application-level unit of transfer (e.g., one RDMA operation). A message comprises one or more packets.

Entropy:

Variability in the packet header fields used by ECMP hashing to select among equal-cost paths, as discussed in .

Problem Statement Several operational approaches are deployed today to mitigate uneven link utilization caused by low-entropy RDMA traffic. Each involves trade-offs. Increasing the number of connections: spreading traffic over more RDMA connections (queue pairs) increases entropy and reduces the probability that large flows collide on a single link; however, additional queue pairs consume NIC resources and add scheduling overhead. Placement affinity: scheduling communicating workers under the same top-of-rack switch or leaf reduces the volume of traffic crossing the tiers where collisions occur; this reduces exposure but does not change the load-distribution behavior itself. Per-hop dynamic load balancing: switches can forward packets of the same flow across different links based on real-time link load; this achieves fine-grained balance but introduces packet reordering that receivers must tolerate. A complementary approach is for endpoints to steer traffic across paths using mechanisms that ECMP fabrics already support, informed by an endpoint view of per-path load. The remainder of this document discusses the considerations for this approach.

Transport Considerations

Path Load Awareness Endpoint-driven path steering benefits from knowledge of the relative load of the candidate paths. Two sources of this knowledge are available with existing building blocks. On-path telemetry: in networks where devices support In situ Operations, Administration, and Maintenance (IOAM) , probe or data packets can collect per-hop information along their forwarding path, including transit delay and queue depth; the export of collected data is described in . Endpoint estimation: where on-path support is not available, endpoints can estimate relative path load from end-to-end measurements such as round-trip time and delivery rate per path, in the manner familiar from delay-based congestion control. A sender MAY combine both sources where available. Load information is advisory input to path steering and MUST NOT be interpreted as a congestion signal in the sense of ; existing congestion control behavior is unchanged.

Path Steering Using ECMP Entropy ECMP path selection is a function of packet header fields. For RoCEv2 traffic, the UDP source port is commonly included in the hash input, and varying it changes the selected path without any change to network devices; the fabric continues to perform standard flow-based ECMP. The use of header entropy for load distribution is discussed in , and an analogous technique using the IPv6 Flow Label is described in . A sender that observes uneven path load MAY change the entropy value (e.g., the UDP source port) used for subsequent traffic on a connection, causing that traffic to be hashed onto a different path. Senders SHOULD rate-limit such changes; frequent repathing can itself induce load oscillation across the fabric.

Ordering Considerations Changing the path of in-flight traffic reorders packets across the change. The disruption can be confined by aligning path changes with application-level message boundaries: all packets of a given message SHOULD carry the same entropy value, so that each message traverses a single path and arrives in order within itself; the entropy value MAY differ between messages, distributing successive messages across paths. With this alignment, the receiver observes reordering only between messages, not within a message. Receivers of multipath traffic MUST tolerate inter-message arrival reordering. For transports where each message is independently placed in receiver memory (as with RDMA operations carrying explicit placement information), inter-message reordering does not require reassembly buffering. The message size determines the granularity of load distribution. Smaller messages distribute load more evenly but increase per-message overhead; larger messages reduce overhead but coarsen the distribution. A sender MAY adjust message sizing based on observed path balance, preferring larger messages on lightly loaded paths.

Differentiated Reliability Inference traffic is not uniformly sensitive to loss. The sensitivity of model state to perturbation varies with position in the model; loss affecting early-layer data propagates through all subsequent computation, while loss affecting late-layer data has more bounded effect on output quality. This creates an opportunity for differentiated reliability, for which the IETF has established precedents: partial reliability in SCTP allows a sender to abandon delivery of selected data, and the QUIC DATAGRAM extension provides unreliable delivery within a reliable connection. When the application indicates the loss sensitivity of the data it submits (for example, by model layer), the transport MAY apply full retransmission to loss-sensitive data and bounded or no retransmission to loss-tolerant data, particularly under high path load. The mapping from data category to reliability level is an application policy decision; it SHOULD be set so that service quality objectives (such as response accuracy and token latency) are preserved. How the application communicates this indication to the transport is a local interface matter outside the scope of this document.

Deployment Considerations The path steering requires only standard ECMP in the fabric and is therefore deployable incrementally: endpoints that implement it coexist with endpoints that do not. On-path telemetry is an optimization, not a dependency; endpoint estimation suffices where IOAM support is absent. Path steering and per-hop dynamic load balancing should not operate on the same traffic simultaneously without coordination, as independent repathing decisions at both the endpoint and the fabric can interact unpredictably. Differentiated reliability should be introduced conservatively, with loss-tolerant treatment applied only where its effect on inference quality has been validated for the model in use.

Security Considerations Path load information, whether collected via IOAM or estimated at endpoints, reveals aspects of fabric topology and utilization. The security considerations of and apply to telemetry collection and export; access to collected data SHOULD be restricted to authorized components. Entropy-based path steering uses header fields that are part of normal traffic; it does not introduce new spoofing surface beyond that of the underlying transport. However, an endpoint that repaths aggressively can concentrate load deliberately; fabrics serving multiple tenants SHOULD apply the usual per-tenant rate and resource isolation. Differentiated reliability relies on application indications of loss sensitivity. A compromised or misconfigured application could mark loss-sensitive data as tolerant, degrading its own service quality; this is contained within the application's own traffic.

IANA Considerations This document has no IANA actions.

Acknowledgements The authors acknowledge ongoing IETF discussion of AI workload networking, including problem statements on training-network load balancing and congestion, which provides context for the inference-specific considerations in this document.