China Mobile

Beijing 100053 China yangjinwl@chinamobile.com

China Mobile

Beijing 100053 China chengweiqiang@chinamobile.com

China Mobile

Beijing 100053 China hantingting@chinamobile.com

Centec

Suzhou 215000 China wangjj@centec.com

Centec

Suzhou 215000 China zhanggy@centec.com

Internet 6MAN IPv6 Extension Header WPS Packet Spraying Multipath Load Balancing Wide Area Network In-Order Delivery RDMA AI Data Center Interconnect This document specifies the Wide-area Packet Spraying (WPS) option, an IPv6 option that can be carried in the Destination Options header or, where required, the Hop-by-Hop Options header. The option conveys the scheduling metadata that is needed to distribute the packets of a single high-volume flow across multiple parallel wide-area paths at the granularity of packet groups, and to restore the original packet ordering at the egress boundary of a controlled domain. The mechanism provides order-preserving multipath load balancing and bandwidth aggregation for wide-area interconnection of distributed computing sites, such as wide-area interconnects between Artificial Intelligence (AI) data centers, and is intended for use within limited domains.

Introduction Per-flow load distribution over parallel wide-area links cannot exceed the capacity of one member link for any single flow, while per-packet spraying resolves the utilization problem but, over wide-area paths of unequal latency, introduces substantial packet reordering that severely degrades loss- and order-sensitive transports. This document therefore specifies Wide-area Packet Spraying (WPS), a mechanism that sprays traffic at an intermediate granularity: the packets of a flow are partitioned at the domain ingress into consecutive packet groups; all packets of one group are forwarded over the same path, while different groups are sprayed over different parallel paths. Intra-group ordering is preserved by path pinning, and inter-group ordering is restored at the domain egress using metadata carried in an IPv6 option defined in this document. The unit of spraying is thus the packet group rather than the individual packet, which retains the load-balancing benefit of spraying while bounding reordering. WPS operates between an ingress node and an egress node at the boundaries of a single administrative domain. It does not modify end-to-end transport or congestion-control behavior, and it is transparent to the endpoints of the flows.

Problem Statement Training of large models increasingly spans several AI data centers, because a single site is constrained by power, cooling, floor space, and cluster size. Wide-area interconnection between such computing sites has therefore become a core part of the infrastructure for distributed training, and the loss of training efficiency attributable to the wide-area segment is commonly required to stay within a small budget (on the order of ten percent). The traffic exchanged between such sites differs fundamentally from general Internet traffic and from intra-data-center traffic. It consists of a small number of concurrent flows, each of very high bandwidth and long duration, typically carried over RDMA-based transports, and it is highly sensitive to congestion, packet reordering, even small amounts of loss, and delay variation: loss triggers RDMA retransmission and rate reduction, which directly reduces the effective compute of a distributed training job. Wide-area backbones commonly use per-flow hash-based Equal-Cost Multipath (ECMP) techniques that hash on invariant packet fields or on the IPv6 Flow Label . For the traffic pattern above, this approach has the following deficiencies:

• Hash collisions and load polarization: with few concurrent flows, the hash values of several large flows can collide, concentrating those flows on a subset of the parallel links. The affected links become congested while others remain lightly loaded, and the aggregate utilization of the backbone stays low.
• Congestion loss directly reduces compute efficiency: bursts of large RDMA flows combined with uneven load cause congestion and loss, which trigger retransmission and rate back-off and prevent the training-efficiency target from being met.
• Single-port bandwidth ceiling: the bandwidth of one flow can already exceed the forwarding capacity of a single router port, and link speeds are not growing as fast as the bandwidth demand of AI workloads. Because per-flow hashing uses the flow as its smallest placement unit, it is structurally unable to distribute one flow over several links; aggregation of multiple links into a logical pipe of larger capacity is required.

Design Rationale Several approaches can aggregate the bandwidth of multiple wide-area links for a single flow. This section records the trade-offs, so that the choice made in this document is documented rather than assumed. Per-packet spraying maximizes link utilization but exposes the receiver to per-packet reordering. Over a wide area, the delay difference between parallel paths is far larger than within a data center, so the buffering and sorting effort at the receiver grows with the bandwidth-delay product, and the reordering directly disturbs transports that expect in-order delivery. Flowlet-based switching relies on inter-packet gaps larger than the path delay difference to switch paths without reordering. Wide-area delay differences are on the order of milliseconds, and a large flow under sustained throughput exhibits almost no such gaps, so flowlet switching degenerates to per-flow hashing in this environment. Entropy mechanisms such as the MPLS entropy label and the IPv6 Flow Label improve the hash distribution among multiple flows but keep the flow as the placement unit; they do not address a single flow whose rate exceeds the capacity of one link. Distributing encapsulated or aggregated traffic over multiple paths by adding per-packet metadata is nevertheless established practice in these mechanisms; WPS extends the idea from per-flow placement to packet-group sequencing for wide-area multipath networks. WPS distributes load at packet-group granularity, between the two extremes. Packets of a group follow one path and stay in order; reordering is confined to group boundaries and is repaired by the egress node with a bounded buffer. The group granularity is a tuning parameter that trades link balance against reordering effort (see ). The sequencing metadata is carried in an IPv6 option of an encapsulating header rather than in a new IP protocol header or a UDP encapsulation. Options are the native IPv6 extension mechanism , and the choice between the Destination Options header and the Hop-by-Hop Options header matches the two deployment modes described in . Encapsulation is required in any case, because an extension header cannot be inserted into a packet in flight (Section 4 of ); it also gives the egress node an unambiguous decapsulation point.

Relationship to Existing Mechanisms Several existing and proposed mechanisms address load distribution or reordering for multipath traffic. WPS differs from each of them in the unit of placement or in the layer at which it operates.

• The Control Word option carries a flow identifier and a per-packet sequence number in the Destination Options header to support per-packet reordering and duplicate elimination, in the manner of the pseudowire and DetNet control words. WPS does not sequence individual packets: it sequences packet groups, relies on path pinning to keep each group internally in order, and thereby bounds the state and buffering that the egress node needs at high rates. The two options serve different operating points and can coexist in one domain.
• The MPT multipath library provides a GRE-in-UDP tunnel over multiple paths and reorders packets at the receiver using per-packet GRE sequence numbers. Like the Control Word option, it sequences individual packets, so the receiver sorts every packet; WPS sequences groups and confines sorting to group boundaries. WPS also stays within the IPv6 header chain instead of adding GRE and UDP encapsulations.
• A Geneve option for per-packet spraying carries a per-packet sequence number in an overlay encapsulation. WPS operates at the IPv6 layer without an overlay protocol and at group rather than packet granularity.
• Transport-layer multipath spraying, such as Multipath Reliable Connection over SRv6 source routing , places path selection and reordering in the host or NIC transport stack. WPS is a network-layer mechanism applied by border gateways; it requires no change to the end hosts or NICs of the computing clusters and is therefore applicable where the operator of the wide-area segment does not control the host transport.
• Load-aware ECMP, such as , improves the placement of flows or flowlets onto paths but keeps the flow as the placement unit. WPS divides one flow into groups in order to aggregate the bandwidth of several links for that flow. The two mechanisms operate at different points and can be combined.

Applicability Statement WPS is designed for limited domains : administered networks such as the wide-area interconnect between AI data centers, in which the ingress and egress border nodes are under one operational authority and the parallel paths between them are known. WPS is not intended for use across the open Internet. The option-type encoding () and the boundary filtering requirements () together keep WPS packets confined to the domain.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

WPS:

Wide-area Packet Spraying, the mechanism specified in this document. In this document, the abbreviation WPS always refers to this mechanism.

WPS Domain:

A limited domain under a single administration within which the WPS option is inserted, transported, and removed.

Ingress Node:

The node at the entry boundary of the WPS domain that classifies eligible flows, partitions them into packet groups, inserts the WPS option, and selects a member path for each packet group. Typically the wide-area gateway of the local computing site.

Egress Node:

The node at the exit boundary of the WPS domain that processes the WPS option, restores inter-group ordering, and removes all WPS-related encapsulation before forwarding packets onward. Typically the border device of the remote computing site.

Packet Group:

A set of consecutive packets of one flow that the Ingress Node assigns to the same Group ID and forwards over the same path through the WPS domain. The packet group is the smallest scheduling unit of WPS.

Member Path:

One of the parallel paths between an Ingress Node and an Egress Node over which packet groups are distributed.

Overview of Operation The reference architecture is shown in . An Ingress Node and an Egress Node are connected by multiple parallel wide-area member paths.

WPS Reference Architecture | | | | ==== Path 1 ==== | +--------+ +-+------+ ==== Path 2 ==== +--------+-+ +--------+ | Site |--| Ingress| ... | Egress |--| Site | | A | | Node | ==== Path N ==== | Node | | B | +--------+ +--------+ +----------+ +--------+ group-based group-order spraying restoration ]]>

Operation proceeds in four steps:

1. The Ingress Node identifies flows that are eligible for WPS treatment. The classification method is a local matter and is outside the scope of this document; possible methods include classification on the five-tuple, deeper packet inspection, and a flow table built from packet features together with burst and average rate measurements. Traffic to which WPS is not applied is forwarded according to existing behavior and does not carry the WPS option.
2. For each eligible flow, the Ingress Node assigns a Flow ID and partitions the packet sequence into packet groups according to a configured quantum (a number of packets, a number of bytes, or a time interval). Each group is assigned a Group ID that increases by one for each successive group.
3. Each packet is encapsulated for transport across the domain, carrying the WPS option defined in . All packets of one group MUST be forwarded over the same member path; successive groups are distributed over the member paths according to a local scheduling discipline (e.g., round-robin or least-loaded).
4. The Egress Node uses the Flow ID and Group ID to deliver packet groups in their original order, buffering groups that arrive ahead of their predecessors, and then removes the encapsulation and the WPS option before forwarding the packets out of the domain.

Because packets of the same group traverse the same path, their relative order is preserved by normal forwarding. The reordering problem is thereby reduced from per-packet to per-group, which bounds the egress buffering requirement.

Wide-Area Packet Spraying Option

Option Format The WPS option has the following format:

WPS Option Format

Option Type:

8-bit identifier of the type of option, TBD1 (suggested value 0x81, to be assigned by IANA; see ). The two highest-order bits are set to "10", so that a node that does not recognize the option discards the packet and sends an ICMPv6 Parameter Problem, Code 2, message to the packet's Source Address, as specified in . The third-highest-order bit is set to "0", indicating that the Option Data does not change en route.

Opt Data Len:

8-bit unsigned integer. Length of the Option Data field of this option, in octets. MUST be set to 12. Packets in which this field has any other value MUST be discarded.

S (Start, 1 bit):

Set to 1 on the first packet of a packet group; set to 0 otherwise.

E (End, 1 bit):

Set to 1 on the last packet of a packet group; set to 0 otherwise. A packet group consisting of a single packet carries both S and E set to 1.

R (Reorder, 1 bit):

Set to 1 if the Egress Node is required to restore inter-group ordering for this flow (see ); set to 0 if the Egress Node MAY forward packets without inter-group order restoration.

Rsvd (5 bits) and Reserved (8 bits):

Reserved for future use. MUST be set to 0 on transmission and MUST be ignored on receipt.

Flow ID (16 bits):

Identifier of the flow, assigned by the Ingress Node. The tuple (Ingress Node ID, Flow ID) uniquely identifies a flow within the WPS domain for the lifetime of the flow state.

Ingress Node ID (16 bits):

Domain-unique identifier of the Ingress Node that inserted this option.

Egress Node ID (16 bits):

Domain-unique identifier of the intended Egress Node. The Egress Node uses this field to validate that it is the intended processing node.

Group ID (16 bits):

Identifier of the packet group within the flow. The Ingress Node increments the Group ID by one for each successive group of the same flow. The field wraps around modulo 2^16; comparisons between Group ID values MUST use serial number arithmetic with SERIAL_BITS = 16.

Ingress Port ID (8 bits):

Identifier, local to the Ingress Node, of the port or logical interface on which the flow entered the domain. Together with the Ingress Node ID, it MAY be used by the Egress Node for diagnostics and state indexing.

Egress Port ID (8 bits):

Identifier, local to the Egress Node, of the port or logical interface on which the reassembled flow is to be forwarded out of the domain.

The WPS option has an alignment requirement of 2n, which places all 16-bit fields of the Option Data on 16-bit boundaries. When the WPS option is the only option in an extension header, the resulting header is exactly 16 octets long, a multiple of 8 octets, and requires no padding options. The assignment of Node ID and Port ID values within a WPS domain is an administrative matter for the domain operator and is outside the scope of this document.

Carrying Extension Header Because an extension header cannot be added to a packet in flight (Section 4 of ), the Ingress Node MUST place the WPS option in an encapsulating IPv6 header ; it MUST NOT insert the option, or an options header carrying it, into a received packet. The WPS option is designed to be examined only by the Egress Node. Therefore, the WPS option SHOULD be carried in a Destination Options header of the outer IPv6 header whose Destination Address identifies the Egress Node; transit nodes then forward the packet without examining the option. Deployments in which transit nodes of the WPS domain need visibility of the scheduling metadata (for example, for path-aware queuing or telemetry) MAY carry the WPS option in a Hop-by-Hop Options header instead. Such deployments need to take into account the operational limitations of Hop-by-Hop option processing described in and : a source cannot assume that every node on the path processes the option, and such deployments MUST ensure that every node on the member paths either processes the option or forwards it unmodified without punting the packet to a slow path. A packet MUST NOT carry the WPS option in both the Hop-by-Hop Options header and the Destination Options header, and MUST NOT carry more than one WPS option.

Ingress Node Procedures For each packet of an eligible flow, the Ingress Node performs the following steps:

1. Determine the flow context and its Flow ID. If no context exists, allocate a Flow ID that is not currently in use for this Ingress Node, and initialize the Group ID to an arbitrary value. A Flow ID MUST NOT be reused while packets of its previous flow can still be in flight within the domain.
2. Assign the packet to the current packet group. If the configured group quantum has been reached, close the current group (setting the E flag on its final packet), increment the Group ID using serial number arithmetic, and open a new group (setting the S flag on this packet).
3. Encapsulate the packet toward the Egress Node and insert the WPS option as specified in , populating all fields. When the Egress Node is required to restore inter-group ordering, the R flag MUST be set to 1.
4. Transmit the packet on the member path associated with the current group. All packets carrying the same (Flow ID, Group ID) MUST be transmitted on the same member path, and the Ingress Node MUST NOT reorder packets within a group.

The mapping of packet groups to member paths is a local scheduling decision. Uniform distribution (e.g., round-robin over equal-capacity paths) is RECOMMENDED as the default; weighted or load-aware scheduling based on measured path load MAY be used. Where the member paths are realized by ECMP inside the domain rather than by explicit tunnels, the Ingress Node MAY derive the outer IPv6 Flow Label from the Group ID so that on-path ECMP keeps all packets of a group on a single path . Smaller group quanta yield finer load balancing but increase the reordering work at the egress; larger quanta approach per-flow behavior. The group quantum SHOULD be configurable, and the operator can set it according to the delay difference between the member paths and the buffering capacity of the Egress Node. An Ingress Node MUST bound the number of groups of one flow that are simultaneously in flight so that the egress reordering buffer requirement () is bounded and Group ID wraparound ambiguity cannot occur.

Transit Node Procedures Transit nodes of the WPS domain forward WPS packets according to normal IPv6 forwarding procedures. When the WPS option is carried in a Destination Options header, transit nodes do not examine it. When the option is carried in the Hop-by-Hop Options header and a transit node is configured to process that header and recognizes the option, the node MAY use the option content for monitoring or for scheduling within the WPS domain, but it MUST NOT modify the option content. A node that examines the option and does not recognize it discards the packet and sends an ICMPv6 Parameter Problem message, as required by the option-type encoding (). This contributes to keeping packets that carry the WPS option inside the WPS domain; it does not replace the boundary filtering of , because nodes that skip option processing forward such packets unchanged. A transit node MUST NOT redistribute packets of the same outer flow across multiple paths on a per-packet basis; per-flow or flow-label-based placement, which keeps a group on a single path, is REQUIRED behavior for member-path internals.

Egress Node Procedures Upon receiving a packet carrying the WPS option, the Egress Node performs the following steps:

1. Validate the option: the Opt Data Len MUST be 12, and the Egress Node ID MUST match a local identifier. Packets failing validation MUST be discarded, and the event SHOULD be logged subject to rate limiting.
2. Locate or create reordering state for the tuple (Ingress Node ID, Flow ID). The state comprises the next expected Group ID (NEG), a reordering buffer, and a release timer.
3. If the packet's Group ID equals NEG, decapsulate and forward the packet immediately (cut-through), preserving arrival order within the group. When the packet carrying the E flag for group NEG has been forwarded, advance NEG by one and release any buffered successor groups that are now in order.
4. If the packet's Group ID is greater than NEG (serial number arithmetic), buffer the packet and start or restart the release timer for the flow.
5. If the packet's Group ID is less than NEG, the packet belongs to a group already released (e.g., a late arrival after timer expiry); the Egress Node SHOULD forward it immediately rather than discard it.

Packets within one group arrive in order because they followed one path; no per-packet sorting is needed. If the release timer expires while groups are buffered, the Egress Node MUST advance NEG to the smallest buffered Group ID and release buffered groups in Group ID order. This bounds the latency penalty when a group boundary marker is lost or a member path stalls, so that a lost group does not stall the flow indefinitely. The timer value SHOULD be configurable and SHOULD default to a small multiple of the measured maximum latency skew between member paths. If the reordering buffer for a flow is exhausted, the Egress Node MUST NOT discard buffered packets; it MUST instead release the buffered groups in Group ID order and advance NEG accordingly. Flow state SHOULD be reclaimed after a configurable idle period. If the R flag is 0, the Egress Node MAY skip steps 3-5 and forward packets on arrival. Before forwarding a packet out of the WPS domain, the Egress Node MUST remove the outer encapsulation, including the extension header carrying the WPS option, restoring the packet as received at the Ingress Node.

Path MTU Considerations WPS adds 56 octets to each packet: 40 octets of encapsulating IPv6 header and a 16-octet options header carrying the WPS option (). An implementation that uses WPS MUST account for these 56 octets when it performs Path MTU Discovery or when it determines the effective MTU toward the Egress Node, following the tunnel MTU and fragmentation procedures of . When PMTUD is not available, the implementation MUST use a conservative MTU estimate that accounts for the 56-octet overhead. Operators of a WPS domain SHOULD provision the MTU of all member paths to accommodate this overhead for the largest expected ingress packet. When the traffic is RDMA, the service-side MTU or MSS configuration SHOULD be adjusted so that no fragmentation occurs on the wide-area segment.

Deployment Considerations WPS requires an upgrade only of the border devices of the two sites. When the Destination Options header is used, transit nodes need no knowledge of WPS at all; they forward ordinary IPv6 packets between the two border nodes. WPS is most effective when the member paths have comparable capacity and bounded latency skew. Persistent capacity or latency asymmetry SHOULD be handled by weighting the group scheduler rather than by enlarging the egress reordering buffer. Authorization and scheduling signaling of the WPS domain, if present, MAY be mapped to a high-priority forwarding class so that it is not blocked by ordinary traffic and remains timely and reliable.

Manageability Considerations Implementations SHOULD expose counters for groups transmitted and received per flow, out-of-order group arrivals, release-timer expiries, buffer occupancy high-water marks, and validation failures. These counters allow the operator to tune the group quantum, the release timer, and the scheduler weights.

Security Considerations The general security considerations for IPv6 extension headers discussed in apply to the WPS option.

Domain Isolation WPS is confined to a limited domain (). An operator MUST filter packets that carry the WPS option at the WPS domain boundary, in both directions: nodes at the boundary MUST be configured to prevent packets carrying the WPS option from leaking out of the domain and to discard packets arriving from outside the domain that carry the option. The option-type encoding, with the highest-order two bits set to "10", provides an additional line of defense: a node outside the domain that examines the option and does not recognize it discards the packet and returns an ICMPv6 Parameter Problem message. Because many nodes forward packets without examining options, this encoding alone does not guarantee confinement, which is why the boundary filtering requirement above is stated as a MUST. Packets leaking to non-supporting nodes trigger ICMPv6 Parameter Problem messages. Nodes originating such messages are subject to the ICMPv6 rate-limiting requirements of , which limits the utility of the option as an ICMPv6 reflection vector.

Information Exposure The WPS option carries only scheduling metadata; it does not contain user payload, and it does not expose user data beyond what the encapsulated packet already exposes. The metadata (Flow ID and node and port identifiers) reveals limited information about the topology and the flow structure of the WPS domain to an on-path observer. An operator whose threat model includes on-path observation of the wide-area segment MAY protect that segment with encryption such as IPsec; such usage is outside the scope of this document.

Metadata Tampering and Injection An attacker able to inject packets with forged WPS options into the domain, or an on-path attacker that modifies the Group ID or Flow ID of packets, could cause an Egress Node to buffer packets needlessly, mis-order or advance flow state incorrectly, or exhaust reordering resources. Boundary filtering as required in is the primary mitigation, and the Egress Node's validation of the Egress Node ID rejects packets that do not belong to it. Where the threat model includes on-path adversaries inside the domain, the encapsulation between Ingress and Egress Nodes can be protected with IPsec.

Resource Exhaustion Egress Nodes MUST bound per-flow and aggregate reordering buffer usage, so that malicious or malfunctioning traffic consisting of incomplete packet groups cannot exhaust the buffer, and SHOULD police the rate of flow-state creation attributable to any single Ingress Node ID. The release-timer rule of bounds the time for which a missing group can hold buffered packets.

IANA Considerations IANA is requested to assign a new IPv6 option type from the "Destination Options and Hop-by-Hop Options" registry within the "Internet Protocol Version 6 (IPv6) Parameters" registry group, as follows: New IPv6 Option Type

Hex Value	Binary Value (act/chg/rest)	Description	Reference
TBD1 (suggested 0x81)	10 / 0 / 00001	Wide-area Packet Spraying (WPS)	This document

act = 10 requires a node that examines the option and does not recognize it to discard the packet and send an ICMPv6 Parameter Problem, Code 2, message to the packet's Source Address; chg = 0 indicates that the Option Data does not change en route .

References Normative References Informative References

Acknowledgements The authors thank the members of the 6MAN working group for discussions on the safe use of IPv6 extension headers that informed the design choices in this document.