]> Futurewei

mmcbride7@gmail.com

China Mobile

liuyisong@chinamobile.com

Huawei

monica.zhangli@huawei.com

BIER multicast AI P2MP inference data center RoCE RDMA AI workloads in data centers exhibit inherently point-to-multipoint (P2MP) communication patterns. During distributed training, collective operations such as AllReduce, AllGather and Broadcast require identical data delivery to many receivers. During inference serving, P2MP patterns also arise from mechanisms such as KV-cache distribution in disaggregated serving architectures and speculative-decoding verifier fan-out. Unicast replication of these flows does not scale to large GPU clusters. This document evaluates two architectural mechanisms for addressing this problem: extending BIER (Bit Index Explicit Replication) to support AI P2MP requirements, or defining a new purpose-built protocol. The evaluation is grounded in the transport-layer requirements this P2MP communication pattern places on a multicast solution, i.e., the requirements imposed by RDMA and RoCEv2 semantics rather than by the choice of network-layer replication mechanism, including considerations around ACK aggregation, congestion control, RoCE/RDMA compatibility and operational complexity. This document does not define a protocol but is intended instead to help the mcast4ai community evaluate this problem space.

Introduction AI workloads generate communication patterns that are structurally point-to-multipoint (P2MP), both during distributed training and during inference serving. During training collective operations, a single sender must deliver identical data to a large number of GPU receivers simultaneously. During inference serving, similar P2MP patterns arise from mechanisms such as KV-cache distribution in disaggregated serving architectures and speculative-decoding verifier fan-out. At the scale of modern GPU clusters, hundreds to thousands of endpoints, replicating these flows via unicast imposes significant and unnecessary burden on the network. BIER provides a stateless multicast forwarding architecture that has seen deployment in WAN and some DC environments. A natural question is whether BIER can be extended to satisfy the requirements of AI P2MP traffic, or whether the unique characteristics of AI workloads, including tight latency requirements, RDMA transport semantics, ACK aggregation and congestion control, necessitate a new protocol. This document explores both approaches. While IETF practice has historically favored extending existing protocols where feasible, there are cases where the requirements diverge sufficiently from the design assumptions of an existing protocol that a new protocol is more appropriate. This document aims to provide an objective basis for that determination. This document's scope is specifically the transport-layer requirements for AI P2MP communication, i.e., the constraints that RDMA and RoCEv2 semantics place on any solution, such as ACK aggregation, congestion control interaction and BTH-aware processing. These requirements apply regardless of which network-layer replication mechanism is ultimately selected to deliver P2MP traffic, whether a BIER extension, a new protocol, or some other approach.

AI P2MP Requirements The following requirements are specific to AI data center P2MP and distinguish this problem space from traditional multicast use cases. These are transport-layer requirements, complementary to network-layer requirements and gap-analysis work such as , which addresses multicast tree and forwarding-plane requirements (e.g., dynamic receiver-set churn) rather than the transport-layer semantics this document focuses on.

Collective Communication Patterns AI training collectives (AllReduce, AllGather, ReduceScatter, Broadcast) require simultaneous delivery of data to all members of a communication group. For dense model training, group membership is established at job initialization and remains stable throughout the training run. In Mixture of Experts (MoE) architectures, All-to-All collectives implement token-dependent expert routing where each token is dispatched to a small subset of experts (typically top-K of N total), meaning the active receiver set changes with every token or batch. The full set of expert endpoints is fixed for the duration of the job, but the per-packet routing destination is highly dynamic. Both cases include P2MP communication patterns. In MoE, while the All-to-All expert dispatch collective is many-to-many rather than P2MP, gradient synchronization across expert replicas uses AllReduce over fixed groups, benefiting from P2MP multicast. MoE's expert dispatch introduces the additional requirement that the forwarding mechanism support dynamic, per-token receiver subsets.

Inference-Serving P2MP Patterns P2MP communication patterns are not limited to training. Several mechanisms increasingly used in AI inference serving exhibit the same structural requirement, that a single sender deliver identical data to multiple receivers, and are therefore in scope for this evaluation alongside training collectives:

• KV-cache / prefix-cache distribution: In disaggregated inference serving architectures, a prefill node computes the key-value cache for a request and must deliver it to one or more decode replicas. When multiple decode replicas require the same cache (e.g., for load balancing or replicated serving of a popular prompt prefix), this is a P2MP delivery problem structurally similar to a training broadcast, though it occurs on the inference-serving data path rather than within a training job.
• Speculative-decoding verifier fan-out: In speculative decoding, a smaller draft model proposes candidate tokens that must be verified by one or more larger verifier model instances. Where verification is parallelized across multiple verifier replicas, the draft model's proposed tokens must be delivered identically to each, a P2MP pattern with tighter latency sensitivity than most training collectives, since it sits on the per-token inference critical path.
• Checkpoint and weight distribution: Delivering a trained or updated model checkpoint from a coordinator to many training workers or inference-serving replicas is P2MP in shape. This is sometimes implemented as a collective broadcast, but is also frequently implemented via out-of-band mechanisms (e.g., shared object storage, peer-to-peer distribution) that do not use RDMA/RoCEv2 transport. This document's transport-layer analysis, including the ACK aggregation discussion in , applies to checkpoint and weight distribution only where it is implemented over an RDMA/RoCEv2 transport; out-of-band distribution mechanisms are outside this document's scope.

Unlike training collectives, where group membership is typically static for the duration of a job, inference-serving P2MP patterns often involve receiver sets that change per-request or per-token, resembling the dynamic receiver behavior already noted for MoE expert dispatch above. Any mechanism evaluated in this document should account for this dynamic membership requirement, not only the more stable membership pattern of dense-model training collectives.

Latency and Jitter Sensitivity GPU training steps are synchronization points. Straggler effects, where a single slow receiver delays the entire collective, make tail latency a critical metric. The multicast mechanism should minimize added latency and jitter relative to optimal unicast paths.

RDMA and RoCE Compatibility AI workloads primarily rely on RDMA (Remote Direct Memory Access) today for high-throughput, low-latency data transfer. In Ethernet-based AI data centers, RoCEv2 (RDMA over Converged Ethernet) is the dominant transport. RoCEv2 uses the InfiniBand Base Transport Header (BTH) encapsulated in UDP. Any multicast solution should be compatible with this transport model, or should define a clear interworking point.

ACK Aggregation In unicast RDMA, each receiver generates an ACK per received packet or message. In P2MP multicast, N receivers generating individual ACKs back toward the sender creates an ACK implosion problem. This imposes load on both the network (N reverse unicast flows) and the sender's RDMA NIC (processing N individual ACKs). An ACK aggregation mechanism is required that collapses N ACKs into a single ACK before reaching the sender's RDMA NIC, while preserving RoCE ACK semantics including BTH opcode and PSN fields.

Congestion Control RoCEv2 environments typically rely on Priority Flow Control (PFC) and DCQCN for congestion management. A multicast solution should either integrate with these mechanisms or define equivalent behavior. Packet spraying, increasingly adopted for load balancing (e.g., Nvidia AR, Broadcom DLB/GLB), complicates per-flow congestion signals.

Scalability GPU clusters in large AI data centers may contain thousands of endpoints organized into multiple communication groups running simultaneously. The multicast solution should scale in terms of group state, replication overhead and control plane complexity.

Extending BIER

BIER Overview BIER encodes the set of intended receivers as a bitstring in the packet header. Each router in the BIER domain forwards and replicates packets based on the bitstring without maintaining per-flow or per-group state. This stateless property is a key operational advantage. describes optimizations to BIER targeting AI data center environments, including mechanisms for ACK aggregation within the BIER forwarding plane.

Arguments for Extending BIER

• BIER is an existing IETF standard with implementations in multiple vendor platforms, reducing time to deployment.
• BIER's stateless forwarding model aligns well with AI P2MP groups whose membership is known at the source, as is the case in dense model training, avoiding the need for network-wide group state. MoE workloads introduce additional complexity, as the subset of active expert endpoints varies per training step, even though the full set of experts is fixed for the duration of the job.
• IETF practice favors reuse of existing protocols where requirements can be met, reducing fragmentation of the standards landscape.
• Extending BIER leverages existing operational familiarity and tooling in networks where BIER is already deployed.
• The BIER header can carry arbitrary payloads, including RoCEv2/BTH, without modification to the forwarding plane.

Limitations of Extending BIER

• BIER was designed for IP multicast replacement and does not natively address ACK aggregation, which is a fundamental requirement for AI P2MP. Adding ACK aggregation to BIER requires significant new machinery that may be architecturally inconsistent with BIER's stateless forwarding model.
• BIER bitstring size limits the number of BFERs (endpoints) in a BIER sub-domain. Large GPU clusters may require hierarchical BIER domains, adding complexity.
• BIER does not define congestion control. Layering RoCEv2 congestion semantics over BIER requires additional specification work.
• BIER is a forwarding plane mechanism. The control plane (BGP-BIER, OSPF-BIER) was not designed with the dynamic, job-scoped group lifecycle typical of AI training workloads, where groups may be set up and torn down thousands of times per day in a busy cluster.
• ACK aggregation in a BIER context requires nodes to maintain per-group ACK state, which contradicts BIER's stateless design principle.

Defining a New Protocol

Arguments for a New Protocol

• A purpose-built protocol can be designed from the ground up to satisfy AI P2MP requirements including ACK aggregation, RoCE/BTH compatibility, and congestion control integration, without being constrained by the design assumptions of BIER.
• AI data center networks are a distinct environment from the WAN and campus networks where BIER was originally targeted. The scale, traffic patterns, and transport semantics differ significantly enough to justify a purpose-built solution.
• A new protocol can define a clean ACK aggregation mechanism that is first-class rather than bolted on, with well-defined semantics for BTH PSN handling and aggregation point placement.
• A new protocol can be co-designed with the RoCEv2 transport model, defining explicit interworking points (e.g., source ToR, smart NIC/DPU) for ACK aggregation without ambiguity.
• Avoiding overloading BIER with AI-specific extensions preserves the clarity and deployability of BIER in its existing use cases.

Limitations of a New Protocol

• A new protocol requires new implementations across the entire stack, switch ASICs, NICs, control plane software, increasing time to deployment and adoption risk.
• A new protocol requires a new IETF working group charter or significant expansion of an existing charter, adding process overhead.
• Operators may be reluctant to deploy a new protocol in production networks without a substantial track record.
• There is a risk of fragmentation if multiple vendors define proprietary solutions in the absence of a timely standard.

Comparative Analysis The following table summarizes the tradeoffs across key dimensions.

Dimension	Extend BIER	New Protocol
ACK aggregation	Requires significant extension; conflicts with stateless model	Can be designed as first-class feature
RoCE/BTH compatibility	Payload-agnostic; interworking not specified	Can be co-designed with RoCEv2
Congestion control	Not in scope for BIER; requires additional work	Can integrate DCQCN natively
Scalability	Bitstring size limits; hierarchical domains add complexity	Designed for large GPU cluster scale
Deployment speed	Faster; builds on existing implementations	Slower; requires new implementations
Operational familiarity	Higher where BIER is deployed	Lower; new operational model
Standards risk	Lower; incremental extension	Higher; new charter/scope required
Architectural cleanliness	Lower; AI requirements are a poor fit for BIER assumptions	Higher; purpose-built for use case

Extending BIER appears to offer a faster path to a first implementation, but this apparent advantage does not survive contact with the ACK aggregation requirement. ACK aggregation is not an optional feature; it is necessary for correctness at scale in any RDMA/RoCEv2-based P2MP solution. Any extension to BIER that adds the stateful, per-group ACK aggregation state this requires has already abandoned BIER's core stateless-forwarding design. What results is, in substance, a new protocol that merely reuses the BIER forwarding header, while inheriting none of the architectural benefit of a design built around the requirement from the outset, and all of the cost of carrying forwarding-plane assumptions BIER was never designed to support. The authors conclude that a new, purpose-built protocol is the architecturally sound choice for AI P2MP. The deployment-speed and standards-risk advantages of extending BIER are outweighed by the fundamental mismatch between BIER's stateless model and a requirement that is not optional. Reusing the BIER header without its stateless design does not produce a lower-risk protocol; it produces a new protocol built on the wrong foundation. The authors recommend that the mcast4ai community focus its design effort accordingly. Further input on implementation and deployment considerations, particularly from operators who have deployed or are evaluating BIER in AI DC environments, remains valuable to that design process, not to whether it should proceed.

Open Issues The following open issues represent challenges that any solution must address, regardless of whether BIER is extended or a new protocol is defined. They are not arguments for or against either approach, but rather constraints that any design must satisfy.

ACK Aggregation Point The aggregated ACK delivered to the sender's RDMA NIC must conform to RoCEv2 ACK semantics, including a valid BTH with opcode 0x11 (ACKNOWLEDGE) and a PSN representing the collective acknowledgment state of all receivers (typically the minimum PSN across all receivers to ensure no data is incorrectly freed). Candidate aggregation points include, but are not limited to:

• The source ToR switch, which is a natural convergence point for returning ACKs under packet spraying, as all receiver ACKs are destined for the source host and must traverse the source ToR. This requires the ToR to implement BTH-aware ACK aggregation, which is not supported by standard merchant silicon today.
• A software agent on the aggregation source host, which intercepts returning ACKs in software before they reach the RDMA stack. This reintroduces CPU involvement and may compromise the kernel-bypass properties of RDMA.
• A smart NIC or DPU (Data Processing Unit) on the aggregation source host, which performs BTH-aware ACK aggregation in NIC hardware without involving the host CPU. This preserves kernel-bypass semantics and is likely the most architecturally clean option, but requires DPU programmability.

Each option has different tradeoffs in terms of where intelligence is placed, whether host CPU is involved, and what hardware capabilities are required. Further analysis is needed.

BTH Classification in the Network Data center switches, without additional hardware capabilities, treat RoCE ACKs as normal UDP data payload and cannot distinguish them from RoCE data packets, since the BTH opcode is carried inside the UDP payload and is not visible to standard switch forwarding logic. Classification of ACK packets in the network requires either P4-capable ASICs with custom parsers or vendor-specific deep packet inspection features. This is a practical constraint on any in-network ACK aggregation mechanism.

Interaction with Packet Spraying Packet spraying is increasingly adopted in AI DC networks for load balancing (e.g., Nvidia Adaptive Routing, Broadcom DLB/GLB, China Mobile GSE). Under packet spraying, ACK paths cannot be assumed to be symmetric with data paths, which complicates in-network ACK aggregation at arbitrary points. The source ToR aggregation point is robust to packet spraying since all ACKs must converge there regardless of the spray path taken.

Security Considerations This document does not define a protocol and introduces no new security considerations beyond those already discussed in and related RDMA/RoCEv2 specifications. Security considerations for any protocol defined based on this analysis should address spoofing of aggregated ACKs, which could cause a sender to incorrectly advance its send window.

IANA Considerations This document has no IANA actions.

References Informative References

Acknowledgements The authors thank Kefei Liu for valuable discussion on ACK identification and aggregation point selection on the mcast4ai mailing list.