AI P2MP Multicast in Scale-UP Network

Introduction The mainstream large language models, such as Llama4, Mixtral, and DeepSeekV3, have adopted MoE (Mixture-of-Experts) as the key technique. MoE has two procedures, dispatch and combine, to serve AI training and inference. The growing AlltoAll collective communication, performed during the dispatch procedure, has become the dominant in MoE, where tokens are required to be delivered to multiple selected expert nodes. This type of AI workload exhibits inherently point-to-multipoint (P2MP) communication patterns along with new requirements. Some valuable work has been done to address the requirements and gap analysis for supporting AI multicast communication. Referring to , it identifies the key requirements of multicast in AIDCs, including the particular ones for MoE. gives a gap analysis of the BIER protocol and proposes to introduce a new protocol. However, it has become a trend that MoE is primarily applied in scale-up scenarios where GPUs are connected in a more limited scale than scale-out scenarios. AI multicast in scale-up scenarios presents particular features, and the target optimization approaches should receive more attention. This document's scope is to make an analysis of the particular AI multicast features of MoE in scale-up scenarios, and give the recommended target multicast protocol evolution approaches.

Requirements of multicast communication supporting MoE Several requirements of multicast communication supporting MoE have been addressed in , including:

Bidirectional Interactivity

Networks in AIDCs are required to support closed-loop control for MoE, including efficient P2MP multicast forwarding and P2MP forwarding for feedback acknowledgments (ACKs). Moreover, it is required to support MP2P packet aggregation in large AIDCs with numerous receivers to reduce the excessive reverse traffic that can lead to network congestion.

High Dynamics

MoE-based AI training and inference use token dispatch, where gating networks select expert nodes per token at microsecond timescales, dynamically determining real-time multicast receiver sets with no fixed groups. This ultra-fast selection leaves no time for traditional multicast to establish, update, or tear down trees, leading to delays, packet loss, or AI task failure . Therefore, multicast in AIDC should meet high dynamics requirements, including fast change of multicast members and low overhead for dynamic change in both the control plane and data plane.

Sparseness

Multicast in AIDCs frequently involves multicast groups where only a small fraction of the total nodes in the cluster are multicast members, a characteristic closely tied to the sparse activation mechanism of modern AI models such as MoE. For example, DeepSeekV3 uses 256 experts and activates 9 experts at a time. The multicast should be efficient when the group size is small relative to the network size, and meet the sparseness requirements, including efficient sparse member identification to avoid unnecessary scanning or signaling of non-member nodes, and low overhead for sparse state maintenance to reduce state maintenance burden and ensure no additional latency during state updates. Besides these requirements, there is one more: memory address mapping. For LLMs, RDMA write operations require specifying the memory addresses of remote GPUs for data writes. In MoE dispatch, the randomly selected experts reside on different destination nodes, each with a distinct memory address. Since the dispatch process inherently involves sending data to a multicast group, the address mapping requirement refers to an efficient method for conveying the address information of these multicast group nodes to the source node.

Differences between MoE in scale-out and scale-up scenarios MoE in scale-out and scale-up scenarios exhibits different features.

Scale

With the growth of LLMs in parameter size, the scale-out network is predicted to reach a cluster of 128K nodes. Although the scale‑up network keeps enlarging, its scale is far smaller than that of the scale‑out network, no more than 1K nodes.

RDMA communication modes

In scale‑out scenarios, message semantics (Send/Receive) are primarily used in RDMA communication, whereas memory semantics (RDMA Read/Write) is the primary mode in scale‑up scenarios.

Challenges for AI multicast communication of MoE in scale-out and scale-up scenarios Since there are critical differences between MoE in scale-out and scale-up scenarios given in Section 3, the challenges of AI multicast protocol evolution also differ.

Challenges for ACK problems

In scale-out scenarios, ACK problems should be comprehensively considered to solve, because ACK is the key process within the message semantics mode of RDMA communication. Referring to , solving ACK aggregation problems requires significant new machinery that may increase the complexity. However, in scale-up scenarios, these problems no longer exist under the memory semantics mode.

Challenges for high dynamics

Since high dynamics exist in both scale-out and scale-up scenarios, the challenges to introduce BIER-like protocols are the same.

Challenges for sparseness

Since scale‑out scenarios may involve thousands of endpoints, far larger than the size of a single sub‑domain, hierarchical domains become a critical implementation requirement in AI multicast protocol evolution. In contrast, scale‑up scenarios are typically limited to no more than 1K nodes, where a single sub‑domain suffices. As a result, the complexity of AI multicast protocol evolution in scale‑up scenarios is much lower than that in scale‑out scenarios, and can therefore yield greater performance gains.

Challenges for memory address mapping

Memory address mapping problems exist in both scale-out and scale-up scenarios, so that the approach is the same. As the scale-up scenarios are increasingly adopted by the mainstream LLMs, meeting the AI multicast requirements in this scenario becomes more urgent and valuable.

Target multicast protocol evolutions in scale-up network BIER's stateless forwarding model aligns well with AI P2MP groups whose membership is known at the source, as is the case in MoE; the subset of active expert endpoints varies per training step, even though the full set of experts is fixed for the duration of the job. However, evolutions of BIER to support MoE are still necessary.

Since BIER BitStrings deliver poor efficiency for MoE multicast groups containing only a small number of selected experts, a promising evolution is to adopt XPU-ID Lists as a replacement for BitStrings. Instead of using a fixed-length BitString to encode the entire sub-domain, this approach employs a compact XPU-ID list to explicitly identify the intended recipients. This replaces the fixed overhead of BitString with a variable-length, compact list that scales with the actual number of participants. It is particularly well-suited for sparse scenarios, as it only encodes the receivers that are actually involved in each dispatch, rather than the entire domain.
BIER doesn't natively address ACK aggregation, which is a fundamental requirement for AI P2MP. Adding ACK aggregation to BIER requires significant new machinery that may be architecturally inconsistent with BIER's stateless forwarding model. In scale-up scenarios, a potential evolution is ACK-Free Reliable Transmission. Leveraging the inherent properties of memory-semantic communication, a lightweight acknowledgment mechanism that avoids the complexity of MP2P multicast ACK aggregation could be designed. Alternatively, batch acknowledgment or cumulative ACK schemes can be adopted as simpler substitutes, further reducing the overhead associated with per-receiver confirmations.

Beyond these target evolutions of AI multicast protocols, the memory address mapping problems remain unresolved. Addressing the question of how to efficiently convey multiple destination addresses persists as an inescapable requirement that cannot be circumvented.

Open Issues While the dispatch phase can be independently optimized—for instance, through compact XPU-ID lists or lightweight acknowledgment mechanisms—such isolated improvements offer only marginal benefits. A balanced, co-optimized approach that synchronously addresses both dispatch and combine is essential to achieve meaningful end-to-end performance improvements.

Security Considerations This document does not introduce any new security considerations.

IANA Considerations This document has no IANA actions.

Informative References