Fully Adaptive Routing Ethernet in Multi-Plane Scale-Out Networks

Large AI training clusters (approaching or even exceeding 100,000 GPUs) are increasingly using multi&nbhy;plane scale&nbhy;out network topologies (see Figure 1) to reduce the total number of switches and links. In such a network, each RNIC is partitioned into multiple interfaces at either port or sub&nbhy;port granularity (Note that a port can be further split into multiple sub&nbhy;ports using breakout cables or shuffles), with each interface connected to an independent CLOS fabric (referred to as a "plane"). Because there are no links between planes, the RNIC itself must decide which plane to use for each packet or flow. In other words, the RNIC needs to determine the reachability and available bandwidth of each plane, and then perform global load-balancing across them.

(For simplicity, the diagram above omits the connections between RNICs and leaf switches, as well as the connections between leaf switches and spine switches within the same plane. In practice, each RNIC is multi&nbhy;homed to one leaf switch in every plane. Additionally, each leaf switch is connected to all spine switches of its own plane.) FARE&nbhy;in&nbhy;SUN extends the FARE&nbhy;BGP protocol from switches to GPUs for scale&nbhy;up networks, which are typically multi&nbhy;plane. Since multi&nbhy;plane scale&nbhy;out networks share the same architectural pattern, the adaptive routing approach defined in FARE&nbhy;in&nbhy;SUN is directly applicable to them. The solution described in this document is almost identical to that in FARE&nbhy;in&nbhy;SUN, with two essential differences. First, FARE&nbhy;BGP is extended from switches to RNICs rather than to GPUs. Second, In a scale&nbhy;up network, the number of route entries is small (typically a few hundred) and can be installed directly on GPUs. In contrast, consider an isolated multi&nbhy;plane scale&nbhy;out network with 100,000 GPUs (assuming a 1:1 GPU&nbhy;to&nbhy;RNIC ratio) and four planes. If the loopback addresses of RNICs are used for QP establishment, each plane MUST propagate up to 100,000 host routes for RNICs to avoid the blackholing issue associated with route aggregation. Even when interface addresses (with different prefixes configured for interfaces attached to different planes) are used instead of loopback addresses, it may still be desirable to propagate those host routes to speed up failover. However, storing all these routes on an RNIC is impractical, and maintaining such a large number of host routes on switches is also suboptimal. Therefore, routing tables on RNICs MUST be suppressed, and routing tables on switches SHOULD be suppressed as well.

This memo makes use of the terms defined in .

In an isolated multi&nbhy;plane scale&nbhy;out network, an RNIC is connected to each plane and configured as a stub BGP speaker per plane. It MUST establish separate BGP sessions with the attached leaf switches of each plane. The BGP neighbor discovery mechanism MAY be used to simplify configuration. Through these sessions, the RNIC learns routes to remote RNICs together with the path&nbhy;bandwidth extended community and then performs WECMP load-balancing as defined in . In this manner, the RNIC provides almost the same Weighted Equal&nbhy;Cost Multi&nbhy;Path (WECMP) load-balancing functionality as a FARE&nbhy;capable GPU as defined in , distributing traffic in proportion to the weight of each ECMP route.

Per&nbhy;flow weighted load balancing is recommended when ordered packet delivery is essential. For per&nbhy;flow weighted load balancing, at least one Queue Pair (QP) per plane MUST be established between a pair of RNICs. Furthermore, the following requirements SHOULD be met: If QPs are established using the loopback address assigned to each RNIC, each QP SHOULD be assigned a unique UDP source port to differentiate traffic flows across all available planes between the RNIC pair. If QPs are established using the physical addresses assigned directly to interfaces, there is no need to assign a unique UDP source port for each QP, because the interface address inherently distinguishes traffic flows across all available planes between the RNIC pair. Switches within each plane SHOULD also perform per&nbhy;flow weighted load balancing to ensure ordered packet delivery for all QPs.

Per-packet weighted load balancing is recommended when disordered packet delivery is acceptable (e.g., through the Direct Data Placement mechanism ). For per&nbhy;packet weighted load balancing, a single QP per RNIC pair is sufficient. Therefore, it is RECOMMENDED to use the loopback address assigned to each RNIC for QP establishment. The traffic of that QP is distributed across all available planes according to the weight of each plane. Switches within each network plane are RECOMMENDED to perform per&nbhy;packet weighted load balancing, as disordered packet delivery is acceptable for all QPs.

In an isolated multi&nbhy;plane scale&nbhy;out network with 100,000 GPUs and four planes, each plane may propagate up to 100,000 host routes – a total of 400,000 routes. Storing all these routes on an RNIC is impractical. Moreover, maintaining roughly 100,000 host routes on the switches of each plane is also suboptimal. Consequently, the following two complementary approaches can be employed to reduce the number of routes that both the RNIC and the switches need to store.

A straightforward approach is to aggregate host routes for RNICs, especially when advertising them from leaf switches to RNICs. However, naive aggregation can create route blackholes: if a remote RNIC becomes unreachable via a given plane, the aggregated route to that RNIC over that plane remains on the local RNIC. Consequently, traffic destined for that remote RNIC will be forwarded by the local RNIC to that plane and then dropped within the plane. To address this issue, when an RNIC becomes disconnected from a given plane, the switch in that plane that performs route aggregation for the RNIC's host route (e.g., the leaf switch to which the RNIC was previously connected) MUST explicitly advertise the unreachability of that RNIC within the plane, while keeping the aggregated route intact. Specifically, the switch SHOULD advertise this unreachability using one of the following two methods: Path Bandwidth value of 0: The leaf switch advertises the host route (NLRI) with the Path Bandwidth extended community set to 0. The RNIC interprets this as "unreachable". Specific BGP unreachability advertisement: The leaf switch sends a dedicated BGP unreachability advertisement message as defined in or . This message is distinct from a standard BGP route withdrawal and explicitly marks the host as unreachable via that plane. When the corresponding specific prefix becomes reachable again, the unreachability advertisement MUST be withdrawn immediately. Upon receiving such an unreachability advertisement, the RNIC updates its forwarding table as follows: It locates the longest&nbhy;matching aggregated route that covers the unreachable host (e.g., a default route or a subnet prefix route). From that aggregated route's set of next&nbhy;hops (which originally includes multiple planes), it removes the next&nbhy;hop associated with the plane over which the unreachable advertisement was received. It then installs a host&nbhy;specific route for the unreachable destination, using the remaining next&nbhy;hops from the aggregated route. For example, suppose an RNIC has an aggregated route (a.b.c.0/24) with next&nbhy;hops pointing to planes A, B, C, and D. Host X (a.b.c.d/32) becomes unreachable via plane A. The RNIC receives an unreachable advertisement for X and then installs a host&nbhy;specific route for X with next&nbhy;hops set to {B, C, D} — i.e., the next&nbhy;hop set of the longest&nbhy;matching aggregate route minus the next&nbhy;hop associated with plane A. As a result, traffic destined for X is never sent to plane A, thereby avoiding blackholes. This technique dramatically reduces the routing table size on the RNIC: the RNIC needs to store only aggregated routes plus a small number of host routes for RNICs that are unreachable via some planes. The majority of RNICs reachable across all planes are covered by the aggregated routes and therefore require no host routes. This approach is especially effective when unreachability is rare, which is typical in well&nbhy;managed clusters. Switches within each plane do not need to install the unreachable host route into their FIB tables.

Since a given RNIC communicates only with a limited subset of GPUs (due to collective communication patterns in distributed AI training, such as data, pipeline, and tensor parallelism), it can filter routes to retain only those it actually needs. The RNIC sends Address Prefix ORF entries to its BGP peer (leaf switch) per plane. These entries indicate the host routes for remote RNICs that the local RNIC is interested in. The peer filters outbound route updates accordingly, sending only the requested routes. Thus, the RNIC stores only a limited number of routes. For switches, there is no need to install host routes for remote RNICs. Therefore, the FIB suppression mechanism as described in can be leveraged. More specifically, upon receiving host routes from the attached RNICs, leaf switches MAY tag those routes with a "FIB-Suppress" Extended Community attribute as defined in Section 4.2.1. Compared to the approach described in Section 4.1, this method enables fine&nbhy;grained WECMP load balancing. For example, some modern transceivers with partial lane failures may continue operating, though at reduced capacity. In such cases, even though each RNIC remains multi&nbhy;homed to multiple planes at the same nominal interface speed, the actual available bandwidth can differ across planes. By obtaining host routes for the communicating RNICs along with their associated path&nbhy;bandwidth attributes, fine&nbhy;grained WECMP load balancing is achieved.

The FIB-Suppress Extended Community indicates that the associated routes MAY be suppressed from the FIB (i.e., not installed in the forwarding table). It is a new AS&nbhy;Specific Extended Community and MUST be transitive. The low&nbhy;order octet of the Type field is to be assigned (TBD). The Value field consists of two sub-fields: Global Administrator sub-field: This sub-field contains the AS number of the advertising router that appends the FIB-Suppress Extended Community. Local Administrator sub-field: This sub-field contains the Router ID of the advertising router that appends the FIB-Suppress Extended Community.

TBD.

IANA is requested to allocate a low-order octet value for the FIB-Suppress Extended Community from the registry of Transitive Two-Octet AS-Specific Extended Community Sub-Types. Upon allocation, IANA is requested to reference this document.

TBD.