RIFT in Dragonfly Topologies

Introduction RIFT today is standardized to deal with CLOS variant fabrics with some horizontal link exceptions. Given that interconnecting multiple CLOS via a dragonfly and its variants is an interesting topology (whether it's a full mesh or some kind of non-completely meshed regular lattice) this document addresses the resulting changes necessary to base RIFT specification to support dragonfly interconnected CLOS fabrics. The reader is advised that due to complexity of figures involved the ASCII version of the document may present those in simplified fashion.

Sparse Dragonfly of CLOS Fabrics +-----+ +-----+ +-----+ +-----+ | LA1 | | | | | | | +-----+ +-----+ +-----+ +-----+ | \ / | | \ / | | X | | X | | / \ | | / \ | +-----+ +-----+ +-----+ +-----+ | SA1 | | SA2 | | SC1 | | SC2 | +-----+ +-----+ +-----+ +-----+ | | | | | | | | | +-------|-|----------------------+ | | | | +---------|-|----------------------+----------+ | | | | | | +----------+ | | | +----------------------|-|-------------------+ | +--------------+ +-------|-|---------------------+ | | | | +-----+ +-----+ | SB2 | | SB1 | +-----+ +-----+ \ / X / \ +-----+ +-----+ | | | LB1 | +-----+ +-----+ To start with, visualizes three simple single plane fabrics interconnected via a DragonFly+ backbone. The behavior of standard RIFT is better understood if we look at the homomorphic version of the same topology in . We can see that it is nothing else but a multi-plane CLOS with a lot of broken links for standard RIFT. The planes consist of S_x_1 and S_x_2 ToFs in each CLOS. Given this, leaf LB1 should be connected to SA1 to be in the plane and since it is not, SA1 will deduct that leaf LB1 fell off the plane 1 and negatively disaggregate it. Unfortunately the same is true for leaf LB1 from the view the SA2 in 2nd plane and it will negatively disaggregate it as well. Hence, leaf LA1 will not have any possibility to forward to LB1 using standard RIFT computed forwarding. This points us already to the first modification needed; we have to relax RIFT to forward through the horizontal links on ToFs and this will be the starting point of the next section.

Homomorphic View of Sparse Dragonfly as a Multi-Plane CLOS Fabric A Fabric B ********************* ********************* * * * * * +-*--*-----------+ +-*--------------+ * +---------|-*--*-+ +---|-----|-*----------| | * | | * * | | | | * | | * +-----+ +-----+ * * +-----+ +-----+ * +-----+ +-----+ * | SA2 | | SA1 | * * | SB2 | | SB1 | * | | | | * +-----+ +-----+ * * +-----+ +-----+ * +-----+ +-----+ * | | \ / | | * * | \ / | * | | \ / | | * +--|---\-/---|--|-*--*----|---\-/---|----*----+ | \ / | | * | X | +-*--*----|----X----|----*-------|----X----|--+ * | / \ | * * | / \ | * | / \ | * +-----+ +-----+ * * +-----+ +-----+ * +-----+ +-----+ * | | | LA1 | * * | | | LB1 | * | | | | * +-----+ +-----+ * * +-----+ +-----+ * +-----+ +-----+ * * * * * * * * ********************* *********************

Glossary The following terms are used in this document.

DF+ capable ToF:: ToF that provides DF+ extensions, both in recognizing the inter fabric links and computation procedures necessary to support those. The resulting combination allows the use of RIFT with dragonfly topologies overall.
Horizon:: We define horizon as a concept differentiating between inter fabric links and southbound pointing standard RIFT intra fabric links on a ToF. Both type of links need a different FIB to support alternate next hop when routing between fabrics. A link can only be on one side of the horizon but not both sides.
Inter Fabric Planes or IFL-planes:: Multi-Plane that spans multiple fabrics.
Inter Fabric link or IFL links:: A horizontal ToF link between two fabrics.
K-Alternate Next Hops or KNHs:: Next hop used by the originating fabric on a k-shortest path to the destination fabric. The receiver on the other side must subsequently use as next hop shortest path next hop to the destination fabric.

Horizontal Link Behavior at ToF Level Dragonfly+, being basically, when seen a single fabric, a multi-plane CLOS with many broken links (which we will call inter fabric planes or IFL planes to distinguish them from multi-plane within a fabric later) will somehow need to change the behavior of RIFT to allow forwarding via horizontal links at ToF level lest we end up inverting the fabric and force leaves to deal with transit traffic. Moreover, the necessity to deal with new mis-cabling concepts leads us to change the solution framework and consider this configuration not as a single fabric but as a multi-fabric setup with dragonfly links building inter fabric planes now. Additionally we will have to allow adjacencies on ToF horizontal links to another fabric and permit those to forward through such inter fabric planes while distinguishing such inter fabric (or IF) links from normal horizontal ToF "multi-plane ringing". Hence in instead of the first assumption of a single fabric we break out fabric A and fabric B and consider the links SA2-SB2 and SA1-SB1 as two "inter fabric DF+" links, or in short, as already introduced, IFL links. And fortunately enough, IFL links, just like all other horizontal ToF links, are considered northbound from both sides and northbound flooding rules apply, an ideal thing since with that ToFs will see full topology of their inter fabric plane. RIFT used in such DF+ configuration will require on ToF not only a DF+ capability flag but a fabric ID now which has to be distinct in each of the CLOS or dragonfly cliques. In case of non-DF+ mode a ToF will declare such links miscabled, once enabled to operate in DF+ it will mark those links as IFL links. Given `fabric_id` is an optional schema element a ToF operating in DF+ mode will reject all links to other ToFs without `fabric_id` value set or not indicating DF+ mode as mis-cabled to prevent a mixture of non-DF+ and DF+ ToFs in a setup. On the other hand, a ToF indicating DF+ capability and showing matching fabric id is clearly a normal horizontal multi plane ring in the same fabric.

First Route Computation Change Now that we can detect IFL links reliably we can also remove those from the computations used in negative disaggregation as first step. This will prevent ToFs in fabric A negatively disaggregating Fabric B prefixes, a desirable behavior. Not being able to forward from Fabric A to fabric B is obviously a far less desirable behavior and hence a ToF in DF+ mode needs to extend its route computation by a special southbound DF+ computation where we use SPF taking in first step all IFL links and the nodes behind them as candidates. This computation will result in a "direct inter fabric forwarding database" containing amongst others shortest path to prefixes in fabric B or in other words, direct inter fabric next hops.

Additional Bi-Sectional Bandwidth Route Computation Change One of the DF+ properties is that it not only provides a direct path to a destination but guarantees that destinations are reachable via KNHs to increase the bi-sectional bandwidth. In our first simple example SB1 forwarding to LA1 can take instead of SA1 directly a path through SC1 relying on it forwarding to SA1. And in less dense DF+ backbones we can even encounter longer indirect paths. To support this we introduce an additional SPF computation which computes for each IFL interface k-shortest-paths whereas K should be obviously reasonably constrained. To illustrate this by an example introduces a 17 fabrics rather sparse dragonfly mesh where a 3 hops alternate paths are viable (if so chosen). FID 0 forwarding to FID 2 has obviously the shortest path via FID1 but a k-shortest path computation will yield FID 13 and FID 16 as viable 3 hops k-shortest-paths. However, only the originating fabric can choose a k-shortest path and every subsequent non originator MUST follow equal cost shortest paths to prevent looping or excessive bow-tying through the fabric. Not only MUST it follow the shortest path to the destination fabric, the shortest path MUST be actually computed on a topology where the sender node is excluded to prevent looping. For the sake of clarity does not visualize the full path diversity since, e.g. FID 13 can viably choose FIDs 3, 6, 9 beside FID 12 as shortest path next hops providing very significant path diversity on a 3 hops path. Further "steering" of chosen nexthops can be achieved by e.g. DSCP marking or congestion notification schemes but such approaches are outside the scope of this document. Computing such alternate next hops will have the other beneficial effect of actually providing a backup path in case the direct IFL plane link to another fabric becomes unavailable.

Multi-Plane CLOS Fabrics Connected in Sparse Dragonfly missing TDB

Dragonfly with Multi-Plane CLOS Fabrics Most complex case of RIFT deployment would be a dragonfly topology of CLOS fabrics which are in themselves already multi-plane fabrics. To present it as homomorphic graph is included. The symmetry is obvious, we end up with the normal RIFT ringing within the fabric, e.g. r_A for fabric A and then for the inter fabric planes dragonfly is basically the according ringing itself, here IR_1 and IR_2. Observe that the northbound flooding occurring on all those links will present each ToF with the full topology of the dragonfly, a necessary condition for proper disaggregation and further reachability computations. If the intra fabric ToF ringing should be avoided a tunnel between the ToFs within a fabric are necessary and may go all the way down to the leaves. How such tunnels are provisioned is outside the specification here but it will necessitate basically flat distribution of the loopbacks of the ToFs across whole fabric via e.g. redistribution of some RIFT routes in northbound and southbound direction or an equivalent scheme.

Multi-Plane CLOS Fabrics Connected in Sparse Dragonfly Fabric A ******************************************* * * * +-----+ +-----+ +-----+ +-----+ * * | | | | | | | | * * +-----+ +-----+ +-----+ +-----+ * * | \ / | | \ / | * * | X | | X | * * | / \ | | / \ | * * +-----+ +-----+ +-----+ +-----+ * * | | | | | | | | * * +-----+ +-----+ +-----+ +-----+ * * \ \ / / * * \ X / * * +-------\---+ / \ +---/---------+ * * | +-----+ +-----+ | * * | +---| SA2 | | SA1 |---+ | * * | | +-----+ +-----+ | | * * | | | | | r_A | * * | +------|---------|------+ | * * | | | | * ****|***********|*********|*************|** | | | | IR_2 | | | | IR_1 | | | | ****|***********|*********|*************|** * | | | | * * | +------|---------|------+ | * * | | | | | r_B | * * | | +-----+ +-----+ | | * * | +---| SB2 | | SB1 |---+ | * * | +-----+ +-----+ | * * +-------/---+ \ / +---\---------+ * * / X \ * * / / \ \ * * +-----+ +-----+ +-----+ +-----+ * * | | | | | | | | * * +-----+ +-----+ +-----+ +-----+ * * | \ / | | \ / | * * | X | | X | * * | / \ | | / \ | * * +-----+ +-----+ +-----+ +-----+ * * | | | | | | | | * * +-----+ +-----+ +-----+ +-----+ * * * ******************************************* Fabric B

Forwarding Considerations Since RIFT is being extended with the concept of KNH and IP packets do not carry any marking as the path they have taken indiscriminate forwarding using non-shortest paths at ToF level may loop in inter fabric case. To prevent this the ToFs have to maintain the concept of a "split horizon" on the arriving traffic. Any traffic arriving at the ToF that is targeted at the prefix within its fabric can be forwarded without any further considerations. On the other hand, traffic not targeted at the fabric of the ToF arriving at a inter fabric link MUST use a FIB generated by . The solution will naturally limit any non-shortest inter fabric path in ToF case to shortest paths computed without including the sender. As last case, traffic arriving from within its own fabric but targeted at another fabric can use any of the k-shortest paths as next hop as described in . Observe that per interface specific FIB is nothing particularly special, any technology supporting VPN or trunking today is already capable of provisioning interface specific forwarding behavior.

Partitioning of inter Fabric Planes A special case where a plane within a remote fabric breaks down is not noticeable in another fabric and hence the traffic can black hole since we do suppress the IFL links during negative disaggregation normally. To detect the condition reliably a ToF has to compute the inter fabric view of all the other ToFs in its own fabric while including IFL links and consider the resulting difference as "inter fabric negative disaggregation". This is possible but at scale can present significant computational load and is left therefore as optional behavior. Additionally, even when the fabric is a single plane fabric it must be then ringed at ToF level since otherwise the ToFs do not see the inter fabric planes they are not part of as an IFL ring. The same computation will deal with an even stranger case of a double failure on the IF links where a ToF becomes completely separated from the other fabrics. It will detect this and initiate negative disaggregation for the according prefixes.

Specification Precise schema changes and computation algorithms are to be provided in future version of the draft in detail. Basically the LIEs and Node TIEs need to be extended by fabric_id and DF+ mode indication and computations described conceptually in former chapters tightly specified.

Summary Overview A final is provided to map things back to the usual dragonfly sparse topology and show the concepts in action. We see three fabrics, each of them multi-plane (though mixes are absolutely possible as long the number of ToFs connected to dragonfly are kept the same). The fat links represent the "IFL horizon", i.e. any traffic coming from those links cannot use alternate next hops to the destination. In this example traffic from LA11 going through PA11 and SA2 towards LC11 is given two choices of next hops, either SC2 or SB2. Now that it entered the IFL horizon in case SB2 receives it no further alternate next hops will be used but traffic will be handed off to SC2 which applies the same rule and in this case actually forwards the traffic into the fabric.

Multi-Plane CLOS Fabrics Connected in Sparse Dragonfly [[ Overview of the solution, Refer to PDF for Picture ]] For the more complex case of alternate paths longer than 2 hops and the resulting forwarding behavior and path diversity has been described in the document.

IANA Considerations This document requests allocation for the following RIFT codepoints. TBD

Security Considerations TBD

Acknowledgements Dmitry Afanasiev's ideas around his work with BGP and dragonfly started interesting discussions, and he provided the crucial split horizon forwarding idea. Jeff Tantsura encouraged the work from its initial conception. Many thanks to Benson Muite for ASCII figures.