EVPN Interoperability Modes

EVPN Interoperability Modes Cisco

170 W. Tasman Drive San Jose 95134 CA USA lkrattig@cisco.com

Cisco

170 W. Tasman Drive San Jose 95134 CA USA sajassi@cisco.com

Cisco

170 W. Tasman Drive San Jose 95134 CA USA sthoria@cisco.com

Nokia

520 Almanor Avenue Sunnyvale 94085 CA USA jorge.rabadan@nokia.com

Juniper

drake@juniper.net

Routing BESS Working Group Ethernet VPN (EVPN) provides different functional modes in the area of Service Interface, Integrated Route and Bridge (IRB) and IRB Core connectivity. This document specifies how the different EVPN functional modes and types can interoperate with each other. This document does not redefine the existing functional modes but describes how these modes interoperate. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Ethernet VPN (EVPN) provides different functional modes in the area of Service Interface, Integrated Route and Bridge (IRB) and IRB connection model. It is understood that the different modes are defined with different use-cases in mind. Even with the specific use- cases and the resulting mode definition, the aim of interoperability is critical. The following EVPN modes are considered for interoperability. It is limited to the most pertinent interop modes as opposed to all permutations.

For Service Interfaces: VLAN-Aware Bundle and VLAN-Based.
For Integrated Routing and Bridging (IRB): Asymmetric IRB and Symmetric IRB.
For IRB connectivity: interface-less and interface-ful unnumbered IRB.

Future revisions of this Internet-Draft might address further variations of interoperability.

The tables below provide an overview of the valid combinations for interoperability described in this Internet-Draft. For the Service Interface Types as described in section 6 and section 5.1.2. Interoperability considerations are provided for the VLAN-Based Service interface (, section 6.1) and the VLAN-Aware Bundle Service Interface type ( section 6.3). The VLAN Bundle Service Interface ( section 6.2) is not considered at this time.

Service Interface Type Interoperability With regards to Integrated Route and Bridge (IRB), two different modes are defined in , with section 5 describing Symmetric IRB and section 6 Asymmetric IRB: The interoperability considerations between both IRB modes, Asymmetric IRB and Symmetric IRB, are represented within this Internet-Draft. For the IRB Core Connectivity, from all the available modes defined in , considered for interoperability is the interface-less mode (section 4.4.1) in conjunction with only one of the interface- ful modes, namely interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB (section 4.4.3). The close functional approximation between the two interface-ful modes, considerations for interoperability between interface-less and interface-ful Numbered are currently not considered. Similarly, the interoperability between the two interface-ful modes is currently not being considered, given the close functional relation and to limit permutations. Future revisions of this Internet-Draft might address further variations of interoperability.

IRB Core Connectivity interoperability

section 6 describes three different Service Interface Types. The two modes in focus for interoperability are namely the VLAN-Based Service Interface as defined in section 6.1 and the VLAN-Aware Bundle Service Interface as defined in section 6.3. The VLAN Bundle Service Interface is not considered. The VLAN-Based Service Interface defines an EVPN instance consisting of only a single broadcast domain, or Single Broadcast Domain per EVI, as defined in section 5.1.2 Option 1. In this mode, individual BGP Route Distinguisher (RD) and Route Target (RT) are required for each EVI. Each EVI corresponds to a single MAC-VRF identified by the RT. This mode has the advantage of a BGP RT constraint mechanism in order to limit the propagation and import of routes to only the PE that are interested. With VLAN-Based, the MAC- VRF corresponds to only a single bridge table. The VLAN-Based Service Interface uses the EVPN MAC/IP Advertisement route (, section 7.2) with the MUST requirement of the Ethernet Tag ID being set to zero. Differently, the VLAN-Aware Bundle Service Interface follows a bundling of multiple broadcast domains, with each having its own bridge table, into a single EVI. This refers to the definition of Multiple Broadcast Domain per EVI as described in section 5.1.2 Option 2. The advantage of this model allows a single RD/RT per broadcast domain, which becomes less relevant when VLAN-Based uses auto- derivation of RD/RT. With VLAN-Aware Bundle Service, RT Constraint, as defined in , does not help to reduce the dissemination of routes for a BD to the PEs attached to that BD. This is given by the nature of the bundle service where the RT is not sufficient to identify the MAC-VRF and corresponding bridge table. The differences between the two modes of Service Interfaces, namely VLAN-Based and VLAN-Aware Bundle Service Interface, is in the definition of the Ethernet Tag field within the EVPN routes. While VLAN-Based Service Interface defines the EtherTag must be set to zero, the VLAN- Aware Bundle Service Interface uses the VID within the EtherTag to identify the bridge table within the MAC-VRF. These two requirements are orthogonal and as a result make the interoperability of the two types mutually exclusive, interoperability is not achievable (Figure 1).

Interoperability of Service Interface Types <----- 2|EthTag [0]| | | | | +------+ +------+ | +-|M1/IP1! |M2/IP2!-+ +------+ +------+ ]]> As illustrated in Figure 1, the MAC/IP routes exchanged by PE1 and PE2 contain Ethernet Tags 2 and 0 respectively. The receiving PE will not process these routes and will normally discard them (treat-as- withdraw). By extending the requirements currently present, an interoperability is achievable. The adjustment would be as follows.

In case of VLAN Aware Bundle Service Interface on the receiving PE and with the consideration of VLAN Based Service Interface on the advertising PE:

For Service Interfaces: VLAN-Aware Bundle and VLAN-Based.
For Integrated Routing and Bridging (IRB): Asymmetric IRB and Symmetric IRB.
For IRB connectivity: interface-less and interface-ful unnumbered IRB.
MUST operate in Single Broadcast Domain per EVI.
Multiple Broadcast Domain per EVI case is not considered.
Must allow to send and receive zero EtherTag.
The import of routes is performed based on the import policy (route-target).
With single bridge table per MAC-VRF, additional evaluation of the EtherTag field is not required; the bridge table is sufficiently defined by the import route-target. Using a single MAC-VRF with per- BD route-target would deviate from the VLAN-Based Service Interface and would create a new interoperability permutation.
No Change to data-plane operation, the MPLS label identifies MAC- VRF + bridge-table, or the VNI identifies the MAC-VRF + the bridge- table.

Operates in Single Broadcast Domain per EVI.

In case of VLAN Based Service Interface on the receiving PE and with the consideration of VLAN Based Service Interface on the advertising PE:

MUST allow receiving of non-zero EtherTag.

- No Change in control-plane operation, the EVI import policy (route- target) identifies the broadcast domain (bridge-table) within a MAC- VRF. - No Change to data-plane operation, the MPLS label identifies MAC- VRF + bridge-table, or the VNI identifies the MAC-VRF + the bridge- table. While the expansion introduces additional configuration requirement for the VLAN-Aware Bundle Service Interface, it also allows for broader interoperability in the eventuality of vendor "A" only implemented VLAN-Based while vendor "B" only implemented VLAN-Aware Bundle Service Interface.

When Service Interface interoperability is required, a given PE should follow this section's procedures for all its broadcast domains (BDs) and not just the BDs that need interoperability. For those BDs where interoperability between VLAN-Aware Bundle and VLAN-Based Service Interface is needed, ignoring the presence of the EVPN routes Ethernet Tag ID on the PEs supporting VLAN-Based mode is not enough. Each PE needs to clearly signal what mode it supports, so that all the PEs attached to the same EVI can understand in what mode the EVI operates. Consider a scenario where PE1 is attached to the BD range BD1-10 and it operates in VLAN-Aware mode, whereas PE2 is attached to the BD range BD7-20 and operates in VLAN-Based mode. Interoperability is required for the intersecting BDs, i.e., BD7-10. For PE1, this means BD7-10 need to be separated into a dedicated MAC- VRF each. EVPN routes for each of these four MAC-VRFs MUST be advertised by PE1 with an Ethernet Tag ID of zero. In this way, PE1 indicates the use of VLAN-Based mode for those BDs. On reception, PE1 imports the BD7-10 routes based on the Route Target and ignoring the Ethernet Tag ID, as the Route Target alone is sufficient to identify the correct MAC-VRF and Bridge Table. The remaining BDs on PE1 (range BD1-6) continue operating in VLAN-Aware Bundle mode. In the same example, other PEs attached to BD1-6 must still process the received Ethernet Tag ID in the EVPN routes from PE1, so that they can identify the correct Bridge Table in a given MAC-VRF. PE2 operates in VLAN-Based mode for BD7-20, as per and . PE2's EVPN route advertisements for BD7-20 will include individual Route Targets per BD and an Ethernet Tag ID of zero. On reception, PE2 identifies the MAC-VRF and Bridge Table solely based on the Route Target.

The differences in the two inter-subnet forwarding modes, namely Asymmetric IRB and Symmetric IRB, are beyond just the information difference in the control-plane from an EVPN Route Type 2 perspective (EVPN MAC/IP Advertisement route). The two IRB modes have significant differences in inter-subnet forwarding behavior and as a result different operation during label imposition or encapsulation. With the Asymmetric IRB mode, the ingress PE performs a "bridge-and- route" operation while the egress PE follows a "bridge-only" approach. Differently, the forwarding behavior in Symmetric IRB mode performs a "bridge-and-route" operation on the ingress PE followed by a "route-and bridge" operation at the egress PE. The significance in difference is not only in the forwarding behavior itself but also around how the respective EVPN attribute are used for driving the inter-subnet operation. More specifically, in the case of inter- subnet forwarding with Asymmetric IRB, next to the MAC and IP address present in the EVPN Route Type 2, only MPLS Label1 is used towards the egress PE to specify the MAC-VRF and respective Bridge-Domain for forwarding. In inter-subnet forwarding with Symmetric IRB, next to the MAC and IP address present in the EVPN Route Type 2, MPLS Label2 associated with the IP-VRF is used for the inter-subnet forwarding operation towards egress PE. To facilitate interoperability between Asymmetric and Symmetric IRB, the usage of EVPN Route Type 2 with both MAC and IP address populated is considered. This is to not expand the requirement of additional capability exchange between PEs. Complementing Asymmetric IRBs absence of MPLS Label2 with an EVPN Route Type 5 (IP Prefix Advertisement) is not part of this procedure. The respective forwarding behaviors are described in . The following steps are required to ensure the interoperability between the Asymmetric and Symmetric IRB modes.

Asymmetric IRB and Symmetric IRB <----- 2|MAC/IP, 2 Label| | | | | +------+ +------+ | +-|M1/IP1! |M2/IP2!-+ +------+ +------+ ]]> Figure 2 illustrates the overview of an Asymmetric IRB PE (PE1) and a Symmetric IRB PE (PE2) within an interoperability deployment scenario. Attached to PE1, end-point M1/IP1 is attached to BD0 within MAC-VRF1. Respectively, on PE2 end-point M2/IP2 is connected via attachment circuit to BD2 positioned within MAC-VRF2. IRB0 and IRB2 represent the host-facing IRB interface for inter-subnet communication between the end-points located in the different IP subnet. The IRB interfaces for a common MAC-VRF/BD on PE1 and PE2 use the same IP address. With the IRB modes on PE1 being Asymmetric IRB and PE2 being Symmetric IRB, the MPLS Label fields, as part of the MAC/IP routes exchanged between the PEs, are different. PE1's update contains a single label, representing MPLS Label1 used for bridging purposes. PE2's advertisement contains two labels, one for bridging and one for routing, as part of the MAC/IP route. While PE1 receives all information necessary from PE2, PE2 is missing information necessary for its routing operation. As a result, inter-subnet routing between PE1 and PE2 is not achieved. By following the current existing forwarding behavior as described in , interoperability is theoretically achievable without changes in the control-plane format. Nevertheless, there are additional steps required that involves the local forwarding behavior of the PE with Symmetric IRB mode.

In case of Asymmetric IRB as the advertising PE and with Symmetric IRB on the receiving PE: - Asymmetric IRB PE MUST send MAC and IP information with MPLS Label1 as described in . In case of Symmetric IRB as the advertising PE and with Asymmetric IRB on the receiving PE: - Asymmetric IRB PE MUST be able to ignore MPLS Label2; already considers this.

In case of Symmetric IRB as the advertising PE and with Asymmetric IRB on the receiving PE:

Symmetric IRB PE has no additional requirements.

In case of Asymmetric IRB as the advertising PE and with Symmetric IRB on the receiving PE: - Symmetric IRB PE requires to add the host-binding information, MAC and IP, and associates them to the adjacency (ARP/ND) table facing the PE with Asymmetric IRB; this is in addition of adding the MAC address into the MAC-VRF table, using MPLS Label1. Since there is no MPLS Label2 or Route-Target for the IP-VRF, the Host IP is not specifically added to IP-VRF table. - Symmetric IRB PE must have defined all BD, MAC-VRF and IRB interfaces of the Asymmetric IRB PE.

Interoperability between the Asymmetric IRB and Symmetric IRB mode follows specific defined behavior that is predominantly required on the PE that operates in the Symmetric IRB mode. Nevertheless, in support for the interoperability, the PE operating in Asymmetric IRB must accommodate the following two minimum requirements (with references to Figure 2): 1) The PE that operates in Asymmetric IRB mode (PE1), MUST send the MAC/IP route including the Host IP address and only MPLS Label1. 2) The PE with Asymmetric IRB (PE1) must accept the MAC/IP routes sent from PE2 (Symmetric IRB), while ignoring the additional information of MPLS Label2 and Route-Target of the IP-VRF. In reference to 1), the PE MUST always send the end-point MAC address, Host IP address and related MPLS Label1 as part of the MAC/IP route towards the PE with Symmetric IRB (PE2). This route will be sent only with MPLS Label1 and the Route-Target of the matching MAC-VRF to achieve bridging. In reference to the illustration in Figure 2, PE1 must generate and advertise an EVPN MAC/IP route using:

MAC Length of 48
MAC Address of M1
IP Length of 32 / 128
IP Address of IP1
Label for MAC-VRF1
Route-Target of MAC-VRF1
Next-Hop PE1

For completeness of the requirements and in reference of 2), the MAC/IP route advertised from the PE operating in Symmetric IRB (PE2) is as follow:

MAC Length of 48
MAC Address of M2
IP Length of 32 /128
IP Address of IP2
Label for MAC-VRF2, IP-VRF1
Route-Target of MAC-VRF2, IP-VRF1
Next-Hop PE2

As defined in 2), the Label and Route-Target information for IP-VRF1 MUST be ignored by PE1 (PE with Asymmetric IRB). With PE2 operating in Symmetric IRB and with enabled interop mode, the MAC/IP route from PE1 (Asymmetric IRB) is processed in the respective bridging, routing and adjacency table. Based on the Route- Target for MAC-VRF1, the MAC address M1 will be imported into MAC- VRF1 respectively and placed within BD0. In addition, the host- binding information M1/IP1 MUST be installed within PE2's adjacency table. Subsequently, on PE2 the MAC address M1 and the host-binding information (adjacency table entry) of M1/IP1 MUST point towards PE1 as the next-hop. With no presence of the Route-Target for IP-VRF1, the IP address IP1 will not be specifically imported into IP-VRF1 and is not associated with a MPLS Label2. As a result of the interoperability, the additional efficiency provided by Symmetric IRB with regards to preserving adjacency table exhaustion is reduced; this is specifically when communicating with an Asymmetric IRB based egress PE. In contrast, the interop mode allows for communication between the different IRB modes. As a result, in the eventuality that vendor "A" only provides Asymmetric IRB, while vendor "B" only has Symmetric IRB available, interoperability for inter-subnet forwarding can be seamlessly achieved. In addition, two further benefits are present by implementing an Asymmetric/Symmetric IRB Co-Existence on the same PE (dual-mode PE). - A dual-mode PE can seamlessly communicate with PEs that are either in Asymmetric or in Symmetric IRB mode. - A dual-mode PE can act as Anchor for interconnecting Symmetric IRB and Asymmetric IRB based PE's (deployment restrictions might apply).

The two modes, namely interface-less and interface-ful Unnumbered SBD IRB, are closely related with regards to the information required in the EVPN Route Type 5. While interface-less provides all information for the IP prefix advertisement within the EVPN Route Type 5, in the case of interface-ful Unnumbered SBD IRB, an additional EVPN Route Type 2 is required for the next-hop recursive lookup. From a forwarding behavior, both approaches are similar and follow a symmetric routing approach but are not interoperable. Note as per the interface-ful Unnumbered SBD IRB is an OPTIONAL mode.

Interoperability of different IRB Core Connectivity Mode (unnumbered) | | <----- 5|No Label| | | | | +-------+ +-------+ | +-|TS1/SN1| |TS2/SN2!-+ +-------+ +-------+ ]]> The illustration in Figure 3 represents the possible deployment scenario between two different Core IRB Connectivity modes. Specifically, PE1 is operating with interface-less Core IRB Mode while PE2 operates with the interface-ful Unnumbered SDB IRB mode; both operate without interoperability capabilities. Attached to PE1 and PE2 respectively, Tenant System 1 (TS1) and Tenant System 2 (TS2) with different IP subnets are present. TS1 attached on PE1 as well as TS2 attached to PE2 are represented in a common IP-VRF (IP-VRF1), sharing a common Route-Target between the PEs. With the different IRB Core Connectivity modes on PE1 and PE2 respectively, the differences in IP prefix advertisements as described in are present. PE1 advertises only a single EVPN Route Type 5 (IP Prefix Route) for TS1 using the fields defined for the interface-less mode: EVPN Route Type 5:

IP Length of 0 to 32 / 0 to 128
IP Address of SN1
Label for IP-VRF1
GW IP Address set to zero
Route-Target of IP-VRF1
Router's MAC Extended Community of PE1
Next-Hop PE1

Differently, PE2 advertises an EVPN Route Type 2 (MAC/IP Route) next to the EVPN Route Type 5 (IP Prefix Route). The MAC/IP Route supports the requirement for recursive next-hop resolution for the next-hop used in the IP Prefix Route. Below the fields used in the Route Type 5 and respective Route Type 2 according to the interface-ful Unnumbered IRB mode: EVPN Route Type 5:

IP Length of 0 to 32 / 0 to 128
IP Address of SN1
Label SHOULD be set to 0
GW IP Address SHOULD be set to zero
Route-Target of IP-VRF1
Router's MAC Extended Community of PE2
Next-Hop PE2

EVPN Route Type 2:

MAC Length of 48
MAC Address of PE2
IP Length of 0
Label for IP-VRF1
Route-Target of IP-VRF1
Next-Hop PE2

While PE1 is missing the MPLS Label for the IP-VRF from PE2, PE2 is missing the MPLS Label information and the necessary info for the next-hop recursion. As a result, Routing with IP Prefix Advertisement between PE1 and PE2 is not achieved. By advertising an additional EVPN Route Type 2 from interface-less (PE1) and by advertising the MPLS Label as part of EVPN Route Type 5 from PE2, interoperability is achievable. The specific mode of operation would be as per the following two sections and refers to Figure 3 and Figure 4.

In case of interface-less on the advertising PE and with the consideration of interface-ful Unnumbered IRB as the receiving PE: The interface-less PE MUST generate and advertise an EVPN Route Type 2

MAC Length of 48
MAC Address with "Router MAC"
IP Length of 0
Label for IP-VRF
Route-Target of IP-VRF

In case of interface-less on the receiving PE and with the consideration of interface-ful Unnumbered IRB as the advertising PE: - MUST ignore EVPN Route Type 2 with MPLS Label and route-target matching the IP-VRF because there is no MAC-VRF defined matching this information.

In case of interface-ful Unnumbered on the advertising PE and with the consideration of interface-less as the receiving PE: - Shall advertise MPLS Label for IP-VRF in EVPN Route Type 5 with matching route-target. In case of interface-ful Unnumbered on the receiving PE and with the consideration of interface-less as the advertising PE:

No Additions Required.

Interop of different IRB Core Connectivity Types (unnumbered) | | <----- 5|Label| | | | | +-------+ +-------+ | +-|TS1/SN1| |TS2/SN2!-+ +-------+ +-------+ ]]> Illustrated in Figure 4 are the additional requirements for interface-less IRB Core Connectivity mode, specifically the MAC/IP Route (EVPN Route Type 2) necessary for PE2's next-hop recursion. Also, the MPLS Label addition within PE2's IP Prefix route (EVPN Route Type 5) is represented, which is required for advertisement by interface-ful Unnumbered IRB towards an interface-less PE (PE1) The interop mode introduces additional control-plane advertisements from an Interface-less perspective. This is necessary to allow interface-ful Unnumbered SBD IRB to perform the recursive lookup required. From a EVPN Type 5 perspective between the two types, most of the fields are already equally defined and populated as per . Exception is the IP-VRF Label, which is required to be added in the interface-ful Unnumbered SBD IRB's EVPN Type 5. In addition, the Interface-less addition allows the Co-Existence of both types on the same PE (dual-mode PE). Such a dual-mode PE can communicate at the same time with PEs that are in Interface-less or in interface-ful Unnumbered SBD IRB mode. The disadvantage of the additional advertisement has to be put into relation to advantage of successful interoperability where eventually vendor "A" only implemented interface-less while vendor "B" only implemented interface-ful Unnumbered SBD IRB.

With regards to IRB core connectivity both solutions, namely interface- less and interface-ful, provide a solution for Layer 3 connectivity among the IP-VRFs. Even as the functional result of both modes is the same, there are important considerations with regards to tunnel encapsulations. section 4 considers the choice for the NVO tunnel should be dictated by the tunnel capabilities. For example for the IP-VRF-to- IP-VRF model with interface-less, the NVO tunnel for MPLS needs to be IP NVO and for VXLAN needs to be Ethernet NVO. With the "IP-VRF-to-IP-VRF" model that is used in interface-ful (numbered or unnumbered), section 4.4.2 or 4.4.3 respectively describe the solution to accommodate Ethernet NVO tunnels (VXLAN or GPE, GENEVE, MPLS with MAC payload) only. In the case of interface- ful unnumbered, the Router-MAC Extended Community is always signaled via EVPN update message, which implies the presence of a MAC payload. IP NVO tunnels are not applicable to these two use-cases/models Depending on the use of NVO tunnels, interoperability between interface-less and interface-ful unnumbered requires additional changes on the Tunnel Encapsulation mode. This Internet-Draft considers the usage of a compatible NVO Tunnel mode between a PE operating in interface-less and a PE operating interface-ful unnumbered mode.

The security considerations of , , and apply to this document. This document defines procedures that rely on existing BGP control-plane and data-plane mechanisms and therefore introduce no new security considerations beyond those described in the referenced documents.

This document has no IANA actions.

With minimal additions, the most common EVPN types for Virtual Identifiers to EVI Mapping, Integrated Routing and Bridging and IP Prefix Advertisement can be made interoperable. The aim for interoperability doesn't remove the requirement for optimized types for different use-cases but allows flexibility and basic interoperability.

Cisco, Juniper and Nokia demonstrated successfully the ability of EVPN interoperability modes during EANTC's yearly "Multi-Vendor Interoperability Test". The Whitepaper can be obtained through EANTC with the latest version being available at .

A proof of the benefit with this interoperability mode has already been demonstrated during EVPN Multi-Vendor interoperability testing and also, in production environments. Specifically, Cisco and Nokia's VLAN-Based Service Interface successfully demonstrated interoperability with Juniper's VLAN-Aware Bundle Service Interface.

A proof of the benefit with this interoperability mode has already successfully demonstrated during EVPN Multi-Vendor interoperability testing. Specifically, Cisco operated in a Hybrid IRB (Dual-Mode) mode while other vendors operated in an Asymmetric IRB mode. Forwarding was achieved through dynamic detection of the alternate vendor PE's mode and adjustment to Asymmetric IRB for these specific BDs. Communication for all other BDs continued to be Symmetric IRB.

A proof of an interoperability mode between interface-less and interface-ful Unnumbered SBD IRB has already been demonstrated in production environments and during EVPN Multi-Vendor interoperability testing. Specifically, Cisco's addition for Interface-less is successfully deployed with Nokia's and Nuage's interface-ful Unnumbered SBD IRB at customers

Multi-Vendor Interoperability Test EANTC

In addition to the authors listed, the following individuals also contributed to this document: Cisco

170 W. Tasman Drive San Jose 95134 CA USA kriswamy@cisco.com