]> Cisco

djsmith@cisco.com

Cisco

sekrier@cisco.com

CoreWeave

sgaw@coreweave.com

AT&T

im8327@att.com

Routing Inter-Domain Routing BGP-4 BGP routing RR route reflector In BGP networks, Route Reflectors (RRs) help to simplify IBGP control plane provisioning as well as mitigate the session scale challenges associated with an IBGP full mesh between border routers. Given these operational benefits, RRs are widely deployed in modern BGP network architectures. This document describes common RR deployment models, including their respective trade-offs, and provides best practice suggestions based on years of industry experience. Operators of BGP networks should consider and/or adopt these best practices, as BGP control plane stability is critical to overall network stability. Status information for this document may be found at . Discussion of this document takes place on the Inter-Domain Routing Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction In BGP networks , Route Reflectors (RRs) help to simplify IBGP control plane provisioning as well as mitigate the session scale challenges associated with an IBGP full mesh between border routers. Given these operational benefits, RRs are widely deployed in modern BGP network architectures. This document describes common RR deployment models, including their respective trade-offs, and provides best practice suggestions based on years of industry experience. Operators of BGP networks should consider and/or adopt these best practices, as BGP control plane stability is critical to overall network stability.

Performance and Scale Considerations BGP RR performance and scale depends on a variety of factors including but not limited to: Available system resources: CPU performance, memory speed and quantity available. Number of BGP sessions, associated timers and update-groups. BGP address families configured , . Number of routes and associated paths for each BGP address family enabled as well as route/path churn. Whether the RR operates in the BGP control plane only or also in the IP forwarding plane. BGP capabilities enabled including but not limited to, for example, inbound/outbound route policies (simple versus complex), Extended Message support , Additional Paths , Graceful Restart , Optimal Route Reflection , Route Target Constraint , automatic detection and isolation of slow peers, and RIB/FIB filtering. Transport layer optimizations including but not limited to Maximum Segment Size (MSS), window size, Selective Acknowledgement (SACK) , and TCP Path MTU discovery , as well as TCP-AO session authentication . Network latency to and from clients and non-client peers. BGP implementation and performance of the network operating system itself. Operators should consider all these factors as part of their RR deployment. Further, since these factors vary widely among network operators, specific BGP RR performance and scale guidelines are outside the scope of this document. Other techniques and deployment best practices that help ensure RR and wider BGP control plane stability, including but not limited to Diffserv QoS and protecting the RR control plane , are also outside the scope of this document. Lastly, operational best practices related to the management, monitoring and maintenance of RRs are also out of scope.

Cluster Design Considerations The placement of a RR within an Interior Gateway Protocol (IGP) topology is important because it impacts network latency, which is a key factor in TCP and BGP performance as previously stated. Further, if the IGP metric to the NEXT_HOP influences the RR's best path selection, then the RR's location affects this metric. For these reasons, RRs have traditionally been deployed in close IGP proximity to their clients to minimize latency and ensure that the IGP metric to the NEXT_HOP is comparable to that of its clients. While network latency remains a key factor in TCP and BGP performance, modern techniques such as BGP Add-Paths and BGP Optimal Route Reflection (ORR) are now available to improve path selection. See below for more details.

Redundancy BGP RR redundancy enhances network stability by eliminating single points of failure and protecting against RR-level outages. Note, however, that excessive RR redundancy can be detrimental. This is because the more RRs a client (e.g., border router) peers with, the more IBGP sessions, updates, and paths the client must carry and process in its BGP table. If we consider two RR clusters (1 and 2), as shown in Figure 1, the RR clients (C-2A, C-2B, C-2C) in cluster 2 must process 50% more sessions, updates, and paths compared to cluster 1 clients. For example, if each of the RRs advertised the same 1.5 million (1.5M) routes to the clients, then the BGP tables on cluster 1 clients would carry 3M paths, while the BGP tables on cluster 2 clients would carry 4.5M paths. That is, the incremental overhead of BGP paths, sessions, and updates increases linearly with every redundant RR in the cluster.

Further, when a route is no longer available and needs to be withdrawn, an RR client cannot withdraw a route learned from its RRs until it receives a withdrawal from each of the RRs. As a result, increasing the number of redundant RRs could also lead to slower BGP convergence. Again, the incremental overhead of BGP sessions, updates, and paths increases linearly with every redundant RR in the cluster. Hence, for optimal redundancy and scaling, it is suggested that clients peer with at least two redundant RRs, but no more than four total - typically structured as two clusters, each containing a pair of redundant RRs. Also, depending on the network services and scale, different RR clusters may be suggested for different BGP address families. See below for more discussion on this topic. Consequently, this general guideline — limiting clients to peering with at most four RRs — applies per BGP address family.

CLUSTER_ID Configuration Redundant RRs within a cluster do not need to share the same CLUSTER_ID value. When RRs within a cluster peer as non-client IBGP peers with different CLUSTER_IDs, they exchange and process BGP routes with each other. However, if they share the same CLUSTER_ID, they exchange routes but discard each other's reflected routes to prevent loops, as specified in . This behavior allows flexibility in deploying RR redundancy without requiring identical cluster IDs, while ensuring loop prevention through the CLUSTER_LIST attribute mechanism. Consequently, there is no benefit to IBGP peering between redundant RRs sharing a CLUSTER_ID, unless either of the RRs have EBGP sessions or redistribute local IGP, static, or connected routes into BGP. This is because the CLUSTER_ID is not prepended to such non-IBGP learned routes as these are not reflected routes per . As a result, these non-reflected routes would be exchanged between redundant RRs configured with the same CLUSTER_ID but not discarded.

Cluster Sizing In addition to managing the number of redundant RRs per cluster, it's also important to maintain moderate RR cluster sizes (i.e., the number of clients per cluster) to limit the blast radius in the event of an RR cluster failure. The upper limit suggested is no more than one thousand (1K) clients per cluster. Accordingly, BGP networks often require multiple RR clusters - not only for fault containment but also for horizontal scaling (sessions, path selection), geo-redundancy, and improved BGP best-path selection.

Multi-Cluster Designs While the BGP control plane is critical to network stability, the RR design is critical to BGP control plane stability. In general, there are three (3) common multi-cluster RR designs: Full mesh Multi-tier Multi-plane Each of these designs has its own set of trade-offs which are described below. These are also not the only design options, as hybrid deployments combining these approaches are also possible.

Full Mesh A full mesh, multi-cluster RR design employs a full IBGP peering mesh between all RRs in all clusters as shown in Figure 2. This is the most common and widely deployed multi-cluster RR design, as it effectively balances scalability and fault isolation while optimizing the distribution of BGP routes and paths within the network. It achieves this by minimizing the number of IBGP hops that updates traverse, which leads to improved BGP performance at scale.

However, for massive-scale BGP deployments with many RR clusters, the IBGP full mesh between RRs becomes a scalability bottleneck. This is due to high RR session counts and the provisioning complexity involved when adding new RR clusters to the topology. For example, introducing a new RR requires configuring new IBGP sessions on all existing RRs, which increases overhead and BGP session density. This N^2 scaling challenge can also degrade BGP update processing and network convergence, much like a full IBGP mesh between a large number of border routers without RRs.

Multi-Tier A multi-tier, multi-cluster RR design is a hierarchical architecture involving multiple tiers of RRs, as illustrated in Figure 3. This design is intended for massively large-scale BGP deployments with many RR clusters.

At the lowest level, border routers remain clients of their assigned RR clusters, similar to the full mesh multi-cluster design described above. In this multi-tier model, the RRs peering with border routers are referred to as Tier 2 RRs. Instead of a full IBGP mesh between them, Tier 2 RRs are deployed as clients of Tier 1 RRs, which are themselves fully meshed. While this design reduces IBGP session density on RRs and simplifies their provisioning, it may adversely affect BGP convergence time given the incremental RR hops within the IBGP control plane. Furthermore, a failure of a Tier 1 RR cluster typically results in a large blast radius.

Multi-Plane A multi-plane, multi-cluster RR design comprises multiple independent RR planes (e.g., RED and BLUE). Within each plane, clusters can be fully meshed or multi-tier as described above, and redundant RRs within the same cluster do not peer with one another. Note that while this document uses a two-plane full mesh model for illustration in Figure 4, the architecture is not limited to two planes or a full mesh. As shown in Figure 4, RRs within the RED plane (RR-R1 through RR-R6) are fully meshed as are RRs in the BLUE plane (RR-B1 through RR-B6), except no IBGP peering between redundant RRs in the same cluster. In a two-plane design, RR clients (typically border routers) peer with the redundant RRs of one specific cluster in each plane, resulting in four IBGP sessions per client. Multi-plane RR designs can be used in different ways, and each has a different failure behavior. For example, if multi-plane is used for increased redundancy, each plane should carry enough routing information to maintain service if another plane fails. In this deployment model, the failure of one plane should reduce redundancy but not remove required IP reachability. Further, a multi-plane design used for redundancy also facilitates software or hardware upgrades. Because the planes operate independently and provide mutual redundancy, they can be maintained or upgraded separately without disrupting IP reachability. Alternatively, if planes are used for horizontal scaling, different routes or paths may be distributed across different planes. In this deployment model, the failure of one plane may remove the routes or paths carried only by that plane.

Address Family Considerations BGP supports multiple route types defined by Address Family Identifier (AFI) and Subsequent Address Family Identifier (SAFI) numbers. When a BGP control plane carries multiple AFI/SAFI route types at high scale, it is suggested to consider whether deploying separate RRs for different sets of AFI/SAFI route types offers any benefits in terms of performance, scalability, and fault isolation. Such an approach isolates BGP update processing and path computation within specific address families, preventing resource contention and prioritization conflicts across AFI/SAFIs that can otherwise degrade RR performance. This separation not only enhances scalability and efficiency in BGP deployments by reducing the processing load on individual RRs, but also provides increased fault isolation both at the RR level and at the IBGP session level, since individual sessions do not have to carry multiple sets of AFI/SAFIs. With that said, for multi-service BGP networks it is suggested to consider deploying Internet service routes (IPv4, IPv6, 6PE) on separate RRs from VPN service routes. Similarly, consider deploying L3VPN service routes (VPNv4, VPNv6, MVPN) on separate RRs from L2-based VPN service routes (L2VPN, EVPN). This is not only suggested for better BGP scaling, convergence and fault isolation between services but also because different services rely on different BGP features to function effectively. Specifically, Internet RRs often use features like BGP Add-Path (primary/backup, n-way, all) and/or Optimal Route Reflection . Conversely, a VPN RR often relies on different RDs for path diversity and uses RT constraint to limit the propagation of VPN routes to peers that have advertised their respective Route Targets. In addition, RRs configured for BGP IPv4 labelled unicast and deployed in the MPLS forwarding plane as part of a seamless MPLS architecture require next-hop-self configuration and MPLS label allocation. Additionally, BGP Link-State , which is used for topology collection of IGP link-state and traffic engineering information (such as RSVP-TE, SR-TE, SR EPE and SR Flexible Algorithms) by SDN controllers, is highly verbose and not used by RRs for BGP best path computation or route reflection. Therefore, it is suggested to deploy BGP-LS on dedicated RRs separate from those computing and reflecting BGP best paths to further optimize BGP control plane performance. The approach described above aligns with industry best practices by isolating resource-intensive or distinct BGP address families and services on dedicated RRs, which enhances fault isolation as well as BGP scalability and convergence for each network service. It is also worth noting that separating services by AFI/SAFI across RRs increases the number of RR loopback addresses carried in the link-state IGP.

Policy and Update-Group Considerations Minimal inbound and outbound policies are suggested for RRs, if any at all, to avoid extra processing overhead that can adversely affect scale and convergence. However, a significant benefit of RRs is the ability to influence network-wide routing policies from a limited number of central locations. Hence, the deployment of routing policies on RRs must carefully balance scale and convergence requirements with simplified provisioning.

Input Policy RRs can be configured to modify attributes, such as communities , within received updates, for example, to signal geographic location or service preferences.

Output Policy and Update-Group RRs can be configured to restrict path propagation for all or a subset of clients, such as those within a particular geographic region. Note, however, that use of outbound policies on a RR may expand the number of update-groups , which can adversely affect BGP convergence.

Slow Peer Considerations It is common for RRs to dynamically establish BGP update-groups across IBGP peers that share identical outbound routing policies , including instances where no outbound policies are applied. This improves RR performance, as a RR formats a single update advertised to all peers within an update-group rather than advertising unique updates for each individual peer. This also enhances overall BGP control plane performance. However, a slow peer within an update-group that cannot keep up with the rate at which a RR generates BGP update messages can adversely affect other members of the group. A single slow client can force a RR to throttle, delaying updates to other peers in the same group. This degrades BGP convergence and may increase traffic disruption during BGP convergence events. It is suggested to isolate known slow clients, such as legacy routers with limited CPU and memory resources, into their own IBGP update-group. This isolation can be performed manually via static configuration or, if supported by the RR software, dynamically. Dynamic isolation involves the RR automatically detecting and moving slow peers out of an update-group so as not to adversely impact other members. This dynamic behavior can also return previous slow peers back into an update-group after their BGP processing is caught up. However, for permanently known slow peers it is suggested to statically assign them to dedicated update-groups. This approach helps to prevent any adverse impact on other peers and minimizes the RR churn associated with the automatic detection, isolation, and reintegration of slow peers. The procedure to identify slow peers - a process that can be automated - involves monitoring the BGP table version and BGP output queue of the peers in each update-group. This information should then be used to verify whether the BGP table version of the peers ever catches up with the main BGP table version or whether it is always behind. Similarly, check whether any peers have a very high output queue value for a long duration. If so, it is likely that these are slow peers and should be isolated.

Path Selection Considerations By default, RRs compute and advertise only a single best path per route. Furthermore, when the IGP metric to the NEXT_HOP influences best path selection, the RR uses its own IGP metric rather than that of its clients. These default behaviors cause path hiding, which can lead to suboptimal routing and, in specific configurations, the route oscillations described in . In BGP L3VPN and EVPN deployments, it is common to configure each provider edge (PE) router with a unique Route Distinguisher (RD). Consequently, best paths are computed per prefix per RD, allowing RRs to advertise multiple paths (if available) to clients by default. This approach facilitates optimal routing, load balancing, and improved network redundancy for these address families. However, RDs do not apply to Internet routes carried in the global routing table (GRT) via the BGP IPv4, IPv6, or 6PE address families. Historically, for Internet routing with RRs routing policies were used to ensure redundant RRs selected different best paths (if available). In this way, Internet border routers would learn multiple paths (one per RR), thereby, facilitating more optimal routing, load balancing and network redundancy. With the advent of BGP Add-Paths , however, such routing policies are no longer necessary. BGP Add-Paths allows a BGP speaker to exchange multiple paths (primary/backup, n-way, all) per prefix, providing clients with a wider range of candidate paths from each of its RRs. If the IGP metric to the NEXT_HOP is a factor, clients can then evaluate these multiple candidate paths using their own IGP metrics. While BGP Add-Paths is suggested for IBGP sessions to enable more optimal routing, load balancing and improved network redundancy, it may also increase the number of BGP paths on some nodes. Therefore, operators must carefully evaluate the impact on network and router scale before enabling it. Alternatively, if a RR is configured to only advertise a single best path and the IGP metric to the NEXT_HOP is a factor during its best path selection, a RR will by default use its own IGP metric rather than that of its clients. As stated earlier, this can lead to suboptimal routing. To avoid this RRs have traditionally been deployed in close IGP proximity to their clients. With the advent of BGP ORR , however, a RR can be configured to use the IGP metric of its clients instead of its own during best path selection. This enables a RR to select the most optimal best paths on behalf of its clients, regardless of its proximity to its clients. Note, however, that BGP ORR increases BGP processing overhead on the RR, which may impact scalability. Additionally, the RR can only compute ORR-based shortest-path trees for root locations that are part of its link-state IGP.

Route Target Constraint Considerations BGP Route Target Constraint (RTC) can significantly reduce the volume of VPN updates an RR advertises to edge routers (PEs). With RTC, it is suggested that RRs advertise an RT-filter default to clients, while clients send specific RT-filters to the RR, rather than the reverse. This is particularly important in multi-tier RR designs: upper-tier RRs should advertise an RT-filter default to lower-tier RRs, while lower-tier RRs send specific RT-filters to the upper-tier RRs. Failure to do so may cause clients and lower-tier RRs to receive all VPN routes, potentially exceeding their scale capacity. Further, RTC is most effective when activated end-to-end to build precise outbound route filters, including those for single-domain and multi-domain network-based VPNs. This also includes multi-tier, multi-cluster RR designs so that Tier 1 RRs only send the required VPN prefixes to Tier 2 RRs, drastically reducing BGP VPN processing and overhead. Operators should also be aware of known issues associated with RTC in specific scenarios. These are documented in and , both of which also propose solutions.

Transport Optimizations

TCP Path MTU Discovery To improve BGP control plane performance, it is suggested to enable TCP Path MTU Discovery (PMTUD) on both RRs and clients; this allows TCP to select an appropriate Maximum Segment Size (MSS) for each BGP session. In this way, BGP sessions use the largest packet size that does not require IP fragmentation anywhere along the path to client and non-client IBGP peers.

Ethernet MTU The Ethernet maximum transmission unit (MTU) is the size of the largest frame, minus the 4-byte frame check sequence (FCS), that can be transmitted on a Ethernet network. Every physical network along the path to the destination of a packet can have a different MTU. The default Ethernet MTU (Maxiumum Transmission Unit) value is 1514 bytes for standard Ethernet frames and 1518 bytes for 802.1Q tagged frames. These numbers exclude the 4-byte frame check sequence (FCS). When the MTU size is larger, more data can be sent in fewer number of packets. As each packet contains a header, a small number of packets means less header data to be transmitted. Thus, a larger size MTU helps in transmitting more actual data faster and efficiently. To ensure the RR is not the limiting factor related to PMTUD and BGP update packet sizes, it is suggested to enable Jumbo frame support on RR interfaces with the largest packet size value available.

Miscellaneous Considerations

BGP Router ID To prevent IBGP session instability caused by changes in the BGP router ID when a physical interface's state changes, it is suggested to statically configure the BGP router ID using the IP address of a virtual loopback interface. This approach is preferred because loopback interfaces are more stable than physical interfaces, which may go down or change state, causing the router ID to change and potentially leading to BGP session resets. This IBGP best practice applies to both RRs and clients.

BGP RIB/FIB Filtering For RRs operating exclusively in the control plane, configuring RIB/FIB filtering policies to suppress BGP route installation is suggested. Restricting these tables on non-forwarding devices: Minimizes RIB and FIB resource consumption Eliminates unnecessary RIB and FIB processing overhead Improves a RR's overall efficiency and scalability

Form Factor Considerations Traditionally, RRs were deployed using physical routing devices. With the advent of Network Function Virtualization (NFV), RRs that operate exclusively in the control plane are now commonly deployed as Virtual Network Functions (VNFs) or virtual RRs. Beyond cost savings, virtual RRs offer superior scalability and convergence performance compared to physical RRs by providing substantially more CPU performance, memory speed and quantity available for BGP protocol processing. Consequently, deploying control-plane-only RRs as virtual RRs is suggested. In contrast, RRs that participate in the IP forwarding plane are typically deployed as physical RRs, unless a virtual RR can satisfy the required forwarding capacity. It is also worth noting that virtual RRs rely on independent compute, hypervisor, storage, and virtual networking infrastructure; however, vendors offer integrated appliances that bundle these components.

Non-Stop Routing/Forwarding A virtual RR is deployed on a compute server which has no concept of redundant route processors. A physical RR can be configured with redundant route processors and leverage this redundancy to enable BGP Non-Stop Routing (NSR). NSR preserves BGP session and control-plane state during a route processor failover, ensuring that IBGP peers do not detect the switchover and eliminating the need for session resets, BGP Graceful Restart, or its helper mode. While physical RRs uniquely support NSR, they generally offer significantly lower BGP scaling compared to virtual RRs as described earlier. Consequently, deploying multiple redundant virtual RRs is suggested for networks with high BGP control plane scale.

Security Considerations This document introduces no new security considerations to the BGP protocol itself. It describes deployment considerations and best practices that BGP network operators should consider or adopt when deploying RRs to maximize BGP control plane performance, scalability and stability. While general BGP operational security guidance is provided in , several RR considerations with security relevance are described below.

Session Authentication BGP session cryptographic authentication is suggested to ensure that BGP updates are only accepted from trusted, verified peers, which helps to reduce the risk of unauthorized route injection and session hijacking. MD5 is the legacy method for BGP authentication and has been obsoleted by TCP-AO . While widely supported, MD5 is considered cryptographically weak by modern standards. MD5 is only suggested if a BGP speaker does not support TCP-AO . TCP-AO is the modern standard for BGP session security and is the suggested method as it provides stronger cryptographic algorithms and improved security features.

BGP Session Prefix Limits Configuring BGP session prefix limits is an important measure to protect a network from an EBGP peer unexpectedly advertising an excessive number of routes, such as the full Internet routing table. BGP session prefix limits enforce a maximum receive prefix count from a neighbor for each address-family, helping safeguard a network from out-of-resource (OOR) conditions which could lead to traffic disruption. In the event a session prefix limit is hit, BGP implementations typically provide several actions in response: Bring down session Restart session Discard extra paths SYSLOG warning Beyond generating a SYSLOG warning, it is suggested to implement one of the listed actions for EBGP sessions based on your specific network requirements. BGP session prefix limits can be configured on IBGP sessions as well; however, if configured for IBGP, it is suggested to use only a SYSLOG warning without taking further action. This approach helps monitor prefix counts while avoiding IBGP session resets or disruptions, which could result in a more significant detrimental impact.

Control Plane Protection It is common for modern IP routers to support control plane protection , including but not limited to control plane policing. Typically enabled by default on modern IP routers, this capability helps protect a router from excessive control plane traffic by classifying and policing different packet types, thereby preventing potential denial-of-service conditions. With regard to RRs, it is important for large-scale deployments to ensure that BGP control plane policer rates are optimally configured to prevent BGP packet drops, which could adversely affect both BGP stability and overall network stability.

IANA Considerations This document has no IANA actions.

This document discusses the Border Gateway Protocol (BGP), which is an inter-Autonomous System routing protocol. The primary function of a BGP speaking system is to exchange network reachability information with other BGP systems. This network reachability information includes information on the list of Autonomous Systems (ASes) that reachability information traverses. This information is sufficient for constructing a graph of AS connectivity for this reachability from which routing loops may be pruned, and, at the AS level, some policy decisions may be enforced. BGP-4 provides a set of mechanisms for supporting Classless Inter-Domain Routing (CIDR). These mechanisms include support for advertising a set of destinations as an IP prefix, and eliminating the concept of network "class" within BGP. BGP-4 also introduces mechanisms that allow aggregation of routes, including aggregation of AS paths. This document obsoletes RFC 1771. [STANDARDS-TRACK] The Border Gateway Protocol (BGP) is an inter-autonomous system routing protocol designed for TCP/IP internets. Typically, all BGP speakers within a single AS must be fully meshed so that any external routing information must be re-distributed to all other routers within that Autonomous System (AS). This represents a serious scaling problem that has been well documented with several alternatives proposed. This document describes the use and design of a method known as "route reflection" to alleviate the need for "full mesh" Internal BGP (IBGP). This document obsoletes RFC 2796 and RFC 1966. [STANDARDS-TRACK] This memo describes a technique for dynamically discovering the maximum transmission unit (MTU) of an arbitrary internet path. It specifies a small change to the way routers generate one type of ICMP message. For a path that passes through a router that has not been so changed, this technique might not discover the correct Path MTU, but it will always choose a Path MTU as accurate as, and in many cases more accurate than, the Path MTU that would be chosen by current practice. [STANDARDS-TRACK] This document describes an extension to BGP which may be used to pass additional information to both neighboring and remote BGP peers. [STANDARDS-TRACK] This document defines an architecture for implementing scalable service differentiation in the Internet. This memo provides information for the Internet community. This memo proposes an implementation of SACK and discusses its performance and related issues. [STANDARDS-TRACK] This document describes a method by which a Service Provider may use an IP backbone to provide IP Virtual Private Networks (VPNs) for its customers. This method uses a "peer model", in which the customers' edge routers (CE routers) send their routes to the Service Provider's edge routers (PE routers); there is no "overlay" visible to the customer's routing algorithm, and CE routers at different sites do not peer with each other. Data packets are tunneled through the backbone, so that the core routers do not need to know the VPN routes. [STANDARDS-TRACK] This document defines Multi-Protocol BGP (MP-BGP) procedures that allow BGP speakers to exchange Route Target reachability information. This information can be used to build a route distribution graph in order to limit the propagation of Virtual Private Network (VPN) Network Layer Reachability Information (NLRI) between different autonomous systems or distinct clusters of the same autonomous system. This document updates RFC 4364. [STANDARDS-TRACK] This document describes a mechanism for BGP that would help minimize the negative effects on routing caused by BGP restart. An End-of-RIB marker is specified and can be used to convey routing convergence information. A new BGP capability, termed "Graceful Restart Capability", is defined that would allow a BGP speaker to express its ability to preserve forwarding state during BGP restart. Finally, procedures are outlined for temporarily retaining routing information across a TCP session termination/re-establishment. The mechanisms described in this document are applicable to all routers, both those with the ability to preserve forwarding state during BGP restart and those without (although the latter need to implement only a subset of the mechanisms described in this document). [STANDARDS-TRACK] This document specifies the TCP Authentication Option (TCP-AO), which obsoletes the TCP MD5 Signature option of RFC 2385 (TCP MD5). TCP-AO specifies the use of stronger Message Authentication Codes (MACs), protects against replays even for long-lived TCP connections, and provides more details on the association of security with TCP connections than TCP MD5. TCP-AO is compatible with either a static Master Key Tuple (MKT) configuration or an external, out-of-band MKT management mechanism; in either case, TCP-AO also protects connections when using the same MKT across repeated instances of a connection, using traffic keys derived from the MKT, and coordinates MKT changes between endpoints. The result is intended to support current infrastructure uses of TCP MD5, such as to protect long-lived connections (as used, e.g., in BGP and LDP), and to support a larger set of MACs with minimal other system and operational changes. TCP-AO uses a different option identifier than TCP MD5, even though TCP-AO and TCP MD5 are never permitted to be used simultaneously. TCP-AO supports IPv6, and is fully compatible with the proposed requirements for the replacement of TCP MD5. [STANDARDS-TRACK] This memo provides a method for protecting a router's control plane from undesired or malicious traffic. In this approach, all legitimate router control plane traffic is identified. Once legitimate traffic has been identified, a filter is deployed in the router's forwarding plane. That filter prevents traffic not specifically identified as legitimate from reaching the router's control plane, or rate-limits such traffic to an acceptable level. Note that the filters described in this memo are applied only to traffic that is destined for the router, and not to all traffic that is passing through the router. This document is not an Internet Standards Track specification; it is published for informational purposes. The Border Gateway Protocol (BGP) is the protocol almost exclusively used in the Internet to exchange routing information between network domains. Due to this central nature, it is important to understand the security measures that can and should be deployed to prevent accidental or intentional routing disturbances. This document describes measures to protect the BGP sessions itself such as Time to Live (TTL), the TCP Authentication Option (TCP-AO), and control-plane filtering. It also describes measures to better control the flow of routing information, using prefix filtering and automation of prefix filters, max-prefix filtering, Autonomous System (AS) path filtering, route flap dampening, and BGP community scrubbing. This document defines a BGP extension that allows the advertisement of multiple paths for the same address prefix without the new paths implicitly replacing any previous ones. The essence of the extension is that each path is identified by a Path Identifier in addition to the address prefix. Some network operators build and operate data centers that support over one hundred thousand servers. In this document, such data centers are referred to as "large-scale" to differentiate them from smaller infrastructures. Environments of this scale have a unique set of network requirements with an emphasis on operational simplicity and network stability. This document summarizes operational experience in designing and operating large-scale data centers using BGP as the only routing protocol. The intent is to report on a proven and stable routing design that could be leveraged by others in the industry. This document describes Path MTU Discovery (PMTUD) for IP version 6. It is largely derived from RFC 1191, which describes Path MTU Discovery for IP version 4. It obsoletes RFC 1981. This document specifies a set of procedures for using BGP to advertise that a specified router has bound a specified MPLS label (or a specified sequence of MPLS labels organized as a contiguous part of a label stack) to a specified address prefix. This can be done by sending a BGP UPDATE message whose Network Layer Reachability Information field contains both the prefix and the MPLS label(s) and whose Next Hop field identifies the node at which said prefix is bound to said label(s). This document obsoletes RFC 3107. The BGP specification (RFC 4271) mandates a maximum BGP message size of 4,096 octets. As BGP is extended to support new Address Family Identifiers (AFIs), Subsequent AFIs (SAFIs), and other features, there is a need to extend the maximum message size beyond 4,096 octets. This document updates the BGP specification by extending the maximum message size from 4,096 octets to 65,535 octets for all messages except for OPEN and KEEPALIVE messages. This document defines an extension to BGP route reflectors. On route reflectors, BGP route selection is modified in order to choose the best route from the standpoint of their clients, rather than from the standpoint of the route reflectors themselves. Depending on the scaling and precision requirements, route selection can be specific for one client, common for a set of clients, or common for all clients of a route reflector. This solution is particularly applicable in deployments using centralized route reflectors, where choosing the best route based on the route reflector's IGP location is suboptimal. This facilitates, for example, a "best exit point" policy ("hot potato routing"). The solution relies upon all route reflectors learning all paths that are eligible for consideration. BGP route selection is performed in the route reflectors based on the IGP cost from configured locations in the link-state IGP. This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. In many environments, a component external to a network is called upon to perform computations based on the network topology and the current state of the connections within the network, including Traffic Engineering (TE) information. This is information typically distributed by IGP routing protocols within the network. This document describes a mechanism by which link-state and TE information can be collected from networks and shared with external components using the BGP routing protocol. This is achieved using a BGP Network Layer Reachability Information (NLRI) encoding format. The mechanism applies to physical and virtual (e.g., tunnel) IGP links. The mechanism described is subject to policy control. Applications of this technique include Application-Layer Traffic Optimization (ALTO) servers and Path Computation Elements (PCEs). This document obsoletes RFC 7752 by completely replacing that document. It makes some small changes and clarifications to the previous specification. This document also obsoletes RFC 9029 by incorporating the updates that it made to RFC 7752. Huawei Technologies Huawei Technologies Arrcus Cisco HPE Arrcus Deutsche Telekom AG Orange Cisco Systems Cisco Systems Steinberg Consulting

Acknowledgments Authors would like to thank Ketan Talaulikar, Jakob Heitz and Lokesh Khanna for their valuable guidance and collaboration on the numerous topics covered in this draft.

Contributors Cisco

slitkows@cisco.com