Multi-level Congestion Response Framework with Long-haul Congestion Notification for DCI Networks

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 31 August 2026.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

RDMA over Converged Ethernet v2 (RoCEv2) is widely deployed in data center networks for high-performance computing and AI training workloads. Within a single data center, congestion control mechanisms such as DCQCN [DCQCN] and ECN-based schemes provide effective flow control. However, when RoCEv2 traffic traverses Data Center Interconnect (DCI) wide-area networks, the existing congestion notification path ("switch marks ECN, receiver generates CNP, CNP returns to sender") introduces feedback latency proportional to the WAN round-trip time, which can reach tens of milliseconds.¶

Recent work on Fast CNP [I-D.xiao-rtgwg-rocev2-fast-cnp] has addressed the fundamental latency issue by enabling switches to generate CNP packets directly to the sender. This document builds upon that foundation by specifying three complementary mechanisms that are particularly relevant for long-haul DCI scenarios:¶

1. A graduated multi-level trigger framework that distinguishes between incipient and severe congestion, avoiding unnecessary control packet generation for transient queue buildup.¶
2. A BDP-adaptive dynamic threshold calculation that automatically adjusts congestion detection sensitivity based on link characteristics (bandwidth, distance, RTT).¶
3. A multi-device collaborative suppression mechanism that coordinates congestion responses across multiple intermediate nodes on a data flow path, preventing redundant or conflicting control instructions.¶

Additionally, this document defines an extended packet format (Long-haul CNP) that carries explicit control instructions with quantified congestion metrics, enabling the source to perform precise rate adjustments rather than relying on generic rate reduction heuristics.¶

2. Terminology

RDMA:

Remote Direct Memory Access.¶

RoCEv2:

RDMA over Converged Ethernet version 2 [RoCEv2].¶

CNP:

Congestion Notification Packet, as defined in the RoCEv2 specification. The standard CNP uses BTH Opcode 0x81.¶

Fast CNP:

Fast Congestion Notification Packet, a switch-originated CNP as defined in [I-D.xiao-rtgwg-rocev2-fast-cnp].¶

Long-haul CNP:

Long-haul Congestion Notification Packet, the extended packet type defined in this document, carrying explicit control instructions and congestion metrics.¶

ECN:

Explicit Congestion Notification [RFC3168].¶

QP:

Queue Pair, a fundamental RDMA communication abstraction.¶

BTH:

Base Transport Header, the common header in all RoCEv2 packets.¶

RTT:

Round-Trip Time.¶

PFC:

Priority-based Flow Control.¶

DCI:

Data Center Interconnect.¶

BDP:

Bandwidth-Delay Product.¶

Congestion-Aware Intermediate Node:

A network device deployed on the DCI path that implements the multi-level congestion monitoring and Long-haul CNP generation functions specified in this document. This may be a router, switch, or dedicated DCI gateway device.¶

3. Related Work and Positioning

This section describes the relationship between this document and existing congestion notification mechanisms for RoCEv2 networks.¶

[I-D.xiao-rtgwg-rocev2-fast-cnp] defines the Fast CNP mechanism, which enables a switch to generate a CNP packet directly to the sender when it detects congestion, without waiting for the receiver to generate the CNP. Fast CNP provides the foundational mechanism for switch-originated congestion notification.¶

This document extends the Fast CNP concept in three directions:¶

Graduated Response:

While Fast CNP triggers upon detecting congestion, this document defines a two-level response where lightweight ECN marking handles incipient congestion and Long-haul CNP generation is reserved for severe or rapidly worsening conditions. This reduces control packet overhead in mildly congested scenarios.¶

Instructional Control:

While Fast CNP advises the sender to reduce its rate, the Long-haul CNP packet format defined here carries explicit action codes (notify, pause, rate reduce, resume) with quantified parameters (e.g., reduction percentage, pause duration) and congestion metrics, enabling more precise source-side rate adjustment.¶

Multi-device Coordination:

In DCI scenarios where a flow traverses multiple congestion-aware nodes, this document defines coordination rules to prevent duplicate or conflicting control instructions from reaching the same source.¶

The mechanisms defined in this document are complementary to SRv6-based congestion control approaches such as those described in [I-D.liu-spring-srv6-cc] and [I-D.hu-rtgwg-rocev2-fcn]. When used within SRv6-based DCI networks [RFC8754], the Long-haul CNP can be encapsulated within the applicable SRv6 transport framework.¶

RTT-based congestion control approaches such as TIMELY [TIMELY] provide an alternative signal (delay) for inferring congestion severity; the Congestion Metric field defined in this document can optionally carry RTT-derived information to complement queue-based metrics.¶

4. Applicability Statement

The mechanisms specified in this document are primarily designed for Data Center Interconnect (DCI) scenarios where RoCEv2 traffic traverses wide-area network paths with non-trivial propagation delay (typically RTT greater than 1 ms). In such environments, the receiver-mediated CNP feedback path introduces significant latency, and the BDP-adaptive threshold mechanism provides meaningful dynamic range.¶

For intra-data-center deployments where RTT is sub-millisecond and paths traverse few hops, the standard ECN/CNP or Fast CNP mechanisms are typically sufficient, and the additional complexity of the multi-level framework may not be warranted.¶

The multi-device collaborative suppression mechanism is most beneficial when data flows traverse two or more congestion-aware intermediate nodes, which is common in multi-hop DCI topologies.¶

Regarding IP version applicability: the ICMPv6 packet format defined in Section 6.1 is applicable only to IPv6 network deployments. For DCI environments that operate over IPv4, implementations MUST use the RoCEv2 backward-compatible extension format defined in Section 6.2. A future document may define an ICMPv4-based format if there is sufficient demand for ICMP-based Long-haul CNP in IPv4-only deployments.¶

5. Protocol Specification

  +------+     +------+                   +------+     +------+
  |Source|---->| Node |======WAN Path=====>| Node |---->| Dest |
  | NIC  |    |  N1  |                    |  N2  |    | NIC  |
  +------+    +------+                    +------+    +------+
     ^            |                           |
     |            | Long-haul CNP             |
     +------------+ (unicast to source)       |
     |                                        |
     +----------------------------------------+
       Long-haul CNP (if N1 control insufficient)

   K_max = max(K_base, alpha * R_port * RTT_est / 8)

6. Packet Formats

This document defines two Long-haul CNP packet formats. Implementations MUST support at least one format and SHOULD indicate the supported format(s) through out-of-band configuration or capability exchange.¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Type      |     Code      |           Checksum            |
+---------------+---------------+-------------------------------+
|  Cong. Level  | Action Flags  |          Parameter            |
+---------------+---------------+-------------------------------+
|                      Source QP Number                         |
+---------------------------------------------------------------+
|  Metric Type  |         Congestion Metric Value (24 bits)     |
+---------------+-----------------------------------------------+
|          Extension Header (optional, see below)               |
+---------------------------------------------------------------+
|          Extension Object(s) (optional, variable)             |
~                                                               ~
+---------------------------------------------------------------+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Version (4)  |   Reserved    |         Checksum              |
+---------------+---------------+-------------------------------+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Length             |  Class-Num    |    C-Type     |
+-------------------------------+---------------+---------------+
|                  Object Payload (variable)                    |
~                                                               ~
+---------------------------------------------------------------+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    MAC Header (112 bits)                      |
+---------------------------------------------------------------+
|              IPv4/IPv6 Header (160/320 bits)                  |
+---------------------------------------------------------------+
|               UDP Header (64 bits, DstPort=4791)              |
+---------------------------------------------------------------+
 BTH (Base Transport Header, 96 bits):
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| OpCode (0x81) |S|M|Pad| TVer  |       Partition Key           |
+---------------+-+-+---+-------+-------------------------------+
|F|B|E| RSV(5b) |              DestQP (24 bits)                 |
+-+-+-+-+-+-+-+-+-----------------------------------------------+
|A|  RSV (7b)   |              PSN (24 bits)                    |
+-+-------------+-----------------------------------------------+
 Long-haul CNP Extension Fields (present when E=1):
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Cong. Level   | Action Flags  |         Parameter             |
+---------------+---------------+-------------------------------+
|                   Source QP Number (32 bits)                   |
+---------------------------------------------------------------+
|  Metric Type  |     Congestion Metric Value (24 bits)         |
+---------------+-----------------------------------------------+
|                   Optional Extension Objects (variable)        |
~                                                               ~
+---------------------------------------------------------------+
|                      ICRC (32 bits)                           |
+---------------------------------------------------------------+
|                      FCS  (32 bits)                           |
+---------------------------------------------------------------+

7. Operational Example

Consider a DCI path: Source (DC-A) -> N1 -> N2 -> Dest (DC-B), where N1 and N2 are congestion-aware intermediate nodes, and the WAN RTT is 10 ms.¶

  +--------+    +----+    +----+    +--------+
  | Source  |--->| N1 |--->| N2 |--->|  Dest  |
  | DC-A   |    |    |    |    |    |  DC-B  |
  +--------+    +----+    +----+    +--------+
     10.0.0.1            10.0.0.4
     QP=100              QP=200

1. N1 learns the flow: {Src=10.0.0.1, Dst=10.0.0.4, SrcQP=100, DstQP=200}.¶
2. N1 calculates K_max for its egress port (100 Gbps link): K_max = max(64KB, 1.0 * 100e9 * 0.010 / 8) = max(64KB, 125MB) = 125 MB. K_min = 62.5 MB.¶
3. Queue depth at N1 reaches 70 MB (exceeds K_min). N1 applies ECN marking to traversing packets (first-level response).¶
4. Queue depth continues to grow to 130 MB (exceeds K_max). N1 generates a Long-haul CNP: {Action=Rate Reduce, Parameter=30, SourceQP=100, CongLevel=180, MetricType=1 (Queue Depth in KB), MetricValue=130000} and unicasts it to 10.0.0.1. In RoCEv2 format, the BTH uses Opcode=0x81, BECN=1, E=1, DestQP=100.¶
5. Source receives the Long-haul CNP within approximately 5 ms (half RTT). If the source supports Long-haul CNP, it checks E=1 and parses the extension fields, reducing QP 100's sending rate by 30%. If the source is a legacy RNIC that does not recognize the E bit, it processes the packet as a standard CNP and applies its default rate reduction algorithm (see Section 8 for frame length considerations).¶
6. N2 also detects mild congestion but observes that the flow arrival rate from N1 has decreased. N2 defers Long-haul CNP generation.¶
7. After 20 ms without new Long-haul CNP, the source begins additive rate recovery on QP 100.¶
8. N1 observes queue depth has fallen below K_min for more than 10 ms (1 * RTT_est). N1 generates a Long-haul CNP: {Action=Resume, Parameter=50, SourceQP=100, CongLevel=20, MetricType=1, MetricValue=30000}. The source increases QP 100's rate by 50% of the previously applied reduction.¶

8. Backward Compatibility

The Long-haul CNP mechanism is designed for incremental deployment:¶

Non-supporting intermediate nodes:

Intermediate nodes that do not implement this specification simply forward data packets without generating Long-haul CNP. The standard ECN/CNP feedback loop continues to operate as the baseline congestion control mechanism.¶

Non-supporting sources (ICMPv6 format):

If a source receives an ICMPv6 Long-haul CNP but does not recognize the Type value, it MUST process it according to standard ICMPv6 unknown-type handling rules [RFC4443]. For informational messages (Type values 128-255), the unknown message is silently discarded.¶

Non-supporting sources (RoCEv2 format):

Because the Long-haul CNP in RoCEv2 format uses the standard CNP Opcode (0x81) with all mandatory BTH fields set to their standard CNP values, a legacy RNIC that does not recognize the E bit will process the packet as a standard CNP. The legacy RNIC will ignore the extension fields (which appear after the expected CNP boundary) and apply its default rate reduction behavior. This provides a graceful degradation path: precise control for supporting endpoints, and standard CNP rate reduction for legacy endpoints.¶

However, the Long-haul CNP frame is larger than a standard CNP frame due to the extension fields (at least 12 additional octets for the base extension, plus any Optional Extension Objects). Legacy RNIC implementations that perform strict frame length validation against the expected standard CNP size may reject the Long-haul CNP packet. Deployments SHOULD verify that legacy endpoints in the network tolerate CNP frames with additional trailing data beyond the standard BTH before enabling the RoCEv2 Long-haul CNP format. In environments where legacy endpoints are known to perform strict length validation, the ICMPv6 format SHOULD be used instead, or all endpoints should be upgraded to support the Long-haul CNP extension.¶

Mixed deployment:

In networks where only some intermediate nodes support this specification, the supporting nodes generate Long-haul CNP while non-supporting nodes rely on ECN marking alone. The multi-level framework degrades gracefully to standard ECN/CNP behavior in portions of the path without Long-haul CNP capability.¶

9. Security Considerations

Long-haul CNP packets carry control instructions that directly affect the source's sending behavior. The following security considerations apply:¶

Packet Forgery:

A malicious entity could forge Long-haul CNP packets to cause a source to reduce its rate or pause transmission, resulting in denial of service. To mitigate this, sources SHOULD validate that the IP source address of received Long-haul CNP packets belongs to a configured set of known congestion-aware intermediate node addresses. For the ICMPv6 format, IPsec Authentication Header (AH) MAY be used to provide packet authentication. For the RoCEv2 format, the ICRC provides integrity protection but not authentication; additional authentication mechanisms are RECOMMENDED in security-sensitive deployments.¶

Amplification:

A single congested packet could potentially trigger Long-haul CNP generation targeting multiple sources. Congestion-aware intermediate nodes MUST rate-limit Long-haul CNP generation on a per-flow basis (RECOMMENDED minimum interval: RTT_est per flow) and MUST impose a global rate limit on total Long-haul CNP output per port.¶

Information Disclosure:

The Congestion Metric Value field reveals internal queue state information. In deployments where this is considered sensitive, both the Metric Type and Congestion Metric Value fields MUST be set to zero while still providing actionable control via the Action Flags and Parameter fields.¶

Reserved Bit Manipulation (RoCEv2 format):

A malicious entity that can modify packets in transit could set the E bit on standard CNP packets and append forged extension fields. The ICRC field provides integrity protection against in-transit modification for RoCEv2 packets. Additionally, sources SHOULD validate that the combination of extension field values is consistent before applying control actions.¶

Resume Action Abuse:

A malicious entity could forge Long-haul CNP packets with the Resume action to cause a source to prematurely increase its sending rate during actual congestion. The source address validation described under Packet Forgery above mitigates this risk. Additionally, sources SHOULD apply a maximum rate increase cap per Resume action to limit the impact of any single forged Resume instruction.¶

10. IANA Considerations

Authors' Addresses