SRv6 Extensions for RDMA Multicast Delivery

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 1 September 2026.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

Large-scale distributed computing deployments, including data center interconnection, distributed AI training and inference, and national-scale computing networks, rely on high-throughput data transport. RDMA (Remote Direct Memory Access) provides kernel-bypass data transfer with low CPU overhead on both sending and receiving hosts. RDMA one-sided operations, where receive buffers are pre-registered at the receiver's Network Interface Card (NIC), further reduce CPU involvement at the receiving end.¶

Many distributed applications exhibit one-to-many traffic patterns, including multi-replica storage writes, HPC collective communications (broadcast, scatter), AI training parameter distribution, and KV cache distribution in inference pipelines. IP multicast delivery of such traffic can reduce total network bandwidth consumption compared to per-receiver unicast replication at the source.¶

The RDMA Reliable Connection (RC) transport mode is the most widely adopted RDMA mode because it supports the complete set of RDMA operations: Read, Write, and Atomic. However, each RC Queue Pair (QP) is a point-to-point association between exactly one sending QP and one receiving QP. RC packets carry per-connection identifiers in the Base Transport Header (BTH), specifically the Destination Queue Pair Number (QPN) and Packet Sequence Number (PSN). These per-connection fields prevent direct application of IP multicast replication to RC traffic because each receiver requires its own QPN and independently tracks PSN state.¶

Existing application-layer approaches in distributed frameworks (MPI, NCCL, Spark) address this limitation in two ways: by opening separate RC QP connections to each receiver, which results in source bandwidth consumption proportional to the number of receivers, or by constructing application-layer relay trees (tree or ring topologies), which introduce per-hop host-stack traversal latency and additional memory copy overhead at relay nodes.¶

This document specifies SRv6 extensions that bridge the gap between RDMA RC point-to-point semantics and IP multicast one-to-many delivery. Edge nodes of the multicast tree execute a new SRv6 endpoint behavior (End.MT) that rewrites per-receiver RDMA BTH fields in replicated packet copies. Intermediate and edge nodes aggregate reverse-path RDMA response messages (ACK, NACK, CNP) before they reach the source. RDMA RC endpoints are not required to implement any multicast-specific extensions.¶

2. Terminology

This document uses the following terms. Familiarity with SRv6 terminology from [RFC8402], [RFC8754], and [RFC8986] is assumed.¶

RDMA:

Remote Direct Memory Access, as specified in the InfiniBand Architecture [ROCEV2].¶

RoCEv2:

RDMA over Converged Ethernet version 2, an RDMA transport encapsulated in UDP/IPv6 (or UDP/IPv4).¶

RC:

Reliable Connection, an RDMA transport mode providing connection-oriented reliable delivery.¶

QP:

Queue Pair, the RDMA communication endpoint consisting of a Send Queue (SQ) and a Receive Queue (RQ).¶

QPN:

Queue Pair Number, a 24-bit identifier for a QP.¶

BTH:

Base Transport Header, the RDMA transport header containing the opcode, Destination QPN, PSN, and other fields.¶

AETH:

ACK Extended Transport Header, an RDMA header carrying acknowledgment information.¶

PSN:

Packet Sequence Number, a 24-bit sequence number used for ordering and acknowledgment in RDMA reliable transport.¶

ACK:

Acknowledgment, an RDMA response confirming successful reception.¶

NACK:

Negative Acknowledgment, an RDMA response requesting retransmission.¶

CNP:

Congestion Notification Packet, used in the RoCEv2 ECN-based congestion control mechanism.¶

SRv6:

Segment Routing over IPv6 [RFC8402].¶

SRH:

Segment Routing Header [RFC8754].¶

SID:

Segment Identifier, a 128-bit IPv6 address in SRv6.¶

End.MT:

A new SRv6 endpoint behavior defined in this document for RDMA multicast header transformation at edge nodes.¶

Designated QPN:

A QPN value agreed upon by all multicast group participants during group setup, used as the Destination QPN in data packets traversing the multicast tree.¶

Proxy Address:

An IPv6 address used as the common destination by the source and all receivers to represent the multicast group at the RDMA layer.¶

3. Applicability

The extensions specified in this document apply to networks that meet all of the following conditions:¶

1. RDMA transport uses RoCEv2 in Reliable Connection (RC) mode over IPv6.¶
2. The network underlay supports SRv6 as defined in [RFC8754] and [RFC8986].¶
3. IP multicast forwarding is available in the underlay between the source and the edge nodes, using any combination of PIM, static multicast routing, or SRv6 Replication segments [RFC9524].¶
4. Edge nodes are capable of maintaining per-receiver RDMA connection state and performing BTH field modification at line rate.¶

These extensions do not modify RDMA endpoint behavior. Hosts run unmodified RoCEv2 protocol stacks and establish standard RC QP connections. All multicast-related packet transformations occur within the network.¶

4. Architecture

                            S1
                             |
                            N6 (transit)
                          /    \
               (transit) N4     N5 (transit)
                        /  \    /  \
              (edge)  N1   N2    N3 (edge)
                     / \    |   / \
                   R1   R2  R3 R4  R5

5. Data Plane Specification

6. SRv6 End.MT Behavior

When N receives a packet destined to S, where S is a local
End.MT SID, N does:

  S01. If NH=SRH and SL > 0 {
  S02.   Decrement SL
  S03.   Update the IPv6 DA with SRH[SL]         ;; Ref1
  S04.   Parse the End.MT TLV associated with S
  S05.   Let RecvList = list of (IPv6_Addr, QPN) from TLV
  S06.   For each entry (Addr_i, QPN_i) in RecvList {
  S07.     Copy the packet                        ;; Ref2
  S08.     In the copy, set IPv6 DA = Addr_i
  S09.     In the copy, set BTH.DestQPN = QPN_i
  S10.     Recompute ICRC over the modified headers
  S11.     Forward the copy based on Addr_i       ;; Ref3
  S12.   }
  S13. } Else {
  S14.   Drop the packet                          ;; Ref4
  S15. }

Ref1: Standard SRH processing per RFC 8754 Section 4.3.1.1.

Ref2: The copy includes all payload beyond the outer IPv6
      and SRH headers that are relevant to the inner RDMA
      frame.

Ref3: FIB lookup on Addr_i determines the outgoing
      interface.

Ref4: A packet arriving with SL=0 or without SRH is not
      valid for End.MT processing.

7. Packet Formats

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Type     |     Length    |           Reserved            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|          Edge Node Address (128 bits IPv6)                    |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Num Receivers |                  Reserved                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|         Receiver 1 Address (128 bits IPv6)                    |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Receiver 1 QPN (24 bits)            |   Reserved   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~                            ...                                ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|         Receiver N Address (128 bits IPv6)                    |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Receiver N QPN (24 bits)            |   Reserved   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

8. Intermediate Node State Requirements

Each intermediate node (both edge and transit) participating in reverse-path response aggregation MUST maintain the following per-multicast-group state:¶

Per-branch AckPSN:

The most recent AckPSN value received from each downstream branch. Initial value: 0.¶

Per-branch ePSN:

The most recent expected PSN from any NACK received from each downstream branch. For branches that have not sent a NACK, this value SHOULD be set to AckPSN + 1.¶

Per-branch CCount:

A counter of CNP messages received from each downstream branch within the current time window T. Reset to zero at the start of each new time window.¶

The amount of state is proportional to the number of downstream branches at each node, not to the total number of receivers in the multicast group. Edge nodes additionally maintain the receiver information (addresses and QPNs) learned from the End.MT TLV or from control-plane provisioning.¶

9. Security Considerations

The security considerations of [RFC8754] and [RFC8986] apply to all SRv6 aspects of this specification. The following additional considerations are specific to RDMA multicast delivery.¶

The End.MT TLV carries receiver IPv6 addresses and QPNs in the SRH. An on-path attacker able to read SRH contents can obtain receiver topology and RDMA connection identifiers. Implementations operating outside a single administrative trust domain SHOULD protect SRH integrity and confidentiality using the HMAC TLV defined in Section 7 of [RFC8754] or IPsec Encapsulating Security Payload (ESP) encapsulation.¶

Intermediate nodes maintain per-branch ACK/NACK/CNP aggregation state. An attacker injecting forged response messages could corrupt this state, causing the source to prematurely consider data as acknowledged (via inflated AckPSN) or to trigger unnecessary retransmissions (via forged NACKs). Nodes SHOULD validate that reverse-path response messages originate from addresses within the expected downstream receiver set. BCP 38 [RFC2827] ingress filtering SHOULD be applied at network boundaries.¶

An attacker injecting a high volume of forged CNP messages could force the source into continuous rate reduction, creating a denial-of-service condition. Intermediate nodes SHOULD implement per-branch CNP rate limiting. The configurable time window T for CNP aggregation provides an inherent dampening effect.¶

If End.MT TLV contents are modified in transit, packets could be delivered to incorrect RDMA QPs, resulting in data corruption or information disclosure at unintended receivers. The SRH HMAC TLV [RFC8754] provides integrity protection for this purpose. Edge nodes SHOULD verify HMAC before processing End.MT TLVs when operating across trust domain boundaries.¶

10. IANA Considerations

Acknowledgments

The authors thank the members of the SPRING and RTGWG working groups for their review and feedback.¶

Authors' Addresses