UnifiedBus over Ethernet (UBoE)

Introduction UnifiedBus (UB) is a high-performance layered interconnection protocol designed for large-scale AI and HPC clusters, providing ultra-low-latency data transmission and advanced remote memory access capabilities. To enable interworking between the UB domains and legacy standard Ethernet/IP networks, the UBoE protocol is defined to encapsulate native UB upper-layer protocol data within standard IPv4/IPv6 and UDP packets over Ethernet. All UBoE deployment scenarios adopt a unified encapsulation format. Cross-protocol translation between UB and UBoE occurs in UB2E switch based deployment scenarios, where network-layer header adaptation is required to bridge UB-specific network layer metadata and standard IP/Ethernet forwarding semantics. This document illustrates the UBoE encapsulation format, IP/UDP header field value configurations, and ICRC integrity verification rules. It specifies the UB-to-UBoE header adaptation behaviors at UB2E switches, including bidirectional adaptation logic and mapping rules between UB-specific network layer fields (including routing type, congestion markings and traffic isolation partitions) and Eth/IP network layer. It also analyzes the congestion control algorithm selection impacts introduced by semantic translation of congestion markings during packet conversion. This document does not modify the UnifiedBus (UB) specification. While some information regarding UBoE was implicitly scattered throughout UB , this document consolidates such content and supplements it with additional details necessary for UBoE but not explicitly defined in the original specification. It provides a comprehensive and explicit reference to assist the Internet community in understanding and implementing the UBoE endpoints or UB2E switches for interconnecting UB domains and Ethernet/IP domain.

Terminologies The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. CNP: Congestion Notification Packet ECN: Explicit Congestion Notification ICRC: Invariant Cyclic Redundancy Check TPH: Transport Header TAH: Transaction Header UB: UnifiedBus UBoE: UnifiedBus over Ethernet UBPU: UB Processing Unit

UBoE Usage UB system offers a rich set of I/O and memory services based on Unified Remote Memory Access(URMA) and/or load/store-like access methods. In a single UB domain, UB processing units (UBPU) always use native and full stack UB protocol to communicate with each other . However, in some deployments, UBPUs across multiple UB domains want to enjoy such advanced features via an interconnected Ethernet/IP network to better reuse the existing network equipments. Three typical UBoE deployment scenarios are defined in this document, covering endpoint direct encapsulation, single-ended switch conversion, and dual-ended bidirectional switch conversion. All scenarios share the same UBoE encapsulation formats, while switch-based scenarios introduce additional network header conversion and adaptation procedures. depicts the UBoE direct pass-through encapsulation scenario. UBoE-capable endpoints (e.g. UBoE NIC) directly perform UBoE encapsulation, overlaying the UB transport layer and upper-layer protocols directly on top of the standard Ethernet/IP network layer. No intermediate protocol conversion devices are deployed. Cross-domain UB service communication is fully implemented by endpoint-based UBoE encapsulation and decapsulation. All Ethernet/IP intermediate devices forward UBoE packets transparently.

UBoE endpoint: A UBoE capable endpoint where at least one port is configured for UBoE packet exchange, while the rest are configured for native UB and/or other packet types. It can be some processing units.
UBoE NIC: A network interface card (NIC) capable of transmitting and receiving UBoE packets over the network. It is a specific UBoE endpoint.

UBoE Direct Pass-Through Encapsulation Deployment illustrates the single-ended UB2E switch protocol conversion scenario, which introduces a single UB2E switch as the protocol conversion node between UB native domain and Ethernet/IP domain, different from the endpoint-only encapsulation mode in . For any single data flow in this scenario, only unidirectional protocol conversion is supported: either native UB packets from the UB domain are encapsulated into UBoE packets for Ethernet/IP transmission, or UBoE packets from the IP domain are decapsulated into native UB packets to access the local UB domain. Limited by the single-ended conversion mechanism, some UB-specific network header field information is discarded when converting to IP network header.

UBPU: UB Processing Unit, designed to generate, transmit, and receive native UB packets.
UB2E switch: A UB switch with an integrated UB-to-UBoE packet conversion function, enabling seamless interoperability between UB and Ethernet/IP networks.

Single-Ended UB2E Switch Conversion Deployment shows the the dual-ended UB2E switch protocol conversion scenario, deploying UB2E switches at both ingress and egress of the Ethernet/IP domain. Different from the single point conversion in , the ingress UB2E switch converts native UB packets to UBoE packets, while the egress UB2E switch parses UBoE packets and potentially restores them to a more complete native UB packets for peer UB domain access.

Dual-Ended UB2E Switch Conversion Deployment All three deployment scenarios use UBoE encapsulation over standard IP network layers. Detailed specifications on IP and UDP header fields for UBoE packets are defined in . Only (Single-Ended UB2E Switch Conversion) and (Double-Ended UB2E Switch Conversion) deploy UB2E switches to perform bidirectional packet translation between native UB network encapsulation and UBoE encapsulation. The protocol adaptation rules, field mapping logic, and header conversion behaviors of UB2E switches are elaborated in .

UBoE Packet Format and IP Header Fields shows a protocol stack overview of UB and UBoE. UBoE encapsulation wraps the UB transport, transaction and payload data inside the IPv4/IPv6 and UDP headers, forming IP-routable packets that can traverse general Ethernet/IP fabrics.

UB and UBoE Protocol Stack The UBoE frame structure is shown in , used by all three deployment scenarios described in . UB transport header(TPH) follows UDP header in UBoE packet. defines three transport modes, reliable transport (RTP), unreliable transport (UTP) and compact transport (CTP). It also supports transport bypass configuration, meaning no transport header is present. Readers may refer to Section 6 of for details regarding the rationale and method for selecting each transport enablement. UBoE only uses UB RTP or UTP transport mode. TPH is 16 bytes in length in both modes. Compact transport (CTP) mode and transport bypass are not support in UBoE.

IP and UDP Header fields for UBoE Packets UBoE supports both IPv4 and IPv6. The formats of the IPv4 header and IPv6 header are conformant with , , and . and below show the values for relevant fields in IPv4 header and IPv6 header of UBoE packets respectively. UBoE with IPv4 header

Relevant IPv4 Header Field in UBoE	Value
Internet Header Length (IHL)	5
Differentiated Services Codepoint (DSCP)	Set to a proper value for UB traffic.
Explicit Congestion Notification (ECN)	Set to '01' or '10' to indicate that the packet can be marked in the network to indicate congestion ; otherwise, set to '00'. Refer to for field mapping rules when UB2E switch conversion is used.
Total Length	Set to the length of the IPv4 packet in bytes including the IPv4 header and up to and including the ICRC.
Flags	'010'. Don't fragment bit is set.
Fragment Offset	0
Time to Live	Set to a value greater than the network diameter.
Protocol	0x11 for UDP
Source Address	IP address of the source UB end point
Destination Address	IP address of the destination UB end point

UBoE with IPv6 header

Relevant IPv6 Header Field in UBoE	Value
Differentiated Services Codepoint (DSCP)	Set to a proper value for UB traffic.
Explicit Congestion Notification (ECN)	Set to '01' or '10' to indicate that the packet can be marked in the network to indicate congestion ; otherwise, set to '00'. Refer to for field mapping rules when UB2E switch conversion is used.
Payload Length	Set to the length of the IPv6 packet payload starting from the first byte after the IPv6 header up to and including the ICRC.
Next Header	0x11 for UDP
Hop Limit	Set to a value greater than the network diameter.
Source Address	IP address of the source UB end point
Destination Address	IP address of the destination UB end point

shows the UDP header fields for UBoE :

UDP Header for UBoE Packet

Destination Port: IANA has assigned the value 4792 for IP Routable UB UDP port, and this value SHOULD be used by default as the destination UDP port for UBoE. This well-known port is also used to carry UB transport and upper-layer information over UDP over IP-address-compatible UB network layer. This draft only focuses on UBoE.
Source Port: It is recommended that the UDP source port number be used as load balance factor. This is to enable a level of entropy for the traffic load-balancing. The implementation is free to choose a proper granularity for load-balancing, e.g., flow-based or packet-based.
UDP Checksum: It SHOULD be transmitted as zero. When a packet is received with a UDP checksum of zero, it MUST be accepted for decapsulation. Optionally, if the UBoE encapsulating end point includes a non-zero UDP checksum, it MUST be correctly calculated across the entire packet as per . When a UBoE decapsulating end point receives a packet with a non-zero checksum, it MAY choose to verify the checksum value. If it chooses to perform such verification, and the verification fails, the packet MUST be dropped. If the decapsulating destination chooses not to perform the verification, or performs it successfully, the packet MUST be accepted for decapsulation.

ICRC for UBoE Packets UBoE implements a 32-bit ICRC to offer the end-to-end integrity protection of the entire IP packets. Calculation starts with the first byte of the IP header up until and including the last UB Payload byte right before the ICRC field itself. The following variant fields in the IP header and UDP header are replaced with all ones in ICRC calculation/check to ensure the changes to these fields along the way donot affect the calculated ICRC value.

IPv4 for UBoE
- Time to Live
- Header Checksum
- Type of Service (DSCP and ECN)
IPv6 for UBoE
- Traffic Class (DSCP and ECN)
- Flow Label
- Hop Limit
UDP for UBoE
- Checksum

The ICRC detailed calculation rules follow the ICRC section in .

UB and UBoE Conversion and Adaptation As introduced in , (Single-Ended UB2E Switch Conversion) and (Double-Ended UB2E Switch Conversion) leverage UB2E switches to perform the cross-protocol translation between native UB packets and UBoE packets. This section specifies such packet conversion and header adaptation logic. The key of UB/UBoE conversion lies in the conversion of network-layer headers. The native UB protocol stack supports three distinct types of network layer addresses, each with a specific length corresponding to a slightly different network layer header format. Among them, one type carries a standard IP header; therefore, we refer to this specific format as the IP-compliant UB network header. illustrates the logical view of the IP-compliant UB network header. It essentially consists of two parts: the UB-specific network information fields and a standard IPv4/IPv6 header format. The former part is intended to provide richer or more specific information within the native UB domain as follows.

RT (2b): Routing type.
CCI (16b): Congestion control indicator.
NPI (25b): Network partition identifier.

Logical View of the IP-Compliant UB Network Header When interconnecting the UB domain with Eth/IP networks via a switch, only the IP-compliant UB network header is permitted for data packet within the UB domain. This constraint ensures that the UB2E switch can seamlessly extract the IP header from the UB network header, and vice versa. The UB domain MUST employ appropriate configuration, management tools, and software to enforce this requirement.

Conversion and adaption by UB2E switch | UB2E | ------------------> <------------------------------ | switch | <----------------- UB domain +--------+ Eth/IP domain ]]> When forwarding a packet between the UB domain and the Eth/IP domain, as shown in , as the IP-compliant UB network header inherently contains a standard IP header, the switch directly reuses this standard IP header portion during packet conversion. The fields within this preserved IP header SHOULD remain consistent with the rules defined in the tables of . The fundamental structural difference between the two network-layer headers is that the IP-compliant UB network header carries additional UB-specific metadata RT, CCI and NPI. The switch strips these three fields and embed the required information in other standard UBoE fields with adaptation when converting UB packets to UBoE for egress to IP networks, and reconstructs them when converting inbound UBoE packets back to native UB format using some rules. The following subsections break down the adaptation required for RT, CCI and NPI information.

Adaptation of RT (Routing Type) The UB IP-compliant network header includes a 2-bit Routing Type (RT) field that controls the load balancing mode and multipath route selection policy within the UB domain. The RT field settings are defined as follows: Routing Type (RT) field in UB IP-Compliant network Header

RT Value	Load Balancing Mode	Multipath Route Selection Policy
00	Per-flow	All available paths
01	Per-packet	All available paths
10	Per-flow	The Shortest paths only
11	Per-packet	The Shortest paths only

Since the standard IP header and Ethernet frame do not have equivalent fields to carry routing type information, the RT field requires special handling during conversion. UB to Eth/IP Direction: The RT information in the UB network header is discarded. Eth/IP to UB Direction: RT field must be populated. Value set to 10 (Per-flow, Shortest paths) as default. The standard Ethernet routers/switches typically perform per-flow load balancing based on 5-tuple hash and select among the shortest paths, this default value aligns with that behavior, so that the UB domain will not misinterpret how the packet is forwarded in the Eth/IP domain. The RT field MAY be configurable by network administrators to accommodate specific deployment requirements for Eth/IP to UB Direction. However, the default value of 10 is RECOMMENDED for most deployments to ensure consistent behavior with standard Ethernet switching.

Adaptation of CCI (Congestion Control Indicator) UB 2.0 defines multiple CCI modes for congestion indicators in the UB domain. Only the FECN mode supports compatible mapping with standard IP ECN natively, which is mandatory for UB/UBoE interworking via UB2E switches. summarizes the CCI modes and their suitability for UB to UBoE conversion network layer. CCI Mode Suitability for UB-to-UBoE by Switch

CCI Mode	Description	Suitability	Reason
CAQM	Active queue management with bandwidth control	No	Relies on richer UB-specific fields like Hint or Increase for bandwidth control, unmapable to standard IP ECN
FECN	Forward ECN with severity levels	Yes	The 2-bit FECN field naturally maps to the 2-bit IP ECN field, aligning with standard mechanisms with some adaptations.
FECN_RTT	FECN with Timestamp for	No	Requires a Timestamp field for RTT measurement, which is not available in standard IP ECN.

As shown in , the UB endpoints SHOULD configure the CCI field to use the FECN mode (CCI.Mode = 3'b100). This ensures that the congestion information can be seamlessly mapped to the standard IP ECN field, allowing intermediate standard Ethernet switches to correctly recognize and process congestion markings. If the other CCI mode is configured, the administrator need ensure that the switch implements the compatible protocol extension in Eth/IP domain.

FECN and ECN Semantic Comparison and Mapping Rules The IP ECN field defined in uses a 2-bit codepoint to indicate congestion. While UB FECN also uses a 2-bit field for congestion marking, their semantic definitions and the amount of information carried differ. summarizes the semantic comparison between the two. CE: Congestion Experienced Semantic Comparison of UB FECN and IP ECN Codepoints

2-bit (F)ECN value	UB FECN Meaning	IP ECN Meaning [RFC3168]	Semantic Equivalence
00	Unmarkable	Not-ECT	Yes
01	Light CE	ECT(1), ECN-Capable	No. Note: Various usages exist for ECT(1), some are proprietary.
10	Markable	ECT(0), ECN-Capable	Yes
11	Severe CE	CE	Yes

When converting packets between UB and Eth/IP domains, the UB2E switch performs bidirectional mapping between the 2-bit FECN field in UB FECN CCI mode and the 2-bit IP ECN field. The detailed mapping rules are as follows: Congestion Experienced (CE) Mapping:

FECN to ECN: The mapping behavior for Light CE (01) and Severe CE (11) in UB FECN to the CE (11) codepoint in IP ECN is configurable. The switch can be configured to map both Light CE and Severe CE to IP ECN-CE, or to map only Severe CE to IP ECN-CE (while mapping Light CE to an ECN-Capable state). The default to map both to IP ECN-CE. When both are mapped to IP ECN-CE, the severity distinction is lost during this conversion, and both are treated as equivalent to the standard CE state.
ECN to FECN: The CE (11) codepoint in IP ECN can be uniformly configured to map to either Light CE (01) or Severe CE (11) in UB FECN. The default is mapping to Severe CE (11).

ECN-Capable / Markable Mapping:

FECN to ECN: The Markable (10) state in UB FECN can be configured to map to either ECT(0) (10) or ECT(1) (01) in IP ECN. The default mapping is ECT(0) (10).
ECN to FECN: Compliant with , both ECT(0) (10) and ECT(1) (01) in IP ECN are mapped to the Markable state (10) in UB FECN by default.

Some proprietary implementations utilize the IP ECN ECT(1) (01) codepoint to indicate a lower severity of congestion, which is equivalent to Light CE (01) in UB FECN. Then a strict one-to-one semantic mapping can be established between UB FECN and IP ECN codepoints. Handling of Additional UB Congestion Related Information: Beyond the 2-bit congestion marking, the UB FECN CCI mode contains additional congestion information, i.e. the Location of Congestion (LoC) bit. Since the standard IP ECN field has no equivalent mechanism to carry this extra information, the LoC bit in the FECN CCI mode will be discarded when converting a packet to the Eth/IP domain. Conversely, when converting from the Eth/IP domain to the UB domain, LoC bit is restored to 0 by default, indicating congestion, if any, occurs at an intermediate switch .

Impact on Transport Congestion Control Loop FECN/ECN mapping at network-layer affects the end-to-end UB transport congestion control loop, which relies on receiver-side congestion marking feedback to trigger sender rate adjustment. As specified in , the endpoint is restricted to use congestion control algorithms based on FECN CCI mode for switch based UBoE scenarios, while CAQM and FECN_RTT CCI modes are not mappable without specific extensions in congestion marking. The native UB protocol supports multiple customizable congestion control algorithms, including Low Delay Control Protocol (LDCP), Confined Active Queue Management (CAQM), and DCQCN-like mechanisms. CAQM is not viable for UB/UBoE conversion scenario without specific extension of CAQM CCI mode. Therefore LDCP and DCQCN-like mechanisms are applicable in UB/UBoE interconnection scenarios. Two types of receivers exist in switch-based UBoE deployments, UBoE endpoints (direct IP termination) and native UB endpoints behind UB2E switches. UBoE endpoints directly use IP ECN-CE as the congestion signal, while native UB endpoints behind UB2E switches use the restored UB FECN markings. Receivers adopt whichever valid congestion marking signal is available according to their deployment type. The DCQCN-like Algorithm in UB uses the Congestion Notification Packet (CNP) to echo back the congestion markings and optionally supports two congestion levels, namely light congestion and severe congestion. Therefore, if a receiver can only get single congestion level based IP ECN-CE in UBoE scenarios, it shall either disable the two-level congestion reporting in CNP feedback or treat all ECN-CE markings as equivalent to severe congestion in CNP.

Adaptation of NPI (Network Partition Identifier) The UB IP-compliant network header includes a 25-bit Network Partition Identifier (NPI) field to enforce logical traffic isolation among network partitions in UB domains. shows the NPI field format.

NPI field format in UB IP-Compliant network Header

Permission bit (1 bit): The most significant bit indicating endpoint privilege level. 0 indicates high privilege, 1 indicates low privilege.
Partition ID (24 bits): Identifies the network partition. ID value 0 is invalid, and ID value 1 represents the default partition.

In native UB domains, endpoints belonging to different Partition IDs cannot intercommunicate. For endpoints sharing the same Partition ID, the Permission bit provides secondary fine-grained access control: communication is permitted for high-to-high and high-to-low privileged pairs, while low-to-low communication is prohibited. Notably, UB switches only validate the Partition ID during forwarding and do not check the Permission bit; privilege is enforced at UB endpoints. Both UB NPI and Ethernet VLAN mechanisms provide traffic isolation, these two fields are mapped at the UB2E switch in UB/UBoE conversion. Since standard Ethernet VLAN has no equivalent privilege concept, the permission bit is discarded in baseline UB-to-UBoE conversion and set to high-privilege (0) by default in the reverse direction. The detailed mapping rules at UB2E switch are defined as follows: UB to Eth/IP Direction:

The 24-bit Partition ID is mapped to the 12-bit VLAN ID. The mapping takes the lower 12 bits of the Partition ID. If the Partition ID exceeds 12 bits (4095), the upper bits are truncated, and administrators MUST ensure that Partition IDs in the same UB domain do not collide after truncation.
The 1-bit Permission bit is removed during UB-to-UBoE translation and is not carried in Ethernet/IP frames.
UB2E switches support configurable NPI-to-VLAN translation.

Eth/IP to UB Direction:

The 12-bit VLAN ID from the Ethernet frame is mapped to the lower 12 bits of the 24-bit Partition ID in the UB domain. The upper 12 bits of the Partition ID are set to 0 by default.
The Permission bit is set to 0 (high privilege) by default, allowing maximum compatibility and communication capability.
UB2E switches support configurable VLAN-to-NPI translation.

UB2E switches in (Double-Ended UB2E Switch Conversion) may support an optional enhanced mapping to restore the original Permission bit information instead of to use the default. In this optional mode, the ingress UB2E switch maps one Partition ID with different Permission bit values (0/1) to two distinct VLAN IDs. The egress UB2E switch restores both the original Partition ID and the original Permission bit from the VLAN ID. However it consumes more VLAN IDs and has more complex configurations. Administrators can enable it based on deployment requirements.

Security Considerations UBoE inherits all security constraints and isolation mechanisms defined in the UB 2.0 base specification . Encapsulating UB upperlayer data payload in standard UDP/IP packets does not introduce new security vulnerabilities. Standard IP security practices, including access control lists, VLAN isolation, and traffic filtering, can be deployed to protect UBoE cross-domain traffic. Misconfiguration of NPI-to-VLAN mapping rules may cause partition isolation or privilege control anomalies. Operators shall strictly manage UB2E switch mapping policies and partition allocation to avoid cross-domain unauthorized traffic leakage. No new security extensions or threats specific to UBoE are introduced in this specification.

IANA Considerations A well-known UDP port (4792) has been assigned by the IANA in the Service Name and Transport Protocol Port Number Registry for IP Routable UB. UBoE uses this UDP port number.

Acknowledgements Thanks to Liudong Xiong and Chuanning Cheng for their review and comments. Thanks to Wei Pan for sharing his knowledge on the registered UDP port (4792).