China Mobile

Beijing 100053 China liupengyjy@chinamobile.com

Huawei

Nanjing 210012 China liyizhou@huawei.com

Futurewei Technologies USA

Santa Clara 95014 United States of America tte@cs.fau.de

ZTE Corporation

Wuhan 430223 China xiong.quan@zte.com.cn

ETRI

Daejeon 34129 South Korea ryoo@etri.re.kr

New H3C Technologies

Beijing 100094 China zhushiyin@h3c.com

Huawei

Beijing 100095 China gengxuesong@huawei.com

Networking Deterministic Networking Working Group Scaling; Deterministic; Requirement To support the scaling of deterministic networks, this document describes the technical and operational requirements for networks that exhibit large hop-to-hop latency variation, a large number of flows, and/or multiple domains that do not share a common time source. Applications with varying levels of determinism coexist and are transported in such a network. This document also describes the corresponding Deterministic Networking (DetNet) data plane enhancement requirements.

Packet networks are evolving from bandwidth-guaranteed Quality of Service (QoS) to latency-guaranteed QoS, which ensures both bounded and definite latency. Bounded latency and definite latency can be interpreted as in-time delivery (a packet arrives within a predetermined time) and and on-time delivery (a packet arrives exactly at a predetermined time). In addition, network survivability, which has traditionally guaranteed traffic recovery within 50 ms in the event of a network failure, is evolving toward lossless recovery. To support this evolution of QoS and network survivability, Time-Sensitive Networking (TSN) and Deterministic Networking (DetNet) technologies are considered essential. TSN, a set of standards developed by the IEEE 802.1 TSN Task Group (TG) , specifies mechanisms and protocols necessary to realize highly available IEEE 802.1 networks with bounded latency to carry time-sensitive, real-time application traffic. DetNet, whose architecture is defined in RFC 8655 , supports the transport of specified unicast or multicast data flows for real-time applications with extremely low data loss rates and bounded latency, operating under a single administrative control or within a closed group of administrative control. The overall framework for the DetNet data plane is described in , and several documents have been standardized for various data plane technologies and their interworking, extending TSN's service capabilities to IP (Internet Protocol) and MPLS (Multi-Protocol Label Switching) networks. As deterministic networks scale or span multiple interconnected domains, DetNet solutions must consider one or more of the following aspects:

• Clock synchronization across different domains may be relaxed or entirely absent. (Section 3.1)
• The end-to-end path may comprise both low- and high-latency hops. (Section 3.2)
• Various transmission rates may be supported across different ports and network nodes. (Section 3.3)
• A large number of flows may make per-flow state identification difficult and lead to high bandwidth utilization. (Section 3.4)
• Topology changes and link or node failures may occur more frequently. (Section 3.5)
• Flow fluctuations caused by the large number of flows may occur frequently. (Section 3.6)
• The end-to-end path may include a large number of hops and involve complex topologies. (Section 3.7)
• Mechanisms used to ensure bounded latency (e.g., queuing methods) may vary or be differently configured across multiple domains. (Section 3.8)

Such domains are typically under a single administrative control or consist of multiple cooperating administrative networks within a closed group . However, they may belong to different Autonomous System (AS) domains and be controlled by multiple peering or cascaded controllers, and they may not share the same time source. Maintaining per-flow status becomes impractical in a large-scale network. Aggregation and disaggregation of flows occur both at the boundaries and within DetNet domains, governed by various rules. Flow identification and packet treatment should address individual or combined changes introduced by the scaling of deterministic networks. Based on the considerations above, this document describes the requirements for scaling deterministic networks. Scaling deterministic networks demands enhancements to the existing bounded latency mechanisms and the corresponding data plane in order to ensure DetNet service across a single administrative network or multiple cooperating administrative networks, as defined in the DetNet charter. Deterministic networking for the open Internet is outside the current scope.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. While and describe interpretations of these key words in terms of protocol specifications and implementations, they are used in this document to describe technical and operational requirements to support the scaling of deterministic networks.

Given the attributes described in , the corresponding technical requirements should be taken into account.

A deterministic network may span multiple networks with one or more cooperating administrative domains. The presence of diverse network nodes in these domains can result in disparate local time variations, which require tolerance for time asynchrony.

One of DetNet's objectives is to stitch TSN islands together. All devices within a TSN domain are time-synchronized, and most TSN technologies rely on precise time synchronization . However, different TSN islands may operate with unsynchronized clocks, as shown in Figure 1, where the time difference between two TSN domains is denoted as D. DetNet needs to connect these two TSN domains and provide end-to-end deterministic latency services. The mechanism adopted by a deterministic network MUST be prepared to traverse multiple unsynchronized TSN domains. This can be achieved, for example, by adding extra buffer space at the ingress of a new domain, increasing the dead time as a guard band , or using some timing compensation mechanisms. This document does not intend to provide an exhaustive list of such methods.

: ]]>

Within a single time synchronization domain, varying levels of clock accuracy are expected. For example, the crystal oscillator used in Ethernet is specified at 100 ppm , Synchronous Ethernet (SyncE) can achieve 50 ppb , and even higher timing accuracy is expected in 5G mobile backhaul . These clocks experience different levels of jitter and wander, which may result in different levels of path asymmetry. Deterministic networks SHOULD be capable of compensating for or absorbing such time variations within a domain.

It is usually difficult to achieve full time synchronization as the scale of the networks increases. Some networks, such as mobile backhaul, use frequency synchronization (e.g., SyncE) instead of strict time synchronization. The same deterministic performance in terms of bounded latency and jitter SHOULD be achieved even when full time synchronization is not available, i.e., when only partial synchronization is in use, with SyncE being one example.

A deterministic network can contain a large number of traffic flows, some of which are non-periodic. Asynchronous methods can meet the requirements of such traffic flows. Moreover, mechanisms that do not require time and/or frequency synchronization can reduce hardware cost and complexity at network nodes. conceptually introduces per-flow-based asynchronous shaper to achieve bounded latency. The effectiveness of the per-flow-based asynchronous shaper can be demonstrated through mathematical analysis. It naturally tolerates time variation, but raises concerns about per-flow state buffer management. When it is in use, the requirement in Section 3.3 SHOULD be carefully considered.

In some deterministic networks, a single hop can introduce significant latency. Optical transmission in fiber travels at approximately 200 km/ms. Thus, the propagation delay of a single hop can be on the order of a few milliseconds. This is much greater than the single hop propagation delays found in typical Local Area Networks (LANs), and can have a substantial impact on queuing mechanisms, such as cyclic or time-aware scheduling methods. Therefore, propagation delay must be taken into account in end-to-end computations. For example, it should be considered when setting the period in both time- and frequency-synchronized methods, or when setting the extra buffer in the asynchronous methods, to meet the requirements of deterministic forwarding between the network nodes. Here, we use an example to illustrate the impact of large single-hop propagation delay on cycle-based methods, without proposing any specific solution. In a cycle-based method, suppose a deterministic network aims to maintain a simple cycle mapping relationship, but the link distance between two nodes is relatively long. Moreover, a downstream node may have multiple upstream nodes with different link propagation delays (e.g., 9 us, 10 us, 11 us, 15 us and 20 us). In order to absorb the longest link propagation delay, the length of the cycle must be set to more than 20 us. In , propagation delay is part of the dead time imposed within a cycle, which negatively affects bandwidth utilization. However, since packet arrival times can vary within the receiving cycle, a longer cycle leads to greater delay variation.

(in us) 0 |<-->| 10 20 link propagation delay ]]>

Note that Figure 2 is provided solely as an example to illustrate latency caused by large single-hop propagation. CQF is not limited to two queues. Instead, using more than two queues can be an option to address latency-related concerns associated with long links. provides a more detailed discussion of this problem and proposes a mechanism to address it.

A deterministic network can use higher-speed links, especially for its backbone. At the time of publication, deterministic mechanisms in local networks typically operate at link speeds of 10 Mbps, 1 Gbps, or possibly 10 Gbps. In contrast, data rates of 10Gbps, 100Gbps, 400Gbps, and even higher are commonly used in wide area networks. With increasing data rates, the network scheduling cycle can be shortened if the same amount of data is required to be sent in each cycle for each application. Alternatively, more data can be sent if the cycle time remains the same. The former case requires more precise time control, such as cycles on the order of a few microseconds or even sub-microseconds, for the input stream gate and the timed output buffer. The latter requires more buffer space, which imposes more complex buffer and queue management and greater memory consumption. Another aspect to consider is the aggregation of flows. In a deterministic network, the number of flows can reach hundreds or even tens of thousands. These flows can be aggregated into a small number of deterministic paths or tunnels. It is practical to maintain a small amount of flow-based or aggregated-flow-based state in local networks. However, in higher-speed and larger-scale networks, this becomes much less feasible. If is in use, it requires more buffers compared to other mechanisms that are fully or partially time-synchronized. Therefore, further optimization is needed to support higher link speeds.

A deterministic network may carry a large number of traffic flows. According to , per-flow state identification in the network becomes infeasible as flow count scales up. As a result, identifying individual IP flows in the DetNet data plane or configuring per-flow state at each node becomes nearly impossible at scale. DetNet supports flow aggregation. However, in large-scale networks, individual flows may dynamically join or leave aggregated flows, which causes instability in the identification of these aggregated DetNet flows. Wildcards and value ranges used for flow identification may need to be adjusted to ensure that aggregated flows maintain consistent deterministic characteristics. For scaling networks, proper provisioning at the control plane is required to accommodate higher levels of aggregation. Provisioning on aggregated flows normally improves scalability, but at the cost of coarse-grained filtering and protection of incoming traffic. It is desirable that adding a flow to the network does not affect the latency of existing flows or require complex recalculations, and instead requires as little work as possible. For instance, filtering and policing configurations are applied only at ingress nodes, not at intermediate ones. uses traffic classes to divide flows, typically limited to eight. As a result, the forwarding mechanism itself remains relatively simple even with a large number of flows or a higher degree of aggregation. However, when new flows are added, the Gate Control List may need to be updated, and the associated recalculation can be complex. There may be methods to simplify the calculation or configuration, but additional work is needed to enhance them. Meanwhile, traffic requiring deterministic networking can significantly utilize the capacity of a link, or the portion of the link that is dedicated to such traffic, for example, exceeding 75% or even approaching full (near 100%) utilization. Typically, resources required for DetNet are reserved. However, over-provisioning of link capacity may not work in such cases. To guarantee deterministic latency and jitter in such environments, it is preferable to provide scalable queuing solutions that improve bandwidth utilization, based on existing methods, including TSN and other published standards. For instance, when the bandwidth utilization is high, the guard band in each cycle in is a type of over-provisioning and can be optimized through more scalable queuing enhancements.

Deterministic networks may involve a greater number of network devices, and the addition or removal of such devices tends to occur more frequently than in LANs. A representative use case is ultra-low-latency public 5G transport networks, which would require DetNet to extend to every 5G base station. For some network operators, their networks may need to connect approximately 100,000 base stations, serving multiple mobile network operators. This number is expected to increase further as 5G deployment continues. On the one hand, a large number of network devices may increase the likelihood of network link failures. Path switching or routing reconvergence can result in significant packet loss and retransmission delays, often lasting several seconds before the network stabilizes. As the delays involved are too large to be mitigated by jitter compensation techniques, it is necessary to support mechanisms that can adapt to link or node failures and topology changes. On the other hand, changes in the number of devices may affect the implementation and adjustment of deterministic networking mechanisms, such as topology discovery, queuing mechanisms, and packet replication and elimination. For instance, having multiple fully disjoint paths when implementing the Packet Replication, Elimination, and Ordering Functions (PREOF) increases survivability in the event of a node or link failure on any path. However, this also introduces challenges in finding paths with similar lengths and/or hop counts, ensuring sufficient buffer space to absorb latency differences between paths at scale. Therefore, a method is needed to support flexible multi-path planning and provide a reliable foundation for implementing PREOF.

A wider variety of DetNet traffic flows described in Section 3.4 will lead to more frequent dynamic joining and leaving of flows. This, in turn, increases flow fluctuations and the overall unpredictability of DetNet traffic. Examples include the following:

• Various high-volume traffic flows of different applications in scaling networks easily cause more bursty traffic.
• An increasing number of aggregation nodes receive flows from more upstream nodes, introducing additional nondeterministic delay in packet handling.
• Bursts of flows accumulate as the flows traverse, merge, and diverge across multiple hops. Even a minor error in packet treatment at one node can have a cascading effect on downstream nodes.
• Loops formed in the network topology increase the maximum bursts of flows exponentially .
• Node and link failures, which are more common in large-scale networks (Section 3.5), require dynamic traffic steering to alternate paths. This can further contribute to flow fluctuations.

It should be noted that non-DetNet flows can also be high in volume and may impact the scalability of DetNet flows. For example, they may lead to high bandwidth utilization, limiting the ability to reserve resources or to perform effective traffic steering for DetNet flows. However, it is assumed that appropriate strategies will be deployed at the ingress to manage non-DetNet traffic and prevent real-time interference with DetNet traffic. Support for bursty traffic is essential, along with mechanisms to mitigate micro-bursts. To accommodate flow fluctuations, pre-planned configurations, ingress traffic conditioning, scalable queuing, and enhanced buffer capacity are necessary. In addition, the time required for network reconfiguration to respond to such changes should be strictly controlled. For example, it may need to be limited to a specified threshold.

Scaling networks often results in situations where an end-to-end flow traverses a large number of hops, for example, 15 or more. The network topology can also be complex, including star, ring, mesh, and their combinations, and may be hierarchical. It is required to support networks with such diverse topologies and large hop counts. Delivering DetNet QoS in large and complex networks requires end-to-end bounds on both latency and jitter, as discussed in Section 3.1 of . Achievable end-to-end latency bounds necessarily depend on the number of hops for a flow. In the best case, the additional latency introduced by queuing mechanisms for DetNet QoS is bounded by a fixed per-hop maximum value, making the resulting end-to-end latency bounds a linear function of the number of hops in addition to the inherent forwarding latency of the nodes involved. In contrast, it is possible to achieve fixed end-to-end jitter bounds that are independent of the number of hops. Such fixed jitter bounds are strongly preferable to those that increase with hop count. DetNet QoS requires resource allocation in advance (e.g., link bandwidth and node buffer resources), as discussed in Section 3.2.1 of . The complexity of resource allocation can range from linear (e.g., allocating resources for each hop via a path-based resource reservation protocol such as RSVP ) to potentially exponential (e.g., if solving a complex graph optimization problem is required). Although this complexity does not directly affect the achievable end-to-end latency and jitter bounds, it does influence other aspects such as the computational effort and the time required to admit a new flow without disrupting the QoS of existing DetNet flows. Different queuing mechanisms exhibit different properties in terms of achievable end-to-end jitter bounds, achievable end-to-end latency bounds, and the complexity of resource allocation. All three factors should be considered in the evaluation and selection of scalable DetNet queuing mechanisms.

As described in Section 3.4, there will be a large number of flows, each potentially having a different level of demand for DetNet services. provides various use cases and their requirements in the areas of industry, electricity, buildings, etc. Some of these use cases clearly specify requirements for both latency and jitter, while others omit jitter requirements. Different types of users have different demands, just as a network provider offers different network services for personal or enterprise business. Some applications have critical Service Level Agreement (SLA) requirements, such as remote control in manufacturing, cloud-based Programmable Logic Controllers (PLCs) for industrial automation, and manufacturing and differential protection in power systems. For these services, exceeding latency or jitter bounds can result in property loss or security risks. Therefore, users of these services cannot tolerate any non-deterministic behavior and are typically willing to pay more for higher-quality network services. Other applications have more relaxed SLA requirements, such as cloud gaming, cloud-based virtual reality (VR), and online meetings in "consumer" networks. While users of these services seek a good quality of experience, they can tolerate some level of performance variation. For example, occasional violations of latency bounds may be acceptable if they occur infrequently. Those different applications expect different types of solutions, often corresponding to varying cost levels. Different SLA requirements and the presence of multiple domains in scaling networks necessitate the use of diverse DetNet technologies and queuing mechanisms. For services that demand strict determinism, highly deterministic queuing technologies need to be deployed, which may require upgrading all network devices. In contrast, for less stringent services, it may be sufficient to upgrade only certain devices, such as edge nodes, or to share existing network resources. As more queuing mechanisms are developed, it becomes increasingly desirable to provide queuing solutions that support multiple levels of deterministic performance and to allocate resources appropriately to optimize overall network resource utilization. These different queuing technologies may be used in the following scenarios:

• Within the same network, where some devices are equipped with multiple queuing technologies. This allows the operator to select the appropriate service type or level as needed.
• Across multiple network domains, where different domains support different queuing technologies and coordination between them is required.
• In separate network implementations tailored for distinct application domains, where no additional coordination is necessary.

| Industrial, tight jitter, strict latency limit | |<----->| Industrial, strict latency limit | |<---.....--->| non-periodic, relatively loose latency requirements | |<-------............------->| Best effort | +----------------------------------------------------------> 0 latency ]]>

A deterministic network should support a variety of deterministic services to meet the diverse requirements of various applications. This includes supporting corresponding queuing mechanisms at multiple DetNet QoS levels, if necessary. For example, different queuing mechanisms can offer varying levels of latency, jitter, and other guarantees, and a single network device may implement multiple mechanisms concurrently. An aggregation device, for instance, may employ mechanisms defined in and , and other mechanisms to forward traffic along different paths simultaneously. The need to support multiple queuing mechanisms becomes especially prominent in large-scale networks, compared to small-scale environments. In such cases, more than eight queues or sub-queues may be required to support complex queuing strategies, exceeding the typical eight traffic classes available in TSN-enabled networks. Accordingly, configuring multiple queuing mechanisms in deterministic networks can be complex. Such configurations MUST support unified or simplified approaches to scheduling and managing multiple queuing mechanisms. For example, in distributed scenarios without a controller, information about the queuing mechanisms, including types and associated algorithms, and queue forwarding capabilities with different levels of latency and jitter guarantees, can be advertised within the domain. In centralized scenarios, this information can be reported to the controller, allowing it to build a deterministic network resource topology pool for path computation. In addition, to enable fine-grained resource management and ensure resource guarantees during session setup and teardown, it is necessary to establish standardized methods for quantifying and measuring resources associated with interfaces and queues.

In deterministic networks, the end-to-end service may span multiple network domains. Each domain may adopt different queuing mechanisms or may operate at different link speeds (see Section 3.3) for the same queuing mechanism. Both cases may generate additional end-to-end latency, jitter, and packet loss, as different queuing mechanisms and link speeds may result in varying scheduling granularities or phases between domains. In the case of a queuing mechanism switchover, such as from a time-synchronized mechanism like to an asynchronous mechanism like , a collaboration mechanism across multiple domains MUST be considered. Similarly, when switching between different link speeds, such as from 1 Gbps to 10 Gbps (or vice versa), the quantification of deterministic forwarding resources (such as time slots) of the queuing mechanism MUST be aligned with the link speed. A device that connects multiple domains requires a flexible queue stitching function. This may include increasing buffer capacity at inter-domain devices to provide sufficient adjustment space when flows are forwarded across different queuing mechanisms, applying jitter compression to decouple queuing behaviors between domains, or using additional traffic shaping solutions to smooth traffic. These techniques help ensure that the same scale of service flows can be forwarded successfully across domains that use different queuing mechanisms and/or operate at different link speeds.

According to , the DetNet data plane can provide or carry two metadata items in MPLS and IP data planes: Flow-ID and sequence number. The Flow-ID is used to identify individual DetNet flows or aggregated flows, while the sequence number is used by PREOF for each DetNet flow. The Flow-ID is used by both the service and forwarding sub-layers, whereas the sequence number is only used by the service layer. Metadata can also be used for OAM indications and instrumentation related to DetNet data plane operation. In general, support for additional data plane metadata and related processing SHOULD be provided in deterministic networks. With the introduction of IPv6 Extension Headers and Segment Routing over IPv6 , native IPv6 data plane SHOULD be supported along with the IPv6-specific enhancements. This section lists the data plane enhancement requirements that are based on, but not limited to, the technical requirements in Section 3. It describes how IP and/or MPLS, along with related OAM, can be used to support flow identification and packet treatment over Layer 3. Some enhancements to the control plane may also be required to meet the requirements in Section 3. However, these are out of scope for this document and are expected to be addressed in or other documents.

Current IPv6 aggregated flow identification is generally based on 5 or 6 tuples, IP prefixes, or wildcards as indicated in . However, in deterministic networks, the number of individual flows can be huge, and these flows may dynamically join or leave an aggregated flow at each hop, as described in Section 3. Such behaviors lead to the difficulty in identifying aggregated flows by relying on prefixes or wildcards. In addition, deterministic services may demand different deterministic QoS requirements according to different levels of application requirements. To address this, flow identification with service-level aggregation should be supported. Flow identification also enables packets to be quickly directed to the appropriate queue. In scaling networks, a large number of flows may require either deterministic latency or normal forwarding services. Explicit flow identification makes it easier to quickly distinguish the DetNet flows without relying on the longest-match rule across multiple header fields in IP data plane. Therefore, explicit aggregated flow identification SHOULD be supported.

According to Section 3.1, deterministic networks should support synchronized or asynchronous queuing mechanisms. Different queuing mechanisms require different types of DetNet-specific metadata to support functions, such as traffic regulation and queue management, which are used to ensure deterministic latency. For instance, the data plane needs to support the identification of the cycle for cyclic queuing and forwarding, or latency-related information for time-based queuing, in order to enable the use of corresponding queuing and/or scheduling mechanisms in a large-scale network. When different queuing mechanisms are deployed at the same network node, the corresponding metadata for each queuing mechanism should be provided simultaneously. When multiple types of metadata are carried in a single packet, the metadata format should be self-descriptive and extensible to support a variable number of metadata fields. However, carrying additional metadata increases processing overhead, such as requiring fixed- or variable-length lookups, and increasing the number of read/write operations on packet headers. Therefore, data plane processing efficiency also needs to be considered when ensuring deterministic latency. The specific methods or solutions are out of scope of this document. This document does not specify what metadata and formats should be carried in the data plane. Solution documents should clearly explain why and how the metadata carried as the data plane interacts with queuing or other functions, and how it contributes to deterministic network deployment.

This document specifies the technical requirements for scaling deterministic networks and enhancing the corresponding data plane. Some of the proposed queuing mechanisms and trials are cited, as they provide useful insights for improving existing queuing mechanisms to meet the requirements of scaling deterministic networks.

When DetNet flows span multiple domains and require multi-domain collaboration, security guarantees need to be strengthened.

No IANA actions are requested.

The authors would like to thank David Black, Jinoo Joung, Lou Berger, Balazs Varga, Fan Yang, Tianran Zhou, and Yaakov Stein for their helpful suggestions. The authors also would like to thank Liang Geng, Peter Willis, Shunsuke Homma, and Li Qiang for their previous works.

The following people have contributed substantially to this document:

Zongpeng Du China Mobile Email: duzongpeng@chinamobile.com Lei Zhou New H3C Technologies Email: zhou.leih@h3c.com

IEEE

International Telecommunication Union International Telecommunication Union IEEE IEEE IEEE IEEE IEEE IEEE Standards Association

Some trials have been conducted to validate the concept of large-scale deterministic networks. To validate deterministic networking technology in large-scale networks, a trial of Deterministic IP on China Environment for Network Innovations (CENI), which is a network built for new network technology trials, was conducted. The trial covered a distance of 3,000 km over 13 hops, and jitter was controlled within 100 us. To validate remote control over Deterministic IP with strict performance requirements, a trial was conducted in cooperation with Baosteel, a Chinese steel company that proposed latency requirements of less than 4 ms and jitter under 20 us. The trial network spanned 600 km. Both the first and second trials were based on a frequency synchronization solution. To realize synchronization of multiple flows across an inter-provincial network during an exhibition, Emergen proposed a requirement in which two flows, one carrying video and the other carrying VR content, would be transmitted from Province A and would arrive at Province B at the same time. This was intended to enable synchronized playback of camera-captured video alongside the corresponding VR model. This requirement was proposed to support the deployment of virtual industry products. Due to time constraints and other limitations, the requirement was fulfilled using edge network devices and was delivered with a lower level of SLA. ETRI, in collaboration with a smart factory operator, network operators, equipment vendors, and universities, demonstrated an ultra-low-latency, high-reliability 5G wired and wireless network-based remote Industrial Internet of Things (IIoT) service. The demonstration connected a control center and a smart factory across the networks of three different operators over a distance of 280 km. In this trial, it was shown that real-time remote smart manufacturing is feasible, with round-trip delay maintained below 3 ms within the smart factory and below 10 ms between remote 5G industrial devices. In the future, the team plans to examine the feasibility of large-scale deterministic networking by interconnecting smart factories in Gyeongsan, South Korea, and Oulu, Finland. These trials indicate that both operators and enterprise users increasingly demand deterministic performance in large-scale networks. However, the implementation technologies used to achieve this are not the same across deployments.