Framework for Flow Aggregation in Scaling Deterministic Networking (DetNet)

Introduction The clearly states that Deterministic Networking (DetNet) operates at the IP layer and delivers service which provides extremely low data loss rates and bounded latency. The DetNet data plane must provide the aggregation of DetNet flows in order to support larger numbers of DetNet flows and improve scalability by reducing the per-hop states. The introduces that the flow aggregation is the ability to aggregate individual flows along with their associated resource control into a large aggregate. It is recommended that the DetNet flow aggregation be enabled for compatible flows with the same or very similar QoS and CoS characteristics via the use of wildcards, masks, prefixes, and ranges. It also suggests the reduction of per-hop states help avoid the per DetNet-flow specific state maintenance in a transit node. It further provides arguments on how DetNet services might be realized in term of delay bound, delay jitter and bandwidth provisioning. Furthermore, the has proposed and expanded two methods of flow aggregation, one being the aggregation via LSP hierarchy and the other to aggregate DetNet flows as a new combined DetNet flow. For enhanced DetNet, has described the data plane enhancement requirements such as the aggregated flow identification in section 4.1. For example, explicit aggregated flow identification in IPv6 networks and the flow identification with service-level aggregation should be supported. In scaling networks, it also should consider the aggregated flows over multi-domains and achieve different levels of co-existed applications with different SLAs requirements which requiring the fine-grained QoS provisioning through flow aggregation. Moreover, the aggregated flows still requires to improve the scalability to avoid the large amount of control signaling and the states maintaining of DetNet flows in enhanced DetNet. This document describes the specific requirements of flow aggregation in enhanced DetNet and provides the enhancement considerations. It also discusses the realization of DetNet flow aggregation for 5GS as well.

Motivation This framework document is informational and is intended to complement, not replace, existing DetNet data plane specifications and . The aggregation mechanisms defined in existing DetNet standards remain normative. The objectives of this framework are: - To provide a structured analysis of the common challenges and motivations for flow aggregation in large-scale DetNet - To illustrate the requirements of flow aggregation for large-scale DetNet deployments - To refine the enhanced considerations of flow aggregation that transcend specific data plane implementations In summary, this draft serves as the starting reference document that revolves around the fundamentals of the flow aggregation in scaling DetNet. It also paves the way for any potential future enhancements while maintaining the compatibility with the current standards.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology The terminology is defined as .

Requirements for Flow Aggregation in Enhanced DetNet

Background A flow aggregation is a process of merging multiple flows into a single aggregated flow. An aggregated flow should be treated, both in control and data planes, as if it is a single flow. A flow is defined as the set of packets having the identical source, destination, and application. A flow can be characterized by TSpec and RSpec, in which traffic characteristics and service requirements are defined, respectively. In order to be treated as a single flow, within an aggregation region, the packets in an aggregated flow should have the same starting point, ending point, and service requirements. The TSpec of an aggregated flow can be inferred from the TSpecs of the individual flows. The problems that arise are the followings; 1) How to determine an aggregation region, 2) Which flows to aggregate, and 3) Whether the aggregation is beneficial. The answer to the problem 3) will be dependent upon the decision on 1) and 2). For example, in case of C-SCORE and N-SCORE, when the clock discrepancies and the propagation delays between the nodes are ignored, the E2E latency bound for a flow is given as,

where Dh(p) is the latency experienced by p from the arrival at the node 0 to the departure from node h C-SCORE . B is the maximum burst size of the flow under observation that p belongs to. Lj is the observed maximum packet length in the node j over all the flows, Rj is the link capacity of the node j, L is the maximum packet length of the flow, and r is the service rate of the flow. When two flows with identical parameters B, L, and r are aggregated, then the max burst size and the service rate becomes twice, 2B and 2r. Then the E2E latency bound of the aggregated flow becomes

which can be less than the E2E latency bound of a single flow, in most cases. This example clarifies the benefits of flow aggregation. However, there are more things to consider; 1) the effect of the aggregation region size, 2) the effect of de-aggregation from and re-aggregation into a different set of flows over aggregation regions, 3) the complexity of control plane configuration such as admission control, and 4) the necessity flow reshaping (for rate-based solutions) or slot reallocation (for slot-based solutions). This document is to answer these questions.

Aggregated Flows across Multi-domains Flow aggregation is recommended in the multi-domain scenario to achieve the end-to-end QoS guarantees for aggregated flow(s) that span across multiple domains. As per , different network implementations may be intended for different application domains, where there is no additional requirements for the coordination. As defined in [ITU-T Y.2122], the network operating parameters of a flow aggregate should be exchanged among different network domains. As shown in Figure 1, the DetNet domain B receiving an aggregated flow should identify the flow and get the service requirements such as the QoS parameters of the flow aggregate.

Individual Flows +-----------------+ +-----------------+ -------> | | | | ...... | DetNet Domain A | Aggregated Flow | DetNet Domain B | -------> | | --------------> | | +-----------------+ +-----------------+

Aggregated Flows with Fine-grained QoS Provisioning The draft specifies that different levels of applications differ in the SLAs requirements such as tight jitter, strict latency, loose latency and so on. While these types of aggregated requirements might bear the coarse-grained nature, individual flows demand differentiated DetNet treatments and more granular QoS forwarding behaviors. A DetNet node or domain adopting multiple forwarding technologies needs to transmit individual flows by aggregating them into a selected treatment solution that corresponds to one of some pre-defined per-hop QoS behaviors, as shown in Figure 2. The DetNet flows with the same level of service requirements can be aggregated to receive collective treatments and forwarding behaviors. The DetNet flows can be aggregated to several pre-defined classes. For example, as per , a 5GS as a logical DetNet node requires to achieve the service requirements and service levels of the aggregated flows, along with the provisioning of fine-grained per-hop behavior (PHB) to each individual flow.

Aggregated Flows with Bursts Flows across Multiple Hops As per , the treatment solutions in data plane can be categorized based on performance and functional characteristics. For example, the category of a solution can be classified based on the traffic granularity, e.g., flow aggregate vs. class aggregate. The class aggregate is provided to simplify the control and accommodate traffic fluctuations by combining flows requiring the same or similar levels of service requirements. The flow aggregation based on the class aggregate could further improve the scalability. As per , there may be a large number of traffic flows in a scaling network, which makes it nearly impossible to achieve the flow-specific state identification. As shown in the Figure 3, the flow identification of aggregated-class can be used to indicate the required treatment and forwarding behaviors without the need to maintain excessive states at transit nodes.

Individual Aggregated Flows +-------------+ Flow(s) +-------------+ +-------------+ -------> | | | | | | .... |DetNet Node A|---------->|DetNet Node B|----->...|DetNet Node N| -------> | | | | | | +-------------+ +-------------+ +-------------+ 'Bucketed' into Large number of Fewer number of classes Individual Flows -----------------> consisting of aggregated flows When DetNet flows are aggregated based on service-class, transit nodes provide deterministic services to a flow aggregate and go through the per-class scheduling without the burden to maintain excessive per-flow states. Still, a transit node performing aggregation should ensure all per-flow service requirements within an aggregated class are achieved. For example, the latency or jitter bounds of an aggregated class shall not exceed the corresponding metrics of any individual flow that has been bucketed into the class. The aggregation based on the class aggregate has data plane and controller plane aspects.

Enhancement Considerations for Flow Aggregation

Aggregated-flow Classification The deterministic services may also demand different deterministic QoS requirements according to different levels of application and service requirements. The individual flows may be aggregated based on a sharing aggregated level of traffic specification and service requirements which could be identified by pre-defined aggregation levels or classes. For example, the DetNet flows MAY be classified based on the service SLAs requirements of applications in scaling networks as per . And the services can also be classified into tight/loose latency, large/small burst, periodic/non-periodic and large/small scale services as per . Several classes can be predefined to indicate the different levels of applications with SLAs requirements and each class demands differentiated QoS behaviors and treatment as well as different DetNet capabilities in scaling networks. The aggregation information may be used alone or together with other metadata to guide the queueing and forwarding behaviors that have been specified in C-SCORE , TQF , EDF , TCQF , gLBF , N-SCORE and PIFO . The encoding of the class-based aggregation information may reuse the DSCP/TC or existing field such as the TC field in A-Label as per . And it also can be encapsulated with the aggregation-based metadata as per .

Aggregated-flow Identification It is required to be dynamic and simplified to ensure the aggregated flows have compatible DetNet flow-specific QoS characteristics. As per , the aggregated flow identification is used to explicitly identify the aggregated flow such as an Flow ID or an Aggregation ID for SRv6 and IPv6 network, or an aggregation label, which is referred to as an A-Label as defined in in MPLS network. The encoding of the aggregation information, as reflected by flow identification, may be an A-Label encapsulated in MPLS header as per or an Aggregation ID encapsulated in IPv6 Options or SRv6 SRH as per .

Aggregated-flow Coordination In scaling networks, flow aggregations become more prevalent, with flows frequently joining and leaving, which may potentially lead to accumulated bursts of flows across multiple hops. Such challenges can be mitigated by coordinating packets within aggregated flows such as proportional scheduling and interleaving. * Proportional scheduling could allocate transmission opportunities based on flow weights, ensuring that each flow receives a fair share of network resources. * Interleaving could achieve micro burst smoothing by rotating the transmission of packets across different flows through timed gates as described in .

Aggregated-flow Admission Control and Resource Allocation Flow aggregation may interact differently with various DetNet forwarding and queuing mechanisms. This section highlights considerations for major categories: * Aggregate-level Admission Control: it should support admission control decisions based on aggregate characteristics while ensuring individual flow requirements within the aggregate can be met. * Resource Allocation: different forwarding and queuing mechanisms for allocating resources to aggregates must consider the composite requirements of member flows, including worst-case latency, jitter, and bandwidth demands.

Aggregated-flow Control Plane Flow aggregation may require to support more control plane extensions such as: * As described in . TSN networks can be interconnected over a DetNet Network. Flow Aggregation during DetNet flow to TSN stream mapping will be accomplished by BGP Flowspec in control plane as per . * Path computation should consider the end-to-end budget of the aggregated flow, which must cover the requirements of all its member flows. * The network parameters of an aggregated flow should be exchanged among different domain controllers as per .

Security Considerations Security considerations for DetNet are covered in the DetNet Architecture and DetNet security considerations .

IANA Considerations This document makes no requests for IANA action.

Acknowledgements The authors would like to thank Lou Berger, Janos Farkas and Toerless Eckert for their review, suggestions and comments to this document.

References Normative References Infomative References 3GPP TS 23.501 (V19.0.0): System Architecture for the 5G System (5GS) 3GPP TS 23.502 (V19.0.0): Procedures for the 5G System (5GS) 3GPP TS 23.503 (V19.0.0): Policy and charging control framework for the 5G System (5GS); Stage 2

Realization of Flow Aggregation for 5GS DetNet The 3GPP in its document has defined and standardized how a 5G system (5GS) may behave as a logical DetNet node, as well as how a 5GS DetNet node may integrate into the IP-domain DetNet as described in . 3GPP has realized the functionalities of the DetNet forwarding sub-layer. As a logical DetNet transit node, a 5GS behaves as a transparent box to external DetNet entities. It can connect to either DetNet end systems or full-fledged IP DetNet domain(s) or both. The 3GPP has demonstrated a ‘composite’ architecture in that a 5GS could act as one or more DetNet nodes upon the integration into the IP DetNet domain. The granularity of determining a 5GS DetNet node is per UPF for each network instance, with the corresponding UPF-id identified as the 5GS DetNet node-id. The 3GPP has implicitly specified two types of DetNet related traffic parameters. One type is the higher-level per-(logical)-node QoS requirements like the node max-latency, max-loss, etc., while the other is more granular settings with which DetNet flow attributes and specifications are mapped to the Flow Description information. The DetNet flow specifications could be based on IP-tuple information, e.g., including IP address, protocol type, ToS, TCP/DUP ports, etc. The document has provided more details. Please note that this draft revolves around the general discussions of the flow aggregations in enhanced DetNet across multiple domains. It emphasizes the objectives & requirements, along with insightful considerations for the possible enhancement to the matter. This indicates the generic principles that are related to the cross-domain flow aggregation as raised in the draft. While the 3GPP defines a 5GS may behave as a logical DetNet (transit) node and the 5GS does own certain advantageous features for a 'composite' DetNet instantiation, the (DetNet) flow aggregation is not an intrinsic characteristics that has been fulfilled in the 5GS. As we explain in the following subsection , the realization of flow aggregation for a 5GS DetNet 'composite' node participating in an enhanced DetNet domains requires the seamless interactions between the IETF domain (DetNet) controller and the 5GS domain counterpart.

Realization of 5GS DetNet Service across Domains 3GPP has so far standardized the forward sub-layer functionality for 5GS. It indicates a 5GS (logical) DetNet node could connect to other end systems and/or IP DetNet domains, together to form a holistic end-to-end DetNet. Thanks to the 'composite' architecture of a 5GS node, along with the interaction between an CPF:DetNet controller in IETF domain and the NF TSCTSF in 3GPP domain , a 5GS node might realize much more advanced DetNet services than a traditional IP DetNet transit node. In scenarios where the (IETF-domain) CPF:Detnet Controller could well divide the DetNet QoS service requirements that are in reality associated with an integrated DetNet domain into multiple QoS sub-requirements that together form the original end-to-end QoS equivalence, a 5GS might be considered as a standalone DetNet (sub-)domain with its own DetNet QoS (sub-)requirements that would be provisioned from the CPF:DetNet controller. The 5GS DetNet QoS (sub-)requirements serve a portion of the original requirements of the integrated DetNet domain. These together form a scaling network to realize the 5GS DetNet service across domains.

5GS QoS Provisioning: Aggregated vs. Fine-grained We have explained previously that the 3GPP has implicitly specified two categories of DetNet related traffic parameters. One type bears the aggregated nature for (5GS DetNet) node-level requirements, while the other addresses the more granular DetNet flow-level attributes and specifications. Evidently, with this kind of two-hierarchy architecture, a 5GS DetNet node could achieve not only the node-level aggregated QoS requirements, but also the more fine-grained flow-level QoS provisioning. This reflects the true value of applying our flow aggregation model in scaling networks to realizing advanced DetNet services for 5GS. Here, we want to point out that the feasibility of applying our flow aggregation scheme indeed depends on the hierarchical nature of a 5GS DetNet node. Had the same aggregation scheme been applied to DetNet nodes that do not have the similar intrinsic hierarchy, the effectiveness could be certainly impaired.

State Maintenance at a 5GS Transit node The 5GS QoS architecture is roughly comprised of three levels, i.e., the UE, the PDU-session, and the QoS-flow levels. Technically, a 5GS DetNet node is of 'composite' nature with a large number of configuration, provisioning, operation and runtime states to maintain. At first glance, this might undermine the state-reduction objective via the flow aggregation for a 5GS DetNet transit node. Fortunately, the 5GS DetNet work owns intrinsically a couple of aspects to handle the challenges: First, also as we have mentioned before, a 5GS node behaves as a transparent entity participating in the DetNet domain. Thus, even having a significant number of states, this can naturally have the 5GS DetNet related states remain hidden from the external entities(and domains). Second, the 3GPP NF TSCTSF exchanges only traffic parameters with the IETF CPF:Detnet Controller, but not the states that are maintained inside a 5GS DetNet node. The external DetNet domain does not care the inside status of a 5GS, nor can it.

Flow Classification & Identification at 5GS node As we have explained so far, the IETF domain CPF:DetNet controller provides traffic parameters & specifications to 3GPP NF TSCTSF. Thus, the SLA requirements of applications in scaling networks could be readily pre-specified in the IETF DetNet CPF, which would then apply the flow classification mapping (to aggregated service classes) and send them over to a 5GS DetNet node to enforce. This model can also relieve the classification burden of a 5GS node in reality. The 5GS has excellent control logics to address flow identification. For example, PDRs (Packet Detection Rules), SDF (Service Data Flow) filters (e.g., IP-filter, MAC-filter, etc.), etc., are all good tools to differentiate flows . Further, the 5GS has standardized powerful procedures for the establishment & update of PDU sessions/QoS flows, which accordingly achieves the flow dynamics (e.g., flow joining & leaving a flow-aggregate as manifested potentially by a PDU session) . Moreover, some QoS parameters, e.g., Aggregated Bit Rate (ABR), may stand at different levels, including UE-ABR, Session-ABR, flow-ABR, etc., that would make the service differentiation & sharing on the aggregated-class (A-Class) level feasible.