Network Measurement Problem Statement

Introduction With the rapid advancement of cloud computing, big data, and AI/ML, data center networks have become the fundamental infrastructure supporting modern Internet applications and services. Data centers host critical tasks such as e-commerce transactions, search engines, online social networking, and AI/ML model training, which impose stringent requirements on the underlying network, characterized by large scale, high bandwidth, low latency, high dynamics, and heterogeneity. These characteristics introduce challenges to network operation and management, particularly in the areas of accurate perception of network state, real-time monitoring of operational status, and efficient localization of performance bottlenecks. In order to ensure network performance, resource utilization efficiency, and service quality, while supporting network system design and operation, it is necessary to achieve efficient perception and measurement of network dynamics. Network measurement provides the essential observability to address these challenges. It encompasses a wide range of techniques to collect and analyze data about network traffic, device status, and performance metrics. Key measurement tasks include:

Flow size estimation
Heavy hitter (large flow) detection
Latency and jitter measurement
Packet loss detection
Path tracing
Cardinality estimation (number of distinct flows)
Entropy and distribution estimation
Anomaly detection (e.g., sudden changes, overspeed flows)

However, existing network measurement techniques face critical limitations when deployed in high-speed, large-scale, and distributed environments. Prevalent network measurement paradigms such as sketch-based measurement, in-band network telemetry (INT), sampling, and probing, each struggle with fundamental trade-offs between accuracy, resource overhead, and timeliness. This document analyzes the shortcomings of these prevalent measurement paradigms. It further highlights the unique challenges introduced by distributed systems, such as multi-node data aggregation, resource heterogeneity, and the need for fairness in distributed queries. The scope of this work is limited to a problem statement and gap analysis. Its goal is to articulate the pressing issues that prevent current network measurement solutions from meeting the responsiveness and scalability demands of modern networks, thereby encouraging the development of more effective network measurement systems.

Terminology CC: Congestion Control ECN: Explicit Congestion Notification. Specified in . INT: In-band Network Telemetry RTT: Round-Trip Time SRAM: Static Random-Access Memory TE: Traffic Engineering WAN: Wide Area Network

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Why Network Measurement is Needed Network measurement provides the essential observability of networks by continuously monitoring, collecting, and analyzing the operational status, behavioral characteristics, and resource usage of network devices and data flows. It provides a continuous stream of data necessary for understanding, managing, and optimizing complex network infrastructures. By collecting and analyzing information about device states, traffic characteristics, and performance metrics, measurement systems transform raw network operations into actionable insights. These insights are important for a wide range of network management tasks:

Traffic Engineering (TE): Effective traffic engineering requires accurate and fine-grained knowledge of traffic distribution information at the link level, port level, and flow level. Network measurement provides this visibility, it allows administrators to accurately identify congested and bottleneck links, thereby formulating refined traffic scheduling and path optimization strategies to achieve reasonable allocation and efficient utilization of network resources.
Load Balancing: Dynamic load balancing depends on real-time awareness of the load status of network devices and links. Network measurement supplies this awareness, it allows the system to dynamically distribute incoming traffic across available paths. By shifting traffic away from overloaded nodes or links toward idle resources, measurement-informed load balancing prevents performance degradation, improves responsiveness, and maximizes overall network throughput.
Congestion Control (CC): High-precision congestion control requires fine-grained, real-time visibility into dynamic network conditions, such as per-hop queue occupancy, per-hop forwarding delay, packet loss, and link utilization. Network measurement provides this visibility, it allows the system to make precise rate adjustments and enable closed-loop control that proactively prevents congestion before it degrades performance, thereby improving overall network throughput and reducing latency.
Resource Accounting: In multi-tenant environments such as cloud data centers, accurate accounting of resource usage is essential for both operational efficiency and business fairness. Network measurement provides precise resource usage data, such as bandwidth consumption, traffic volume, and service request counts. This enables cloud service providers to implement usage-based billing models that are transparent and equitable, while also informing capacity planning and resource allocation decisions.
Fault Diagnosis: When network failures or performance anomalies occur, rapid fault identification and repair are critical. Network measurement enables real-time monitoring of device status and traffic behavior, supporting early detection of abnormal traffic patterns. By pinpointing the location and nature of faults, whether equipment failure, misconfiguration, or security incident, measurement data enables operators to take swift corrective measures to minimize downtime and preserve service reliability.

The tasks described above share a common dependency: timely, accurate, and efficient access to network state information. However, existing network measurement techniques face fundamental limitations that prevent them from fully meeting the requirements of modern high-speed, large-scale, and distributed network environments.

Problems with Existing Measurement Techniques This section outlines the limitations of four common measurement approaches.

Sketch-based measurement Network measurement tasks often require per‑flow tracking, where a flow is typically identified by a five‑tuple (source and destination IP addresses, source and destination ports, and protocol type). However, such fine‑grained tracking is impractical in high‑speed environments due to the prohibitive storage and computation overhead associated with storing and processing every flow. Sketch‑based measurement addresses this challenge by providing a lightweight approximate solution. Using sub‑linear memory, sketch‑based approaches can rapidly process and aggregate measurement data on switch chips, gateway devices, or control planes with much fewer resources, enabling scalable flow monitoring. Despite its widespread adoption, sketch‑based measurement has several fundamental limitations in modern networks:

Accuracy Trade‑offs: Sketch‑based measurement inherently introduces estimation errors due to hash collisions. Although error bounds are provable, the errors can be significant for small flows, especially when memory is constrained. Moreover, the error is often data‑dependent and difficult to predict without post‑processing.
Limited Information: Most sketch‑based approaches capture only simple metrics like packet or byte counts. They cannot directly measure more informative per‑hop attributes such as queuing delay, jitter, or path information without substantial extensions.
Static Resource Allocation: Traditional sketch‑based measurement allocates fixed memory across all flows, which is inefficient given the heavy‑tailed nature of flow size distributions.
Lack of Flexibility in Distributed Environments: When sketch‑based measurements from multiple devices must be aggregated to obtain a global view, they require identical parameters (e.g., array size, hash functions) to be mergeable, which prevents adaptation to varying network conditions and available bandwidth across nodes.
Fairness Issues in Distributed Top‑K Queries: In distributed settings where each device reports its local top‑K frequent items, naive aggregation of sketch‑based measurements can be biased toward larger streams. This unfairness leads to inaccurate global views.

In-band Network Telemetry In-band Network Telemetry (INT) leverages the programmability of modern data planes to embed detailed, per-packet network state information directly into data flows. INT-capable devices interpret header fields as telemetry instructions, collecting and writing network states (such as queue length, port statistics, and timestamps) into packets as they traverse the network. This enables fine-grained, real-time network monitoring without requiring separate out-of-band channels. INT supports multiple operation modes (INT-XD, INT-MX, INT-MD), which present trade-offs between bandwidth overhead and flexibility. Despite these advantages, INT faces several fundamental limitations:

High Bandwidth Overhead: Embedding telemetry information requires each packet to carry additional headers, increasing packet size. Even with small per-packet INT metadata, the cumulative overhead can be significant in high-speed networks, consuming a substantial fraction of link capacity.
Processing Overhead: Switches must parse, modify, and generate INT headers, which consumes network device resources and impacts forwarding performance. This overhead scales with the number of collected telemetry items and the volume of packets carrying INT metadata.
Limited Scalability: As network size (i.e., number of hops) grows, the amount of telemetry data embedded in each packet grows linearly, exacerbating overhead. Collecting and aggregating INT data at receivers or analyzers also becomes a bottleneck.
Lack of Reliability: INT data is attached to the data packets, which means that if a packet is lost, the telemetry for that path will also be lost. This makes INT less robust in lossy environments.
Deployment Constraints: Current INT mechanisms have limited support for multi-domain and Layer-2-centric broadband infrastructures. In addition, existing INT-based measurement cannot effectively adapt to network topology changes caused by link or device failures.

Sampling-based Measurement Sampling-based measurement is a widely deployed technique that estimates traffic characteristics by selecting a subset of packets from the transmission stream. It is valued for its low storage overhead and high processing efficiency, making it particularly suitable for high-speed links and resource-constrained observation points. Sampling-based measurement is often deployed as an interface between the control plane and data plane, assisting with traffic engineering and anomaly detection. Despite its widespread adoption, sampling-based measurement suffers from several fundamental limitations:

Limited Coverage and Inaccuracy: Sampling misses many packets, especially short-lived flows, leading to biased estimates. Flow size distributions derived from sampled data are often inaccurate for small flows, and critical events like microbursts or transient congestion may not be detected if they occur between samples.
Parameter Sensitivity: The performance of sampling-based measurement heavily depends on the sampling rate and strategy, which are difficult to tune optimally for dynamic traffic patterns.
Loss of Correlation: Sampling individual packets breaks the temporal and spatial correlations that are essential for tasks such as path tracing, and causality analysis. Without per-flow continuity, it becomes difficult to reconstruct the complete behavior of network flows.

Probing-based Measurement Probing-based measurement inject dedicated probe packets into the network to actively measure path properties such as delay, loss, and reachability. Unlike passive measurement methods that observe existing traffic, probing-based measurement provides on-demand, targeted visibility and is particularly useful for fault isolation and performance verification in operational networks. It is widely deployed for link health monitoring and rapid failure detection. Despite these advantages, probing-based measurement faces several fundamental limitations:

Limited Coverage: Probe packets only measure the paths they traverse, missing the majority of data traffic. They cannot provide per-flow statistics or capture the behavior of specific application flows.
Unrepresentative Traffic: Probe packets may not experience the same queuing and forwarding behavior as normal data packets, for example, they may be prioritized or rate-limited, making them unrepresentative of actual data plane performance.
Additional Overhead: Frequent probing consumes bandwidth and switch processing resources. In large-scale networks, the probing rate must be carefully controlled to avoid interfering with normal traffic.
Lack of Flow-level Information: Probe packets lack flow-level granularity; they cannot attribute measured performance degradation to specific flows or tenants, limiting their usage in fine-grained troubleshooting and root cause analysis.

Distributed Network Measurement Challenges Modern networks are inherently distributed: they consist of multiple devices (e.g., routers, switches) and may span multiple data centers or geographic regions. Distributed network measurement introduces additional complexities.

Data Aggregation and Communication Overhead In a distributed system, each node collects local measurements, which must be aggregated at a central analyzer to obtain a global view. The aggregation process faces:

Bandwidth Bottlenecks: Transmitting raw measurement data from many nodes to a central point can overwhelm the network, especially over Wide Area Networks (WANs) with limited bandwidth. For example, collecting full flow records from hundreds of routers is infeasible. As the number of flows and switches grows, aggregating telemetry from distributed sources becomes a bottleneck.
Compression and Accuracy Trade-off: To reduce transmission costs, nodes must compress their measurements (e.g., using sketches). However, compression introduces errors that accumulate during aggregation, potentially leading to unacceptable global inaccuracies.
Adaptability: Available bandwidth between nodes and the central analyzer may vary over time. Measurement systems should adapt the size of transmitted summaries accordingly, but existing schemes often require fixed-size summaries.

Heterogeneity and Dynamic Resource Allocation Distributed nodes may have different hardware capabilities (e.g., CPU, memory, programmable ASICs) and run different software stacks. Measurement tasks may also have varying resource demands. Key issues include:

Resource Efficiency: Static allocation of measurement resources across nodes is inefficient. Some nodes may be underutilized while others are overloaded. Dynamic resource allocation based on network states and task importance is needed but not yet standardized.
Hardware Constraints: Programmable switches have limited Static Random-Access Memory (SRAM) and strict pipeline constraints (single-stage memory access, limited concurrency). Many sophisticated measurement algorithms cannot be deployed on such hardware without significant modification.

Fairness in Distributed Top-K Queries As mentioned before, when aggregating local top-K frequent items from multiple disjoint data streams, the resulting global top-K list can be biased if the streams have vastly different sizes. For example, a truly globally frequent item that appears in a small stream may be overlooked because its local estimate is overshadowed by collisions from large streams. This "Top-K unfairness" undermines the accuracy of applications like network monitoring and anomaly detection. Existing unbiased sketches do not guarantee fairness for top-K queries because the selection of top-K elements is correlated with estimation errors.

Synchronization and Consistency Distributed measurements often require time synchronization across nodes to correlate events (e.g., delay measurements, one-way delay). Inaccuracies in clock synchronization can lead to misleading results. Moreover, when measurements are used for closed-loop control (e.g., congestion control), consistency across nodes is critical; stale or inconsistent information can cause oscillations or instability.

Measurement for Closed-Loop Control Emerging data center transport protocols leverage real-time network telemetry to adjust sending rates at microsecond timescales. In distributed environments, closed-loop control introduces additional challenges:

Multi-hop Feedback Latency: Telemetry must traverse multiple network segments before reaching the decision point. Cumulative delay from collection to delivery can exceed the tolerable control loop latency.
Cross-domain Coordination: Consistent telemetry collection across heterogeneous devices and administrative domains is difficult due to differing capabilities and formats, hindering end-to-end visibility.
Robustness to Packet Loss: Individual switch or link failures may cause loss of telemetry-carrying packets. Control loops must remain stable even with incomplete measurement data.

Security Considerations TBD.

IANA Considerations This document has no IANA actions.