]> Everything OPS & Arrcus

benoit@everything-ops.net

Ciena

paitken@ciena.com

Deutsche Telekom

holger.keller@telekom.de

ZTE Corporation

liu.yao71@zte.com.cn

Operations and Management OPSAWG IPFIX ordered encapsulation MPLS label stack tunneling The IP Flow Information Export (IPFIX) protocol allows the same Information Element to appear multiple times in a Template Record. RFC 7011 states that multiple occurrences of the same Information Element SHOULD follow the logical order of their treatments by the Metering Process; however, no protocol mechanism exists for the Exporting Process to explicitly signal to the Collecting Process that this ordering is guaranteed. Without such a guarantee, the Collecting Process cannot safely interpret repeated Information Elements as ordered observations. This document describes the problem, presents use cases, analyzes candidate solutions, and specifies a mechanism -- a new Ordered Set type -- that establishes an end-to-end ordering guarantee: the Metering Process records repeated Information Element values in the order the corresponding observations are made, the Exporting Process preserves and signals this guarantee, and the Collecting Process interprets the n-th occurrence of an Information Element as corresponding to the n-th observation by the Metering Process at the Observation Point. This document updates RFC 7011, RFC 5476, and RFC 6183. About This Document Status information for this document may be found at .

Introduction The IPFIX protocol ("Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of Flow Information") exports Flow information from an Exporting Process to one or more Collecting Processes. An (Options) Template Record defines the structure of a Data Record by specifying an ordered list of Information Elements. When the same Information Element appears more than once in a Template Record, Section 8 of states:

• If an Information Element is required more than once in a Template, the different occurrences of this Information Element SHOULD follow the logical order of their treatments by the Metering Process.

The use of SHOULD rather than MUST means that an Exporting Process is not required to guarantee this ordering. Furthermore, Section 4.2 and Section 4.3 of each explicitly state that "the order of the Information Elements in the Template Records is not guaranteed." Nevertheless, the Metering Process SHOULD record repeated Information Element values in the order in which the corresponding observations are made, as this is the natural basis for any ordering guarantee. It is important to note that IPFIX is fundamentally an export protocol: defines the behavior of the Exporting Process and the Collecting Process in detail, but places very few normative constraints on the Metering Process. The IPFIX architecture ("Architecture for IP Flow Information Export") explicitly scopes IPFIX as governing "the export of measured IP Flow information from an IPFIX Exporting Process to a Collecting Process" and defines the Metering Process as an internal component of the IPFIX Device -- alongside the Exporting Process, but not a protocol peer (see Figure 2 of ). An end-to-end ordering guarantee requires that all three components act consistently: the Metering Process records values in the order in which they are observed at the Observation Point, the Exporting Process preserves that order during export, and the Collecting Process interprets the received ordering accordingly. There is consequently no IPFIX protocol mechanism by which the Exporting Process can signal to the Collecting Process that the ordering of repeated Information Elements is guaranteed to reflect the Metering Process observation order -- even when the Metering Process correctly records those observations in order. Without such a signal, a Collecting Process cannot safely interpret the n-th occurrence of an Information Element as corresponding to the n-th observation made by the Metering Process at the Observation Point. This limitation has several practical consequences, of which the following are the most prominent:

1. Information Element Proliferation: position-specific Information Elements accumulate in the IANA IPFIX registry to work around the missing ordering guarantee (see ).
2. Multi-Layer Encapsulation Ambiguity: without an ordering guarantee, the Collecting Process cannot determine which encapsulation layer a given Information Element occurrence corresponds to (see ).

The ordering problem also arises in the Packet Sampling (PSAMP) protocol family , which uses IPFIX as its export protocol and relies on IPFIX IEs to describe packet selection and observation metadata. PSAMP defines a Composite Selector as an ordered sequence of Primitive Selectors applied in succession to a Packet Stream. Section 6.5.1 of already requires that selectorId Information Elements in the Selection Sequence Report Options Template MUST be listed in the order the corresponding Primitive Selectors are applied -- making the ordering requirement for Options Template Records older than this document. However, this requirement cannot be enforced at the IPFIX protocol level: the Options Template Set (Set ID 3) of provides no mechanism for the Exporting Process to signal to the Collecting Process that the ordering of repeated selectorId occurrences is guaranteed. The Ordered Options Template Set (Set ID 5) defined in this document closes this gap. ("Export of Structured Data in IP Flow Information Export (IPFIX)") has sometimes been cited as a solution to this problem, through the "ordered" semantic defined for its basicList and subTemplateMultiList structured data types. However, as analyzed in detail in , the "ordered" semantic is an export ordering guarantee only -- it does not assert that the order reflects the Metering Process observation order at the Observation Point. Furthermore, structured data still falls under the SHOULD of Section 8 of and does not upgrade it to a MUST. therefore does not solve the problem described in this document. This document specifies a mechanism to address these use cases by providing an explicit ordering guarantee for repeated Information Elements in an IPFIX (Options) Template Record. The scope of this document is deliberately limited to this specific problem. Other potential IPFIX improvements are explicitly out of scope.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology This document uses the IPFIX-specific terminology defined in , the PSAMP-specific terminology defined in , and the IPFIX Mediation terminology defined in . As in , the first letter of each IPFIX-specific term is capitalized. The following terms are used as defined in :

Collecting Process:

The process in a Collector that receives and processes IPFIX Messages.

Data Record:

A record that contains values of the parameters corresponding to a Template Record.

Data Set:

A Set that contains one or more Data Records defined by the same Template Record.

Exporting Process:

The process in an Exporter that sends IPFIX Messages to a Collecting Process.

Flow Record:

A record that contains information about a specific Flow.

Information Element:

A protocol-independent description of an attribute that has been observed or metered.

Metering Process:

The process that generates Flow Records from packet observations at an Observation Point.

Observation Domain:

The largest set of Observation Points for which Flow information can be aggregated by a Metering Process.

Observation Point:

A location in the network where IP packets can be observed.

Options Template Record:

A Template Record that defines the structure of an Options Data Record.

Options Template Set:

A collection of one or more Options Template Records that have been grouped together in an IPFIX Message.

Set:

A collection of records that have a similar structure, prefixed by a header.

Template Record:

A record that defines the structure of a Data Record.

Template Set:

A collection of one or more Template Records that have been grouped together in an IPFIX Message.

Transport Session:

In SCTP, the Transport Session is the SCTP association. In TCP, the Transport Session is the TCP connection. In UDP, the Transport Session is the UDP session, uniquely identified by the combination of IP addresses and UDP ports used.

Note: A Flow Record is a specific type of Data Record. Options Data Records, which carry metadata rather than Flow measurements, are also Data Records but not Flow Records. The following term is modified from :

Set ID:

A field in the Set header that identifies the Set type. Value 2 identifies a Template Set, value 3 an Options Template Set, values 4 and 5 identify Ordered Template Sets and Ordered Options Template Sets respectively as defined in this document, and values 256 and above identify Data Sets. Values 6-255 are reserved for future use.

The following terms are used as defined in :

Composite Selector:

An ordered composition of Selectors, in which the output Packet Stream issuing from one Selector forms the input Packet Stream to the succeeding Selector.

Primitive Selector:

A Selector that applies exactly one selection method and is the elemental component of a Composite Selector.

Selection Sequence:

A specific combination of an Observation Point and a Composite Selector, identified by a selectionSequenceId Information Element.

The following term is used as defined in :

IPFIX Mediator:

An IPFIX Device that receives IPFIX Messages from one or more Exporting Processes or other IPFIX Mediators, optionally transforms them through one or more Intermediate Processes, and forwards the results to one or more Collecting Processes.

The following new terms are defined in this document:

Ordered Template Set:

A Set with Set ID 4 that carries Ordered Template Records. It uses the same wire format as the Template Set (Set ID 2) defined in , with the addition that the ordering of repeated Information Elements in the corresponding Data Records is guaranteed to reflect the Metering Process observation order at the Observation Point.

Ordered Options Template Set:

A Set with Set ID 5 that carries Ordered Options Template Records. It uses the same wire format as the Options Template Set (Set ID 3) defined in , with the same ordering guarantee as an Ordered Template Set.

Ordered Template Record:

A Template Record transmitted in an Ordered Template Set (Set ID 4). The order of Information Elements in an Ordered Template Record is significant: when the same Information Element appears more than once, its occurrences MUST reflect the order in which the corresponding values are observed by the Metering Process at the Observation Point.

Ordered Options Template Record:

An Options Template Record transmitted in an Ordered Options Template Set (Set ID 5), with the same ordering guarantee as an Ordered Template Record.

Ordered Data Set:

A Data Set whose Template ID refers to an Ordered Template Record or an Ordered Options Template Record. The ordering of fields in each Data Record within the Set reflects the Metering Process observation order.

Problem Statement & Use Cases

The Ordering Ambiguity The IPFIX protocol allows any Information Element to appear multiple times in a Template Record. Section 8 of provides the following guidance and example:

• If an Information Element is required more than once in a Template, the different occurrences of this Information Element SHOULD follow the logical order of their treatments by the Metering Process. For example, when exporting the two source IP addresses of an IPv4-in-IPv4 packet, the first sourceIPv4Address Information Element occurrence should be the IPv4 address of the outer header, while the second occurrence should be the address of the inner header.

The use of SHOULD rather than MUST means that a conformant Exporting Process is not required to respect this ordering. Even when the Exporting Process does follow the ordering by convention, there is no protocol field by which it can communicate this intent to the Collecting Process. Furthermore, the ordering of Information Elements may be affected by optimizations within the Metering Process itself (for example, software that processes packet headers from different encapsulation layers in parallel rather than in strict observation order) or by optimizations that intervene between the Metering Process and the Exporting Process. Section 2.2 ("IPFIX Limitations") of notes that "some encoding optimizations are based on the permutation of Information Element order", which conflicts with any strict ordering requirement applied universally to all existing IPFIX Sets.

Information Element Proliferation The absence of an ordering guarantee has led implementors to define position-specific Information Elements, effectively polluting the IANA IPFIX registry with entries that exist solely to work around the lack of an ordering mechanism. The MPLS label stack is the most prominent example. Rather than using a single mplsLabelStackSection (IPFIX IE TBD1) Information Element appearing multiple times in ordered fashion, the IPFIX Information Elements registry ("Information Model for IP Flow Information Export (IPFIX)") defines ten separate Information Elements:

• mplsTopLabelStackSection (IPFIX IE 70)
• mplsLabelStackSection2 (IPFIX IE 71)
• mplsLabelStackSection3 (IPFIX IE 72)
• mplsLabelStackSection4 (IPFIX IE 73)
• mplsLabelStackSection5 (IPFIX IE 74)
• mplsLabelStackSection6 (IPFIX IE 75)
• mplsLabelStackSection7 (IPFIX IE 76)
• mplsLabelStackSection8 (IPFIX IE 77)
• mplsLabelStackSection9 (IPFIX IE 78)
• mplsLabelStackSection10 (IPFIX IE 79)

This approach has some drawbacks:

• It consumes ten Information Element identifiers where one could suffice.
• It artificially limits the exportable label stack depth to ten entries.
• It cannot be generalized to other stacked or layered structures without defining further sets of numbered Information Elements.

The same pattern extends beyond the label stack entries. A cluster of Information Elements exists for attributes of the top-of-stack label only: mplsTopLabelType (IPFIX IE 46), mplsTopLabelIPv4Address (IPFIX IE 47), mplsTopLabelPrefixLength (IPFIX IE 91), mplsTopLabelIPv6Address (IPFIX IE 140), mplsTopLabelTTL (IPFIX IE 200), mplsTopLabelExp (IPFIX IE 203), and postMplsTopLabelExp (IPFIX IE 237). These exist because no general mechanism allows expressing "the attribute of the label at stack position N." With ordered export, a single repeated Information Element per attribute would suffice. A Metering Process observing MPLS traffic may observe a label stack of arbitrary depth. An operator may wish to export the entire label stack as a sequence of mplsLabelStackSection values, where the first occurrence corresponds to the top of the stack, the second to the next label, and so on. With the mechanism specified in this document, the Exporting Process can use a single mplsLabelStackSection Information Element repeated N times in an Ordered Template Record and signal that the sequence reflects the top-to-bottom order of the label stack as observed by the Metering Process. This eliminates the need for the ten numbered label-stack Information Elements. Since N is a free parameter determined by the Exporting Process, label stacks of depth greater than ten can be exported in full. This lifts the artificial cap inherent in the positional naming scheme.

Ordering Ambiguity in PSAMP Options Template Records The ordering requirement extends beyond Template Records: it also applies to Options Template Records used by the Packet Sampling (PSAMP) protocol ("A Framework for Packet Selection and Reporting") ("Sampling and Filtering Techniques for IP Packet Selection") ("Packet Sampling (PSAMP) Protocol Specifications"). This is the primary use case for the Ordered Options Template Set (Set ID 5) specified in this document. PSAMP defines a Composite Selector as "an ordered composition of Selectors, in which the output Packet Stream issuing from one Selector forms the input Packet Stream to the succeeding Selector" (). The word "ordered" in the definition is architecturally significant: a Composite Selector is a sequence, not a set. The ordering matters semantically. Section 8.1 of demonstrates that filtering followed by sampling and sampling followed by filtering produce different statistical results:

• Filter then Sample: the filter creates population subsets with potentially different intensities, allowing the sampling rate to be tuned per subset. Packets not matching the filter never load the Sampling function.
• Sample then Filter: the same sampling rate is applied to the entire stream before filtering; packets ultimately discarded by the filter still consume sampling capacity.

A Composite Selector is exported using the Selection Sequence Report Options Template defined in Section 6.5.1 of . When the Selection Sequence consists of multiple Primitive Selectors, the selectorId Information Element appears once per Primitive Selector in the same Options Template Record. Section 6.5.1 of is explicit:

• "the selectorId's MUST be identified in the order they are used."

Section 6.5.1 of illustrates this with a concrete example: Selection Sequence 7 (Filter selectorId 5, then Sampler selectorId 10) and Selection Sequence 9 (Sampler selectorId 10, then Filter selectorId 5) are distinct, produce different results, and are differentiated solely by the order of the two selectorId occurrences in the Options Template Record. RFC 5476 therefore already requires ordered Options Template Records, and a PSAMP-aware Collecting Process can infer this guarantee from receipt of the Selection Sequence Report Options Template. However, IPFIX-layer components that do not implement PSAMP -- forwarders, middleboxes, or generic analysis tools -- have no means to apply that constraint. The Ordered Options Template Set (Set ID 5) specified in this document expresses the ordering guarantee at the IPFIX protocol level, making it visible to any IPFIX component regardless of PSAMP awareness.

Multi-Layer Encapsulated Traffic Modern networks increasingly carry traffic with multiple encapsulation layers. SRv6 deployments, in particular, may result in packets with two or three nested IPv6 headers, each with its own source and destination addresses. When an Exporting Process exports fields from such packets, repeating Information Elements such as sourceIPv6Address and destinationIPv6Address, the Collecting Process has no way to determine which occurrence corresponds to which encapsulation layer unless an ordering guarantee is explicitly provided. This problem is described in , which proposes several mechanisms to address the specific case of encapsulation layer identification. That draft brings renewed attention to an ordering ambiguity that has been present in IPFIX since was first published. The present document addresses the underlying, more general problem of ordered Information Element export, of which multi-layer encapsulation is one important use case. When a packet carries multiple IP encapsulation layers -- for example, an SRv6 packet where a Provider Edge adds an outer IPv6 header encapsulating an inner IPv6 packet from the customer -- an Exporting Process may wish to export fields from multiple layers. With the mechanism specified in this document, the Exporting Process can include destinationIPv6Address twice in an Ordered Template Record and signal that the first occurrence corresponds to the outermost encapsulation layer and the second to the innermost, reflecting the order observed by the Metering Process. This use case covers IPv4-in-IPv4, IPv4-in-IPv6, IPv6-in-IPv6, and any other IP-in-IP encapsulation scenario, as well as SRv6 traffic with nested Segment Routing Headers as described in ("Export of Segment Routing over IPv6 Information in IP Flow Information Export (IPFIX)"). Further examples include:

• MAC-in-MAC (IEEE 802.1ah Provider Backbone Bridging): both sourceMacAddress and destinationMacAddress repeat -- once for the outer backbone MAC (B-MAC) and once for the inner customer MAC (C-MAC).
• VXLAN ("Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks") and GENEVE ("Geneve: Generic Network Virtualization Encapsulation"): outer IP transport addresses (UDP/IP encapsulation) and inner tenant IP addresses both map to Information Elements such as sourceIPv4Address and destinationIPv4Address; without an ordering guarantee the Collecting Process cannot distinguish transport layer from tenant layer.
• GRE: outer and inner IP header fields repeat for the same Information Elements, with the same ambiguity.
• IEEE 802.1ad Q-in-Q (stacked VLAN tags): vlanId (IPFIX IE 58) repeats for each VLAN layer -- the outer Service VLAN (S-TAG) and inner Customer VLAN (C-TAG). Workaround IPFIX IEs dot1qVlanId (IPFIX IE 243) and dot1qCustomerVlanId (IPFIX IE 245) already exist in the IANA IPFIX registry as positional substitutes, following the same pattern as the MPLS label stack IPFIX IEs.

Without the mechanism specified in this document, unambiguous export of these layered fields would require defining new position-specific Information Elements, perpetuating the same registry proliferation already seen with MPLS and IEEE 802.1ad. The mechanism specified in this document addresses the common case where all encapsulation layers are exported consecutively, with the n-th occurrence of an Information Element corresponding to the n-th layer. Cases where only a sparse subset of layers is exported, or where backward compatibility with Collecting Processes that do not support Ordered Sets is required, may benefit from the explicit layer-marking approach described in , which is complementary to the mechanism defined here.

Solution Space This section analyzes candidate solutions. The analysis is presented to inform discussion and justify the specification in .

• Note: The detailed analysis in this section is intended to support Working Group discussion. Once the Working Group has reached consensus on the recommended solution, this section may be condensed into a summary or moved to an informative appendix.

Updating RFC 7011 SHOULD to MUST Alone (Rejected) The most minimal approach would be to update Section 8 of by strengthening the existing SHOULD to a MUST, requiring all Exporting Processes to guarantee that repeated Information Elements follow the Metering Process observation order. The fundamental limitation of this approach is that it constrains the Exporting Process but provides no signal to the Collecting Process. Even if every Exporting Process were compliant with the updated requirement, the Collecting Process would still have no per-Template indication of whether a given Template Record was sent by an updated Exporting Process (ordering guaranteed) or by an older implementation that predates the update (ordering not guaranteed). Without such a signal, the Collecting Process cannot safely interpret the n-th occurrence of a repeated Information Element as the n-th Metering Process observation. Two additional considerations reinforce the rejection. First, Sections 4.2 and 4.3 of explicitly state that "the order of the Information Elements in the Template Records is not guaranteed"; a SHOULD-to-MUST change in Section 8 alone would leave these statements in direct conflict with the new requirement. Second, Section 2.2 ("IPFIX Limitations") of notes that "some encoding optimizations are based on the permutation of Information Element order"; a universal MUST would prohibit such optimizations across all Templates containing repeated Information Elements, regardless of whether ordering is semantically significant. This approach is therefore rejected.

RFC 6313 Structured Data with the "ordered" Semantic (Rejected) This option is analyzed in detail because has been cited as a solution to the ordering problem (see Introduction). The analysis below shows that it does not provide the required guarantee and is therefore rejected. defines structured data types -- basicList, subTemplateList, and subTemplateMultiList -- and associates them with list semantics. One defined semantic is:

• The "ordered" structured data type semantic specifies that elements from the list in the structured data are ordered.

Using this mechanism, an Exporting Process could encode a sequence of ordered MPLS label values as a basicList with the "ordered" semantic: , , ) ]]> Similarly, multi-layer encapsulation fields could be encoded in a subTemplateMultiList with the "ordered" semantic. The fundamental limitation of this approach is that the "ordered" semantic is an export ordering guarantee, not a Metering Process observation ordering guarantee. Section 4.4.6 of states only that "elements from the list in the structured data are ordered" -- it does not state that the order reflects the sequence in which the Metering Process makes its observations. Furthermore, structured data, when carried in an IPFIX Template Record, still falls under the general rule of Section 8 of : "the different occurrences of this Information Element SHOULD follow the logical order of their treatments by the Metering Process." does not upgrade this SHOULD to a MUST, nor does it bind the "ordered" semantic to Metering Process observation order. Since IPFIX is an export protocol, governing the wire format between the Exporting Process and the Collecting Process, it cannot by itself make claims about the internal behavior of the Metering Process. A Collecting Process that receives a basicList or subTemplateMultiList with the "ordered" semantic therefore cannot safely conclude that element N corresponds to the N-th observation by the Metering Process. This is precisely the guarantee this document requires, and does not provide it. This approach is therefore rejected.

New IPFIX Version 11 (Rejected) A new IPFIX version implies a broad overhaul of the protocol and would carry community expectations far beyond the narrow scope of this document. The risk of scope creep is significant. Beyond scope, a version bump would be far more disruptive than new Set IDs. A Collecting Process that receives an IPFIX Message with an unknown version number has no choice but to discard the entire message. This means that introducing a new IPFIX version would cause every existing Collecting Process to reject all IPFIX Messages -- including those carrying unordered Data Sets -- until it is updated to support the new version. The deployment impact would be severe and disproportionate to the problem being solved. By contrast, new Set IDs are additive. Section 3.3.2 of reserves Set IDs 4 through 255 for future use and mandates that "the Length value MUST be used to determine the position of the next Set." A Collecting Process that does not support Set ID 4 or 5 can therefore use the Set Length field to skip the unknown Set and continue processing the remainder of the IPFIX Message normally. All prior IPFIX extensions -- including -- have been deployed without requiring a version bump. This approach is therefore rejected.

Marker Information Element (Evaluated) Define a new Information Element whose presence in an (Options) Template Record indicates that the Information Elements following it (until the next marker) are to be interpreted as ordered. This approach is similar to the encapLayerTop and encapLayer2 Information Elements proposed in for encapsulation layer identification. Advantages:

• A Collecting Process that does not understand the marker Information Element MAY discard only that one Information Element (per Section 9 of ), retaining the other fields in the Data Record.

Disadvantages:

• The semantic of other Information Elements becomes dependent on the presence and position of the marker Information Element, creating an implicit dependency between IPFIX IEs that is not expressible in the IPFIX Information Model .
• The approach is less clean than a dedicated Set type from a protocol design perspective.

Options Template Indicator (Evaluated) Define a new Options Template Record with a scope of Template ID, containing a bitmap Information Element that identifies, for each field in the referenced Template Record, whether that field belongs to an ordered sequence. This is analogous to the flowKeyIndicator Information Element (IPFIX IE 173), which uses an Options Template to convey which fields of a referenced Template Record are Flow Keys. Advantages:

• Does not require changes to the Data Set format.
• The Options Template Record can be transmitted once per Template and cached by the Collecting Process.

Disadvantages:

• Requires the Collecting Process to correlate the Options Template record with the Data Template before it can correctly interpret Data Records, adding a level of indirection and a potential for synchronization issues.
• More complex to implement than a dedicated Set type.

New Ordered Set Type (Recommended) Define new Set IDs to indicate that the ordering of Information Elements within the corresponding Data Records is guaranteed to reflect the Metering Process observation order. This approach extends the existing Set ID structure of :

Set ID	Name
2	Template Set (existing)
3	Options Template Set (existing)
4	Ordered Template Set (new)
5	Ordered Options Template Set (new)

Ordered Template Records (Set ID 4) and Ordered Options Template Records (Set ID 5) use the same wire format as their existing counterparts (Set ID 2 and Set ID 3 respectively), with the addition of the ordering semantic on the corresponding Data Records. Advantages:

• Provides an explicit, unambiguous per-Set signal that ordering is guaranteed, without relying on the Collecting Process to parse an in-band marker or to maintain an out-of-band correlation table.
• No structured data complexity is required; existing flat Template and Data Record formats are reused.
• Backwards compatible: existing Set IDs 2 and 3 are unaffected and continue to be used for cases where ordering is not guaranteed or not relevant.
• Generalizable to any stacked or layered protocol structure without requiring new position-specific Information Element definitions.

Disadvantages:

• Requires both Exporting Process and Collecting Process updates to support the new Set IDs.
• A Collecting Process that does not support Set ID 4 or 5 may discard the entire Set upon receipt, resulting in a complete loss of the Data Records in that Set without any log information. Section 3.3.2 of does not specify the behavior of a Collecting Process upon receipt of an unknown Set ID.

New Ordered Set Specifications The recommended approach is the New Ordered Set Type: the definition of two new Ordered Set types, Ordered Template Set (Set ID 4) and Ordered Options Template Set (Set ID 5). The rationale is as follows:

1. Explicit protocol semantics: A dedicated Set type provides an unambiguous, per-Set signal that ordering is guaranteed. The Collecting Process requires no additional correlation or inference to determine whether a given Template is ordered.
2. Implementation practicality: The RFC 6313 structured data approach is technically sound, but its usefulness depends on the availability of implementations, which are not known to be widespread. The New Ordered Set Type requires a more targeted change to existing IPFIX implementations.
3. Clean protocol design: Ordered data is a distinct semantic concept that warrants a first-class protocol signal. A new Set type is the natural extension point within the IPFIX message format, consistent with the principle that Set IDs 4 through 255 are reserved in for exactly this purpose.
4. Backwards compatibility: The existing Set IDs 2, 3, and 256 to 65535 are unaffected. Exporting Processes that do not require ordered export continue to use the existing Set IDs without modification.

Wire Format The Ordered Template Set (Set ID 4) uses exactly the same wire format as the Template Set (Set ID 2) defined in Section 3.4.1 of . The Ordered Options Template Set (Set ID 5) uses exactly the same wire format as the Options Template Set (Set ID 3) defined in Section 3.4.2 of . The Set ID value is the sole distinguishing element: it signals to the Collecting Process that the ordering of repeated Information Elements in corresponding Data Records is guaranteed. No new header fields, flags, or record structures are introduced. Implementations that already support Set ID 2 and Set ID 3 can support Set ID 4 and Set ID 5 by reusing the same parsing code and adding the ordering semantic to the Template Records so identified.

Template Withdrawal The Template Withdrawal mechanism defined in Section 8.1 of applies without modification to Ordered Template Records and Ordered Options Template Records. An Ordered Template Record is withdrawn by sending an Ordered Template Set (Set ID 4) containing a Template Record with the target Template ID and a Field Count of 0. An Ordered Options Template Record is withdrawn by sending an Ordered Options Template Set (Set ID 5) with a Field Count of 0. Upon receipt of a withdrawal, the Collecting Process MUST cease treating Data Sets referencing the withdrawn Template ID as Ordered Data Sets and MUST discard all stored information about that Template Record. This applies to the IPFIX transport state only; it does not retroactively alter the ordering interpretation of Data Records already received and decoded under that Template ID. A withdrawn Ordered Template ID MAY be reused by the Exporting Process within the same Transport Session and Observation Domain, consistent with Section 8.1 of . If the Template ID is subsequently registered via an unordered Template Set (Set ID 2), the Collecting Process MUST treat it as unordered. If registered again via an Ordered Template Set (Set ID 4), the ordering guarantee applies anew. The same applies for Options Template Records (Set IDs 3 and 5).

Normative Behavior The Exporting Process MUST preserve the Metering Process observation order when constructing Data Records for an Ordered Data Set. A Collecting Process identifies an Ordered Data Set by tracking the Set ID under which the corresponding Template Record was received. If the Template was received in an Ordered Template Set (Set ID 4) or an Ordered Options Template Set (Set ID 5), all subsequent Data Sets referencing that Template ID MUST be treated as Ordered Data Sets. If the Template was received in a Template Set (Set ID 2) or an Options Template Set (Set ID 3), the ordering of repeated Information Elements in the corresponding Data Records is not guaranteed. Section 3.4.1 of requires Template IDs to be unique within an Observation Domain and Transport Session. A given Template ID can therefore appear in only one Set type -- it cannot be registered as both an Ordered Template Set (Set ID 4) and a Template Set (Set ID 2), nor as both Set ID 5 and Set ID 3. There is consequently no ambiguity in the Collecting Process's per-Template-ID tracking of the ordering guarantee. The Collecting Process MUST interpret the n-th occurrence of an Information Element in a Data Record from an Ordered Data Set as corresponding to the n-th observation of that Information Element by the Metering Process at the Observation Point. This ordering guarantee applies independently to each Information Element: the n-th occurrence of IE X corresponds to the n-th observation of IE X, and the n-th occurrence of IE Y corresponds to the n-th observation of IE Y. The guarantee does not impose any relative ordering between occurrences of different Information Elements. This guarantee covers the ordering of exported occurrences only. The Exporting Process is expected to export all observations consecutively; where only a sparse subset of layers is exported, the Collecting Process cannot determine whether exported occurrences are contiguous from the first observation. For sparse export scenarios, the explicit layer-marking approach of is more appropriate. An IPFIX Mediator that receives Ordered Template Sets (Set ID 4) or Ordered Options Template Sets (Set ID 5) MUST preserve the ordering guarantee when forwarding. The Mediator MUST forward the corresponding Template Records in Ordered Template Sets or Ordered Options Template Sets, and MUST preserve the observation order of fields in the corresponding Data Records. A Mediator that cannot preserve the ordering guarantee MUST NOT forward Ordered Template Sets as unordered Template Sets (Set ID 2 or Set ID 3).

Updates to RFC 7011 This recommendation implies the following updates to . The following text in Section 8 of : OLD:

• If an Information Element is required more than once in a Template, the different occurrences of this Information Element SHOULD follow the logical order of their treatments by the Metering Process. For example, if a selected packet goes through two hash functions, and if the two hash values are sent within a single Template, the first occurrence of the hash value should belong to the first hash function in the Metering Process. For example, when exporting the two source IP addresses of an IPv4-in-IPv4 packet, the first sourceIPv4Address Information Element occurrence should be the IPv4 address of the outer header, while the second occurrence should be the address of the inner header. Collecting Processes MUST properly handle Templates with multiple identical Information Elements.

NEW:

• If an Information Element is required more than once in a Template transmitted in a Template Set (Set ID 2) or Options Template Set (Set ID 3), the different occurrences of this Information Element SHOULD follow the logical order of their treatments by the Metering Process. For example, if a selected packet goes through two hash functions, and if the two hash values are sent within a single Template, the first occurrence of the hash value should belong to the first hash function in the Metering Process. For example, when exporting the two source IP addresses of an IPv4-in-IPv4 packet, the first sourceIPv4Address Information Element occurrence should be the IPv4 address of the outer header, while the second occurrence should be the address of the inner header. Collecting Processes MUST properly handle Templates with multiple identical Information Elements. If the Template Record is transmitted in an Ordered Template Set (Set ID 4) or an Ordered Options Template Set (Set ID 5), the Metering Process MUST record the repeated Information Element values in the order in which the corresponding observations are made at the Observation Point, the Exporting Process MUST preserve and follow this order when constructing the corresponding Data Records, and the Collecting Process MUST interpret the n-th occurrence as corresponding to the n-th observation by the Metering Process at the Observation Point. The Metering Process MUST record observations from outermost to innermost for encapsulated traffic and from top to bottom for label stacks.

The following sentence in Section 4.2 of : OLD:

• Since the Metering Process Reliability Statistics Options Template contains two identical timestamp Information Elements, and since the order of the Information Elements in the Template Records is not guaranteed, the Collecting Process interprets the time interval of ignored packets as the range between the two values; see Section 5.2 for wraparound considerations.

NEW:

• Since the Metering Process Reliability Statistics Options Template contains two identical timestamp Information Elements, and since the order of the Information Elements in Options Template Records transmitted in Sets with Set ID 3 is not guaranteed, whereas the order in Options Template Records transmitted in Sets with Set ID 5 is guaranteed to reflect the Metering Process observation order, the Collecting Process interprets the time interval of ignored packets as the range between the two values; see Section 5.2 for wraparound considerations.

The following sentence in Section 4.3 of : OLD:

• Since the Exporting Process Reliability Statistics Options Template contains two identical timestamp Information Elements, and since the order of the Information Elements in the Template Records is not guaranteed, the Collecting Process interprets the time interval of dropped packets as the range between the two values; see Section 5.2 for wraparound considerations.

NEW:

• Since the Exporting Process Reliability Statistics Options Template contains two identical timestamp Information Elements, and since the order of the Information Elements in Options Template Records transmitted in Sets with Set ID 3 is not guaranteed, whereas the order in Options Template Records transmitted in Sets with Set ID 5 is guaranteed to reflect the Metering Process observation order, the Collecting Process interprets the time interval of dropped packets as the range between the two values; see Section 5.2 for wraparound considerations.

The following sentence in Section 3.3.2 of : OLD:

• Values from 4 to 255 are reserved for future use.

NEW:

• Values 4 and 5 are used for Ordered Template Sets and Ordered Options Template Sets, respectively, as specified in this document. Values from 6 to 255 are reserved for future use.

The following normative text is added to Section 3.3.2 of , after the sentence "The Length value MUST be used to determine the position of the next Set.": NEW:

• A Collecting Process that receives a Set with an unknown Set ID SHOULD log a warning to assist operators in detecting configuration or capability mismatches between Exporting and Collecting Processes.

Updates to RFC 5476 Section 6.5.1 of already mandates that selectorId occurrences in a Selection Sequence Report Interpretation Options Template Record MUST be listed in the order the corresponding Primitive Selectors are applied. However, predates the Ordered Options Template Set mechanism defined in this document, and therefore uses the regular Options Template Set (Set ID 3), which provides no protocol-level signal to the Collecting Process that the ordering is guaranteed. The following sentence in Section 6.5.1 of : OLD:

• If multiple Selectors are contained in the Selection Sequence Report Interpretation, the selectorId's MUST be identified in the order they are used.

NEW:

• If multiple Selectors are contained in the Selection Sequence Report Interpretation, the selectorId's MUST be identified in the order they are used. The corresponding Options Template Record MUST be transmitted in an Ordered Options Template Set (Set ID 5) as defined in this document, so that the Collecting Process receives an explicit signal that the ordering of selectorId occurrences reflects the Composite Selector application order.

Updates to RFC 6183 ("IP Flow Information Export (IPFIX) Mediation: Framework") defines the behavior of IPFIX Mediators but predates the Ordered Template Set mechanism defined in this document. This document adds the following normative requirement to Mediator behavior: An IPFIX Mediator that receives Ordered Template Sets (Set ID 4) or Ordered Options Template Sets (Set ID 5) MUST preserve the ordering guarantee when forwarding. The Mediator MUST forward the corresponding Template Records in Ordered Template Sets or Ordered Options Template Sets, and MUST preserve the observation order of fields in the corresponding Data Records. A Mediator that cannot preserve the ordering guarantee MUST NOT forward Ordered Template Sets as unordered Template Sets (Set ID 2 or Set ID 3).

New IPFIX Information Elements Five new IPFIX Information Elements are required to support ordered export for MPLS label stacks and label attributes; their full definitions are in the IANA Considerations of this document. For other protocol layers, no new IPFIX IEs are required: vlanId (IPFIX IE 58) and postVlanId (IPFIX IE 59) already serve IEEE 802.1ad Q-in-Q; sourceMacAddress (IPFIX IE 56) and destinationMacAddress (IPFIX IE 80) already serve MAC-in-MAC; and IP-level IPFIX IEs (sourceIPv4Address, destinationIPv4Address, and their IPv6 equivalents) already serve VXLAN, GENEVE, GRE, and IP-in-IP encapsulation. In each of these cases, the existing IPFIX IEs can simply be repeated in an Ordered Template Record.

• Note 1: No separate generic IPFIX IEs for TTL and TC (Exp) are defined in this document, because the TTL and TC (Exp) fields are already encoded as subfields of the 4-octet label stack entry exported by mplsLabelStackSection (TBD1). mplsTopLabelTTL (IPFIX IE 200) and mplsTopLabelExp (IPFIX IE 203) are therefore superseded by mplsLabelStackSection when used with ordered export; see . Note 2: The existing positional IPFIX IEs mplsTopLabelStackSection (IE 70) through mplsLabelStackSection10 (IE 79) encode only 3 octets (label, TC, and S fields), omitting the TTL field. mplsLabelStackSection (TBD1) encodes the full 4-octet label stack entry as defined in Section 2.1 of , making a separate TTL Information Element unnecessary at any stack depth.

The generic IPFIX IEs defined in this document do not immediately obsolete the existing positional IPFIX IEs. Implementations that do not support ordered export (Set IDs 4 and 5) continue to require the positional IPFIX IEs. Once ordered export is widely deployed, the positional IPFIX IEs listed in -- as well as the Q-in-Q workaround IPFIX IEs (dot1qVlanId IE 243, dot1qPriority IE 244, dot1qCustomerVlanId IE 245, dot1qCustomerPriority IE 246, postDot1qVlanId IE 254, postDot1qCustomerVlanId IE 255) -- may be deprecated in a future document.

Operational Considerations This section follows the guidelines of ("Guidelines for Considering Operations and Management of New Protocols and Protocol Extensions") for documenting operational and management considerations in IETF specifications. This document is a protocol extension to ; operational and management considerations that are not specific to ordered export are inherited from and are not repeated here.

Deployment and Coexistence Ordered export is strictly additive. Existing Set IDs 2 and 3 are unaffected, and Exporting Processes that do not require ordered export continue to operate without modification. Deployment can therefore proceed incrementally: ordered export can be enabled on a per-Template basis toward specific Collecting Processes, without disrupting any existing IPFIX infrastructure. Operators SHOULD enable ordered export only toward Collecting Processes that are known to support Set ID 4 and Set ID 5. Deployment against unaware Collecting Processes may result in silent loss of entire Ordered Sets, as discussed in .

Configuration Ordered export is opt-in. An Exporting Process is configured to use an Ordered Template Set (Set ID 4) or an Ordered Options Template Set (Set ID 5) for specific Template Records. No protocol-level negotiation is currently defined. Operators are responsible for ensuring that the Collecting Process is capable of processing the selected Set IDs before enabling ordered export. Ordered export may allow simplification of existing Template Records: where a deployment previously required multiple position-specific Information Elements (e.g., mplsTopLabelStackSection through mplsLabelStackSection10), a single repeated Information Element in an Ordered Template Record may suffice. This can reduce the number of Information Element identifiers consumed and the size of Template Records.

Verification of Correct Operation End-to-end verification of ordered export requires confirming correct behavior across all three components of the ordering chain:

• Metering Process: records repeated Information Element values in the order they are observed. This is internal to the implementation and is not directly observable on the wire.
• Exporting Process: transmits Ordered Template Sets (Set ID 4 or 5) and constructs the corresponding Data Records with field values in Metering Process observation order. This can be verified by inspecting the IPFIX Messages on the wire.
• Collecting Process: tracks which Template IDs were received in Ordered Template Sets and interprets the n-th occurrence of a repeated Information Element as the n-th Metering Process observation. Correct interpretation can be verified by comparing collected data against known traffic (e.g., known MPLS label stack or encapsulation layer ordering in a test environment).

Fault Management The primary fault scenario for ordered export is a mismatch between the Exporting Process and the Collecting Process: the Exporting Process sends Ordered Sets that the Collecting Process does not understand. Section 3.3.2 of mandates that "the Length value MUST be used to determine the position of the next Set," providing the mechanism for a Collecting Process that does not support Set ID 4 or 5 to skip the unknown Set and continue processing normally. However, does not specify what a Collecting Process MUST or SHOULD do upon receipt of an unknown Set ID -- whether to skip silently, log the event, or terminate the Transport Session. As a result, a RFC 7011 compliant Collecting Process may silently discard the Set without any notification to the Exporting Process or to the operator. This document addresses this gap; see . Operators SHOULD monitor for unexpected loss of Flow Records that may indicate silent Set discard at an unaware Collecting Process. If suspected, ordered export SHOULD be disabled for that destination until the Collecting Process is confirmed to support Set ID 4 and Set ID 5.

Performance The ordered export mechanism specified in this document introduces no additional overhead on the wire: Ordered Template Sets and Ordered Options Template Sets use the same wire format as their Set ID 2 and Set ID 3 counterparts. The only additional cost is the Collecting Process maintaining per-Template-ID state to track whether each Template was received in an Ordered or unordered Set, which is negligible. The Metering Process may incur an implementation cost to preserve observation order when recording repeated Information Element values, depending on its internal architecture. This is an implementation consideration and is outside the scope of this document.

Security Considerations This document does not change the trust model of the IPFIX protocol. The security considerations of apply. An Exporting Process that uses an Ordered Template Set (Set ID 4 or 5) but fails to preserve the Metering Process observation order will cause the Collecting Process to misinterpret the exported data. In particular, analytics systems that rely on the ordering to identify encapsulation layers or label stack positions may produce incorrect results. Operators SHOULD ensure that the Exporting Process and Metering Process are correctly coordinated before using ordered export.

IANA Considerations

IPFIX Set IDs Registry This document requests IANA to allocate the following values in the "IPFIX Set IDs" subregistry of the "IP Flow Information Export (IPFIX) Entities" registry :

Value	Name	Reference
4	Ordered Template Set	This document
5	Ordered Options Template Set	This document

IPFIX Information Elements Registry This document requests IANA to allocate the following Information Elements in the "IPFIX Information Elements" registry :

ElementID	Name
TBD1	mplsLabelStackSection
TBD2	mplsLabelType
TBD3	mplsLabelIPv4Address
TBD4	mplsLabelPrefixLength
TBD5	mplsLabelIPv6Address

mplsLabelStackSection

ElementID:

TBD1

Name:

mplsLabelStackSection

Abstract Data Type:

octetArray

Data Type Semantics:

default

Description:

The contents of a label stack entry at any position in the MPLS label stack, encoded as the 4-octet MPLS label stack entry format defined in Section 2.1 of . The encoding comprises a 20-bit label value, a 3-bit Traffic Class (TC) field, a 1-bit Stack Bottom (S) flag, and an 8-bit Time to Live (TTL) field. When this Information Element appears multiple times in an Ordered Template Record, the N-th occurrence corresponds to the label stack entry at stack position N (counting from the top) as observed by the Metering Process at the Observation Point.

Additional Information:

See Section 2.1 of for the MPLS label stack entry encoding. The ten positional IPFIX IEs mplsTopLabelStackSection (IPFIX IE 70) through mplsLabelStackSection10 (IPFIX IE 79) exist to work around the absence of an ordered export mechanism and are superseded by this IPFIX IE when used with the ordered export mechanism specified in this document.

Reference:

This document

mplsLabelType

ElementID:

TBD2

Name:

mplsLabelType

Abstract Data Type:

unsigned8

Data Type Semantics:

identifier

Description:

The type of the MPLS label at the given stack position, using the same encoding as mplsTopLabelType (IPFIX IE 46). Values are assigned by IANA in the "Multi-Protocol Label Switching (MPLS) Label Types" registry. When this Information Element appears multiple times in an Ordered Template Record, the N-th occurrence corresponds to the label type at stack position N (counting from the top) as observed by the Metering Process at the Observation Point.

Additional Information:

See the IANA registry "Multi-Protocol Label Switching (MPLS) Label Types" for the list of values.

Reference:

This document

mplsLabelIPv4Address

ElementID:

TBD3

Name:

mplsLabelIPv4Address

Abstract Data Type:

ipv4Address

Data Type Semantics:

default

Description:

The IPv4 address of the system that the MPLS label at the given stack position will cause this Flow to be forwarded to, using the same encoding as mplsTopLabelIPv4Address (IPFIX IE 47). When this Information Element appears multiple times in an Ordered Template Record, the N-th occurrence corresponds to the next-hop IPv4 address for the label at stack position N (counting from the top) as observed by the Metering Process at the Observation Point.

Additional Information:

mplsTopLabelIPv4Address (IPFIX IE 47) exists to work around the absence of an ordered export mechanism and is superseded by this IPFIX IE when used with the ordered export mechanism specified in this document.

Reference:

This document

mplsLabelPrefixLength

ElementID:

TBD4

Name:

mplsLabelPrefixLength

Abstract Data Type:

unsigned8

Data Type Semantics:

identifier

Description:

The prefix length of the IPv4 subnet that the MPLS label at the given stack position will cause the Flow to be forwarded to, using the same encoding as mplsTopLabelPrefixLength (IPFIX IE 91). When this Information Element appears multiple times in an Ordered Template Record, the N-th occurrence corresponds to the prefix length for the label at stack position N (counting from the top) as observed by the Metering Process at the Observation Point.

Units:

bits

Range:

0-32

Additional Information:

mplsTopLabelPrefixLength (IPFIX IE 91) exists to work around the absence of an ordered export mechanism and is superseded by this IPFIX IE when used with the ordered export mechanism specified in this document.

Reference:

This document

mplsLabelIPv6Address

ElementID:

TBD5

Name:

mplsLabelIPv6Address

Abstract Data Type:

ipv6Address

Data Type Semantics:

default

Description:

The IPv6 address of the system that the MPLS label at the given stack position will cause this Flow to be forwarded to, using the same encoding as mplsTopLabelIPv6Address (IPFIX IE 140). When this Information Element appears multiple times in an Ordered Template Record, the N-th occurrence corresponds to the next-hop IPv6 address for the label at stack position N (counting from the top) as observed by the Metering Process at the Observation Point.

Additional Information:

mplsTopLabelIPv6Address (IPFIX IE 140) exists to work around the absence of an ordered export mechanism and is superseded by this IPFIX IE when used with the ordered export mechanism specified in this document.

Reference:

This document

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document specifies the encoding to be used by an LSR in order to transmit labeled packets on Point-to-Point Protocol (PPP) data links, on LAN data links, and possibly on other data links as well. This document also specifies rules and procedures for processing the various fields of the label stack encoding. [STANDARDS-TRACK] This document specifies the export of packet information from a Packet SAMPling (PSAMP) Exporting Process to a PSAMP Collecting Process. For export of packet information, the IP Flow Information eXport (IPFIX) protocol is used, as both the IPFIX and PSAMP architecture match very well, and the means provided by the IPFIX protocol are sufficient. The document specifies in detail how the IPFIX protocol is used for PSAMP export of packet information. [STANDARDS-TRACK] This document describes a framework for IP Flow Information Export (IPFIX) Mediation. This framework extends the IPFIX reference model specified in RFC 5470 by defining the IPFIX Mediator components. This document is not an Internet Standards Track specification; it is published for informational purposes. This document specifies the IP Flow Information Export (IPFIX) protocol, which serves as a means for transmitting Traffic Flow information over the network. In order to transmit Traffic Flow information from an Exporting Process to a Collecting Process, a common representation of flow data and a standard means of communicating them are required. This document describes how the IPFIX Data and Template Records are carried over a number of transport protocols from an IPFIX Exporting Process to an IPFIX Collecting Process. This document obsoletes RFC 5101. Informative References This memo defines the IP Flow Information eXport (IPFIX) architecture for the selective monitoring of IP Flows, and for the export of measured IP Flow information from an IPFIX Device to a Collector. This memo provides information for the Internet community. This document specifies a framework for the PSAMP (Packet SAMPling) protocol. The functions of this protocol are to select packets from a stream according to a set of standardized Selectors, to form a stream of reports on the selected packets, and to export the reports to a Collector. This framework details the components of this architecture, then describes some generic requirements, motivated by the dual aims of ubiquitous deployment and utility of the reports for applications. Detailed requirements for selection, reporting, and exporting are described, along with configuration requirements of the PSAMP functions. This memo provides information for the Internet community. This document describes Sampling and Filtering techniques for IP packet selection. It provides a categorization of schemes and defines what parameters are needed to describe the most common selection schemes. Furthermore, it shows how techniques can be combined to build more elaborate packet Selectors. The document provides the basis for the definition of information models for configuring selection techniques in Metering Processes and for reporting the technique in use to a Collector. [STANDARDS-TRACK] This document specifies an extension to the IP Flow Information Export (IPFIX) protocol specification in RFC 5101 and the IPFIX information model specified in RFC 5102 to support hierarchical structured data and lists (sequences) of Information Elements in data records. This extension allows definition of complex data structures such as variable-length lists and specification of hierarchical containment relationships between Templates. Finally, the semantics are provided in order to express the relationship among multiple list elements in a structured data record. [STANDARDS-TRACK] The Intermediate Flow Selection Process is the process of selecting a subset of Flows from all observed Flows. The Intermediate Flow Selection Process may be located at an IP Flow Information Export (IPFIX) Exporter or Collector, or within an IPFIX Mediator. It reduces the effort of post-processing Flow data and transferring Flow Records. This document describes motivations for using the Intermediate Flow Selection process and presents Intermediate Flow Selection techniques. It provides an information model for configuring Intermediate Flow Selection Process techniques and discusses what information about an Intermediate Flow Selection Process should be exported. This document defines the data types and management policy for the information model for the IP Flow Information Export (IPFIX) protocol. This information model is maintained as the IANA "IPFIX Information Elements" registry, the initial contents of which were defined by RFC 5102. This information model is used by the IPFIX protocol for encoding measured traffic information and information related to the traffic Observation Point, the traffic Metering Process, and the Exporting Process. Although this model was developed for the IPFIX protocol, it is defined in an open way that allows it to be easily used in other protocols, interfaces, and applications. This document obsoletes RFC 5102. This document describes Virtual eXtensible Local Area Network (VXLAN), which is used to address the need for overlay networks within virtualized data centers accommodating multiple tenants. The scheme and the related protocols can be used in networks for cloud service providers and enterprise data centers. This memo documents the deployed VXLAN protocol for the benefit of the Internet community. Network virtualization involves the cooperation of devices with a wide variety of capabilities such as software and hardware tunnel endpoints, transit fabrics, and centralized control clusters. As a result of their role in tying together different elements of the system, the requirements on tunnels are influenced by all of these components. Therefore, flexibility is the most important aspect of a tunneling protocol if it is to keep pace with the evolution of technology. This document describes Geneve, an encapsulation protocol designed to recognize and accommodate these changing capabilities and needs. This document introduces new IP Flow Information Export (IPFIX) Information Elements (IEs) to identify a set of information related to Segment Routing over IPv6 (SRv6) such as data contained in a Segment Routing Header (SRH), the SRv6 control plane, and the SRv6 Endpoint behavior that traffic is being forwarded with. ZTE Corporation ZTE Corporation Everything OPS Cisco Old Dog Consulting Nokia Blue Fern Consulting ZTE New Protocols or Protocol Extensions are best designed with due consideration of the functionality needed to operate and manage them. Retrofitting operations and management considerations is suboptimal. The purpose of this document is to provide guidance to authors and reviewers on what operational and management aspects should be addressed when defining New Protocols or Protocol Extensions. This document obsoletes RFC 5706, replacing it completely and updating it with new operational and management techniques and mechanisms. It also introduces a requirement to include an "Operational Considerations" section in new RFCs in the IETF Stream. IANA

Migration from Positional to Generic Information Elements This appendix is informative. It provides guidance for migrating existing deployments from positional Information Elements to generic Information Elements used with the ordered export mechanism specified in this document. Positional Information Elements were defined as a workaround for the absence of an ordering guarantee in IPFIX (see ). With ordered export, a single generic Information Element repeated N times in an Ordered Template Record replaces N positional Information Elements, where the N-th occurrence corresponds to the N-th observation at the Observation Point. Migration requires both Exporting Process and Collecting Process updates to support Ordered Template Sets (Set ID 4) or Ordered Options Template Sets (Set ID 5). Until ordered export is universally deployed, both positional and generic forms must be supported concurrently.

MPLS Information Elements The following positional IPFIX IEs for MPLS label stacks and label attributes are replaced by the generic IEs defined in this document when used with ordered export.

Positional IE	IE ID	Generic IE	IE ID	Notes
mplsTopLabelType	46	mplsLabelType	TBD2	Top-of-stack only; generic IE covers position N
mplsTopLabelIPv4Address	47	mplsLabelIPv4Address	TBD3	Top-of-stack only; generic IE covers position N
mplsTopLabelStackSection	70	mplsLabelStackSection	TBD1	Occurrence 1 (top of stack)
mplsLabelStackSection2	71	mplsLabelStackSection	TBD1	Occurrence 2
mplsLabelStackSection3	72	mplsLabelStackSection	TBD1	Occurrence 3
mplsLabelStackSection4	73	mplsLabelStackSection	TBD1	Occurrence 4
mplsLabelStackSection5	74	mplsLabelStackSection	TBD1	Occurrence 5
mplsLabelStackSection6	75	mplsLabelStackSection	TBD1	Occurrence 6
mplsLabelStackSection7	76	mplsLabelStackSection	TBD1	Occurrence 7
mplsLabelStackSection8	77	mplsLabelStackSection	TBD1	Occurrence 8
mplsLabelStackSection9	78	mplsLabelStackSection	TBD1	Occurrence 9
mplsLabelStackSection10	79	mplsLabelStackSection	TBD1	Occurrence 10
mplsTopLabelPrefixLength	91	mplsLabelPrefixLength	TBD4	Top-of-stack only; generic IE covers position N
mplsTopLabelIPv6Address	140	mplsLabelIPv6Address	TBD5	Top-of-stack only; generic IE covers position N
mplsTopLabelTTL	200	mplsLabelStackSection	TBD1	TTL is encoded in octet 4 of the 4-octet label stack entry
mplsTopLabelExp	203	mplsLabelStackSection	TBD1	TC (Exp) is encoded in bits 21-23 of the 4-octet label stack entry
postMplsTopLabelExp	237	(not in scope)	--	Post-policy TC; no generic replacement defined in this document

IEEE 802.1ad Q-in-Q Information Elements The following positional IPFIX IEs for IEEE 802.1ad Q-in-Q (stacked VLAN tags) are replaced by repeated use of the existing vlanId (IPFIX IE 58) and postVlanId (IPFIX IE 59) Information Elements in an Ordered Template Record. No new generic IPFIX IEs are required for VLAN IDs.

Positional IE	IE ID	Generic IE	IE ID	Notes
dot1qVlanId	243	vlanId	58	Occurrence 1 (outer S-TAG)
dot1qPriority	244	(not in scope)	--	No generic priority IE defined in this document
dot1qCustomerVlanId	245	vlanId	58	Occurrence 2 (inner C-TAG)
dot1qCustomerPriority	246	(not in scope)	--	No generic priority IE defined in this document
postDot1qVlanId	254	postVlanId	59	Occurrence 1 (outer S-TAG, post-NAT)
postDot1qCustomerVlanId	255	postVlanId	59	Occurrence 2 (inner C-TAG, post-NAT)

Acknowledgements The authors thank Yao Liu and Taoran Zhou for their work on , which identified the multi-layer encapsulation use case and motivated the broader problem statement addressed in this document.