| Internet-Draft | Augmented-by Addition into the IETF-YANG | July 2024 |
| Lin, et al. | Expires 6 January 2025 | [Page] |
This document augments the ietf-yang-library to provide the augmented-by list. It facilitates the process of obtaining the entire dependencies between YANG modules, by directly querying the server's YANG module.¶
This note is to be removed before publishing as an RFC.¶
Source for this draft and an issue tracker can be found at https://github.com/Zephyre777/draft-lincla-netconf-yang-library-augmentation.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 6 January 2025.¶
Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The YANG Library [RFC8525] specifies a YANG module that provides the information about the YANG modules and datastores to facilitate a client application to fully utilize and understand the YANG data modelling language. To know the YANG dependencies, [RFC8525] has defined and provided the submodule list and the YANG modules deviation list. However, the YANG modules augmentation is not provided.¶
According to [RFC7950], both augmentations and deviations are defining contents external to the model, but applying internally for the model. It is important to know the augmentation and deviation as they are dependencies between modules, but it is also difficult because they are defined externally. When we try to use the ietf-yang-library in [RFC8525] to obtain the reverse dependencies (i.e., augmentations and deviations), the augmentations are not defined in it.¶
However, both the augmentation and the deviation work as YANG module dependencies. Therefore, it is reasonable to document them the same way in the IETF YANG Library. Besides, it will be easier to determine the reverse dependency if the augmentation is directly available in the YANG Library.¶
This draft augments the ietf-yang-library YANG module to include the YANG module augmentation information.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The terminology from [RFC8525] is used in this document¶
Tree diagrams in this document use the notation defined in [RFC8340] .¶
When using a YANG module, it is necessary to make sure that all its dependencies are presented. [RFC7950] identifies four types of dependencies between YANG modules:¶
The import and include are direct dependencies while the augmentation and deviation are reverse dependencies. To know the direct dependencies of specific YANG module, we can parse this YANG module as the dependencies are directly specified (import and include statements").¶
As for the reverse dependencies, since they are defined externally, we cannot parse the YANG module itself to discover them. Among all the methods for finding reverse dependencies, getting the content from YANG Library is one of the most convenient ways.¶
However, YANG Library only provides the deviation list, but not the augmentation. Thus, YANG Library could be extended to provide the augmentation information since both augmentation and deviation have a similar way of working (both are applied to the original module but invisible to them). A noticeable difference between deviations and augmentations is that deviations are required to understand the API contract between the client and the server. But with some use cases arise as the requirement of automate network management, the augmentation becomes essential information for understanding the network.¶
As the demand for YANG-based telemetry [RFC8641] arises, there is a need for real-time knowledge of a specific YANG module's dependency list when a specific YANG-Push notification is received.¶
The alternative for a YANG-Push receiver is to collect and store the entire module set for every single server who could be streaming data. This approach is not always practical due to the following reasons:¶
This section introduces two use cases that reflect the motivation for extending YANG Library. One targets solving dependency problems in a data mesh telemetry system while the other aims at building a data catalog that makes YANG module information easily accessible.¶
A network analytics architecture that integrates YANG-Push and Kafka is proposed in 2022 and is continuously growing and gaining influence, refer to the draft: An Architecture for YANG-Push to Apache Kafka Integration [I-D.netana-nmop-yang-kafka-integration]. This open-source project encompasses contributions such as Support of Versioning in YANG Notifications Subscription [I-D.ietf-netconf-yang-notifications-versioning] or Support of Network Observation Timestamping in YANG Notifications [I-D.netconf-tgraf-yang-push-observation-time], among others.¶
The purpose of this project is to provide adequate information in the YANG-Push notification so that when it is received, the module and its dependency can be parsed and found automatically from the vantage point. The architecture relies on the information of YANG module and their dependency to realize, as one of its main goals is to solve the problem of missing YANG semantics when data is received in Time Series Database in the end. To solve the problem, a schema registry is introduced to store YANG modules and all their relationships (direct and reverse dependencies). The schema is obtained by the NETCONF <get-schema> of the subscribed YANG module, which is obtained by parsing the <subscription-started> message of each YANG-Push subscription.¶
The scope of this draft is limited to configured subscriptions as defined in Section 2.5 of [RFC8639]. Configured subscriptions are configured by a YANG client on the YANG server via the supported network protocol. In this scenario, once the subscription is set up, the YANG-Push notification (or event record) is sent over the connections specified by the transport and receiver of the configured subscription. This technique differs from dynamic subscriptions, where the notification messages are sent over the session that has been used to establish the subscription.¶
Section 3 of draft [I-D.netana-nmop-yang-kafka-integration], defines a separate network orchestrator and data collector in its architecture, which means subscription and data collection are done separately. Therefore, only configured subscription, with which user can configure the subscription from one YANG client and receive the telemetry data in another YANG collector indicated in the subscription, could work with this architecture.¶
As a method for massively streaming telemetry data, the UDP-based Transport for configured Subscription defined in draft [I-D.ietf-netconf-udp-notif](UDP-notif) has been applied in [I-D.netana-nmop-yang-kafka-integration] as the transport method and streaming message type. With the same spirit as applying the configured subscription, the UDP-notif has introduced more flexibility into the architecture by defining useful metadata in the message content such as the receiver address, port etc. In this way, at the same time when the Data Mesh architecture is handling massive data, it has the ability to trace the publisher of each message.¶
By explaining the above, we have gone back to the beginning of this section, where we explained the schema registry, that contains the YANG modules concerned in each YANG-Push subscription which are obtained by NETCONF <get-schema> operation. UDP-notif has provided the ability to know the publisher of message. Therefore, an independent process containing multiple <get-schema> operations is launched after each new YANG-Push subscription module has been known. However, the complexity still remains at:¶
Currently, the method used for obtaining modules and finding module dependencies is "get-all-schemas", where the YANG client retrieves all YANG modules from the network device to enable later the client can fully understand and utilize all modules and module dependencies of device. This process is very heavy because in a real situation, each device may implement hundreds of YANG modules, requiring up to several minutes to complete, in the worse cases. Besides, the need of parsing all YANG modules and finding all the dependencies adds a small extra delay. Applying this method to obtain YANG module will make the operation very costly, since after each subscribed module is learned, "get-all-schemas" needs to be re-performed.¶
Therefore, considering the telemetry real-time aspects, this extra delay in collecting (and processing) the dependencies through a get- all-schemas approach is not ideal.¶
It's more efficient to get dependencies only for the required modules in the telemetry.¶
By using the provided the augmentation information in ietf-yang-library, the collector can directly obtain the YANG reverse dependencies by fetching the contents of YANG Library, saving collection (and processing time) at the collector, and therefore helping with the near real-time aspects of the closed loop action.¶
Finding the YANG modules implemented by a network device is paramount for configuring and monitoring the status of a network. However, since the inception of YANG the network industry has experienced a tsunami of YANG modules developed by SDOs, open-source communities, and network vendors. This heterogeneity of YANG modules, that vary from one network device model to another, makes the management of a multi-vendor network a big challenge for operators. [Martinez-Casanueva2023]¶
In this regard, a data catalog provides a registry of the datasets exposed by remote data sources for consumers to discover data of interest. Besides the location of the dataset (i.e., the data source), the data catalog registers additional metadata such as the data model (or schema) followed in the dataset or even related terms defined in a business glossary.¶
Data catalog solutions typically implement collectors that ingest metadata from the data sources themselves and also external metadata sources. For example, a Kafka Schema Registry is a metadata source that provides metadata about the data models followed by some data stored in a Kafka topic.¶
In this sense, a YANG-enabled network device can be considered as another kind of data source, which the Data Catalog can pull metadata from. For instance, the data catalog can include a connector that fetches metadata about the YANG modules implemented by the network device. Combining these metadata with other such as the business concept "interface", would enable data consumers to discover which datasets related to the concept "interface" are exposed by the network device.¶
Network devices that implement YANG Library expose metadata about which YANG modules are implemented, and which are only imported. However, what a data consumer needs at the end are the YANG modules implemented by the device, hence, the combination of implemented YANG modules with other YANG modules that might deviate or augment the formers.¶
Coming back to the example of datasets related to the "interface" concept, say we have a network device that implements the ietf-interfaces module [RFC8343] and the ietf-ip module [RFC8344], where the latter augments the former. For a data catalog to collect these metadata, a connector would retrieve YANG Library data from the target device. However, the current version of YANG Library would not satisfy the use case as it would tell that the device implements both ietf-interfaces and ietf-ip modules, but will miss the augment dependency between them.¶
The current workaround to this limitation is to, in combination with the YANG Library data, additionally fetch both YANG modules and process them to discover that there is an augment dependency. This adds extra burden on the connector, which is forced to combine multiple metadata collection mechanisms. This process could be softened by extending YANG Library to also capture augment dependencies, in a similar fashion to deviation dependencies.¶
This YANG module augments the ietf-yang-library module by adding the augmented-by list in the "yang-library/module-set". The name "augmented-by" indicates the modules by which the current module is being augmented. Note that this module only augments the ietf-yang-library defined in [RFC8525]. At the time of writing this document, most vendors support [RFC7895], a previous revision of the ietf-yang-library YANG module; The module that augments [RFC7895] is provided in the Appendix B.¶
The following is the YANG tree diagram for model ietf-yang-library-augmentedby.¶
module: ietf-yang-library-augmentedby
augment /yanglib:yang-library/yanglib:module-set/yanglib:module:
+--ro augmented-by* -> ../../yanglib:module/name
¶
The following is the YANG tree diagram[RFC8340] for the ietf-yang-library with the augmentation defined in module ietf-yang-library-augmentedby, including the RPCs and notifications.¶
module: ietf-yang-library
+--ro yang-library
| +--ro module-set* [name]
| | +--ro name string
| | +--ro module* [name]
| | | +--ro name yang:yang-identifier
| | | +--ro revision? revision-identifier
| | | +--ro namespace inet:uri
| | | +--ro location* inet:uri
| | | +--ro submodule* [name]
| | | | +--ro name yang:yang-identifier
| | | | +--ro revision? revision-identifier
| | | | +--ro location* inet:uri
| | | +--ro feature* yang:yang-identifier
| | | +--ro deviation* -> ../../module/name
| | | +--ro yanglib-aug:augmented-by*
-> ../../yanglib:module/name
| | +--ro import-only-module* [name revision]
| | +--ro name yang:yang-identifier
| | +--ro revision union
| | +--ro namespace inet:uri
| | +--ro location* inet:uri
| | +--ro submodule* [name]
| | +--ro name yang:yang-identifier
| | +--ro revision? revision-identifier
| | +--ro location* inet:uri
| +--ro schema* [name]
| | +--ro name string
| | +--ro module-set* -> ../../module-set/name
| +--ro datastore* [name]
| | +--ro name ds:datastore-ref
| | +--ro schema -> ../../schema/name
| +--ro content-id string
x--ro modules-state
x--ro module-set-id string
x--ro module* [name revision]
x--ro name yang:yang-identifier
x--ro revision union
+--ro schema? inet:uri
x--ro namespace inet:uri
x--ro feature* yang:yang-identifier
x--ro deviation* [name revision]
| x--ro name yang:yang-identifier
| x--ro revision union
x--ro conformance-type enumeration
x--ro submodule* [name revision]
x--ro name yang:yang-identifier
x--ro revision union
+--ro schema? inet:uri
notifications:
+---n yang-library-update
| +--ro content-id -> /yang-library/content-id
x---n yang-library-change
x--ro module-set-id -> /modules-state/module-set-id
¶
The YANG module source code of ietf-yang-library-augmentedby in which augmentation to the ietf-yang-library of [RFC8525] is defined.¶
<CODE BEGINS> file "ietf-yang-library-augmentedby@2023-10-27.yang"
module ietf-yang-library-augmentedby {
yang-version 1.1;
namespace "urn:ietf:params:xml:ns:yang:ietf-yang-library-augmentedby";
prefix yanglib-aug;
import ietf-yang-library {
prefix yanglib;
reference
"RFC 8525: YANG Library";
}
organization
"IETF NETCONF (Network Configuration) Working Group";
contact
"WG Web: <https://datatracker.ietf.org/wg/netconf/>
WG List: <mailto:netconf@ietf.org>
Author: Zhuoyao Lin
<mailto:zhuoyao.lin1@huawei-parteners.com>
Benoit Claise
<mailto:benoit.claise@huawei.com>
IGNACIO DOMINGUEZ MARTINEZ-CASANUEVA
<matilto:ignacio.dominguezmartinez@telefonica.com>";
description
"This module augments the ietf-yang-library defined in
[RFC8525] to provide not only the deviation list, but also
the augmented-by list, in order to give sufficient
information about the YANG modules reverse dependency. It
facilitates the process of obtaining the entire
dependencies of YANG module.
The key words 'MUST', 'MUST NOT', 'REQUIRED', 'SHALL',
'SHALL NOT', 'SHOULD', 'SHOULD NOT', 'RECOMMENDED',
'NOT RECOMMENDED', 'MAY', and 'OPTIONAL' in this document
are to be interpreted as described in BCP 14 (RFC 2119)
(RFC 8174) when, and only when, they appear in all
capitals, as shown here.
Copyright (c) 2022 IETF Trust and the persons identified as
authors of the code. All rights reserved.
Redistribution and use in source and binary forms, with or
without modification, is permitted pursuant to, and subject
to the license terms contained in, the Revised BSD License
set forth in Section 4.c of the IETF Trust's Legal Provisions
Relating to IETF Documents
(https://trustee.ietf.org/license-info).
This version of this YANG module is part of RFC XXXX; see the
RFC itself for full legal notices. ";
revision 2023-10-27 {
description
"Added list augmented-by in yang-library/module-set/module to
make the module store the entire reverse dependency information
(augmented-by and deviation).";
reference
"RFC XXXX: Support of augmentedby in ietf-yang-library";
}
augment "/yanglib:yang-library/yanglib:module-set/yanglib:module" {
description
"Augment the augmented-by list from module info with the
module-augmented-by grouping" ;
leaf-list augmented-by {
type leafref {
path "../../yanglib:module/yanglib:name";
}
description
"Leaf-list of the augmentation used by this server to
modify the conformance of the module associated with
this entry. Note that the same module can be used for
augmented-by for multiple modules, so the same
entry MAY appear within multiple 'module' entries.
This reference MUST NOT (directly or indirectly)
refer to the module being augmented.
Robust clients may want to make sure that they handle a
situation where a module augments itself (directly or
indirectly) gracefully.";
}
}
}
<CODE ENDS>¶
Note to the RFC-Editor: Please remove this section before publishing (This follows the template in RFC7942).¶
Zhuoyao Lin did the prototype implementation of the augmented-by list feature of this draft and demonstrated it based on Netopeer2 in IETF 119 Hackathon.¶
Netopeer2 is a NETCONF server & client implementation developed by CESNET. Source code is here: [NTP17]. The actual feature is implemented by extending the libyang [LY16] and sysrepo [SR16] which are the base libraries for Netopeer2 to support populating the augmented-by list.¶
The list name has been updated from "augmentation" to "augmented-by", in order to represent the usage clearly.¶
The leafref has been changed from absolute path "/yanglib:yang-libraray/yanglib:module-set/yanglib:module/yanglib:name" to relative path "../../yanglib:module/yanglib:name". The YANG validation in the appendix A shows that this path can work as expected.¶
Section 5 Implementation and section 6 Changes has been added.¶
Updated the Use case content in Section 3.1. Add explanation: the scope of use case "Data Mesh Architecture" is limited to configured subscription.¶
Updated Implementation status content.¶
Updated affiliations¶
Update content of Section 3.1 Data Mesh use case. Explain the limitation of applying get-all-schemas solution under the background of using UDP-notif of configured subscription, and how the feature proposed in the draft can improve the solution.¶
Full review of document. Nits and refinement of sections.¶
This document has no actions for IANA.¶
This section gives a few examples that the user can try themselves with yanglint. This is created to prove the syntax correctness.¶
The valid example should pass the validation while the invalid one will not, because the module has augmented a module in another module-set, which is illegal according to the YANG source code.¶
<CODE BEGINS> file "example_valid.xml"
<yang-library xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library">
<content-id>1</content-id>
<module-set>
<name>ms1</name>
<module>
<name>module1</name>
<revision>2024-02-29</revision>
<namespace>urn:ietf:params:xml:ns:yang:module1</namespace>
<augmented-by
xmlns="urn:ietf:params:xml:ns:yang:
ietf-yang-library-augmentedby">module2</augmented-by>
<augmented-by
xmlns="urn:ietf:params:xml:ns:yang:
ietf-yang-library-augmentedby">module3</augmented-by>
</module>
<module>
<name>module2</name>
<revision>2024-02-29</revision>
<namespace>urn:ietf:params:xml:ns:yang:module2</namespace>
</module>
<module>
<name>module3</name>
<revision>2024-02-29</revision>
<namespace>urn:ietf:params:xml:ns:yang:module3</namespace>
</module>
</module-set>
</yang-library>
<modules-state xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library">
<module-set-id>0</module-set-id>
</modules-state>
<CODE ENDS>¶
<CODE BEGINS> file "example_invalid.xml"
<yang-library xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library">
<content-id>1</content-id>
<module-set>
<name>ms1</name>
<module>
<name>module1</name>
<revision>2024-02-29</revision>
<namespace>urn:ietf:params:xml:ns:yang:module1</namespace>
<augmented-by
xmlns="urn:ietf:params:xml:ns:yang:
ietf-yang-library-augmentedby">module2</augmented-by>
<augmented-by
xmlns="urn:ietf:params:xml:ns:yang:
ietf-yang-library-augmentedby">module3</augmented-by>
</module>
<module>
<name>module2</name>
<revision>2024-02-29</revision>
<namespace>urn:ietf:params:xml:ns:yang:module2</namespace>
</module>
<module>
<name>module3</name>
<revision>2024-02-29</revision>
<namespace>urn:ietf:params:xml:ns:yang:module3</namespace>
</module>
</module-set>
</yang-library>
<modules-state xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library">
<module-set-id>0</module-set-id>
</modules-state>
<CODE ENDS>¶
This section defines the ietf-yang-library-rfc7895-augmentedby that augments the ietf-yang-library defined in [RFC7895]. The module-state/module list of this YANG module version is also defined in the [RFC8525] version though deprecated.¶
The following is the YANG tree diagram for ietf-yang-library-rfc7895-augmentedby augmenting RFC7895.¶
module: ietf-yang-library-rfc7895-augmentedby
augment /yanglib:modules-state/yanglib:module:
x--ro augmentedby* [name revision]
+--ro name -> /yanglib:modules-state/module/name
+--ro revision -> /yanglib:modules-state/module/revision
¶
The following is the full YANG tree diagram of ietf-yang-library-rfc7895-augmentedby augmenting ietf-yang-library defined in RFC7895.¶
module: ietf-yang-library
+--ro modules-state
+--ro module-set-id string
+--ro module* [name revision]
+--ro name yang:yang-identifier
+--ro revision union
+--ro schema? inet:uri
+--ro namespace inet:uri
+--ro feature* yang:yang-identifier
+--ro deviation* [name revision]
| +--ro name yang:yang-identifier
| +--ro revision union
+--ro conformance-type enumeration
+--ro submodule* [name revision]
| +--ro name yang:yang-identifier
| +--ro revision union
| +--ro schema? inet:uri
x--ro yanglib-aug:augmented-by* [name revision]
+--ro yanglib-aug:name
-> /yanglib:modules-state/module/name
+--ro yanglib-aug:revision
-> /yanglib:modules-state/module/revision
notifications:
+---n yang-library-change
+--ro module-set-id -> /modules-state/module-set-id
¶
The YANG module that augments the ietf-yang-library RFC7895.¶
<CODE BEGINS> file "ietf-yang-library-rfc7895-augmentedby@2023-10-27.yang"
module ietf-yang-library-rfc7895-augmentedby {
yang-version 1.1;
namespace "urn:ietf:params:xml:ns:yang:ietf-yang-library-rfc7895-augmentedby";
prefix yanglib-aug;
import ietf-yang-library {
prefix yanglib;
revision-date 2016-06-21;
reference
"RFC 7895: YANG Module Library.";
}
organization
"IETF NETCONF (Network Configuration) Working Group";
contact
"WG Web: <https://datatracker.ietf.org/wg/netconf/>
WG List: <mailto:netconf@ietf.org>
Author: Zhuoyao Lin
<mailto:zhuoyao.lin1@huawei-partners.com>
Author: Benoit Claise
<mailto:benoit.claise@huawei.com>
Author: IGNACIO DOMINGUEZ MARTINEZ-CASANUEVA
<matilto:ignacio.dominguezmartinez@telefonica.com>";
description
"This document augments the ietf-yang-library to provide the
augmented-by list. It facilitates the process of obtaining
the entire dependencies between YANG modules, by directly
querying the server's YANG module.
The key words 'MUST', 'MUST NOT', 'REQUIRED', 'SHALL',
'SHALL NOT', 'SHOULD', 'SHOULD NOT', 'RECOMMENDED',
'NOT RECOMMENDED', 'MAY', and 'OPTIONAL' in this document
are to be interpreted as described in BCP 14 (RFC 2119)
(RFC 8174) when, and only when, they appear in all
capitals, as shown here.
Copyright (c) 2022 IETF Trust and the persons identified as
authors of the code. All rights reserved.
Redistribution and use in source and binary forms, with or
without modification, is permitted pursuant to, and subject
to the license terms contained in, the Revised BSD License
set forth in Section 4.c of the IETF Trust's Legal Provisions
Relating to IETF Documents
(https://trustee.ietf.org/license-info).
This version of this YANG module is part of RFC XXXX; see the
RFC itself for full legal notices. ";
revision 2023-10-27 {
description
"Added list augmentedby in yang-library/modules-state/module to
make the module store the entire reverse dependency information
(augmentedby and deviation).";
reference
"RFC XXXX: Support of augmentedby in ietf-yang-library
defined in RFC7895";
}
augment "/yanglib:modules-state/yanglib:module" {
description
"Augment the augmentedby from module info with the
module-augmented-by grouping" ;
uses yanglib-aug:module-state-augmented-by;
}
/*
* Groupings
*/
grouping module-state-augmented-by {
description
"This grouping defines a list with keys being the module
name and revison. The list contains the augmented-by list.";
list augmented-by {
key "name revision";
status deprecated;
description
"List of YANG augmented-by module names and revisions
used by this server to modify the conformance of
the module associated with this entry. Note that
the same module can be used for augmented-by for
multiple modules, so the same entry MAY appear
within multiple 'module' entries.
The augment module MUST be present in the 'module'
list, with the same name and revision values.
The 'conformance-type' value will be 'implement' for
the augment module.";
leaf name {
type leafref {
path "/yanglib:modules-state/yanglib:module/yanglib:name";
}
description
"Identifies a given module in the YANG Library by
its name.";
}
leaf revision {
type leafref {
path "/yanglib:modules-state/yanglib:module/yanglib:revision";
}
description
"Revision of the module";
}
}
}
}
<CODE ENDS>¶
The following people all contributed to creating this document:¶
The author would like to thanks Jan Lindblad and Jean Quilbeuf for his help during the design of the YANG module.¶