]> Hammerspace

loghyr@gmail.com

General Network File System Version 4 Internet-Draft Network File System version 4.2 (NFSv4.2) clients may cache directory-entry (dirent) metadata returned by READDIR to improve performance. In some deployments, the rate at which directory contents change at the server exceeds the validity of typical NFSv4.2 client caches, so that a client serving READDIR responses from its local cache returns entries or attribute values that no longer reflect the current state of the directory. This document introduces an uncacheable dirent metadata attribute for NFSv4.2 that allows a server to indicate that a client honouring the attribute is required to retrieve directory-entry metadata from the server on each READDIR rather than serving the response from a local cache. Note to Readers Note to RFC Editor: please remove this section prior to publication. Discussion of this draft takes place on the NFSv4 working group mailing list (nfsv4@ietf.org), which is archived at . Source code and issues list for this draft can be found at . Working Group information can be found at .

Introduction Clients of remote filesystems may cache directory-entry metadata to improve performance. This caching presumes that directory contents and the attributes of its entries remain stable for the duration of a client cache lifetime. In some deployments that presumption does not hold; the conditions under which this is the case for the deployment that motivates this work are described in . In this document, the term directory is used to describe the context in which directory entries are retrieved. The uncacheable dirent metadata attribute applies to the caching of directory-entry metadata, including names and associated file object metadata such as size and timestamps. It does not prohibit caching of the directory object itself, nor does it affect caching of file data. Caching of directory-entry metadata can result in incorrect size and timestamp information when files are modified concurrently, reducing the effectiveness of uncacheable file data semantics () when directory-entry metadata is stale. This can lead to applications observing inconsistent metadata and data views even when file data caching is disabled. This document introduces the uncacheable dirent metadata attribute to NFSv4.2 to allow servers to advise clients that caching of directory-entry metadata is unsuitable. Using the process detailed in , the revisions in this document become an extension of NFSv4.2 . They are built on top of the external data representation (XDR) generated from .

Deployment Motivation A class of deployment exposes a single file namespace concurrently through multiple file-access protocols, typically including NFSv4.2, NFSv3, and the Server Message Block (SMB) protocol family . The same directories are reachable from all of those protocols and, in many such deployments, from server-side policy components (placement, replication, archival, tiering) that act on the namespace without a client connection. Changes to a directory's contents -- entries added, entries removed, entries whose attributes have been updated -- may therefore originate at any time from any of those sources. NFSv4.2 client implementations typically cache directory-entry metadata returned by READDIR for a period bounded by either the directory's change attribute or a heuristic timeout. In deployments of the kind described above, the rate of server-side directory churn can exceed what those caches can track, so that an NFSv4.2 client serving READDIR responses from its local cache will, with some regularity, return entries or attribute values that no longer reflect the current state of the directory at the server. The fattr4_uncacheable_dirent_metadata attribute is the server's mechanism to identify a directory for which this risk is high enough that client-side caching is not safe. When the server sets the attribute on a directory, an honouring client retrieves directory-entry metadata from the server on each READDIR rather than from a local cache. In the deployments that motivate this work, no attribute values returned for a given dirent vary across users. The server returns the same set of entries with the same attribute values to all NFSv4 clients of a given directory regardless of the requesting user's identity. The SMB protocol family includes a related concept called Access-Based Enumeration (ABE) , in which an SMB server filters the visible set of directory entries by the requesting user's access rights. ABE is named here as background for readers familiar with the SMB ecosystem. The NFSv4.2 attribute defined in this document is not an NFS surface for ABE: it does not require, specify, or depend on per-user directory filtering, and the servers that motivate this work do not apply such filtering to NFSv4.2 READDIR responses.

Definitions

Access Based Enumeration (ABE)

A Server Message Block (SMB) feature in which the server filters the visible set of directory entries by the requesting user's access rights . Used in this document only as background for readers familiar with the SMB ecosystem; the NFSv4.2 attribute defined here is not an NFS surface for ABE.

dirent

A directory-entry representing a file or subdirectory and its associated attributes.

dirent caching

A client-side cache of directory-entry names and associated file object metadata, used to avoid repeated directory lookup and attribute retrieval.

uncacheable dirent metadata attribute

An NFSv4.2 file attribute that advises clients not to cache directory-entry metadata associated with file objects, including names, size, and timestamps.

This document assumes familiarity with NFSv4.2 operations, attributes, and error handling as defined in and .

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Caching of Directory-Entry Metadata The fattr4_uncacheable_dirent_metadata attribute is a read-write boolean attribute that applies to directory objects and has a data type of boolean. The attribute is not set on individual file objects and applies only to directory-entry metadata returned from the directory on which it is set. The uncacheable dirent metadata attribute enables correct presentation of directory-entry metadata in deployments where directory contents may change at the server at a rate that exceeds the validity of typical client caches. As such, it is an OPTIONAL attribute to implement for NFSv4.2. If both the client and the server support this attribute, and the attribute is set on a directory, the client MUST retrieve directory-entry metadata from the server on each READDIR rather than serving the response from a local cache. This document specifies the required observable behavior rather than mandating a particular internal implementation strategy. Clients MAY employ more sophisticated mechanisms, such as time-limited caches that revalidate against the server on each READDIR, provided that the externally visible behavior is equivalent to retrieving directory-entry metadata from the server on each READDIR. Allowing clients to set this attribute provides a portable mechanism to request that directory-entry metadata not be cached, without requiring changes to application behavior or out-of-band administrative configuration. A client can determine whether the uncacheable dirent metadata attribute is supported for a given directory by examining the supported_attrs attribute for that directory's filesystem or by probing support using the procedures described in . The only way that the server can determine that the client supports the attribute is if the client sends either a GETATTR or a SETATTR with the uncacheable dirent metadata attribute. The uncacheable dirent metadata attribute governs caching behavior of directory-entry metadata returned by READDIR and related operations, not the directory object itself. The uncacheable dirent metadata attribute addresses a different aspect of client-side caching than fattr4_uncacheable_file_data (). The file data attribute governs caching of file contents, while the dirent metadata attribute governs caching of directory-entry metadata returned by READDIR and related operations. The attributes are independent and may be used separately. This attribute does not define behavior for positive or negative name caching or for caching of LOOKUP results outside the scope of directory-entry metadata returned by READDIR and related operations. Directory delegations grant a client exclusive caching rights subject to server recall. In deployments where directory contents change at a rate that makes per-change recall impractical, a directory delegation does not provide the always-refetch semantics defined by the uncacheable dirent metadata attribute. These mechanisms are independent. Clients MUST NOT assume that directory-entry metadata is valid beyond the READDIR that produced it.

Uncacheable Directory-Entry Metadata The fattr4_uncacheable_dirent_metadata attribute is a read-write boolean attribute that applies to directory objects. Authorization to query or modify this attribute is governed by existing NFSv4.2 authorization mechanisms. If a directory object has the uncacheable dirent metadata attribute set, the client MUST retrieve directory-entry metadata from the server on each READDIR rather than serving the response from a local cache. This ensures that the returned metadata reflects the current state of the directory as determined by the server. The uncacheable dirent metadata attribute does not modify the semantics of the NFSv4.2 change attribute. Clients MUST continue to use the change attribute to detect directory modifications and to determine when directory contents may have changed, even when directory-entry metadata caching is suppressed. Suppressing caching of directory-entry metadata does not remove the need for change-based validation. Servers SHOULD assume that clients which do not query or set this attribute may cache directory-entry metadata, and therefore SHOULD NOT rely on this attribute for correctness unless client support is confirmed. A directory delegation grants a client the right to cache directory-entry metadata until the server recalls the delegation. The always-refetch rule of this attribute is incompatible with that grant. If a directory has both the uncacheable dirent metadata attribute set and an outstanding directory delegation, the server MUST recall the delegation, after which the client follows the always-refetch rule on each subsequent READDIR. A server MUST NOT grant a new directory delegation on a directory while the uncacheable dirent metadata attribute is set on that directory.

Example: Directory Enumeration With and Without Dirent Metadata Caching This example illustrates the difference in client-visible behavior when directory-entry metadata caching is enabled versus when the uncacheable dirent metadata attribute is set on a directory. In both scenarios, the set of entries in the directory does not change between the two calls; an attribute value of one entry is updated at the server between calls. The difference is whether the second call observes the updated attribute value.

Classic Directory Enumeration (Directory-Entry Metadata Cached) In this scenario, the client caches directory-entry metadata obtained from the server and reuses it for the second readdir.

Directory-Entry Metadata Cached ------------------------> | entries: | a (size=100) | b (size=200) | c (size=300) |<--------------------<------------------------ | (entries cached in client) (concurrent writer extends a from size=100 to size=500) readdir("/dir") | | (no network traffic) | entries returned from | client cache: | a (size=100) | b (size=200) | c (size=300) ]]>

In this case, shows that directory-entry metadata retrieved by the first READDIR is reused to satisfy the second. The cached response reflects entry a's size as it was at the time of the first call; it does not reflect the update that occurred at the server between calls. This behavior is typical of legacy NFSv4.2 clients and maximizes performance, but it can result in applications observing dirent attribute values that do not reflect the current state of the server.

Directory Enumeration With Uncacheable Dirent Metadata In this scenario, the directory has the uncacheable dirent metadata attribute set. The client retrieves directory-entry metadata from the server on each READDIR.

Directory-Entry Metadata Not Cached ------------------------> | entries: | a (size=100) | b (size=200) | c (size=300) |<--------------------<------------------------ | (no directory-entry metadata retained) (concurrent writer extends a from size=100 to size=500) readdir("/dir") | | READDIR |-------------------->------------------------> | entries: | a (size=500) | b (size=200) | c (size=300) |<--------------------<------------------------ ]]>

In this case, shows that each readdir request results in a READDIR operation sent to the server, so the second call observes the updated size of entry a. The set of entries returned is unchanged between calls; only the attribute value differs. The client may still cache other information, provided the externally observable behavior is equivalent to retrieving directory-entry metadata from the server on each READDIR.

Discussion This example demonstrates that the uncacheable dirent metadata attribute does not mandate a particular client implementation, but it does require the always-refetch behavior specified in . The attribute ensures that NFSv4.2 clients observe directory contents reflecting the current state of the server in deployments where the rate of directory change exceeds the validity of typical client caches.

Implementation Status Note to RFC Editor: please remove this section prior to publication. There is a prototype Hammerspace server which implements the uncacheable dirent metadata attribute and a prototype Linux client which treats the attribute as an indication to retrieve directory- entry metadata from the server on each READDIR rather than from a local cache. In the prototype, directories whose contents change at the server at a rate exceeding typical client cache lifetimes are marked with the fattr4_uncacheable_dirent_metadata attribute. The Linux client decodes the attribute in fs/nfs/nfs4xdr.c into a per-inode flag (nfsi->uncacheable). The readdir path in fs/nfs/dir.c consults this flag to skip the readdir cache and refetch from the server on each readdir call. Clients may employ more sophisticated mechanisms, such as time-limited caches that revalidate against the server on each READDIR, provided that the externally observable behavior matches the always-refetch semantics described in this document. At the time of writing, the prototype encodes the attribute on the wire alongside the companion file-data attribute defined in as a single shared flag. A follow-on prototype branch separates the two attributes as the two documents specify. Experience with the prototype indicates that the attribute enables servers to identify directories whose contents change faster than typical NFSv4.2 client cache lifetimes can track, while remaining compatible with existing NFSv4.2 semantics.

XDR for Uncacheable Dirents Attribute

Extraction of XDR This document contains the external data representation (XDR) description of the uncacheable dirent metadata attribute. The XDR description is presented in a manner that facilitates easy extraction into a ready-to-compile format. To extract the machine-readable XDR description, use the following shell script: For example, if the script is named 'extract.sh' and this document is named 'spec.txt', execute the following command: uncacheable_prot.x ]]> This script removes leading blank spaces and the sentinel sequence '///' from each line. XDR descriptions with the sentinel sequence are embedded throughout the document. Note that the XDR code contained in this document depends on types from the NFSv4.2 nfs4_prot.x file (generated from ). This includes both nfs types that end with a 4, such as offset4, length4, etc., as well as more generic types such as uint32_t and uint64_t. While the XDR can be appended to that from , the code snippets should be placed in their appropriate sections within the existing XDR.

Security Considerations This attribute is not a security mechanism. It addresses correctness of client-side caching when client-cached directory-entry metadata can become stale relative to the current state of the directory at the server. It does not change NFSv4.2 authentication or authorization semantics, and it does not impose access controls on the entries it describes. Authorization to set or modify the fattr4_uncacheable_dirent_metadata attribute is governed by existing NFSv4.2 authorization mechanisms. Servers MAY restrict modification of this attribute based on local policy, file ownership, or access control rules. This document does not define a new authorization model. If the client supports Labeled NFS (see for background), the client MUST locally enforce the MAC security policies defined by NFSv4.2 (, Section 9). This obligation is independent of whether directory-entry metadata is cached or refetched. The uncacheable dirent metadata attribute allows servers to indicate that directory-entry metadata should not be assumed to remain valid beyond the READDIR that produced it.

IANA Considerations This document has no IANA actions. Numbered FATTR4 attributes for NFSv4 are assigned by WG coordination through IETF publication rather than by an IANA registry, consistent with the assignment of attributes 81 and 82 in and . Attribute number 88 was selected on the NFSv4 working group mailing list in conjunction with the assignment of number 87 in .

References Normative References This document describes the External Data Representation Standard (XDR) protocol as it is currently deployed and accepted. This document obsoletes RFC 1832. [STANDARDS-TRACK] This document describes NFS version 4 minor version 2; it describes the protocol extensions made from NFS version 4 minor version 1. Major extensions introduced in NFS version 4 minor version 2 include the following: Server-Side Copy, Application Input/Output (I/O) Advise, Space Reservations, Sparse Files, Application Data Blocks, and Labeled NFS. This document provides the External Data Representation (XDR) description for NFS version 4 minor version 2. This document describes the rules relating to the extension of the NFSv4 family of protocols. It covers the creation of minor versions, the addition of optional features to existing minor versions, and the correction of flaws in features already published as Proposed Standards. The rules relating to the construction of minor versions and the interaction of minor version implementations that appear in this document supersede the minor versioning rules in RFC 5661 and other RFCs defining minor versions. This document describes the Network File System (NFS) version 4 minor version 1, including features retained from the base protocol (NFS version 4 minor version 0, which is specified in RFC 7530) and protocol extensions made subsequently. The later minor version has no dependencies on NFS version 4 minor version 0, and is considered a separate protocol. This document obsoletes RFC 5661. It substantially revises the treatment of features relating to multi-server namespace, superseding the description of those features appearing in RFC 5661. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References Hammerspace Network File System version 4.2 (NFSv4.2) clients commonly perform client-side caching of file data in order to improve performance. On some systems, applications may influence client data caching behavior, but there is no standardized mechanism for a server or administrator to indicate that particular file data should not be cached by clients for reasons of performance or correctness. This document introduces a new file data caching attribute for NFSv4.2. Files marked with this attribute are intended to be accessed with client-side caching of file data suppressed, in order to support workloads that require predictable data visibility. This document extends NFSv4.2. This memo outlines high-level requirements for the integration of flexible Mandatory Access Control (MAC) functionality into the Network File System (NFS) version 4.2 (NFSv4.2). It describes the level of protections that should be provided over protocol components and the basic structure of the proposed system. The intent here is not to present the protocol changes but to describe the environment in which they reside. In many environments, inheritable NFSv4 Access Control Entries (ACEs) can be rendered ineffective by the application of the per-process file mode creation mask (umask). This can be addressed by transmitting the umask and create mode as separate pieces of data, allowing the server to make more intelligent decisions about the permissions to set on new files. This document proposes a protocol extension to accomplish that. This document describes an optional feature extending the NFSv4 protocol. This feature allows extended attributes (hereinafter also referred to as xattrs) to be interrogated and manipulated using NFSv4 clients. Xattrs are provided by a file system to associate opaque metadata, not interpreted by the file system, with files and directories. Such support is present in many modern local file systems. New file attributes are provided to allow clients to query the server for xattr support, with that support consisting of new operations to get and set xattrs on file system objects. Microsoft Microsoft Corporation n.d.

Acknowledgments Trond Myklebust, Mike Snitzer, Jon Flynn, Keith Mannthey, and Thomas Haynes all worked on the prototype at Hammerspace. Rick Macklem, Chuck Lever, and Dave Noveck reviewed the document. Chris Inacio, Chuck Lever, Brian Pawlowski, and Gorry Fairhurst helped guide this process.