Parallel NFS (pNFS) Flexible File Layout Version 2

Parallel NFS (pNFS) Flexible File Layout Version 2 Hammerspace

loghyr@gmail.com

General Network File System Version 4 Internet-Draft Parallel NFS (pNFS) allows a separation between the metadata (onto a metadata server) and data (onto a storage device) for a file. The Flexible File Version 2 Layout Type is defined in this document as an extension to pNFS that allows the use of storage devices that require only a limited degree of interaction with the metadata server and use already-existing protocols. Data protection is also added to provide integrity. Both Client-side mirroring and the erasure coding algorithms are used for data protection. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Network File System Version 4 Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at . Note to Readers This is an individual submission and does not reflect Working Group consensus. The "About This Document" section above has the current discussion venue, latest rendering, and source location.

Introduction In Parallel NFS (pNFS) (see ), the metadata server returns layout type structures that describe where file data is located. There are different layout types for different storage systems and methods of arranging data on storage devices. defined the Flexible File Version 1 Layout Type used with file-based data servers that are accessed using the NFS protocols: NFSv3 , NFSv4.0 , NFSv4.1 , and NFSv4.2 . A metadata server that supports the Flexible File Version 2 Layout Type MUST be an NFSv4.2 server. The new operations defined by this document for the metadata server (the TRUST_STATEID family on the metadata server / storage device control session, and the CB_CHUNK_REPAIR back-channel callback to clients) are NFSv4.2 operations and have no representation in NFSv4.1 or earlier minor versions. Storage devices can speak NFSv3, NFSv4.1, or NFSv4.2, but some encoding types and coupling configurations narrow that choice; see for the exact rules. To provide a global state model equivalent to that of the files layout type, a back-end control protocol might be implemented between the metadata server and NFSv4.1+ storage devices. An implementation can either define its own proprietary mechanism or it could define a control protocol in a Standards Track document. The requirements for a control protocol are specified in and clarified in . The control protocol described in this document is based on NFS. It does not provide for knowledge of stateids to be passed between the metadata server and the storage devices. Instead, the storage devices are configured such that the metadata server has full access rights to the data file system and then the metadata server uses synthetic ids to control client access to individual data files. In traditional mirroring of data, the server is responsible for replicating, validating, and repairing copies of the data file. With client-side mirroring, the metadata server provides a layout that presents the available mirrors to the client. The client then picks a mirror to read from and ensures that all writes go to all mirrors. The client only considers the write transaction to have succeeded if all mirrors are successfully updated. In case of error, the client can use the LAYOUTERROR operation to inform the metadata server, which is then responsible for the repairing of the mirrored copies of the file. This client side mirroring provides for replication of data but does not provide for integrity of data. In the event of an error, a user would be able to repair the file by silvering the mirror contents. I.e., they would pick one of the mirror instances and replicate it to the other instance locations. However, lacking integrity checks, silent corruptions are not able to be detected and the choice of what constitutes the good copy is difficult. This document defines the Flexible File Version 2 Layout Type, an independent layout type that adds error-detection integrity (checksum) for erasure coding. It does not modify the Flexible File Version 1 Layout Type (); the two coexist. Data blocks are transformed into a header and a chunk. This document also introduces new operations that allow the client to roll back writes to the data file. Using the process detailed in , the revisions in this document become an extension of NFSv4.2 . They are built on top of the external data representation (XDR) generated from . This document defines LAYOUT4_FLEX_FILES_V2, a new and independent layout type that coexists with the Flexible File Version 1 Layout Type (LAYOUT4_FLEX_FILES, ). The two layout types are wire-format-incompatible: a flexible file v1 layout receiver cannot parse flexible file v2 layout bytes, and a flexible file v2 layout receiver cannot parse flexible file v1 layout bytes. Semantically, however, the flexible file v2 layout is a superset of the flexible file v1 layout: any flexible file v1 layout has a natural flexible file v2 layout representation using a single FFV2ENCODING_PASSTHROUGH mirror (see ), with the flexible file v1 layout's data servers mapped into ffv2_stripes4 and the flexible file v1 layout's layout-level ffl_stripe_unit mapped into per-mirror ffv2m_striping_unit_size and ffv2m_striping. The reverse does not hold: flexible file v2 layouts that use any CHUNK-based encoding (any FFV2_ENCODING* value other than FFV2_ENCODING_PASSTHROUGH) have no flexible file v1 layout representation, because the flexible file v1 layout has neither the chunk envelope (chunk_guard4, per-chunk checksum) nor the per-mirror encoding choice that those encodings require. A server MAY support both layout types simultaneously; a client selects the desired layout type in its LAYOUTGET request.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Motivation Workloads that need both the throughput of parallel pNFS data servers and the durability of erasure coding have driven the work in this draft. The deployments span scientific-instrumentation pipelines that produce petabytes of detector data per year, training-checkpoint files written hot from machine-learning jobs, archive workloads in which a single file may hold the only copy of an experiment's result, and ordinary production filesystems where read patterns evolve across a file's lifetime. These deployments are documented in detail in ; the protocol shape that follows is the result of looking at them and asking what a pNFS layout type would have to provide. The first thing the deployments share is that erasure coding moves work off the data servers that are already the bottleneck in the write-heavy parallel case. Server-side erasure coding makes each data server compute its share of the parity transform on every write, multiplying the per-write CPU cost by (k + m) across the storage tier and serializing on the data server's limited compute. Client-side erasure coding shifts that compute to the writers, which scale horizontally with the workload, and lets the data servers stay close to their strength -- storing and serving bytes. Flexible file v1 layout () already chose client-side compute by placing replication at the writer; this draft extends that choice to client-side erasure coding. Benchmark measurements summarized in confirm that the resulting overhead is competitive with server-side encoding on realistic workloads and that the encoding compute scales with the writer population rather than with the data-server count. Client-side erasure coding has a corollary that protocol designers cannot avoid: when a client fans a stripe out across multiple data servers and fails mid-write, no single data server has whole-transaction visibility. The state left behind on each data server is a partial fragment of a write that may or may not have completed elsewhere. A server-side coordinator that holds the whole stripe -- the flexible file v1 case -- can resilver from a surviving copy without any client involvement. In the v2 case there is no such coordinator, and the on-wire protocol specifies how the partial state is reconciled. This is the load-bearing constraint that shapes the rest of the design. A natural-looking answer is to add a distributed-consensus protocol between the data servers and have them agree on which write committed. That answer is rejected here. Distributed consensus is operationally expensive, introduces a synchronization cost on every write, and makes the data servers themselves stateful peers in a way that closes off the simpler implementations the protocol is designed to accommodate. Instead, this draft uses two narrowly-scoped primitives that together provide just enough on-wire reconciliation: the chunk_guard4 compare-and-swap (CAS) and the CB_CHUNK_REPAIR callback. Every CHUNK_WRITE carries a cohort header (a 64-bit writer-chosen chunk_cohort_id4 plus the writer's 32-bit layout-granted client id) once for the batch, and the data server performs a per-chunk compare-and-swap against its own chunk_guard4 (a 32-bit per-chunk generation counter plus the last writer's client id) when the client sets cwa_guard on receipt. If two writers race for the same chunk, exactly one wins on each data server, the loser receives NFS4ERR_CHUNK_GUARDED for that chunk, and the chunks the loser intended to write are left unchanged. No data server needs to consult its peers; the CAS is local. The cost on the metadata server is bounded by the 12-byte cohort header per CHUNK_WRITE plus a 32-bit per-layout client identifier (, ). Independent collisions on different chunks resolve independently; there is no file-wide lock and no global ordering across writes. When the per-chunk CAS detects that a stripe ended up non-atomic -- some shards under writer A's guard, others under writer B's, or a writer crashed mid-fan-out -- the metadata server selects a repair actor via CB_CHUNK_REPAIR, and that actor drives the repair: it acquires CHUNK_LOCK on the affected range, reads the surviving shards, decodes through the erasure transform, writes the reconstructed shards via CHUNK_WRITE_REPAIR, and clears the errored state via CHUNK_REPAIRED. The repair actor may be a client, a data server (in a tightly coupled deployment), or a proxy server; exactly one holds the chunk-range lock per repair episode so the data servers still do not coordinate among themselves. The repair-actor selection rule is given in . These two primitives -- the per-chunk CAS and the callback-driven repair -- replace what would otherwise require a distributed-consensus protocol. The CAS handles the common case at local cost, where independent writers racing for different chunks resolve independently and concurrent writers on the same chunk get a clean win/loss decision. CB_CHUNK_REPAIR handles the rare case of partial-failure non-atomicity with a single coordinator selected per repair episode. The cost model is asymmetric on purpose: the hot path pays for an 8-byte header and a local CAS; the cold path pays for a selected actor and a small number of round-trips. The CHUNK operation set in this draft is the minimum sufficient to drive the chunk state machine () plus the repair flow above. CHUNK_WRITE places PENDING content; CHUNK_FINALIZE signals that the writer is done with a generation; CHUNK_COMMIT promotes that generation to durable, globally visible state; CHUNK_READ retrieves it; CHUNK_HEADER_READ provides the fast probe that lets repair coordinators and recovering writers inspect chunk metadata without reading payloads. CHUNK_LOCK, CHUNK_UNLOCK, CHUNK_ERROR, CHUNK_REPAIRED, CHUNK_WRITE_REPAIR, and CHUNK_ROLLBACK together drive the repair flow. Each operation does one well-scoped job; the complexity is in the state machine the operations drive, not in the operation set itself. Each of these primitives closes a specific gap in the lifecycle or the repair path. The detailed treatment of the operation set is in . The same design discipline shapes the rest of the specification. The protocol describes wire format and server obligations; it does not pin a data-server backend, a control protocol between metadata server and data server, a checksum algorithm, or a file-attribute representation on the data server. Different implementations resolve these choices differently and remain conformant. TRUST_STATEID () is one such control protocol that this draft defines; storage devices with their own established control protocols are conformant without implementing it. The tagged checksum4 () lets the metadata server pick any registered checksum algorithm per file. The authorization-outcome parity rule () lets data servers that do not expose a POSIX file namespace satisfy the tight coupling requirements without materializing POSIX uid/gid bits. A protocol-level consequence of placing erasure coding at the client is that the layout is able to describe a file's storage shape over its full lifetime -- including the transition windows when the file is being assimilated from a non-erasure-coded source, re-encoded from one encoding to another, or recovered from a correlated encoding failure through a mirror under a different encoding. This draft allows a single file's layout to contain mirrors under different encodings. The heterogeneous-mirror capability is not a steady-state expectation; most files have one encoding most of the time. It is the protocol shape that lets transitions happen while the file remains readable. The deployment cases that drive the allowance are catalogued in . Scope note: the consistency goal of flexible file v2 layout is RAID consistency across the shards that make up an encoded stripe, not POSIX write ordering across arbitrary application writes. The protocol does not attempt to make overlapping application writes from different clients atomic; that is the province of file locking () and of application-level coordination. What the protocol does guarantee is that the shards comprising a given stripe agree on which write produced them -- expressed on the wire as agreement on the chunk_guard4 value of every chunk that carries those shards -- so that readers and repair actors never observe a half-applied stripe. Readers who need cross-write ordering beyond a single stripe MUST use the existing NFSv4 locking primitives.

Use Cases The protocol is designed around three workload classes. The labels below reflect the relative frequency of each class in installations that choose flexible file v2 layout for its combination of integrity and performance; individual deployments may diverge.

Single writer, multiple readers (the common case):: A file written by one client and subsequently read by many. Examples include artifacts deposited by batch jobs, container images, and media files. The protocol is optimized for this case; see .
Multiple writers without sustained contention (occasional):: Files with multiple concurrent writers where races on the same chunk are rare. Examples include shared-directory append-only logs and distributed builds. The chunk_guard4 CAS primitive and per-chunk locking cover this case without penalizing the common single writer path.
Multiple writers, disjoint regions (rare):: High-performance computing (HPC) checkpoint workloads, in which many ranks write disjoint regions of the same file in lockstep. The protocol relies on block alignment to keep per-chunk contention rare despite overall high writer count. Contention that does occur is resolved via the deterministic tiebreaker rule defined in . Deployments that use an XOR-based erasure encoding (see ) and expect frequent small edits from this workload class MAY use the optional delta-write protocol defined in , which lets the client forward per-projection XOR deltas directly to each data server, avoiding client-side read-modify-write of the full stripe on the small-edit path.

Scale targets include multi-thousand-client deployments (on the order of tens of thousands of concurrent clients for HPC checkpointing), parallel-filesystem replacements, and multi-rack shared-storage clusters. The repair protocol (see ) is designed to let such deployments tolerate data-server failures and concurrent writer races without blocking the critical path for the first two workload classes.

Definitions

block:: the application's view of file data. A block is a unit of file content as observed by an NFS client at the POSIX layer or by the local file system. A chunk's payload, after any decoding the client performs, is presented to the application as one or more blocks.
shard:: the encoding's view. A shard is a single piece of an encoded stripe produced by an erasure-coding (or replication) transformation. A stripe of k data shards plus m parity shards is the unit an encoding encodes and decodes. The word "shard" only has meaning while the encoding is reasoning about a stripe; once a shard is at rest on a data server it is, by virtue of having been transmitted, the payload of a chunk.
chunk:: the protocol's unit of file data on the wire, carrying an envelope that distinguishes it from a block: a compare-and-swap guard (chunk_guard4 -- atomicity, see ), a checksum (per-chunk integrity), a provenance identifier (chunk_owner4, see ), a lifecycle state (PENDING / FINALIZED / COMMITTED via the chunk state machine, see ), and per-chunk locking that survives stateid revocation through lock escrow. A chunk is the addressable unit named in the CHUNK operations defined in this document and durably persisted by a data server. A chunk's payload may be a block (mirrored layout) or a shard (erasure-coded layout); the wire protocol does not distinguish. The chunk size MAY differ from the size of the block or shard it carries. See for the load-bearing role each envelope property plays in the protocol's consistency story.

The three terms describe the same data at three different layers and should be used accordingly. The encoding transforms blocks into shards; the wire protocol transmits shards as chunk payloads; the data server persists chunks. On read the path reverses. A protocol-internal note: the chunk state machine () and several CHUNK operations refer to the per-chunk-offset state records as "blocks" (PENDING / FINALIZED / COMMITTED / errored). This is a finer-grained use of the word, internal to the data server's chunk metadata, and should not be confused with the application-layer "block" defined above. Where ambiguity matters, this document writes "chunk-state block" or relies on context (operation names, state names) to disambiguate.

client-side erasure coding:: Erasure coding in which the encode and decode transforms are performed on the pNFS client, and encoded shards are written directly to the data servers (rather than being computed at a server-side coordinator). This is the deployment shape Flexible File Version 2 Layout Type is designed for; the chunk substrate and CHUNK operations that make it safe are specified in and . Contrast: server-side erasure coding, where a coordinator holds the entire stripe and produces parity, is out of scope for this document.
client-side mirroring:: a feature in which the client, not the server, is responsible for updating all of the mirrored copies of a layout segment.
compare-and-swap (CAS):: an atomic primitive from concurrent programming in which an update is conditional on a prior observed value: the operation succeeds only if the current value matches an expected prior value, and otherwise fails so the caller can retry. In this document, the chunk_guard4 mechanism (see ) implements CAS at the chunk level; the "expected prior value" is the chunk_guard4 the writer observed at read time, and the "fail" outcome is NFS4ERR_CHUNK_GUARDED.
control communication requirements:: the specification for information on layouts, stateids, file metadata, and file data that must be communicated between the metadata server and the storage devices. There is a separate set of requirements for each layout type.
control protocol:: the particular mechanism that an implementation of a layout type would use to meet the control communication requirement for that layout type. This need not be a protocol as normally understood. In some cases, the same protocol may be used as a control protocol and storage protocol.
(file) data:: that part of the file system object that contains the data to be read or written. It is the contents of the object rather than the attributes of the object.
data block:: A block (as defined above) in the client's cache for a file.
data file:: The data portion of the file, stored on the data server.
data server:: a pNFS server that provides the file's data when the file system object is accessed over a file-based protocol.
erasure coding:: A data protection scheme where a stripe of data is encoded into shards (k data shards and m parity shards) so that the original content can be reconstructed from any sufficient subset of the shards. Shards are transmitted as the payload of CHUNK operations and stored on different data servers.
escrow (lock escrow, metadata-server escrow):: a state in which a chunk lock is held by the metadata server on behalf of an as-yet-unselected future owner. Escrow preserves the lock-continuity invariant across two survivability scenarios: (a) client failure while holding chunk locks (client-revocation case: the locks are transferred to the metadata server rather than dropped), and (b) metadata-server-instance change (restart / HA failover: a new incarnation reclaims the escrows the departing incarnation installed). The escrowed lock is marked on the wire by the reserved cg_client_id value CHUNK_GUARD_CLIENT_ID_MDS (see ) and named by an escrow_id4 (). A repair actor adopts an escrow via CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT driven by CB_CHUNK_REPAIR; a new metadata-server incarnation reclaims outstanding escrows via CHUNK_ESCROW_TAKEOVER. A "metadata-server escrow owner" is the metadata server acting in this placeholder role; "in escrow" describes a lock in this state. The full model is .
fencing:: the process by which the metadata server prevents the storage devices from processing I/O from a specific client to a specific file.
file layout type:: a layout type in which the storage devices are accessed via the NFS protocol (see ).
gid:: the group id, a numeric value that identifies to which group a file belongs.
layout:: the information a client uses to access file data on a storage device. This information includes specification of the protocol (layout type) and the identity of the storage devices to be used.
layout iomode:: a grant of either read-only or read/write I/O to the client.
layout segment:: a sub-division of a layout. That sub-division might be by the layout iomode (see Sections 3.3.20 and 12.2.9 of ), a striping pattern (see ), or requested byte range.
layout stateid:: a 128-bit quantity returned by a server that uniquely defines the layout state provided by the server for a specific layout that describes a layout type and file (see ). Further, describes differences in handling between layout stateids and other stateid types.
layout type:: a specification of both the storage protocol used to access the data and the aggregation scheme used to lay out the file data on the underlying storage devices.
loose coupling:: when the control protocol is a storage protocol.
(file) metadata:: the part of the file system object that contains various descriptive data relevant to the file object, as opposed to the file data itself. This could include the time of last modification, access time, EOF position, etc.
metadata server:: the pNFS server that provides metadata information for a file system object. It is also responsible for generating, recalling, and revoking layouts for file system objects, for performing directory operations, and for performing I/O operations to regular files when the clients direct these to the metadata server itself.
mirror:: a copy of a layout segment. Note that if one copy of the mirror is updated, then all copies must be updated.
non-systematic encoding:: An erasure coding scheme in which the encoded shards do not contain verbatim copies of the original data. Every read requires decoding, even when no shards are lost. The Mojette non-systematic transform is an example.
proxy server:: a peer of the metadata server, defined in (hereafter "the proxy server draft"), that admits client I/O on the metadata server's behalf -- either as a translator for clients that cannot speak the file's native encoding, or as a proxy-mediated data path during whole-file move and repair operations. A proxy server may have additional roles. This document cites the proxy server draft only when referencing a specific rule or section within it; other mentions of "proxy server" or "the proxy server draft" in this document refer to the same reference without repeating the citation.
recalling a layout:: a graceful recall, via a callback, of a specific layout by the metadata server to the client. Graceful here means that the client would have the opportunity to flush any WRITEs, etc., before returning the layout to the metadata server.
replication of data:: Data replication is making and storing multiple copies of data in different locations.
resilvering:: the act of rebuilding a mirrored copy of a layout segment from a known good copy of the layout segment. Note that this can also be done to create a new mirrored copy of the layout segment.
revoking a layout:: an invalidation of a specific layout by the metadata server. Once revocation occurs, the metadata server will not accept as valid any reference to the revoked layout, and a storage device will not accept any client access based on the layout.
rsize:: the data transfer buffer size used for READs.
stateid:: a 128-bit quantity returned by a server that uniquely defines the set of locking-related state provided by the server. Stateids may designate state related to open files, byte range locks, delegations, or layouts.
storage device:: the target to which clients may direct I/O requests when they hold an appropriate layout. See for further discussion of the difference between a data server and a storage device.
storage protocol:: the protocol used by clients to do I/O operations to the storage device. Each layout type specifies the set of storage protocols.
systematic encoding:: An erasure coding scheme in which the first k of the k + m encoded shards are identical to the original k data blocks. A healthy read (no failures) requires no decoding -- the data shards are read directly. Decoding is triggered only when data shards are missing. Reed-Solomon Vandermonde and Mojette systematic are examples.
tight coupling:: an arrangement in which the control protocol is one designed specifically for control communication. It may be either a proprietary protocol adapted specifically to a particular metadata server or a protocol based on a Standards Track document. The specific tight coupling variant defined by this document, in which the control protocol is the TRUST_STATEID family, is referred to as trusted stateid tight coupling (see ).
trusted stateid tight coupling:: the specific tight coupling control protocol defined in this document, consisting of the operations TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID. Within the scope of this document, unqualified references to "tight coupling" or "tightly coupled" refer to trusted stateid tight coupling unless the context explicitly discusses the general concept. Other tight coupling control protocols (proprietary or future Standards Track) may exist but are not covered by this specification.
uid:: the user id, a numeric value that identifies which user owns a file.
write hole:: A write hole is a data corruption scenario where either two clients are trying to write to the same chunk or one client is overwriting an existing chunk of data.
wsize:: the data transfer buffer size used for WRITEs.

Naming the layout type (reviewer note) The layout type defined by this document is referred to by several forms in the surrounding prose. This note fixes the vocabulary so the reader (and the sweep in the source) can keep them straight. It follows the register RFC 8435 established for its own layout type: a formal title-case name for headings and IANA registrations, a lowercase running-text form, and separate XDR / struct-prefix identifiers. The parallel is exact -- everywhere RFC 8435 says "flexible file layout" this document says "flexible file v2 layout", and everywhere RFC 8435 says "Flexible File Layout Type" this document says "Flexible File Version 2 Layout Type". Formal name (headings, IANA registrations, the abstract): "Flexible File Version 2 Layout Type". This is the form to use when naming the layout type as such -- for example, the Section 5 heading, the IANA registry title, and the sentence that introduces the layout type in the abstract. A short formal variant "Flexible File Version 2 Layout" drops "Type" when "Type" would be redundant with the surrounding noun (as in the IANA-registered creation-hint name). Running-text form (body prose): "flexible file v2 layout". This is the form for sentences that mention the layout in passing rather than naming it as an object of definition -- "the flexible file v2 layout supports multipathing to multiple storage devices", "the flexible file v2 layout does not use lou_body". It parallels RFC 8435's lowercase "flexible file layout" and is preferred over the title-case form for sentences where the emphasis is on the mechanism, not the name. Do NOT write "flexible file v2 layout version 2" -- the "v2" already carries the version, and appending "version 2" is a rendering hazard. Short form (narrow tables and reviewer-aid appendix material where cell width or repetition would otherwise dominate): "FFv2", with "FFv1" reserved for the predecessor defined by . The short forms are NOT appropriate in body prose, headings, IANA registrations, or the abstract; body prose that would otherwise repeat the layout name uses the running-text form above. Their scoped use is in tables where "flexible file v2 layout" would break the column layout (e.g., the four-variant wire-path comparison table in the Implementation Status appendix). XDR identifier: LAYOUT4_FLEX_FILES_V2. This is the name assigned to the layout type by the layout-type registry () and is the constant a receiver compares against when dispatching on layout type. It parallels LAYOUT4_FLEX_FILES for the predecessor layout type (). Use the XDR identifier whenever the sentence is about the value that appears on the wire; use the "Flexible File Version 2 Layout Type" formal name whenever the sentence is about the layout type as a specification. Struct-name prefix: ffv2_ (as in ffv2_layout4, ffv2_mirror4, ffv2_stripes4). This parallels RFC 8435's ff_ prefix (as in ff_layout4, ff_device_addr4). The per-struct field prefix follows the RFC 8435 pattern of struct-initials plus underscore: ffv2l_ for ffv2_layout4, ffv2m_ for ffv2_mirror4, ffv2ds_ for ffv2_data_server4, and so on. Every field prefix carries the full ffv2 stem; a bare f plus struct initials is not used, so that a reader scanning an XDR block can tell at a glance which specification a field belongs to. Where two struct names would reduce to the same initials, the prefix takes one letter from each word of the struct name so that the prefixes stay distinct: ffv2l_ for ffv2_layout4 but ffv2lu_ for ffv2_layoutupdate4, ffv2lh_ for ffv2_layouthint4, and ffv2lr_ for ffv2_layoutreturn4. This matters most for ffv2_layout4 and ffv2_layoutupdate4, which both carry a field named for flags: ffv2l_flags is the ffv2_flags4 bitmask on the layout, while ffv2lu_flags is the ffv2_layoutstats_flags4 bitmask on a layout-statistics report. The two are different types with different registries, and an implementation that conflates them will encode the wrong bitmask. Document title and file abbreviation: the front-matter title is "Parallel NFS (pNFS) Flexible File Layout Version 2" and the abbrev used in the running header is "Flex File Layout v2". These are document-metadata conventions and do not participate in the body-prose vocabulary; the running text follows the forms above, not the front-matter forms.

Coupling of Storage Devices A server implementation may choose either a loosely coupled model or a tightly coupled model between the metadata server and the storage devices. describes the general problems facing pNFS implementations. This document details how the new flexible file v2 layout addresses these issues. To implement the tightly coupled model, a control protocol has to be defined. As the flexible file v2 layout imposes no special requirements on the client, the control protocol will need to provide:

management of both security and LAYOUTCOMMITs; and
a global stateid model and management of these stateids.

When implementing the loosely coupled model, the only control protocol will be a version of NFS, with no ability to provide a global stateid model or to prevent clients from using layouts inappropriately. To enable client use in that environment, this document specifies how security, state, and locking are managed. The loosely and tightly coupled locking models defined in apply equally to this layout type, including the use of anonymous stateids with loosely coupled storage devices, the handling of lock and delegation stateids, and the mandatory byte range lock requirements for the tightly coupled model.

LAYOUTCOMMIT Regardless of the coupling model, the metadata server has the responsibility, upon receiving a LAYOUTCOMMIT (see ) to ensure that the semantics of pNFS are respected (see ). These do include a requirement that data written to a data storage device be stable before the occurrence of the LAYOUTCOMMIT. It is the responsibility of the client to make sure the data file is stable before the metadata server begins to query the storage devices about the changes to the file. If any WRITE to a storage device did not result with stable_how equal to FILE_SYNC, a LAYOUTCOMMIT to the metadata server MUST be preceded by a COMMIT to the storage devices written to. Note that if the client has not done a COMMIT to the storage device, then the LAYOUTCOMMIT might not be synchronized to the last WRITE operation to the storage device.

Fencing Clients from the Storage Device With loosely coupled storage devices, the metadata server uses synthetic uids (user ids) and gids (group ids) for the data file, where the uid owner of the data file is allowed read/write access and the gid owner is allowed read-only access. As part of the layout (see ffv2ds_user and ffv2ds_group in ), the client is provided with the user and group to be used in the Remote Procedure Call (RPC) credentials needed to access the data file. Fencing off of clients is achieved by the metadata server changing the synthetic uid and/or gid owners of the data file on the storage device to implicitly revoke the outstanding RPC credentials. A client presenting the wrong credential for the desired access will get an NFS4ERR_ACCESS error. With this loosely coupled model, the metadata server is not able to fence off a single client; it is forced to fence off all clients. However, as the other clients react to the fencing, returning their layouts and trying to get new ones, the metadata server can hand out a new uid and gid to allow access. It is RECOMMENDED to implement common access control methods at the storage device file system to allow only the metadata server root (super user) access to the storage device and to set the owner of all directories holding data files to the root user. This approach provides a practical model to enforce access control and fence off cooperative clients, but it cannot protect against malicious clients; hence, it provides a level of security equivalent to AUTH_SYS. It is RECOMMENDED that the communication between the metadata server and storage device be secure from eavesdroppers and man-in-the-middle protocol tampering. The security measure could be physical security (e.g., the servers are co-located in a physically secure area), encrypted communications, or some other technique. With tightly coupled storage devices, the metadata server and the storage device agree on the authorization decision for each client access: a client allowed by the metadata server to read or write a file is allowed the same access at the storage device, and a client denied at the metadata server is denied at the storage device. How the storage device reaches that decision is not constrained by this specification. Some storage devices replicate the user, group, mode bits, and ACL of the metadata file onto a POSIX-shaped local representation of the data file and let their native filesystem enforce the decision; others (such as storage devices backed by an object store, a control-protocol-driven backend, or a backend with no exposed file namespace) consult the control protocol directly without ever materializing a POSIX file representation. Both approaches are conformant; the specification's requirement is the authorization-outcome parity, not the mechanism that produces it. The client authenticates with the storage device and receives the same authorization outcome it would have received via the metadata server. In the case of tight coupling, fencing is the responsibility of the control protocol and is not described in detail in this document. Implementations of the tightly coupled locking model (see ) will need a way to prevent access by certain clients to specific files by invalidating the corresponding stateids on the storage device; in such a scenario, the client receives NFS4ERR_BAD_STATEID. The client need not know the model used between the metadata server and the storage device. It need only react consistently to any errors in interacting with the storage device. It SHOULD both return the layout and error to the metadata server and ask for a new layout. At that point, the metadata server can either hand out a new layout, hand out no layout (forcing the I/O through it), or deny the client further access to the file.

Implementation Notes for Synthetic uids/gids The selection method for the synthetic uids and gids to be used for fencing in loosely coupled storage devices is strictly an implementation issue. That is, an administrator might restrict a range of such ids available to the Lightweight Directory Access Protocol (LDAP) 'uid' field . The administrator might also be able to choose an id that would never be used to grant access. Then, when the metadata server had a request to access a file, a SETATTR would be sent to the storage device to set the owner and group of the data file. The user and group might be selected in a round-robin fashion from the range of available ids. Those ids would be sent back as ffv2ds_user and ffv2ds_group to the client, who would present them as the RPC credentials to the storage device. When the client is done accessing the file and the metadata server knows that no other client is accessing the file, it can reset the owner and group to restrict access to the data file. When the metadata server wants to fence off a client, it changes the synthetic uid and/or gid to the restricted ids. Note that using a restricted id ensures that there is a change of owner and at least one id available that never gets allowed access. Under an AUTH_SYS security model, synthetic uids and gids of 0 SHOULD be avoided. These typically either grant super access to files on a storage device or are mapped to an anonymous id. In the first case, even if the data file is fenced, the client might still be able to access the file. In the second case, multiple ids might be mapped to the anonymous ids.

Example of using Synthetic uids/gids The user loghyr creates a file "ompha.c" on the metadata server, which then creates a corresponding data file on the storage device. The metadata server entry may look like:

Metadata's view of ompha.c On the storage device, the file may be assigned some unpredictable synthetic uid/gid to deny access:

Data's view of ompha.c When the file is opened on a client and accessed, the user will try to get a layout for the data file. Since the layout knows nothing about the user (and does not care), it does not matter whether the user loghyr or garbo opens the file. The client has to present an uid of 19452 to get write permission. If it presents any other value for the uid, then it must give a gid of 28418 to get read access. Further, if the metadata server decides to fence the file, it SHOULD change the uid and/or gid such that these values neither match earlier values for that file nor match a predictable change based on an earlier fencing.

Fenced Data's view of ompha.c The set of synthetic gids on the storage device SHOULD be selected such that there is no mapping in any of the name services used by the storage device, i.e., each group SHOULD have no members. If the layout segment has an iomode of LAYOUTIOMODE4_READ, then the metadata server SHOULD return a synthetic uid that is not set on the storage device. Only the synthetic gid would be valid. The client is thus solely responsible for enforcing file permissions in a loosely coupled model. To allow loghyr write access, it will send an RPC to the storage device with a credential of 1066:1067. To allow garbo read access, it will send an RPC to the storage device with a credential of 1067:1067. The value of the uid does not matter as long as it is not the synthetic uid granted when getting the layout. While pushing the enforcement of permission checking onto the client may seem to weaken security, the client may already be responsible for enforcing permissions before modifications are sent to a server. With cached writes, the client is always responsible for tracking who is modifying a file and making sure to not coalesce requests from multiple users into one request.

State and Locking Models The coupling model in effect for a given metadata-server / storage-device pair is not negotiated over the NFS protocol. The metadata server determines the coupling model from out-of-band signals: administrative configuration, the choice and capabilities of the control protocol between the metadata server and the storage device, the storage device's data-path protocol version, and the storage device's backend architecture. At the NFS protocol level, the metadata server's expectations of the storage device follow these classifications:

Storage devices implementing the NFSv3 or NFSv4.0 protocols on the data path are treated as loosely coupled.
NFSv4.1+ storage devices that do not return the EXCHGID4_FLAG_USE_PNFS_DS flag in EXCHANGE_ID indicate that they are to be treated as loosely coupled. From the locking viewpoint, they are treated in the same way as NFSv4.0 storage devices.
NFSv4.1+ storage devices that identify themselves with the EXCHGID4_FLAG_USE_PNFS_DS flag set in EXCHANGE_ID can potentially be tightly coupled. They use a back-end control protocol to implement the global stateid model described in .

Tight coupling additionally requires a control protocol between the metadata server and the storage device, discovered or advertised out-of-band as described above. Some storage devices cannot operate under the loosely coupled model at all. The loose coupling model in this specification relies on the storage device authorizing client access against synthetic uid and gid values (), which presupposes that the data file has a local representation on the storage device against which POSIX-style ownership checks can be applied. Storage devices whose backend has no exposed file namespace -- for example, object-store-backed data servers, or data servers driven entirely through a control protocol against a non-POSIX backend -- do not have that local representation and MUST operate in the tightly coupled model with a control protocol that conveys the authorization decision directly. A metadata server deploying with such a storage device cannot fall back to loose coupling.

Loosely Coupled Locking Model When locking-related operations are requested, they are primarily dealt with by the metadata server, which generates the appropriate stateids. When an NFSv4 version is used as the data access protocol, the metadata server may make stateid-related requests of the storage devices. However, it is not required to do so, and the resulting stateids are known only to the metadata server and the storage device. Given this basic structure, locking-related operations are handled as follows:

OPENs are dealt with by the metadata server. Stateids are selected by the metadata server and associated with the client ID describing the client's connection to the metadata server. The metadata server may need to interact with the storage device to locate the file to be opened, but no locking-related functionality need be used on the storage device.
OPEN_DOWNGRADE and CLOSE only require local execution on the metadata server.
Advisory byte range locks can be implemented locally on the metadata server. As in the case of OPENs, the stateids associated with byte range locks are assigned by the metadata server and only used on the metadata server.
Delegations are assigned by the metadata server that initiates recalls when conflicting OPENs are processed. No storage device involvement is required.
TEST_STATEID and FREE_STATEID are processed locally on the metadata server, without storage device involvement.

All I/O operations to the storage device are done using the anonymous stateid. Thus, the storage device has no information about the openowner and lockowner responsible for issuing a particular I/O operation. As a result:

Mandatory byte range locking cannot be supported because the storage device has no way of distinguishing I/O done on behalf of the lock owner from those done by others.
Enforcement of share reservations is the responsibility of the client. Even though I/O is done using the anonymous stateid, the client MUST ensure that it has a valid stateid associated with the openowner.

In the event that a stateid is revoked, the metadata server is responsible for preventing client access, since it has no way of being sure that the client is aware that the stateid in question has been revoked. As the client never receives a stateid generated by a storage device, there is no client lease on the storage device and no prospect of lease expiration, even when access is via NFSv4 protocols. Clients will have leases on the metadata server. In dealing with lease expiration, the metadata server may need to use fencing to prevent revoked stateids from being relied upon by a client unaware of the fact that they have been revoked.

Tightly Coupled Locking Model When locking-related operations are requested, they are primarily dealt with by the metadata server, which generates the appropriate stateids. These stateids MUST be made known to the storage device using control protocol facilities. This document defines one such control protocol -- the TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID operations in -- for deployments in which the storage devices are NFSv4.2 servers willing to implement the new operations. A storage device with its own established back-end control protocol that provides the equivalent functional capabilities is conformant under this specification without implementing the TRUST_STATEID family; see for the conformance framing. When using the TRUST_STATEID control protocol defined in , the metadata server and a storage device establish that they can use it via a two-part handshake, both parts of which MUST succeed before the metadata server may issue TRUST_STATEID against that storage device for production traffic:

Capability probe:: At control-session setup the metadata server sends a TRUST_STATEID against the anonymous stateid (see ). A storage device that supports tight coupling MUST reject the probe with NFS4ERR_INVAL; a storage device that does not support tight coupling returns NFS4ERR_NOTSUPP and the metadata server falls back to loose coupling. The metadata server records result per storage device by setting the FFV2_COUPLING_TRUSTED_STATEID bit in ffv2dv_coupling on success (leaving the bit clear on NFS4ERR_NOTSUPP).
Control-session gating:: The metadata server presents EXCHGID4_FLAG_USE_PNFS_MDS at EXCHANGE_ID when it opens the control session to the storage device (see ). The storage device MUST reject any incoming TRUST_STATEID, REVOKE_STATEID, or BULK_REVOKE_STATEID that does not arrive on such a session with NFS4ERR_PERM. This is the authorization mechanism that distinguishes the metadata server from ordinary pNFS clients, which connect with EXCHGID4_FLAG_USE_PNFS_DS or EXCHGID4_FLAG_USE_NON_PNFS and are therefore structurally unable to invoke these operations.

Given this basic structure, locking-related operations are handled as follows:

OPENs are dealt with primarily on the metadata server. Stateids are selected by the metadata server and associated with the client ID describing the client's connection to the metadata server. The metadata server needs to interact with the storage device to locate the file to be opened and to make the storage device aware of the association between the metadata-server-chosen stateid and the client and openowner that it represents. OPEN_DOWNGRADE and CLOSE are executed initially on the metadata server, but the state change MUST be propagated to the storage device.
Advisory byte range locks can be implemented locally on the metadata server. As in the case of OPENs, the stateids associated with byte range locks are assigned by the metadata server and are available for use on the metadata server. Because I/O operations are allowed to present lock stateids, the metadata server needs the ability to make the storage device aware of the association between the metadata-server-chosen stateid and the corresponding open stateid it is associated with.
Mandatory byte range locks can be supported when both the metadata server and the storage devices have the appropriate support. As in the case of advisory byte range locks, these are assigned by the metadata server and are available for use on the metadata server. To enable mandatory lock enforcement on the storage device, the metadata server needs the ability to make the storage device aware of the association between the metadata-server-chosen stateid and the client, openowner, and lock (i.e., lockowner, byte range, and lock-type) that it represents. Because I/O operations are allowed to present lock stateids, this information needs to be propagated to all storage devices to which I/O might be directed rather than only to storage device that contain the locked region.
Delegations are assigned by the metadata server that initiates recalls when conflicting OPENs are processed. Because I/O operations are allowed to present delegation stateids, the metadata server requires the ability:
1. to make the storage device aware of the association between the metadata-server-chosen stateid and the filehandle and delegation type it represents
2. to break such an association.
TEST_STATEID is processed locally on the metadata server, without storage device involvement.
FREE_STATEID is processed on the metadata server, but the metadata server requires the ability to propagate the request to the corresponding storage devices.

Scope of this document. The wire-level control-protocol operations this document defines -- TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID () -- carry only the association between a layout stateid, the ffv2m_client_id the metadata server assigned to the client, and (for TRUST_STATEID) the iomode, expiry, and principal. They do NOT carry openowner, lockowner, byte range, lock-type, the identity of an associated open stateid, or delegation-type. The bullets above (OPEN state, advisory byte range lock state, mandatory byte range lock state, and delegation state) enumerate the associations the storage device would need in order to enforce POSIX-conformant OPEN, byte range locking, and delegation semantics against per-client identity rather than against the loose coupling synthetic uid/gid. These are inherited from the general tight coupling locking model in . Trusted-stateid tight coupling as defined by this document satisfies these associations only for the layout stateid; a deployment that requires mandatory byte range locking, delegation recall, or the finer-grained open/lock stateid associations MUST use a back-end control protocol between the metadata server and the storage device that carries this state. Such a back-end control protocol is out of scope for this document. A deployment that does not need those finer-grained associations -- for example, a flexible file v2 layout deployment whose per-file access-control decisions live entirely on the metadata server and whose storage devices see only chunk-level CAS on layout stateids -- is conformant with the trusted stateid tight coupling model using only the TRUST_STATEID family. That scope covers the layout stateid and, transitively via ffv2m_client_id, the writer identity carried on CHUNK operations. Because the client will possess and use stateids valid on the storage device, there will be a client lease on the storage device, and the possibility of lease expiration does exist. The best approach for the storage device is to retain these locks as a courtesy. However, if it does not do so, control protocol facilities need to provide the means to synchronize lock state between the metadata server and storage device. Clients will also have leases on the metadata server that are subject to expiration. In dealing with lease expiration, the metadata server would be expected to use control protocol facilities enabling it to invalidate revoked stateids on the storage device. In the event the client is not responsive, the metadata server may need to use fencing to prevent revoked stateids from being acted upon by the storage device.

Tight Coupling Control Protocol When an NFSv4.2 storage device participates in a tightly coupled deployment, the metadata server and the storage devices need a control protocol that:

registers the layout stateid with each storage device so the storage device can validate client I/O independently; and
revokes trust promptly when the metadata server withdraws the client's authorization -- for example, on CB_LAYOUTRECALL timeout, lease expiry, or layout return after error.

This specification defines one such control protocol, designated trusted stateid tight coupling, as three new NFSv4.2 operations: TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID. These operations are sent by the metadata server to each storage device over a dedicated control session (see ) and MUST NOT be sent by pNFS clients. Other tight coupling control protocols may exist or be defined elsewhere. Existing pNFS server implementations with established back-end control protocols -- for example, dCache , which has its own control protocol between its metadata service and its data servers -- satisfy the tightly coupled locking model () through their own mechanisms and are conformant under this specification provided they meet the functional capabilities described there. Such implementations need not adopt the TRUST_STATEID family, and their interoperability with the TRUST_STATEID family is outside the scope of this document. A storage device that does not implement TRUST_STATEID is treated as not supporting trusted stateid tight coupling specifically; the capability probe in detects this and the metadata server falls back to loose coupling () or, if the storage device's own control protocol is in use, that protocol governs. Within the remainder of and its subsections, unqualified references to "tight coupling" or "tightly coupled" refer to the trusted stateid variant defined here. The receiver of these operations is any server the metadata server delegates client-I/O admission to. In this document that is the storage device (data server). The same mechanism applies to a proxy server -- a proxy server may or may not additionally act as a data server, but in either role it needs the metadata server to register a layout stateid before it can admit client I/O. Where this section says "storage device," read it as "storage device, or proxy server"; the flag check and the three operations are identical for both roles.

Capability Discovery A storage device indicates support for trusted stateid tight coupling implicitly, by processing TRUST_STATEID rather than returning NFS4ERR_NOTSUPP. (A storage device that supports a non-TRUST_STATEID form of tight coupling but not the trusted stateid variant defined here will return NFS4ERR_NOTSUPP on this probe; from this specification's perspective it is treated the same as a storage device that does not support tight coupling at all.) The metadata server probes each storage device during control-session setup:

TRUST_STATEID capability probe The anonymous stateid is used deliberately: a correctly implemented storage device MUST reject it (see ), so the probe cannot accidentally register garbage in the trust table. The metadata server interprets the probe response as follows:

NFS4ERR_NOTSUPP:: trusted stateid tight coupling is not supported on this storage device. The metadata server leaves the FFV2_COUPLING_TRUSTED_STATEID bit clear in ffv2dv_coupling for this storage device. If ffv2dv_coupling has no other tight coupling bits set for this storage device, the metadata server falls back to the synthetic-uid model (anonymous stateid plus fencing).
NFS4ERR_INVAL:: trusted stateid tight coupling is supported. The anonymous stateid was correctly rejected. The metadata server sets the FFV2_COUPLING_TRUSTED_STATEID bit in ffv2dv_coupling for this storage device.
NFS4_OK:: the storage device accepted an anonymous stateid into its trust table. This is a storage device bug. The metadata server MAY treat the capability as confirmed to avoid downgrading to loose coupling, but it MUST immediately issue REVOKE_STATEID to remove the bogus entry.

The capability is recorded per storage device, not per file. Partial support across a mirror set is permitted: each ffv2_device_versions4 entry returned by GETDEVICEINFO carries its own ffv2dv_coupling value, set independently.

Control Session The metadata server establishes an NFSv4.2 session to each tight coupling capable storage device at startup. On this session the metadata server acts as the storage device's client and presents EXCHGID4_FLAG_USE_PNFS_MDS in its EXCHANGE_ID args. That is a repurposing of the flag, and implementers should understand it as one. determines a server's pNFS role from the eir_flags it returns, not from what the client asks for, and treats the argument flags as role requests; read literally, EXCHGID4_FLAG_USE_PNFS_MDS in the args asks the peer to act as this client's metadata server, which is not what a metadata server wants from a storage device. The flag is used here precisely because it carries no other meaning on this path. does name a flag for a client connecting to a data server, EXCHGID4_FLAG_USE_PNFS_DS, but a pNFS client presents that one too, so it cannot separate a metadata server from a client. EXCHGID4_FLAG_USE_PNFS_MDS is the one role request a client has no reason to make of a storage device, and that is what leaves it free to mark the control session. Presenting it does not conflict with . permits a client to request zero or more of the three role flags even where the combination requested is contradictory, and constrains only the combinations a server may return in its eir_flags. A storage device that also serves a metadata server role, and that selects among its roles from the requested ones, should account for this document's use of the flag when composing its eir_flags. The storage device MUST verify that any incoming TRUST_STATEID, REVOKE_STATEID, or BULK_REVOKE_STATEID compound arrives on a session whose owning client presented EXCHGID4_FLAG_USE_PNFS_MDS in its EXCHANGE_ID args. Requests that arrive on any other session MUST be rejected with NFS4ERR_PERM. This is the sole access control on these operations; a pNFS client connecting to the storage device does not present EXCHGID4_FLAG_USE_PNFS_MDS and therefore cannot invoke them. The EXCHGID4_FLAG_USE_PNFS_MDS check replaces any path- or filehandle-level gating. TRUST_STATEID operates on a filehandle that may be any file on the storage device, and the metadata server is the sole authority that can legitimately speak this protocol. Because the EXCHGID4_FLAG_USE_PNFS_MDS check relies on the owning client's self-declaration at EXCHANGE_ID time, the storage device cannot by itself distinguish a legitimate metadata server from any other host that sets the flag. The wire protocol provides no primitive that binds a request to a particular pNFS role; the flag is a hint whose operational meaning is only as strong as the deployment's authentication or isolation choice. The deployment requirement is that only entities the deployment considers legitimate metadata servers can (a) establish a control session with EXCHGID4_FLAG_USE_PNFS_MDS against the storage device and (b) invoke TRUST_STATEID, REVOKE_STATEID, or BULK_REVOKE_STATEID on it. Deployments have historically satisfied this requirement using one or more of the following mechanisms; the wire protocol does not prescribe which is chosen:

RPCSEC_GSS with a machine principal that the storage device has been configured to accept as a metadata server. The storage device validates the principal against a local policy list before accepting the flag.
TLS () with a client certificate that the storage device has been configured to accept as a metadata server. The storage device validates the certificate against a local policy list before accepting the flag.
Network-path isolation: the control-session path runs on a network segment (dedicated management VLAN, private link, or similar) that pNFS clients cannot reach, so only configured metadata servers can open a session at all.
Operating-system-level filesystem access control on the storage device: the underlying export that backs the pNFS data files is configured so that only the metadata server's OS identity (typically root, or a dedicated privileged uid) can access it. A pNFS client that reaches the storage device but presents any other uid receives permission errors before it can invoke any operation, control-plane or otherwise.

A single host MAY legitimately act in multiple pNFS roles against the same storage device -- for example, an entity that is a metadata server for one export and a plain pNFS client for another. The wire protocol does not distinguish these roles at the operation level; the deployment is responsible for arranging that the storage device can tell them apart, typically by using distinct credentials or distinct sessions for the two roles (each session presents EXCHGID4_FLAG_USE_PNFS_MDS according to the role the entity is acting in on that session). The security consequences of these choices -- what an unauthenticated attacker can invoke if none of the mechanisms above is deployed, and what an authenticated caller can still achieve by misrepresenting role -- are discussed in .

Flow at LAYOUTGET For each new or refreshed layout segment, the metadata server:

chooses the layout stateid (as it would without tight coupling);
identifies the trusted stateid capable storage devices in the mirror set (those for which ffv2dv_coupling has the FFV2_COUPLING_TRUSTED_STATEID flag set);
fans out TRUST_STATEID to each such storage device, specifying the layout stateid, the layout iomode, a tsa_expire derived from the metadata server's lease (see ), and the client's authenticated identity in tsa_principal;
waits for all fan-outs to complete (or reach their per-storage-device timeout) before returning the layout.

If every storage device in the mirror set rejects the TRUST_STATEID fan-out, the metadata server MUST NOT return the layout; instead it returns NFS4ERR_LAYOUTTRYLATER. If some storage devices accept and others reject, the metadata server MAY return a layout covering only the accepting storage devices, provided the accepting subset still meets the minimum servable coverage for the file's encoding: at least one replica for FFV2_ENCODING_PASSTHROUGH or FFV2_ENCODING_REPLICATED, or at least k of the k + m storage devices for an erasure-coded encoding at (k, m) parameters. If it does not, the metadata server MUST NOT return a partial layout and instead returns NFS4ERR_LAYOUTTRYLATER as in the all-reject case. A storage device that returns NFS4ERR_DELAY is retried until either success or the metadata server's LAYOUTGET-response budget is exhausted. If a storage device returns NFS4ERR_NOTSUPP at this time (having accepted the probe earlier), the metadata server MUST clear the FFV2_COUPLING_TRUSTED_STATEID flag in ffv2dv_coupling for this storage device. If no tight coupling flags remain set for this device, the metadata server falls back to the synthetic-uid model and re-issues the layout accordingly.

Principal Binding and the Kerberos Gap The flexible file v1 layout has a known gap: a client authenticated to the metadata server with Kerberos has no way to present the same authenticated identity to the storage device, because flexible file v1 layouts carry only ffds_user / ffds_group (POSIX uid/gid for AUTH_SYS). A strict Kerberos deployment on the flexible file v1 layout must either allow AUTH_SYS from the metadata server's subnet or accept that the flexible file v1 layout's data path is not Kerberos-protected. The tsa_principal field in TRUST_STATEID closes that gap. When a client authenticates to the metadata server as a Kerberos principal (e.g., alice@REALM), the metadata server passes that principal name to each storage device in tsa_principal. The storage device then enforces a two-part check on each CHUNK operation that presents the layout stateid: a. the stateid is in the trust table and has not expired; and b. the caller's authenticated identity (the RPCSEC_GSS display name on the CHUNK compound) matches tsa_principal. Both conditions MUST hold. On principal mismatch the storage device MUST return NFS4ERR_ACCESS -- the semantics are "you do not have an authorized layout for this file", which matches the existing fencing error and avoids the confusion of NFS4ERR_WRONGSEC (which directs the client to re-authenticate with a different flavor) or NFS4ERR_BAD_STATEID (which directs the client to return the layout). The metadata server MUST populate tsa_principal with the RPCSEC_GSS display name of the authenticated client when the client authenticated to the metadata server via RPCSEC_GSS. The metadata server MUST set tsa_principal to the empty string only for AUTH_SYS and TLS clients (for which there is no server-verified per-user identity). Setting tsa_principal to the empty string for an RPCSEC_GSS client disables the principal check on the storage device and silently re-opens the flexible file v1 layout Kerberos gap; it is a metadata server bug, not a protocol option. If tsa_principal is the empty string, no principal check applies. This is the expected setting for AUTH_SYS and TLS clients:

AUTH_SYS clients have no server-verified identity. The storage device's stateid check and the AUTH_SYS uid/gid on the data file together constitute the authorization. In a tightly coupled deployment the data file's owner/group need not match the metadata file's, since ffv2ds_user and ffv2ds_group are ignored (see ).
TLS clients have transport-layer authentication via mutual TLS (). The TLS layer authenticates the client machine; the stateid check confirms the metadata server authorized that machine to access this file. The machine-level authentication is handled beneath the RPC layer and is not reflected in tsa_principal. Opportunistic TLS (STARTTLS without certificate verification) provides encryption but not authentication, and therefore has the same authorization properties as plain AUTH_SYS.

When a client's I/O is routed through a proxy server -- that is, the layout the metadata server returns to the client has FFV2_DS_FLAGS_PROXY set on the proxy's ffv2_data_server4 entry -- the storage device observes CHUNK operations arriving from the proxy server's address rather than from the client directly. The tsa_principal the metadata server populates in TRUST_STATEID is the principal the storage device will observe on those CHUNK operations, and 's credential-forwarding rules (in particular rule 1, "Credential pass-through") require the proxy server to forward the client's credentials verbatim on every CHUNK operation it issues on the client's behalf. Therefore:

For an RPCSEC_GSS client whose I/O is proxied through a proxy server, the metadata server MUST set tsa_principal to the client's RPCSEC_GSS display name (identical to the non-proxied case). The storage device's principal check on CHUNK operations will match against the client's principal on the forwarded compound, not the proxy server's service identity.
For an AUTH_SYS client whose I/O is proxied through a proxy server, the metadata server MUST set tsa_principal to the empty string (identical to the non-proxied case). The proxy server forwards the client's AUTH_SYS uid/gid; the storage device's stateid check plus the forwarded AUTH_SYS uid/gid constitute the authorization.

The metadata server MUST NOT set tsa_principal to the proxy server's own service principal. Doing so would require the proxy server to authenticate to the storage device as itself (bypassing credential forwarding) which is explicitly prohibited by rule 4 of ("proxy server service identity is for the control plane only").

Client-Detected Trust Gap A window exists between a successful TRUST_STATEID fan-out and the client's first I/O to the storage device. A transient failure may cause the storage device to forget or reject the entry before the client's first CHUNK_WRITE arrives. The client cannot distinguish this case from legitimate revocation; both surface as NFS4ERR_BAD_STATEID on the storage device. The recovery path:

The client sends LAYOUTERROR(layout_stateid, device_id, NFS4ERR_BAD_STATEID) to the metadata server.
The metadata server retries TRUST_STATEID against the reporting storage device. If the retry succeeds, the metadata server returns NFS4_OK for LAYOUTERROR. The client retries the original I/O.
If the retry fails -- the storage device is unreachable or returns a hard error -- the metadata server issues CB_LAYOUTRECALL for that device and the client returns the layout segment covering that storage device. The client is expected to re-request via LAYOUTGET.

This is the same LAYOUTERROR path used for NFS4ERR_ACCESS or NFS4ERR_PERM in the fencing model (see ), with the metadata server's action being "retry TRUST_STATEID" instead of "rotate uid/gid".

Lease and Renewal tsa_expire in a TRUST_STATEID request is a wall-clock expiry instant expressed as an nfstime4. The metadata server MUST set tsa_expire to the current wall-clock time plus the metadata server's client lease period. Clock-synchronization assumption: the metadata server and each storage device MUST maintain wall-clock synchronization within one lease period, e.g., via NTP or an equivalent mechanism. Under this assumption, a tsa_expire computed by the metadata server and evaluated by the storage device is interpreted consistently within the storage device's local clock. Deployments unable to guarantee sub-lease-period clock synchronization MUST either (a) shorten the effective TRUST_STATEID lease so it exceeds the worst-case skew by at least 2x, or (b) route I/O through the metadata server as the fallback path (no tight coupling control session, no TRUST_STATEID) so lease enforcement stays on the metadata server's clock alone. A storage device that detects sustained clock divergence from the metadata server (e.g., via periodic wall-clock exchange as part of its tight coupling control-session heartbeats) SHOULD log the divergence and MAY refuse further TRUST_STATEID entries with NFS4ERR_DELAY until the divergence is corrected. The metadata server MUST re-issue TRUST_STATEID for an entry before tsa_expire while the corresponding layout is outstanding. The RECOMMENDED trigger is: when an entry is within half the lease period of its tsa_expire, re-issue TRUST_STATEID with a refreshed tsa_expire. Renewing on every SEQUENCE that keeps the layout stateid alive is correct but produces metadata-server-to-storage-device traffic proportional to the client's SEQUENCE rate, which is undesirable in steady state. If an entry expires on the storage device before the metadata server renews it -- for example, because the metadata server is partitioned from the storage device for longer than the lease period -- the storage device MUST return NFS4ERR_BAD_STATEID to the client on the next CHUNK operation. The client returns the layout to the metadata server and re-requests. This is the same recovery path as the trust gap described above.

Storage Device Crash Recovery A storage device MAY persist its trust table across restarts. An implementation that does so MUST also persist its server-instance identity, returning the same eir_server_owner.so_minor_id on EXCHANGE_ID after the restart (per ), so that clients and the metadata server observe the device as continuously available and the persisted trust entries remain valid against the layout stateids that were issued before the restart. A storage device that does NOT persist its trust table empties the table on restart and MUST present a new server instance (incremented so_minor_id) so that clients detect the restart. The remainder of this section describes the recovery path for the volatile case. The client detects a volatile storage device restart via NFS4ERR_BADSESSION or NFS4ERR_STALE_CLIENTID on its data server session. The client returns the affected layout segment to the metadata server via LAYOUTRETURN and re-requests via LAYOUTGET. The metadata server then fans out fresh TRUST_STATEID operations to the recovered storage device. Planned storage device restarts (software upgrade, etc.) SHOULD drain in-flight CHUNK operations before shutting down.

Metadata Server Crash Recovery A metadata server MAY persist all its trust-management state across restarts. An implementation that does so MUST also persist its server-instance identity, returning the same eir_server_owner.so_minor_id on EXCHANGE_ID after the restart (per ), so that storage devices observe the metadata server as continuously available and accept incoming TRUST_STATEID and REVOKE_STATEID operations against the existing trust entries without revalidation. No grace period is required. A metadata server that presents a new server instance (incremented so_minor_id) on restart follows the recovery path in the remainder of this section. When the metadata server restarts as a new instance, its control sessions to the storage devices are lost. Trust entries remain on the storage devices until tsa_expire, but the metadata server is no longer renewing them; the entries are effectively orphaned until the metadata server completes grace. When the metadata server reconnects to a storage device with a new boot epoch -- that is, the EXCHANGE_ID returns a new server owner on the storage device's view of the metadata server -- the storage device SHOULD mark all trust entries established under the prior metadata-server epoch as pending-revalidation. While an entry is pending-revalidation:

I/O that presents the entry's stateid MUST receive NFS4ERR_DELAY, not NFS4ERR_BAD_STATEID. NFS4ERR_DELAY tells the client to retry with the same stateid -- the metadata server is recovering and may yet revalidate the entry. NFS4ERR_BAD_STATEID would instead cause the client to return the layout immediately, producing a thundering herd against the metadata server during grace.
An entry remains pending-revalidation until the metadata server either re-issues TRUST_STATEID for it (which transitions it back to trusted) or until the entry's tsa_expire elapses (which removes it).

The metadata server's recovery sequence is:

Reconnect to each storage device and establish a fresh control session.
Optionally issue BULK_REVOKE_STATEID with an all-zeros clientid to each storage device. This clears the prior trust table eagerly; skipping this step is correct, because orphan entries expire via tsa_expire.
Enter grace and accept RECLAIM operations from clients. For each reclaimed layout, fan out TRUST_STATEID to the relevant storage devices.
Exit grace. Clients that did not reclaim in time have their state revoked; the metadata server issues REVOKE_STATEID or BULK_REVOKE_STATEID on their behalf.

Metadata servers SHOULD persist the set of outstanding TRUST_STATEID entries (clientid, layout stateid, storage device address, tsa_expire) to stable storage. With this persistence the metadata server can re-issue TRUST_STATEID for all known entries immediately upon reconnecting to each storage device, before clients begin reclaiming. This shrinks the window during which the storage device returns NFS4ERR_DELAY for client I/O. Persistence is a latency optimization, not a correctness requirement: the re-layout path handles recovery in all cases.

Backward Compatibility

NFSv3 storage devices are unchanged. They are always treated as loosely coupled; TRUST_STATEID does not exist on NFSv3 servers.
NFSv4.2 storage devices for which the TRUST_STATEID probe returns NFS4ERR_NOTSUPP are treated as loosely coupled; fencing is the only revocation mechanism, the same as for NFSv3.
NFSv4.2 storage devices for which the probe returns NFS4ERR_INVAL support tight coupling; the metadata server uses TRUST_STATEID at LAYOUTGET and REVOKE_STATEID or BULK_REVOKE_STATEID for revocation instead of fencing.

A single deployment MAY contain a mix of tight-coupled and loose-coupled storage devices; each is negotiated independently via the probe.

Device Addressing and Discovery Data operations to a storage device require the client to know the network address of the storage device. The NFSv4.1+ GETDEVICEINFO operation () is used by the client to retrieve that information.

ffv2_device_addr4 The ffv2_device_addr4 data structure (see ) is returned by the server as the layout type specific opaque field da_addr_body in the device_addr4 structure by a successful GETDEVICEINFO operation for LAYOUT_FLEX_FILES_V2. ffv2_device_addr4 and ffv2_device_versions4 are the flexible file v2 layout counterparts to ff_device_addr4 and ff_device_versions4 in . The two structures are similar in shape but carry a flexible-file-v2-specific enrichment: the boolean ffdv_tightly_coupled from RFC 8435 has been widened to the uint32_t bitfield ffv2dv_coupling, which lets a storage device advertise more than one coupling capability at the same time (). Because the field type has changed, the flexible file v2 layout structs are named distinctly to avoid confusion with the RFC 8435 originals.

ffv2_device_versions4

ffv2_device_addr4 ; }; ]]> The ffv2da_netaddrs field is used to locate the storage device. It MUST be set by the server to a list holding one or more of the device network addresses. The ffv2da_versions array allows the metadata server to present choices as to NFS version, minor version, and coupling capabilities to the client. The ffv2dv_version and ffv2dv_minorversion represent the NFS protocol to be used to access the storage device. This layout specification defines the semantics for ffv2dv_versions 3 and 4. If ffv2dv_version equals 3, then the server MUST set ffv2dv_minorversion to 0 and ffv2dv_coupling to FFV2_COUPLING_SYNTHETIC_UIDS. The client MUST then access the storage device using the NFSv3 protocol . If ffv2dv_version equals 4, then the server MUST set ffv2dv_minorversion to 1 or 2, and the client MUST access the storage device using NFSv4 with the specified minor version. Three additional constraints narrow the valid set of (ffv2dv_version, ffv2dv_minorversion, ffv2dv_coupling) tuples in specific cases:

When a mirror's encoding type uses CHUNK operations (that is, any FFV2ENCODING* value other than FFV2_ENCODING_PASSTHROUGH), the corresponding storage device MUST be advertised with ffv2dv_version = 4 and ffv2dv_minorversion = 2. CHUNK operations are NFSv4.2 ops defined in this document; NFSv3 and NFSv4.1 storage devices cannot serve a non-PASSTHROUGH mirror.
When ffv2dv_coupling has the FFV2_COUPLING_TRUSTED_STATEID flag set, the storage device MUST be advertised with ffv2dv_version = 4 and ffv2dv_minorversion = 2. The TRUST_STATEID family of operations is defined as NFSv4.2; NFSv4.1 storage devices cannot participate in trusted stateid tight coupling.
When a mirror's encoding type uses CHUNK operations, the corresponding storage device MUST be advertised with ffv2dv_coupling having at least one tight coupling flag set (FFV2_COUPLING_TIGHTLY_COUPLED or FFV2_COUPLING_TRUSTED_STATEID, or both). The chunk lifecycle depends on the metadata-server-registered layout stateid and the per-client identity conveyed to the data server (); a synthetic-uid-only storage device has neither the trust-table entry required to validate a presented stateid nor the client-id binding required to authorize the writer, and therefore cannot serve a non-PASSTHROUGH mirror.

PASSTHROUGH is the only encoding that admits loose coupling (FFV2_COUPLING_SYNTHETIC_UIDS); every non-PASSTHROUGH encoding requires ffv2dv_version = 4, ffv2dv_minorversion = 2, and at least one tight coupling flag set in ffv2dv_coupling. PASSTHROUGH itself may be advertised under any of the following (ffv2dv_version, ffv2dv_minorversion) tuples: (3, 0), (4, 1), or (4, 2); the first two of these tuples are valid only for PASSTHROUGH under FFV2_COUPLING_SYNTHETIC_UIDS, because they precede or lack the NFSv4.2 features on which the CHUNK operations and the TRUST_STATEID family depend. Note that while the client might determine that it cannot use any of the configured combinations of ffv2dv_version, ffv2dv_minorversion, and ffv2dv_coupling, when it gets the device list from the metadata server, there is no way to indicate to the metadata server as to which device it is version incompatible. However, if the client waits until it retrieves the layout from the metadata server, it can at that time clearly identify the storage device in question (see ). The ffv2dv_rsize and ffv2dv_wsize are used to communicate the maximum rsize and wsize supported by the storage device. As the storage device can have a different rsize or wsize than the metadata server, the ffv2dv_rsize and ffv2dv_wsize allow the metadata server to communicate that information on behalf of the storage device. ffv2dv_coupling informs the client which tight coupling capabilities the storage device supports. The two tight coupling flags are orthogonal:

FFV2_COUPLING_TIGHTLY_COUPLED asserts that the deployment has a back-end control protocol between the metadata server and the data servers (the RFC 8435 general tight coupling concept); this document does not specify what that protocol is or how it operates. A dCache deployment, for example, would set this flag based on its own metadata-server / pool control plane.
FFV2_COUPLING_TRUSTED_STATEID asserts that the storage device implements the TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID operations defined in this document (). This is the tight coupling mechanism this specification adds; a storage device MUST NOT advertise this flag until the metadata server has confirmed the capability via the probe in .

A storage device MAY advertise either, both, or neither flag. When both are set, the deployment supports two tight coupling paths concurrently and MAY use either for a given operation. When neither is set (ffv2dv_coupling equals FFV2_COUPLING_SYNTHETIC_UIDS), the storage device is loosely coupled and the RFC 8435 synthetic-uid model applies (see ). If ffv2dv_coupling has no tight coupling flag set, then the client MUST commit writes to the storage devices for the file before sending a LAYOUTCOMMIT to the metadata server. That is, the writes MUST be committed by the client to stable storage via issuing WRITEs with stable_how = FILE_SYNC or by issuing a COMMIT after WRITEs with stable_how != FILE_SYNC (see ).

Storage Device Multipathing The flexible file v2 layout supports multipathing to multiple storage device addresses. Storage-device-level multipathing is used for bandwidth scaling via trunking and for higher availability of use in the event of a storage device failure. Multipathing allows the client to switch to another storage device address that may be that of another storage device that is exporting the same data stripe unit, without having to contact the metadata server for a new layout. To support storage device multipathing, ffv2da_netaddrs contains an array of one or more storage device network addresses. This array (data type multipath_list4) represents a list of storage devices (each identified by a network address), with the possibility that some storage device will appear in the list multiple times. The client is free to use any of the network addresses as a destination to send storage device requests. If some network addresses are less desirable paths to the data than others, then the metadata server SHOULD NOT include those network addresses in ffv2da_netaddrs. If less desirable network addresses exist to provide failover, the RECOMMENDED method to offer the addresses is to provide them in a replacement device-ID-to-device-address mapping or a replacement device ID. When a client finds no response from the storage device using all addresses available in ffv2da_netaddrs, it SHOULD send a GETDEVICEINFO to attempt to replace the existing device-ID-to-device-address mappings. If the metadata server detects that all network paths represented by ffv2da_netaddrs are unavailable, the metadata server SHOULD send a CB_NOTIFY_DEVICEID (if the client has indicated it wants device ID notifications for changed device IDs) to change the device-ID-to-device-address mappings to the available addresses. If the device ID itself will be replaced, the metadata server SHOULD recall all layouts with the device ID and thus force the client to get new layouts and device ID mappings via LAYOUTGET and GETDEVICEINFO. Generally, if two network addresses appear in ffv2da_netaddrs, they will designate the same storage device. When the storage device is accessed over NFSv4.1 or a higher minor version, the two storage device addresses will support the implementation of client ID or session trunking (the latter is RECOMMENDED) as defined in . The two storage device addresses will share the same server owner or major ID of the server owner. It is not always necessary for the two storage device addresses to designate the same storage device with trunking being used. For example, the data could be read-only, and the data consist of exact replicas.

Flexible File Version 2 Layout Type The original layouttype4 introduced in is extended as shown in . The layout_content4 and layout4 structures are reused unchanged from ; the layouttype4 enum is extended with the new LAYOUT4_FLEX_FILES_V2 value. The full enum and surrounding structures below are reproduced for reader convenience; only the new constant LAYOUT4_FLEX_FILES_V2 is part of the XDR extracted from this document (see ).

The original layout type (illustrative; reused from RFC 5662 with extension) ; }; struct layout4 { offset4 lo_offset; length4 lo_length; layoutiomode4 lo_iomode; layout_content4 lo_content; }; ]]> The extracted XDR contribution for this extension is the new layouttype4 constant alone:

New layouttype4 value (extracted) This document defines structures associated with the layouttype4 value LAYOUT4_FLEX_FILES_V2. specifies the loc_body structure as an XDR type "opaque". The opaque layout is uninterpreted by the generic pNFS client layers but is interpreted by the flexible file layout type implementation. This section defines the structure of this otherwise opaque value, ffv2_layout4.

ffv2_encoding_type4

The encoding type The ffv2_encoding_type4 (see ) encompasses a new IANA registry for 'Flexible File Version 2 Layout Type Erasure Coding Type Registry'. I.e., instead of defining a new Layout Type for each erasure coding, we define a new Erasure Encoding Type. The encoding types this document defines fall into two groups:

FFV2_ENCODING_PASSTHROUGH is the non-chunked, non-integrity on-ramp from flexible file v1 layout. It uses NFSv3 WRITE / READ or NFSv4 READ / WRITE directly against each replica's data server. No CHUNK_WRITE, no CHUNK_READ, no per-chunk CRC. See .
Every other standards-track encoding (any FFV2ENCODING* value other than FFV2_ENCODING_PASSTHROUGH; see ) uses the new operations defined here: in particular CHUNK_WRITE and CHUNK_READ, which carry the per-chunk checksum this version of the layout type relies on for end-to-end integrity. The encoding type selects how chunks are produced from application data; the wire and the storage device are the same in every case. See the individual encoding sections for the mathematical constructions of each encoding and for the wire-compatibility relationships among the Galois Field GF(2^8) family.

The 32-bit ffv2_encoding_type4 value space is partitioned by intended scope -- Standards Track, Experimental, Vendor (open), and Private / proprietary -- with different allocation policies per range, so that vendors can assign encoding values without consuming standards-track codepoints. See and the accompanying prose in for the range assignments and allocation policies.

Heterogeneous Mirror Sets A single flexible file v2 layout's ffv2l_mirrors array MAY carry mirror entries of different encoding types. The protocol does not require the entries to agree -- one mirror can be FFV2_ENCODING_PASSTHROUGH, another can be FFV2_ENCODING_RS_VANDERMONDE, both describing the same file's data range. This combination is the structural primitive for three operations that motivate keeping PASSTHROUGH in the layout type's vocabulary:

Assimilate:: A file that exists today as a plain copy on storage outside flexible file v2 layout -- no chunk envelope, no per-chunk CRC -- enters the namespace as a PASSTHROUGH mirror against the source bytes as they are. The metadata server then adds one or more encoded mirrors (MIRRORED, RS, Mojette) to the same layout and synchronizes them from the PASSTHROUGH source. Clients can read the file via the PASSTHROUGH mirror immediately; the encoded mirrors become available as they are populated. No "rewrite before serve" step is required.
Copy / migrate between encodings:: Changing a file from one encoding to another is "add a mirror in the target encoding to the same layout, let it sync from any healthy source mirror, retire the source mirror." PASSTHROUGH is the special case where one endpoint of that migration is "no encoding."
Repair across encodings:: When an encoded mirror has a chunk whose CRC fails and whose parity cannot reconstruct, a PASSTHROUGH mirror in the same layout is an authoritative source: CHUNK_READ a peer encoded mirror or read the PASSTHROUGH byte range, then CHUNK_WRITE the repaired chunk. The reverse is also true: a byte range on the PASSTHROUGH copy whose contents the metadata server suspects has drifted can be repaired by reconstructing from the verified-CRC chunks of an encoded peer. Two encodings of the same file are two independent recovery paths.

The metadata server is responsible for keeping the entries in a heterogeneous mirror set in sync; the protocol does not require client awareness of which encoding produced which mirror beyond what the layout already states. The wire-level coordination that makes a heterogeneous mirror set safe to operate -- in particular, the rule that a client arriving during a transition sees a single layout naming the proxy server rather than two layouts naming the source and destination encodings, and the rule that the metadata server's commit of a transition is a single transaction -- is specified in the proxy server draft (), in the sections "Layout Shape During a Proxy Operation" and "Atomic commit on PROXY_DONE". This document specifies the per-mirror encoding naming primitive; the proxy server document specifies the transactional machinery that uses it. The full description of each encoding type is deferred to its own section. The mapping between the enum values above and those sections is: Encoding type value to section mapping

Value	Encoding type	Description
1	FFV2_ENCODING_PASSTHROUGH	On-ramp from the flexible file v1 layout; direct NFSv3/v4 I/O, no chunk envelope
2	FFV2_ENCODING_MOJETTE_SYSTEMATIC	Discrete Radon projections, systematic (data shards passed through)
3	FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC	Discrete Radon projections, non-systematic (all shards transformed)
4	FFV2_ENCODING_RS_VANDERMONDE	Reed-Solomon Vandermonde over GF(2^8); arbitrary (k, m)
5	FFV2_ENCODING_REPLICATED	Chunked replication; N-way redundancy at N x storage
6	FFV2_ENCODING_XOR_PARITY	Single-parity RAID-5 shape; k + 1, m=1, pure XOR
7	FFV2_ENCODING_LINUX_MD_RAID	Linux md/raid6 P + Q double-parity; k + 2, m=2, GF(2^8)

Encoding Type Interoperability The data servers do not interpret erasure-coded data -- they store and return opaque chunks. The NFS wire protocol likewise does not depend on the encoding mathematics. However, a client that writes data using one encoding type MUST be able to read it back, and a different client implementation MUST be able to read data written by the first client if both claim to support the same encoding type. This interoperability requirement means that each registered encoding type MUST fully specify the encoding and decoding mathematics such that two independent implementations produce byte-identical encoded output for the same input. The specification of a new encoding type MUST include one of the following:

A complete mathematical specification of the encoding and decoding algorithms, including all parameters (e.g., field polynomial, matrix construction, element size) sufficient for an independent implementation to produce interoperable results.
A reference to a published patent or pending patent application that contains the algorithm specification. Implementors can then evaluate the licensing terms and decide whether to support the encoding type.
A declaration that the encoding type is a proprietary implementation. In this case, the encoding type name SHOULD include an organizational prefix (e.g., FFV2_ENCODING_ACME_FOOBAR) to signal that interoperability is limited to implementations licensed by that organization.

Option 1 is RECOMMENDED for encoding types intended for broad interoperability. Options 2 and 3 allow vendors to register encoding types for use within their own ecosystems while preserving the encoding type namespace. The rationale for this requirement is that erasure coding moves computation from the server to the client. If the client cannot determine how data was encoded, it cannot decode it. Unlike layout types (where the server controls the storage format), encoding types require client-side agreement on the mathematics.

ffv2_layout4

ffv2_flags4

The ffv2_flags4 The ffv2_flags4 in is a bitmap that allows the metadata server to inform the client of particular conditions that may result from more or less tight coupling of the storage devices. Each flag below describes both the semantics when set and the normative requirement it places on the client. When a flag is not set, the client MUST follow the default behavior described for its unset state.

FFV2_FLAGS_NO_LAYOUTCOMMIT:: When set, the client MAY omit the LAYOUTCOMMIT to the metadata server. When unset, the client MUST send LAYOUTCOMMIT per
FFV2_FLAGS_NO_IO_THRU_MDS:: When set, the client MUST NOT proxy I/O operations through the metadata server, even after detecting a network disconnect to a storage device. When unset, the client MAY retry failed I/O via the metadata server.
FFV2_FLAGS_NO_READ_IO:: When set, the client MUST NOT issue READ against layouts of iomode LAYOUTIOMODE4_RW, and MUST instead request a separate layout of iomode LAYOUTIOMODE4_READ for any read I/O. When unset, the client MAY issue READ against either iomode.
FFV2_FLAGS_WRITE_ONE_MIRROR:: When set, the client MAY update only one mirror of each layout segment (see ) and rely on the metadata server (or a proxy server acting on its behalf) or a peer data server to propagate the update to the remaining mirrors. When unset, the client MUST update all mirrors. The metadata server MUST NOT set FFV2_FLAGS_WRITE_ONE_MIRROR on a layout whose ffv2l_mirrors carry more than one distinct ffv2m_encoding_type_data value unless a propagation actor is available that speaks every encoding present in the layout. Cross-encoding propagation requires the actor to decode through the source mirror's encoding transform and re-encode for each target mirror's transform, which the metadata server itself cannot do for chunked encodings (it does not hold the encoded shards); a proxy server is the entity that performs cross-encoding translation. On a mixed-encoding layout without a proxy server for the affected file, the metadata server MUST leave FFV2_FLAGS_WRITE_ONE_MIRROR unset and require the client to update all mirrors directly.
FFV2_FLAGS_ONLY_ONE_WRITER:: When set, the client is the exclusive writer for the layout and MAY issue CHUNK_WRITE without setting cwa_guard, retaining the ability to use CHUNK_ROLLBACK in the event of a write hole caused by overwriting. When unset, the client MUST set cwa_guard on every CHUNK_WRITE so that chunk_guard4 CAS can prevent collisions across concurrent writers.

ffv2_file_info4

The ffv2_file_info4 The ffv2_file_info4 is a new structure that resolves the stateid-vs-fh_vers pairing issue discussed in . In 's flexible file v1 layout, a singleton ffv2ds_stateid was paired with an ffv2ds_fh_vers array, forcing every fh_vers on a data server to share one stateid. In each fh_vers has its own stateid alongside it. The stateid value ffv2fi_stateid MUST carry depends on the coupling mode advertised for the corresponding (ffv2dv_version, ffv2dv_minorversion, ffv2dv_coupling) tuple (see ):

If ffv2dv_coupling for this entry equals FFV2_COUPLING_SYNTHETIC_UIDS (loose coupling), ffv2fi_stateid MUST be the anonymous stateid; the client authenticates to the data server via the synthetic ffv2ds_user / ffv2ds_group () rather than by presenting a meaningful stateid.
If ffv2dv_coupling for this entry has the FFV2_COUPLING_TRUSTED_STATEID flag set, ffv2fi_stateid MUST be the layout stateid the metadata server issued in the LAYOUTGET that produced this layout. The data server validates presented stateids against its per-file trust table populated by TRUST_STATEID ().
If ffv2dv_coupling for this entry has only FFV2_COUPLING_TIGHTLY_COUPLED set (a back-end control protocol other than trusted stateid, with no TRUST_STATEID support advertised), ffv2fi_stateid carries whatever stateid the deployment's back-end control protocol expects the client to present; this document does not further specify that value.

Because ffv2ds_file_info<> has one element per (version, minorversion, coupling) tuple advertised on the data server (parallel to ffv2da_versions<>), a single data server that exposes both loose and tight combinations carries multiple ffv2_file_info4 entries with different stateid values. The client selects one tuple to use for I/O; it presents that tuple's stateid on subsequent CHUNK operations.

ffv2_ds_flags4

The ffv2_ds_flags4 The ffv2_ds_flags4 (in ) flags detail the state of the data servers. With erasure coding algorithms, there are both Systematic and Non-Systematic approaches. In the Systematic approach, the bits for integrity are placed amongst the resulting transformed chunk. Such an implementation would typically see FFV2_DS_FLAGS_ACTIVE data servers with FFV2_DS_FLAGS_REPAIR entries added by the metadata server when a failed ACTIVE has been replaced. With the Non-Systematic approach, the data and integrity live on different data servers. Such an implementation would typically see FFV2_DS_FLAGS_ACTIVE and FFV2_DS_FLAGS_PARITY data servers, again with FFV2_DS_FLAGS_REPAIR entries appearing as needed. The FFV2_DS_FLAGS_REPAIR flag informs the client that the indicated data server is a replacement for a previously failed ACTIVE data server, whose content has been (or is being) reconstructed from the surviving shards of the mirror set. Its payload was placed there by a repair actor executing the flow in rather than directly by the original writer. The flag is the client's indication that reads from this data server return erasure-decoded content rather than content produced by the original write. Clients that rely on write-provenance information (for example, deployments that track which client wrote which generation) SHOULD be aware of the REPAIR flag so they do not treat the reconstructed payload as if it had been written directly by the cg_client_id recorded in the chunk_guard4; the guard values still match across the mirror set by construction, but the physical write path differs. Over the lifetime of a file, a single data server MAY transition ACTIVE -> REPAIR (on replacement) or REPAIR -> ACTIVE (once the metadata server has accepted the reconstructed content as authoritative and the fail-over is complete); the metadata server reflects the current flag set in the next layout it returns. The following paragraphs describe the mechanics of these transitions. The client is not the driver in either direction; both are metadata-server-initiated changes that a client observes only by refreshing its layout.

ACTIVE -> REPAIR:: Triggered when a client (or the metadata server itself, via scrub) reports a failure via LAYOUTERROR against a particular shard. The metadata server picks a target for the reconstructed content -- either an existing FFV2_DS_FLAGS_REPAIR-flagged entry already in the layout, or a fresh data server drawn from an out-of-band deployment pool that the metadata server adds to the layout with FFV2_DS_FLAGS_REPAIR set -- and initiates the client-driven repair flow at . Once the reconstructed shard is written and the metadata server has accepted it (via CHUNK_REPAIRED), the metadata server updates the affected layout to remove the failed entry and mark the target entry as FFV2_DS_FLAGS_REPAIR. The metadata server SHOULD then issue CB_LAYOUTRECALL against any client that holds an outstanding layout for the affected file, so those clients refresh via a subsequent LAYOUTGET and observe the updated flag set. Clients that never issued LAYOUTGET during the incident window observe the new layout on first fetch.
REPAIR -> ACTIVE:: Triggered when the metadata server confirms the reconstructed content has been durably committed and the replaced data server is not returning. This is a metadata-server-internal state change; no client-visible operations are required for the transition itself. The flag update is reflected in the next layout the metadata server hands out (either the layout returned to the next LAYOUTGET, or the layout delivered after a CB_LAYOUTRECALL that the metadata server MAY issue to accelerate the transition).

Both transitions preserve the mirror set's array indexing: the shard formerly held by the failed entry lives at the same array position in the layout, under the REPAIR-flagged entry. A client reads by array position; the flag informs the client of the read's provenance (originally written vs reconstructed) but does not change the shard-index addressing. A client MUST NOT infer that a REPAIR-flagged entry serves a different shard than the ACTIVE entry it replaced. If the same payload identifier appears at the same shard position across an ACTIVE entry (about to be retired) and a REPAIR entry (being promoted) during a transition window, the two entries are guaranteed to carry identical chunk contents (the reconstructed content matches the original by the erasure-coding correctness invariant, and the checksum verifies). There is no client-driven failover to a passive data server; a client that observes a CHUNK_WRITE failure against an ACTIVE MUST report the failure via LAYOUTERROR and rely on the metadata-server-initiated repair flow above to promote a replacement. The FFV2_DS_FLAGS_PROXY flag identifies a data-server entry that names a proxy server rather than a real storage device. A client whose local encoding capabilities cannot cover the file's mirror set receives a layout in which one or more mirror entries have FFV2_DS_FLAGS_PROXY set on their ffv2_data_server4; the client directs I/O for that mirror to the proxy, which translates on behalf of the client. The proxy server protocol itself is specified in the proxy server draft; this document defines only the layout flag (this bit) that lets the metadata server mark a data-server entry as proxy-mediated.

ffv2_data_server4

The ffv2_data_server4 ; /// fattr4_owner ffv2ds_user; /// fattr4_owner_group ffv2ds_group; /// ffv2_ds_flags4 ffv2ds_flags; /// }; ]]> The ffv2_data_server4 (in ) describes a data file and how to access it via the different NFS protocols.

ffv2ds_deviceid names the data server; see the flexible file v1 layout () for the deviceid model this layout inherits.
ffv2ds_efficiency is the metadata-server-assigned mirror ranking used for read-mirror selection ().
ffv2ds_file_info<> pairs a filehandle and a stateid for each (version, minorversion, coupling) tuple the layout advertises on this data server (see and the discussion at ffv2_file_info4 above for the stateid-value rules per coupling mode).
ffv2ds_user and ffv2ds_group are the synthetic uid/gid the client presents in the RPC credentials to the data server under loose coupling (see ). They are present in every ffv2_data_server4 regardless of the coupling advertised on this data server, because the underlying file on the data server has a single uid/gid irrespective of which NFS protocol combination the client uses to reach it. If ffv2dv_coupling for the tuple the client selects has any tight coupling flag set (FFV2_COUPLING_TIGHTLY_COUPLED or FFV2_COUPLING_TRUSTED_STATEID), the client MUST ignore ffv2ds_user and ffv2ds_group; the data server authorizes the write via the trusted stateid table or the back-end control protocol instead of via the synthetic uid. If the client selects a tuple with ffv2dv_coupling = FFV2_COUPLING_SYNTHETIC_UIDS, the client MUST present ffv2ds_user and ffv2ds_group in the RPC credentials.
ffv2ds_flags carries the ffv2_ds_flags4 state (ACTIVE, PARITY, REPAIR, PROXY; see ).

ffv2_data_protection4

The ffv2_data_protection4 The ffv2_data_protection4 (in ) describes the data protection geometry as a pair of counts: the number of data shards (ffv2dp_data, also known as k) and the number of parity or redundancy shards (ffv2dp_parity, also known as m). This structure is used in both layout hints and layout responses, and applies uniformly to all encoding types: Example data protection configurations

Protection Mode	ffv2dp_data	ffv2dp_parity	Total Data Servers	Description
Mirroring (3-way)	1	2	3	3 copies, no encoding
Striping (6-way)	6	0	6	Parallel I/O, no redundancy
RS Vandermonde 4+2	4	2	6	Tolerates 2 data-server failures
Mojette-sys 8+2	8	2	10	Tolerates 2 data-server failures

By expressing all protection modes as (ffv2dp_data, ffv2dp_parity) pairs, a single structure serves mirroring, striping, and all erasure encoding types. The encoding type () determines how the shards are encoded; the protection structure determines how many shards there are. The total number of data servers required is ffv2dp_data + ffv2dp_parity. The storage overhead is ffv2dp_parity / ffv2dp_data (e.g., 50% for 4+2, 25% for 8+2).

ffv2_encoding_type_data4

The ffv2_encoding_type_data4 The ffv2_encoding_type_data4 (in ) describes the data protection geometry for the layout. All encoding types carry an ffv2_data_protection4 () specifying the number of data and parity shards. The encoding type enum determines how the shards are encoded; the protection structure determines how many shards there are. Although every arm of the union currently carries the same type, the union form is intentional. Future revisions of this specification may assign distinct arm types to specific coding types; using a union now avoids an incompatible change to the XDR at that time. The (data, parity) tuple is interpreted per encoding type:

FFV2_ENCODING_PASSTHROUGH preserves the flexible file v1 layout-style notation for backward compatibility: ffv2dp_data is 1 and ffv2dp_parity is the number of additional copies (e.g., ffv2dp_parity=2 for 3-way mirroring). The "1" data carrier is the file as stored; the ffv2dp_parity additional copies are the flexible file v1 layout mirror replicas.
FFV2_ENCODING_REPLICATED uses the N + 0 notation: ffv2dp_data is the number of replicas (e.g., ffv2dp_data=3 for 3-way mirroring) and ffv2dp_parity MUST be 0. Every replica is a full, independent data carrier; mirroring carries no parity reconstruction.
Erasure encoding types (FFV2_ENCODING_RS_VANDERMONDE, FFV2_ENCODING_MOJETTE_SYSTEMATIC, FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC, and any future types subsequently registered in the IANA registry established by this document) use ffv2dp_data >= 2 and ffv2dp_parity >= 1.

ffv2_stripes4

The ffv2_stripes4 structure ; /// }; ]]> Each stripe contains a set of data servers in ffv2s_data_servers. If the stripe is part of a ffv2_encoding_type_data4 of FFV2_ENCODING_PASSTHROUGH or FFV2_ENCODING_REPLICATED, then the length of ffv2s_data_servers MUST be 1: under both encoding types each stripe's data lives on a single data server, with replica multiplicity expressed in ffv2l_mirrors rather than in ffv2s_data_servers. ffv2_stripes4 has no direct counterpart in . In the flexible file v1 layout, a mirror's data servers are listed directly on the mirror (via ffm_data_servers<> on ff_mirror4). The flexible file v2 layout introduces this intermediate stripes level so a single mirror MAY carry multiple stripe groups, and pushes the striping-mode metadata (ffv2m_striping, ffv2m_striping_unit_size) down onto the mirror (see ) rather than onto the layout as does with the layout-level ffl_stripe_unit. The ffv2_striping4 enum (FFV2_STRIPING_NONE / _SPARSE / _DENSE) inherits its meaning from and ; the flexible file v2 layout evolution is that the striping mode is a per-mirror decision rather than a per-layout one.

ffv2_mirror4

The ffv2_mirror4 ; /// }; ]]> The ffv2_mirror4 (in ) is the flexible file v2 layout counterpart to ff_mirror4 in . The flexible file v2 layout is a semantic superset of the flexible file v1 layout: any ff_mirror4 can be re-expressed as an ffv2_mirror4 by setting ffv2m_encoding_type_data = FFV2_ENCODING_PASSTHROUGH, ffv2m_striping and ffv2m_striping_unit_size to the flexible file v1 layout's layout-level values, ffv2m_checksum_algorithm to CHECKSUM_ALG_NONE, and wrapping the flexible file v1 layout's ffm_data_servers<> in a single-element ffv2m_stripes<>. The reverse does not hold: ffv2_mirror4 instances whose ffv2m_encoding_type_data is anything other than FFV2_ENCODING_PASSTHROUGH have no ff_mirror4 representation. Relative to ff_mirror4, ffv2_mirror4 adds the following per-mirror fields:

ffv2m_encoding_type_data: per-mirror encoding type choice (see ). This enables a single layout to carry mirrors under different encodings (e.g., a PASSTHROUGH mirror alongside a Reed-Solomon mirror over the same file; see ) -- the transition-window and per-mirror-optimization patterns that motivated the flexible file v2 layout.
ffv2m_striping and ffv2m_striping_unit_size: pull the striping-mode decision from the layout level down to the mirror level. The flexible file v1 layout's ffl_stripe_unit is layout-wide; in the flexible file v2 layout different mirrors of the same file MAY use different striping configurations.
ffv2m_client_id: writer identity for chunk_guard4 CAS (see ). No flexible file v1 layout counterpart; introduced for the CHUNK operation set that the flexible file v2 layout adds.
ffv2m_checksum_algorithm: per-mirror integrity-checksum algorithm. No flexible file v1 layout counterpart; introduced for the per-chunk checksum integrity the flexible file v2 layout adds.
ffv2m_stripes<>: replaces the flexible file v1 layout's flat ffm_data_servers<> with an array of ffv2_stripes4 (see ), allowing a single mirror to carry multiple stripe groups.

The ffv2m_checksum_algorithm field names the checksum algorithm the client MUST use when computing cwa_checksums on CHUNK_WRITE and cwra_checksums on CHUNK_WRITE_REPAIR, and the algorithm the client MUST expect in cr_checksum on CHUNK_READ responses, for chunks in this mirror. The metadata server picks the algorithm at LAYOUTGET time; the value is one of the registered checksum_algorithm4 codes (see ). Different mirrors of the same file MAY name different checksum algorithms, supporting transition cases where one mirror is being migrated to a stronger algorithm while others retain the previous algorithm. A client that does not implement the algorithm named in ffv2m_checksum_algorithm MUST return the layout with NFS4ERR_LAYOUT_CHECKSUM_NOT_SUPPORTED; the metadata server may then issue a new layout naming a different algorithm the client supports, or deny the layout request. The ffv2m_client_id is a 32-bit value, assigned by the metadata server at layout-grant time, that the client MUST use as the cg_client_id field of chunk_guard4 (see ) in every CHUNK_WRITE it issues against the mirror's data servers. Its purpose is to satisfy the 32-bit-per-field budget of chunk_guard4 while preserving the guarantee that concurrent writers on the same file are distinguishable:

The NFSv4 clientid4 () is a 64-bit identifier; does not constrain how a server populates its bits, and the bit-layout choices made by any particular metadata server implementation are not visible to the client and MUST NOT be assumed by the client. Folding clientid4 to 32 bits locally at the client therefore risks colliding with another client's folded value, which would violate the uniqueness contract on chunk_guard4.
Only the metadata server has the information needed to avoid such collisions: it sees every layout it grants on a file and can assign a dense 32-bit ffv2m_client_id that is guaranteed distinct from the ffv2m_client_ids assigned to other clients holding concurrent write layouts on the same file. The metadata server MUST assign ffv2m_client_id subject to this uniqueness rule.
Because cg_client_id participates in the deterministic tiebreaker for racing writers (see ), having the metadata server assign it also lets the metadata server influence which client wins contention by choosing the numeric ordering of the values it hands out. Specific ordering policies are implementation-defined and out of scope for this document, but the protocol mechanism is present.

An ffv2m_client_id is scoped to the file and layout for which it was granted. A client that holds layouts on two different files may receive two different ffv2m_client_ids from the same metadata server, and a client that relinquishes and later re-acquires a layout on a given file MAY be assigned a different ffv2m_client_id. ffv2m_client_id does NOT survive a metadata server restart: the metadata server reassigns values as clients reclaim layouts during the grace period. The ffv2m_encoding_type_data is which encoding type is used by the mirror. The ffv2m_striping selects the striping method used by the mirror. The three permissible values are FFV2_STRIPING_NONE (the mirror is not striped), FFV2_STRIPING_SPARSE (stripe units are mapped to the same physical offset on every data server, leaving holes), and FFV2_STRIPING_DENSE (stripe units are packed contiguously on each data server without holes). See for the mapping math for each option. The ffv2m_striping_unit_size is the stripe unit size used by the mirror. The minimum stripe unit size is 64 bytes. If the value of ffv2m_striping is FFV2_STRIPING_NONE, then the value of ffv2m_striping_unit_size MUST be 1. The ffv2m_stripes is the array of stripes for the mirror; the length of the array is the stripe count. If there is no striping or the ffv2m_encoding_type_data is FFV2_ENCODING_PASSTHROUGH, then the length of ffv2m_stripes MUST be 1. Under FFV2_ENCODING_REPLICATED the file MAY be striped within each replica; the constraint that ffv2s_data_servers length is 1 still applies, but ffv2m_stripes can carry multiple stripes.

ffv2_layout4

The ffv2_layout4 ; /// ffv2_flags4 ffv2l_flags; /// uint32_t ffv2l_stats_collect_hint; /// }; ]]> The ffv2_layout4 (in ) describes the Flexible File Layout Version 2. The ffv2l_mirrors field is the array of mirrored storage devices that provide the storage for the current stripe; see . The ffv2l_stats_collect_hint field provides a hint to the client on how often the server wants it to report LAYOUTSTATS for a file. The time is in seconds.

The Relationship between Metadata Server and Data Servers As shown in if the ffv2m_encoding_type_data is FFV2_ENCODING_PASSTHROUGH or FFV2_ENCODING_REPLICATED, then each of the stripes MUST only have 1 storage device. I.e., the length of ffv2s_data_servers MUST be 1. The erasure-coding encoding types distribute shards across multiple storage devices and so carry multiple entries in ffv2s_data_servers. The abstraction here is that for FFV2_ENCODING_PASSTHROUGH and FFV2_ENCODING_REPLICATED, each stripe describes exactly one data server. And for the erasure-coded encoding types, each of the stripes describes a set of data servers to which the shards are distributed. Further, the payload length can be different per stripe.

ffv2_layouthint4

The ffv2_layouthint4 ; /// ffv2_data_protection4 ffv2lh_preferred_protection; /// uint32_t ffv2lh_stripe_unit; /// uint64_t ffv2lh_expected_file_size; /// }; ]]> The ffv2_layouthint4 (in ) describes the layout_hint (see ) that the client can provide to the metadata server. The client provides four hints. All four are advisory; the metadata server MAY honor any subset and MAY override any of them per administrative policy.

ffv2lh_supported_types: An ordered list of encoding types the client supports, with the most preferred type first. The server SHOULD select a type from this list but MAY choose any type it supports. If the server does not support any of the listed types, it returns NFS4ERR_ENCODING_NOT_SUPPORTED, and the client can retry with a different list to discover the overlapping set.
ffv2lh_preferred_protection: The client's preferred data protection geometry as a (ffv2dp_data, ffv2dp_parity) pair. The server SHOULD honor this hint but MAY override it based on server-side policy. A server that manages data protection via administrative policy (e.g., per-directory or per-export objectives) will typically ignore this hint and return the geometry dictated by policy.
ffv2lh_stripe_unit: The client's preferred stripe unit size in bytes. A value of zero means "no hint" -- the metadata server selects a stripe unit from policy or from file history. When the value is non-zero, the metadata server SHOULD use it as the per-mirror stripe unit when policy permits. The metadata server MAY round to a server-supported alignment. The metadata server MAY return NFS4ERR_INVAL when a non-zero value is below an implementation-defined floor or above an implementation-defined ceiling. The hint SHOULD be a power of two.
ffv2lh_expected_file_size: The client's hint at the file's eventual size in bytes. A value of zero means "no hint." When the value is non-zero, the metadata server MAY use it to decide whether to issue a striped layout (typically for files large enough that striping pays for itself) versus a non-striped layout (typically for files expected to remain small). The hint applies to the layout being requested and has no retroactive effect on layouts already issued. The hint is never required to be accurate; a file is free to grow beyond its hint without protocol penalty, and the metadata server is free to ignore the hint.

The stripe_unit and expected_file_size hints are most useful at the LAYOUTGET that follows file creation, when the file is at size zero and the metadata server has no usage history to drive its own striping decision. On a LAYOUTGET against an already-grown file, the metadata server SHOULD ignore both hints and use the file's actual size and access history. For example, a client that prefers Mojette systematic with 8+2 protection, 1 MiB stripe units, and is creating a file expected to grow to 16 GiB would send: A server with a policy of RS 4+2 for this directory would ignore both encoding hints and return a layout with FFV2_ENCODING_RS_VANDERMONDE and (ffv2dp_data=4, ffv2dp_parity=2). A server without erasure coding might return FFV2_ENCODING_REPLICATED with (ffv2dp_data=3, ffv2dp_parity=0) for 3-way mirroring with per-chunk integrity, or FFV2_ENCODING_PASSTHROUGH with (ffv2dp_data=1, ffv2dp_parity=2) for 3-way flexible file v1 layout-compatible mirroring without per-chunk integrity. A server may also use ffv2lh_expected_file_size as a striping gate: a deployment that wants to avoid the runway and bookkeeping overhead of striping small files (which dominate file-count even when they do not dominate byte count) can use a single-mirror non-striped layout for any LAYOUTGET whose hint is below a configured threshold, and a striped layout above it. Without the hint the metadata server must either always stripe, never stripe, or rely on observing the file's growth -- which is exactly the friction the hint exists to remove.

Encoding Negotiation Because the encoding type registry is expected to grow over time (new erasure encoding types are added, older ones fall out of favor, vendors register private codes; see ), neither clients nor metadata servers are required to implement every registered encoding. The protocol negotiates encoding capabilities via ffv2_layouthint4:

Client-side advertisement:

A client that wishes to influence encoding selection SHOULD send the set of encodings it actually implements in ffv2lh_supported_types. A client MUST NOT claim support for an encoding it cannot encode or decode: a false advertisement produces silent data unavailability when the resulting layout is issued.

Metadata-server selection at file creation:

When the LAYOUTGET is against a newly-created file (the file has no committed data yet), the metadata server has real discretion. It SHOULD select an encoding from the client's ffv2lh_supported_types list when the server's policy permits. The server MAY override the hint when its policy dictates a specific encoding (for example, per-export objectives); in that case the server issues a layout with the policy-dictated encoding and the client MUST either honor it or fail its I/O with NFS4ERR_ENCODING_NOT_SUPPORTED.

Metadata-server selection for an existing file:

When the LAYOUTGET is against a file that already has committed data on the data servers, the file's encoding is fixed by the bytes already written; the metadata server does not choose an encoding at layout-issue time. Re-encoding an existing file is a separate operation (see the heterogeneous mirror set primitive in and the migration paths in the proxy server draft), not a consequence of a LAYOUTGET hint. In this case ffv2lh_supported_types is not a selection input; it is an admissibility check. The metadata server issues a layout with the file's actual encoding and evaluates whether the client can consume it:

If the file's encoding is in the client's ffv2lh_supported_types list, the metadata server issues the layout normally.
If it is not, the metadata server takes one of the fallback actions enumerated in "Fallback when no overlap exists" below (return NFS4ERR_ENCODING_NOT_SUPPORTED, route I/O through the metadata server, or route through a translating proxy server).

Fallback when no overlap exists:

If the server's policy cannot be satisfied by any encoding the client supports, the metadata server has three options:

Return NFS4ERR_ENCODING_NOT_SUPPORTED on the LAYOUTGET. The client MAY retry with a different (possibly empty) ffv2lh_supported_types list to learn the server's encoding repertoire through the errors returned.
Fall back to I/O via the metadata server itself, so the client's reads and writes are satisfied by the metadata server translating to the underlying data server encoding on the client's behalf (see for the metadata-server I/O fallback). This is correct but serializes all I/O for the encoding-ignorant client through a single actor.
Route the client through a translating proxy that understands both the file's native encoding and an encoding the client does support. The metadata server issues a layout with the proxy's data-server entry carrying FFV2_DS_FLAGS_PROXY and a coding_type the client does support (typically FFV2_ENCODING_REPLICATED for a minimal NFSv4.2 client, or FFV2_ENCODING_PASSTHROUGH / a flat NFSv3 view for an NFSv3 client). The proxy encodes and decodes on the fly against the real data servers. This preserves parallel I/O for the encoding-ignorant client that the metadata-server I/O fallback loses. The proxy registration, directive, and credential-forwarding rules are defined in the proxy server draft; this draft defines only the layout flag (FFV2_DS_FLAGS_PROXY in ) that makes the proxy visible to the client.

Options (1), (2), and (3) are not mutually exclusive: a given deployment MAY implement any combination. A deployment that supports (3) covers all the clients that (1) and (2) would cover and additionally preserves parallel I/O for encoding-ignorant clients.

Runtime encoding change:

If a metadata server changes its encoding policy after layouts have been issued (for example, a deployment upgrade that retires an older encoding), the metadata server MUST recall the affected layouts via CB_LAYOUTRECALL and may re-issue new layouts with the new encoding. Clients that do not support the new encoding LAYOUTRETURN with NFS4ERR_ENCODING_NOT_SUPPORTED, and the server either grants a layout using a mutually-supported encoding or the client falls back to I/O via the metadata server.

This mechanism deliberately avoids a separate capability-bit handshake at EXCHANGE_ID. ffv2_layouthint4 already provides per-request negotiation; adding a session-level capability set would duplicate it and would complicate encoding upgrades without additional value, because a client that genuinely upgrades its encoding set at runtime can simply update the ffv2lh_supported_types on its next LAYOUTGET. Note: In ffv2_encoding_type_data4 is an enumerated union with the payload of each arm being defined by the protection type. ffv2m_client_id tells the client which id to use when interacting with the data servers. The ffv2_layout4 structure (see ) specifies a layout in that portion of the data file described in the current layout segment. It is either a single instance or a set of mirrored copies of that portion of the data file. When mirroring is in effect, it protects against loss of data in layout segments. While not explicitly shown in , each layout4 element returned in the logr_layout array of LAYOUTGET4res (see ) describes a layout segment. Hence, each ffv2_layout4 also describes a layout segment. It is possible that the file is concatenated from more than one layout segment. Each layout segment MAY represent different striping parameters. The ffv2m_striping_unit_size field (inside each ffv2_mirror4) is the stripe unit size in use for that mirror. The stripe width W is given by the number of elements in ffv2s_data_servers within each ffv2_stripes4 (the count of data servers over which each stripe is spread). If ffv2m_striping is FFV2_STRIPING_NONE the mirror is unstriped and ffv2m_striping_unit_size MUST be 1 (matching the FFV2_STRIPING_NONE rule in and ); when ffv2m_striping is FFV2_STRIPING_SPARSE or FFV2_STRIPING_DENSE the field carries the stripe unit size in bytes with a minimum of 64. The mapping scheme (sparse or dense) is selected per mirror by ffv2m_striping and is detailed in . Stripe unit size and stripe count MAY differ between mirrors in the same layout segment. In particular, mirrors of different encoding types (see ) have stripe counts determined by their respective (ffv2dp_data, ffv2dp_parity) protection structures, and there is no requirement that those structures match across mirrors. Each mirror is self-consistent internally; cross-mirror coherence is at the byte level (every mirror represents the same file bytes), not at the stripe-geometry level. The ffv2l_mirrors field represents an array of state information for each mirrored copy of the current layout segment. Each element is described by a ffv2_mirror4 type. ffv2ds_deviceid provides the deviceid of the storage device holding the data file. ffv2ds_file_info is an array of ffv2_file_info4 structures, each pairing a filehandle (ffv2fi_fh_vers) with a stateid (ffv2fi_stateid). There MUST be exactly as many elements in ffv2ds_file_info as there are in ffv2da_versions. Each element of the array corresponds to a particular combination of ffv2dv_version, ffv2dv_minorversion, and ffv2dv_coupling provided for the device. The array allows for server implementations that have different filehandles and stateids for different combinations of version, minor version, and coupling strength. See for how to handle versioning issues between the client and storage devices. For tight coupling, ffv2fi_stateid provides the stateid to be used by the client to access the file. The metadata server registers ffv2fi_stateid with each tight coupling capable storage device via TRUST_STATEID (see ) before returning the layout; the storage device validates subsequent CHUNK operations against its trust table. For loose coupling and an NFSv4 storage device (necessarily a PASSTHROUGH mirror per , since non-PASSTHROUGH encodings require tight coupling), the client MUST use the anonymous stateid to perform I/O on the storage device, because the metadata server stateid has no meaning to a storage device that is not participating in the control protocol. In this case the metadata server MUST set ffv2fi_stateid to the anonymous stateid. For an NFSv3 storage device (ffv2dv_version = 3), the tight coupling model does not apply: requires ffv2dv_coupling to equal FFV2_COUPLING_SYNTHETIC_UIDS whenever ffv2dv_version equals 3, because NFSv3 has no wire encoding for stateids. The corresponding ffv2fi_stateid element in the ffv2ds_file_info array MUST therefore be the anonymous stateid and is unused; an NFSv3 data server uses the synthetic-uid fencing model (see ) rather than a stateid-based trust table. This specification of the ffv2fi_stateid restricts both models for NFSv4.x storage protocols:

loosely couple: the stateid has to be an anonymous stateid
tightly couple: the stateid has to be a global stateid

By pairing each ffv2fi_fh_vers with its own ffv2fi_stateid inside ffv2_file_info4, the flexible file v2 layout addresses a limitation in the flexible file v1 layout where a single stateid was shared across all filehandles. Whether the ffv2fi_stateid values across an ffv2_file_info4 array are distinct depends on each entry's coupling mode per the rules above. Loose-coupling and NFSv3 entries MUST carry the anonymous stateid; those entries are therefore byte-identical by mandate. Tight-coupling entries carry stateids the metadata server assigned and registered via TRUST_STATEID; the metadata server MAY assign these distinctly per filehandle version or MAY reuse the same stateid across entries. The client MUST treat each (ffv2fi_fh_vers, ffv2fi_stateid) pair as an opaque, independent authorization unit. The client MUST NOT compare ffv2fi_stateid values across entries in the array and MUST NOT infer any relationship between two entries whose stateid values are byte-identical. When the client selects an entry to use for I/O, it presents that entry's stateid with that entry's filehandle; other entries in the array are unused for that I/O. For loosely coupled storage devices, ffv2ds_user and ffv2ds_group provide the synthetic user and group to be used in the RPC credentials that the client presents to the storage device to access the data files. For tightly coupled storage devices, the user and group on the storage device will be the same as on the metadata server; that is, if ffv2dv_coupling has any tight coupling flag set (see ), then the client MUST ignore both ffv2ds_user and ffv2ds_group. The allowed values for both ffv2ds_user and ffv2ds_group are specified as owner and owner_group, respectively, in . For NFSv3 compatibility, user and group strings that consist of decimal numeric values with no leading zeros can be given a special interpretation by clients and servers that choose to provide such support. The receiver may treat such a user or group string as representing the same user as would be represented by an NFSv3 uid or gid having the corresponding numeric value. Note that if using Kerberos for security, the expectation is that these values will be a name@domain string. ffv2ds_efficiency describes the metadata server's evaluation as to the effectiveness of each mirror. Note that this is per layout and not per device as the metric may change due to perceived load, availability to the metadata server, etc. Higher values denote higher perceived utility. The way the client can select the best mirror to access is discussed in .

Error Codes from LAYOUTGET provides little guidance as to how the client is to proceed with a LAYOUTGET that returns an error of either NFS4ERR_LAYOUTTRYLATER, NFS4ERR_LAYOUTUNAVAILABLE, and NFS4ERR_DELAY. Within the context of this document:

NFS4ERR_LAYOUTUNAVAILABLE:: there is no layout available and the I/O is to go to the metadata server. Note that it is possible to have had a layout before a recall and not after.
NFS4ERR_LAYOUTTRYLATER:: there is some issue preventing the layout from being granted. If the client already has an appropriate layout, it should continue with I/O to the storage devices.
NFS4ERR_DELAY:: there is some issue preventing the layout from being granted. If the client already has an appropriate layout, it should not continue with I/O to the storage devices.

Client Interactions with FFV2_FLAGS_NO_IO_THRU_MDS FFV2_FLAGS_NO_IO_THRU_MDS is normative: when the metadata server sets FFV2_FLAGS_NO_IO_THRU_MDS on a layout, the client MUST NOT proxy I/O for that layout through the metadata server, even after detecting a network disconnect to a storage device (). A client that cannot reach a storage device on which it holds a NO_IO_THRU_MDS layout MUST return the layout via LAYOUTRETURN and reacquire (via LAYOUTGET), at which point the metadata server chooses whether to grant a new layout with the flag cleared, grant a layout naming a different storage device, or fall back to metadata-server-terminated I/O via the encoding-negotiation path () with the flag cleared. The NO_IO_THRU_MDS flag is not advisory; it is an instruction the client MUST honor. When I/O through the metadata server is required (for example, via the encoding-negotiation fallback path in ), the metadata server MUST clear NO_IO_THRU_MDS on the fallback layout it issues. A client MUST NOT interpret a set NO_IO_THRU_MDS flag as advisory or bypass it.

LAYOUTCOMMIT The flexible file v2 layout does not use lou_body inside the loca_layoutupdate argument to LAYOUTCOMMIT. If lou_type is LAYOUT4_FLEX_FILES_V2, the lou_body field MUST have a zero length (see ).

Interactions between Devices and Layouts The file layout type is defined such that the relationship between multipathing and filehandles can result in either 0, 1, or N filehandles (see ). Some rationales for this are clustered servers that share the same filehandle or allow for multiple read-only copies of the file on the same storage device. In the flexible file v2 layout, while there is an array of filehandles, they are independent of the multipathing being used. If the metadata server wants to provide multiple read-only copies of the same file on the same storage device, then it should provide multiple mirrored instances, each with a different ffv2_device_addr4. The client can then determine that, since each of the ffv2fi_fh_vers values within ffv2ds_file_info are different, there are multiple copies of the file for the current layout segment available.

Handling Version Errors When the metadata server provides the ffv2da_versions array in the ffv2_device_addr4 (see ), the client is able to determine whether or not it can access a storage device with any of the supplied combinations of ffv2dv_version, ffv2dv_minorversion, and ffv2dv_coupling. However, due to the limitations of reporting errors in GETDEVICEINFO (see Section 18.40 in ), the client is not able to specify which specific device it cannot communicate with over one of the provided ffv2dv_version and ffv2dv_minorversion combinations. Using ffv2_ioerr4 () inside either the LAYOUTRETURN (see ) or the LAYOUTERROR (see and of this document), the client can isolate the problematic storage device. The error code to return for LAYOUTRETURN and/or LAYOUTERROR is NFS4ERR_MINOR_VERS_MISMATCH. It does not matter whether the mismatch is a major version (e.g., client can use NFSv3 but not NFSv4) or minor version (e.g., client can use NFSv4.1 but not NFSv4.2), the error indicates that for all the supplied combinations for ffv2dv_version and ffv2dv_minorversion, the client cannot communicate with the storage device. The client can retry the GETDEVICEINFO to see if the metadata server can provide a different combination, or it can fall back to doing the I/O through the metadata server.

Striping The flexible file v2 layout inherits the dense and sparse striping dispositions defined by the file layout type in . The disposition for a given mirror is selected by the ffv2m_striping field (see ) and applies to every data server in that mirror's ffv2s_data_servers list. Three values are permitted:

FFV2_STRIPING_NONE:: The mirror is not striped. ffv2m_striping_unit_size MUST be 1 and ffv2m_stripes MUST contain exactly one stripe. The entire mirror lives on that stripe's single data server list, with no offset transformation.
FFV2_STRIPING_SPARSE:: Logical offsets within the file map to the same numeric offset on each data server. A data server that does not own the stripe unit at a given logical offset presents a hole at that offset. This is the simpler model and matches the mental picture of "the file is laid out end-to-end on each data server, but each data server stores only its stripe units".
FFV2_STRIPING_DENSE:: Stripe units owned by a given data server are packed contiguously on that data server, with no holes. The logical offset is transformed into a compact physical offset on the target data server. This matches pre-existing deployments that follow the dense layout convention of .

The mapping math for sparse and dense is given in . Common definitions apply to both.

Sparse and dense stripe mapping math

Recovering from Client I/O Errors The pNFS client may encounter errors when directly accessing the storage devices. However, it is the responsibility of the metadata server to recover from the I/O errors. When the LAYOUT4_FLEX_FILES_V2 layout type is used, the client MUST report the I/O errors to the server at LAYOUTRETURN time using the ffv2_ioerr4 structure (see ). The metadata server analyzes the error and determines the required recovery operations such as recovering media failures or reconstructing missing data files. The metadata server MUST recall any outstanding layouts to allow it exclusive write access to the stripes being recovered and to prevent other clients from hitting the same error condition. In these cases, the server MUST complete recovery before handing out any new layouts to the affected byte ranges. The client's retry disposition depends on which encoding the affected mirror uses. The two subsections below split the encoding types into two families: mirrored / PASSTHROUGH (where the storage device holds the file's bytes directly, so retrying the I/O through the metadata server is possible), and the chunked encodings (where the storage device holds encoded shards, so retrying the I/O through the metadata server is meaningful only if a proxy server is available to translate).

Retry policy for mirrored and PASSTHROUGH encodings For a mirror using FFV2_ENCODING_REPLICATED or FFV2_ENCODING_PASSTHROUGH, the storage device holds the file's bytes directly (no chunk envelope, no encoding transform), and an ordinary NFS READ or WRITE on the metadata server accesses the same bytes. Although the client implementation has the option to propagate a corresponding error to the application that initiated the I/O operation and drop any unwritten data, the client should attempt to retry the original I/O operation by either requesting a new layout or sending the I/O via regular NFSv4.1+ READ or WRITE operations to the metadata server. The client SHOULD attempt to retrieve a new layout and retry the I/O operation using the storage device first and only retry the I/O operation via the metadata server if the error persists.

Retry policy for chunked (erasure-coded) encodings For a mirror using any chunked encoding (any FFV2ENCODING* value other than FFV2_ENCODING_PASSTHROUGH), the storage device holds encoded shards inside chunk envelopes rather than the file's bytes, and retrying the I/O through the metadata server as a regular NFS READ or WRITE against the file is not an equivalent fallback: the metadata server does not hold the encoded shards, and the file's bytes are recoverable only by decoding through the mirror's erasure transform. The client's retry disposition is correspondingly different. For a CHUNK_READ error, the client SHOULD attempt local reconstruction from surviving shards before returning the layout, provided the encoding is a k + m code and the client holds (directly or by fetching from unaffected data servers in the same stripe) at least k surviving shards. A successful local reconstruction satisfies the read; the client MUST still record the ioerr and report it at LAYOUTRETURN so the metadata server can drive repair via CB_CHUNK_REPAIR and the repair-actor flow (). For a CHUNK_WRITE error, or when a CHUNK_READ error cannot be satisfied by local reconstruction (fewer than k surviving shards, or a non-recoverable chunk_guard4 CAS failure), the client SHOULD return the layout with the ioerr recorded after its own storage-device-directed retries (multipath, transient error) have been exhausted. The client MUST NOT retry the same I/O through the metadata server as a regular NFSv4.1+ READ or WRITE against the file: the metadata server does not hold the encoded shards, so an I/O through the metadata server would either be rejected or would bypass the encoding transform and corrupt the file. Retrying the I/O through the metadata server is meaningful for a chunked encoding only when a proxy server is available to translate on the metadata server's behalf: the proxy server admits the client's I/O, performs the encoding transform, and issues the corresponding CHUNK operations to the data servers. When no proxy server is available for the affected file, the client's remaining option is to re-request a layout after the metadata server has driven repair to completion.

Client-Side Protection Modes

Client-Side Mirroring The flexible file v2 layout has a simple model in place for the mirroring of the file data constrained by a layout segment. Each mirror in ffv2l_mirrors is an independent representation of the file's contents for that segment: the XDR (see ) lets each mirror carry its own encoding (ffv2m_encoding_type_data), its own striping pattern (ffv2m_striping and ffv2m_stripes), and its own set of data servers. A single layout MAY combine dissimilar mirrors -- for example, one FFV2_ENCODING_REPLICATED mirror and one FFV2_ENCODING_RS_VANDERMONDE mirror of the same file contents, as sketched in -- and there is no cross-mirror constraint that striping patterns or encoding choices match. There is likewise no assumption that each copy of the mirror is stored identically on the storage devices. For example, one device might employ compression or deduplication on the data. What each mirror MUST provide is the same reconstructed file contents on read (after any decoding through the mirror's encoding transform); the on-disk representation and the wire representation per mirror are consequences of that mirror's independently selected encoding. The metadata server is responsible for determining the number of mirrored copies and the location of each mirror. While the client may provide a hint to how many copies it wants (see ), the metadata server can ignore that hint; in any event, the client has no means to dictate either the storage device (which also means the coupling and/or protocol levels to access the layout segments) or the location of said storage device. The updating of mirrored layout segments is done via client-side mirroring. With this approach, the client is responsible for making sure modifications are made on all copies of the layout segments it is informed of via the layout. If a layout segment is being resilvered to a storage device, that mirrored copy will not be in the layout. Thus, the metadata server (or a proxy server acting on its behalf, if one is available for the file) MUST update that copy until the client is presented it in a layout. If the FFV2_FLAGS_WRITE_ONE_MIRROR is set in ffv2l_flags, the client need only update one of the mirrors (see ). If the client is writing to the layout segments via the metadata server, then the metadata server (or a proxy server acting on its behalf) MUST update all copies of the mirror; see the encoding constraint on FFV2_FLAGS_WRITE_ONE_MIRROR in for the case where doing so requires the propagation actor to translate across encodings. As seen in , during the resilvering, the layout is recalled, and the client has to make modifications through the metadata-server side.

Selecting a Mirror When the metadata server grants a layout to a client, it MAY let the client know how fast it expects each mirror to be once the request arrives at the storage devices via the ffv2ds_efficiency member. While the algorithms to calculate that value are left to the metadata server implementations, factors that could contribute to that calculation include speed of the storage device, physical memory available to the device, operating system version, current load, etc. However, what should not be involved in that calculation is a perceived network distance between the client and the storage device. The client is better situated for making that determination based on past interaction with the storage device over the different available network interfaces between the two; that is, the metadata server might not know about a transient outage between the client and storage device because it has no presence on the given subnet. As such, it is the client that decides which mirror to access for reading the file. The requirements for writing to mirrored layout segments are presented below.

Writing to Mirrors

Single Storage Device Updates Mirrors If the FFV2_FLAGS_WRITE_ONE_MIRROR flag in ffv2l_flags is set, the client MAY update just one of the copies of the layout segment. For this case, the storage device MUST ensure that all copies of the mirror are updated when any one of the mirrors is updated. If the storage device gets an error when updating one of the mirrors, then it MUST inform the client that the original WRITE had an error. The client then MUST inform the metadata server (see ). The client's responsibility with respect to COMMIT is explained in . The client may choose any one of the mirrors and may use ffv2ds_efficiency as described in when making this choice.

Client Updates All Mirrors If the FFV2_FLAGS_WRITE_ONE_MIRROR flag in ffv2l_flags is not set, the client is responsible for updating all mirrored copies of the layout segments that it is given in the layout. A single failed update is sufficient to fail the entire operation. If all but one copy is updated successfully and the last one provides an error, then the client MUST inform the metadata server about the error. The client can use either LAYOUTRETURN or LAYOUTERROR to inform the metadata server that the update failed to that storage device. If the client is updating the mirrors serially, then it SHOULD stop at the first error encountered and report that to the metadata server. If the client is updating the mirrors in parallel, then it SHOULD wait until all storage devices respond so that it can report all errors encountered during the update.

Handling Write Errors When the client reports a write error to the metadata server, the metadata server is responsible for determining if it wants to remove the errant mirror from the layout, if the mirror has recovered from some transient error, etc. When the client tries to get a new layout, the metadata server informs it of the decision by the contents of the layout. The client MUST NOT assume that the contents of the previous layout will match those of the new one. If it has updates that were not committed to all mirrors, then it MUST resend those updates to all mirrors. There is no provision in the protocol for the metadata server to directly determine that the client has or has not recovered from an error. For example, if a storage device was network partitioned from the client and the client reported the error to the metadata server, then the network partition would be repaired, and all of the copies would be successfully updated. There is no mechanism for the client to report that fact, and the metadata server is forced to repair the file across the mirror. If the client supports NFSv4.2, it can use LAYOUTERROR and LAYOUTRETURN to provide hints to the metadata server about the recovery efforts. A LAYOUTERROR on a file is for a non-fatal error. A subsequent LAYOUTRETURN without a ffv2_ioerr4 indicates that the client successfully replayed the I/O to all mirrors. Any LAYOUTRETURN with a ffv2_ioerr4 is an error that the metadata server needs to repair. The client MUST be prepared for the LAYOUTERROR to trigger a CB_LAYOUTRECALL if the metadata server determines it needs to start repairing the file.

Handling Write COMMITs When stable writes are done to the metadata server or to a single replica (if allowed by the use of FFV2_FLAGS_WRITE_ONE_MIRROR), it is the responsibility of the receiving node to propagate the written data stably, before replying to the client. In the corresponding cases in which unstable writes are done, the receiving node does not have any such obligation, although it may choose to asynchronously propagate the updates. However, once a COMMIT is replied to, all replicas MUST reflect the writes that have been done, and this data MUST have been committed to stable storage on all replicas. In order to avoid situations in which stale data is read from replicas to which writes have not been propagated:

A client that has outstanding unstable writes made to single node (metadata server or storage device) MUST do all reads from that same node.
When writes are flushed to the server (for example, to implement close-to-open semantics), a COMMIT must be done by the client to ensure that up-to-date written data will be available irrespective of the particular replica read.

Metadata Server Resilvering of the File The metadata server may elect to create a new mirror of the layout segments at any time. This might be to resilver a copy on a storage device that was down for servicing, to provide a copy of the layout segments on storage with different storage performance characteristics, etc. As the client will not be aware of the new mirror and the metadata server will not be aware of updates that the client is making to the layout segments, the metadata server MUST recall the writable layout segment(s) that it is resilvering. If the client issues a LAYOUTGET for a writable layout segment that is in the process of being resilvered, then the metadata server can deny that request with an NFS4ERR_LAYOUTUNAVAILABLE. The client's fallback while the layout is withheld follows the per-encoding rules in and : for a mirror whose I/O reduces to FFV2_ENCODING_REPLICATED or FFV2_ENCODING_PASSTHROUGH the client MAY perform the I/O through the metadata server as an ordinary NFSv4.1+ READ or WRITE; for a mirror using any chunked encoding the metadata server itself cannot service that I/O (it does not hold the encoded shards), so the client's only fallback path is through a proxy server if one is available for the file. If no proxy server is available, the client MUST wait for the metadata server to complete resilvering and re-issue LAYOUTGET rather than attempt to route the I/O through the metadata server.

Client-Side Erasure Coding Erasure coding takes a data block and transforms it to a payload to send to the data servers (see ). It generates a metadata header and transformed block per data server. The header is metadata information for the transformed block. From now on, the metadata is simply referred to as the header and the transformed block as the chunk. The payload of a data block is the set of generated headers and chunks for that data block. The chunk header carries four conceptual fields, each stored on the data server with the chunk and carried on the wire by every op that reads or writes the chunk. This document does not define a single XDR struct for the chunk header; instead each op names the four fields directly, and the fields appear on the wire under an op-specific prefix.

Guard:: an XDR chunk_guard4 (cg_gen_id, cg_client_id) pair maintained by the data server to serialize concurrent writers (see ). Each accepted CHUNK_WRITE advances the target chunk's guard, and readers observe the accepted guard on the read path.
Owner:: an XDR chunk_owner4 owner triple identifying the writer's cohort (see ). The owner is the identity that lifecycle operations (CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK) address.
Payload identifier:: a writer-chosen uint32_t naming the chunk's position within the payload's shard array. Unlike the guard and owner, the payload identifier is not a sub-struct with named fields; it is a bare scalar carried directly under an op-specific prefix on each op's wire arguments: cwa_payload_id on CHUNK_WRITE, cr_payload_id in the CHUNK_READ result (), and cwra_payload_id on CHUNK_WRITE_REPAIR. The value the three carry names the same conceptual header field; only the wire prefix differs across ops.
Checksum:: a 32-bit CRC computed over the header and the chunk. Because the checksum field is itself part of the header, the computation treats the bytes of that field as zero so that the result is independent of the field's wire value; the writer then stores the computed CRC into the checksum field for transmission. To validate on the read path, the receiver saves the received checksum, treats those bytes as zero, recomputes the CRC over the header and chunk, and compares against the saved value. By combining the two parts of the payload in the CRC, integrity is ensured for both parts.

While the data block might have a length of 4kB, that does not necessarily mean that the length of the chunk is 4kB. That length is determined by the erasure encoding type algorithm. For example, Reed Solomon might have 4kB chunks with the data integrity being compromised by parity chunks. Another example would be the Mojette Transformation, which might have 1kB chunk lengths. The payload contains redundancy which will allow the erasure encoding type algorithm to repair chunks in the payload as it is transformed back to a data block (see ). The protocol provides two levels of payload integrity, consumed at different points in the read path:

Atomicity:: A payload is atomic when all of the chunks that belong to it carry the same chunk_guard4 value (see ). Atomicity alone does NOT imply the bytes are free of corruption; it means only that every chunk in the payload came from one write transaction. A reader detects a non-atomic payload (a torn read across writes) when it assembles a payload and finds differing chunk_guard4 values across chunks.
Integrity:: A payload has integrity when it is atomic AND every contained chunk passes its checksum check. Integrity is the precondition for returning the payload's data block to the application.

The separation matters because the two checks detect different failure modes. Atomicity detects protocol-level failures (racing writers, partial writes, rollback windows); the checksum detects byte-level corruption (network errors, media errors, software bugs in the erasure transform). Neither subsumes the other. The two-level integrity model also reflects a deeper property of distributed writes: last-writer-wins does not apply to a payload spread across independent data servers. The ordering of writes arriving at one data server may differ from the ordering arriving at another; the "last" write on one data server may well be the "first" on another. The chunk_guard4 CAS primitive (see ) resolves this by serializing concurrent writers per chunk rather than by imposing a global order. The erasure coding algorithm itself might not be sufficient to detect all byte-level errors in the chunks. The checksum checks allow the data server to detect chunks with integrity issues; the erasure decoding algorithm can then reconstruct the affected chunks from the remaining integral chunks in the payload.

Encoding a Data Block

Encoding a Data Block Each data block of the file resident in the client's cache of the file will be encoded into N different payloads to be sent to the data servers as shown in . As CHUNK_WRITE (see ) can encode multiple write_chunk4 into a single transaction, a more accurate description of a CHUNK_WRITE is in .

Example of CHUNK_WRITE_args This describes a 3 block write of data from an offset of 1 block in the file. All three chunks share cwa_cohort_id and cwa_client_id, so the cohort identity is presented once; each chunk's writer-chosen opaque co_id appears in the co-indexed cwa_co_ids array (here the client chose 1, 2, 3, but any distinct-per-cohort uint32_t values would be equally valid per ). The data server can construct the cohort record for the i'th chunk from cwa_chunks using cwa_payload_id, cwa_cohort_id + cwa_client_id + cwa_co_ids[i], and the i'th checksum from cwa_checksums. The cwa_chunks are sent together as a byte stream to increase performance. Assuming that there were no issues, illustrates the results. The payload sequence id is implicit in the CHUNK_WRITEargs.

Example of CHUNK_WRITE_res

Worked Example: Calculating the CRC32 The examples in this section and in illustrate checksum computation and verification using CHECKSUM_ALG_CRC32 as the worked algorithm. The other registered checksum algorithms (see ) follow the same pattern -- the algorithm names a function over the header and chunk bytes, the writer fills cs_value with the computed output, and the reader recomputes and compares. Only the algorithm and the cs_value length differ.

CRC32 Before Calculation Assuming the header and payload as in , the crc32 needs to be calculated in order to fill in the cwa_checksums entry field. In this case, the crc32 is calculated over the 4 fields as shown in the header and the cw_chunk. In this example, it is calculated to be 0x21de8. The resulting CHUNK_WRITE is shown in .

CRC32 After Calculation

Decoding a Data Block

Decoding a Data Block When reading chunks via a CHUNK_READ operation, the client will decode them into data blocks as shown in . At this time, the client could detect issues in the integrity of the data. The handling and repair are out of the scope of this document and MUST be addressed in the document describing each erasure encoding type.

Worked Example: Checking the CRC32

CRC32 on the Wire Assuming the CHUNK_READ results as in , the crc32 needs to be checked in order to detect accidental corruption. Conceptually, a header and payload can be built as shown in . The crc32 is calculated over the 4 fields as shown in the header and the cr_chunk. In this example, it is calculated to be 0x21de8; because the calculated value matches the received cr_checksum, no accidental corruption was detected on this payload. (Content authentication -- protection against adversarial modification -- requires a keyed MAC or signature; see .)

CRC32 Being Checked

Write Modes There are three writing modes for erasure coding, aligned with the three workload classes in . The mode is selected by the metadata server using FFV2_FLAGS_ONLY_ONE_WRITER in the ffv2l_flags in the ffv2_layout4 (see ) to inform the client whether it is the only writer to the file or not, and by the client's own understanding of its workload class when the flag is unset.

Single writer:: When FFV2_FLAGS_ONLY_ONE_WRITER is set, the client is the only writer to the file. CHUNK_WRITE with cwa_guard not set can be used to write chunks. There is no write contention, but write holes can occur as the client overwrites old data. The client does not need guarded writes, but it does need the ability to rollback writes. This mode corresponds to Use Case 1 (single writer, multiple readers) in .
Concurrent writers with occasional contention:: When FFV2_FLAGS_ONLY_ONE_WRITER is not set, the client is one of several possible concurrent writers. CHUNK_WRITE with cwa_guard set MUST be used to write chunks. Write holes can be caused by multiple clients writing to the same chunk, so the client needs guarded writes to prevent overwrites and also needs the ability to rollback writes. Racing writers that lose the chunk_guard4 CAS receive NFS4ERR_CHUNK_GUARDED and retry with a refreshed guard. This mode corresponds to Use Case 2 (multiple writers without sustained contention) in .
Concurrent writers on disjoint regions:: A specialization of the concurrent-writers mode above, targeting Use Case 3 (multiple writers, disjoint regions -- the HPC checkpoint pattern) in . The wire primitives are the same as the concurrent-writers mode (cwa_guard set, chunk_guard4 CAS), but the deployment relies on block alignment to keep per-chunk contention rare despite a high overall writer count. Contention that does occur is resolved via the deterministic tiebreaker rule defined in , so racing writers get a stable winner across the mirror set without additional round trips. Deployments that use an XOR-based erasure encoding and expect frequent small edits from this workload class MAY additionally use the delta-write protocol defined in , which lets the client forward per-projection XOR deltas directly to each data server, avoiding client-side read-modify-write of the full stripe on the small-edit path.

In both modes, clients MUST NOT overwrite payloads which already contain non-atomicity. This directly follows from and MUST be handled as discussed there. Once atomicity in the payload has been detected, the client can use those chunks as a basis for read/modify/update. CHUNK_WRITE is a two-pass operation in cooperation with CHUNK_FINALIZE and CHUNK_ROLLBACK. It writes new bytes into the chunk and transitions the chunk to the PENDING state; the data server is responsible for retaining the prior COMMITTED content until the chunk reaches its next stable state. While a chunk is in PENDING or FINALIZED, a subsequent CHUNK_READ does NOT observe the new content (visibility rules of apply: PENDING and FINALIZED chunks are not globally visible; CHUNK_READ returns only COMMITTED content). Concurrent CHUNK_WRITE against a PENDING or FINALIZED chunk is regulated by chunk_guard4 (), not by an implicit lock. A racing writer whose guard check fails receives NFS4ERR_CHUNK_GUARDED; an explicit CHUNK_LOCK holder is signaled by NFS4ERR_CHUNK_LOCKED. No implicit chunk-write lock is acquired by CHUNK_WRITE. A client that requires exclusive access to a chunk MUST invoke CHUNK_LOCK explicitly (). If the CHUNK_WRITE results in a atomic data block, then the client will send a CHUNK_FINALIZE in a subsequent compound to inform the data server that the chunk is finalized and can be overwritten by another CHUNK_WRITE. If the CHUNK_WRITE results in an non-atomic data block, or if the data server returns NFS4ERR_CHUNK_LOCKED, the client reports the condition to the metadata server via LAYOUTERROR with an error code of NFS4ERR_PAYLOAD_NOT_ATOMIC.

Selecting the Repair Actor The repair topology involves three actors communicating along distinct paths, as shown in .

Repair topology | | | client | LAYOUTERROR | Metadata Server | | (detects | | | | error) | | | +-------------+ +--------+--------+ | | (2b) | CB_CHUNK_REPAIR | (RACE or SCRUB) v +-------------+ (4) +-----------------+ | Repair | ---------------> | Data Servers | | actor | CHUNK ops | (mirror set | | (selected | | for affected | | per (2a), | | ranges) | | adopts | | | | lock (3)) | | | +-------------+ +-----------------+ (1) Reporting client LAYOUTERRORs to the metadata server. (2a) Metadata server selects a repair actor (may be a client -- possibly the reporting client -- a data server under tight coupling, or a proxy server). (2b) Metadata server escrows the chunk lock and issues CB_CHUNK_REPAIR to the selected repair actor. (3) Repair actor adopts the lock and drives the repair. (4) Repair actor issues CHUNK_LOCK_ADOPT, CHUNK_WRITE_REPAIR, CHUNK_FINALIZE, CHUNK_COMMIT, and CHUNK_REPAIRED against the mirror set. ]]> The metadata server is the authority that selects the repair actor for a non-atomic payload. The candidate set is any client, any data server (in a tightly coupled deployment), or the proxy server (in a proxy server deployment); the selection is analogous to the way the metadata server assigns per-mirror priority via ffv2ds_efficiency (see ): the protocol does not prescribe the selection algorithm, and each deployment MAY tune its policy. Implementations MAY consider factors such as:

Whether a client holds an active write layout on the affected payload (the client most likely to hold surviving shards in cache).
Whether a client has previously reported atomic shards to the metadata server via LAYOUTSTATS or a prior LAYOUTERROR.
Whether the layout exposes a data server carrying FFV2_DS_FLAGS_REPAIR as a target for reconstructed shards.
Network proximity, observed latency, or recent client load -- the same class of information that informs ffv2ds_efficiency.

The selection algorithm is not normative. What is normative is that every client MUST be prepared to:

Receive a repair request for a payload that the client does not have an outstanding write layout on, and did not write; and
Continue its own workload after reporting NFS4ERR_PAYLOAD_NOT_ATOMIC without itself being selected to repair the payload it reported.

The metadata server signals the selected client via the CB_CHUNK_REPAIR callback (), which identifies the file, the affected ranges (each with its own triggering nfsstat4), and a wall-clock deadline. A client that receives CB_CHUNK_REPAIR for a file for which it does not already hold a layout MUST acquire a layout via LAYOUTGET before attempting the repair. Operational expectations for CB_CHUNK_REPAIR: CB_CHUNK_REPAIR is an exceptional path, triggered only by concurrent writer races or data-server failures. It is not a steady-state operation and its frequency is a function of racing-writer and data-server-failure rates in the deployment rather than of normal client workload. Implementations SHOULD treat the CB_CHUNK_REPAIR handler as rare-path code and avoid over-optimizing it. Implementations SHOULD, however, provision enough client-side compute to handle a repair transaction without stalling their foreground I/O, because foreground throughput during repair is the externally observable cost of this callback.

Repair Protocol: Normative vs. Informative The selection algorithm is non-normative and deployment-tunable. The externally-observable state transitions of the repair flow are normative. The line between the two is drawn at what another party on the wire -- the metadata server, another client, a reader -- can observe. What no other party can see (client-internal ordering, retry policy, whether to CHUNK_READ first to confirm the failure) is left to implementations. The following requirements are normative. An implementation that violates any of these can leak inconsistency or write-holes into the cluster:

Final state flat:: Every shard in every range identified in a CB_CHUNK_REPAIR MUST reach either the COMMITTED state (repaired) or the EMPTY state (rolled back). No shard is left in PENDING or FINALIZED indefinitely.
Lock before write:: The repair actor MUST adopt the lock on every affected range via CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT () before issuing any CHUNK_WRITE_REPAIR, CHUNK_ROLLBACK, or CHUNK_WRITE on a chunk in that range. The lock on the affected chunks is held continuously from the failure that triggered CB_CHUNK_REPAIR through the adoption; at no point is the range unlocked.
Clear the errored state:: On the reconstruction path, the repair actor MUST issue CHUNK_REPAIRED after CHUNK_COMMIT. Without it, readers continue to see holes regardless of on-disk state.
Release locks explicitly:: CHUNK_ROLLBACK does not release chunk locks. On the rollback path the client MUST issue CHUNK_UNLOCK on each affected chunk. A client that walks away without either completing CHUNK_REPAIRED or issuing CHUNK_UNLOCK holds the locks until lease expiry, blocking progress for other writers.
Deadline honored:: The client MUST drive every range to its final flat state before ccra_deadline, or MUST respond to the CB_CHUNK_REPAIR with NFS4ERR_DELAY (requesting an extension), NFS4ERR_ENCODING_NOT_SUPPORTED (declining), or NFS4ERR_PAYLOAD_LOST (declaring the data unrecoverable). A deadline that elapses without any of these leaves the metadata server free to re-select; the client MUST NOT continue repair-related CHUNK operations after the deadline without first re-verifying its layout and the chunk lock state.
Terminal return codes:: NFS4ERR_ENCODING_NOT_SUPPORTED MUST mean "decline; select another client." NFS4ERR_PAYLOAD_LOST MUST mean "the data is not recoverable; do not retry." The metadata server relies on these to decide whether to re-issue.

The following aspects are informative / implementation-defined:

Choice between the reconstruction path (CHUNK_WRITE_REPAIR) and the rollback path (CHUNK_ROLLBACK) on a given range. The protocol MUST support both; the client MAY use either based on its local state and whether reconstruction is feasible from surviving shards.
Ordering among multiple affected ranges in a single CB_CHUNK_REPAIR (parallel or serial).
Whether to issue CHUNK_READ to confirm the failure mode before reconstructing.
Retry policy on transient CHUNK_WRITE_REPAIR errors below the deadline cutoff.
How the repair status is surfaced to local filesystem API callers.

Carrying Out the Repair With the normative framing above, the reconstruction path is:

CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT on each affected range ().
CHUNK_WRITE_REPAIR with the reconstructed data for each non-atomic shard. The client's chunk_owner4 on this and all subsequent operations is the one it presented in the CHUNK_LOCK ADOPT above; prior owners' generation ids are now historical.
CHUNK_FINALIZE and CHUNK_COMMIT to persist the repaired shards.
CHUNK_REPAIRED to clear the errored state.

The rollback path, when reconstruction is not possible:

CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT on each affected range.
CHUNK_ROLLBACK on each affected shard to restore the previously committed content.
CHUNK_UNLOCK on each shard.

In both paths, the repair actor SHOULD target reconstructed shards according to the following fallback order: first, any data server in the layout carrying FFV2_DS_FLAGS_REPAIR; then the data server that reported the failure (the one carrying the failing shard at the range identified by ccr_offset and ccr_count in the CB_CHUNK_REPAIR argument). If neither is available, the client MUST return NFS4ERR_PAYLOAD_LOST on the CB_CHUNK_REPAIR response; the metadata server is then responsible for adding a new REPAIR-flagged data server to the layout (drawn from its out-of-band pool) and re-driving the repair.

Single Writer Mode In single writer mode, the metadata server sets FFV2_FLAGS_ONLY_ONE_WRITER in ffv2l_flags, indicating that no other client holds a write layout for the file. The client sends CHUNK_WRITE with cwa_guard.cwg_check set to FALSE, omitting the guard value. Because only one writer is active, there is no risk of two clients overwriting the same chunk concurrently. The single writer write sequence is:

The client issues CHUNK_WRITE (cwa_guard.cwg_check = FALSE) for each shard. The data server places the written block in the PENDING state and retains a copy of the previous block for rollback.
The client issues CHUNK_FINALIZE to advance the blocks from PENDING to FINALIZED, validating the per-block checksum.
The client issues CHUNK_COMMIT to advance the blocks from FINALIZED to COMMITTED, persisting the block metadata to stable storage.

If the client detects an error after CHUNK_WRITE but before CHUNK_FINALIZE (e.g., a CRC mismatch on a subsequent CHUNK_READ), it issues CHUNK_ROLLBACK to restore the previous block content. CHUNK_ROLLBACK does not lock the chunk; the next CHUNK_WRITE is permitted immediately.

Repairing Single Writer Payloads In single writer mode, non-atomic blocks arise from a client or data server failure during a CHUNK_WRITE / CHUNK_FINALIZE sequence. Because no other writer is active, the original writer is the typical choice for repair, but the metadata server MAY designate any client according to the rules in . A designated client that did not originate the writes MUST follow the rollback path of that section if it cannot reconstruct the payload from surviving shards. The repair sequence when the selected client is the original writer is:

The repair actor issues CHUNK_READ to identify which blocks are in a failed state (PENDING with a CRC mismatch, or in the errored state set by a prior CHUNK_ERROR).
For each errored chunk, the repair actor reconstructs the correct data using the erasure coding algorithm from the surviving atomic chunks (treating each chunk's payload as a shard of the stripe).
The repair actor issues CHUNK_WRITE_REPAIR to write the reconstructed data. CHUNK_WRITE_REPAIR bypasses the guard check and applies different data server policies (e.g., allowing writes to blocks in the errored state).
The repair actor issues CHUNK_FINALIZE and CHUNK_COMMIT to persist the repaired blocks.
The repair actor issues CHUNK_REPAIRED to clear the errored state and make the blocks available for normal reads.

Transitioning from Single Writer Mode to Multiple Writer Mode When a second writer requests a write layout for a file currently covered by a single writer layout (FFV2_FLAGS_ONLY_ONE_WRITER set), the metadata server recalls the existing layout before granting the new request. The sequence is:

The metadata server issues CB_LAYOUTRECALL to the single writer client.
The single writer client drains its outstanding I/O issued under the single writer assumption (CHUNK_WRITE with cwa_guard.cwg_check = FALSE). Operations already underway complete under the layout that authorized them: CHUNK_FINALIZE and CHUNK_COMMIT proceed normally for blocks already written.
Once drained, the single writer client issues LAYOUTRETURN.
The metadata server grants the new writer a layout without FFV2_FLAGS_ONLY_ONE_WRITER set. When the original writer next issues LAYOUTGET, it also receives a layout without the flag. Both clients then operate in multiple writer mode (), supplying cwa_guard.cwg_check = TRUE and a chunk_guard4 on every CHUNK_WRITE.

The transition uses standard NFSv4.1 layout recall semantics (). Drained single writer I/O does not need to be re-issued under multiple writer rules; it completed under the layout that authorized it. If the single writer client fails to return the layout within the recall window, the metadata server escalates to layout revocation (); any single writer writes that did not complete before revocation are repaired via the multiple-writer repair path on subsequent access.

Multiple Writer Mode In multiple writer mode, the metadata server does not set FFV2_FLAGS_ONLY_ONE_WRITER, indicating that concurrent writers may hold write layouts for the file. The client sends CHUNK_WRITE with cwa_guard.cwg_check set to TRUE, supplying the expected prior chunk_guard4 in cwa_guard.cwg_guard so the data server can perform per-chunk CAS. The client obtains the expected prior chunk_guard4 by observing the chunk's current guard first: either cr_guard from CHUNK_READ when the payload is also required, or the corresponding chrr_guards entry from CHUNK_HEADER_READ when only the guard is needed. The write transaction is separately identified by the cohort pair (cwa_cohort_id, cwa_client_id) supplied on the same CHUNK_WRITE. The multiple writer write sequence is:

The client selects a unique cohort pair (cwa_cohort_id, cwa_client_id) for this transaction. cwa_client_id is the client's layout-granted ffv2m_client_id (see ); the cwa_cohort_id is a per-writer opaque identifier the client picks distinct per transaction.
The client issues CHUNK_WRITE (cwa_guard.cwg_check = TRUE) for each chunk. The data server checks that no other client's chunk is in the PENDING state at this offset. If another client's chunk is already pending, the data server returns NFS4ERR_CHUNK_LOCKED with the clr_owner field identifying the lock holder.
On NFS4ERR_CHUNK_LOCKED, the client MUST back off. It issues CHUNK_ROLLBACK for any chunks it has already written in this transaction, then retries after a delay.
If all CHUNK_WRITEs succeed, the client issues CHUNK_FINALIZE and CHUNK_COMMIT as in single writer mode.

The cohort pair ensures that the chunks carrying the shards of an atomic erasure-coded stripe all carry the same (co_cohort_id, co_client_id). A reader that encounters chunks with different cohort pairs knows the stripe is not yet atomic and MUST either retry or report NFS4ERR_PAYLOAD_NOT_ATOMIC.

Repairing Multiple Writer Payloads In multiple writer mode, non-atomic chunks can arise from two sources: a client failure leaving some chunks in PENDING state, or two clients writing different data to the same chunk before one has committed. The metadata server coordinates repair by designating a repair client according to the rules in . The FFV2_DS_FLAGS_REPAIR flag, when present on a data server in the layout, identifies the target data server into which reconstructed shards should be written; it does not by itself identify the repair actor. The repair sequence is:

The repair actor issues CHUNK_LOCK on the affected block range of each data server. If any lock attempt returns NFS4ERR_CHUNK_LOCKED, the repair actor records the existing lock holder's chunk_owner4 and proceeds; the lock holder's data is a candidate for the winning payload.
The repair actor issues CHUNK_READ on all data servers to retrieve the current payload. It examines the chunk_owner4 of each shard to identify which transaction (if any) produced a atomic set across all k data shards.
If a atomic set is found (all k data shards carry the same chunk_guard4), that payload is the winner. The repair actor issues CHUNK_WRITE_REPAIR to copy the winner's data to any data servers whose shard is non-atomic, followed by CHUNK_FINALIZE and CHUNK_COMMIT.
If no atomic set exists (all available payloads are partial), the repair actor selects one transaction's payload as authoritative (typically the one with the most complete set of shards, or the most recent cg_gen_id) and proceeds as above.
After all data servers carry atomic, finalized, committed data, the repair actor issues CHUNK_REPAIRED to clear the errored state and CHUNK_UNLOCK to release the locks acquired in step 1.
The repair actor reports success to the metadata server via LAYOUTRETURN.

Transitioning from Multiple Writer Mode to Single Writer Mode The reverse transition is optional. When the metadata server determines that only one writer holds a write layout for a file (for example, because other writers' layouts have been returned or their leases have expired), it MAY recall the remaining writer's layout and grant a fresh layout with FFV2_FLAGS_ONLY_ONE_WRITER set, restoring the single writer optimization. The metadata server MAY also leave the writer in multiple writer mode indefinitely; single writer mode is an optimization, not a correctness requirement. The metadata server's choice of when to grant FFV2_FLAGS_ONLY_ONE_WRITER is policy and is implementation-defined. A metadata server that aggressively grants single writer mode and then must recall it each time a second writer appears can produce recall churn under workloads with irregular concurrent access: each single writer to multiple writer transition costs a CB_LAYOUTRECALL round trip and drain time for in-flight I/O. Strategies to limit churn include withholding FFV2_FLAGS_ONLY_ONE_WRITER until sustained single writer behavior is observed, deferring the single writer grant after a recent recall, or never granting single writer mode for files with a history of concurrent access.

Reading Chunks The client reads chunks from the data file via CHUNK_READ. The number of chunks in the payload that need to be atomic depend on both the Erasure Encoding Type and the level of protection selected. If the client has enough atomic chunks in the payload, then it can proceed to use them to build a data block. If it does not have enough atomic chunks in the payload, then it can either decide to return a LAYOUTERROR of NFS4ERR_PAYLOAD_NOT_ATOMIC to the metadata server or it can retry the CHUNK_READ until there are enough atomic chunks in the payload. As another client might be writing to the chunks as they are being read, it is entirely possible to read the chunks while they are not atomic. As such, it might even be the non-atomic chunks which contain the new data and a better action than building the data block is to retry the CHUNK_READ to see if new chunks are overwritten.

Whole File Repair Whole-file repair is the case in which too many data servers have failed, or too many chunks have been lost, for the per-range repair flow defined in to reconstruct the file in place. In this case the metadata server MUST either:

Construct a new layout backed by replacement data servers and drive the reconstruction via the proxy server mechanism (a designated data server acts as the source of truth for client I/O during the transition, pushing reconstructed content to the replacement data servers in the background). The proxy server mechanism also covers the non-repair cases where a file's layout must change while remaining available to clients -- policy-driven layout transitions, data server maintenance evacuation, administrative ingest, TLS coverage transition, and filehandle-backend migration.
If the metadata server has no proxy server capable data server available, or the surviving shards are insufficient to reconstruct any portion of the file, terminate the affected byte ranges with NFS4ERR_PAYLOAD_LOST (see ).

Implementations that do not support the proxy server mechanism can still perform recovery for cases where per-range repair suffices, using CB_CHUNK_REPAIR and the repair client selection rules in . Such implementations will surface NFS4ERR_PAYLOAD_LOST on any failure that exceeds per-range repair's reach, including the multi-data-server failure scenarios the proxy server mechanism is intended to handle.

Mixing of Encoding Types Multiple encoding types can be present in a Flexible File Version 2 Layout Type layout. The ffv2_layout4 has an array of ffv2_mirror4, each of which has a ffv2_encoding_type4. Mixing encoding types in a single file's mirror set addresses several use cases:

Assimilation and export:: assimilation of a non-erasure-coded file into an erasure-coded representation, or export of an erasure-coded file to a non-erasure-coded representation.
Online migration between encodings:: for example, from a Reed-Solomon Vandermonde encoding to a Mojette systematic encoding when a read-access-pattern change makes the new encoding a better fit. Both representations remain addressable through the layout throughout the transition.
Cross-encoding recovery:: when one encoding loses data to a correlated failure mode (an encoding implementation bug, a memory-corruption pattern that affects parity shards identically), a second mirror in a different encoding provides an independent recovery path.
Client-capability routing:: a proxy server sees the full mirror set and chooses between encodings on behalf of clients that do not implement every encoding the file is represented in.

Consider a layout that exposes a file in two encodings simultaneously: a PASSTHROUGH mirror over the original byte stream and a Reed-Solomon Vandermonde (FFV2_ENCODING_RS_VANDERMONDE) mirror with 4 active data shards plus 2 parity data servers. A layout for such a file might appear as in . Both representations are active and addressable through the layout simultaneously. This is the transition-window pattern: a file may transiently span encodings while it is being assimilated from a non-flexible-file-v2 source or migrated between encodings. Steady state is homogeneous; the multi-encoding window is what the protocol must accommodate. The active mirrors serve different access patterns concurrently:

A client that speaks only the file-layout READ path issues READ () calls to index 0 (the PASSTHROUGH mirror).
A client that speaks the chunked path issues CHUNK_READ calls to index 1 (the RS_VANDERMONDE mirror).
A proxy server fronting legacy clients chooses between the two encodings on the client's behalf.

All three patterns coexist during the transition.

Example of Mixed Encoding Types in a Layout When performing I/O via a FFV2ENCODING_PASSTHROUGH encoding type, the non-transformed data will be used; whereas with any of the chunked encoding types (any FFV2_ENCODING* value other than FFV2_ENCODING_PASSTHROUGH; see ), a metadata header and transformed block will be sent. Further, when reading data from the instance files, the client MUST be prepared to have one of the encoding types supply data and the other type not to supply data. I.e., the CHUNK_READ call to the data servers in mirror 1 might return rlr_eof set to true (see ), which indicates that there is no data, where the READ call to the data server in mirror 0 might return eof to be false, which indicates that there is data. The client MUST determine that there is in fact data. An example use case is the active assimilation of a file to ensure integrity. As the client is helping to translate the file to the new coding scheme, it is actively modifying the file. As such, it might be sequentially reading the file in order to translate. The READ calls to mirror 0 would be returning data and the CHUNK_READ calls to mirror 1 would not be returning data. As the client overwrites the file, the WRITE call and CHUNK_WRITE call would have data sent to all of the data servers. Finally, if the client reads back a section which had been modified earlier, both the READ and CHUNK_READ calls would return data. The two-mirror layout shown in is the file's full mirror set as known to the metadata server. A client that arrives during the assimilation or migration window above does not necessarily receive that layout; per the proxy server draft, the client gets a single layout naming the proxy server as a single endpoint, and the proxy server selects which of the file's mirrors to read from or write to on the client's behalf. The two-mirror view in this section describes the metadata server's bookkeeping during the transition; the client directs its I/O to a single endpoint.

Steady-state heterogeneous mirrors The transition-window patterns above (assimilation, migration, repair) are the most visible motivations for heterogeneous mirror sets, but they are not the only ones. A file's mirror set MAY be heterogeneous in steady state -- where no transition is in progress and no transition is planned -- when the deployment's storage pools have different encoding capabilities and the file is too large to fit in any single pool. Consider an operator with three 100-TB storage pools. Pool A is a set of NFSv3-only data servers, which is a hard capability constraint: the chunked encodings require NFSv4.2's CHUNK operations, so an NFSv3-only pool can hold data only under an FFV2_ENCODING_PASSTHROUGH mirror. Pools B and C are NFSv4.2 data servers, which are encoding-agnostic at the wire level (CHUNK_READ and CHUNK_WRITE just move opaque chunk payloads; the encoding transform is client-side). The operator, however, has chosen to write Pool B under FFV2_ENCODING_RS_VANDERMONDE and Pool C under FFV2_ENCODING_MOJETTE_SYSTEMATIC -- a policy decision motivated by durability diversity (hedging against a correlated failure in any single encoder implementation) and by per-workload fit (each encoding's reconstruction cost and failure-tolerance profile suits a different tenant on that pool). A 250-TB file cannot fit in any single pool. Striping the file across all three pools is forced by capacity arithmetic: 250 > 100. Because bytes in each pool were written under the pool's chosen encoding, the layout for this file MUST name the actual per-mirror ffv2_encoding_type4 so that clients decode each segment correctly: PASSTHROUGH for the Pool A segment, RS_VANDERMONDE for the Pool B segment, MOJETTE_SYSTEMATIC for the Pool C segment. The heterogeneity is not a transition window; it is the permanent consequence of striping across pools whose bytes were written under different operator-chosen encodings. In this steady-state case, no proxy server mediated transition machinery is involved. The client receives a layout enumerating the mirrors at different ffv2m_coding values and routes chunk operations to the appropriate data server per segment (or, if the client cannot speak one of the encodings, requests proxy mediation per the proxy server draft's section "Encoding Translation for Encoding-Ignorant Clients" ()). The layout machinery that supports this case is exactly the per-mirror encoding naming primitive described above; no additional protocol elements are required to express it. The transient case (one file moving between encodings) and the steady-state case (one file permanently striped across heterogeneous pools) share a single wire primitive: an ffv2l_mirrors array that admits mixed ffv2_encoding_type4 values. Removing that primitive would foreclose both cases and would force one of three workarounds: (a) split the 250-TB file into three independently-named files (loses single-namespace semantics), (b) require every pool to implement every encoding (forces a single-vendor or single-implementation procurement story), or (c) require an always-on transcoding proxy in front of every read and write (re-introduces the centralized data-plane that the proxy server role is scoped to avoid in steady state). None of these alternatives address the steady-state case while preserving the transition-window flexibility this document specifies.

FFV2_ENCODING_PASSTHROUGH FFV2_ENCODING_PASSTHROUGH is the on-ramp from the flexible file v1 layout () into the flexible file v2 layout. A PASSTHROUGH mirror points at the file's bytes as they exist on the data server, without the chunk envelope, checksum header, or chunk_guard4 fields that the encoded types use. Client I/O against a PASSTHROUGH mirror uses NFSv3 WRITE / READ () or NFSv4 READ / WRITE () directly -- not CHUNK_WRITE / CHUNK_READ. PASSTHROUGH provides:

Replication of data across N data servers, exactly as flexible file v1 layout does. Clients write to every replica; clients read from any one. N-way redundancy tolerates up to N-1 replica losses.
Zero encoding compute at the client and zero chunk-metadata overhead at the server. The on-disk format is the file itself.
Compatibility with files that already exist outside flexible file v2 layout. A PASSTHROUGH mirror can be created over an existing file without rewriting it.

PASSTHROUGH does NOT provide:

Per-chunk integrity. There is no checksum on the data path. Silent corruption is undetectable without out-of-band tooling (e.g., comparing checksums across replicas).
Chunk-grained repair. The repair unit is the whole file: resilvering picks a trusted replica and replicates it end to end to the affected replica(s).
The concurrent writer disambiguation that chunk_guard4 provides for encoded types.

PASSTHROUGH is RECOMMENDED for the assimilation, migration, and heterogeneous-mirror use cases described in . New deployments that do not need a flexible file v1 layout on-ramp SHOULD use FFV2_ENCODING_REPLICATED for the integrity guarantees described in .

FFV2_ENCODING_REPLICATED FFV2_ENCODING_REPLICATED is the chunked-with-integrity peer of PASSTHROUGH. The chunk produced for each replica is the application data verbatim -- no transform, no parity shards -- but it travels on the wire and is stored on the data server through CHUNK_WRITE / CHUNK_READ and so carries every integrity property the encoded encoding types carry. What FFV2_ENCODING_REPLICATED keeps from the mirror model:

Zero encoding compute at the client. Each replica's chunk is the input bytes; there is no transform to apply on write and nothing to decode on read.
Storage cost of N x payload, where N is the replica count. Mirroring trades storage for redundancy without the reconstruction machinery that erasure coding requires.
Reading any one intact replica is sufficient. If a replica fails to verify (see below), the client tries another.

What FFV2_ENCODING_REPLICATED adds beyond PASSTHROUGH, by virtue of using CHUNK_WRITE and CHUNK_READ:

Per-chunk checksum on write and on read. The CRC is computed by the client over the chunk header and chunk payload with the checksum field itself treated as zero (), sent on the wire with the chunk, recomputed by the data server before storing, and recomputed again from disk by the data server on every CHUNK_READ. Wire-level bit flips are caught before the chunk is stored; on-disk bit rot is caught the next time the chunk is read.
Per-chunk repair granularity. When one replica's CRC fails to verify and another replica's verifies, the repair unit is the chunk, not the file: CHUNK_READ the good replica, CHUNK_WRITE to the bad replica, done. No whole-file resilvering is required.
Per-chunk concurrent writer disambiguation. Mirrored writes carry the same chunk_guard4 () the erasure encoding types do. Two clients racing to write the same offset of the same file fan out to every replica with a guard pair (generation, owning-client short-id) per chunk; the CHUNK_FINALIZE step resolves which writer's chunk wins and the other writer observes a deterministic loss instead of an unresolved split-mirror.

What FFV2_ENCODING_REPLICATED is for: files where the deployment wants integrity and replication without the storage savings or the reconstruction story of erasure coding. Small files that do not exceed a single stripe, files whose access pattern is read-mostly and where the N x storage cost is acceptable, and files where the operator prefers the simplicity of "any one replica is the file" over "k of (k + m) shards reconstruct the file." The coding choice is per-file; a deployment can mix mirrored and erasure-coded files in the same namespace and pick whichever fits each file's profile. What FFV2_ENCODING_REPLICATED is not: a substitute for erasure coding when storage efficiency or multi-replica fault tolerance matters. An N-way mirror tolerates up to N-1 replica losses but costs N x the payload; a (k, m) erasure coding tolerates m losses at (k + m)/k x the payload. Both have per-chunk integrity under this document; the choice is the cost-vs-tolerance one.

FFV2_ENCODING_XOR_PARITY

Overview FFV2_ENCODING_XOR_PARITY is a single-parity, systematic, RAID-5-shape encoding: k data shards accompanied by one parity shard computed as the bytewise XOR of every data shard. Parameters: k in the range 1 to 254, m fixed at 1. Unlike Reed-Solomon and Linux md/raid6, XOR_PARITY requires no finite-field arithmetic. The "encoding" is a plain XOR reduction across k shards, so the implementation footprint is trivial and the compute cost scales at memory bandwidth. This makes XOR_PARITY the simplest candidate encoding: any conformant implementation can support it without a GF(2^8) library.

Encoding Given k data shards, each of shard_len bytes, encoding produces a single parity shard of shard_len bytes: All shards (data and parity) are the same size. No table lookups, no multiplications; a straight bitwise XOR reduction.

Recovery XOR_PARITY tolerates the loss of exactly one shard (any one data shard or the parity shard).

Missing data shard data[i]: reconstruct by XOR-ing all surviving shards (k-1 data shards plus the parity shard): data[i][j] = parity[j] XOR data[0][j] XOR ... (skip i) ... XOR data[k-1][j].
Missing parity shard: recompute from the k intact data shards using the encoding step above.

Reconstruction cost is a single XOR reduction over shard_len bytes; memory bandwidth dominates. No matrix inversion is required. Loss of two or more shards is unrecoverable under XOR_PARITY. Deployments requiring tolerance of two or more concurrent losses SHOULD use FFV2_ENCODING_LINUX_MD_RAID (m=2) or FFV2_ENCODING_RS_VANDERMONDE (configurable m).

Interoperability XOR_PARITY produces a parity shard byte-identical to the P (first) parity row of Reed-Solomon Vandermonde encoding at m=1 and to the P row of Linux md/raid6 at any m >= 1. This follows from using the same primitive coefficients:

RS Vandermonde at m=1 uses parity row [1, 1, ..., 1] in GF(2^8), which reduces to bitwise XOR (see ).
LINUX_MD_RAID's P row is defined as the bitwise XOR of every data shard, identical to XOR_PARITY's parity by construction.

A receiver capable of FFV2_ENCODING_RS_VANDERMONDE at m=1 or FFV2_ENCODING_LINUX_MD_RAID at m >= 1 therefore consumes an XOR_PARITY-encoded chunk without re-encoding, provided the k matches. Conversely, an XOR_PARITY receiver consumes the P row of any GF(2^8) family encoding at m >= 1 unchanged.

XOR_PARITY Interoperability Test Vectors Concrete byte-level test vector with k = 3, m = 1, shard_len = 1: XOR_PARITY test vector: k=3, m=1

data[0]	data[1]	data[2]	parity	Notes
`0x00`	`0x00`	`0x00`	`0x00`	zero input
`0x01`	`0x00`	`0x00`	`0x01`	single non-zero data shard
`0x01`	`0x02`	`0x04`	`0x07`	0x01 XOR 0x02 XOR 0x04 = 0x07
`0x37`	`0x91`	`0xac`	`0x0a`	matches P from
`0xff`	`0xff`	`0x00`	`0x00`	0xff XOR 0xff XOR 0x00 = 0x00

The fourth row is byte-identical to the P entry of the same input in -- the wire-compat property in action.

XOR_PARITY Shard Sizes All XOR_PARITY shards (data and parity) are exactly shard_len bytes. chunk_size equals shard_len for every mirror in the layout. Total storage overhead is 1/k of payload. XOR_PARITY shard sizes for common configurations

Configuration	File Size	Shard Size	Total Storage	Overhead
3+1	3 KB	1 KB	4 KB	33%
3+1	1 MB	~342 KB	~1.33 MB	33%
7+1	7 KB	1 KB	8 KB	14%
7+1	1 MB	~147 KB	~1.14 MB	14%

FFV2_ENCODING_LINUX_MD_RAID

Overview FFV2_ENCODING_LINUX_MD_RAID is the Linux kernel md/raid6 P + Q double-parity encoding, evaluated in GF(2^8) with primitive polynomial x^8 + x^4 + x^3 + x^2 + 1 (encoded as 0x1d when the implicit degree-8 term is dropped, 0x11d when it is kept). Parameters: k in the range 2 to 253, m fixed at 2. The encoding produces two parity shards. P is the bitwise XOR of every data shard (identical to FFV2_ENCODING_XOR_PARITY's parity, see ). Q is a weighted sum sum(g^i * data_i) in GF(2^8) with g = 2. The construction is bit-for-bit compatible with Linux kernel lib/raid6 and with FFV2_ENCODING_RS_VANDERMONDE at m<=2 (see ). The k=1 case (a single data shard with P and Q) is degenerate and MUST NOT be used with FFV2_ENCODING_LINUX_MD_RAID. Callers who need triple-mirror semantics MUST use FFV2_ENCODING_REPLICATED with N=3 instead.

LINUX_MD_RAID Galois Field Arithmetic All LINUX_MD_RAID operations are performed over GF(2^8), the Galois field with 256 elements. Each element is represented as a byte.

Irreducible Polynomial:: The field is constructed using x^8 + x^4 + x^3 + x^2 + 1, encoded as 0x1d when the implicit x^8 term is dropped (Linux md convention), or 0x11d when it is kept (RS Vandermonde convention in ). Both encodings refer to the same field; only the notation differs.
Primitive Element:: g = 2. Powers of g cycle through GF(2^8) \ {0} with period 255.
Addition:: Bitwise XOR.
Multiplication:: Via log/antilog tables in the reference implementation, or SIMD vector multiply in modern kernels. See the Linux kernel lib/raid6/{int.uc, sse2.c, altivec.uc, ...} sources for the canonical implementation.

LINUX_MD_RAID Encoding Given k data shards, each of shard_len bytes, encoding produces two parity shards P and Q, each shard_len bytes: where multiplication is in GF(2^8) and g = 2. All shards (data, P, and Q) are the same size. The P row is identical to FFV2_ENCODING_XOR_PARITY's parity by construction; the two encodings share the P computation. The Q row is the m=2 P + Q construction from , and produces byte-identical output to FFV2_ENCODING_RS_VANDERMONDE at m=2.

LINUX_MD_RAID Recovery LINUX_MD_RAID tolerates up to two concurrent shard losses (any two of the k + 2 shards).

Single-shard failure:: If only P or only Q is missing, recompute from the k intact data shards using the encoding step above.
: If exactly one data shard data[i] is missing but both P and Q are intact, prefer the P-only recovery (cheaper): XOR all surviving shards (k-1 data plus P) to recover data[i].
Dual-shard failure:: If two data shards are missing, use both P and Q to solve a 2x2 linear system in GF(2^8) per byte position. See the Linux kernel lib/raid6/recov.c for the canonical solver.
: If one data shard and one parity shard are missing, first recover the data shard using the intact parity row (as in the single-shard case), then recompute the missing parity from the reconstructed data.

Loss of three or more shards is unrecoverable under LINUX_MD_RAID. Deployments requiring m >= 3 fault tolerance MUST use FFV2_ENCODING_RS_VANDERMONDE.

LINUX_MD_RAID Interoperability FFV2_ENCODING_LINUX_MD_RAID is wire-compatible with FFV2_ENCODING_RS_VANDERMONDE at m <= 2 by construction: both encoders emit byte-identical P and Q for the same (k, data) input at m <= 2.

At m = 1, LINUX_MD_RAID's P row equals RS_VANDERMONDE's m=1 parity and equals XOR_PARITY's parity (see ).
At m = 2, LINUX_MD_RAID's P + Q equals RS_VANDERMONDE's m=2 P + Q rows byte-for-byte.

RS_VANDERMONDE joined the m <= 2 wire-compat set as a wire format revision in this document: its m <= 2 parity rows are the hand-crafted P/Q construction rather than the normalized-Vandermonde bottom rows RS uses at m >= 3. See for the RS construction and its interoperability parameters. A client implementing either LINUX_MD_RAID or RS_VANDERMONDE at m <= 2 can consume the other without re-encoding, provided the (k, m) geometry matches.

LINUX_MD_RAID Interoperability Test Vectors Concrete byte-level test vector with k = 3, m = 2, shard_len = 1: LINUX_MD_RAID test vector: k=3, m=2

data[0]	data[1]	data[2]	P	Q	Notes
`0x00`	`0x00`	`0x00`	`0x00`	`0x00`	zero input
`0x01`	`0x02`	`0x03`	`0x00`	`0x09`	1 XOR (22) XOR (43) = 1 XOR 4 XOR 12 = 9
`0x80`	`0x00`	`0x00`	`0x80`	`0x80`	Q = 1 * 0x80 = 0x80
`0x00`	`0x80`	`0x00`	`0x80`	`0x1d`	Q = 2 * 0x80 = 0x100 -> reduce by 0x11d -> 0x1d
`0x00`	`0x00`	`0x80`	`0x80`	`0x3a`	Q = 4 * 0x80 = 2 * 0x1d = 0x3a
`0x37`	`0x91`	`0xac`	`0x0a`	`0x82`	general non-degenerate case

Every row above is byte-identical to the k=3, m=2 test vector for FFV2_ENCODING_RS_VANDERMONDE () at the same input. This is the wire-compat property in action: an implementation whose LINUX_MD_RAID output matches these bytes is producing the same wire encoding as RS_VANDERMONDE at m=2.

LINUX_MD_RAID Shard Sizes All LINUX_MD_RAID shards (data, P, Q) are exactly shard_len bytes. chunk_size equals shard_len for every mirror in the layout. Total storage overhead is 2/k of payload. LINUX_MD_RAID shard sizes for common configurations

Configuration	File Size	Shard Size	Total Storage	Overhead
4+2	4 KB	1 KB	6 KB	50%
4+2	1 MB	256 KB	1.5 MB	50%
8+2	4 KB	512 B	5 KB	25%
8+2	1 MB	128 KB	1.25 MB	25%

Reed-Solomon Vandermonde Encoding (FFV2_ENCODING_RS_VANDERMONDE)

Overview Reed-Solomon (RS) codes are Maximum Distance Separable codes: for a (k + m, k) code, any k of the k + m encoded shards suffice to recover the original data. The code tolerates the simultaneous loss of up to m shards. is a tutorial treatment of RS coding in RAID-like systems and is the recommended background reading for implementers unfamiliar with the construction used here.

Galois Field Arithmetic All RS operations are performed over GF(2^8), the Galois field with 256 elements. Each element is represented as a byte.

Irreducible Polynomial:: The field is constructed using the irreducible polynomial x^8 + x^4 + x^3 + x^2 + 1 (0x11d in hexadecimal). The primitive element (generator) is g = 2, which has multiplicative order 255.
Addition:: Addition in GF(2^8) is bitwise XOR.
Multiplication:: Multiplication uses log/antilog tables. For non-zero elements a and b: a * b = exp(log(a) + log(b)), where the exp table is doubled to 512 entries to avoid modular reduction on the index sum.

These are the classical constructions from Berlekamp (1968) and Peterson & Weldon (1972). The log/antilog table approach for GF(2^8) multiplication predates all known patents on SIMD-accelerated GF arithmetic. Implementors considering SIMD acceleration of GF(2^8) operations should be aware of US Patent 8,683,296 (StreamScale), which covers certain SIMD-based GF multiplication techniques.

Encoding Matrix The systematic encoding matrix E has (k + m) rows and k columns. The top k rows are always the k x k identity, so data shards pass through unchanged. The bottom m parity rows are chosen as follows.

At m = 1: single parity row The single parity row is [1, 1, ..., 1]: Encoded parity is the bitwise XOR of every data shard. This matches the P row of Linux md's RAID6 construction and the sole parity row of FFV2_ENCODING_XOR_PARITY byte-for-byte; a receiver that speaks either of those consumes RS_VANDERMONDE at m=1 without re-encoding.

At m = 2: P + Q parity rows The two parity rows are: where g = 2 is the primitive element of GF(2^8) with polynomial 0x11d. These are exactly the coefficients Linux md RAID6 uses for its P and Q shards. A receiver that speaks FFV2_ENCODING_LINUX_MD_RAID at m <= 2 also consumes RS_VANDERMONDE at m <= 2 byte-for-byte (and vice versa).

At m >= 3: normalized Vandermonde bottom rows The parity rows are the bottom m rows of a normalized Vandermonde encoding matrix, constructed as follows.

Assign each of the k + m shards a distinct non-zero evaluation point in GF(2^8): shard i (for i = 0, 1, ..., k + m - 1) is assigned the point alpha_i = i + 1. This gives evaluation points 1, 2, ..., k + m, all non-zero and distinct. The value k + m MUST NOT exceed 255 so that all points fit in GF(2^8) \ {0}.
Construct a (k + m) x k Vandermonde matrix V where the row for shard i is the geometric progression of alpha_i: Row i is (1, alpha_i, alpha_i^2, ..., alpha_i^(k-1)). The exponent zero is defined as x^0 = 1 for all x in GF(2^8), including x = 0 (this is the standard combinatorial convention; here alpha_i is never zero by step 1's construction, but the convention makes the V[0][0] cell unambiguous). Any k distinct rows form a k x k Vandermonde matrix on k distinct non-zero evaluation points, which is invertible over GF(2^8); this is the property that gives the code its Maximum Distance Separable (any k of k + m shards recover the data) guarantee. The minimum useful geometry is k >= 1 and m >= 1 (k = 0 gives no data and m = 0 gives no redundancy); the maximum is bounded by k + m <= 255 as stated in step 1.
Extract the top k x k sub-matrix T from V. T is the Vandermonde on evaluation points alpha_0 = 1, alpha_1 = 2, ..., alpha_(k-1) = k.
Compute T_inv = T^(-1) using Gaussian elimination in GF(2^8).
Multiply: E = V * T_inv. The result has an identity block on top (rows 0 through k-1) and the parity generation matrix P on the bottom (rows k through k + m - 1).

The identity block makes the code systematic: data shards pass through unchanged, and only the parity sub-matrix P is needed during encoding. These bottom rows do not match any external encoding at m >= 3; this encoding stands on its own at m >= 3.

Encoding Given k data shards, each of shard_len bytes, encoding produces m parity shards, each also shard_len bytes: where the sum and product are in GF(2^8). All shards (data and parity) are the same size.

Decoding When one or more shards are lost (up to m), reconstruction proceeds by matrix inversion:

Select k available shards (from the k + m total).
Form a k x k sub-matrix S of the encoding matrix E by selecting the rows corresponding to the available shards.
Compute S_inv = S^(-1) using Gaussian elimination in GF(2^8).
Multiply S_inv by the vector of available shard data at each byte position to recover the original k data shards.
If any parity shards are also missing, regenerate them by re-encoding from the recovered data shards.

The reconstruction cost is dominated by the matrix inversion, which is O(k^3) in GF(2^8) multiplications.

RS Interoperability Requirements For two implementations of FFV2_ENCODING_RS_VANDERMONDE to interoperate, they MUST agree on all of the following parameters. Any deviation produces a different encoding matrix and renders data unrecoverable by a different implementation.

Irreducible polynomial:: x^8 + x^4 + x^3 + x^2 + 1 (0x11d).
Primitive element:: g = 2.
Evaluation points:: shard i (i = 0, 1, ..., k+m-1) uses alpha_i = i + 1 in GF(2^8) (values 1 through k+m, all non-zero and distinct).
Vandermonde entries:: V[i][j] = alpha_i^j = (i+1)^j in GF(2^8) for i = 0..k+m-1, j = 0..k-1.
Matrix normalization:: E = V * T^(-1) where T is the top k x k sub-matrix (rows for shards 0..k-1).
Parameter bound:: k + m MUST NOT exceed 255.

These parameters fully determine the encoding matrix for any (k, m) configuration in the permitted range.

RS Interoperability Test Vectors The following worked examples pin the encoding matrix and end-to-end encodings for two representative geometries: k=2 m=1 (exercises the m=1 all-ones parity row) and k=3 m=2 (exercises the m=2 P + Q parity rows). An implementation whose encoded output matches these tables is using the same GF(2^8) representation and the same parity-row construction as required by the interoperability parameters above.

k=2, m=1 Encoding matrix E: The parity shard is the bitwise XOR of both data shards: parity[j] = data[0][j] XOR data[1][j]. Concrete byte-level test vector with shard_len = 1: RS Vandermonde test vector: k=2, m=1

data[0]	data[1]	parity	Notes
`0x00`	`0x00`	`0x00`	zero input
`0x01`	`0x00`	`0x01`	0x01 XOR 0x00 = 0x01
`0x00`	`0x01`	`0x01`	0x00 XOR 0x01 = 0x01
`0x01`	`0x01`	`0x00`	0x01 XOR 0x01 = 0x00
`0x80`	`0x80`	`0x00`	0x80 XOR 0x80 = 0x00

k=3, m=2 Encoding matrix E: The two parity shards are computed byte-wise as: where the multiplication is in GF(2^8) with polynomial 0x11d. Concrete byte-level test vector with shard_len = 1: RS Vandermonde test vector: k=3, m=2

data[0]	data[1]	data[2]	P	Q	Notes
`0x00`	`0x00`	`0x00`	`0x00`	`0x00`	zero input
`0x01`	`0x02`	`0x03`	`0x00`	`0x09`	1 XOR (22) XOR (43) = 1 XOR 4 XOR 12 = 9
`0x80`	`0x00`	`0x00`	`0x80`	`0x80`	Q = 1 * 0x80 = 0x80
`0x00`	`0x80`	`0x00`	`0x80`	`0x1d`	Q = 2 * 0x80 = 0x100 -> reduce by 0x11d -> 0x1d
`0x00`	`0x00`	`0x80`	`0x80`	`0x3a`	Q = 4 * 0x80 = (2 * 0x1d) = 0x3a
`0x37`	`0x91`	`0xac`	`0x0a`	`0x82`	general non-degenerate case

Implementations that produce different values for any row of either table disagree with this specification and will not interoperate. m >= 3 test vectors are not included here because the normalized-Vandermonde parity rows are not easily hand-computed; implementations that need m >= 3 verification SHOULD cross-check against a reference implementation.

RS Shard Sizes All RS shards (data and parity) are exactly shard_len bytes. This simplifies the CHUNK operation protocol: chunk_size is exactly the shard size for all mirrors. RS shard sizes for common configurations

Configuration	File Size	Shard Size	Total Storage	Overhead
4+2	4 KB	1 KB	6 KB	50%
4+2	1 MB	256 KB	1.5 MB	50%
8+2	4 KB	512 B	5 KB	25%
8+2	1 MB	128 KB	1.25 MB	25%

Mojette Transform Encoding (FFV2_ENCODING_MOJETTE_SYSTEMATIC, FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC)

Overview The Mojette Transform is an erasure coding technique based on discrete geometry rather than algebraic field operations. It computes 1D projections of a 2D grid along selected directions. Given enough projections, the original grid can be reconstructed exactly. The transform operates on fixed-width words combined with bitwise XOR -- the additive group of (GF(2))^(W*8) where W is the element width in bytes. Encoders and decoders MUST use XOR; modular integer addition is not equivalent and is not interoperable. XOR has no carry chain, is its own inverse (so the residual subtraction in reconstruction is identical to the forward accumulation), and scales straightforwardly to wider SIMD lanes (NEON, SSE, AVX, AVX-512) without requiring multiplicative Galois field operations. For interoperability, this specification pins the element width to W = 8 bytes (64 bits) for both FFV2_ENCODING_MOJETTE_SYSTEMATIC and FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC in this revision. All implementations MUST use W = 8; a future revision (or a distinct registered encoding type) may lift the fixed width, at which point it becomes a wire-visible parameter. Fixing W in this revision prevents an interoperability hazard: the same shard input XOR'd at W = 4 versus W = 8 produces different bin values that appear "close but wrong" to a mismatched decoder.

Grid Structure Data is arranged as a P x Q grid of unsigned integer elements, where P is the number of columns and Q is the number of rows. For k data shards of S bytes each with W-byte elements:

Directions A direction is a pair of coprime integers (p_i, q_i). This specification pins all directions to q_i = 1 for both Mojette encoding types (systematic and non-systematic) in this revision. Non-unity q_i values may be introduced by a future distinct registered encoding type. For n = k + m total shards (Mojette non-systematic) or m parity shards (Mojette systematic), the direction set is determined by the following mandatory rule on the shard count N (N = n for non-systematic, N = m for systematic).

For N even (N = 2t):: directions = { (p, 1) : p in {-t, -t+1, ..., -1, 1, 2, ..., t} } -- symmetric around zero, giving |directions| = 2t = N.
For N odd (N = 2t + 1):: directions = { (p, 1) : p in {-t, -t+1, ..., -1, 1, 2, ..., t, t+1} } -- asymmetric by including one additional positive magnitude so that |directions| = 2t + 1 = N.

Direction slots are then sorted by p_i ascending (most-negative p first, most-positive p last) to give the canonical direction order. The direction in slot i is used to compute the projection for shard slot (k + i) in systematic form, or for shard slot i in non-systematic form. Examples:

N = 4 (even): p = {-2, -1, 1, 2}
N = 6 (even): p = {-3, -2, -1, 1, 2, 3}
N = 3 (odd): p = {-1, 1, 2}
N = 5 (odd): p = {-2, -1, 1, 2, 3}

Two implementations that follow this algorithm on the same N generate identical direction sets in identical slot order and therefore identical shard layouts. Implementations MUST NOT diverge from this algorithm (e.g., by using a different tie-breaking rule for odd N) without registering a distinct encoding type.

Forward Transform (Encoding) For each direction (p_i, q_i), the forward transform computes a 1D projection. Each bin XORs the grid elements that lie on a discrete line through the grid. This specification adopts the bin convention of : a grid cell at (row, col) maps to bin where off is chosen so that the smallest reachable bin index is 0 (off = min over all (row, col) in [0, Q) x [0, P) of row * p + col * q). All implementations MUST use this convention -- the alternative "transposed" convention b = col * p - row * q + off produces a different bin ordering and is not interoperable. The full forward transform along direction (p, q) is then: The number of bins B in a projection is: For q = 1, this simplifies to: The byte size of the projection is B * W.

Katz Reconstruction Criterion Reconstruction from a set of n projections is possible if and only if the Katz criterion holds over those n projections: = Q OR SUM(i=1..n) |p_i| >= P ]]> With q_i = 1 pinned for every direction (see the Directions subsection above), the q-sum simplifies to n >= Q. For the non-systematic form, n = k + m and every direction counts toward the criterion. The criterion holds for the initial encoding (design-time check on the direction set) and must continue to hold after any losses; the surviving projections must satisfy Katz over the same grid dimensions (P, Q). For the systematic form, the raw data rows act as "projections at direction (p=0, q=1)": row r is Grid[r] and contributes q = 1 toward the q-sum of the Katz criterion. For an arbitrary loss pattern with r data-row losses and s parity-projection losses (with r + s <= m), the surviving set comprises k - r data rows (each q_i = 1) and m - s parity projections. Decoding proceeds by:

Subtracting the contributions of the k - r surviving data rows from the m - s surviving parity projections (the "residual").
Applying the corner-peeling algorithm over the residual to recover the r missing rows.

Step 2 succeeds if and only if the Katz criterion holds over the m - s residual projections against the r x P unknown sub-grid. Substituting: the criterion reduces to m - s >= r (q-sum, since q_i = 1 for every parity direction and the unknown-grid Q is r) OR the analogous p-sum condition over the residual. Because the mandatory direction algorithm above generates m projections with distinct nonzero p values, the p-sum condition also holds for any loss pattern with r + s <= m; the systematic form therefore achieves Maximum-Distance-Separable-like recovery up to m combined data-row and parity-projection losses.

Inverse Transform (Decoding) The choice of inverse algorithm is purely an implementer concern: all correct inverses produce byte-identical plaintext from the same shards and bin layout, so the choice has no on-the-wire impact. Two well-known algorithms apply. The corner-peeling algorithm:

Count how many unknown elements contribute to each bin.
Find any bin with exactly one contributor (singleton).
Recover the element, XOR it back through all projections.
Repeat until all elements are recovered.

Corner peeling runs in O(n * P * Q) and is the simplest correct inverse. Implementations MAY instead use the geometry-driven inverse of , which precomputes a recurrence over the sorted projection slopes and walks each line once: it eliminates the inner singleton search and runs substantially faster on the parameter ranges typical of flexible file v2 layout deployments (high redundancy, wide stripes), with no change to the shards or to the reconstructed plaintext.

Systematic Mojette In the systematic form (FFV2_ENCODING_MOJETTE_SYSTEMATIC), the first k shards are the original data rows and the remaining m shards are projections. Healthy reads require no decoding. Reconstruction of missing data rows proceeds via the corner-peeling algorithm of :

Load available parity projections.
Subtract contributions of present data rows (residual).
Corner-peel the residual to recover missing rows.

Reconstruction cost is O(m * k) -- a fundamental advantage over RS at wide geometries (k >= 8).

Non-Systematic Mojette In the non-systematic form (FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC), all k + m shards are projections. Every read requires the full inverse transform. This provides constant performance regardless of failure count, but at higher baseline read cost than systematic.

Mojette Shard Sizes and Layout

Slot-to-direction mapping:

The canonical shard layout for Mojette is:

Systematic (FFV2_ENCODING_MOJETTE_SYSTEMATIC):: shard slots 0..k-1 carry the k data rows (row r in slot r); shard slots k..k+m-1 carry the m parity projections in the canonical direction order defined in the Directions subsection above (direction slot i occupies shard slot k + i).
Non-systematic (FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC):: shard slots 0..k+m-1 carry the n = k + m parity projections in the canonical direction order (direction slot i occupies shard slot i).

Bin ordering within a projection:

Within a projection shard the bins are serialized in ascending bin-index order (bin 0 first, bin B-1 last), with no gap or header. Each bin value is W = 8 bytes wide; the W-byte element is serialized in big-endian byte order (the same order the data-row shards present their W-byte grid elements in).

Projection sizes:

Unlike Reed-Solomon, Mojette parity shard sizes vary by direction:

Mojette projection sizes for 4+2, 4KB shards, W=8

Direction (p, q)	Bins (B) for P=512, Q=4	Size (bytes, W=8)
(-3, 1)	521	4168
(-2, 1)	518	4144
(-1, 1)	515	4120
(1, 1)	515	4120
(2, 1)	518	4144
(3, 1)	521	4168

Chunk sizing for variable-length projections:

When a projection shard is written via CHUNK_WRITE / CHUNK_FINALIZE / CHUNK_COMMIT, the shard is divided into chunks by the following mapping. Let shard_bytes = B * W be the projection shard's total byte size (where B is the number of bins per the B formula above ( uses it) applied to the shard's direction (p, q) and the grid dimensions (P, Q)):

num_chunks = ceil(shard_bytes / chunk_size)
Chunk j (for j = 0..num_chunks-1) covers the shard byte range [j * chunk_size, min((j+1) * chunk_size, shard_bytes)).
The final chunk (chunk num_chunks - 1) MAY be shorter than chunk_size if shard_bytes is not a multiple of chunk_size; all other chunks are exactly chunk_size bytes.

The chunk_size value is a per-mirror parameter and does not vary across the parity projections of a single file, even though the shard sizes vary. For a file with parity projections of sizes S_i = B_i * W, the number of chunks per shard is ceil(S_i / chunk_size) per shard; a reader that requests chunk offset >= S_i on shard i receives NFS4ERR_PAYLOAD_LOST (per ) with a short read reporting the shard's true byte length.

Comparison of Encoding Types Comparison of erasure encoding types

Property	Reed-Solomon	Mojette Systematic	Mojette Non-Systematic
Maximum Distance Separable guarantee	Yes	Yes (Katz)	Yes (Katz)
Shard sizes	Uniform	Variable	Variable
Reconstruction cost	O(k^3) shard ops (matrix inversion)	O(mkP*Q) grid ops (peeling)	O(mkP*Q) grid ops (peeling)
Healthy read cost	Zero	Zero	Full decode
GF operations	Yes (GF(2^8))	No	No

Reed-Solomon uses uniform shard sizes and GF(2^8) operations. Mojette systematic provides zero-cost healthy reads with variable parity shard sizes; reconstruction cost scales as O(m * k) rather than O(k^3). Mojette non-systematic encodes all k + m shards as projections, providing constant decode cost regardless of failure count at a higher baseline read cost than systematic. The choice among these is a deployment decision driven by workload characteristics and operational priorities.

Handling write holes A write hole occurs when a client begins writing a stripe but does not successfully write all k + m shards before a failure. Some data servers will hold new data while others still hold old data, producing an non-atomic payload. The CHUNK_WRITE / CHUNK_ROLLBACK mechanism addresses this. When a client issues CHUNK_WRITE, the data server retains a copy of the previous shard and places the new data in the PENDING state. If any shard write fails, the client issues CHUNK_ROLLBACK to each data server that received a CHUNK_WRITE, restoring the previous content. The payload remains atomic from the reader's perspective throughout, because PENDING blocks carry the new chunk_guard4 value and CHUNK_READ returns the last COMMITTED or FINALIZED block when a PENDING block exists. A single-shard CHUNK_WRITE failure MAY alternatively be handled by reporting the failure to the metadata server via LAYOUTERROR and letting the metadata server initiate the repair flow at . The metadata server adds a REPAIR-flagged replacement data server to the layout (from its out-of-band pool) and drives reconstruction of the missing shard. In the multiple writer model, a write hole can also arise when two clients are racing. The chunk_guard4 value on each chunk identifies which transaction wrote it. A reader that finds chunks with different guard values detects the non-atomicity and either retries (if a concurrent write is still in progress) or reports NFS4ERR_PAYLOAD_NOT_ATOMIC to the metadata server to trigger repair. When CHUNK_ROLLBACK and repair are both unavailable, and the payload cannot be reconstructed because too many shards have been lost (for example, a catastrophic multi-data server failure with no reachable replacement data servers), the repair flow ultimately terminates with NFS4ERR_PAYLOAD_LOST; see .

System Model and Correctness The design decisions in this document -- centralized coordination through the metadata server, CAS semantics via chunk_guard4, pessimistic lock escrow during repair, and erasure-coded reads from any sufficient subset -- depart visibly from a classical distributed-consensus protocol such as Paxos or Raft. This section states the system model those decisions rest on, the consistency and progress guarantees the protocol provides under that model, and how the protocol relates to (and when it relies on) classical consensus. It is intended as the correctness framing for implementers and reviewers; the normative wire behavior is defined in the preceding sections.

Wire Semantics vs Implementation The protocol defines wire semantics, not data-server implementation. The operations introduced in (CHUNK_WRITE, CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK, CHUNK_LOCK / CHUNK_UNLOCK, CHUNK_READ, CHUNK_REPAIRED, CHUNK_ERROR, CHUNK_HEADER_READ, CHUNK_WRITE_REPAIR) together with the per-chunk state machine () and the chunk_guard4 CAS () are everything a peer observes. The data server's internal representation of persistent state is not exposed on the wire, and two data-server implementations that satisfy the same wire semantics MAY differ arbitrarily in their internal structure. In particular, the protocol does NOT exchange:

which on-disk layout (log-structured, append-only, in-place-overwrite, external object store, key-value store, or any other) a data server uses to persist chunks;
whether a data server holds PENDING and FINALIZED chunks in a single blob or in distinct regions;
how a data server represents the CHUNK_LOCK table, the guard epoch, or the escrow owner;
whether a data server's chunk retention beyond COMMIT is implemented via shadow blocks, journals, reference counts, or copy-on-write.

This decoupling is deliberate. It lets the protocol accommodate future smart-data server designs -- including designs that integrate more closely with storage back-ends that already provide atomic replace, multi-version concurrency, or internal erasure coding -- without protocol revisions, provided the wire semantics are preserved. Conversely, a data server implementer is free to pick the representation that best fits the underlying storage stack without fear that some less common implementation choice is disallowed. The counterpart of this rule is that the wire is the entire contract. Any behavior a client relies on MUST be observable via the operations listed above; any behavior that is not observable (cache state, background scrubbing cadence, internal retry ordering, on-disk layout) is implementation detail and MUST NOT be depended upon.

Chunks Are Not Blocks The chunk is a protocol-level primitive distinct from a block. Throughout this document, "block" refers to a byte range in the file's address space (the application's view); "chunk" refers to the addressable unit carried by the CHUNK operations, which has an envelope that blocks do not. A chunk carries five properties that a block does not:

Atomicity:: the chunk_guard4 compare-and-swap guard () sequences concurrent writers and rejects torn-write attempts. Block I/O has no comparable primitive; concurrent block writes either serialize at the storage layer or interleave unpredictably.
Integrity:: the checksum in each chunk header is computed over the header and payload and verified end-to-end on the read path (). Block I/O carries no integrity tag; data-corruption detection is delegated to the underlying storage medium or is absent.
Provenance:: the chunk_owner4 () records which transaction produced the chunk. Block I/O carries no per-write provenance; a block's bytes have no protocol-visible producer.
Lifecycle state:: a chunk progresses through PENDING -> FINALIZED -> COMMITTED via CHUNK_FINALIZE / CHUNK_COMMIT (). Block I/O has no lifecycle states; a block is either present or absent.
Lock continuity across revocation:: the chunk's lock () is transferred to the metadata server in escrow when a holder's stateid is revoked, and adopted by a repair actor via CHUNK_LOCK_FLAGS_ADOPT. Block I/O has no per-block locking and no continuity mechanism; client failure leaves any external lock indeterminate.

Each of these properties is load-bearing for some part of the flexible file v2 layout's consistency story: the chunk_guard4 CAS underlies multi-writer correctness; the checksum underlies end-to-end integrity; lock escrow underlies repair coordination across stateid revocation; the state machine underlies the PENDING / FINALIZED / COMMITTED distinction that enables rollback and repair. Removing any one of them would change what the protocol can guarantee. A protocol that exchanges file data as byte ranges with no envelope -- whether described as "block I/O" or as "generic data movement" -- is not interoperable with this specification's CHUNK operations. The CHUNK operations are not a byte-range I/O protocol with optional integrity bolted on; they are a chunk protocol in which the envelope is the primitive.

Actors and Roles Three actors participate on behalf of any given file:

pNFS client:: Issues CHUNK operations to data servers over the data path; issues LAYOUTGET, LAYOUTRETURN, LAYOUTERROR, and SEQUENCE to the metadata server on the control path. Authenticates to the metadata server via AUTH_SYS, RPCSEC_GSS, or TLS. MAY be selected as a repair actor via CB_CHUNK_REPAIR.
Metadata server:: Is the sole coordinator for the file. Grants, renews, and revokes layouts; issues TRUST_STATEID / REVOKE_STATEID / BULK_REVOKE_STATEID to each tight-coupled data server; selects the repair actor under the rules in ; owns the reserved CHUNK_GUARD_CLIENT_ID_MDS escrow identity for in-flight repair.
Data server:: Persists chunks and enforces the per-file trust table, the per-chunk guard CAS (chunk_guard4), the per-chunk lock state (including the metadata-server escrow owner), and the chunk state machine (EMPTY / PENDING / FINALIZED / COMMITTED). Has no coordinator role. Has no knowledge of the erasure encoding type in use for any file: the erasure transform is performed entirely at the client, and the data server stores the resulting chunks without interpreting their contents.

An entity MAY simultaneously hold more than one of these roles with respect to a given data server, with each role bound to a distinct session. A metadata server that opens a control session to a data server (presenting EXCHGID4_FLAG_USE_PNFS_MDS at EXCHANGE_ID; see ) issues TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID on that session; on a separate client-side session (presenting EXCHGID4_FLAG_USE_NON_PNFS), the same metadata server MAY also issue CHUNK operations as a data-path client. A data server MUST NOT assume that the metadata server is not also one of its clients; it distinguishes metadata-server-only operations from client-side operations by the EXCHANGE_ID flags of the session that carries the operation, not by the requester's IP address or principal. A data server MAY likewise act as a client of another data server -- for example, when selected as the repair actor by an metadata-server-directed CB_CHUNK_REPAIR. Independent of the actor role, any entity may operate as encoding-aware (issuing CHUNK operations directly against data servers) or encoding-unaware (operating through the proxy server mediated READ / WRITE path described in the proxy server draft). Proxy-server registration carries the encoding capability explicitly; direct pNFS clients reveal their encoding posture implicitly through the operations they issue. The protocol does NOT mandate how a data server implements the chunk state machine or stores PENDING chunks. An implementation MAY use per-client staging files, a single append-only instance file with an index, a separate metadata-header file paired with a blocks file, a log-structured store, or any other representation that preserves the normative semantics (the EMPTY / PENDING / FINALIZED / COMMITTED transitions, the chunk_guard4 CAS, lock continuity across revocation, and the integrity checks). The choice is a data-server implementation concern and is transparent to clients and the metadata server.

Failure Model The protocol assumes:

Crash-stop:: Clients, metadata servers, and data servers fail by stopping. A restarted component rejoins the protocol with a fresh epoch and participates in the grace / reclaim path already defined in . Correct components do not exhibit arbitrary (Byzantine) behavior.
Fail-silent data servers:: Data servers report honestly about the state of the data they hold. The protocol detects on-disk bit rot via checksum (see ) but does not defend against a data server that deliberately lies about whether a chunk is COMMITTED or what its contents are. Byzantine data servers are explicitly outside the trust model; see .
Authenticated writers and their own data:: An authenticated client may write arbitrary (even semantically-invalid) bytes into chunks it owns. The checksum check detects transport corruption, not adversarial content. This matches the existing NFSv4 authorization model: once you have write access, you may write anything.
Network partitions:: The protocol is partition-tolerant at the cost of availability during the partition window. A client partitioned from a data server recovers via LAYOUTERROR and may be issued a new layout (with a REPAIR-flagged replacement data server added by the metadata server; see ). A metadata server partitioned from a data server eventually renews trust entries on reconnection; in the interim, the data server returns NFS4ERR_DELAY for affected stateids (see ). Message loss is bounded by RPC retransmit; eventual delivery is assumed once the partition heals. Split-brain scenarios (in which a partitioned minority of the data servers in a mirror set attempts to make progress independently of the majority) cannot drive non-atomic writes to COMMITTED state. The chunk_guard4 CAS on each write requires the guard value from a successor chunk to strictly advance the guard value of its predecessor; on partition heal, any writes attempted on the minority side are detected by the majority because their guard values do not satisfy the CAS precondition, and those writes are discarded. When reconciliation is impossible -- for example, the erasure coding has lost too many shards across both sides of the partition to reconstruct any single atomic generation -- the repair flow terminates with NFS4ERR_PAYLOAD_LOST (see ), which is terminal for the affected ranges.
Lease bound:: All state held by a data server on behalf of a metadata server is bounded by the TRUST_STATEID expiry (see ). An orphaned entry will eventually expire even if the metadata server never returns.

Escrow Model Chunk locks () provide the exclusion primitive that the chunk_guard4 CAS () cannot: a client that needs to hold a chunk against every other writer for the duration of a multi-step operation (repair, migration, delta-write coordination) acquires a CHUNK_LOCK. Once acquired, the lock is held on the data server's state, not on the metadata server. That creates two survivability problems the wire protocol has to answer:

Client failure while holding locks:: a client whose stateid is revoked (lease expiry, CB_LAYOUTRECALL timeout, explicit REVOKE_STATEID) is by definition no longer authorized to speak to the data server. If its outstanding chunk locks were simply dropped, the locked chunks would become writable by any other client immediately -- mid-way through the operation the lost client was performing. That would expose the chunk to concurrent writers before the operation completes, defeating the point of taking the lock.
Metadata-server instance change:: in a highly-available deployment the metadata-server role can move from one physical host to another (restart, planned failover, unplanned failover). If the departing incarnation had installed placeholder locks on the data server (see below), the arriving incarnation needs to reclaim them safely -- reclaim by identity, not by client-connection state, since the connection is gone -- and must simultaneously fence the departing incarnation so that a stale prior host resuming operation cannot double-manage the same escrows.

Both problems resolve to the same primitive: chunk locks CAN be held by an owner other than the client that acquired them, and OWNERSHIP CAN TRANSFER without releasing the lock. The escrow mechanism has three components:

Placeholder ownership:: when a client's stateid is revoked while it holds chunk locks, the data server MUST NOT drop the locks. Instead the locks transfer to the metadata server as a placeholder owner, identified on the wire by the reserved value CHUNK_GUARD_CLIENT_ID_MDS in cg_client_id (). The chunk stays locked -- no concurrent writer can win a CAS against the placeholder -- until a repair actor adopts the lock explicitly via CHUNK_LOCK_FLAGS_ADOPT () driven by CB_CHUNK_REPAIR. This closes the lock-continuity gap across client failure.
Escrow identity:: every placeholder-owned lock the metadata server installs on the data server carries an escrow_id4 (), a 128-bit opaque identifier the metadata server chooses. escrow_id4 is what a repair actor names when adopting a specific escrow, what CHUNK_ESCROW_ENUMERATE returns to inventory outstanding escrows, and what CHUNK_ESCROW_RELEASE names when the metadata server retires an escrow explicitly. Without a stable identity, cross-restart adoption and reclamation could not name what they were operating on.
Incarnation handoff:: when the metadata-server role moves to a new host (restart, HA failover), the new incarnation reclaims escrows the old incarnation installed via CHUNK_ESCROW_TAKEOVER. The TAKEOVER carries an incarnation-lease proof () -- cryptographic evidence that the caller is now the authoritative metadata-server incarnation -- which the data server verifies. On acceptance, subsequent escrow operations from the prior incarnation are rejected with NFS4ERR_STALE_MDS_EPOCH (), fencing the departed instance. Only one metadata-server incarnation is authoritative on any given data server at a time.

This mechanism does not support concurrent multi-metadata-server management of the same file -- see the single-metadata-server-per-file position in . The escrow control plane handles the metadata-server role moving between physical hosts over time (via TAKEOVER), not multiple metadata servers acting on the same escrow simultaneously. Wire mechanics for the operations referenced here (CHUNK_ESCROW_INSTALL, CHUNK_ESCROW_RELEASE, CHUNK_ESCROW_ENUMERATE, CHUNK_ESCROW_TAKEOVER, the incarnation-lease proof format, the escrow_id4 XDR) live in the New NFSv4.2 Common Data Structures and New NFSv4.2 Operations sections.

Chunk State Machine Each chunk on a data server occupies exactly one of four states. The transitions below are the complete set; any implementation of the data server's chunk state table MUST admit these transitions and no others.

EMPTY:: The chunk has no payload. CHUNK_READ returns a zero-filled result; CHUNK_WRITE against an EMPTY chunk is the first write.
PENDING:: The chunk has payload accepted by CHUNK_WRITE but not yet finalized. Not visible to CHUNK_READ (see ). Further CHUNK_WRITEs from the same writer MAY replace the payload in place (same cg_gen_id).
FINALIZED:: The writer has signalled via CHUNK_FINALIZE that it will send no more CHUNK_WRITEs for this generation. Still not visible to CHUNK_READ, but a candidate for CHUNK_COMMIT.
COMMITTED:: The chunk is durable and globally visible. Subsequent CHUNK_READs return this content until a newer COMMITTED generation replaces it. A higher-generation PENDING successor MAY exist concurrently; the rollback invariant in requires the data server to retain the COMMITTED content while that successor exists.

Transitions are driven by the operations named on the arrows. CHUNK_ROLLBACK against a COMMITTED chunk is used only on the repair path (see ) and replaces the chunk with a newer COMMITTED generation chosen by the repair actor, rather than returning the chunk to EMPTY. covers the lifecycle of a chunk's payload but not the lock that may be held on it. The lock has its own state machine, shown in .

Read-Time Generation Status Independent of the chunk lifecycle state () and the lock state (), a data server distinguishes three read-time statuses for any generation the caller names or observes at a chunk index. These are the higher-level categories the CHUNK_READ per-chunk status codes and the CHUNK_HEADER_READ discovery response classify into:

AVAILABLE:: the generation's payload is held by the data server and its integrity check succeeds at read time. A CHUNK_READ returns the payload with NFS4_OK. A CHUNK_HEADER_READ that observes an AVAILABLE predecessor is the input the caller uses to decide whether CHUNK_ROLLBACK's restore case ( "Rollback of COMMITTED Chunks") can succeed under the caller's own retention scope ().
ERRORED:: the owner-to-index association is still recorded but the payload is not readable -- the persisted checksum or guard check fails at read time (), the underlying storage is unreachable, or the chunk carries the errored bit set by an earlier CHUNK_ERROR. A CHUNK_READ returns NFS4ERR_PAYLOAD_NOT_ATOMIC (or another per-chunk status appropriate to the failure); the client reports the fault via LAYOUTERROR and the metadata server arranges repair. An ERRORED generation is observable -- its owner triple can be discovered -- but not consumable.
ABSENT:: the data server holds no generation at the requested chunk index (the chunk is EMPTY) or holds no generation matching the requested owner triple. A CHUNK_READ returns the synthetic-hole response () or per-chunk NFS4ERR_NOENT. A CHUNK_HEADER_READ that names a predecessor and finds ABSENT returns the absence in its per-chunk response fields; a CHUNK_ROLLBACK that names such a predecessor returns NFS4ERR_NO_PREDECESSOR () or NFS4ERR_INVAL per the rules under "Deletion Atomicity and Invalidated Triples" ().

The three statuses are exhaustive at read time: any generation the caller might reference is exactly one of AVAILABLE, ERRORED, or ABSENT. A generation can transition from AVAILABLE to ERRORED (via CHUNK_ERROR or a checksum failure) and from ERRORED back to AVAILABLE (via a successful repair sequence). The transition from AVAILABLE or ERRORED to ABSENT is a release under the retention scope rule () coupled with the payload/association biconditional (): the payload and its owner-to-index association are released together and the generation vanishes from the observable set. There is no transition from ABSENT back to any observable status for the same owner triple: the invalidated-triple rule ("Deletion Atomicity and Invalidated Triples" under ) forbids resurrection.

Consistency Guarantees The protocol provides per-chunk linearizability on COMMITTED state:

Once CHUNK_COMMIT returns success to a writer for a given chunk, every subsequent CHUNK_READ whose stateid postdates the COMMIT observes either that writer's data or the data of a later committed write. A reader MUST NOT observe a rolled-back write as if it had committed.
Concurrent writers on the same chunk in multi-writer mode serialize via chunk_guard4. On guard conflict one writer succeeds; the other receives NFS4ERR_CHUNK_GUARDED and MUST either abandon the write or re-read and retry. At most one generation becomes COMMITTED per serialized decision.
During repair, the chunk's lock is held continuously -- first by the original writer, then transferred to the metadata-server escrow owner on REVOKE_STATEID, and finally adopted by the repair client via CHUNK_LOCK_FLAGS_ADOPT. No writer that did not hold the lock may observe or mutate the chunk. The invariant "a chunk with a live lock has exactly one logical owner at any instant" is preserved across revocation.

Across multiple chunks the protocol makes no multi-chunk atomicity or ordering guarantee. A reader that reads chunk A at one offset and chunk B at another MAY observe A's new value and B's old value simultaneously. Applications that require multi-chunk atomicity MUST layer it above this protocol -- for example, via file-level checksums, application-level generation fields, or external transaction managers.

The chunk is the unit of atomicity:: Two properties follow.

Chunk-aligned writes do not interfere. Two concurrent writers whose writes cover disjoint chunks -- even writes that cover adjacent chunks -- never race. Each write terminates independently at COMMITTED per the per-chunk linearizability rule above.
Sub-chunk overlapping writes from different writers produce chunk-resolution-granularity contention. When two concurrent writers target overlapping byte ranges within a single chunk, chunk_guard4 resolves them: one writer's entire chunk-generation wins and becomes COMMITTED; the other writer sees NFS4ERR_CHUNK_GUARDED and is expected to re-read and retry if it wishes to apply its change on top of the winning generation (see ). The protocol does NOT produce byte-level merges of overlapping sub-chunk writes: the losing writer's bytes are not preserved as a partial update within the winning generation.

Applications that require byte-level write merging or sub-chunk ordering guarantees MUST serialize such writes externally, for example via NFSv4 byte range locks (, Section 12). The chunk size that bounds the atomicity unit for a given file is the product of ffv2m_striping_unit_size and the stripe width W in ; applications can query fattr4_coding_block_size (see ) to learn the effective chunk size and align their writes accordingly. This choice -- chunk-boundary atomicity rather than stripe- or block-boundary atomicity -- is load-bearing for the rest of the consistency story: the chunk_guard4 CAS evaluates at the chunk level, the PENDING / FINALIZED / COMMITTED state machine is per chunk, CHUNK_LOCK is per chunk, and repair via CB_CHUNK_REPAIR operates on chunks. A different atomicity boundary would require redefining those primitives.

Erasure-coded reads:: A reader of an erasure-coded file reconstructs the plaintext from any sufficient subset of k shards of the (k + m)-shard stripe; the guard values on those shards MUST agree. Shards with stale guards are ignored. This is not a quorum read in the Paxos sense -- there is no voting on a value; there is only reconstruction of the single value identified by the current guard.
Rollback invariant:: The data server MUST retain the prior FINALIZED or COMMITTED content of a chunk while any successor PENDING or FINALIZED chunk exists. A corollary of this rule is the lowest-guard-recoverable property: as long as at least k data servers in the mirror set retain the chunk at some generation G or lower, the payload that was COMMITTED at generation G (or earlier) can be reconstructed. This is the correctness basis for CHUNK_ROLLBACK (see ): rollback does not synthesize data, it simply selects the lowest-generation chunks whose guards agree across the mirror set and discards the higher-generation PENDING or FINALIZED chunks that triggered the rollback. The protocol never relies on locating or reconstructing data from outside the mirror set.
Visibility of non-committed state:: PENDING and FINALIZED chunks MUST NOT be globally visible. CHUNK_READ returns only COMMITTED content; a CHUNK_READ whose target chunk is currently PENDING or FINALIZED sees the predecessor COMMITTED content (or an EMPTY chunk if none exists), not the in-progress successor. A writer observing its own PENDING or FINALIZED chunk MAY receive the in-progress content on the same stateid that produced it, but no other stateid -- on the same or a different client -- sees it. The retention window that makes the prior COMMITTED content available to CHUNK_READ and to CHUNK_ROLLBACK is itself bounded; see for the normative scoping rule.

Ownership and Scope of Retained Prior Content The rollback invariant in requires a data server to retain the prior FINALIZED or COMMITTED content of a chunk while any successor PENDING or FINALIZED chunk exists. That retained content -- sometimes informally called the "safe buffer" -- is not global state. It is scoped to the stateid that wrote the PENDING successor, and its retention and visibility are governed by that owning stateid's lease.

Owner:: The data server MUST record, alongside each PENDING chunk, the owning stateid (the stateid presented on the CHUNK_WRITE that produced the PENDING). This is the owning writer's stateid; it identifies the client and openowner/lockowner that the data server will release the PENDING to on CHUNK_FINALIZE or CHUNK_COMMIT, and that the metadata server will treat as the authoritative owner for purposes of .
Visibility:: Before transition to COMMITTED, the PENDING content is visible only on the owning stateid. A CHUNK_READ presenting any other stateid (from the same client or a different client) MUST observe the predecessor COMMITTED or EMPTY state, not the PENDING successor. This is the normative form of the "non-committed data MUST NOT be globally visible" rule stated in the "Visibility of non-committed state" bullet of .
Retention window:: The data server MUST retain the predecessor COMMITTED (or FINALIZED) content that the PENDING is superseding for as long as the owning stateid's lease is valid. If the owning stateid's lease expires without the PENDING reaching COMMITTED, the retention obligation for that PENDING ends (see for the scavenger rule that drives demotion). If the PENDING does reach COMMITTED, the new COMMITTED generation supersedes the prior one under the standard rollback invariant and its own retention is governed by any newer PENDING successor.

The practical effect is that the "safe buffer" for a chunk is not an unbounded chunk-global state but a per-writer window bounded by that writer's lease. The data server always has a rule for discarding retained prior content -- it is the owning stateid's lease expiry -- so a chunk cannot accumulate indefinitely many retained generations even in the presence of dropped or partitioned writers.

Owner-to-Index Persistence Coupling The wire lifecycle operations (CHUNK_COMMIT, CHUNK_FINALIZE, CHUNK_ROLLBACK) name generations by full owner triple () and require the data server to locate the chunk-index each triple was written at. For that lookup to succeed after a data server restart, the data server MUST persist the owner-to-index association with a durability floor that matches the payload's: without the association the payload can never again be addressed by a lifecycle operation.

Uniqueness invariant (normative):

An accepted owner triple is UNIQUE across the live generations the data server holds for a given file: at any instant there is at most one live generation on any chunk of the file that matches that full triple. Two writers with distinct co_client_id values cannot collide. A single writer that reuses a co_cohort_id + co_id pair within the same co_client_id across two distinct chunk indices MUST NOT do so while the earlier generation remains live; the data server MAY reject a CHUNK_WRITE that attempts to create such a collision with NFS4ERR_INVAL in the corresponding cwr_block_status slot. The predecessor-retention rule () is compatible: a retained predecessor and its successor on the same chunk index MUST carry distinct triples (typically distinct co_cohort_id under a shared co_client_id) so that CHUNK_ROLLBACK can name each unambiguously.

Durability floor (normative, per CHUNK_WRITE "Stability and Activation"):

The durability requirement varies by the CHUNK_WRITE stability level:

FILE_SYNC4:: both the chunk payload AND its owner-to-index association MUST survive a data server restart.
DATA_SYNC4:: both the chunk payload AND its owner-to-index association MUST survive a data server restart. (The association is retrieval metadata for the payload; it shares the payload's durability floor. An implementation MAY treat DATA_SYNC4 identically to FILE_SYNC4.)
UNSTABLE4:: the association MAY be lost on restart, but ONLY if the payload is also lost. A data server MUST NOT retain payload without its associated owner triple; a payload whose association was lost is unaddressable by every lifecycle operation and MUST be treated as destroyed. cwr_writeverf changes on any restart that loses UNSTABLE4 state, allowing the client to detect the loss.

A data server that cannot honor the durability floor for a given stability level MUST reject the CHUNK_WRITE with NFS4ERR_IO rather than accepting the payload without its association. The retention scope rule () governs WHEN a predecessor generation's payload + association may be released; the payload/association biconditional in the next subsection governs the INVARIANT that whenever the payload survives, the association survives with it, and vice versa.

Payload and Association Biconditional For every generation the data server holds -- PENDING, FINALIZED, or COMMITTED, including any predecessor retained under the rollback invariant () -- the chunk payload and its owner-to-index association () MUST share a lifetime. A conforming data server MUST NOT release one while retaining the other:

MUST NOT retain payload without association:: A chunk payload whose owner-to-index association has been released is unaddressable by every lifecycle operation (CHUNK_COMMIT, CHUNK_FINALIZE, CHUNK_ROLLBACK) because those operations name generations by full owner triple (). A data server that cannot locate the recorded chunk index for a presented triple returns the appropriate per-entry error under the release-scope split at : NFS4ERR_INVAL when the data server holds a concrete invalidation context that identifies the triple (structurally invalid, or released by an explicit CHUNK_ROLLBACK delete case within the session slot's replay-cache window), NFS4ERR_NO_PREDECESSOR otherwise. Retaining the payload while making it unaddressable serves no purpose and is prohibited. The payload MUST be released atomically with the association.
MUST NOT release association while retaining payload:: A recorded owner-to-index association refers to a specific chunk payload; the data server MUST NOT retain the association after releasing that payload. A subsequent CHUNK_READ or lifecycle operation whose recorded index still points to a released payload would return data whose provenance the client cannot verify against its own writer history; forbidding this releases the client from having to detect such stale associations.

The biconditional is symmetric: released together, retained together. This coupling is what makes CHUNK_ROLLBACK's delete case () atomic -- invalidating a generation drops BOTH the association and the payload in one step -- and what makes CHUNK_ROLLBACK's restore case rely on the predecessor's association surviving with its payload as a single unit. Data servers implementing an on-disk chunk store SHOULD treat the association and payload as the crash-consistent atomic unit at the storage layer. The retention scope rule () governs WHEN the pair may be released (bounded by the owning stateid's lease and any successor's presence); this biconditional governs the INVARIANT that whenever the release happens, the two go together.

Progress and Termination Under the failure model above, the protocol guarantees the following progress properties:

Data-path progress:

If all mirrors are reachable and none are failed, a CHUNK_WRITE followed by CHUNK_FINALIZE followed by CHUNK_COMMIT completes in O(1) round trips independent of cluster size. In particular, there is no consensus round, no leader election, and no quorum voting on the write itself. The three operations MAY be amortized across compounds: a steady-state writer sending a series of CHUNK_WRITEs can piggyback the CHUNK_FINALIZE of the previous write on the compound that carries the next write (for example, SEQUENCE + PUTFH + CHUNK_FINALIZE + CHUNK_WRITE), reducing the data-path happy case to a single round trip per CHUNK_WRITE rather than three. The CHUNK_COMMIT for the final write in a sequence MAY similarly ride on the CLOSE compound. These compound-packing optimizations are permitted by the normal NFSv4.2 compound rules and require no protocol extensions.

Repair termination:

Every CB_CHUNK_REPAIR completes in bounded time. The client selected as the repair actor either:

returns NFS4_OK for every range in ccra_ranges (repair succeeded), or
returns NFS4ERR_PAYLOAD_LOST for one or more ranges (the erasure coding lost too many shards to reconstruct; the data is permanently unrecoverable), or
fails to respond within the ccra_deadline, in which case the metadata server MUST re-select under the rules in or MUST declare the ranges lost.

NFS4ERR_PAYLOAD_LOST is terminal for the affected ranges. The protocol makes no further attempt to recover them.

Eventual trust-table convergence:

After a metadata server restart, each data server's trust table converges to the metadata server's view within one metadata-server lease period. Entries that the metadata server does not re-issue expire naturally via tsa_expire; entries that the metadata server does re-issue transition from pending-revalidation back to active on the next TRUST_STATEID (see ).

Orphaned PENDING scavenger:

A PENDING chunk whose owning stateid (see ) has expired without transition to FINALIZED or COMMITTED is an orphan. The metadata server MUST drive demotion of orphaned PENDINGs so that no chunk remains in a non-terminal state indefinitely:

When an owning stateid's lease expires, the metadata server identifies every PENDING chunk owned by that stateid (either from its own bookkeeping or by query against the data server) and issues the control-plane operations needed to demote each PENDING.
Demotion replaces the PENDING with the predecessor COMMITTED (or EMPTY) content that the data server has been retaining under . The data server MUST NOT wait for a separate client action before performing the demotion.
Any CHUNK_LOCK held in escrow on behalf of the expired stateid (see ) is released after a metadata-server-defined grace period. The grace period exists to let a recovering client reclaim its lock via the grace / reclaim path defined in ; on expiry of the grace period without reclaim, the lock becomes available for new CHUNK_LOCK_FLAGS_ADOPT acquirers.

The scavenger timeout (the delay between lease expiry and demotion) is implementation-defined but SHOULD be tied to the metadata server lease period so that it composes naturally with existing NFSv4 grace / reclaim semantics. A scavenger timeout shorter than the lease risks racing an in-progress client reclaim; a timeout substantially longer than the lease extends the retention budget without a commensurate benefit. A late-arriving client op against a demoted PENDING sees NFS4ERR_BAD_STATEID under trusted-stateid tight coupling () or NFS4ERR_NO_PREDECESSOR under loose coupling (); a CHUNK_LOCK reclaim after the grace period fails per the RFC 8881 grace-reclaim semantics.

The protocol does NOT guarantee progress if the metadata server is unavailable for longer than its lease period -- this is the standard NFSv4 lease assumption and is inherited unchanged.

Relation to Classical Consensus Classical consensus protocols (Paxos, Raft, Viewstamped Replication) solve the problem of reaching agreement among mutually-distrusting replicas in the absence of a trusted coordinator. They typically cost two or three round trips per decision, require a majority of replicas to be live and reachable for progress, and impose the overhead of leader election and log replication. This protocol is not a consensus protocol and does not attempt to be. Its approach instead is:

Designated coordinator:: The metadata server is the coordinator for a file. Clients accept the metadata server's authority for layout grants, stateid registration, repair actor selection, and revocation. This assumption is the same one made by and all pNFS layout types to date.
Per-chunk CAS, not per-chunk voting:: Concurrent writes on the same chunk serialize via chunk_guard4 as a CAS primitive (see ). No replica vote is required; the data server that owns the chunk evaluates the guard locally and rejects stale writes with NFS4ERR_CHUNK_GUARDED.
Pessimistic locks off the critical path:: CHUNK_LOCK is used only during repair, never on the normal write path. Lock escrow (see ) preserves the "exactly one owner" invariant across stateid revocation without requiring a consensus round to elect the next owner.
Erasure-coded reads replace quorum reads:: A reader reconstructs from any k of k + m shards with matching guards. No voting is needed because there is no disagreement to resolve: the guard identifies the single generation that was committed.

The result is a data path with O(1) round-trip cost independent of the number of replicas, and a repair path whose cost is bounded by the number of affected chunks rather than by the cluster size. Metadata-server high availability is orthogonal. Deployments that require a highly-available metadata server MAY replicate metadata-server state across multiple metadata server instances using classical consensus (Raft, Paxos, or equivalent). Such replication is implementation-defined; from a pNFS client's perspective a highly-available metadata server looks like a single metadata server that occasionally resets its session and triggers grace-period reclaim, and the client's behavior is already specified by . This protocol neither requires nor precludes such an implementation.

Non-Goals For clarity, the protocol explicitly does not provide:

Byzantine fault tolerance:: A data server that deliberately misreports its state, or a client that bypasses its own authentication, is outside the trust model. Deployments requiring Byzantine tolerance MUST add it in a layer above or below this protocol.
Metadata server high availability:: Single-metadata-server-per-file is the protocol model. Metadata server high availability, if deployed, is implemented below the wire protocol and transparent to clients.
Cross-file atomicity:: Writes to multiple files are not atomic at the protocol level. File-system-level transactions are not defined.
Multi-chunk atomicity within a single file:: COMMITs on distinct chunks are independent. A reader may observe a partial write across chunks; applications must layer their own consistency if they need otherwise.
Global linearizability across unrelated files:: Each file's COMMITTED state is linearizable in isolation; no total order is defined across files.
Authenticated malicious client protection:: An authenticated client may write garbage into its own chunks with a correctly computed checksum; see . A bit-flip-class checksum is a transport-integrity check, not an adversarial-integrity check; cryptographic-class checksums detect adversarial modification by anyone other than the authenticated writer.

NFSv4.2 Operations Allowed to Data Files In the Flexible File Version 1 Layout Type (), the data path between client and data server was NFSv3 (); the operations a client sent to a data file were limited to READ, WRITE, and COMMIT, and the operations the metadata server sent on its control plane to the data server were limited to GETATTR, SETATTR, CREATE, and REMOVE. An NFSv4.2 data server, as used by the Flexible File Version 2 Layout Type, exposes a much larger operation set. This section defines which operations a client MAY send to a data file, which operations the metadata server MAY send, and which operations a data server MUST reject. The restrictions below apply only to operations directed at a data file on a data server. Clients retain the full NFSv4.2 operation set for files visible through the metadata server, including the operations prohibited below (RENAME, LINK, CLONE, COPY, ACL-scoped SETATTR, and so on). The metadata server MAY internally use operations on data files that clients MUST NOT send, as part of its control-plane duties for the file (see ).

Control Plane: Metadata Server to Data Server When the metadata server acts as a client to a data server, it is managing the data file on behalf of the metadata file's namespace. A data server MUST support the following operations on data files when issued by the metadata server:

SEQUENCE, PUTFH, PUTROOTFH, GETFH ( Sections 18.46, 18.19, 18.21, 18.8): session and filehandle plumbing.
LOOKUP (): directory traversal the metadata server issues when locating data files it has allocated for its use.
GETATTR (): the metadata server issues GETATTR against the data file after a write layout is returned, to pull the post-write size, mtime, and other attributes back and reconcile its cached view. Any other attribute queries the metadata server needs for the same purpose use the same operation.
SETATTR (): data file truncate for metadata-server-level SETATTR(size) fan-out, synthetic uid/gid rotation for fencing, and mode-bit initialization when the metadata server binds a data file to a new metadata-level file.
CREATE (): the metadata server's own allocation of data files on the data server.
REMOVE (): cleanup on metadata server file unlink.
OPEN, CLOSE ( Sections 18.16, 18.2): used by the metadata server when it acts as a client to the data server for I/O routed through the metadata server or through a proxy server on the metadata server's behalf.
EXCHANGE_ID, CREATE_SESSION, DESTROY_SESSION, BIND_CONN_TO_SESSION, DESTROY_CLIENTID ( Sections 18.35, 18.36, 18.37, 18.34, 18.50): control-session management. The metadata server sets EXCHGID4_FLAG_USE_PNFS_MDS in its EXCHANGE_ID. A data server that supports the tight coupling control protocol (see ) identifies the metadata server's session by EXCHGID4_FLAG_USE_PNFS_MDS and accepts TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID on that session.
TRUST_STATEID, REVOKE_STATEID, BULK_REVOKE_STATEID: the metadata-server-to-data-server tight coupling trust-table control operations.

The metadata server MAY also use other NFSv4.2 operations on data files as implementation-defined control-plane actions (for example, COPY or CLONE to migrate a data file between data servers during a proxy server operation). The list above is the minimum set a flexible file v2 layout data server MUST support for the metadata server's use.

Data Path: Client to Data Server A pNFS client with an active flexible file v2 layout MUST restrict the operations it issues against data files to the operations defined below. A data server that has identified the file as a chunked data file (see ) MUST reject any other operation on that file with NFS4ERR_NOTSUPP.

Data-File Identification on the Data Server The "MUST reject" rules in this section apply on the data server side only to files the data server has identified as chunked data files. A data server identifies a file by any of the following means, in decreasing order of authority:

fattr4_chunked_data_file = TRUE:: The metadata server has set this attribute () when it allocated the file as a chunked data file. This is the authoritative per-file identification and is required for a data server that supports the attribute.
Live TRUST_STATEID entry for the file:: Under trusted-stateid tight coupling (), a live trust entry for the file registered via TRUST_STATEID identifies the file as under flexible file v2 layout management. This is a fallback for data servers that do not yet support fattr4_chunked_data_file, and it is per-client rather than per-file, but it is sufficient to trigger the "MUST reject" rules for that client's operations against the file.
Deployment namespace convention:: Deployments SHOULD export chunked data files in a namespace scope not shared with client-accessible file access, as one means of preventing a misconfigured or malicious client from reaching data files through a normal NFS mount. A data server MAY apply the "MUST reject" rules to every file in a namespace configured for flexible file v2 layout data-file service, without per-file classification. Note that a separate NFS export does not close direct filesystem access on the data server host itself; that is outside the scope of this specification.

If none of the above identifies the file, the data server cannot reliably classify it and the client-side "MUST NOT" rules in this section remain the primary defense. Client-side compliance is mandatory in all deployment modes regardless of what the data server can enforce.

Session and Identity Plumbing Required for all protection modes:

SEQUENCE, PUTFH, GETFH, PUTROOTFH ( Sections 18.46, 18.19, 18.8, 18.21).
EXCHANGE_ID, CREATE_SESSION, DESTROY_SESSION, BIND_CONN_TO_SESSION, DESTROY_CLIENTID ( Sections 18.35, 18.36, 18.37, 18.34, 18.50).
RECLAIM_COMPLETE ().
SECINFO, SECINFO_NO_NAME ( Sections 18.29, 18.45): discovery of acceptable security flavors on the data server.

These operations are baseline NFSv4.2 session plumbing and are supported on data files as on any NFSv4.2 file.

Stateid Model on the Data Server The stateid presented on a CHUNK operation is a layout stateid returned by a prior LAYOUTGET against the metadata server (see ), NOT an open stateid, byte range lock stateid, or delegation stateid. A pNFS client does NOT issue OPEN against the data server. This is a meaningful departure from the stateid model in (which states that the WRITE stateid "represents a value returned from a previous byte range LOCK or OPEN request or the stateid associated with a delegation"), and clients implementing Flexible File Version 2 MUST NOT carry over those expectations to the data path. The three roles the RFC 8881 stateid plays on a regular NFSv4 server split apart in the Flexible File Version 2 data-server model:

Open and share-mode tracking:: Lives at the metadata server, established by OPEN () on the metadata-server filehandle. The metadata server's open stateid is NOT exposed to data servers; share-mode conflicts are resolved at the metadata server before LAYOUTGET grants a layout.
Byte-range lock tracking:: Does not apply at the data server. Locking on the data path is chunk-range rather than byte range, expressed via CHUNK_LOCK, and the lock holder is identified by chunk_owner4 (the (co_cohort_id, co_client_id, co_id) triple) rather than by a lock stateid. A client wanting byte range locks on a file MUST acquire them on the metadata-server filehandle, where standard byte range locking applies.
I/O authorization on the data server:: The layout stateid carried on CHUNK operations. encodings that use CHUNK operations require tight coupling (); the metadata server registers each issued layout stateid with the data server via TRUST_STATEID together with the ffv2m_client_id assigned to the writer, and the data server validates subsequent CHUNK operation stateids against the trust table and the presented cwa_client_id against the trust-table's tsa_client_id. Loose coupling applies only to PASSTHROUGH mirrors, which use regular READ/WRITE authorized by the synthetic uid/gid the layout carries (see ).

Because the layout stateid does authorization but does not identify a per-open or per-lock owner, a single client may present the same layout stateid on many CHUNK operations across many parallel writers within the client, without any of the open-owner ordering constraints imposes on regular NFSv4 stateids. Chunk-level write ordering and contention are resolved by the per-chunk chunk_guard4 CAS () and the chunk-range CHUNK_LOCK, not by stateid-owner sequencing.

GETATTR on a Data File GETATTR MAY be issued by a client against a data file. The primary use case is repair: a repair actor selected by CB_CHUNK_REPAIR may need to query the per-server file size or allocation state when reconstructing a payload, and the proxy server described informally in similarly benefits from attribute queries on surviving mirrors. Diagnostic use is also permitted. Clients MUST NOT treat GETATTR values returned by a data server as authoritative for any file attribute (size, timestamps, owner, mode, ACL, and so on). The metadata server is the sole authority for file attributes. Values returned by a data server reflect the per-server data file instance only and MAY diverge from the metadata server's view, particularly during a write layout's lifetime or during a proxy server transition. A client that uses a data-server GETATTR result to determine the file's visible size will observe inconsistencies.

SETATTR on a Data File Clients MUST NOT issue SETATTR against a data file. A data server that has identified the file as a chunked data file (see ) MUST reject a client SETATTR with NFS4ERR_NOTSUPP. Attribute changes on data files MUST be reconciled with the metadata server's view and cannot be applied unilaterally by a client. A client that wants to truncate, change the mode, change ownership, or otherwise modify attributes on a file MUST issue SETATTR to the metadata server for the file's metadata server handle; the metadata server fans the change out to the data files as a control-plane operation. This rule explicitly covers truncate (SETATTR with size in the bitmap): a client MUST NOT truncate a data file directly. See for how the metadata server handles truncate on chunked files. Similarly, a client MUST NOT issue DEALLOCATE against a data file; see the next subsection.

Metadata-Server-Driven Truncate on Chunked Files A client that wants to truncate a chunked file MUST issue SETATTR(FATTR4_SIZE) to the metadata-server filehandle (see ). The metadata server translates the logical truncate into per-shard size changes across the data servers in each mirror. For FFV2_ENCODING_REPLICATED, per-shard size equals the logical truncate size; for erasure-coded encodings the per-shard sizes are derived from the geometry parameters below.

Stripe-aligned truncate:: When the new size lies on a stripe boundary (including zero), no chunk re-encoding is required. The metadata server computes per-shard sizes from the encoding geometry it issued in the layout (k, m, and the projection parameters for Mojette; see ) and issues per-data-server SETATTR(FATTR4_SIZE) with the computed per-shard size. Geometry parameters are sufficient arithmetic; no encoding implementation is required at the metadata server.
Non-stripe-aligned truncate:: When the new size falls within a stripe, the data shards covering the partial stripe must be truncated and the parity shards re-encoded from the truncated data. Because re-encoding requires running the erasure transform, the metadata server MUST delegate this case to an encoding-aware actor: either a proxy server for proxy-mediated truncate, or an encoding-aware client selected per via CB_CHUNK_REPAIR with the affected partial-stripe chunks as the repair target. If neither path is available, the metadata server MUST return NFS4ERR_NOTSUPP to the originating SETATTR.

The metadata server knows encoding geometry from the layout but is not required to include an encoding implementation. The delegation rule above accommodates a metadata server that has geometry knowledge only.

PASSTHROUGH Data Files (FFV2_ENCODING_PASSTHROUGH) For a mirror whose ffv2m_encoding_type_data is FFV2_ENCODING_PASSTHROUGH (see ), client operations on the data file follow the same pattern as the File Layout Type in and the Flexible File Version 1 Layout Type in : Required:

READ ().
WRITE ().
COMMIT ().

Optional (the client MAY send, and the data server MAY support):

READ_PLUS (): hole-aware reads.
SEEK (): hole and data detection.
ALLOCATE (): space reservation hint.

The client MUST NOT send:

DEALLOCATE (): hole punching is a metadata-server responsibility; the client issues DEALLOCATE on the metadata-server filehandle, and the metadata server fans out to the data servers as a control-plane operation.

Chunked Data Files For a mirror whose ffv2m_encoding_type_data is any of the chunked encoding types defined in this document -- i.e., every FFV2ENCODING* value except FFV2_ENCODING_PASSTHROUGH (see ) -- client operations use the CHUNK operations rather than READ / WRITE / COMMIT. This includes FFV2_ENCODING_REPLICATED, whose per-shard payload is the input bytes verbatim but is dispatched via the CHUNK operations to carry the per-chunk checksum and guard ( legend). Required for all chunked clients:

CHUNK_WRITE.
CHUNK_READ.
CHUNK_FINALIZE.
CHUNK_COMMIT.
CHUNK_HEADER_READ.
CHUNK_LOCK and CHUNK_UNLOCK.
CHUNK_ROLLBACK.

Required for clients that participate in repair:

CHUNK_ERROR.
CHUNK_REPAIRED.
CHUNK_WRITE_REPAIR.

Clients MUST NOT send:

READ, WRITE, COMMIT against a chunked data file. A data server MUST reject these with NFS4ERR_NOTSUPP and MAY log the client for operator attention; this case is almost always a client bug in which the client did not inspect the mirror's ffv2m_encoding_type_data before issuing I/O.
READ_PLUS, SEEK, ALLOCATE, DEALLOCATE against a chunked data file. Chunk-level allocation is a metadata-server responsibility.
SETATTR against a chunked data file (the general prohibition in applies to all data files; truncate in particular is handled by the metadata server per ).

Operations That MUST NOT Be Sent to a Data File Clients MUST NOT send the following operations to a data server on a data file, regardless of protection mode. A data server that has identified the file as a chunked data file (see ) MUST return NFS4ERR_NOTSUPP:

OPEN, CLOSE, OPEN_DOWNGRADE, OPEN_CONFIRM ( Sections 18.16, 18.2, 18.18, 18.20). Opens occur on the metadata server; the stateid obtained there is used on the data path.
LOCK, LOCKU, LOCKT, RELEASE_LOCKOWNER ( Sections 18.10, 18.11, 18.13, 18.24). Byte-range locks on data files are not supported; chunked files use CHUNK_LOCK, and PASSTHROUGH files rely on metadata-server coordination.
DELEGPURGE, DELEGRETURN, WANT_DELEGATION ( Sections 18.5, 18.6 and ). Delegations are issued by the metadata server.
Any operation whose purpose is to manipulate the file's namespace: RENAME, LINK, SYMLINK, CREATE (at the client's file-creation use, not the metadata server's own allocation of data files on the data server described above), REMOVE. Namespace operations belong on the metadata server.
Any ACL-scoped SETATTR or GETATTR bit (FATTR4_ACL, FATTR4_DACL, FATTR4_SACL). Access control on data files is delegated to the metadata server.
CLONE, COPY, COPY_NOTIFY, OFFLOAD_CANCEL, OFFLOAD_STATUS ( Sections 15.13, 15.2, 15.3, 15.8, 15.9). File-level data migration is a metadata-server responsibility.
LAYOUTGET, LAYOUTCOMMIT, LAYOUTRETURN, LAYOUTSTATS, LAYOUTERROR, GETDEVICEINFO, GETDEVICELIST ( Sections 18.43, 18.42, 18.44, Sections 15.7, 15.6, Sections 18.40, 18.41). Layout operations belong on the metadata server.
TRUST_STATEID, REVOKE_STATEID, BULK_REVOKE_STATEID (, , ). These are metadata-server-to-data-server control-plane operations; a data server rejects them with NFS4ERR_PERM when received on a client session (see ).
NFSv4 READ, WRITE, and COMMIT ( Sections 18.22, 18.32, 18.3). Chunked data files use CHUNK_READ, CHUNK_WRITE, and CHUNK_COMMIT respectively; NFSv4 byte-range READ / WRITE / COMMIT bypasses the chunk envelope (chunk_guard4 CAS, per-chunk checksum, lifecycle state) that this document's consistency guarantees depend on. A data server that has identified the file as a chunked data file MUST reject NFSv4 READ / WRITE / COMMIT with NFS4ERR_NOTSUPP. PASSTHROUGH data files () are non-chunked and continue to use NFSv4 READ / WRITE / COMMIT normally; their fattr4_chunked_data_file is FALSE ().

CHUNK Operations on Non-Chunked Data Files The reciprocal rule: clients MUST NOT send CHUNK operations (CHUNK_READ, CHUNK_WRITE, CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK, CHUNK_HEADER_READ, CHUNK_LOCK, CHUNK_UNLOCK, CHUNK_ERROR, CHUNK_REPAIRED, CHUNK_WRITE_REPAIR) to a data server on a file that is not a chunked data file. Non-chunked data files (PASSTHROUGH mirrors, and any file the metadata server has not marked as chunked) do not carry a chunk envelope and cannot honor CHUNK operation semantics. A data server that has identified the file as non-chunked (fattr4_chunked_data_file = FALSE) MUST reject CHUNK operations against that file with NFS4ERR_NOTSUPP. A data server that cannot identify the file under any of the paths in relies on the client-side "MUST NOT" as the primary defense; a client with an active layout will not present a chunked-layout stateid against a non-chunked file, so standard stateid validation () catches the misdirected CHUNK operation with NFS4ERR_BAD_STATEID before the chunk-layer rules apply.

Callback Path: Data Server to Client A data server does not call back directly to pNFS clients. Recall notifications and repair coordination flow through the metadata server's backchannel session with the client. The callbacks a client will observe that affect its data files are:

CB_LAYOUTRECALL ().
CB_NOTIFY_DEVICEID ().
CB_RECALL_ANY ().
CB_CHUNK_REPAIR.

A data server influences these callbacks only indirectly, via LAYOUTERROR reports the client issues to the metadata server or by returning error codes that prompt the client to report. A data server MUST NOT attempt to send CB_* operations to clients directly.

Summary Table The classification below adapts the operation taxonomy of (REQUIRED / RECOMMENDED / OPTIONAL / MUST NOT IMPLEMENT) to the two-direction per-operation view a Flexible File Version 2 data server requires. Two of the four labels in the table below match usage; the other two are extensions specific to this document.

REQUIRED:: The data server MUST support the operation on this path. Matches REQ.
OPTIONAL:: The data server MAY support the operation; if it does, the actor in this column MUST tolerate the absence of support. Matches OPT.
MUST NOT:: The actor in this column MUST NOT send the operation, and the data server MUST reject it with NFS4ERR_NOTSUPP. This per-direction prohibition extends 's single-axis MUST NOT IMPLEMENT classification: an operation may be forbidden on one path (client to data server) while required on another (metadata server to data server). SETATTR is the canonical example.
MAY:: The metadata server MAY use the operation as an implementation-defined control-plane action. Not in ; specific to the metadata-server-to-data-server path in this document.

NFSv4.2 operations allowed on data files

Operation	Client -> data server	metadata server -> data server
SEQUENCE, PUTFH, GETFH, PUTROOTFH	REQUIRED	REQUIRED
EXCHANGE_ID, CREATE_SESSION, DESTROY_SESSION, BIND_CONN_TO_SESSION, DESTROY_CLIENTID	REQUIRED	REQUIRED
RECLAIM_COMPLETE	REQUIRED	REQUIRED
SECINFO, SECINFO_NO_NAME	REQUIRED	MAY
GETATTR	OPTIONAL (non-authoritative)	REQUIRED
SETATTR	MUST NOT	REQUIRED
LOOKUP, CREATE, REMOVE	MUST NOT	REQUIRED
READ, WRITE, COMMIT	REQUIRED (PASSTHROUGH); MUST NOT (chunked)	MAY
READ_PLUS, SEEK, ALLOCATE	OPTIONAL (PASSTHROUGH); MUST NOT (chunked)	MAY
DEALLOCATE	MUST NOT	MAY
CHUNK_WRITE, CHUNK_READ, CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_HEADER_READ, CHUNK_LOCK, CHUNK_UNLOCK, CHUNK_ROLLBACK	REQUIRED (chunked); MUST NOT (PASSTHROUGH)	not used
CHUNK_ERROR, CHUNK_REPAIRED, CHUNK_WRITE_REPAIR	REQUIRED (chunked repair actors); MUST NOT (PASSTHROUGH)	not used
OPEN, CLOSE, OPEN_DOWNGRADE, OPEN_CONFIRM	MUST NOT	OPTIONAL (proxy I/O)
LOCK, LOCKU, LOCKT, RELEASE_LOCKOWNER	MUST NOT	MUST NOT
DELEGPURGE, DELEGRETURN, WANT_DELEGATION	MUST NOT	MUST NOT
RENAME, LINK, SYMLINK	MUST NOT	MUST NOT
CLONE, COPY, COPY_NOTIFY, OFFLOAD_CANCEL, OFFLOAD_STATUS	MUST NOT	MAY (data migration)
LAYOUTGET, LAYOUTCOMMIT, LAYOUTRETURN, LAYOUTSTATS, LAYOUTERROR, GETDEVICEINFO, GETDEVICELIST	MUST NOT	MUST NOT
ACL-scoped GETATTR/SETATTR bits	MUST NOT	MAY
TRUST_STATEID, REVOKE_STATEID, BULK_REVOKE_STATEID	MUST NOT	REQUIRED (tight coupling)
CHUNK_ESCROW_INSTALL, CHUNK_ESCROW_RELEASE, CHUNK_ESCROW_ENUMERATE, CHUNK_ESCROW_TAKEOVER	MUST NOT	REQUIRED

The (PASSTHROUGH) and (chunked) qualifiers in the client-to-data-server column select by the mirror's ffv2m_encoding_type_data value. FFV2ENCODING_PASSTHROUGH () uses NFSv3 WRITE / READ or NFSv4 READ / WRITE directly and does not use the CHUNK operations. Every other standards-track encoding (any FFV2_ENCODING* value other than FFV2_ENCODING_PASSTHROUGH; see ) is (chunked): it uses CHUNK_WRITE and CHUNK_READ and MUST NOT use the RFC 8881 READ / WRITE / COMMIT operations against the data server. FFV2_ENCODING_REPLICATED replicates the payload verbatim across N shards but dispatches via the CHUNK operations so each shard carries the per-chunk checksum this version of the layout type relies on for end-to-end integrity ().

Flexible File Version 2 Layout Type Return layoutreturn_file4 is used in the LAYOUTRETURN operation to convey layout type specific information to the server. It is defined in (also shown in ).

Layout Return XDR ; }; union layoutreturn4 switch(layoutreturn_type4 lr_returntype) { case LAYOUTRETURN4_FILE: layoutreturn_file4 lr_layout; default: void; }; struct LAYOUTRETURN4args { /* CURRENT_FH: file */ bool lora_reclaim; layouttype4 lora_layout_type; layoutiomode4 lora_iomode; layoutreturn4 lora_layoutreturn; }; ]]> If the lora_layout_type layout type is LAYOUT4_FLEX_FILES_V2 and the lr_returntype is LAYOUTRETURN4_FILE, then the lrf_body opaque value is defined by ffv2_layoutreturn4 (see ). This allows the client to report I/O error information or layout usage statistics back to the metadata server as defined below. Note that while the data structures are built on concepts introduced in NFSv4.2, the effective discriminated union (lora_layout_type combined with ffv2_layoutreturn4) allows for an NFSv4.1 metadata server to utilize the data.

I/O Error Reporting

ffv2_ioerr4

ffv2_ioerr4 ; /// }; /// ]]> Recall that defines device_error4 as in :

device_error4 The ffv2_ioerr4 structure is used to return error indications for data files that generated errors during data transfers. These are hints to the metadata server that there are problems with that file. For each error, ffv2ie_errors.de_deviceid, ffv2ie_offset, and ffv2ie_length represent the storage device and byte range within the file in which the error occurred; ffv2ie_errors represents the operation and type of error. The use of device_error4 is described in . Even though the storage device might be accessed via NFSv3 and reports back NFSv3 errors to the client, the client is responsible for mapping these to appropriate NFSv4 status codes as de_status. Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4 operations.

Layout Usage Statistics

ff_io_latency4

ff_io_latency4 Both operation counts and bytes transferred are kept in the ff_io_latency4 (see ). As seen in ff_layoutupdate4 (see ), READ and WRITE operations are aggregated separately. READ operations are used for the ff_io_latency4 ffv2lu_read. Both WRITE and COMMIT operations are used for the ff_io_latency4 ffv2lu_write. "Requested" counters track what the client is attempting to do, and "completed" counters track what was done. There is no requirement that the client only report completed results that have matching requested results from the reported period. ffv2il_bytes_not_delivered is used to track the aggregate number of bytes requested but not fulfilled due to error conditions. ffv2il_total_busy_time is the aggregate time spent with outstanding RPC calls. ffv2il_aggregate_completion_time is the sum of all round-trip times for completed RPC calls. In , the nfstime4 is defined as the number of seconds and nanoseconds since midnight or zero hour January 1, 1970 Coordinated Universal Time (UTC). The use of nfstime4 in ff_io_latency4 is to store time since the start of the first I/O from the client after receiving the layout. In other words, these are to be decoded as duration and not as a date and time. Note that LAYOUTSTATS are cumulative, i.e., not reset each time the operation is sent. If two LAYOUTSTATS operations for the same file and layout stateid originate from the same NFS client and are processed at the same time by the metadata server, then the one containing the larger values contains the most recent time series data.

ff_layoutupdate4

ff_layoutupdate4 ffv2lu_addr differentiates which network address the client is connected to on the storage device. In the case of multipathing, ffv2lu_fhandle indicates which read-only copy was selected. ffv2lu_read and ffv2lu_write convey the latencies for both READ and WRITE operations, respectively. ffv2lu_duration is used to indicate the time period over which the statistics were collected. ffv2lu_flags is a 32-bit bitmask reporting per-report properties; when FFV2_LAYOUTSTATS_FLAGS_LOCAL is set, the I/O was serviced by the client's cache. This flag allows the client to inform the metadata server about "hot" access to a file it would not normally be allowed to report on. Unassigned bits are reserved for future revisions; receivers MUST ignore bits they do not recognize.

ff_iostats4

ff_iostats4 defines io_info4 as in .

io_info4 With pNFS, data transfers are performed directly between the pNFS client and the storage devices. Therefore, the metadata server has no direct knowledge of the I/O operations being done and thus cannot create on its own statistical information about client I/O to optimize the data storage location. ff_iostats4 MAY be used by the client to report I/O statistics back to the metadata server upon returning the layout. Since it is not feasible for the client to report every I/O that used the layout, the client MAY identify "hot" byte ranges for which to report I/O statistics. The definition and/or configuration mechanism of what is considered "hot" and the size of the reported byte range are out of the scope of this document. For client implementations, providing reasonable default values and an optional run-time management interface to control these parameters is suggested. For example, a client can define the default byte range resolution to be 1 MB in size and the thresholds for reporting to be 1 MB/second or 10 I/O operations per second. For each byte range, ffv2is_offset and ffv2is_length represent the starting offset of the range and the range length in bytes. ffv2is_read.ii_count, ffv2is_read.ii_bytes, ffv2is_write.ii_count, and ffv2is_write.ii_bytes represent the number of contiguous READ and WRITE I/Os and the respective aggregate number of bytes transferred within the reported byte range. The combination of ffv2is_deviceid and ffv2lu_addr uniquely identifies both the storage path and the network route to it. Finally, ffv2lu_fhandle allows the metadata server to differentiate between multiple read-only copies of the file on the same storage device.

ffv2_layoutreturn4

ffv2_layoutreturn4 ; /// ffv2_iostats4 ffv2lr_iostats_report<>; /// }; /// ]]> When data file I/O operations fail, ffv2lr_ioerr_report<> is used to report these errors to the metadata server as an array of elements of type ffv2_ioerr4. Each element in the array represents an error that occurred on the data file identified by ffv2ie_errors.de_deviceid. If no errors are to be reported, the size of the ffv2lr_ioerr_report<> array is set to zero. The client MAY also use ffv2lr_iostats_report<> to report a list of I/O statistics as an array of elements of type ff_iostats4. Each element in the array represents statistics for a particular byte range. Byte ranges are not guaranteed to be disjoint and MAY repeat or intersect.

Flexible File Version 2 Layout Type LAYOUTERROR If the client is using NFSv4.2 to communicate with the metadata server, then instead of waiting for a LAYOUTRETURN to send error information to the metadata server (see ), it MAY use LAYOUTERROR (see ) to communicate that information. For the flexible file v2 layout, this means that LAYOUTERROR4args is treated the same as ffv2_ioerr4.

Flexible File Version 2 Layout Type LAYOUTSTATS If the client is using NFSv4.2 to communicate with the metadata server, then instead of waiting for a LAYOUTRETURN to send I/O statistics to the metadata server (see ), it MAY use LAYOUTSTATS (see ) to communicate that information. For the flexible file v2 layout, this means that LAYOUTSTATS4args.lsa_layoutupdate is overloaded with the same contents as in ffv2is_layoutupdate.

Flexible File Version 2 Layout Type Creation Hint The layouthint4 type is defined in the as in .

layouthint4 v1 ; }; ]]> The layouthint4 structure is used by the client to pass a hint about the type of layout it would like created for a particular file. If the loh_type layout type is LAYOUT4_FLEX_FILES, then the loh_body opaque value is defined by the ff_layouthint4 type (v1 compatibility). If the loh_type layout type is LAYOUT4_FLEX_FILES_V2, then the loh_body opaque value is defined by the ffv2_layouthint4 type (see ).

ff_layouthint4

ff_layouthint4 (v1 compatibility) The ff_layouthint4 is retained for backwards compatibility with flexible file v1 layouts. For flexible file v2 layouts, clients SHOULD use ffv2_layouthint4 () instead, which provides encoding type selection and data protection geometry hints via ffv2_data_protection4 ().

Recalling a Layout While discusses reasons independent of layout type for recalling a layout, the flexible file v2 layout type metadata server should recall outstanding layouts in the following cases:

When the file's security policy changes, i.e., ACLs or permission mode bits are set.
When the file's layout changes, rendering outstanding layouts invalid.
When existing layouts are inconsistent with the need to enforce locking constraints.
When existing layouts are inconsistent with the requirements regarding resilvering as described in .

CB_RECALL_ANY The metadata server can use the CB_RECALL_ANY callback operation to notify the client to return some or all of its layouts. defines the allowed types of the "NFSv4 Recallable Object Types Registry".

RCA4 masks for v2

CB_RECALL_ANY4args XDR Typically, CB_RECALL_ANY will be used to recall client state when the server needs to reclaim resources. The craa_type_mask bitmap specifies the type of resources that are recalled, and the craa_layouts_to_keep value specifies how many of the recalled flexible file v2 layouts the client is allowed to keep. The mask flags for the flexible file v2 layout are defined as in .

Recall Mask Flags for v2 The flags represent the iomode of the recalled layouts. In response, the client SHOULD return layouts of the recalled iomode that it needs the least, keeping at most craa_layouts_to_keep flexible file layouts. The PNFS_FF_RCA4_TYPE_MASK_READ flag notifies the client to return layouts of iomode LAYOUTIOMODE4_READ. Similarly, the PNFS_FF_RCA4_TYPE_MASK_RW flag notifies the client to return layouts of iomode LAYOUTIOMODE4_RW. When both mask flags are set, the client is notified to return layouts of either iomode.

Layout Revocation and Fencing In cases where clients are uncommunicative and their lease has expired, or when clients fail to return recalled layouts within a lease period, the metadata server MAY revoke client layouts and reassign these resources to other clients (see ). To avoid data corruption from a revoked client continuing to issue I/O, the metadata server MUST fence the revoked client from the affected data files. The mechanism varies by coupling model and by whether the client's layout stateid has been registered with the data servers via TRUST_STATEID:

Loosely coupled, untrusted stateid:: The metadata server rotates the synthetic uid/gid on the affected data files per . The revoked client presents stale RPC credentials and receives NFS4ERR_ACCESS from the data server. This is the flexible-file-v1-layout-style fencing mechanism; it operates per data file and does not distinguish between clients that hold layouts on the same file.
Tightly coupled, trusted stateid:: When the metadata server has registered a client's layout stateid with the data servers via TRUST_STATEID, it can revoke per-client access without rotating credentials by issuing REVOKE_STATEID to each affected data server, or BULK_REVOKE_STATEID when revoking all stateids belonging to a single clientid4 across a data server. Subsequent I/O from the revoked client carrying the revoked stateid receives NFS4ERR_BAD_STATEID. This is the preferred mechanism for chunked layouts because it is per-client and avoids the flexible file v1 layout's limitation of fencing all clients on a data file when only one needs to be revoked.
Mixed:: A metadata server MAY combine the two mechanisms when a file's layout includes both PASSTHROUGH mirrors (where stateid registration is not in play) and chunked mirrors with trusted stateids. The metadata server rotates synthetic ids for the PASSTHROUGH mirror's data file and issues REVOKE_STATEID for the chunked mirror's data servers.

New NFSv4.2 Error Values

Errors XDR The new error codes are shown in .

Error Definitions Error Definitions

Error	Number	Description
NFS4ERR_ENCODING_NOT_SUPPORTED	10097
NFS4ERR_PAYLOAD_NOT_ATOMIC	10098
NFS4ERR_CHUNK_LOCKED	10099
NFS4ERR_CHUNK_GUARDED	10100
NFS4ERR_PAYLOAD_LOST	10101
NFS4ERR_LAYOUT_CHECKSUM_NOT_SUPPORTED	10102
NFS4ERR_NO_PREDECESSOR	10103
NFS4ERR_NO_ADOPTABLE_LOCK	10104
NFS4ERR_STALE_ESCROW	10105
NFS4ERR_STALE_MDS_EPOCH	10106
NFS4ERR_PARTIAL	10107

NFS4ERR_ENCODING_NOT_SUPPORTED (Error Code 10097) The client requested a ffv2_encoding_type4 which the metadata server does not support. I.e., if the client sends a layout_hint requesting an erasure encoding type that the metadata server does not support, this error code can be returned. The client might have to send the layout_hint several times to determine the overlapping set of supported erasure encoding types.

NFS4ERR_PAYLOAD_NOT_ATOMIC (Error Code 10098) The client encountered a payload in which the blocks were non-atomic and stay non-atomic. As the client can not tell if another client is actively writing, it informs the metadata server of this error via LAYOUTERROR. The metadata server can then arrange for repair of the file.

NFS4ERR_CHUNK_LOCKED (Error Code 10099) The client tried an operation on a chunk which resulted in the data server reporting that the chunk was locked. The client will then inform the metadata server of this error via LAYOUTERROR. The metadata server can then arrange for repair of the file.

NFS4ERR_CHUNK_GUARDED (Error Code 10100) The client tried a guarded CHUNK_WRITE on a chunk which did not match the guard on the chunk in the data file. As such, the CHUNK_WRITE was rejected and the client should refresh the chunk it has cached.

NFS4ERR_PAYLOAD_LOST (Error Code 10101) Returned by a repair actor on the CB_CHUNK_REPAIR response (ccrr_status) to indicate that the identified ranges cannot be repaired and the underlying data is no longer recoverable. Causes include: too few surviving shards to meet the reconstruction threshold (Katz criterion for Mojette, any k-of-(k + m) subset for Reed-Solomon Vandermonde), inability to roll back to a previously committed payload because that payload is also lost, or exhaustion of all FFV2_DS_FLAGS_REPAIR data servers available in the layout with no additional replacement reachable from the metadata server's out-of-band pool. On receipt of NFS4ERR_PAYLOAD_LOST from a repair actor, the metadata server MUST NOT retry the repair by selecting a different client -- the payload is damaged. Subsequent client access to the affected ranges MUST fail: a CHUNK_READ (or an NFSv4 READ against a damaged PASSTHROUGH range) returns NFS4ERR_PAYLOAD_LOST; a CHUNK_WRITE that overwrites a damaged range in its entirety MAY succeed and produce a new atomic generation for that range (the write does not depend on the damaged prior content), but a partial-range write that would require reading the damaged bytes returns NFS4ERR_PAYLOAD_LOST as well. Client NFSv4 implementations MUST surface NFS4ERR_PAYLOAD_LOST to the calling application as the POSIX errno EIO; a client MAY additionally record a more specific diagnostic (e.g., Linux EMEDIUMTYPE) in an out-of-band log or attribute, but the application-visible errno is EIO for portability. The metadata server MAY additionally reflect the damage in an attribute-level indicator or operator-facing notification, but those mechanisms are outside the scope of this specification. NFS4ERR_PAYLOAD_LOST is distinct from NFS4ERR_DELAY (transient; metadata server MAY extend the deadline or re-select) and from NFS4ERR_IO (per-operation failure; metadata server MAY retry or re-select). Only NFS4ERR_PAYLOAD_LOST is terminal.

NFS4ERR_LAYOUT_CHECKSUM_NOT_SUPPORTED (Error Code 10102) Returned by the client on LAYOUTRETURN to indicate that the layout's ffv2m_checksum_algorithm () names a checksum_algorithm4 () that the client does not implement. The client returns the layout with this error code rather than attempting CHUNK operations it cannot validate. On receipt, the metadata server MAY:

issue a new layout for the same file naming a different checksum_algorithm4 that the client supports (if the file's policy permits any of the algorithms the client does support); or
deny the layout request, in which case the client MUST either fall back to metadata-server-mediated I/O or report an I/O error to the application.

NFS4ERR_LAYOUT_CHECKSUM_NOT_SUPPORTED is distinct from NFS4ERR_BADLAYOUT (generic "this layout shape is unusable"): the explicit per-checksum-algorithm signal lets the metadata server discriminate "client can't read this layout because of the checksum algorithm" from "client can't read this layout for some other reason" and respond accordingly.

NFS4ERR_NO_PREDECESSOR (Error Code 10103) Returned on a CHUNK_ROLLBACK against a COMMITTED chunk when the caller names a predecessor generation whose owner-to-index association is no longer recorded by the data server, and no other per-entry error covers the situation. This is the "no restorable predecessor" outcome of the restore case described under "Rollback of COMMITTED Chunks": the predecessor's payload+association pair was released some time earlier under the retention scope rule () -- by lease expiry, by an earlier CHUNK_ROLLBACK delete case, or by any other terminal transition -- and the data server cannot restore what it no longer holds. NFS4ERR_NO_PREDECESSOR is a per-entry status (a caller that names several generations in a single CHUNK_ROLLBACK MAY receive NFS4ERR_NO_PREDECESSOR on some slots and NFS4_OK or other per-entry errors on others). It is a data-plane result, NOT a control-plane failure: the caller reached the data server successfully, was authorized, and the operation evaluated normally -- the named predecessor simply does not exist to restore. NFS4ERR_NO_PREDECESSOR is distinct from NFS4ERR_INVAL (the triple was invalidated by an explicit delete case and cannot be resurrected -- see "Deletion Atomicity and Invalidated Triples") and from NFS4ERR_PAYLOAD_LOST (terminal payload loss reported on CB_CHUNK_REPAIR). The data server distinguishes NFS4ERR_NO_PREDECESSOR from NFS4ERR_INVAL on the basis of what the data server can concretely observe about the presented triple, NOT on historical knowledge the data server may no longer retain:

NFS4ERR_INVAL:

returned in exactly the cases where the data server can concretely recognize the presented triple as invalid:

Structurally invalid:: the triple is malformed or truncated, or names a different file or a different data server.
Within-window release:: a prior explicit CHUNK_ROLLBACK delete case released the association and the current request falls within the same session slot's replay-cache window of that CHUNK_ROLLBACK, so the data server still holds the concrete invalidation context that identifies this triple as released-by-delete-case.

NFS4ERR_NO_PREDECESSOR:

covers every other case in which the data server holds no association for the presented triple, including:

Retention-scope release:: release under the retention scope rule (lease expiry, storage-pressure release).
Post-window release:: the eventual release of a triple invalidated by an earlier delete case whose replay-cache window has since elapsed.
No surviving record:: a well-formed triple for this file and this data server for which the data server holds no surviving record and no concrete invalidation context.

An implementation that does not retain a per-triple record after release cannot distinguish "the data server never recorded this triple" from "the data server recorded and later released this triple" outside a live invalidation context, and this specification does not require it to: both cases resolve to NFS4ERR_NO_PREDECESSOR under the "no surviving record" case above. The two errors are not interchangeable: NFS4ERR_INVAL is a caller-side signal that the client MUST NOT retry the same identity, while NFS4ERR_NO_PREDECESSOR is a data-plane signal that the caller MAY fall back to best-effort reconstruction via CHUNK_WRITE_REPAIR. A client that receives NFS4ERR_NO_PREDECESSOR MAY fall back to reconstructing authoritative bytes from surviving shards and writing them via CHUNK_WRITE_REPAIR under a new owner triple; that fallback is best-effort and MAY itself terminate at NFS4ERR_PAYLOAD_LOST if no authoritative source exists. A client that requires the restored generation to retain the predecessor's original owner triple in cases where the retention scope () would otherwise permit release MUST use the metadata-server escrow control plane ( through ), which pins the predecessor's payload and its owner-to-index association jointly against that release rule for as long as the escrow-lock or a client-owned lock adopted from it remains in continuous custody (see ).

NFS4ERR_NO_ADOPTABLE_LOCK (Error Code 10104) Returned by CHUNK_LOCK when a repair actor attempts to adopt a metadata-server escrow lock () and no such adoption is possible on the data server for the requested range. There are four state-level causes:

no metadata-server escrow lock is installed on the data server for the requested range;
a metadata-server escrow lock is installed but its escrow_id4 () does not match the identity the repair actor presents;
the range's escrow lock is currently under a reconciliation hold following a metadata-server incarnation change (see ) and cannot be adopted until the hold clears; or
the escrow was already adopted by a different repair actor whose adoption remains active.

Authorization-level failures -- the caller is not the metadata-server-designated repair actor for this escrow, or the caller lacks credentials for the escrow's scope -- surface as NFS4ERR_ACCESS rather than NFS4ERR_NO_ADOPTABLE_LOCK, so that no data-plane information about the current adopter is leaked to an unauthorized caller. On receipt of NFS4ERR_NO_ADOPTABLE_LOCK, the repair client MUST report the outcome to the metadata server via CB_CHUNK_REPAIR's per-range status array () and MUST NOT unilaterally acquire a fresh CHUNK_LOCK, retry the adoption, or invoke the discovery/fallback path of NFS4ERR_NO_PREDECESSOR () -- the four causes above are all control-plane conditions the metadata server is best placed to resolve (by reissuing the escrow, waiting for the reconciliation hold to clear, or abandoning the repair).

NFS4ERR_STALE_ESCROW (Error Code 10105) Returned by CHUNK_ESCROW_RELEASE when the escrow_id4 the metadata server presents does not match any escrow currently installed on the data server for the referenced range. Two causes lead to the same wire outcome:

no escrow covers the referenced range at the data server (whether never installed, released earlier, or consumed by a repair adoption); or
an escrow covers the range but its recorded escrow_id4 differs from the one presented (the presented identity is a stale reference to an earlier installation).

The data server MUST NOT alter any current lock or escrow state as a side effect of returning NFS4ERR_STALE_ESCROW: the response reports "the identity you presented is not what I hold" without changing what is held. In particular, if the range carries a metadata-server escrow lock whose escrow_id4 differs from the presented identity, that lock survives the call unchanged; and if the range carries a client-owned lock that was previously adopted from an escrow whose identity matches, that adopted lock is not affected by a release of the presented (now-stale) identity.

NFS4ERR_STALE_MDS_EPOCH (Error Code 10106) Returned by any of the CHUNK_ESCROW operations (, , ) when the requesting metadata server presents an epoch value the data server no longer accepts because a newer metadata-server incarnation has completed a CHUNK_ESCROW_TAKEOVER. The metadata server that receives NFS4ERR_STALE_MDS_EPOCH has been fenced from continued escrow operations on this data server; it MUST NOT retry the operation under the same epoch and MUST obtain a fresh incarnation-lease token and reissue via CHUNK_ESCROW_TAKEOVER before resuming. CHUNK_ESCROW_TAKEOVER itself is not subject to this rejection -- it is the recovery path out of an expired epoch and carries its own compare-and-advance semantics per . NFS4ERR_STALE_MDS_EPOCH is distinct from NFS4ERR_ACCESS (credential-level failure) and from NFS4ERR_STALE_ESCROW (identity mismatch on a specific escrow): the stale-epoch response fences the presenter's entire escrow-control session, not a single per-escrow operation.

NFS4ERR_PARTIAL (Error Code 10107) Returned as the top-level status of a CB_CHUNK_REPAIR response () when the repair actor evaluated every named range but at least one range's per-range status is not NFS4_OK. The per-range status array (ccrr_range_status) is authoritative for the outcome of each range; NFS4ERR_PARTIAL is the top-level signal that the metadata server MUST consume the per-range array rather than treating the response as uniformly successful or uniformly failed. NFS4ERR_PARTIAL is distinct from operation-wide errors (decode failures, authorization failures, session state failures) that fail every range at once: those return the operation-wide error at the top level with an EMPTY per-range array. NFS4ERR_PARTIAL requires a co-indexed array with one entry per named range and MAY carry any mix of NFS4_OK and per-range failure codes.

Operations and Their Valid Errors The operations and their valid errors are presented in . All error codes not defined in this document are defined in and . Operations and Their Valid Errors

Operation	Errors
CHUNK_COMMIT	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_NOTSUPP, NFS4ERR_SERVERFAULT, NFS4ERR_STALE
CHUNK_ERROR	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_SERVERFAULT
CHUNK_FINALIZE	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_NOTSUPP, NFS4ERR_SERVERFAULT, NFS4ERR_STALE
CHUNK_HEADER_READ	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_NOTSUPP, NFS4ERR_REP_TOO_BIG, NFS4ERR_SERVERFAULT, NFS4ERR_STALE
CHUNK_LOCK	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_CHUNK_LOCKED, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_NO_ADOPTABLE_LOCK, NFS4ERR_SERVERFAULT
CHUNK_READ	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, NFS4ERR_IO, NFS4ERR_NOTSUPP, NFS4ERR_PAYLOAD_NOT_ATOMIC, NFS4ERR_SERVERFAULT, NFS4ERR_STALE
CHUNK_REPAIRED	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_SERVERFAULT
CHUNK_ROLLBACK	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_NO_PREDECESSOR, NFS4ERR_SERVERFAULT
CHUNK_UNLOCK	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_SERVERFAULT
CHUNK_WRITE	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_CHUNK_GUARDED, NFS4ERR_CHUNK_LOCKED, NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, NFS4ERR_IO, NFS4ERR_NOSPC, NFS4ERR_NOTSUPP, NFS4ERR_SERVERFAULT, NFS4ERR_STALE
CHUNK_WRITE_REPAIR	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_DELAY, NFS4ERR_FHEXPIRED, NFS4ERR_IO, NFS4ERR_NOSPC, NFS4ERR_NOTSUPP, NFS4ERR_SERVERFAULT, NFS4ERR_STALE
TRUST_STATEID	NFS4_OK, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_DELAY, NFS4ERR_INVAL, NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTSUPP, NFS4ERR_PERM, NFS4ERR_SERVERFAULT
REVOKE_STATEID	NFS4_OK, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_DELAY, NFS4ERR_INVAL, NFS4ERR_NOFILEHANDLE, NFS4ERR_NOTSUPP, NFS4ERR_PERM, NFS4ERR_SERVERFAULT
BULK_REVOKE_STATEID	NFS4_OK, NFS4ERR_BADXDR, NFS4ERR_DELAY, NFS4ERR_NOTSUPP, NFS4ERR_PERM, NFS4ERR_SERVERFAULT
CHUNK_ESCROW_INSTALL	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_CHUNK_LOCKED, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_PERM, NFS4ERR_SERVERFAULT, NFS4ERR_STALE_MDS_EPOCH
CHUNK_ESCROW_RELEASE	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_PERM, NFS4ERR_SERVERFAULT, NFS4ERR_STALE_ESCROW, NFS4ERR_STALE_MDS_EPOCH
CHUNK_ESCROW_ENUMERATE	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_PERM, NFS4ERR_SERVERFAULT, NFS4ERR_STALE_MDS_EPOCH
CHUNK_ESCROW_TAKEOVER	NFS4_OK, NFS4ERR_ACCESS, NFS4ERR_BADXDR, NFS4ERR_INVAL, NFS4ERR_NOTSUPP, NFS4ERR_PERM, NFS4ERR_SERVERFAULT, NFS4ERR_STALE_MDS_EPOCH

Callback Operations and Their Valid Errors The callback operations and their valid errors are presented in . All error codes not defined in this document are defined in and . Callback Operations and Their Valid Errors

Callback Operation	Errors
CB_CHUNK_REPAIR	NFS4_OK, NFS4ERR_BADXDR, NFS4ERR_BAD_STATEID, NFS4ERR_DEADSESSION, NFS4ERR_DELAY, NFS4ERR_ENCODING_NOT_SUPPORTED, NFS4ERR_INVAL, NFS4ERR_IO, NFS4ERR_ISDIR, NFS4ERR_LOCKED, NFS4ERR_NOTSUPP, NFS4ERR_OLD_STATEID, NFS4ERR_PARTIAL, NFS4ERR_PAYLOAD_LOST, NFS4ERR_SERVERFAULT, NFS4ERR_STALE

Errors and the Operations That Use Them The operations and their valid errors are presented in . All operations not defined in this document are defined in and . Errors and the Operations That Use Them

Error	Operations
NFS4ERR_ENCODING_NOT_SUPPORTED	CB_CHUNK_REPAIR, LAYOUTGET
NFS4ERR_PAYLOAD_NOT_ATOMIC	CHUNK_READ
NFS4ERR_CHUNK_LOCKED	CHUNK_LOCK, CHUNK_WRITE, CHUNK_ESCROW_INSTALL
NFS4ERR_CHUNK_GUARDED	CHUNK_WRITE
NFS4ERR_PAYLOAD_LOST	CB_CHUNK_REPAIR
NFS4ERR_LAYOUT_CHECKSUM_NOT_SUPPORTED	LAYOUTGET
NFS4ERR_NO_PREDECESSOR	CHUNK_ROLLBACK
NFS4ERR_NO_ADOPTABLE_LOCK	CHUNK_LOCK
NFS4ERR_STALE_ESCROW	CHUNK_ESCROW_RELEASE
NFS4ERR_STALE_MDS_EPOCH	CHUNK_ESCROW_INSTALL, CHUNK_ESCROW_RELEASE, CHUNK_ESCROW_ENUMERATE, CHUNK_ESCROW_TAKEOVER
NFS4ERR_PARTIAL	CB_CHUNK_REPAIR

EXCHGID4_FLAG_USE_ERASURE_DS

The EXCHGID4_FLAG_USE_ERASURE_DS When a data server connects to a metadata server it can via EXCHANGE_ID (see ) state its pNFS role. The data server can use EXCHGID4_FLAG_USE_ERASURE_DS (see ) to indicate that it supports the new NFSv4.2 operations introduced in this document. describes the interaction of the various pNFS roles masked by EXCHGID4_FLAG_MASK_PNFS. However, that does not mask out EXCHGID4_FLAG_USE_ERASURE_DS. I.e., EXCHGID4_FLAG_USE_ERASURE_DS can be used in combination with all of the pNFS flags. If the data server sets EXCHGID4_FLAG_USE_ERASURE_DS during the EXCHANGE_ID operation, then it MUST support all of the operations in . Further, this support is orthogonal to the Erasure Encoding Type selected. The data server is unaware of which type is driving the I/O.

New NFSv4.2 Attributes

Attribute 89: fattr4_coding_block_size

XDR for fattr4_coding_block_size The new attribute fattr4_coding_block_size (see ) is an OPTIONAL to NFSv4.2 attribute which MUST be supported if the metadata server supports the Flexible File Version 2 Layout Type. By querying it, the client can determine the data block size it is to use when coding the data blocks to chunks.

Attribute 90: fattr4_chunked_data_file

XDR for fattr4_chunked_data_file The new attribute fattr4_chunked_data_file (see ) is an OPTIONAL to NFSv4.2 attribute a data server uses to classify a data file as a chunked-encoding data file for the purpose of enforcing the client restrictions in . When set to TRUE, the file is under flexible file v2 layout chunked-encoding management by a metadata server; the data server MUST apply the "MUST reject" rules on client operations against such files (see ). When FALSE or absent, no such enforcement is triggered by this attribute alone. Only the metadata server sets this attribute; the metadata server sets it as part of allocating a chunked-encoding data file on the data server. Clients MUST NOT SETATTR this attribute; a data server MUST reject a client SETATTR of FATTR4_CHUNKED_DATA_FILE with NFS4ERR_INVAL. A data server that does not support this attribute falls back to the other identification methods described in . The attribute describes the format of the file's content rather than a property that can be applied to content already written: when TRUE, the file holds chunks; when FALSE, it holds the data as the writer supplied it. Changing the attribute on a file that is not empty would reinterpret existing content under a format it was not written in, in either direction, and the resulting misinterpretation is not detectable from the attribute alone. The value is therefore fixed once the file holds data. The metadata server MUST NOT change the attribute on a file that is not empty, and a data server MUST reject a SETATTR of FATTR4_CHUNKED_DATA_FILE against a file that is not empty with NFS4ERR_INVAL, whether the request arrives from a client or over the metadata server's control session. A metadata server that allocates a data file before it has selected the encoding for that file MAY set the attribute to its final value at any point while the file remains empty. How a data server determines that a file is not empty is an implementation matter.

New NFSv4.2 Common Data Structures

chunk_cohort_id4

XDR for chunk_cohort_id4 The chunk_cohort_id4 is a 64-bit writer-chosen opaque identifier that names a single write transaction (a "cohort" of chunks written together in one CHUNK_WRITE batch, or across several CHUNK_WRITEs that a single writer chooses to associate as one logical transaction). Under the shared-cohort model (), every chunk in a batch carries the same chunk_cohort_id4 so that lifecycle operations (CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK) can name the transaction as a whole. The data server treats chunk_cohort_id4 opaquely: it does NOT interpret the value beyond equality comparison, and it is NOT required to be monotonic, dense, or ordered in any way. Writers MUST choose a new chunk_cohort_id4 per distinct write transaction so that transactions can be distinguished on the data server; the uniqueness scope required is per-(writer, file), not global. A writer with sufficient state MAY generate chunk_cohort_id4 values from a counter, a random 64-bit source, a hashed transaction identifier, or any other locally convenient scheme, subject to the per-writer uniqueness rule. On the wire, a single CHUNK_WRITE carries a 12-byte cohort header (chunk_cohort_id4 + writer's cg_client_id) once for the batch, followed by, per chunk in the batch, the tagged checksum4 and the opaque chunk payload, as shown in . The payload length is carried separately in the CHUNK_WRITE4args cwa_chunks<> slot; the diagram shows the header once and the per-chunk framing that repeats.

CHUNK_WRITE wire framing: cohort header + per-chunk chunks and are not part of this framing. ]]>

chunk_guard4

XDR for chunk_guard4 The chunk_guard4 (see ) is the per-chunk compare-and-swap (CAS) state used by CHUNK_WRITE to detect concurrent updates. It is state that the data server MAINTAINS per chunk; the writer transaction identity (the "cohort") is carried separately by chunk_cohort_id4 () and is NOT part of chunk_guard4. chunk_guard4 has two fields:

cg_gen_id:

A per-chunk monotonic generation counter, tracked by the data server. Each chunk's gen_id starts at 0 when the chunk is first written and is incremented on each successful CHUNK_WRITE by any client. cg_gen_id is NOT a timestamp -- the protocol does not rely on a global clock, and no interpretation of cg_gen_id as a wall-clock value is supported. cg_gen_id values are NOT comparable across distinct chunks; a given cg_gen_id is only meaningful within the scope of a single chunk on a single file. cg_gen_id is NOT a transaction identifier and MUST NOT be interpreted as naming or ordering write transactions: transaction identity is chunk_cohort_id4 ().

cg_client_id:

A 32-bit value established by the metadata server at the time the client's layout is granted (see and ffv2m_client_id). The metadata server MUST assign distinct cg_client_id values to distinct clients that hold concurrent write layouts on the same file. cg_client_id is opaque with respect to client identity -- a data server MUST NOT interpret its bits as naming or ordering clients in any external sense. The value supports two operations only: equality comparison (to detect whether the current writer is the client that last wrote a chunk) and numeric comparison (to implement the tiebreaker rule below). The same cg_client_id value appears in the cohort record () as co_client_id; the two are redundant carriers of the same layout-granted writer identity so that both CAS state and cohort records remain self-contained.

Uniqueness contract:

The chunk_guard4 pair (cg_gen_id, cg_client_id) identifies the MOST RECENT successful writer of a chunk plus that chunk's generation counter; it does NOT identify a specific write transaction (that role is filled by chunk_cohort_id4). Neither field alone is globally unique; two clients MAY observe the same cg_gen_id on distinct chunks (each chunk's counter is independent), and the cg_client_id is what makes concurrent writers distinguishable at the CAS layer.

Observability:

The data server exposes the chunk's current chunk_guard4 on the read path so that a client preparing a CAS update in multiple writer mode () can obtain the expected prior value without racing. Two operations return it: CHUNK_READ carries cr_guard in each read_chunk4 (), and CHUNK_HEADER_READ returns a chrr_guards<> array co-indexed with the other per-chunk arrays (). A client places the observed pair into cwa_guard.cwg_guard and sets cwa_guard.cwg_check = TRUE on the subsequent CHUNK_WRITE; the data server compares against its currently-held chunk_guard4 and returns NFS4ERR_CHUNK_GUARDED on mismatch.

Deterministic contention resolution for concurrent writers:

When two or more clients race on the same chunk in the multi-writer mode, the losing writer -- the one whose CAS fails at any affected data server -- MUST roll back the chunks it already wrote for the affected transaction and retry under a fresh cohort id. Convergence is achieved by making losing writers back off, not by having the data server pick a winner from the losing writer's state. This is the wire-level mechanism that implements the consistency invariant in ("one writer's entire chunk-generation wins and becomes COMMITTED; the other writer sees NFS4ERR_CHUNK_GUARDED and is expected to re-read and retry"); documents the progress guarantee this rule underwrites.

At CHUNK_WRITE (per data server, arrival-order):: the data server accepts the first CHUNK_WRITE whose chunk_guard4 CAS check succeeds against the chunk's current chunk_guard4 value. Later writers whose CAS fails receive NFS4ERR_CHUNK_GUARDED for that chunk. Because arrival order can differ between data servers, different subsets of the mirror set MAY initially accept different clients' writes for different chunks; that is transient divergence, resolved by the client-driven rollback below.
At NFS4ERR_CHUNK_GUARDED (client-driven rollback and retry):: a client that observes NFS4ERR_CHUNK_GUARDED on any chunk of an in-flight transaction MUST treat the entire transaction as lost. It MUST issue CHUNK_ROLLBACK against every data server that accepted a CHUNK_WRITE under this transaction's cohort pair, so that the data server can release the PENDING state and revert cg_gen_id to the value it held before the losing writer's CAS succeeded. The client then re-reads the chunks with CHUNK_READ, applies its intended change to the as-observed data, chooses a fresh cwa_cohort_id, and re-issues CHUNK_WRITE with cwa_guard.cwg_check = TRUE supplying the refreshed prior chunk_guard4.
At CHUNK_FINALIZE (single winner already established):: by the time the mirror set converges, only one writer's cohort remains as the PENDING state for each affected chunk (all others have rolled back). CHUNK_FINALIZE against that PENDING state succeeds; there is no tiebreaker comparison against the caller's co_client_id at FINALIZE. A caller that attempts CHUNK_FINALIZE against a chunk whose current PENDING state carries a cohort pair not matching the caller's transaction MUST receive NFS4ERR_CHUNK_GUARDED and MUST proceed with the rollback-and-retry flow above. A caller whose own PENDING state was overwritten by a different writer SHOULD also receive NFS4ERR_CHUNK_GUARDED (the caller no longer has a PENDING chunk to finalize).

A client that has rolled back and retried but continues to observe NFS4ERR_CHUNK_GUARDED without forward progress after a bounded number of retries MUST escalate via LAYOUTERROR and the repair coordination flow in . Bounded retry is a client-implementation matter; the protocol does not mandate a specific count, only that unbounded retry is forbidden so that a contentious workload cannot livelock without surfacing the contention to the metadata server. Client cg_client_id ordering is NOT used to pick a winner: the numeric ordering of cg_client_id values is arbitrary with respect to clients' external identities and MUST NOT be interpreted as a preference ordering over the clients themselves. The arrival-order arbitration at each data server, combined with mandatory client rollback on CAS failure, is what makes the mirror set converge; the metadata server MAY still arrange cg_client_id assignment to influence diagnosis but MUST NOT rely on numeric ordering for correctness.

Metadata-Server Assignment Rules for cg_client_id To uphold the uniqueness contract, the metadata server MUST follow these rules when assigning cg_client_id (that is, when populating ffv2m_client_id at layout-grant time):

Two clients holding concurrent write layouts on the same file MUST receive distinct cg_client_id values. A client that holds only a read layout need not be assigned a distinct value.
The reserved sentinel CHUNK_GUARD_CLIENT_ID_NONE (0x00000000) MUST NOT be assigned to any client. Reserving 0 prevents an uninitialized cg_client_id field from passing as a real client and ensures the deterministic tiebreaker (numerically lowest wins) does not encode an implicit priority via assignment of 0.
The reserved sentinel CHUNK_GUARD_CLIENT_ID_MDS (0xFFFFFFFF) MUST NOT be assigned to any client.
A cg_client_id MAY be reused by the metadata server after the prior holder's layout has been fully returned (via LAYOUTRETURN or revocation). The metadata server SHOULD avoid reusing a cg_client_id within a single lease period to simplify diagnosis of stale writes.
cg_client_id values do not persist across metadata-server restart. Clients reclaiming layouts during the grace period receive freshly assigned values; the protocol does not rely on any pre-restart assignment surviving.

Data-Server Collision Handling A (cg_gen_id, cg_client_id) pair that the uniqueness contract would otherwise render unique can nonetheless collide if a client and the metadata server disagree about which cg_client_id the client currently holds, or if a client presents a spoofed cg_client_id. The data server enforces the contract locally:

If the data server receives a CHUNK_WRITE whose chunk_guard4 has the same (cg_gen_id, cg_client_id) as a chunk already in PENDING, FINALIZED, or COMMITTED state AND the presented payload differs from the retained payload, the data server MUST reject the write with NFS4ERR_CHUNK_GUARDED and SHOULD report the collision to the metadata server via LAYOUTERROR. This situation is a protocol violation on one side of the conversation; the metadata server resolves it by revoking the offending client's layout and selecting a repair actor under .
If a client presents CHUNK_GUARD_CLIENT_ID_MDS as cg_client_id in any client-originated operation, the data server MUST reject the operation with NFS4ERR_INVAL (see ).
A cg_client_id that does not match the tsa_client_id recorded for the layout stateid under which the CAS operation is issued MUST be rejected with NFS4ERR_BAD_STATEID. This (stateid, client_id) binding is registered by the metadata server via TRUST_STATEID; an unmatched cg_client_id is treated as a stale-layout condition, see .

Reserved cg_client_id Value: CHUNK_GUARD_CLIENT_ID_NONE The value CHUNK_GUARD_CLIENT_ID_NONE (0x00000000) is reserved. It does not denote any client. Reserving 0 prevents an uninitialized cg_client_id field from passing as a real client and ensures the deterministic tiebreaker (numerically lowest wins, see ) does not encode an implicit priority via assignment of 0. Clients MUST NOT present CHUNK_GUARD_CLIENT_ID_NONE as the cg_client_id of any client-originated chunk_guard4 or chunk_owner4. A data server that receives such a value from a client MUST reject the operation with NFS4ERR_INVAL.

CHUNK_GUARD_CLIENT_ID_MDS and the Metadata-Server Escrow Lock The value CHUNK_GUARD_CLIENT_ID_MDS (0xFFFFFFFF) is reserved. It denotes that the chunk lock is held by the metadata server itself, in escrow during a repair coordination sequence (see ). The data server produces a chunk_guard4 with this cg_client_id when the metadata server revokes the prior holder's stateid while that holder still holds chunk locks; the locks MUST NOT be dropped and are transferred to the metadata-server escrow owner instead. The metadata server does not originate CHUNK_LOCK or CHUNK_WRITE traffic on its own session. Clients MUST NOT present CHUNK_GUARD_CLIENT_ID_MDS as the cg_client_id of any client-originated chunk_guard4 or chunk_owner4. A data server that receives such a value from a client MUST reject the operation with NFS4ERR_INVAL. The metadata-server escrow owner is released only by a CHUNK_LOCK from the client selected via CB_CHUNK_REPAIR, carrying CHUNK_LOCK_FLAGS_ADOPT. See . Each metadata-server escrow lock carries an escrow_id4 () the metadata server chose at CHUNK_ESCROW_INSTALL () time and presents on every subsequent operation that refers to that lock (CHUNK_ESCROW_RELEASE, CB_CHUNK_REPAIR, CHUNK_LOCK adoption via cla_adopt). When a repair client adopts the metadata-server escrow lock (CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT), the data server MUST retain the adopted escrow_id4 as durable custody metadata on the resulting client-owned lock, for as long as a subsequent revocation-transfer can occur that would convert the client-owned lock back to a metadata-server escrow lock (for example, when the client's lease later expires while the lock is still held). If such a revocation-transfer occurs, the resulting new metadata-server escrow lock MUST be installed with the SAME escrow_id4 that was preserved as custody metadata, not a fresh identity: this preserves the full durable key (file, escrow_id, data-server set) across the entire custody chain so that a metadata server's tuple records can re-associate with the reappeared escrow across an incarnation change (see discovery).

chunk_owner4

XDR for chunk_owner4 The chunk_owner4 (see ) is the "cohort record" for a chunk: it identifies the write transaction that produced the chunk and which of that transaction's chunks the record refers to. It is separate from chunk_guard4 (), which is the per-chunk CAS state. The three fields together form the full owner triple. co_cohort_id is the chunk_cohort_id4 shared by every chunk written together in a single logical transaction (see and the discussion of chunk_cohort_id4 above). Two chunks are members of the same write transaction iff their (co_cohort_id, co_client_id) pairs are equal. Lifecycle operations (CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK) name transactions by co_cohort_id + co_client_id. co_client_id is the writer's layout-granted 32-bit identity, the same value that appears as cg_client_id in the chunk_guard4 CAS state (see ). It supports equality comparison (matching cohort records to lifecycle operations) and numeric comparison (implementing the multi-writer tiebreaker described in ). co_id is a writer-supplied opaque per-chunk identifier the client chooses at CHUNK_WRITE time (see ). It is NOT required to equal the chunk's file index; the client MAY choose any uint32_t value, subject only to the uniqueness constraint that within a single (co_cohort_id, co_client_id) all co_id values MUST be distinct so that a subsequent lifecycle operation can name individual chunks unambiguously. The data server treats co_id opaquely; it does NOT interpret the value beyond equality comparison against the co_id values it previously accepted from the client for this cohort. Writers that would otherwise choose file-index values MAY do so, but the wire semantics do not privilege that choice. Each distinct chunk-write transaction from a given client MUST carry a unique co_cohort_id, so lifecycle operations can be correlated with the transactions that produced them across all data files. The uniqueness scope required is per-(co_client_id, file); different clients MAY independently pick colliding co_cohort_id values because co_client_id disambiguates them.

escrow_id4 For the design rationale of the escrow control plane -- what problem it solves and how the components compose -- see . This section defines the identifier only.

XDR for escrow_id4 The escrow_id4 is a 128-bit opaque identifier the metadata server chooses for each metadata-server escrow lock () it installs on a data server. It is distinct from the chunk_owner4 owner triple () that identifies a chunk generation: the escrow identity refers to the custody of a repair-scope lock, while the owner triple refers to a specific generation's bytes. The two namespaces do not overlap and are compared only within their own contexts (an escrow_id4 is never matched against a chunk_owner4 field). The escrow identity is threaded through the metadata-server operations that manage escrow custody: it appears in each escrow-family operation this specification defines (CHUNK_ESCROW_INSTALL / CHUNK_ESCROW_RELEASE / CHUNK_ESCROW_ENUMERATE / CHUNK_ESCROW_TAKEOVER), in the callback that carries the repair instruction (CB_CHUNK_REPAIR), and in the CHUNK_LOCK argument variant that adopts an escrow lock (see ). A conforming metadata server MUST choose escrow_id4 values that are unique across every escrow it has ever installed on any data server that might still recognize a prior installation -- sufficient uniqueness is provided by a 128-bit identifier drawn from a metadata-server-incarnation prefix and a per-incarnation monotonic counter, or by any mechanism whose collision probability is negligible across the lifetime of the deployment. The metadata server MUST NOT reuse an escrow_id4 whose lifecycle is not provably complete. The data server treats escrow_id4 opaquely: it compares two escrow_id4 values for equality when matching a CHUNK_LOCK adoption against an installed escrow, or when reconciling an escrow tuple against a discovery response, and does not interpret the internal structure.

chunk_state_flags4

XDR for chunk_state_flags4 The chunk_state_flags4 is a 32-bit bitmask carried in per-chunk result arrays (chrr_locked in CHUNK_HEADER_READ4resok, cr_locked in read_chunk4) to report the read-time state of a chunk on the data server. The value 0 means "no state flags set"; each defined bit indicates a specific state property. Unassigned bits are reserved for future revisions of this specification; receivers MUST ignore bits they do not recognize so that additions are additive on the wire.

CHUNK_STATE_FLAGS_LOCKED:: the chunk currently has a CHUNK_LOCK held by some chunk_owner4. This state is reported orthogonally to the per-chunk chrr_status / cr_status so that a locked chunk still surfaces its lifecycle state and chunk_owner4 to the inspector.

Incarnation-Lease Proof CHUNK_ESCROW_TAKEOVER accepts a bounded opaque proof payload identified by a proof profile identifier. The proof is not the metadata server's own machine credential (which would only prove the role, not the current exclusive incarnation): it is an assertion issued by a single writer authority external to the metadata server (a high-availability manager, a cluster-consensus service, or an operator-mediated recovery workflow) that only one metadata-server instance currently holds the incarnation lease. The data server verifies the proof before it will compare-and-advance its recorded metadata-server epoch to the value the takeover names.

XDR for proof_profile_id4

Mandatory-to-Implement Profile A conforming implementation MUST support at least one proof profile so that two independent implementations can verify each other's takeover proofs at the wire level without prior out-of-band negotiation. This specification designates the following profile as mandatory-to-implement. The mandatory profile carries a signed token whose signer is the incarnation-lease authority (NOT the metadata server itself). The envelope is a COSE_Sign1 structure (Section 4.2 of ) over a deterministic CBOR payload (Section 4.2 of ) with the following normative choices:

Signature algorithm (mandatory-to-implement):: Ed25519 (algorithm identifier -8 per ). A conforming signer MUST use Ed25519; a conforming verifier MUST accept Ed25519. Other signature algorithms (e.g., ECDSA-P256 with identifier -7, RSASSA-PSS-SHA256 with identifier -37) MAY be registered as additional profiles per the IANA Considerations.
kid header parameter:: OPTIONAL. A deployment with a single trust anchor MAY omit it; a deployment supporting key rotation or multiple trust anchors SHOULD include it so a data server can select the correct verification key. Absent kid, the data server attempts verification against each configured trust anchor and accepts on the first match.
Payload map fields:: the signed CBOR payload is a map with integer-keyed fields (per convention for compact wire size). The mandatory-profile keys are: - 1 = principal (CBOR text string): the metadata-server principal that holds this incarnation. Comparison is byte-for-byte against the RPCSEC_GSS authenticated name of the presenter (Section 5 of ), with no Unicode normalization or case folding; a deployment MUST provision the authority to sign tokens whose principal field is exactly the RPC-authenticated name the data server will observe. - 2 = epoch (CBOR uint): the metadata-server epoch value being claimed. On presentation in CHUNK_ESCROW_TAKEOVER the data server MUST verify that this field equals ceta_new_epoch in the operation arguments; any mismatch is a signature-and-payload check failure at step 4 of "Presentation and Verification" below and returns NFS4ERR_ACCESS. - 3 = scope (CBOR text string): identifier of the data server or data-server set the token is valid for. Comparison is byte-for-byte against the data-server-side value the deployment provisions at trust-anchor setup (see "Trust Anchor Provisioning" below); a data server accepts a token whose scope field exactly matches any scope identifier it has been provisioned to serve, and rejects any other value. The scope namespace is deployment-local -- this specification neither defines a format nor constrains the character set beyond requiring UTF-8 CBOR text. - 4 = issued_at (CBOR tag-1 epoch-based date/time per Section 3.4.2): the instant the authority signed the token. A conforming issuer MUST encode this as a CBOR unsigned integer number of seconds since the POSIX epoch; a verifier MUST reject a token whose issued_at is not an unsigned integer under tag 1 (fractional or negative values are rejected). - 5 = expires_at (CBOR tag-1 epoch-based date/time per Section 3.4.2): the instant the token ceases to be admissible. Same encoding constraints as issued_at. - 6 = token_id (CBOR byte string, 16 bytes): a nonce for replay detection.

Presentation and Verification The metadata server presents the profile identifier and the proof bytes together in the takeover arguments (see ). The data server MUST evaluate the takeover in a fixed order so that no failure discloses state that a prior check would have denied:

session replay-cache lookup: retransmission of a prior request in the current session slot returns the cached response;
presenter authorization: RPCSEC_GSS presenter authentication and credential check (NFS4ERR_ACCESS if the caller lacks metadata-server role);
profile support: unknown proof_profile_id4 returns NFS4ERR_NOTSUPP;
proof verification: the profile's signature and payload checks are applied (NFS4ERR_ACCESS on any failure -- bad signature, mismatched principal, mismatched epoch, mismatched scope, token past expires_at, or token_id already in the data server's replay cache);
epoch compare-and-advance: NFS4ERR_STALE_MDS_EPOCH on mismatch, otherwise the epoch and epoch_expires_at are advanced atomically per .

The strict ordering ensures an unauthenticated caller learns nothing about which profiles the data server supports or which epoch it currently holds. Replay-cache scoping (both layers): the NFSv4.1 session replay cache in step 1 is scoped to the session slot and its lifetime is bounded by session liveness. The token_id replay cache in step 4 is scoped to the (proof_profile_id4, verified issuer) pair and MUST be sized and expired coherently with the token's own expires_at, so that a valid token cannot be replayed after its natural expiry and a recently-observed token cannot be inadvertently retired while still admissible. A data server MAY persist the token_id replay cache across restart. Loss of a cache entry (eviction, non-persisted restart) does not permanently defeat lost-response recovery: the byte-identical uncertain-completion recovery path in covers the cache-miss form and returns the same postcondition-equivalent NFS4_OK when the operation had already completed.

Uncertain-Completion Recovery for TAKEOVER CHUNK_ESCROW_TAKEOVER is the recovery path a metadata server uses after an incarnation change; the compare-and-advance semantics make the successful case observable to the data server, but the metadata server MAY lose the response to a successful TAKEOVER (network drop, RPC retransmission timeout, session loss). The bare strict-ordering rules above would rebuff a byte-identical reissue: either the token_id replay-cache check at step 4 rejects with NFS4ERR_ACCESS (the cache still holds the entry) or the epoch compare-and-advance at step 5 rejects with NFS4ERR_STALE_MDS_EPOCH (the ordinary path, because the reissue's ceta_expected_prior_epoch names the pre-advance epoch that the earlier successful TAKEOVER has already moved past). In either case the metadata server has no way to distinguish "the prior TAKEOVER succeeded and the response was lost" from "the proof is invalid." To close that recovery gap, a data server MUST accept a byte-identical CHUNK_ESCROW_TAKEOVER reissue as postcondition-equivalent success when step 4's signature and payload checks succeed (signature verifies against the deployment-provisioned trust anchor; the signed principal, scope, and epoch match; the token is within its expires_at window; all per "Payload map fields"), with the ordinary step-4 "token_id already in the replay cache" rejection overridden as stated below, and both of the following hold:

the reissue's ceta_new_epoch equals the data server's currently-recorded metadata-server epoch (the takeover the token authorized has already completed); and
the reissue's ceta_expected_prior_epoch is strictly less than the reissue's ceta_new_epoch (the token names a genuine advance, not a no-op).

The recovery rule takes two forms distinguished by the state of the token_id replay cache at the data server:

Cache-hit form (ordinary lost-response case):: the token_id is present in the token_id replay cache. The data server recognizes the cached byte-identical decision and returns the cached NFS4_OK result; step 5 is not re-executed. Cache-miss form (applies after eviction or non-persisted restart):
: the token_id is not present in the replay cache. The data server treats the presentation as a fresh byte-identical proof under the two epoch predicates above and does NOT execute step 5 (the epoch is already at the post-advance state).

Under these predicates the data server returns NFS4_OK without side effect: the epoch and epoch_expires_at are already at the post-advance state, no state changes, and the second observation is idempotent. The rule is safe against a fresh presentation of an already-used token by a different party -- a different party would not present the same proof bytes without stealing the signer's key material, the token was issued to a specific principal matched by step 4's principal check, expiry admissibility windows for successor principals do not overlap ( "Time-Related Bounds"), and a subsequent successful TAKEOVER by any party advances the current epoch past ceta_new_epoch and disqualifies the reissue on the first predicate above. When any predicate fails the data server returns NFS4ERR_ACCESS or NFS4ERR_STALE_MDS_EPOCH per the ordinary strict-ordering rule and the metadata server MUST obtain a fresh incarnation-lease token from the authority. A fresh token has a new token_id and does not collide with the replay cache; the fresh takeover uses the ordinary advance form.

Trust Anchor Provisioning The verification trust anchor (the public key or key set the authority's signatures verify against) is provisioned at each data server at deployment time. Key distribution mechanisms (X.509 chain, JWK set fetch, raw public key push) are deployment concerns outside the scope of this specification; rotation is likewise deployment-local. The wire format is fully specified so that two independent implementations sharing the same trust anchor can interoperate; the trust-anchor bootstrap itself is not a wire-negotiated step.

Time-Related Bounds The token's issued_at MUST NOT be more than skew_tolerance in the future when the data server evaluates it (deployment-configured; recommend NTP-consistent, roughly 10 s). The token's expires_at is compared strictly: the token becomes inadmissible at the first instant now >= expires_at. No skew tolerance is added to the right edge; the incarnation-lease authority MUST NOT issue a successor token to a different metadata-server principal until at least prior.expires_at + max_ds_skew_tolerance, where max_ds_skew_tolerance is a deployment-configured bound that MUST equal or exceed the largest skew_tolerance any data server in the scope is allowed to use. This ordering ensures that no data server admits the prior holder at any instant after the successor's token becomes admissible.

checksum4

XDR for checksum4 ; /// }; ]]> The checksum4 (see ) is a tagged checksum value used to detect transport corruption and on-disk bit rot of chunk payloads. Every chunk on the wire and at rest carries a checksum4 alongside its chunk_owner4.

cs_algorithm:

identifies the checksum algorithm. The values listed above are registered by this document; additional values are managed by the IANA registry (see "Checksum Algorithm Registry" in the IANA Considerations section). CHECKSUM_ALG_NONE indicates the deployment relies on transport-layer (TLS, IPsec) or storage-layer integrity instead of a protocol-level per-chunk checksum.

cs_value:

the checksum bytes. The length is fixed per registered algorithm:

CHECKSUM_ALG_NONE: 0 bytes.
CHECKSUM_ALG_CRC32: 4 bytes.
CHECKSUM_ALG_CRC32C: 4 bytes.
CHECKSUM_ALG_FLETCHER4: 32 bytes (four 64-bit accumulators, matching the ZFS Fletcher4 layout).
CHECKSUM_ALG_SHA256: 32 bytes.
CHECKSUM_ALG_SHA512: 64 bytes.
CHECKSUM_ALG_BLAKE3: 32 bytes (BLAKE3 standard output length).

A checksum4 whose cs_value length does not match the registered length for cs_algorithm MUST be rejected with NFS4ERR_INVAL.

Coverage:

Every registered algorithm covers the same input bytes: the chunk's header immediately followed by the chunk payload, in wire-transmission order, with the bytes of the header's own checksum field (cs_value) treated as zero for the duration of the computation. After computing, the writer stores the resulting bytes into cs_value for transmission and at-rest storage; the reader saves the received cs_value, treats those bytes as zero, recomputes over the same input, and compares against the saved value. Including the header in coverage protects the per-chunk metadata (payload_id, guard, owner, length fields) as well as the payload; treating the checksum field as zero makes the computation independent of the field's wire value so the same input produces the same output on both ends. This coverage rule applies uniformly to every registered cs_algorithm; individual registry entries name the function and any function-specific parameters, but do not restate the covered-bytes rule.

The checksum algorithm for a given file is selected by the metadata server at LAYOUTGET time and carried in the layout (see ). A client that does not implement the algorithm a layout names returns the layout with NFS4ERR_LAYOUT_CHECKSUM_NOT_SUPPORTED (); the metadata server may then offer a layout with a different algorithm.

New NFSv4.2 Operations

Operations XDR The following amendment blocks extend the nfs_argop4 and nfs_resop4 dispatch unions defined in with arms for each of the new operations defined in this document. A consumer that combines this document's extracted XDR with the RFC 7863 XDR applies these amendments at the union's extension point.

nfs_argop4 amendment block

nfs_resop4 amendment block Operations 78 through 88 (the CHUNK operations) are sent by clients to storage devices on the data path. Operations 89 through 91 (TRUST_STATEID, REVOKE_STATEID, BULK_REVOKE_STATEID) and operations 92 through 95 (CHUNK_ESCROW_INSTALL, CHUNK_ESCROW_RELEASE, CHUNK_ESCROW_ENUMERATE, CHUNK_ESCROW_TAKEOVER) are sent by the metadata server to storage devices on the metadata-server-to-data-server control session (see ); they MUST NOT be sent by pNFS clients. The escrow control-plane operations (92 through 95) are available on the metadata-server-to-data-server control session under either loose- or tight coupling deployment; the tight coupling section is cited for its description of the session itself, not to restrict availability to the tight coupling profile. All CHUNK operations MUST be issued under an active flexible file v2 layout obtained via LAYOUTGET against the metadata server. Because encodings that use CHUNK operations require tight coupling (see the three constraints in ), the presented stateid is the tight coupling registered layout stateid, and the data server MUST validate it against its per-file trust table: a stateid not present in the trust table MUST be rejected with NFS4ERR_BAD_STATEID per . The anonymous stateid is reserved for PASSTHROUGH mirrors under loose coupling () and MUST NOT appear on a CHUNK operation; a data server receiving a CHUNK operation with the anonymous stateid MUST reject it with NFS4ERR_BAD_STATEID. The chunk envelope's safety properties (atomicity via chunk_guard4 CAS, integrity via checksum, lock continuity across revocation) depend on metadata-server coordination of layout grants, guard generation, and lock escrow. A client that issues CHUNK operations without an active chunked-encoding layout has no chunked-layout stateid to present; whatever stateid it does send (anonymous, open, lock, forged, expired) is rejected by the data server's normal stateid validation per , typically with NFS4ERR_BAD_STATEID (or NFS4ERR_STALE_STATEID or NFS4ERR_OLD_STATEID as appropriate). See for the distinction between the CHUNK operations and a generic block I/O interface. Protocol OPs

Operation	Number	Target Server
CHUNK_COMMIT	78	data server (client)
CHUNK_ERROR	79	data server (client)
CHUNK_FINALIZE	80	data server (client)
CHUNK_HEADER_READ	81	data server (client)
CHUNK_LOCK	82	data server (client)
CHUNK_READ	83	data server (client)
CHUNK_REPAIRED	84	data server (client)
CHUNK_ROLLBACK	85	data server (client)
CHUNK_UNLOCK	86	data server (client)
CHUNK_WRITE	87	data server (client)
CHUNK_WRITE_REPAIR	88	data server (client)
TRUST_STATEID	89	data server (metadata server control)
REVOKE_STATEID	90	data server (metadata server control)
BULK_REVOKE_STATEID	91	data server (metadata server control)
CHUNK_ESCROW_INSTALL	92	data server (metadata server control)
CHUNK_ESCROW_RELEASE	93	data server (metadata server control)
CHUNK_ESCROW_ENUMERATE	94	data server (metadata server control)
CHUNK_ESCROW_TAKEOVER	95	data server (metadata server control)

Bounds on Chunk-Operation Arrays The chunk-lifecycle operations (CHUNK_WRITE, CHUNK_WRITE_REPAIR, CHUNK_READ, CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK, CHUNK_LOCK, CHUNK_UNLOCK, CHUNK_ERROR, CHUNK_REPAIRED, CHUNK_HEADER_READ) carry variable-length arrays of chunks, owners, status codes, checksums, and payloads. To bound allocation and to guarantee that a valid request always has an encodable response, this document defines the following protocol maxima:

Chunk-operation array maxima CHUNK_MAX_CHUNKS_PER_OP is the maximum number of chunks addressed by a single CHUNK_WRITE, CHUNK_WRITE_REPAIR, CHUNK_READ, CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK, or CHUNK_LOCK request; it bounds cwa_co_ids, cra_ranges, cca_chunks, cfa_chunks, cra_chunks (CHUNK_ROLLBACK), and the equivalent arrays in the other operations. CHUNK_MAX_PAYLOAD_BYTES bounds the aggregate opaque payload (cwa_chunks, cwra_chunks, and the concatenated cr_chunk bytes returned in CHUNK_READ4resok). CHUNK_MAX_CHECKSUMS_PER_OP, CHUNK_MAX_OWNERS_PER_OP, and CHUNK_MAX_STATUS_PER_OP bound the co-indexed result and argument arrays with matching cardinality; each MUST equal the request's chunk count on success, and MUST be present with the requested cardinality on failures for positional correlation. A data server MUST reject a request whose array length exceeds any of these maxima with NFS4ERR_INVAL before performing any mutation. A data server MUST also verify, before performing any mutation, that the response it will construct fits within the session's negotiated ca_maxresponsesize (); if the mandatory response arrays (per-chunk status, per-chunk owner, per-chunk checksum, and payload where applicable) would exceed ca_maxresponsesize, the data server MUST reject with NFS4ERR_TOOSMALL and MUST NOT partially process the request. The client is expected to split the request across multiple compounds when either bound is reached. Short processing (a data server returning fewer chunks than requested at its discretion) is NOT permitted for the lifecycle operations because the co-indexed result arrays would lose positional correlation. A data server that cannot process all requested chunks MUST reject the entire request rather than partially serve it. The rejection uses one of the errors already defined for the two capacity-limit paths above -- NFS4ERR_INVAL when the request array length exceeds a CHUNK_MAX_* bound, NFS4ERR_TOOSMALL when the response would exceed the session-negotiated ca_maxresponsesize (), or NFS4ERR_DELAY when the data server is out of transient resources to process the requested batch -- and the entire request is a no-op with respect to mutation. On receiving any of these rejections, a client that wishes to retry SHOULD halve the batch size and retry, subject to the static maxima in and the session's ca_maxresponsesize. A client that has already reduced the batch to a single chunk and continues to see the rejection MUST escalate via LAYOUTERROR rather than retry unbounded.

Operation 78: CHUNK_COMMIT - Activate Cached Chunk Data

ARGUMENTS

XDR for CHUNK_COMMIT4args ; /// }; ]]>

RESULTS

XDR for CHUNK_COMMIT4resok ; /// }; ]]>

XDR for CHUNK_COMMIT4res

DESCRIPTION The CHUNK_COMMIT operation is based upon the NFSv4.1 COMMIT operation (see ) and similarly commits previously written data to stable storage on the regular file identified by the current filehandle, with the difference that CHUNK_COMMIT operates on the chunk coordinate system used by Flexible File Version 2 layouts rather than on the byte coordinate system, and that CHUNK_COMMIT advances each named chunk through the chunk state machine from FINALIZED to COMMITTED () rather than acting on a byte range without a state-machine context. The client provides cca_offset and cca_count to bound the chunk range, and cca_chunks to name the specific (chunk_owner4) generations within that range to commit:

cca_offset:: starting chunk index in the file (not a byte offset).
cca_count:: number of chunks the range covers, starting at cca_offset. When cca_count is zero, the client MUST also supply an empty cca_chunks array, and the data server returns NFS4_OK with an empty ccr_status array. A cca_offset beyond the data server's highest chunk with a non-zero cca_count is not itself an error at the operation level: the data server evaluates each cca_chunks entry per its normal per-entry rules and returns the co-indexed ccr_status.
cca_chunks:: an array of chunk_owner4 entries () naming the specific generations to commit. For each entry the data server looks up the chunk index it associated with the complete owner triple when the client wrote it via CHUNK_WRITE or CHUNK_WRITE_REPAIR. That recorded chunk index MUST lie in [cca_offset, cca_offset + cca_count); if the triple does not match any recorded owner association on this data server for this file, or the recorded chunk index lies outside the requested range, the entry is rejected with NFS4ERR_INVAL in the corresponding ccr_status slot. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear as the co_client_id of any cca_chunks entry; see and .

cca_offset and cca_count are NOT redundant with cca_chunks: the owner triples in cca_chunks name specific generations (which the data server correlates via its recorded owner-to-index association), while cca_offset and cca_count bound the intended chunk-index scope of the operation. A chunk index MAY have multiple persisted generations at the moment CHUNK_COMMIT arrives -- an older COMMITTED generation retained for the rollback invariant () alongside a newer FINALIZED successor. cca_chunks selects which owner triple to advance to COMMITTED at each affected index; cca_offset and cca_count let the data server reject malformed requests that name generations whose recorded chunk index lies outside the intended commit window. The CHUNK_COMMIT result reports the outcome per chunk in the same order as cca_chunks:

ccr_writeverf:: a verifier identifying the data server's incarnation at the time the commit completed. A client compares ccr_writeverf to the cwr_writeverf returned by the prior CHUNK_WRITE to detect a data server restart that lost UNSTABLE4 writes between the write and the commit; on a mismatch the client MUST re-issue the CHUNK_WRITE before any committed bytes are considered durable. ccr_writeverf changes on every data server restart that loses uncommitted state.
ccr_status:: per-chunk commit status, one entry per cca_chunks entry, co-indexed. NFS4_OK indicates that the named chunk is COMMITTED on return. Other per-entry failure codes are described in "Interaction with CHUNK_FINALIZE" and "Interaction with a Locked Chunk" below. The top-level CHUNK_COMMIT status is NFS4_OK as long as the data server could evaluate each cca_chunks entry; per-chunk failures are reported in ccr_status rather than by failing the whole operation. The top-level status returns a non-OK code only when the request could not be evaluated at all.

Like CHUNK_READ and CHUNK_WRITE, CHUNK_COMMIT carries an explicit layout stateid in cca_stateid. The data server authorizes CHUNK_COMMIT by validating cca_stateid against the file identified by the current filehandle: cca_stateid MUST be the layout stateid the metadata server issued to the caller for the current filehandle, or the special anonymous stateid (see below). Under trusted stateid tight coupling (), the data server rejects CHUNK_COMMIT with NFS4ERR_BAD_STATEID unless cca_stateid is present in the data server's trust table for the current filehandle. The explicit field ensures that a CHUNK_COMMIT in its own standalone compound (typical for recovery and pipelined lifecycle operations) carries the authorization the data server needs without depending on any prior operation's implicit state. Passing the special anonymous stateid is permitted only when the underlying security regime authorizes an unattributed writer (that is, when tight coupling is not in force and the deployment's access-control policy permits it). If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

Interaction with CHUNK_FINALIZE CHUNK_COMMIT transitions a chunk from FINALIZED to COMMITTED (see ). A chunk MUST have previously been transitioned from PENDING to FINALIZED via CHUNK_FINALIZE before CHUNK_COMMIT is accepted:

If the target chunk is PENDING (i.e., the writer never issued CHUNK_FINALIZE), the data server MUST reject the CHUNK_COMMIT entry for that chunk with NFS4ERR_PAYLOAD_NOT_ATOMIC in the corresponding ccr_status slot. The writer is expected to either issue CHUNK_FINALIZE to advance the state or CHUNK_ROLLBACK to abandon the PENDING generation.
If the target chunk is EMPTY (no generation to commit), the data server MUST reject with NFS4ERR_PAYLOAD_NOT_ATOMIC for that chunk.

The two rejection cases above share the error code because the client's local bookkeeping distinguishes them without requiring the data server to differentiate. A client with intact local state that recognizes it holds a PENDING generation for the chunk calls CHUNK_FINALIZE; a client with no local record calls CHUNK_HEADER_READ to inspect the chunk's actual state (owner triple, disposition) and reconcile from there.

If the target chunk is already COMMITTED at the generation identified by the cca_chunks entry's owner triple, the CHUNK_COMMIT is idempotent and MUST succeed. Idempotence preserves the NFSv4 COMMIT contract for duplicate-request retransmission.
If the target chunk is FINALIZED at a different generation than the one named in the cca_chunks entry, the data server MUST reject with NFS4ERR_CHUNK_GUARDED. A client that sees this has lost a race and SHOULD re-read the chunk (see ).

Pipelining Considerations The three-step CHUNK_WRITE -> CHUNK_FINALIZE -> CHUNK_COMMIT sequence MAY be pipelined within a single NFSv4.2 compound (see ) in single writer mode, where no other writer can race the client's per-chunk transitions and the CHUNK_WRITE per-block status array reports only local-failure cases (NFS4ERR_NOSPC, NFS4ERR_IO, and so on). Same-compound pipelining is NOT RECOMMENDED in multiple-writer mode. CHUNK_WRITE reports per-block outcomes in cwr_block_status (); a partial-success outcome (some chunks accepted, others rejected with NFS4ERR_CHUNK_GUARDED on a lost race) leaves the client without an opportunity to react before a same-compound CHUNK_FINALIZE / CHUNK_COMMIT proceeds against whichever chunks happen to be PENDING. The compound-level status is NFS4_OK in this case because per-block failures are reported in the per-op status array rather than as a compound-level error, so NFSv4 compound short-circuit () does not stop the trailing ops. A client that wants atomic-or-none semantics across multiple chunks MUST examine the per-block status returned by each CHUNK_WRITE before issuing the corresponding CHUNK_FINALIZE. For multi-chunk pipelines in multiple-writer mode, the recommended pattern is to stagger the three steps across compounds so each trailing operation acts only on chunks whose preceding operation's status the client has already inspected:

Staggered three-stage chunk pipeline (multiple-writer mode) In each compound, the CHUNK_WRITE acts on the trailing chunk the client wants to enqueue next; the CHUNK_FINALIZE operates on a chunk whose CHUNK_WRITE the client has already inspected in a previous compound; the CHUNK_COMMIT operates on a chunk whose CHUNK_FINALIZE the client has already inspected. If any per-block status in compound N reports a guard loss or other failure, the client abandons the affected chunk (via CHUNK_ROLLBACK in compound N + 1 or later) without ever issuing the trailing FINALIZE / COMMIT for it. This pattern adds two compounds of latency between a chunk's write and its commit (one for the FINALIZE wait, one for the COMMIT wait), but provides the client with the per-step inspection point required for atomic-or-none multi-chunk writes under contention.

Interaction with a Locked Chunk When a chunk is locked via CHUNK_LOCK (see ), CHUNK_COMMIT is permitted only when the submitter owns the lock -- that is, when the stateid carried on the compound matches the lock holder's stateid (or is an CHUNK_LOCK_FLAGS_ADOPT-transferred continuation):

The owning writer MAY issue CHUNK_COMMIT; the chunk transitions from FINALIZED to COMMITTED normally.
A non-owning client MUST receive NFS4ERR_CHUNK_LOCKED in the corresponding ccr_status slot. The chunk's state is not changed.
During repair, the metadata-server escrow owner (CHUNK_GUARD_CLIENT_ID_MDS, see ) holds the lock while the repair actor adopts it via CHUNK_LOCK_FLAGS_ADOPT. CHUNK_COMMIT during the escrow window is permitted only to the holder of the adopted lock.

This rule is what calls "lock continuity across revocation": the COMMIT privilege follows the lock without gaps in which a non-owner could race.

RESPONSE CODES

NFS4_OK:: every named chunk transitioned to COMMITTED.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to commit on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_DELAY:: the data server is temporarily unable to process the request.
NFS4ERR_FHEXPIRED:: the current filehandle has expired.
NFS4ERR_INVAL:: arguments named chunks outside the file's mirror set or in a non-atomic state.
NFS4ERR_IO:: an I/O error occurred while persisting the commit.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_COMMIT.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE:: the current filehandle no longer identifies a valid file.

Operation 79: CHUNK_ERROR - Report Error on Cached Chunk Data

ARGUMENTS

XDR for CHUNK_ERROR4args

RESULTS

XDR for CHUNK_ERROR4res

DESCRIPTION CHUNK_ERROR allows a client that has detected corruption or inconsistency in a chunk to report the condition to the data server, so that the data server can mark the affected chunks as errored. Errored chunks are excluded from subsequent CHUNK_READ responses until they are repaired via CHUNK_WRITE_REPAIR and the repair is confirmed via CHUNK_REPAIRED. CHUNK_ERROR has no direct analog in . The closest parallel is LAYOUTERROR (), which reports layout-level errors to the metadata server. CHUNK_ERROR is the data-path counterpart: it reports a chunk-level integrity finding directly to the data server so that the corrupted chunks are quarantined before the metadata server has had time to coordinate repair. A client SHOULD issue CHUNK_ERROR to the data server holding the bad chunks before issuing LAYOUTERROR to the metadata server. The client provides:

cea_stateid:: the layout stateid the metadata server granted for this file. Under trusted stateid tight coupling (), this stateid MUST be in the data server's trust table; otherwise the data server rejects the operation with NFS4ERR_BAD_STATEID.
cea_offset:: starting chunk index of the affected range (not a byte offset).
cea_count:: number of chunks the affected range covers, starting at cea_offset.
cea_error:: the nfsstat4 error code that describes the integrity finding. Typical values include NFS4ERR_PAYLOAD_NOT_ATOMIC (the chunk's persisted checksum or guard did not match the value the client expected), NFS4ERR_IO (the client's CHUNK_READ returned an I/O error from this data server), and NFS4ERR_INVAL (the chunk's chunk_owner4 did not match the expected generation across mirrors). The data server MAY record the supplied error code in operator logs but does not otherwise interpret it; the chunk-level effect (mark errored) is the same for any cea_error value.
cea_owner:: the chunk_owner4 () the client read when it observed the error, so the data server can record which owner triple generation was reported as corrupted. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear in cea_owner; see and .

CHUNK_ERROR returns a single top-level status in cer_status; there is no per-chunk status array because the data server either accepts the report for the whole range or returns a top-level error. Once a CHUNK_ERROR has been accepted, the affected chunks transition into the errored state described in ; subsequent CHUNK_READ operations against those chunks return NFS4ERR_PAYLOAD_NOT_ATOMIC in the per-chunk cr_status slot until a successful CHUNK_REPAIRED sequence clears the errored flag. If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

RESPONSE CODES

NFS4_OK:: the client's chunk error report has been recorded.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to report errors on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_INVAL:: the reported chunk range or error code was not recognized.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_ERROR.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.

Operation 80: CHUNK_FINALIZE - Transition Chunks from Pending to Finalized

ARGUMENTS

XDR for CHUNK_FINALIZE4args ; /// }; ]]>

RESULTS

XDR for CHUNK_FINALIZE4resok ; /// }; ]]>

XDR for CHUNK_FINALIZE4res

DESCRIPTION CHUNK_FINALIZE transitions chunks from the PENDING state (set by CHUNK_WRITE, see ) to the FINALIZED state in the chunk state machine (). A FINALIZED chunk is visible on the owning stateid for reads () and is eligible for CHUNK_COMMIT; the FINALIZED transition is the writer's signal that it will issue no further CHUNK_WRITEs for the named owner triple generation of each chunk. CHUNK_FINALIZE has no direct analog in : the COMMIT operation in combines the "no more writes" signal and the "make durable and globally visible" step into one operation; the Flexible File Version 2 chunk lifecycle separates them so a writer in multiple-writer mode can validate the per-chunk acceptance status reported by CHUNK_WRITE before committing any chunk to durable storage (see "Pipelining Considerations" in ). The client provides cfa_offset and cfa_count to bound the chunk range, and cfa_chunks to name the specific (chunk_owner4) generations within that range to finalize:

cfa_offset:: starting chunk index in the file (not a byte offset).
cfa_count:: number of chunks the range covers, starting at cfa_offset. When cfa_count is zero, the client MUST also supply an empty cfa_chunks array, and the data server returns NFS4_OK with an empty cfr_status array. A cfa_offset beyond the data server's highest chunk with a non-zero cfa_count is not itself an error at the operation level: the data server evaluates each cfa_chunks entry per its normal per-entry rules and returns the co-indexed cfr_status.
cfa_chunks:: an array of chunk_owner4 entries () naming the specific generations to finalize. For each entry the data server looks up the chunk index it associated with the complete owner triple when the client wrote it via CHUNK_WRITE or CHUNK_WRITE_REPAIR. That recorded chunk index MUST lie in [cfa_offset, cfa_offset + cfa_count); if the triple does not match any recorded owner association on this data server for this file, or the recorded chunk index lies outside the requested range, the entry is rejected with NFS4ERR_INVAL in the corresponding cfr_status slot. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear as the co_client_id of any cfa_chunks entry; see and .

The CHUNK_FINALIZE result reports the outcome per chunk in the same order as cfa_chunks:

cfr_writeverf:

a verifier identifying the data server's incarnation at the time the finalization completed. Semantics match cwr_writeverf in CHUNK_WRITE: a client that observes a different writeverf on a subsequent CHUNK_COMMIT MUST re-issue the CHUNK_WRITE before treating any of the finalized chunks as durable.

cfr_status:

per-chunk finalization status, one entry per cfa_chunks entry, co-indexed. NFS4_OK indicates that the named chunk is FINALIZED on return. Other per-entry failure cases:

NFS4ERR_INVAL -- the named generation is not in the PENDING state at its recorded chunk index (the chunk is EMPTY, FINALIZED at a different generation, or COMMITTED), or the triple does not match any recorded owner association, or the recorded chunk index lies outside [cfa_offset, cfa_offset + cfa_count).
NFS4ERR_CHUNK_GUARDED -- the chunk is PENDING but the current PENDING owner is a different (co_cohort_id, co_client_id) than the one named in the cfa_chunks entry. A client that sees this has lost a race with another writer; see .
NFS4ERR_CHUNK_LOCKED -- the chunk is locked by a CHUNK_LOCK held by a different stateid; the finalize is rejected.

The top-level CHUNK_FINALIZE status is NFS4_OK as long as the data server could evaluate each cfa_chunks entry; per-chunk failures are reported in cfr_status rather than by failing the whole operation. The top-level status returns a non-OK code only when the request could not be evaluated at all.

CHUNK_FINALIZE serves as the CRC validation checkpoint for the chunk lifecycle. The data server SHOULD have validated each chunk's checksum against the value supplied in cwa_checksums at CHUNK_WRITE time; the FINALIZE transition persists the chunk metadata (CRC, owner, state) to stable storage so it survives a data server restart. An implementation MAY defer some metadata persistence to CHUNK_COMMIT instead of CHUNK_FINALIZE; in that case the FINALIZED state is recovered by replay of the data server's local journal on restart. A chunk that has been FINALIZED but not yet COMMITTED MAY be rolled back via CHUNK_ROLLBACK, which returns the chunk to the EMPTY state (or to the prior COMMITTED generation, if one exists). Like CHUNK_COMMIT, CHUNK_FINALIZE carries an explicit layout stateid in cfa_stateid. The data server authorizes CHUNK_FINALIZE by validating cfa_stateid against the file identified by the current filehandle: cfa_stateid MUST be the layout stateid the metadata server issued to the caller for the current filehandle, or the special anonymous stateid (see below). Under trusted stateid tight coupling (), the data server rejects CHUNK_FINALIZE with NFS4ERR_BAD_STATEID unless cfa_stateid is present in the data server's trust table for the current filehandle. The explicit field ensures that a CHUNK_FINALIZE in its own standalone compound (typical for pipelined and recovery cases) carries the authorization the data server needs. Passing the special anonymous stateid is permitted only when the underlying security regime authorizes an unattributed writer. If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

RESPONSE CODES

NFS4_OK:: every named chunk transitioned from PENDING to FINALIZED.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to finalize on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_DELAY:: the data server is temporarily unable to process the request.
NFS4ERR_FHEXPIRED:: the current filehandle has expired.
NFS4ERR_INVAL:: arguments named chunks not in PENDING or outside the file's mirror set.
NFS4ERR_IO:: an I/O error occurred while persisting the transition.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_FINALIZE.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE:: the current filehandle no longer identifies a valid file.

Operation 81: CHUNK_HEADER_READ - Read Chunk Header from File

ARGUMENTS

XDR for CHUNK_HEADER_READ4args

RESULTS

XDR for CHUNK_HEADER_READ4resok ; /// chunk_state_flags4 /// chrr_locked; /// chunk_owner4 /// chrr_owners; /// chunk_guard4 /// chrr_guards; /// optional_retained4 /// chrr_predecessors; /// }; ]]>

XDR for CHUNK_HEADER_READ4res

DESCRIPTION CHUNK_HEADER_READ returns the per-chunk metadata (chunk_owner4, chunk_guard4, lock state, and per-chunk status) for a range of chunks in the target data file without returning the chunk payloads. The operation enables clients and repair coordinators to inspect chunk lifecycle and ownership cheaply, without the data-transfer cost of CHUNK_READ. CHUNK_HEADER_READ has no direct analog in ; it is the chunk-protocol counterpart of a stat-like fast probe and exists because chunks are first-class state-bearing objects whose ownership, lock state, and lifecycle status are not recoverable from a byte-offset query. The client provides:

chra_stateid:: the layout stateid the metadata server granted for this file. Under trusted stateid tight coupling (), this stateid MUST be in the data server's trust table; otherwise the data server rejects the operation with NFS4ERR_BAD_STATEID.
chra_offset:: starting chunk index of the range to inspect (not a byte offset).
chra_count:: number of chunks the inspection range covers, starting at chra_offset. chra_count MUST NOT exceed CHUNK_HEADER_READ_MAX4; a request that exceeds the bound is rejected with NFS4ERR_INVAL. The bound protects the data server against arbitrarily large response construction.

The CHUNK_HEADER_READ result returns five co-indexed arrays, one entry per chunk in the requested range in chunk-offset order from chra_offset:

chrr_eof:

TRUE if the requested range extended at or past the data server's last chunk for this file. Same per-data-server semantics as crr_eof in CHUNK_READ.

chrr_status:

per-chunk lifecycle state encoded as an nfsstat4 (see "Per-Chunk Status Encoding" below).

chrr_locked:

per-chunk chunk_state_flags4 (). CHUNK_STATE_FLAGS_LOCKED set indicates the chunk currently has a CHUNK_LOCK held by some chunk_owner4. Lock state is reported orthogonally to chrr_status so that a locked chunk still surfaces its lifecycle state and chunk_owner4 to the inspector.

chrr_owners:

per-chunk chunk_owner4 (). For a chunk whose chrr_status is NFS4_OK the field is the COMMITTED generation's owner. For NFS4ERR_PAYLOAD_NOT_ATOMIC the field is the writer of the in-progress (PENDING or FINALIZED) generation. For NFS4ERR_NOENT (EMPTY chunk) the chunk_owner4 is unspecified.

chrr_guards:

per-chunk chunk_guard4 () -- the (cg_gen_id, cg_client_id) pair the data server holds as the chunk's current guard at CHUNK_HEADER_READ time. A caller that intends to update the chunk in multiple writer mode uses the corresponding chrr_guards entry as the expected prior value for the guard CAS (), avoiding the payload cost of CHUNK_READ. For a chunk whose chrr_status is NFS4ERR_NOENT (EMPTY) the chunk_guard4 is the all-zeros pair (cg_gen_id = 0 and cg_client_id = CHUNK_GUARD_CLIENT_ID_NONE, see ). chrr_guards is a read-time observation; a concurrent writer MAY advance the guard between the CHUNK_HEADER_READ response and the subsequent CHUNK_WRITE, in which case the CAS returns NFS4ERR_CHUNK_GUARDED and the caller re-observes the guard per the rollback-and-retry flow in .

chrr_predecessors:

per-chunk immediate-predecessor disposition, one optional_retained4 entry per chunk in the returned range, co-indexed with chrr_status, chrr_locked, and chrr_owners. Each entry names the read-time state of the single most recent retained predecessor of the current generation the data server holds for that chunk index under the retention scope rule (); the entry's discriminant is exactly one of:

RETAINED_GENERATION_DISPOSITION_ABSENT:: the data server holds no retained predecessor at that index (either the current chrr_owners generation is the only one, or the chunk is EMPTY). The arm carries no owner triple.
RETAINED_GENERATION_DISPOSITION_PRESENT:: the data server retains an immediate predecessor whose owner triple is carried in the restorable arm and whose payload is in the AVAILABLE read-time state (). A CHUNK_ROLLBACK naming this owner triple satisfies "Rollback of COMMITTED Chunks" case (a) ().
RETAINED_GENERATION_DISPOSITION_ERRORED:: the data server retains the immediate predecessor's owner triple (in the errored arm) but its payload is in the ERRORED read-time state and cannot be restored by CHUNK_ROLLBACK. A CHUNK_ROLLBACK naming this owner triple MUST return NFS4ERR_NO_PREDECESSOR (); the client falls back to best-effort reconstruction via CHUNK_WRITE_REPAIR with an authoritative source of its own choosing. The owner triple is disclosed so a caller can coordinate reconstruction from surviving shards.

The list is informational and MAY change between successive CHUNK_HEADER_READ calls -- the data server MAY release a predecessor between calls under the retention scope rule. A caller that observes a PRESENT disposition and issues CHUNK_ROLLBACK before that release remains guaranteed by the composed rollback guarantee () when it holds a qualifying lock or escrow; without such a lock, the retention scope MAY release the predecessor at any time and a subsequent CHUNK_ROLLBACK MAY return NFS4ERR_NO_PREDECESSOR even though a previous CHUNK_HEADER_READ observed PRESENT.

Cardinality and short responses:

All five response arrays are the same length; the data server MUST NOT sparsify or truncate one array independently of the others. The response array length N MAY be smaller than chra_count when the requested range extends past the data server's last chunk (chrr_eof = TRUE, N = the number of chunks the data server holds within the requested range) or when the fully-populated response would exceed the session-negotiated ca_maxresponsesize (). In the response-size case the data server returns a short response with chrr_eof = FALSE containing as many entries N as fit under ca_maxresponsesize minus COMPOUND/RPC overhead; the client resumes at chra_offset + N. If even the minimum useful response (a single entry) will not fit, the data server returns NFS4ERR_REP_TOO_BIG per ; the client MUST NOT retry with a smaller chra_count and instead uses a session or COMPOUND with more available response budget.

The operation has several uses:

Whole-file repair scan:

A repair actor selected via CB_CHUNK_REPAIR walks the affected chunk range and uses the per-chunk chunk_owner4 returned by each mirror's data server to identify which chunks carry an atomic stripe (all k data shards share the same chunk_guard4) and which require reconstruction. CHUNK_HEADER_READ is the discovery primitive that drives the per-chunk decisions described in ; without it, a repair client would have to issue CHUNK_READ to retrieve the full payload of every chunk merely to inspect its guard.

Client-side recovery from partial writes:

After a network disruption or client restart, a writer that holds the file's layout MAY issue CHUNK_HEADER_READ to learn which of its prior CHUNK_WRITEs reached the data server. Chunks whose chunk_owner4 reports the writer's own (co_cohort_id, co_client_id) pair are PENDING or FINALIZED and recoverable; chunks absent from the response or carrying another writer's owner are not. The writer can then re-issue CHUNK_WRITE for the missing chunks or CHUNK_ROLLBACK for the abandoned ones without reading payloads it has already committed locally.

Read-side atomicity check:

Before issuing a multi-chunk CHUNK_READ in multiple-writer mode, a client MAY issue CHUNK_HEADER_READ to verify that the chunks in the target range share a common (co_cohort_id, co_client_id) pair in chrr_owners (the cohort-atomicity property in ) and MAY additionally inspect chrr_guards as a cheaper generation-level corroboration. If the cohort pairs diverge, the client knows the read will not be atomic and can wait for a writer to commit, retry, or report NFS4ERR_PAYLOAD_NOT_ATOMIC via LAYOUTERROR. This is a hint rather than a guarantee: a concurrent writer MAY advance a chunk's state between the CHUNK_HEADER_READ response and the subsequent CHUNK_READ.

Predecessor-guided rollback discovery:

A caller preparing a CHUNK_ROLLBACK against a COMMITTED chunk inspects the corresponding chrr_predecessors entry to decide whether CHUNK_ROLLBACK will succeed.

PRESENT:: name the disclosed owner triple in the cra_chunks entry of the subsequent CHUNK_ROLLBACK. "Rollback of COMMITTED Chunks" case (a) will succeed subject to the composed rollback guarantee's continuous-custody condition ().
ERRORED:: do NOT issue CHUNK_ROLLBACK against the disclosed owner triple. The data server MUST return NFS4ERR_NO_PREDECESSOR for that owner; use CHUNK_WRITE_REPAIR directly with a reconstructed authoritative source. The disclosed owner triple lets the caller coordinate reconstruction from other sources.
ABSENT:: no restorable predecessor exists. Skip CHUNK_ROLLBACK; use CHUNK_WRITE_REPAIR if reconstruction is possible, or defer to a guaranteed-pinning mechanism when the caller requires the original owner triple be preserved.

As with the atomicity check, a subsequent lifecycle event MAY change a chunk's disposition between the CHUNK_HEADER_READ response and the CHUNK_ROLLBACK (a PRESENT observation MAY become ABSENT if the retention scope releases the predecessor and the caller does not hold a qualifying lock or escrow).

Lock probe before write:

A client MAY issue CHUNK_HEADER_READ and inspect the chrr_locked array to discover whether any chunk in the target range is currently held by a CHUNK_LOCK before attempting CHUNK_WRITE, avoiding the round-trip cost of receiving NFS4ERR_CHUNK_LOCKED. As above, this is a hint; a lock MAY be acquired between the header read and the write.

CHUNK_HEADER_READ does not change any chunk state. If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

Per-Chunk Status Encoding The per-chunk chrr_status field reports the chunk's lifecycle state encoded as an nfsstat4:

NFS4_OK:: the chunk is COMMITTED and the chunk_owner4 in the corresponding chrr_owners slot is the COMMITTED generation's owner.
NFS4ERR_PAYLOAD_NOT_ATOMIC:: the chunk is PENDING or FINALIZED (a non-globally-visible generation is in progress). The chunk_owner4 in the corresponding chrr_owners slot names the writer of that in-progress generation.
NFS4ERR_NOENT:: the chunk is EMPTY (no COMMITTED generation has been written at this offset). The chunk_owner4 in the corresponding chrr_owners slot is unspecified.

CHUNK_HEADER_READ never returns NFS4ERR_CHUNK_LOCKED in chrr_status; lock state is reported orthogonally via chrr_locked so that locked chunks still surface their chunk_owner4 to the inspector.

RESPONSE CODES

NFS4_OK:: the chunk headers have been returned.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to read chunk headers on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_DELAY:: the data server is temporarily unable to process the request.
NFS4ERR_FHEXPIRED:: the current filehandle has expired.
NFS4ERR_IO:: an I/O error occurred while reading chunk headers.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_HEADER_READ.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE:: the current filehandle no longer identifies a valid file.

Operation 82: CHUNK_LOCK - Lock Cached Chunk Data

ARGUMENTS

XDR for CHUNK_LOCK4args

RESULTS

XDR for CHUNK_LOCK4res

DESCRIPTION CHUNK_LOCK acquires an exclusive chunk-range lock on the range specified by cla_offset and cla_count. While the lock is held, CHUNK_WRITE, CHUNK_WRITE_REPAIR, CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK, and CHUNK_UNLOCK operations on any of the locked chunks from any other chunk_owner4 receive NFS4ERR_CHUNK_LOCKED in the corresponding per-chunk status slot. The lock is associated with the chunk_owner4 in cla_owner. CHUNK_LOCK is loosely analogous to LOCK () in that it acquires an exclusive guard against concurrent modification, but the two operate on different coordinate systems and use different naming: LOCK is byte range and stateid-based; CHUNK_LOCK is chunk-range and chunk_owner4-based. CHUNK_LOCK is used in multiple-writer mode () to serialize racing writers on a common chunk range, and in the repair flow () to transfer lock ownership to a repair actor via CHUNK_LOCK_FLAGS_ADOPT. The client provides:

cla_stateid:: the layout stateid the metadata server granted for this file. Under trusted stateid tight coupling (), this stateid MUST be in the data server's trust table; otherwise the data server rejects the operation with NFS4ERR_BAD_STATEID.
cla_offset:: starting chunk index of the lock range (not a byte offset).
cla_count:: number of chunks the lock range covers, starting at cla_offset.
cla_flags:: bitmask of CHUNK_LOCK_FLAGS_* values. Defined: CHUNK_LOCK_FLAGS_ADOPT (adopt a metadata-server escrow lock; see "Lock Transfer via CHUNK_LOCK_FLAGS_ADOPT" below); CHUNK_LOCK_FLAGS_TAKEOVER (transfer ownership of a live client-held lock, distinct from adoption; see "Live-Client Lock Takeover via CHUNK_LOCK_FLAGS_TAKEOVER" below). The two flags are mutually exclusive; a request with both bits set MUST be rejected with NFS4ERR_INVAL. Unknown bits MUST be rejected with NFS4ERR_INVAL.
cla_owner:: the chunk_owner4 () that will hold the lock on success. The reserved sentinel values CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear as the co_client_id of cla_owner; see and . (A client requesting CHUNK_LOCK_FLAGS_ADOPT MUST use its own co_client_id, not the metadata-server escrow sentinel, even when adopting from a metadata-server escrow holder.)
cla_adopt:: a discriminated union carrying the escrow_id4 () that identifies the specific metadata-server escrow lock the caller is adopting. When cla_flags carries CHUNK_LOCK_FLAGS_ADOPT, cla_adopt MUST be the TRUE arm and cla_escrow_id MUST match the escrow_id4 the metadata server installed on this data server for the requested range (identity mismatch is one of the state-level causes of NFS4ERR_NO_ADOPTABLE_LOCK per ). When cla_flags does not carry CHUNK_LOCK_FLAGS_ADOPT, cla_adopt MUST be the FALSE arm. The two conditions (bit-flag value and discriminant value) MUST agree; a mismatch is rejected with NFS4ERR_INVAL.

The CHUNK_LOCK result returns:

clr_status:: NFS4_OK if the lock was acquired (or transferred via ADOPT). NFS4ERR_CHUNK_LOCKED if one or more chunks in the range are already locked and the request does not carry CHUNK_LOCK_FLAGS_ADOPT.
clr_owner (NFS4ERR_CHUNK_LOCKED case only):: the chunk_owner4 of the current lock holder, so the caller can identify the blocking writer.

The lock is released by CHUNK_UNLOCK or implicitly when the holder's lease expires; on lease expiry without explicit release, the data server transitions the lock to the metadata-server escrow owner if the metadata server has revoked the holder's stateid via REVOKE_STATEID, per the lock-continuity-across-revocation invariant in . If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

Lock Transfer via CHUNK_LOCK_FLAGS_ADOPT The CHUNK_LOCK_FLAGS_ADOPT flag in cla_flags requests an atomic transfer of lock ownership to cla_owner for every chunk in [cla_offset, cla_offset+cla_count). The data server MUST perform the transfer as a single atomic step per chunk: there is no window in which the chunk is unlocked. After a successful ADOPT, subsequent CHUNK_WRITE, CHUNK_WRITE_REPAIR, CHUNK_ROLLBACK, and CHUNK_UNLOCK operations MUST present cla_owner as their chunk_owner4. CHUNK_LOCK_FLAGS_ADOPT is the sole mechanism by which a chunk lock can change hands without first being released. The lock ordering invariant -- that every chunk in a payload transitioning through repair is held by exactly one owner continuously from failure detection to repair completion -- depends on it. CHUNK_LOCK_FLAGS_ADOPT is valid only when the caller has been selected as the repair actor for the range by the metadata server, typically via CB_CHUNK_REPAIR. A data server that receives CHUNK_LOCK with the ADOPT flag from a client that has not been so designated MAY reject the operation with NFS4ERR_ACCESS. Because CHUNK_LOCK is a CHUNK operation and encodings that use CHUNK operations require tight coupling (), the metadata server notifies the data server of the ADOPT designation via the control protocol (e.g., TRUST_STATEID with the new client's stateid or a similar facility); no loose coupling ADOPT path exists. The current lock holder at the moment of ADOPT MAY be:

Another client whose stateid remains valid (for example, a client that has stopped making progress but has not yet lost its lease). The prior owner's PENDING or FINALIZED shards remain on disk until the new owner issues CHUNK_WRITE_REPAIR, CHUNK_ROLLBACK, or CHUNK_COMMIT.
The metadata server itself, acting through the CHUNK_GUARD_CLIENT_ID_MDS escrow owner (). This occurs when the metadata server has revoked the prior holder's stateid in a tightly coupled deployment.

In either case, ADOPT's effect from the repair actor's perspective is the same: after the successful return the caller holds the lock and may drive the range to consistency. The data server MUST reject CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT if cla_owner's co_client_id equals CHUNK_GUARD_CLIENT_ID_MDS -- that value is reserved for server production and MUST NOT be presented by a client. The operation returns NFS4ERR_INVAL in that case.

Live-Client Lock Takeover via CHUNK_LOCK_FLAGS_TAKEOVER The CHUNK_LOCK_FLAGS_TAKEOVER flag in cla_flags requests an atomic transfer of lock ownership from a currently live client-held lock to cla_owner, distinct from CHUNK_LOCK_FLAGS_ADOPT which transfers from an metadata-server escrow lock. The two flags are mutually exclusive: ADOPT names the metadata server's escrow identity via cla_adopt, while TAKEOVER names another client's already-held lock and is used only when the metadata server has designated cla_owner as the successor to a displaced live client (for example, when a repair sequence must proceed while the prior writer's session remains valid). Under CHUNK_LOCK_FLAGS_TAKEOVER, cla_adopt MUST be the FALSE arm (there is no escrow identity being adopted); the metadata server's designation is what authorizes the transfer, and the data server verifies the designation by the same coupling-model-dependent mechanism used for ADOPT above. A data server that receives TAKEOVER from a client not designated as successor MAY reject with NFS4ERR_ACCESS. As with ADOPT, TAKEOVER is atomic: no window exists in which the chunk is unlocked, and after a successful TAKEOVER subsequent operations on the range MUST present cla_owner as their chunk_owner4.

RESPONSE CODES

NFS4_OK:: the requested chunk range has been locked.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to lock chunks on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_CHUNK_LOCKED:: one or more chunks in the requested range are already locked by another writer.
NFS4ERR_INVAL:: the requested range was malformed or outside the file's mirror set.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_LOCK.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.

Operation 83: CHUNK_READ - Read Chunks from File

ARGUMENTS

XDR for CHUNK_READ4args

RESULTS

XDR for read_chunk4 ; /// }; ]]>

XDR for CHUNK_READ4resok ; /// }; ]]>

XDR for CHUNK_READ4res

DESCRIPTION The CHUNK_READ operation is based upon the NFSv4.1 READ operation (see ) and similarly reads data from the regular file identified by the current filehandle, with the difference that CHUNK_READ operates on the chunk coordinate system used by Flexible File Version 2 layouts rather than on the byte coordinate system. The client provides a cra_offset of where the CHUNK_READ is to start and a cra_count of how many chunks are to be read. cra_offset is the starting chunk index in the file (not a byte offset); the chunk at index N occupies the bytes [N * chunk_size, (N + 1) * chunk_size) for encodings with a uniform chunk size, where chunk_size is taken from ffv2m_striping_unit_size in the file's layout (). For encodings whose parity shards have variable sizes (the Mojette family), the parity-shard chunks on a given data server may use a smaller per-shard chunk size; see . cra_count is a count of chunks to read and not bytes to read. A cra_offset of zero starts reading at the first chunk of the file. If cra_offset is greater than or equal to the number of chunks the data server holds for this file, the status NFS4_OK is returned with crr_chunks empty and crr_eof set to TRUE. If cra_count is zero, the CHUNK_READ succeeds and returns zero chunks. In all situations the data server MAY choose to return fewer chunks than the client requested; the client must be prepared to handle a short read and reissue CHUNK_READ for the remaining chunks. The CHUNK_READ result is comprised of an array of read_chunk4, each describing the metadata and payload of one chunk. The array entries are in chunk-index order starting from cra_offset. Within each read_chunk4 ():

cr_checksum:: the checksum4 () the data server recomputed over the chunk header and cr_chunk (with the checksum field's bytes treated as zero, per the uniform coverage rule in ) at CHUNK_READ time, from the persisted value it recorded at CHUNK_FINALIZE or CHUNK_COMMIT time. The cs_algorithm field matches the layout's ffv2m_checksum_algorithm (); the cs_value carries the computed bytes at the length registered for that algorithm. The client uses cr_checksum to detect transport corruption between the data server and the client; see for the scope and limits of checksum protection per algorithm class.
cr_effective_len:: the byte length of cr_chunk. This may be smaller than the layout's chunk_size when the chunk is the final chunk of a file whose size is not chunk-aligned, or when the chunk belongs to a variable-size Mojette parity shard.
cr_owner:: the full owner triple of the COMMITTED generation being returned (see ); co_id is the opaque writer-supplied per-chunk identifier the client provided at CHUNK_WRITE or CHUNK_WRITE_REPAIR time, not a chunk index. A client reading from multiple data servers in an erasure-coded layout MUST compare the pair (cr_owner.co_cohort_id, cr_owner.co_client_id) across data servers; agreement of the cohort pair across the k data shards is the atomicity invariant on which reconstruction depends. See .
cr_guard:: the (cg_gen_id, cg_client_id) pair the data server holds as the chunk's current chunk_guard4 () at CHUNK_READ time. A client that intends to update this chunk in multiple writer mode uses cr_guard as the expected prior value for the guard CAS: it supplies cwa_guard.cwg_check = TRUE with cwa_guard.cwg_guard set to the observed cr_guard on the subsequent CHUNK_WRITE (, ). A client reading from multiple data servers in an erasure-coded layout MAY also compare cr_guard values across shards as an auxiliary check on payload atomicity; the cr_owner cohort-pair comparison (see the bullet above and ) is the normative atomicity invariant and cr_guard adds a cheaper generation-level check. For the NFS4ERR_NOENT synthetic zero-filled chunk the cr_guard is set to the all-zeros pair (cg_client_id = CHUNK_GUARD_CLIENT_ID_NONE, see ).
cr_payload_id:: the payload-id the writer associated with the chunk at CHUNK_WRITE time, used by repair coordinators to correlate chunks across mirrors.
cr_locked:: chunk_state_flags4 (). CHUNK_STATE_FLAGS_LOCKED set indicates the chunk currently has a CHUNK_LOCK held against it. Lock state does not block the read.
cr_status:: per-chunk status. NFS4_OK indicates that cr_chunk is the COMMITTED payload. NFS4ERR_PAYLOAD_NOT_ATOMIC indicates the chunk's persisted checksum or guard check failed at read time, in which case cr_chunk content is undefined; see . NFS4ERR_NOENT indicates the chunk is EMPTY (no COMMITTED generation has been written at this offset).
cr_chunk:: the chunk payload bytes. Empty for cr_status values other than NFS4_OK.

A chunk that is EMPTY at the requested offset is returned as a synthetic zero-filled chunk: cr_status is NFS4ERR_NOENT, cr_chunk is zero-filled to the layout's chunk_size, cr_owner is set to all-zeros (with co_client_id = CHUNK_GUARD_CLIENT_ID_NONE, see ), cr_guard is set to the all-zeros pair (cg_gen_id = 0 and cg_client_id = CHUNK_GUARD_CLIENT_ID_NONE), and cr_checksum is the checksum of the synthetic zero-filled payload. This lets a client reconstruct holes without a special-casing path. The data server MAY signal end-of-file by setting crr_eof to TRUE. If the CHUNK_READ ended at the last chunk that exists on this data server (the read returned chunks up to and including the data server's last chunk) or extended beyond it, crr_eof MUST be TRUE. Otherwise crr_eof is FALSE. A successful CHUNK_READ of an empty file always returns crr_eof as TRUE with crr_chunks empty. Note that crr_eof reflects the state at the data server only; in a multi-data-server erasure-coded layout the file's logical size is reconstructed at the client from the chunk-index positions at which surviving shards hold non-EMPTY chunks (observed via successive CHUNK_READs at known offsets), not from any single data server's crr_eof. Because co_id is opaque per , the reconstructing client MUST NOT derive positional information from the chunk_owner4 values themselves. Except when special stateids are used, the cra_stateid value represents a layout stateid returned by a prior LAYOUTGET against the metadata server (see ). The data server uses cra_stateid to verify that the client holds a valid layout that authorizes reading this file. Under trusted stateid tight coupling (), the data server additionally checks that the metadata server has registered the stateid via TRUST_STATEID; an unregistered stateid (other than a special stateid) returns NFS4ERR_BAD_STATEID. For a CHUNK_READ with a cra_stateid value of all bits equal to zero, the data server MAY allow the CHUNK_READ to be serviced subject to the chunk-lock state recorded in cr_locked. For a CHUNK_READ with a cra_stateid value of all bits equal to one, the data server MAY allow CHUNK_READ to bypass lock-state reporting at the data server. These special-stateid behaviors mirror the corresponding READ semantics in adapted to the chunk-locking model () rather than the byte range locking model of If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ). shows a client requesting 4 chunks starting at chunk index 2. Data Server 2 responds as in : there is valid data for chunks 2 and 4, a synthetic zero-filled hole at chunk 3, and no data for chunk 5 (the data server's last chunk is chunk 4, so crr_eof is TRUE). The data server calculates a valid cr_checksum for chunk 3 based on the synthetic zero-filled payload.

Example: CHUNK_READ4args parameters

Example: Resulting CHUNK_READ4resok reply

RESPONSE CODES

NFS4_OK:: the requested chunks have been returned.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to read this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_DELAY:: the data server is temporarily unable to process the request.
NFS4ERR_FHEXPIRED:: the current filehandle has expired.
NFS4ERR_IO:: an I/O error occurred while reading the chunks.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_READ.
NFS4ERR_PAYLOAD_NOT_ATOMIC:: one or more chunks failed their persisted guard or CRC check. See .
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE:: the current filehandle no longer identifies a valid file.

Operation 84: CHUNK_REPAIRED - Confirm Repair of Errored Chunk Data

ARGUMENTS

XDR for CHUNK_REPAIRED4args

RESULTS

XDR for CHUNK_REPAIRED4res

DESCRIPTION CHUNK_REPAIRED signals that chunks previously marked as errored (via CHUNK_ERROR, see ) have been repaired and the errored state can be cleared. The repair client writes replacement data via CHUNK_WRITE_REPAIR, advances the new chunks through CHUNK_FINALIZE and CHUNK_COMMIT, and only then issues CHUNK_REPAIRED to make the repaired chunks visible to normal CHUNK_READ traffic again. CHUNK_REPAIRED has no direct analog in ; it is the chunk-protocol equivalent of clearing a "needs scrub" flag after a RAID controller has rewritten a parity stripe. Together with CHUNK_ERROR it forms the data-server-side state-bit pair that quarantines damaged chunks from ordinary reads during the repair window. The client provides:

cra_stateid:: the layout stateid the metadata server granted to the repair actor. Under trusted stateid tight coupling (), this stateid MUST be in the data server's trust table; otherwise the data server rejects the operation with NFS4ERR_BAD_STATEID.
cra_offset:: starting chunk index of the repaired range (not a byte offset).
cra_count:: number of chunks the repaired range covers, starting at cra_offset. The cra_offset / cra_count range MUST match the range named in the original CHUNK_ERROR that marked these chunks errored; mismatched ranges are rejected with NFS4ERR_INVAL.
cra_owner:: the chunk_owner4 () identifying the repair actor. The data server uses this to record which actor cleared the errored state. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear in cra_owner; see and .

CHUNK_REPAIRED returns a single top-level status; there is no per-chunk status array because the data server either accepts the confirmation for the whole range or returns a top-level error. The data server MUST verify before accepting the confirmation that:

every chunk in [cra_offset, cra_offset + cra_count) is currently in the errored state, and
every chunk in the range has been advanced to COMMITTED by a CHUNK_WRITE_REPAIR / CHUNK_FINALIZE / CHUNK_COMMIT sequence since the CHUNK_ERROR that marked them errored.

If either precondition fails, the data server returns NFS4ERR_INVAL and the errored state is left in place. A repair actor that sees NFS4ERR_INVAL SHOULD verify the chunks via CHUNK_HEADER_READ before retrying. If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

RESPONSE CODES

NFS4_OK:: the repair confirmation has been recorded.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to confirm repair on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_INVAL:: the chunks named were not in an errored state, or the repair did not match the recorded error.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_REPAIRED.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.

Operation 85: CHUNK_ROLLBACK - Rollback Changes on Cached Chunk Data

ARGUMENTS

XDR for CHUNK_ROLLBACK4args ; /// }; ]]>

RESULTS

XDR for CHUNK_ROLLBACK4resok ; /// }; ]]>

XDR for CHUNK_ROLLBACK4res

DESCRIPTION CHUNK_ROLLBACK reverts chunks from the PENDING or FINALIZED state to their previous state, effectively undoing a CHUNK_WRITE that has not yet reached COMMITTED via CHUNK_COMMIT. The reversion target is the prior COMMITTED generation, if one exists for the affected chunk; otherwise the chunk returns to the EMPTY state (). CHUNK_ROLLBACK against a chunk already in the COMMITTED state is permitted only on the repair path; see "Rollback of COMMITTED Chunks" below. CHUNK_ROLLBACK has no direct analog in : NFS WRITE has no separate finalization or commit step that a client could undo without contacting other components. CHUNK_ROLLBACK exists because the chunk state machine exposes the PENDING and FINALIZED states explicitly, and the writer needs a way to abandon a non-committed generation without committing it. The client provides cra_offset and cra_count to bound the chunk range, and cra_chunks to name the specific (chunk_owner4) generations within that range to roll back:

cra_offset:: starting chunk index in the file (not a byte offset).
cra_count:: number of chunks the range covers, starting at cra_offset.
cra_chunks:: an array of chunk_owner4 entries () naming the specific generations to roll back. For each entry the data server looks up the chunk index it associated with the complete owner triple when the client wrote it via CHUNK_WRITE or CHUNK_WRITE_REPAIR. That recorded chunk index MUST lie in [cra_offset, cra_offset + cra_count); if the triple does not match any recorded owner association on this data server for this file, or the recorded chunk index lies outside the requested range, the entry is rejected with NFS4ERR_INVAL in the corresponding crr_chunk_status slot. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear as the co_client_id of any cra_chunks entry; see and .

The CHUNK_ROLLBACK result returns:

crr_writeverf:: a verifier identifying the data server's incarnation. Semantics match cwr_writeverf in CHUNK_WRITE.
crr_chunk_status:: per-chunk rollback status, one entry per cra_chunks entry, co-indexed. NFS4_OK indicates that the named generation was reverted (either to a retained predecessor COMMITTED or FINALIZED generation, or to EMPTY when no predecessor exists). NFS4ERR_INVAL indicates the data server holds a concrete invalidation context that identifies the presented triple (structurally invalid, or released by an explicit CHUNK_ROLLBACK delete case within the session slot's replay-cache window), or the recorded chunk index lies outside [cra_offset, cra_offset + cra_count). NFS4ERR_NO_PREDECESSOR indicates the data server holds no association for the presented triple and no concrete invalidation context -- the release-scope split is defined normatively at . Other per-entry failures use the appropriate error code; the top-level operation status is NFS4_OK as long as the data server could evaluate each entry.

CHUNK_ROLLBACK has two principal scenarios:

A writer in multiple-writer mode that observed per-chunk failures in the CHUNK_WRITE response (e.g., NFS4ERR_CHUNK_GUARDED on a subset of chunks) needs to abandon the partial write before issuing CHUNK_FINALIZE on the chunks that did succeed. CHUNK_ROLLBACK on the abandoned chunks releases their PENDING generation cleanly.
A repair actor that wrote reconstructed data via CHUNK_WRITE_REPAIR and subsequently discovered the reconstruction was wrong (for example, a CRC mismatch detected during cross-mirror verification) needs to abandon the repair before any client commits it.

The data server effects the rollback as follows:

Chunks in PENDING with a matching chunk_owner4:: the data server deletes the PENDING payload and restores the chunk to its prior state (EMPTY, or the prior COMMITTED generation if the rollback invariant in required retention).
Chunks in FINALIZED with a matching chunk_owner4:: the data server deletes the FINALIZED payload and the persisted finalization metadata, restoring the chunk to its prior state.
Chunks not in PENDING or FINALIZED at the named generation, or whose chunk_owner4 does not match:: the corresponding crr_chunk_status slot reports NFS4ERR_INVAL or NFS4ERR_NO_PREDECESSOR per the release-scope split at , and the chunk is left unchanged.

Deletion Atomicity and Invalidated Triples CHUNK_ROLLBACK's delete case (successful rollback of a PENDING or FINALIZED generation, or the abandonment sub-case of the rollback of COMMITTED chunks below where a displaced successor is discarded) removes BOTH the payload AND the recorded owner-to-index association for that generation, in one atomic step, per the payload/association biconditional (). The generation's owner triple is then INVALIDATED at that chunk index: it names no generation the data server holds, and no future CHUNK_WRITE recreates an association under the same triple. A subsequent lifecycle operation (CHUNK_COMMIT, CHUNK_FINALIZE, CHUNK_ROLLBACK) that names an invalidated triple MUST NOT be treated as resurrecting the deleted generation; the data server MUST NOT attempt to match the triple against any other record. The error returned in the corresponding per-chunk slot depends on the release scope observable to the caller:

Within the session slot's replay-cache window of the CHUNK_ROLLBACK that performed the delete case (or an equivalent slotted retransmission of it), the slot reports NFS4ERR_INVAL -- the caller could have observed the specific invalidating operation.
After that replay-cache window has elapsed, or for release under any other terminal transition (lease expiry, storage-pressure release, retention-scope release of an already-invalidated triple), the slot reports NFS4ERR_NO_PREDECESSOR -- the data server holds no association for the presented triple and cannot distinguish it from any other released generation.

Both errors carry the same non-resurrection guarantee above; they differ only in what the caller can observe about how the association was released (see ). The invalidation is on the FULL owner triple, not on any sub-part. A client that legitimately reuses the same co_id value under a different co_cohort_id (or against a different co_client_id) creates a distinct triple and therefore a distinct generation identity; that new triple is unaffected by the earlier deletion. This is what allows a writer that abandoned a generation via CHUNK_ROLLBACK to retry with a fresh co_cohort_id -- the retry is a new generation, not a resurrection of the deleted one. The uncertain-replay carve-out (the CHUNK_ROLLBACK "Idempotence and Uncertain-Replay Carve-Out" text below) is a narrow exception: an EXACT reissue of the same CHUNK_ROLLBACK op whose prior completion was uncertain MAY be treated as postcondition-equivalent success when the client independently verifies the deletion postcondition holds. This is separate from -- and does NOT resurrect -- the invalidated triple; it merely acknowledges that the prior op already achieved the deletion the caller requested.

Rollback of COMMITTED Chunks CHUNK_ROLLBACK against a COMMITTED chunk is permitted ONLY on the repair path, when a repair actor is restoring a prior COMMITTED generation that another client incorrectly advanced. Two cases separate by whether the predecessor generation the caller wants to restore is still present on the data server:

Case (a) -- Retained Predecessor The predecessor generation named in the cra_chunks entry is still held by the data server (typically the prior COMMITTED retained under the rollback invariant () alongside the displaced successor, and MUST have survived with its payload+association pair intact per the biconditional ()). The data server:

restores the retained predecessor as the current COMMITTED generation UNDER ITS ORIGINAL owner triple (the owner triple the predecessor was written with) -- its owner-to-index association is preserved, so the restored generation MUST remain recognizable to subsequent lifecycle operations by the same triple; AND
atomically invalidates the displaced successor's triple via the delete case above ("Deletion Atomicity and Invalidated Triples") -- the displaced successor's payload+association pair is released as one unit; a subsequent lifecycle op naming the displaced triple returns NFS4ERR_INVAL within the session slot's replay-cache window of this CHUNK_ROLLBACK, and NFS4ERR_NO_PREDECESSOR after that window has elapsed, per the release-scope split at .

The restore is atomic with the delete: no intermediate state exposes both generations as current, and no intermediate state exposes neither.

Case (b) -- Predecessor No Longer Retained If the predecessor generation named in the cra_chunks entry is NOT held by the data server -- its payload+association pair was released some time earlier under the retention scope rule (lease expiry, an earlier CHUNK_ROLLBACK delete case, or any other terminal transition) -- the CHUNK_ROLLBACK CANNOT restore it. The deleted association is not recreated by any operation defined in this document. The corresponding crr_chunk_status slot reports:

NFS4ERR_INVAL:: when the caller has named a triple released by an explicit CHUNK_ROLLBACK delete case within the current session slot's replay-cache window.
NFS4ERR_NO_PREDECESSOR:: in the ordinary case -- release under the retention scope rule, or eventual release of an already-invalidated triple after the delete case's replay-cache window has elapsed. See for the choice between the two codes.

The caller then consults whatever fallback the deployment provides. A repair actor MAY reconstruct authoritative bytes from surviving shards and issue CHUNK_WRITE_REPAIR to write a new generation carrying those bytes under a new owner triple. That new generation is a distinct generation for lifecycle purposes; it is not the deleted predecessor resurrected. A client that requires the restored generation to retain the predecessor's original owner triple, in cases where the retention scope () would otherwise permit release, MUST use the metadata-server escrow control plane ( through ). Escrow pins the predecessor's payload and its owner-to-index association jointly against that release rule for as long as the escrow-lock or a client-owned lock adopted from it remains in continuous custody (see ). A non-repair CHUNK_ROLLBACK against a COMMITTED chunk is rejected with NFS4ERR_INVAL regardless of case.

Stateid and Authorization Like CHUNK_COMMIT, CHUNK_ROLLBACK carries an explicit layout stateid in cra_stateid. The data server authorizes CHUNK_ROLLBACK by validating cra_stateid against the file identified by the current filehandle: cra_stateid MUST be the layout stateid the metadata server issued to the caller for the current filehandle, or the special anonymous stateid (see below). Under trusted stateid tight coupling (), the data server rejects CHUNK_ROLLBACK with NFS4ERR_BAD_STATEID unless cra_stateid is present in the data server's trust table for the current filehandle. The explicit field ensures that a CHUNK_ROLLBACK in its own standalone compound (typical for retry after a failed CHUNK_WRITE cohort) carries the authorization the data server needs. Passing the special anonymous stateid is permitted only when the underlying security regime authorizes an unattributed writer. If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

Idempotence and Uncertain-Replay Carve-Out CHUNK_ROLLBACK is idempotent on its target: a second CHUNK_ROLLBACK naming the same generations after the first has succeeded finds those generations already in the state the first produced. Under the ordinary per-entry rules the second call returns NFS4ERR_INVAL in each crr_chunk_status slot within the session slot's replay-cache window of the first call (the first call's delete case invalidated the triples per "Deletion Atomicity and Invalidated Triples" above, or the retained-predecessor restore invalidated the displaced successor's triple per "Rollback of COMMITTED Chunks" above); after that window has elapsed the second call returns NFS4ERR_NO_PREDECESSOR per the release-scope split at .

Uncertain-replay carve-out:: When the first CHUNK_ROLLBACK completed at the data server but its response was lost (network error, dropped connection, data server restart before the reply was received), the client cannot distinguish "the op did not run" from "the op ran and its reply was lost." A client that issues an EXACT REISSUE of the same CHUNK_ROLLBACK op under these conditions MAY treat the resulting per-chunk NFS4ERR_INVAL (within the replay-cache window) or NFS4ERR_NO_PREDECESSOR (after the window has elapsed) as POSTCONDITION-EQUIVALENT SUCCESS on a slot-by-slot basis, PROVIDED all of the following hold for that slot:

the reissue is byte-identical to the original op (same cra_offset, same cra_count, same cra_chunks array entry) -- a fresh op with an accidentally-matching triple does NOT qualify;
the prior completion is genuinely uncertain (the client never observed a per-entry response for that slot); AND
the client INDEPENDENTLY verifies that the target postcondition holds. For the delete case, this means observing that the named generation is absent at the target chunk index (via CHUNK_HEADER_READ or CHUNK_READ with the retention-scope rules understood). For the retained-predecessor restore case, this means observing that the predecessor's original triple is now the current COMMITTED generation.

The carve-out is narrow by construction: a fresh op receiving NFS4ERR_INVAL or NFS4ERR_NO_PREDECESSOR never qualifies, and a reissue that cannot verify the postcondition also does not qualify -- the client MUST treat the error as a terminal per-entry failure in either case. This prevents the carve-out from being conflated with the underlying non-resurrection guarantee, which is unconditional and prohibits the data server from resurrecting any deleted generation ("Deletion Atomicity and Invalidated Triples" above). Similar reasoning applies to exact uncertain reissues of CHUNK_COMMIT and CHUNK_FINALIZE, which are also idempotent on their targets under the same three predicates.

RESPONSE CODES

NFS4_OK:: the named chunks have been rolled back.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to roll back chunks on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_INVAL:: arguments named chunks not eligible for rollback or outside the file's mirror set.
NFS4ERR_NO_PREDECESSOR:: the named predecessor has no recorded owner-to-index association on the data server (retention scope released it, or the delete case's replay-cache window has elapsed). See for the split against NFS4ERR_INVAL.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_ROLLBACK.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.

Operation 86: CHUNK_UNLOCK - Unlock Cached Chunk Data

ARGUMENTS

XDR for CHUNK_UNLOCK4args

RESULTS

XDR for CHUNK_UNLOCK4res

DESCRIPTION CHUNK_UNLOCK releases the exclusive chunk-range lock previously acquired by CHUNK_LOCK. CHUNK_UNLOCK is loosely analogous to LOCKU () in that it releases an exclusive guard, but it operates on chunk-range coordinates and is matched against the chunk_owner4 that acquired the lock rather than against an open / lock stateid. The client provides:

cua_stateid:: the layout stateid the metadata server granted for this file. Under trusted stateid tight coupling (), this stateid MUST be in the data server's trust table; otherwise the data server rejects the operation with NFS4ERR_BAD_STATEID.
cua_offset:: starting chunk index of the range to unlock (not a byte offset).
cua_count:: number of chunks the unlock range covers, starting at cua_offset. The range MUST match exactly the range of an outstanding CHUNK_LOCK held by cua_owner; partial-range unlock is not supported.
cua_owner:: the chunk_owner4 () that holds the lock. The co_client_id MUST match the chunk_owner4 that was supplied on the CHUNK_LOCK that acquired the lock (including the case of a lock transferred via CHUNK_LOCK_FLAGS_ADOPT, in which the adopter's chunk_owner4 is the current holder). The reserved sentinel values CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear as the co_client_id of cua_owner; see and . In particular, a repair actor releasing a lock it adopted from the metadata-server escrow owner uses its own co_client_id in cua_owner, not CHUNK_GUARD_CLIENT_ID_MDS.

The CHUNK_UNLOCK result returns a single top-level status; there is no per-chunk status array because the unlock either succeeds for the whole range or returns a top-level error. CHUNK_UNLOCK is idempotent in the sense that releasing chunks that are not currently locked returns NFS4_OK without effect. Releasing chunks that are locked by a different cua_owner returns NFS4ERR_INVAL and leaves the lock in place. A client SHOULD issue CHUNK_UNLOCK promptly after completing the write, write-repair, or commit sequence that the lock guarded. Locks not explicitly released are released implicitly when the holder's lease expires; if the metadata server has revoked the holder's stateid via REVOKE_STATEID before the lease lapses, the lock transitions to the metadata-server escrow owner per the lock-continuity invariant in rather than being released outright. If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

RESPONSE CODES

NFS4_OK:: the named chunks have been unlocked, or no lock was held (idempotent).
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to unlock chunks on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_INVAL:: arguments named chunks not in a locked state owned by this caller.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_UNLOCK.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.

Operation 87: CHUNK_WRITE - Write Chunks to File

ARGUMENTS

XDR for write_chunk_guard4

XDR for CHUNK_WRITE4args ; /// uint32_t cwa_payload_id; /// uint32_t cwa_flags; /// write_chunk_guard4 cwa_guard; /// uint32_t cwa_chunk_size; /// checksum4 cwa_checksums<>; /// opaque cwa_chunks<>; /// }; ]]>

RESULTS

XDR for CHUNK_WRITE4resok ; /// bool cwr_block_activated<>; /// chunk_owner4 cwr_owners<>; /// }; ]]>

XDR for CHUNK_WRITE4res

DESCRIPTION The CHUNK_WRITE operation is based upon the NFSv4.1 WRITE operation (see ) and similarly writes data to the regular file identified by the current filehandle, with the difference that CHUNK_WRITE operates on the chunk coordinate system used by Flexible File Version 2 layouts rather than on the byte coordinate system. Successful chunk writes initially enter the PENDING state in the chunk state machine (); a subsequent CHUNK_FINALIZE and CHUNK_COMMIT (or the activation shortcut described below) progress them to COMMITTED. The client provides a cwa_offset of where the CHUNK_WRITE is to start and a payload consisting of one or more chunks packed into the cwa_chunks opaque field. cwa_offset is the starting chunk index in the file (not a byte offset); each chunk occupies cwa_chunk_size bytes within cwa_chunks except the last, which MAY be shorter when the file size is not chunk-aligned or when the payload encodes a variable-size Mojette parity shard (). The number of chunks in the payload is ceil(len(cwa_chunks) / cwa_chunk_size). The writer supplies the cohort identity of the payload in compact form: cwa_cohort_id carries the shared chunk_cohort_id4 for every chunk in the batch, cwa_client_id carries the writer's layout-granted identity, and cwa_co_ids is an array of writer-chosen opaque per-chunk co_id values, one per chunk in the payload (see ). The full cohort triple recorded on the data server for the chunk at zero-based position i within the payload is (cwa_cohort_id, cwa_client_id, cwa_co_ids[i]). cwa_client_id identifies the writer. It is client-presented, metadata-server-assigned: the client presents the 32-bit layout-granted identity that the metadata server established in ffv2m_client_id (see ) at layout-grant time; the client MUST NOT substitute any other value. Because encodings that use CHUNK operations require tight coupling (), the data server always has an authoritative binding for this identity: the metadata server registers it via tsa_client_id in TRUST_STATEID alongside the layout stateid. The data server MUST compare cwa_client_id against the tsa_client_id recorded in its trust table for the presented layout stateid, and MUST reject a mismatch with NFS4ERR_BAD_STATEID: a client that presents a cwa_client_id different from its layout's ffv2m_client_id is spoofing another writer's identity. cwa_client_id MUST NOT be the reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE or CHUNK_GUARD_CLIENT_ID_MDS (see and ); those sentinels are reserved for data-server-observed cg_client_id values in chunk_guard4 CAS state, not for use as a writer's own identity. cwa_cohort_id is the writer's opaque per-transaction identifier, shared by every chunk in the payload; the writer chooses a new cwa_cohort_id per distinct write transaction so that lifecycle operations can distinguish transactions across all data files (see the discussion of chunk_cohort_id4 in ). The per-chunk CAS state (chunk_guard4) is carried separately via cwa_guard.cwg_guard when compare-and-swap is required (). cwa_co_ids<> contains exactly one entry per chunk in the payload (i.e., cwa_co_ids_len MUST equal ceil(len(cwa_chunks) / cwa_chunk_size)); the data server rejects any other length with NFS4ERR_INVAL. Within a single (cwa_cohort_id, cwa_client_id), the cwa_co_ids values in this request MUST be distinct so that lifecycle operations (CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK) can name individual chunks unambiguously by full triple. The data server treats each cwa_co_ids[i] as opaque; it does NOT interpret the value beyond equality comparison, and it MUST NOT require cwa_co_ids[i] to equal cwa_offset + i or any other derived index. A writer that finds it convenient to use monotonic per-writer serials or file-index-derived values MAY do so, but the wire semantics do not privilege that choice. cwa_payload_id is a writer-chosen identifier that lets a repair coordinator correlate chunks of the same logical write across data servers. cwa_checksums, when non-empty, MUST contain one checksum entry per chunk in the payload. Each entry's cs_algorithm MUST match ffv2m_checksum_algorithm of the mirror named in the layout (see ); a mismatch is rejected with NFS4ERR_INVAL. The data server validates each chunk's checksum at CHUNK_WRITE time and rejects mismatched chunks with NFS4ERR_IO in the corresponding cwr_block_status slot. An empty cwa_checksums array (cwa_checksums_len = 0) indicates the client did not supply per-chunk checksums; the data server still computes and persists per-chunk checksums from the payload bytes for later integrity verification but cannot detect transport corruption at CHUNK_WRITE time without the client's reference values. cwa_flags carries CHUNK_WRITE_FLAGS_ACTIVATE_IF_EMPTY (see "Stability and Activation" below). cwa_guard () controls the chunk-guard CAS check (see "Guarding the Write" below). A cwa_offset of zero starts writing at the first chunk of the file. Unlike READ in , a CHUNK_WRITE whose cwa_offset extends beyond the current end of the file is not an error: the data server extends the file's chunk store to cover the new chunks, with intervening offsets remaining EMPTY () until they too are written. If the cwa_chunks payload is empty (zero bytes), the CHUNK_WRITE succeeds and writes zero chunks (cwr_count = 0). In all situations the data server MAY choose to write fewer chunks than the client requested; the client must be prepared to handle a short write and reissue CHUNK_WRITE for the remaining chunks. The CHUNK_WRITE result includes per-chunk outcomes in cwr_block_status, cwr_block_activated, and cwr_owners, all co-indexed and one entry per chunk in the payload:

cwr_count:: the number of chunks the data server successfully accepted. Chunks that failed their guard check, checksum check, or any other local precondition do not contribute to cwr_count.
cwr_committed:: the stable_how4 level the data server actually applied for accepted chunks. This MUST be at least as durable as cwa_stable; see "Stability and Activation" below.
cwr_writeverf:: a verifier identifying the data server's incarnation. A client uses cwr_writeverf to detect a data server restart that lost UNSTABLE4 writes: if the client's subsequent CHUNK_COMMIT returns a different writeverf than was returned by an UNSTABLE4 CHUNK_WRITE earlier, the chunks may have been lost and the client SHOULD re-issue CHUNK_WRITE. cwr_writeverf changes on every data server restart that loses uncommitted state.
cwr_block_status:: per-chunk acceptance status; see "Per-Block Acceptance Semantics" below.
cwr_block_activated:: per-chunk activation flag. TRUE indicates that the chunk is COMMITTED on return from CHUNK_WRITE -- the activation shortcut described under "Stability and Activation" below. FALSE indicates that the chunk is in the PENDING state and requires a subsequent CHUNK_FINALIZE and CHUNK_COMMIT to become COMMITTED.
cwr_owners:: per-chunk chunk_owner4 the data server recorded, one entry per chunk in the payload (co-indexed with cwa_co_ids and cwr_block_status). Each entry is the complete cohort triple (cwa_cohort_id, cwa_client_id, cwa_co_ids[i]) that the data server recorded (or, for a rejected slot, the requested triple echoed for positional correlation with the failure status in cwr_block_status[i]). The data server does NOT synthesize per-chunk identity by modifying the writer's cohort; the returned triple is the value the client supplied, so a client that lost track of its own transmitted co_ids array can recover the data server's view. Under the compact carrier the cohort pair (cwa_cohort_id, cwa_client_id) is shared across every returned entry -- the per-chunk distinction is carried entirely by the co_id.

Except when special stateids are used, cwa_stateid represents a layout stateid returned by a prior LAYOUTGET against the metadata server (see ) that authorizes write access to this file. Under trusted stateid tight coupling (), the data server additionally checks that the metadata server has registered the stateid via TRUST_STATEID; an unregistered stateid (other than a special stateid) returns NFS4ERR_BAD_STATEID. For a CHUNK_WRITE with a cwa_stateid value of all bits equal to zero, the data server MAY allow the CHUNK_WRITE to be serviced subject to any CHUNK_LOCK currently held on the target chunks. For a CHUNK_WRITE with a cwa_stateid value of all bits equal to one, the data server MAY allow CHUNK_WRITE to bypass lock-state checking at the data server. These special-stateid behaviors mirror the corresponding WRITE semantics in adapted to the chunk-locking model () rather than the byte range locking model of If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

Stability and Activation The cwa_stable field controls the durability level the data server guarantees before returning. The durability floor for each level covers BOTH the chunk payload AND its owner-to-index association, so a subsequent lifecycle operation naming the full owner triple can locate the recorded chunk index; see for the normative coupling rule and the "MUST NOT retain payload without association" invariant.

FILE_SYNC4:: The data server MUST commit all written chunks plus all chunk-store metadata to stable storage before returning.
DATA_SYNC4:: The data server MUST commit all written chunk payloads to stable storage and enough of the chunk-store metadata to retrieve the data before returning. An implementation MAY treat DATA_SYNC4 identically to FILE_SYNC4 at a possible performance cost.
UNSTABLE4:: The data server is free to commit any portion of the chunk payload and metadata to stable storage before returning, including all or none. The data server makes only two guarantees: it will not destroy any chunk payload it accepted without changing cwr_writeverf, and the durability level it ultimately applies will not be less than that requested.

The CHUNK_WRITE_FLAGS_ACTIVATE_IF_EMPTY flag in cwa_flags requests an activation shortcut for first-time writes: a chunk that was EMPTY before the CHUNK_WRITE and whose write reaches FILE_SYNC4 or DATA_SYNC4 durability MAY be transitioned directly to COMMITTED by the data server, with the corresponding cwr_block_activated entry set to TRUE in the response. Without the flag, or for chunks that were not EMPTY before the write, or for writes at UNSTABLE4 durability, the chunk enters the PENDING state and reaches COMMITTED only after a subsequent CHUNK_FINALIZE and CHUNK_COMMIT. The activation shortcut interacts with concurrent writers and unstable writes in subtle ways:

A chunk written with cwa_stable = UNSTABLE4 cannot be activated by CHUNK_WRITE_FLAGS_ACTIVATE_IF_EMPTY because the payload has not been committed to stable storage; the chunk enters the PENDING state regardless of the flag.
In the multi-writer mode () a client MUST NOT set CHUNK_WRITE_FLAGS_ACTIVATE_IF_EMPTY on any CHUNK_WRITE. The activation shortcut bypasses CHUNK_FINALIZE, so a chunk that reached COMMITTED via the shortcut cannot be reverted by the loser's CHUNK_ROLLBACK flow described in ; two racing writers whose EMPTY activations landed on disjoint subsets of the mirror set would leave the stripe permanently non-atomic with no protocol path to recovery. A data server that receives a multi-writer-mode CHUNK_WRITE with CHUNK_WRITE_FLAGS_ACTIVATE_IF_EMPTY set MUST reject the request with NFS4ERR_INVAL in the corresponding cwr_block_status slot; the client MUST re-issue without the flag, allowing the chunk to enter PENDING and proceed through the standard CHUNK_FINALIZE + CHUNK_COMMIT path. The shortcut remains available in single writer mode where CAS contention cannot arise.
A client that issues an UNSTABLE4 CHUNK_WRITE and observes a FALSE entry in cwr_block_activated for a chunk MAY still find that chunk COMMITTED on a subsequent CHUNK_READ -- another client could have activated it via the shortcut after this one's response was sent. cwr_block_activated reflects the state at the moment the CHUNK_WRITE result was constructed, not a commitment to that state's persistence.

Guarding the Write A guarded CHUNK_WRITE is when the writing of a block MUST fail if cwa_guard.cwg_check is TRUE and the target chunk does not have the same cg_gen_id as cwa_guard.cwg_guard.cg_gen_id. This is useful in read-update-write scenarios. The client reads a block, updates it, and is prepared to write it back. It guards the write such that if another writer has modified the block, the data server will reject the modification. As the chunk_guard4 (see ) does not have a chunk_id and the CHUNK_WRITE applies to all blocks in the range of cwa_offset to the length of cwa_data, then each of the target blocks MUST have the same cg_gen_id and cg_client_id. The client SHOULD present the smallest set of blocks as possible to meet this requirement.

Per-Block Acceptance Semantics A CHUNK_WRITE targets a contiguous range of blocks on a single data server. The data server evaluates each block independently and reports the outcome per block in cwr_block_status (see ):

Each block is subjected to the guard check (when cwa_guard.cwg_check is TRUE), the cg_client_id validation (see ), and any other local preconditions (storage-space limits, tight coupling trust-table state, etc.).
Blocks that pass their preconditions are written and their cwr_block_status entry is NFS4_OK. Blocks that fail produce the appropriate error code (NFS4ERR_CHUNK_GUARDED, NFS4ERR_NOSPC, etc.) in the corresponding cwr_block_status slot, and their data is NOT persisted.
cwr_count reflects only the blocks that were written successfully; failed blocks do not contribute.
The top-level cwr_status is NFS4_OK when the call itself was structurally valid and the data server could evaluate each block. Per-block failures are reported in cwr_block_status, not by failing the whole operation. The data server returns a top-level error only if it could not evaluate the request at all.

This is the "continue and report" discipline. It is intentionally not all-or-none: atomicity is already per-chunk (see ), so there is no file-level correctness reason to reject the entire compound because of a single chunk guard failure. Per-block reporting gives the client the information it needs to construct a targeted CHUNK_ROLLBACK or CHUNK_WRITE retry that covers only the blocks that failed. The data server does not hold a file-wide lock across the per-block evaluation. The chunk_guard4 CAS is evaluated atomically per chunk at the point the data server updates that chunk's state, so an interleaving CHUNK_WRITE from a different client that arrives mid-compound will either win its own CAS race (and the losing client sees NFS4ERR_CHUNK_GUARDED for the contested block) or be rejected itself, without introducing data-server-level locking beyond the per-chunk scope.

RESPONSE CODES

NFS4_OK:: the chunks have been written and are in the PENDING state.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to write to this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_CHUNK_GUARDED:: the chunk_guard4 condition supplied by the client did not match the persisted state.
NFS4ERR_CHUNK_LOCKED:: one or more chunks in the requested range are locked by another writer.
NFS4ERR_DELAY:: the data server is temporarily unable to process the request.
NFS4ERR_FHEXPIRED:: the current filehandle has expired.
NFS4ERR_IO:: an I/O error occurred while persisting the chunks.
NFS4ERR_NOSPC:: there is insufficient space at the data server.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_WRITE.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE:: the current filehandle no longer identifies a valid file.

Operation 88: CHUNK_WRITE_REPAIR - Write Repaired Cached Chunk Data

ARGUMENTS

XDR for CHUNK_WRITE_REPAIR4args ; /// uint32_t cwra_payload_id; /// uint32_t cwra_chunk_size; /// checksum4 cwra_checksums<>; /// opaque cwra_chunks<>; /// }; ]]>

RESULTS

XDR for CHUNK_WRITE_REPAIR4resok ; /// chunk_owner4 cwrr_owners<>; /// }; ]]>

XDR for CHUNK_WRITE_REPAIR4res

DESCRIPTION CHUNK_WRITE_REPAIR is the repair-path variant of CHUNK_WRITE. It writes reconstructed chunk data to a data server whose chunks have been reported errored (via CHUNK_ERROR, see ) or to a replacement data server selected during whole-file repair. The data server applies repair-specific policies to the write that are not appropriate for normal client writes: the chunk_guard4 CAS check is bypassed (the repair actor is writing a reconstructed value rather than competing in a multiple-writer race), and the data server MAY log the repair separately for operator audit. CHUNK_WRITE_REPAIR has no direct analog in ; it is the chunk-protocol equivalent of writing reconstructed data into a RAID stripe whose other members are known healthy. The reconstructed data is produced by the repair client from surviving shards via the erasure-coding algorithm of the file's layout. The repair workflow that invokes CHUNK_WRITE_REPAIR is:

The repair actor (selected per ) reads surviving chunks from the remaining data servers via CHUNK_READ.
The repair actor reconstructs the missing chunks using the erasure-coding algorithm of the file's layout.
The repair actor acquires a CHUNK_LOCK on the target data server to prevent concurrent writes during repair. For repair that adopts a metadata-server escrow lock, the CHUNK_LOCK carries CHUNK_LOCK_FLAGS_ADOPT ().
The repair actor writes the reconstructed data via CHUNK_WRITE_REPAIR.
The repair actor issues CHUNK_FINALIZE and CHUNK_COMMIT to persist the repair.
The repair actor issues CHUNK_REPAIRED to clear the errored state.
The repair actor releases the lock via CHUNK_UNLOCK.

CHUNK_WRITE_REPAIR is also the fallback path used by a client that received NFS4ERR_NO_PREDECESSOR () from CHUNK_ROLLBACK's restore case ( "Rollback of COMMITTED Chunks", case (b)) -- the named predecessor was released under the retention scope () and CHUNK_ROLLBACK cannot restore it. Under this fallback the client reconstructs authoritative bytes from surviving shards (or from any other authoritative source it holds) and writes them via CHUNK_WRITE_REPAIR under a new owner triple of its own choosing. The result is a distinct generation for lifecycle purposes: it carries the repair actor's own owner triple, NOT the released predecessor's triple. The released predecessor is not resurrected by any operation defined in this document, including this fallback; a subsequent lifecycle operation naming the released predecessor's triple returns NFS4ERR_NO_PREDECESSOR under the release-scope split at -- the predecessor was released under the retention scope, not by an explicit CHUNK_ROLLBACK delete case within a live replay-cache window. A client that requires the restored generation to carry the released predecessor's original triple MUST use a guaranteed-pinning mechanism as noted in , since CHUNK_WRITE_REPAIR's semantics only produce a new generation. When no authoritative source exists for reconstruction, the fallback itself may terminate at NFS4ERR_PAYLOAD_LOST () via CB_CHUNK_REPAIR. The arguments mirror CHUNK_WRITE except that CHUNK_WRITE_REPAIR has no cwa_flags field (the activation-shortcut behavior is not offered on the repair path) and no cwa_guard field (the guard CAS is bypassed by construction):

cwra_stateid:: the layout stateid the metadata server granted to the repair actor. Under trusted stateid tight coupling (), this stateid MUST be in the data server's trust table; otherwise the data server rejects the operation with NFS4ERR_BAD_STATEID.
cwra_offset:: starting chunk index in the file (not a byte offset).
cwra_stable:: the stable_how4 durability level the data server MUST apply before returning. Semantics match cwa_stable in CHUNK_WRITE (see "Stability and Activation").
cwra_cohort_id / cwra_client_id / cwra_co_ids:: the compact cohort-identity carrier the repair actor uses for the reconstructed payload, mirroring CHUNK_WRITE's cwa_cohort_id + cwa_client_id + cwa_co_ids per . cwra_cohort_id carries the shared chunk_cohort_id4 for every reconstructed chunk in the batch; cwra_client_id is the repair actor's layout-granted identity; cwra_co_ids is a co-indexed array of writer-chosen opaque co_id values, one per chunk in cwra_chunks (exactly ceil(len(cwra_chunks) / cwra_chunk_size) entries; a length mismatch is rejected with NFS4ERR_INVAL). The full cohort triple for reconstructed chunk position i is (cwra_cohort_id, cwra_client_id, cwra_co_ids[i]). cwra_client_id MUST be the repair actor's own ffv2m_client_id (not CHUNK_GUARD_CLIENT_ID_MDS); the cwra_cohort_id is the repair actor's locally chosen per-transaction opaque identifier. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear in cwra_client_id; see and . Within a single (cwra_cohort_id, cwra_client_id), the cwra_co_ids values MUST be distinct so that lifecycle operations can name individual reconstructed chunks unambiguously by full triple.
cwra_payload_id:: the payload-id the repair actor associates with the reconstructed payload, used by the repair coordinator to correlate this repair across mirrors.
cwra_chunk_size:: the nominal chunk size of the reconstructed payload, in bytes.
cwra_checksums:: per-chunk checksum4 () array. Semantics match cwa_checksums in CHUNK_WRITE: each entry's cs_algorithm MUST match ffv2m_checksum_algorithm of the mirror named in the layout (), with NFS4ERR_INVAL on mismatch.
cwra_chunks:: the reconstructed chunk payload as an opaque blob, packed identically to cwa_chunks in CHUNK_WRITE.

The CHUNK_WRITE_REPAIR result reports per-chunk outcomes:

cwrr_count:: the number of chunks the data server successfully accepted.
cwrr_committed:: the stable_how4 level the data server actually applied. MUST be at least as durable as cwra_stable.
cwrr_writeverf:: a verifier identifying the data server's incarnation. Semantics match cwr_writeverf in CHUNK_WRITE.
cwrr_block_status:: per-chunk acceptance status, one entry per chunk in the payload, co-indexed with cwra_co_ids and cwrr_owners. The top-level CHUNK_WRITE_REPAIR status is NFS4_OK as long as the data server could evaluate each chunk; per-chunk failures are reported in cwrr_block_status rather than by failing the whole operation.
cwrr_owners:: per-chunk chunk_owner4 the data server recorded, one entry per chunk in the payload (co-indexed with cwra_co_ids and cwrr_block_status). Each entry is the complete cohort triple (cwra_cohort_id, cwra_client_id, cwra_co_ids[i]) that the data server recorded (or, for a rejected slot, the requested triple echoed for positional correlation with the failure status in cwrr_block_status[i]). The data server does NOT synthesize per-chunk identity by modifying the repair actor's cohort.

The target chunks SHOULD be in the errored state (set by a prior CHUNK_ERROR) or EMPTY. If a target chunk is COMMITTED with valid data, the data server MAY reject the repair-write with NFS4ERR_INVAL in the corresponding cwrr_block_status slot to prevent overwriting good data; the repair actor SHOULD re-verify the chunk before attempting another repair-write on the same range. If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

RESPONSE CODES

NFS4_OK:: the repair-write succeeded.
NFS4ERR_ACCESS:: the layout stateid or credentials are not permitted to write repair data to this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See .
NFS4ERR_DELAY:: the data server is temporarily unable to process the request.
NFS4ERR_FHEXPIRED:: the current filehandle has expired.
NFS4ERR_IO:: an I/O error occurred while persisting the repair data.
NFS4ERR_NOSPC:: there is insufficient space at the data server.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_WRITE_REPAIR.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE:: the current filehandle no longer identifies a valid file.

Operation 89: TRUST_STATEID - Register Layout Stateid on Data Server

ARGUMENTS

XDR for TRUST_STATEID4args

RESULTS

XDR for TRUST_STATEID4res

DESCRIPTION TRUST_STATEID registers a layout stateid with the data server so that subsequent CHUNK operations presenting that stateid can be validated against the data server's per-file trust table. The registration also binds the stateid to the ffv2m_client_id (tsa_client_id) the metadata server assigned to the client at LAYOUTGET time; the data server uses that binding to authorize the writer identity carried in cwa_client_id on CHUNK_WRITE and in cg_client_id in any chunk_guard4 CAS (). TRUST_STATEID is the mechanism by which tight coupling (see ) is established between the metadata server and the data server for a particular layout. TRUST_STATEID has no analog in : pNFS layouts in do not register the layout stateid with data servers; data servers in the loose coupling model trust the synthetic uid/gid the metadata server inserts on each I/O (). TRUST_STATEID is the new metadata-server-to-data-server control-plane operation that replaces synthetic-uid fencing with per-client stateid-table validation for deployments that opt into tight coupling. TRUST_STATEID is a metadata-server-to-data-server operation; pNFS clients MUST NOT send it. The data server rejects any TRUST_STATEID received on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS at EXCHANGE_ID with NFS4ERR_PERM. The flag is self-declared, so this wire-level check is a hint, not authentication; the deployment-level requirement -- that only entities considered legitimate metadata servers can establish such a session -- is stated in . TRUST_STATEID operates on the current filehandle; a PUTFH naming the data server's file MUST precede it in the same compound (except in the capability probe case, where the current filehandle is the root). The metadata server provides:

tsa_layout_stateid:: the stateid the metadata server issued in the LAYOUTGET that produced this layout. MUST NOT be a special stateid (anonymous, invalid, read-bypass, or current). The sole exception is the capability probe described in : when the metadata server sends TRUST_STATEID with tsa_layout_stateid set to the anonymous stateid against the root filehandle, the data server MUST reject the request with NFS4ERR_INVAL -- that rejection is the positive response to the probe.
tsa_client_id:: the ffv2m_client_id () the metadata server assigned to the client in the layout that produced tsa_layout_stateid. The data server records tsa_client_id alongside tsa_layout_stateid in the trust-table entry and uses it as the authoritative writer identity for subsequent CHUNK operations: cwa_client_id on CHUNK_WRITE MUST equal the trust-table's tsa_client_id, and cg_client_id in any chunk_guard4 CAS submitted with the same stateid MUST also match. A mismatch MUST be rejected with NFS4ERR_BAD_STATEID. In the capability probe the metadata server SHOULD set tsa_client_id to zero (which is the reserved CHUNK_GUARD_CLIENT_ID_NONE, see ); the data server's NFS4ERR_INVAL response to the probe is unaffected by the field's value.
tsa_iomode:: the iomode of the layout (LAYOUTIOMODE4_READ or LAYOUTIOMODE4_RW). The data server MAY enforce this against the CHUNK operation presented later: a READ-iomode trust entry does not authorize CHUNK_WRITE.
tsa_expire:: the absolute time at which the trust entry becomes invalid if not renewed; see for the clock-synchronization assumption and lease-computation rule. The data server MUST reject a TRUST_STATEID whose tsa_expire has tv_nseconds >= 10^9 with NFS4ERR_INVAL.
tsa_principal:: the client's authenticated identity as verified by the metadata server at LAYOUTGET time. For RPCSEC_GSS clients this is the GSS display name (e.g., "alice@REALM"). For AUTH_SYS and TLS clients, tsa_principal MUST be the empty string, indicating that no principal binding is enforced on subsequent CHUNK operations. See .

If a trust entry already exists for the same tsa_layout_stateid on the same current filehandle, TRUST_STATEID atomically updates tsa_expire and tsa_principal; this is the renewal path (see ). At registration time the data server tags the new trust entry with the identity of the metadata server, derived from the clientid of the owning client of the control session on which TRUST_STATEID arrived. This tag is consulted by REVOKE_STATEID and BULK_REVOKE_STATEID so that revocation only affects entries registered by the same metadata server. In a multi-metadata-server deployment sharing a single data server, each metadata server registers and revokes only its own entries; the tag is opaque to pNFS clients and is not carried on the wire. TRUST_STATEID returns only a top-level status; there is no result body beyond the nfsstat4 discriminant. If the current filehandle is not an ordinary file (except in the capability-probe case, where the current filehandle is the root and the operation is expected to be rejected with NFS4ERR_INVAL), an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

RESPONSE CODES

NFS4_OK:: the trust entry is registered (or updated).
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: tsa_layout_stateid was a special stateid other than the anonymous stateid on the root filehandle.
NFS4ERR_DELAY:: the data server is temporarily unable to process the request; the metadata server SHOULD retry.
NFS4ERR_INVAL:: tsa_layout_stateid was the anonymous stateid and the current filehandle is not the root filehandle; tsa_expire is malformed; or the current filehandle is a directory (except in the capability-probe case).
NFS4ERR_NOFILEHANDLE:: no current filehandle is set.
NFS4ERR_NOTSUPP:: the data server does not implement TRUST_STATEID. This is the capability-probe response (see ).
NFS4ERR_PERM:: the request arrived on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.

Operation 90: REVOKE_STATEID - Revoke Registered Stateid on Data Server

ARGUMENTS

XDR for REVOKE_STATEID4args

RESULTS

XDR for REVOKE_STATEID4res

DESCRIPTION REVOKE_STATEID invalidates a single trust entry on the data server. Subsequent CHUNK operations that present the revoked stateid MUST fail with NFS4ERR_BAD_STATEID. REVOKE_STATEID is the per-file revoke counterpart to TRUST_STATEID -- registration and revocation form the matched pair that drives the per-file trust table for a tightly coupled deployment. REVOKE_STATEID has no analog in . revokes pNFS layouts via LAYOUTRETURN with a special all-files marker or via implicit lease expiry; REVOKE_STATEID is the new metadata-server-to-data-server operation that lets the metadata server force per-client invalidation at the data server without waiting for tsa_expire and without unsetting other clients' trust entries. REVOKE_STATEID is a metadata-server-to-data-server operation; pNFS clients MUST NOT send it. The data server rejects any REVOKE_STATEID received on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS at EXCHANGE_ID with NFS4ERR_PERM. The flag is self-declared, so this wire-level check is a hint, not authentication; the deployment-level requirement is stated in . REVOKE_STATEID operates on the current filehandle; a PUTFH naming the data server's file MUST precede it in the same compound. The metadata server provides:

rsa_layout_stateid:: the stateid to revoke. Together with the current filehandle this identifies the trust entry to remove. MUST NOT be a special stateid; the anonymous stateid is rejected with NFS4ERR_INVAL and other special stateids with NFS4ERR_BAD_STATEID.

The metadata server calls REVOKE_STATEID in any of the following situations:

CB_LAYOUTRECALL timeout:: the client did not return the layout within the recall timeout. REVOKE_STATEID terminates the client's ability to issue further I/O to the data server without waiting for tsa_expire.
LAYOUTERROR with NFS4ERR_ACCESS or NFS4ERR_PERM:: the data server rejected the client's I/O; the trust entry is stale and must be removed. This mirrors the fencing case in the loose-coupled model ().
Explicit LAYOUTRETURN:: the client returned the layout cleanly. The metadata server MAY issue REVOKE_STATEID at this time or MAY rely on tsa_expire; either is correct.

In-flight CHUNK operations that arrived before REVOKE_STATEID completes MAY be allowed to finish. The data server MUST NOT process new CHUNK operations presenting rsa_layout_stateid after REVOKE_STATEID returns. Lock state (see ) held by the revoked stateid is NOT released as part of REVOKE_STATEID; the data server MUST transfer each held lock to the metadata-server escrow owner (see ). Dropping a chunk lock during revocation would permit a write hole and is prohibited; the repair coordination sequence in assumes that locks held by a revoked writer remain held until a repair client adopts them via CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT. REVOKE_STATEID is scoped to the issuing metadata server's entries (see the tagging rule in ). The data server MUST NOT remove an entry that was registered by a different metadata server, even if rsa_layout_stateid happens to match. In a multi-metadata-server deployment, one metadata server therefore cannot revoke another metadata server's entries. REVOKE_STATEID is idempotent: revoking a stateid that has no matching trust entry (either no entry exists, or the entry was registered by a different metadata server) returns NFS4_OK. The metadata server therefore does not need to track precisely which entries are currently live on which data server in order to revoke safely. REVOKE_STATEID returns only a top-level status; there is no result body beyond the nfsstat4 discriminant. If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per ).

RESPONSE CODES

NFS4_OK:: the trust entry was removed, or no matching entry existed (idempotent).
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_BAD_STATEID:: rsa_layout_stateid was a special stateid.
NFS4ERR_DELAY:: the data server is temporarily unable to process the request.
NFS4ERR_INVAL:: rsa_layout_stateid was the anonymous stateid.
NFS4ERR_NOFILEHANDLE:: no current filehandle is set.
NFS4ERR_NOTSUPP:: the data server does not implement REVOKE_STATEID.
NFS4ERR_PERM:: the request arrived on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.

Operation 91: BULK_REVOKE_STATEID - Revoke All Stateids for a Client

ARGUMENTS

XDR for BULK_REVOKE_STATEID4args

RESULTS

XDR for BULK_REVOKE_STATEID4res

DESCRIPTION BULK_REVOKE_STATEID removes every trust entry on the data server that was registered on behalf of a single named client. Unlike REVOKE_STATEID, which removes one entry identified by a (filehandle, stateid) pair, BULK_REVOKE_STATEID does not target a specific filehandle or stateid; it instructs the data server to scan its trust table and remove every entry whose owning pNFS client matches brsa_clientid (and whose issuing metadata server matches the calling metadata server). BULK_REVOKE_STATEID has no analog in . recall-all is expressed at the layout layer (CB_LAYOUTRECALL with LAYOUTRECALL4_ALL); the per-client trust-table sweep introduced here is the data-server-side complement, replacing the N per-file REVOKE_STATEID compounds that per-entry revocation would require with a single round-trip. BULK_REVOKE_STATEID is a metadata-server-to-data-server operation; pNFS clients MUST NOT send it. The data server rejects any BULK_REVOKE_STATEID received on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS at EXCHANGE_ID with NFS4ERR_PERM. The flag is self-declared, so this wire-level check is a hint, not authentication; the deployment-level requirement is stated in . BULK_REVOKE_STATEID does not operate on the current filehandle; no PUTFH is required in the compound. The metadata server provides:

brsa_clientid:: the clientid of the pNFS client whose trust entries are to be removed. The special all-zeros value means "remove every trust entry owned by the calling metadata server, regardless of which pNFS client registered it"; the data server MUST interpret this value as a sweep of its own entries only, NOT as the pNFS client whose clientid happens to be zero and NOT as a global cross-metadata-server table clear.

The metadata server calls BULK_REVOKE_STATEID in any of the following situations:

Client lease expiry:: when a client's lease on the metadata server expires, the metadata server revokes all of that client's layouts. A single BULK_REVOKE_STATEID with brsa_clientid set to the expired client's clientid sweeps every per-file trust entry the metadata server had registered for that client.
CB_LAYOUTRECALL with LAYOUTRECALL4_ALL:: the metadata server is recalling all layouts for a client. BULK_REVOKE_STATEID is the data-server-side complement.
Metadata server restart cleanup:: after the metadata server reconnects to a data server, it MAY issue BULK_REVOKE_STATEID with brsa_clientid set to all-zeros to clear the prior trust table before re-issuing TRUST_STATEID as clients reclaim. See .

BULK_REVOKE_STATEID is scoped to the issuing metadata server's entries (see the tagging rule in ). The data server MUST NOT affect entries registered by a different metadata server. Consequently, in a multi-metadata-server deployment sharing a single data server, one metadata server cannot clear another metadata server's entries via BULK_REVOKE_STATEID. Like REVOKE_STATEID, BULK_REVOKE_STATEID is idempotent (no error is returned if there are no matching entries) and preserves chunk locks held under any revoked stateid by transferring them to the metadata-server escrow owner (see ), rather than dropping them. Subsequent CHUNK operations from the revoked client fail with NFS4ERR_BAD_STATEID; locks held under those revoked stateids remain until adopted by a repair client via CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT (). BULK_REVOKE_STATEID returns only a top-level status; there is no result body beyond the nfsstat4 discriminant.

RESPONSE CODES

NFS4_OK:: the matching entries were removed, or there were none (idempotent).
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_DELAY:: the data server is temporarily unable to process the request.
NFS4ERR_NOTSUPP:: the data server does not implement BULK_REVOKE_STATEID.
NFS4ERR_PERM:: the request arrived on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS.
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.

Operation 92: CHUNK_ESCROW_INSTALL - Install a metadata-server escrow lock

ARGUMENTS

XDR for CHUNK_ESCROW_INSTALL4args

RESULTS

XDR for CHUNK_ESCROW_INSTALL4res

DESCRIPTION CHUNK_ESCROW_INSTALL is sent by the metadata server to a data server on the metadata-server-to-data-server control session to install a metadata-server escrow lock () over the chunk range [ceia_offset, ceia_offset+ceia_count) on the file selected by CURRENT_FH. The lock is created with the specified ceia_escrow_id, which the metadata server will later present to reference this specific installation on CHUNK_ESCROW_RELEASE, in CB_CHUNK_REPAIR, and in the corresponding CHUNK_LOCK adoption (). ceia_mds_epoch is the metadata server's current epoch on this data server; the data server rejects the operation with NFS4ERR_STALE_MDS_EPOCH () if the epoch does not match its recorded value or if the current epoch's expires_at has passed. The install is all-or-nothing across the range: if any chunk in [ceia_offset, ceia_offset+ceia_count) carries an incompatible existing lock, the operation fails with the corresponding per-chunk error at the operation level (typically NFS4ERR_CHUNK_LOCKED). On success, ceir_escrow_id echoes ceia_escrow_id to confirm the identity the data server recorded. The metadata server MUST retain this identity in its durable escrow-tuple set before issuing the call and MUST NOT reuse an escrow_id4 whose lifecycle is not provably complete ().

RESPONSE CODES

NFS4_OK:: the metadata-server escrow lock has been installed over the requested range with the specified escrow_id4.
NFS4ERR_ACCESS:: the metadata server is not authorized to install escrows on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_CHUNK_LOCKED:: a chunk in [ceia_offset, ceia_offset + ceia_count) carries an incompatible existing lock; the install is all-or-nothing across the range.
NFS4ERR_INVAL:: ceia_offset or ceia_count is malformed, or ceia_escrow_id is the reserved all-zeros value.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_ESCROW_INSTALL.
NFS4ERR_PERM:: the request arrived on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS (see ).
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE_MDS_EPOCH:: ceia_mds_epoch does not match the data server's recorded value, or the current epoch's epoch_expires_at has passed.

Operation 93: CHUNK_ESCROW_RELEASE - Release a metadata-server escrow lock

ARGUMENTS

XDR for CHUNK_ESCROW_RELEASE4args

RESULTS

XDR for CHUNK_ESCROW_RELEASE4res

DESCRIPTION CHUNK_ESCROW_RELEASE is sent by the metadata server to release a metadata-server escrow lock it previously installed with CHUNK_ESCROW_INSTALL. The release is compare-and-release: the operation succeeds only when an escrow lock covering the requested range exists on the data server AND its escrow_id4 matches cera_escrow_id. If no escrow covers the range, or the covering escrow's identity differs, the data server returns NFS4ERR_STALE_ESCROW () and MUST NOT alter any current lock or escrow state as a side effect. The compare-and-release rule ensures that a metadata server which has been superseded by a newer incarnation cannot release an escrow the newer incarnation has since re-installed under a different identity. cera_mds_epoch is treated as in ; a stale epoch returns NFS4ERR_STALE_MDS_EPOCH. Interaction with adopted locks (see ): if a repair actor has already adopted this escrow via CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT (), the escrow no longer exists on the data server as a distinct lock (the adoption consumes it and transfers ownership to the client); a subsequent CHUNK_ESCROW_RELEASE naming that identity returns NFS4ERR_STALE_ESCROW and MUST NOT affect the client-owned adopted lock. The metadata server interprets NFS4ERR_STALE_ESCROW after possible adoption as "the escrow was consumed by ADOPT; wait for the CB_CHUNK_REPAIR response or a subsequent revocation-transfer signal before removing durable escrow-tuple bookkeeping."

RESPONSE CODES

NFS4_OK:: the metadata-server escrow lock matching cera_escrow_id has been released.
NFS4ERR_ACCESS:: the metadata server is not authorized to release escrows on this file.
NFS4ERR_BADXDR:: arguments could not be decoded.
NFS4ERR_INVAL:: cera_offset or cera_count is malformed.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_ESCROW_RELEASE.
NFS4ERR_PERM:: the request arrived on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS (see ).
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE_ESCROW:: no escrow covers the requested range, or the covering escrow's identity differs from cera_escrow_id (which may indicate the escrow was consumed by CHUNK_LOCK adoption); the data server MUST NOT alter any current lock or escrow state as a side effect.
NFS4ERR_STALE_MDS_EPOCH:: cera_mds_epoch does not match the data server's recorded value, or the current epoch's epoch_expires_at has passed.

Operation 94: CHUNK_ESCROW_ENUMERATE - Enumerate metadata-server escrow locks on a file

ARGUMENTS

XDR for CHUNK_ESCROW_ENUMERATE4args ; /// }; ]]>

RESULTS

XDR for CHUNK_ESCROW_ENUMERATE4res ; /// escrow_enum_entry4 /// ceer_entries; /// }; /// /// union CHUNK_ESCROW_ENUMERATE4res /// switch (nfsstat4 ceer_status) { /// case NFS4_OK: /// CHUNK_ESCROW_ENUMERATE4resok ceer_resok4; /// default: /// void; /// }; ]]>

DESCRIPTION CHUNK_ESCROW_ENUMERATE is sent by the metadata server to discover which metadata-server escrow locks the data server currently holds on the file selected by CURRENT_FH. Each returned entry names an installed escrow's range and identity. ceea_offset and ceea_count bound the inspection range; ceea_maxcount is the maximum number of entries the caller is willing to receive (ceea_maxcount = 0 is legal and is the op-family capability probe: the data server MUST return NFS4_OK with an empty ceer_entries array and ceer_eof = TRUE when it implements the operation, regardless of any escrows present. A data server that does not implement the operation returns NFS4ERR_OP_ILLEGAL at COMPOUND decode time per , which is distinct from NFS4ERR_NOTSUPP). Pagination uses ceea_cookie: on the first call the caller supplies an empty cookie; the data server returns a snapshot verifier-plus-cursor in ceer_cookie that the caller supplies on the next call to continue enumeration. The data server MUST return a verifier-consistent snapshot: no escrow installed after the first call in a pagination sequence appears in subsequent calls, and no escrow released after the first call disappears from subsequent calls (implementations may achieve this by snapshotting the escrow set at first-call time and streaming from the snapshot). ceer_eof = TRUE signals the end of the enumeration. Non-mutating: the operation observes but does not modify escrow state. ceea_mds_epoch is treated as in the sibling operations; a stale epoch returns NFS4ERR_STALE_MDS_EPOCH.

RESPONSE CODES

NFS4_OK:: the requested escrow enumeration entries have been returned; ceer_eof indicates whether more remain.
NFS4ERR_ACCESS:: the metadata server is not authorized to enumerate escrows on this file.
NFS4ERR_BADXDR:: arguments could not be decoded, or ceea_cookie is malformed.
NFS4ERR_INVAL:: ceea_offset, ceea_count, or ceea_maxcount is malformed, or ceea_cookie references a snapshot the data server no longer holds.
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_ESCROW_ENUMERATE. See the ceea_maxcount = 0 op-family capability probe in the description for the distinction from NFS4ERR_OP_ILLEGAL at COMPOUND decode time.
NFS4ERR_PERM:: the request arrived on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS (see ).
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE_MDS_EPOCH:: ceea_mds_epoch does not match the data server's recorded value, or the current epoch's epoch_expires_at has passed.

Operation 95: CHUNK_ESCROW_TAKEOVER - Advance metadata-server epoch after incarnation change

ARGUMENTS

XDR for CHUNK_ESCROW_TAKEOVER4args ; /// }; ]]>

RESULTS

XDR for CHUNK_ESCROW_TAKEOVER4res

DESCRIPTION CHUNK_ESCROW_TAKEOVER is the recovery path a metadata server uses to assume the escrow-control role on a data server after an incarnation change (a failover, a restart, or an operator-mediated recovery). It carries an incarnation-lease proof issued by an authority external to the metadata server (see ) which the data server verifies before advancing its recorded metadata-server epoch. The operation is a compare-and-advance:

ceta_expected_prior_epoch MUST equal the data server's currently-recorded metadata-server epoch;
ceta_new_epoch MUST either exceed the prior epoch (ordinary takeover / advance) or equal the prior epoch when the caller is renewing the lease under the same incarnation (renewal form; both values equal the data server's current epoch, and only the epoch_expires_at field is refreshed);
ceta_proof_profile MUST name a profile the data server supports; and
ceta_proof_data MUST verify under that profile per .

All four conditions are evaluated atomically; on success the data server updates its recorded epoch and epoch_expires_at values in a single step, and subsequent CHUNK_ESCROW_INSTALL, CHUNK_ESCROW_RELEASE, and CHUNK_ESCROW_ENUMERATE operations from the new epoch are accepted. Concurrent takeovers serialize: exactly one same-expected-prior request wins, and any other sees NFS4ERR_STALE_MDS_EPOCH on its second attempt because the prior has advanced. The strict evaluation order per is followed: session-replay-cache lookup first, then presenter authorization (NFS4ERR_ACCESS on failure), then profile support (NFS4ERR_NOTSUPP on unknown profile), then proof verification (NFS4ERR_ACCESS on failure), then epoch compare-and-advance (NFS4ERR_STALE_MDS_EPOCH on mismatch). This ordering ensures unauthenticated callers learn nothing about supported profiles or current epoch state. Unlike the other CHUNK_ESCROW operations, CHUNK_ESCROW_TAKEOVER is EXEMPT from the ongoing epoch_expires_at check applied to INSTALL, RELEASE, and ENUMERATE: TAKEOVER is itself the recovery path out of an expired epoch and carries its own compare-and-advance + proof-verification semantics. A metadata server that has been fenced by a superseding takeover cannot recover by presenting another CHUNK_ESCROW_TAKEOVER unless it obtains a valid fresh incarnation-lease proof from the authority. Older escrow state is NOT invalidated by an epoch advance: escrows installed under the prior epoch survive; the new incarnation reconciles them via CHUNK_ESCROW_ENUMERATE and adopts, releases, or reissues each per its durable recovery state. Fencing applies to control traffic on old epochs, not to the state those epochs installed.

RESPONSE CODES

NFS4_OK:: the metadata server epoch has been advanced to ceta_new_epoch, or refreshed under the renewal form.
NFS4ERR_ACCESS:: presenter authorization or incarnation-lease proof verification failed. Per , this outcome is returned both for authorization failure and for proof failure so that unauthenticated callers learn nothing about supported profiles or current epoch state.
NFS4ERR_BADXDR:: arguments could not be decoded, or ceta_proof_data is structurally malformed.
NFS4ERR_INVAL:: ceta_expected_prior_epoch or ceta_new_epoch is malformed (for example, ceta_new_epoch precedes ceta_expected_prior_epoch and does not equal it under the renewal form).
NFS4ERR_NOTSUPP:: the data server does not implement CHUNK_ESCROW_TAKEOVER, or the presented ceta_proof_profile is unknown to the data server.
NFS4ERR_PERM:: the request arrived on a session whose owning client did not present EXCHGID4_FLAG_USE_PNFS_MDS (see ).
NFS4ERR_SERVERFAULT:: the data server failed while processing the request.
NFS4ERR_STALE_MDS_EPOCH:: ceta_expected_prior_epoch does not match the data server's recorded value; a concurrent TAKEOVER has already advanced past it, or the caller has been fenced by a superseding takeover.

New NFSv4.2 Callback Operations

Callback Operations XDR The following amendment blocks extend the nfs_cb_argop4 and nfs_cb_resop4 dispatch unions defined in with arms for the new callback operation defined in this document.

nfs_cb_argop4 amendment block

nfs_cb_resop4 amendment block

Callback Operation 16: CB_CHUNK_REPAIR - Request Repair of Inconsistent Chunk Ranges

ARGUMENTS

XDR for CB_CHUNK_REPAIR4args ; /// }; ]]>

RESULTS

XDR for CB_CHUNK_REPAIR4res ; /// }; ]]>

DESCRIPTION CB_CHUNK_REPAIR is the back-channel by which the metadata server delivers a repair work item to a repair actor it has selected per . The work item names one or more non-atomic chunk ranges on the file's data servers for the repair actor to reconstruct or roll back. The selection rules in are normative for how a client MUST respond on receipt of this callback. CB_CHUNK_REPAIR has no analog in . back-channel callbacks operate at the layout layer (CB_LAYOUTRECALL) or the file-state layer (CB_RECALL, CB_NOTIFY); CB_CHUNK_REPAIR is the new chunk-layer callback that drives reconstruction or rollback of non-atomic chunks without requiring a full layout return. The metadata server provides:

ccra_fh:

the filehandle of the file whose chunks are non-atomic. The callback compound carries the filehandle directly; there is no preceding PUTFH in callback compounds.

ccra_layout_stateid:

the recipient client's current layout stateid for the file if one is held. A client that does not hold a layout on ccra_fh MUST ignore ccra_layout_stateid (it will be the anonymous stateid in that case) and MUST acquire one via LAYOUTGET before issuing any CHUNK operation on the ranges.

ccra_deadline:

an absolute nfstime4 (seconds and nanoseconds since the epoch, as defined in ) by which the client is expected to have driven every range to completion (CHUNK_REPAIRED on the reconstruction path, or CHUNK_UNLOCK on the rollback path). As with tsa_expire (), the wall-clock representation assumes the metadata server and the repair actor maintain clock synchronization within one metadata-server lease period (see the clock-sync paragraph in for the deployment options when this cannot be guaranteed); under skew, missing the deadline is not safety-critical because state cannot be corrupted, but spurious deadline expiry SHOULD be avoided by setting ccra_deadline to at least (current-wall-clock + deadline-budget + expected-skew). Missing the deadline does not corrupt state -- the metadata server MAY re-select another repair actor after the deadline elapses -- but a client that has missed the deadline MUST re-verify its layout and the chunk lock state before continuing any repair-related CHUNK operation.

ccra_reason:

distinguishes the two flows that cause the metadata server to issue a repair callback:

CB_REPAIR_REASON_RACE:: A live-race repair. A client (not necessarily the recipient of this callback) detected a chunk-level non-atomicity at write or read time and reported it via LAYOUTERROR. The metadata server is driving repair synchronously because the affected chunk is on the critical path of some I/O. The recipient SHOULD prioritize the callback over background work.
CB_REPAIR_REASON_SCRUB:: A background scrub. The metadata server has detected stale or non-atomic payloads during a scheduled integrity sweep and is opportunistically driving repair. No client is currently blocked on these ranges. The recipient MAY schedule the callback at lower priority than CB_REPAIR_REASON_RACE, and MAY return NFS4ERR_DELAY to defer repair to a more convenient time; the metadata server will retry.

The two reasons share all other semantics: the same ccra_ranges encoding, the same response codes, the same deadline contract. Only the priority and retry behavior differs.

ccra_escrow_id:

the escrow_id4 () of the metadata-server escrow lock the metadata server installed to cover every range in this callback. The client presents this same escrow_id4 in the cla_adopt discriminant of the CHUNK_LOCK request that adopts the escrow (see "Lock Transfer via CHUNK_LOCK_FLAGS_ADOPT"); the data server matches on full identity, so a callback ranges over a single escrow identity. If the repair spans ranges the metadata server installed under distinct escrow identities (for example, when a multi-data-server repair needs its own escrow per data server), the metadata server MUST issue one CB_CHUNK_REPAIR per escrow identity rather than mixing them into a single callback.

ccra_ranges:

the list of every chunk range the metadata server requests the client to repair. Each entry carries its own ccr_error describing the failure mode the client is being asked to remedy. The repair strategy depends on the error code; see for the normative and guidance split. The array is bounded by CB_CHUNK_REPAIR_MAX_RANGES4; if a repair set exceeds that bound, the metadata server splits it across multiple callbacks.

The metadata server SHOULD keep each CB_CHUNK_REPAIR compound within the back-channel maximum (ca_maxrequestsize) negotiated in CREATE_SESSION (see ). If the set of affected ranges would exceed that maximum, the metadata server MAY issue multiple CB_CHUNK_REPAIR callbacks to the same client. Each callback is independent; the client drives each to completion before the deadline on that callback's ranges. The metadata server MUST NOT name a range in ccra_ranges unless the data server holds a chunk lock on that range at the time the callback is issued (the failure occurred in or around a PENDING or FINALIZED state that established the lock). The repair actor MUST use CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT () to take ownership of the lock before issuing CHUNK_WRITE_REPAIR, CHUNK_ROLLBACK, or CHUNK_WRITE on any chunk in a requested range. Because callback delivery and lock state can race, a repair actor that receives NFS4ERR_NO_ADOPTABLE_LOCK on the adoption MUST report the mismatch via ccrr_range_status per rather than attempt recovery. CB_CHUNK_REPAIR returns a top-level ccrr_status plus a co-indexed per-range status array ccrr_range_status. When the entire callback fails at once -- decode failure, authorization failure, session-stale, and other conditions that fail every range -- the operation-wide error is placed in ccrr_status and ccrr_range_status MUST be empty (zero entries). When the callback is well-formed and evaluated per range, ccrr_range_status carries exactly one nfsstat4 per entry in ccra_ranges, co-indexed, and ccrr_status is one of:

NFS4_OK:: every range reached CHUNK_REPAIRED or CHUNK_UNLOCK per the completion contract; every ccrr_range_status entry is NFS4_OK.
NFS4ERR_PARTIAL ():: at least one range did not reach the completion state. The metadata server MUST consume the ccrr_range_status array to determine per-range outcome; the array is authoritative. An NFS4ERR_PARTIAL response with an empty array is malformed and MUST be rejected.

A non-empty ccrr_range_status pairs only with NFS4_OK or NFS4ERR_PARTIAL at the top level; every other top-level nfsstat4 carries an empty ccrr_range_status. See "RESPONSE CODES" below for the normative meanings the metadata server attaches to each returned top-level nfsstat4.

RESPONSE CODES The ccrr_status value returned by the client has the following normative meanings to the metadata server:

NFS4_OK:: The client has accepted the request and driven every range in this callback to completion (CHUNK_REPAIRED or CHUNK_UNLOCK on every affected chunk). The metadata server clears the repair queue entry.
NFS4ERR_DELAY:: The client has accepted the request but requires more time. The metadata server MAY extend the deadline by issuing a new CB_CHUNK_REPAIR with a later ccra_deadline, or MAY re-select another client. The client continues to hold any locks it has adopted until the original or extended deadline.
NFS4ERR_ENCODING_NOT_SUPPORTED:: The client does not implement the encoding type of the layout and cannot reconstruct. The metadata server MUST NOT retry with the same client and SHOULD select a different client.
NFS4ERR_PAYLOAD_LOST:: The client has concluded that the identified ranges cannot be repaired -- there are not enough surviving shards to reconstruct and rollback is also impossible. The metadata server MUST NOT retry the repair and transitions the affected ranges into an implementation-defined damaged state. See .
NFS4ERR_PARTIAL:: At least one range in the callback did not reach the completion contract. The metadata server MUST consume the co-indexed ccrr_range_status array to determine the per-range outcome; NFS4ERR_PARTIAL is not a whole-callback retriable error and MUST NOT be treated as one. The array is authoritative: each ccrr_range_status entry maps directly to the co-indexed entry in ccra_ranges and is evaluated on its own terms (NFS4_OK, NFS4ERR_DELAY, NFS4ERR_ENCODING_NOT_SUPPORTED, NFS4ERR_PAYLOAD_LOST, or another per-range disposition) per the result contract above. An NFS4ERR_PARTIAL response with an empty ccrr_range_status array is malformed and the metadata server MUST reject the callback response. See .

All other error codes listed in are treated by the metadata server as retriable: the metadata server MAY issue a subsequent CB_CHUNK_REPAIR to the same or a different client. If the client becomes unreachable (no response within the deadline), the metadata server re-selects per .

Composed Rollback Guarantee and Error Decision Tree The flexible file v2 layout chunk protocol combines three related mechanisms -- writer-supplied opaque owner identity (), best-effort predecessor discovery ( / ), and metadata-server escrow control-plane pinning ( / / / , ) -- that together deliver a CONDITIONAL rollback guarantee. This section states the guarantee scope precisely and specifies the client-side decision tree over the composed error set.

Guarantee Scope The composed guarantee is against protocol-level garbage collection and owner-association release only, NOT against storage or integrity failures. It applies only when three conditions all hold at the moment a CHUNK_ROLLBACK is issued against the named predecessor:

Present at acquisition. The predecessor generation must have existed on the data server at the time the qualifying CHUNK_LOCK or metadata-server escrow was acquired. Predecessors that had already been released under the retention scope () before any qualifying lock or escrow was acquired are not covered.
Continuous custody. The lock or escrow custody chain must have remained uninterrupted through the rollback decision window. Custody handoffs are permitted (a client-owned lock adopted from a metadata-server escrow lock preserves the escrow_id4 per , and a subsequent revocation-transfer re-emits the same escrow_id4 to a new metadata-server escrow lock, keeping the custody chain continuous), but any interval in which the predecessor was covered by neither a qualifying lock nor a metadata-server escrow lock breaks continuity.
Payload remains AVAILABLE. The predecessor's payload MUST be in the AVAILABLE read-time state (). A predecessor whose payload has become ERRORED through media loss, unrecoverable corruption, loss of all redundant data servers, or non-conforming data-server behavior is not covered -- an ERRORED predecessor follows the best-effort reconstruction path () and MAY terminate at NFS4ERR_PAYLOAD_LOST via CB_CHUNK_REPAIR.

When all three conditions hold, a CHUNK_ROLLBACK that names the predecessor's original owner triple is guaranteed to restore the predecessor as the current COMMITTED generation with its original triple intact ( "Rollback of COMMITTED Chunks", case (a)). When any condition fails, the call may return NFS4ERR_NO_PREDECESSOR () or NFS4ERR_INVAL and the caller falls back to best-effort reconstruction as described in -- best-effort reconstruction into a NEW generation under a NEW owner triple, or terminal NFS4ERR_PAYLOAD_LOST when no authoritative source exists. The guarantee is a protocol-level protection against premature release: flexible file v2 layout implementations MUST NOT release the payload or owner-to-index association of a predecessor covered by an active qualifying lock or metadata-server escrow lock, per the payload/association biconditional () and the retention scope () as extended by the escrow-pin mechanism. It is not a guarantee against exogenous failures of the storage substrate.

Client-Side Error Decision Tree A client operating over the composed error set distinguishes control-plane failures (which the metadata server must resolve) from data-plane failures (which the client can address via fallback). Ordering matters: control-plane failures must be resolved before data-plane recovery is attempted.

CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT -> NFS4ERR_NO_ADOPTABLE_LOCK ():: custody / control-plane failure BEFORE any usable lock is in hand. The four state causes (no escrow / identity mismatch / reconciliation hold / already-adopted) are all conditions the metadata server is best placed to resolve. The client MUST report the outcome via the ccrr_range_status array () and MUST NOT unilaterally retry the adoption or begin the CHUNK_WRITE_REPAIR fallback path (); that fallback assumes a usable lock, and no lock exists here to fall back under.
CHUNK_LOCK -> NFS4ERR_ACCESS:: presenter authorization failure. Report to the metadata server; do not retry.
After successful CHUNK_LOCK / ADOPT -> CHUNK_HEADER_READ:: read the primary owner and chrr_predecessors array. If the intended predecessor's triple appears in the list, proceed to CHUNK_ROLLBACK. If absent, the CHUNK_WRITE_REPAIR fallback path may begin directly (do not issue CHUNK_ROLLBACK against a predecessor known-absent).
CHUNK_ROLLBACK -> NFS4ERR_NO_PREDECESSOR ():: data-plane result AFTER a usable lock is in hand. The actor has the lock; there is simply no restorable predecessor. The client MAY invoke best-effort reconstruction via CHUNK_WRITE_REPAIR under a new owner triple (); fallback MAY terminate at NFS4ERR_PAYLOAD_LOST () if no authoritative source exists.
CHUNK_ROLLBACK -> NFS4ERR_INVAL or NFS4ERR_NO_PREDECESSOR for a fresh op naming a released triple:: terminal per-entry failure. Caller holds a stale reference; no operation defined in this document resurrects the deleted generation. The data server returns NFS4ERR_INVAL within the delete case's session-slot replay-cache window and NFS4ERR_NO_PREDECESSOR after the window has elapsed or for any other terminal release (per the release-scope split at ). Compare either code to the uncertain-replay carve-out ( "Idempotence and Uncertain-Replay Carve-Out") which permits an EXACT reissue after uncertain prior completion to treat NFS4ERR_INVAL or NFS4ERR_NO_PREDECESSOR as postcondition-equivalent success, but ONLY when the caller independently verifies the postcondition holds.
Any CHUNK_ESCROW operation -> NFS4ERR_STALE_ESCROW ():: control-plane identity mismatch or no covering escrow on the metadata server's own CHUNK_ESCROW_RELEASE. The response never authorizes tuple removal by itself when adoption may have consumed the escrow (see ).
Any CHUNK_ESCROW operation -> NFS4ERR_STALE_MDS_EPOCH ():: the metadata server has been fenced by a superseding CHUNK_ESCROW_TAKEOVER. The metadata server MUST obtain a fresh incarnation-lease token and reissue via CHUNK_ESCROW_TAKEOVER; TAKEOVER is exempt from this rejection.
CB_CHUNK_REPAIR response -> NFS4ERR_PARTIAL ():: at least one named range did not reach completion; the metadata server MUST consume the per-range ccrr_range_status array () to determine per-range outcome.

The essential distinction is that NFS4ERR_NO_ADOPTABLE_LOCK and NFS4ERR_NO_PREDECESSOR both prevent the direct rollback path, but at different lifecycle stages and with different recovery authorities: NFS4ERR_NO_ADOPTABLE_LOCK is a custody / control-plane failure BEFORE the actor obtains usable custody (unrecoverable unilaterally), while NFS4ERR_NO_PREDECESSOR is a data-plane result AFTER successful adoption where the actor already holds the lock but finds no restorable predecessor data (the actor MAY invoke best-effort reconstruction via CHUNK_WRITE_REPAIR). A client that receives one MUST NOT treat it as the other.

Worked Example: Composed Rollback and Fallback The following worked example exercises the three composed conditions. A single file has chunk index 5 previously written by client 7 with co_cohort_id 41 and co_id 100 (owner triple (41, 7, 100)) and then overwritten by client 7 with co_cohort_id 42 and co_id 101 (owner triple (42, 7, 101)), which reached the COMMITTED state. The metadata server determines that the (42, 7, 101) commit was incorrect and initiates rollback. The trace shows the happy path, the lost-callback branch, and the fallback contrast. Note that the owner triple is (co_cohort_id, co_client_id, co_id) throughout, per ; per-chunk CAS state (cg_gen_id, cg_client_id) is data-server-managed and is distinct from these owner-identity numbers.

Happy Path (all three conditions hold)

The metadata server proactively installs an metadata-server escrow lock over chunk index 5 by sending CHUNK_ESCROW_INSTALL with escrow_id4 E1 while the (41, 7, 100) predecessor is still AVAILABLE on the data server. The data server accepts and echoes E1 in ceir_escrow_id. Per conditions 1 and 3, the predecessor is now under continuous metadata-server escrow custody and remains AVAILABLE.
The metadata server sends CB_CHUNK_REPAIR to a selected repair actor with ccra_escrow_id = E1 and a range naming chunk index 5.
The repair actor issues CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT and cla_adopt = { TRUE, cla_escrow_id = E1 }. The data server validates identity and atomically transfers custody: E1 dissolves as a metadata-server escrow lock, the client owns the lock, the predecessor's payload and owner association survive intact, and E1 is retained as durable custody metadata on the client-owned lock (per ). Condition 2 (continuous custody) is preserved through the adoption.
The repair actor issues CHUNK_HEADER_READ over chunk index 5. The response shows chrr_owners as the (42, 7, 101) current generation and chrr_predecessors[0] as the (41, 7, 100) retained predecessor ().
The repair actor issues CHUNK_ROLLBACK naming the predecessor's triple (41, 7, 100). The data server executes "Rollback of COMMITTED Chunks" case (a) (): the retained predecessor is restored as the current COMMITTED generation under its original triple (41, 7, 100), and the displaced successor's triple (42, 7, 101) is atomically invalidated ("Deletion Atomicity and Invalidated Triples").
The repair actor issues CHUNK_UNLOCK. The client-owned lock is released; the preserved escrow_id4 custody metadata is cleared.
The repair actor returns NFS4_OK on the CB_CHUNK_REPAIR response with a co-indexed NFS4_OK in ccrr_range_status. The metadata server issues CHUNK_ESCROW_RELEASE with cera_escrow_id = E1. Because E1 was consumed by the adoption in step 3, the data server returns NFS4ERR_STALE_ESCROW; the metadata server interprets this as "the escrow was consumed by ADOPT and the callback response confirms completion," and clears its durable escrow-tuple record for (file, E1, {this data server}).

Lost-Callback Branch Steps 1-6 proceed as above but the CB_CHUNK_REPAIR response is lost in transit (data server restart or network failure before the metadata server receives the reply). The metadata server never observes step-7 confirmation. The repair actor's lock lease eventually expires without an explicit release, and the data server transitions the lock through the revocation-transfer path (see ): a new metadata-server escrow lock is installed on the same range with the same preserved escrow_id4 = E1. On the next CHUNK_ESCROW_ENUMERATE the metadata server observes the reappeared E1 and reissues repair under it.

Fallback Contrast (condition 1 fails) Under an alternative setup where the metadata server did NOT install a metadata-server escrow before the retention scope () released the (41, 7, 100) predecessor, condition 1 fails. By the time repair is initiated the predecessor is ABSENT on the data server, and the escrow the metadata server installs pins only what still exists (there is no predecessor to pin). The repair actor's CHUNK_LOCK / CHUNK_HEADER_READ sequence discovers no retained predecessor for index 5. A subsequent CHUNK_ROLLBACK against the (41, 7, 100) triple returns NFS4ERR_NO_PREDECESSOR (). The client falls back to CHUNK_WRITE_REPAIR, reconstructing authoritative bytes from surviving data-server shards and writing them under a new owner triple of its own choosing -- for example (43, 7, 102). The resulting COMMITTED generation carries the new triple, NOT the released (41, 7, 100). Any subsequent lifecycle operation that names the released (41, 7, 100) triple returns NFS4ERR_NO_PREDECESSOR -- the (41, 7, 100) association was released under the retention scope, not by an explicit CHUNK_ROLLBACK delete case within a live replay-cache window, so the release-scope split at routes to the NO_PREDECESSOR arm consistent with the previous CHUNK_ROLLBACK outcome in this trace: the fallback creates a new generation, it does not resurrect the released predecessor. When no authoritative source exists for reconstruction, the fallback itself terminates at NFS4ERR_PAYLOAD_LOST () via CB_CHUNK_REPAIR.

Security Considerations The combination of components in a pNFS system is required to preserve the security properties of NFSv4.1+ with respect to an entity accessing data via a client. The pNFS feature partitions the NFSv4.1+ file system protocol into two parts: the control protocol and the data protocol. As the control protocol in this document is NFS, the security properties are equivalent to the version of NFS being used. The flexible file v2 layout further divides the data protocol into metadata and data paths. The security properties of the metadata path are equivalent to those of NFSv4.1x (see Sections 1.7.1 and 2.2.1 of ). And the security properties of the data path are equivalent to those of the version of NFS used to access the storage device, with the provision that the metadata server is responsible for authenticating client access to the data file. The metadata server provides appropriate credentials to the client to access data files on the storage device. It is also responsible for revoking access for a client to the storage device. The metadata server enforces the file access control policy at LAYOUTGET time. The client MUST use RPC authorization credentials for getting the layout for the requested iomode (LAYOUTIOMODE4_READ or LAYOUTIOMODE4_RW), and the server verifies the permissions and ACL for these credentials, possibly returning NFS4ERR_ACCESS if the client is not allowed the requested iomode. If the LAYOUTGET operation succeeds, the client receives, as part of the layout, a set of credentials allowing it I/O access to the specified data files corresponding to the requested iomode. When the client acts on I/O operations on behalf of its local users, it MUST authenticate and authorize the user by issuing respective OPEN and ACCESS calls to the metadata server, similar to having NFSv4 data delegations. Under loose coupling, the combination of filehandle, synthetic uid, and gid in the layout is the way that the metadata server enforces access control to the data server. The client only has access to filehandles of file objects and not directory objects. Thus, given a filehandle in a layout, it is not possible to guess the parent directory filehandle. Further, as the data file permissions only allow the given synthetic uid read/write permission and the given synthetic gid read permission, knowing the synthetic ids of one file does not necessarily allow access to any other data file on the storage device. Under loose coupling, the metadata server can also deny access at any time by fencing the data file, which means changing the synthetic ids. In turn, that forces the client to return its current layout and get a new layout if it wants to continue I/O to the data file. Under tight coupling (), the metadata server does not rely on synthetic uid and gid values to enforce access at the data server. Instead, the metadata server injects a per-stateid trust entry into the data server via TRUST_STATEID (); the data server validates each CHUNK operation against the registered trust entry and rejects operations that do not present a registered stateid. The metadata server rescinds that authorization via REVOKE_STATEID () for a single stateid or BULK_REVOKE_STATEID () for every stateid a client currently holds; both are the tight-coupling equivalent of a synthetic-id change and achieve the same effect (the client MUST return its current layout and obtain a new one to continue I/O to the data file). The security implications of the trust-stateid channel are covered in . If access is allowed, the client uses the corresponding (read-only or read/write) credentials to perform the I/O operations at the data file's storage devices. When the metadata server receives a request to change a file's permissions or ACL, it SHOULD recall all layouts for that file and then MUST fence off any clients still holding outstanding layouts for the respective files by implicitly invalidating the previously distributed credential on all data file comprising the file in question. It is REQUIRED that this be done before committing to the new permissions and/or ACL. By requesting new layouts, the clients will reauthorize access against the modified access control metadata. Recalling the layouts in this case is intended to prevent clients from getting an error on I/Os done after the client was fenced off.

Checksum Integrity Scope The checksum values carried in CHUNK_WRITE and returned from CHUNK_READ defend against accidental data corruption during storage or transmission -- bit flips on storage media, network errors, software bugs in the erasure transform. The threat model an individual deployment achieves depends on which checksum_algorithm4 () the metadata server selects for the file's mirrors via ffv2m_checksum_algorithm ():

Bit-flip-class algorithms (CRC32, CRC32C, Fletcher4):: Detect accidental bit-level corruption with high probability. Do NOT defend against an adversary who can modify the payload and recompute a valid checksum, because these algorithms are not cryptographic and the algorithm identifier and parameters are public. Suitable when the threat model excludes adversaries on the wire and at rest.
Cryptographic-strength algorithms (SHA-256, SHA-512, BLAKE3):: Detect accidental corruption with cryptographic-strength collision resistance. These are UNKEYED hashes carried alongside the payload on the wire, so they do NOT by themselves defend against an adversary who can modify a chunk and recompute a valid hash: the attacker knows the algorithm and can substitute a matching digest. Content authentication against active adversaries requires a keyed MAC or signature scheme (e.g., RPCSEC_GSS_KRB5I, RPC-over-TLS with mutual authentication, or an application-layer signed manifest) applied at the trust boundary; the checksum mechanism defined here provides corruption detection, not content authentication. Suitable when chunks may be at rest on storage the deployment does not fully control and the deployment layers cryptographic transport or storage integrity on top of the checksum for adversarial protection.
CHECKSUM_ALG_NONE:: No protocol-level integrity check. The deployment is relying on transport-layer integrity (RPC-over-TLS , RPCSEC_GSS_KRB5I) or storage-layer integrity (filesystem checksums on the data server, RAID-level integrity) instead. Suitable when those other layers are reliably present end-to-end and the per-chunk wire protection would be redundant.

Deployments requiring protection against active attackers SHOULD select one of the cryptographic algorithms, OR use CHECKSUM_ALG_NONE in conjunction with RPC-over-TLS () or RPCSEC_GSS, whichever fits the deployment's existing security architecture. An authenticated client is in the "active attacker" role with respect to its own chunks, in a restricted sense. The data server validates the checksum against the bytes the client provided, so an authenticated client that chooses to send semantically-invalid bytes with a correctly computed checksum will have those bytes accepted. The residual risk differs per authentication model:

Under AUTH_SYS with loose coupling, the residual risk is essentially the pre-existing attack surface of NFSv3 writes: any host that can reach the data server with a valid uid can write nonsense to chunks that uid owns. This is the Flex Files v1 authorization model, which flexible file v2 layout inherits without modification for this path.
Under RPCSEC_GSS or TLS with mutual authentication, the residual risk reduces to: only the authenticated client can write nonsense into chunks it owns. Cross-client corruption is prevented because the data server verifies the principal before accepting the write. The remaining exposure is at the client's own integrity: any deployment that relies on data integrity above the wire MUST apply application-level content validation.

Flexible file v2 layout does not attempt to defend against this authenticated-but-malicious case. The checksum mechanism is a transport-integrity check, not a content-integrity check; the system trust model assumes that an authenticated principal is entitled to destroy the content of chunks it owns.

Chunk Lock and Lease Expiry When a client holds a chunk lock (acquired via CHUNK_LOCK) and its lease expires or the client crashes, the lock is released implicitly by the data server. This opens a window in which another client may write to the previously locked range before the original client's repair is complete. Implementations SHOULD ensure that the lease period for chunk locks is sufficient to complete repair operations, and SHOULD implement CHUNK_UNLOCK explicitly on abort paths. The metadata server's LAYOUTERROR and LAYOUTRETURN mechanisms provide the coordination point for detecting and resolving such races.

Error Code Information Disclosure The new error codes NFS4ERR_CHUNK_LOCKED (10099) and NFS4ERR_PAYLOAD_NOT_ATOMIC (10098) convey information about chunk state to the caller. Both of these errors MAY be returned to callers whose credentials have not been verified by the data server (e.g., when the AUTH_SYS uid presented does not match the synthetic uid on the data file). The information they reveal -- that a chunk is locked, or that a CRC mismatch occurred -- does not directly disclose file contents but may indicate concurrent write activity. Implementations that are concerned about this level of disclosure SHOULD require that CHUNK operations only succeed after credential verification and return NFS4ERR_ACCESS for unverified callers rather than the more specific error codes. This document adds NFS4ERR_NO_PREDECESSOR (10103), NFS4ERR_NO_ADOPTABLE_LOCK (10104), NFS4ERR_STALE_ESCROW (10105), NFS4ERR_STALE_MDS_EPOCH (10106), and NFS4ERR_PARTIAL (10107). All five reveal control-plane or discovery state that a compromised or curious caller could aggregate into a picture of which files are under repair, which are pinned, and which epoch is current. A data server SHOULD enforce presenter authorization before returning any of these codes so that an unauthorized caller receives NFS4ERR_ACCESS rather than the specific state indicator. In particular, NFS4ERR_STALE_MDS_EPOCH and unknown-profile NFS4ERR_NOTSUPP on CHUNK_ESCROW_TAKEOVER are gated behind presenter authorization by the strict evaluation order in ; that ordering is a security requirement, not merely a diagnostic preference.

Escrow Control Plane and Incarnation Proofs This document's escrow control plane introduces four operations (CHUNK_ESCROW_INSTALL, CHUNK_ESCROW_RELEASE, CHUNK_ESCROW_ENUMERATE, CHUNK_ESCROW_TAKEOVER -- through ) and one proof envelope () whose security properties merit dedicated treatment.

Authorization scope CHUNK_ESCROW_INSTALL, CHUNK_ESCROW_RELEASE, and CHUNK_ESCROW_ENUMERATE MUST be invoked only by the metadata server currently holding the incarnation lease. A data server MUST authenticate the presenter via RPCSEC_GSS () and verify the presenter's principal against the deployment-configured metadata-server principal(s) for the file's device before accepting any escrow-family operation; presenter authorization failure returns NFS4ERR_ACCESS with no side effect on escrow state. This authorization is orthogonal to the ordinary client-facing NFSv4 access controls: an authenticated pNFS client's credentials MUST NOT authorize an escrow-family operation even if the client is otherwise permitted to read or write the file. A data server that receives any escrow-family operation on a session belonging to a non-metadata-server principal MUST reject with NFS4ERR_ACCESS or NFS4ERR_PERM per the ordinary NFSv4 rules.

Trust anchor lifecycle The proof profile depends on a deployment-provisioned trust anchor at each data server ( "Trust Anchor Provisioning"). Anchor compromise permits an adversary to forge incarnation-lease tokens and stage takeovers of the escrow control plane; anchor unavailability blocks legitimate takeovers and inhibits recovery from an incarnation failure. Deployments SHOULD:

provision separate anchors per proof profile so revocation of one profile does not require replacing keys for others;
rotate anchors on a schedule consistent with the authority's own key-management posture (this specification does not mandate a rotation interval); and
retain the ability to revoke an anchor out-of-band without requiring a wire-level protocol message, because a compromised anchor's tokens will otherwise satisfy the data server's strict-ordering checks by construction.

An incarnation-lease authority MUST NOT delegate signing to any component the metadata server can spoof; the security of the proof mechanism rests on the assumption that the authority is a distinct entity from any metadata server and can independently attest current single writer ownership.

Replay-cache exhaustion and durability The token_id replay cache at the data server ( "Presentation and Verification") is a finite resource. A hostile presenter that can obtain many valid tokens for the same profile MAY attempt to grow the cache without bound. A data server SHOULD bound the cache size and expire entries no later than their token's expires_at; entries older than the longest admissible issued_at-to-expires_at window MAY be evicted. Eviction does not sacrifice lost-response recovery: the byte-identical uncertain-completion recovery path () recognizes both the cache-hit and the cache-miss form of the same reissue, so a token whose replay entry has aged out still resolves to NFS4_OK when its ceta_new_epoch equals the data server's currently-recorded epoch and its ceta_expected_prior_epoch is strictly prior. A hostile presenter cannot exploit the cache-miss form to advance the epoch: proof verification still gates on the token's signature and principal, and once a subsequent successful TAKEOVER advances past ceta_new_epoch the reissue no longer satisfies the postcondition-equivalent predicates. Data servers that persist the token_id cache across restart MUST persist and evict entries coherently with expires_at values. A data server that does not persist the cache returns to the freshly-initialized state after restart; lost-response reissues still resolve through the cache-miss form of the uncertain-completion recovery path.

Discovery information disclosure CHUNK_ESCROW_ENUMERATE and the CHUNK_HEADER_READ predecessor arm () disclose control-plane and discovery information. CHUNK_ESCROW_ENUMERATE is gated behind the escrow-role authorization above; CHUNK_HEADER_READ's predecessor list is available to any pNFS client holding a valid layout for the file. A deployment where predecessor identity is itself sensitive (for example, because owner triples encode information about the writer's identity that ordinary read authorization does not disclose) SHOULD scope the layout so that only appropriately-authorized clients can obtain it. The specification does not require the data server to suppress predecessor disclosure on a per-client basis; where that is required, deploy at the layout-authorization layer.

Resource exhaustion via long-lived escrows A compromised or faulty metadata server can install an arbitrary number of long-lived metadata-server escrow locks via CHUNK_ESCROW_INSTALL, each of which forces retention of a predecessor generation the data server would otherwise release under the retention scope rule (). Sustained abuse could exhaust data-server storage. A data server MAY apply implementation-defined admission control on the total number of metadata-server escrow locks it holds concurrently, rejecting further CHUNK_ESCROW_INSTALL with NFS4ERR_SERVERFAULT or a resource-exhaustion error when the bound is reached; the metadata server can then release older escrows or wait for their natural completion before retrying. This specification does not prescribe a concrete bound; the choice is deployment-local and interacts with the data-server's storage-management policy.

Consequence of losing durable adopted-lock escrow identity The client-owned lock that adopted a metadata-server escrow retains the adopted escrow_id4 as durable custody metadata per ; on data-server restart this metadata must survive so a subsequent revocation-transfer re-emits the same identity. A data server that fails to persist the adopted escrow_id4 (whether by implementation defect or storage corruption) forces the metadata server's tuple bookkeeping into an unresolvable state: the recovered metadata-server escrow appears with a fresh identity that no durable tuple matches, and the composed rollback guarantee () is broken for repairs that crossed the restart. A deployment SHOULD monitor for missing-adopted-id events and treat them as a data-server failure requiring operator investigation.

Transport Layer Security RPC-over-TLS MAY be used to protect traffic between the client and the metadata server and between the client and data servers. When RPC-over-TLS is in use on the data server path, the synthetic uid/gid credentials carried in AUTH_SYS remain the access control mechanism; TLS provides confidentiality and integrity for the transport but does not replace the fencing model described in . Servers that require transport security SHOULD advertise this via the SECINFO mechanism rather than silently dropping connections.

RPCSEC_GSS and Security Services This document does not specify how RPCSEC_GSS is used between the client and a storage device in the loosely coupled model, and the reasons differ between the two coupling models. Because the loosely coupled model uses synthetic credentials that are managed by the metadata server rather than shared with the storage device, a full RPCSEC_GSS integration would require protocol work (RPCSEC_GSSv3 structured privilege assertions, per ) on all three of the metadata server, the storage device, and the client. In the tightly coupled model the principal used to access the data file is the same as the one used to access the metadata file, so RPCSEC_GSS applies unchanged. The two subsections below treat each model in turn.

Loosely Coupled RPCSEC_GSS version 3 (RPCSEC_GSSv3) contains facilities that would allow it to be used to authorize the client to the storage device on behalf of the metadata server. Doing so would require that each of the metadata server, storage device, and client would need to implement RPCSEC_GSSv3 using an RPC-application-defined structured privilege assertion in a manner described in . The specifics necessary to do so are not described in this document. This is principally because any such specification would require extensive implementation work on a wide range of storage devices, which would be unlikely to result in a widely usable specification for a considerable time. As a result, the layout type described in this document will not provide support for use of RPCSEC_GSS together with the loosely coupled model. However, future layout types could be specified, which would allow such support, either through the use of RPCSEC_GSSv3 or in other ways.

Tightly Coupled With tight coupling, the principal used to access the metadata file is exactly the same as used to access the data file. The storage device can use the control protocol to validate any RPC credentials. As a result, there are no security issues related to using RPCSEC_GSS with a tightly coupled system. For example, if Kerberos V5 Generic Security Service Application Program Interface (GSS-API) is used as the security mechanism, then the storage device could use a control protocol to validate the RPC credentials to the metadata server.

Trusted Stateids The TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID operations (, , ) introduce a per-stateid authorization channel between the metadata server and the data server. The security implications of that channel are distinct from those of the loosely coupled synthetic-uid model () and warrant their own treatment.

Interaction with Kerberos and RPCSEC_GSS Trusted stateids decouple the credential the data server uses to authorize I/O from the credential the client uses to authenticate to the data server. Under loose coupling (), the metadata server inserts a synthetic uid/gid into the layout and the client presents that synthetic credential on every data-server RPC; the data server has no independent verification of the client's identity, and a client that learns another client's synthetic uid/gid can impersonate it on the data path. Tight coupling via TRUST_STATEID changes this in three ways:

The metadata server records the client's authenticated principal in the trust entry via tsa_principal at TRUST_STATEID time (). Under RPCSEC_GSS (typically Kerberos V5 GSS-API per ), tsa_principal is the GSS display name (for example, "alice@REALM"); under AUTH_SYS and TLS, tsa_principal is the empty string.
The client presents its own RPCSEC_GSS context on each CHUNK operation against the data server. Under tight coupling with GSS, the data server MUST verify that the principal carried in the inbound RPC's RPCSEC_GSS context matches the tsa_principal recorded for the stateid in its trust table; a mismatch returns NFS4ERR_ACCESS. A client that learned another client's layout stateid (from a log file, a packet capture of cleartext RPC, or any other leak) cannot use it because their own GSS principal would not match.
The data server does NOT need its own Kerberos keytab to validate each client principal individually. In a loose coupling Kerberos deployment the data server would have to be a service principal in every realm it serves clients from; under tight coupling the data server's keytab is only required for its session with the metadata server (the control session, ). Operational complexity of Kerberos deployment is meaningfully reduced.

The mechanism does not authenticate the metadata server to the client; it authenticates the client to the data server using credentials the metadata server vouched for at LAYOUTGET time. Compromise of the metadata server allows an attacker to register arbitrary trust entries; the metadata server is the trust anchor for the layout grant, unchanged from the existing pNFS layout-issuance model.

Attack Surfaces and Mitigations

Compromised metadata server:: An attacker controlling the metadata server can issue TRUST_STATEID for any (layout stateid, principal) pair. This is the same trust assumption pNFS already makes -- the metadata server grants layouts and the data servers honor them. Deployment defense is the same: restrict administrative access to the metadata server, require RPCSEC_GSS or RPC-over-TLS () with mutual authentication on the control session (), and monitor for anomalous TRUST_STATEID volume.
Compromised data server:: A compromised data server sees plaintext chunk payloads at rest and on the wire (subject to whatever the deployment uses for at-rest encryption and transport security). It can return arbitrary content on CHUNK_READ with a correctly computed checksum; the checksum protects against transport corruption, not adversarial content (). This is the same as the RAID-stripe trust model: each shard host can lie about its shard. Deployment defenses are encryption at rest, an integrity-protected transport (RPCSEC_GSS_KRB5I or TLS), and physical or logical isolation of data servers.
Stateid leak from client to attacker:: Under tight coupling with RPCSEC_GSS, a leaked stateid is not exploitable: the attacker's own RPC principal will not match tsa_principal in the trust table, and the data server returns NFS4ERR_ACCESS. Under tight coupling with AUTH_SYS over TLS (where tsa_principal is empty), a leaked stateid is exploitable by any attacker who can also forge the source-address binding the data server's TLS session expects; this is the standard AUTH_SYS-over-TLS trust model, unchanged.
Replay of revoked stateid:: After REVOKE_STATEID or BULK_REVOKE_STATEID the data server removes the trust entry and subsequent CHUNK operations presenting the revoked stateid fail with NFS4ERR_BAD_STATEID (). An in-flight CHUNK operation that arrived before the revoke completed MAY be allowed to finish; the chunk_guard4 CAS () bounds the worst-case damage from such in-flight I/O to the chunks already PENDING at revocation time, and the lock-transfer-to-metadata-server escrow rule () prevents a write hole from opening during revocation.
Compromised control session:: An attacker who controls the metadata-server-to-data-server control session can register or revoke arbitrary trust entries. The control session is the most security-sensitive element introduced by tight coupling. Deployment MUST protect it with RPCSEC_GSS () using a service principal both sides trust, or with RPC-over-TLS () using mutual authentication and allowlisted certificates. The data server enforces that TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID only arrive on sessions whose owning client presented EXCHGID4_FLAG_USE_PNFS_MDS at EXCHANGE_ID (), but that flag alone does not authenticate the metadata server.
Resource exhaustion via trust-table flood:: A misbehaving metadata server could register an unbounded number of TRUST_STATEID entries to exhaust the data server's trust-table memory. The mechanism defending against this is the tsa_expire lease on each entry: trust entries that are not renewed before expiry are reaped by the data server. A data server under memory pressure MAY also return NFS4ERR_DELAY on new TRUST_STATEID requests, forcing the metadata server to back off.
Cross-metadata-server isolation:: In a deployment where two metadata servers share a single data server, the per-entry metadata-server tag (derived from the control session's owning client; see ) ensures that REVOKE_STATEID and BULK_REVOKE_STATEID from one metadata server cannot remove entries registered by the other. A compromised metadata server can, however, register entries against any filehandle the data server exposes to it. Deployments concerned about cross-metadata-server isolation MUST partition the data server's filesystem namespace into per-metadata-server exports at the data server, rather than rely on the trust table alone to enforce file-level boundaries between metadata servers.

A repair actor reconstructs and writes shards on behalf of other clients via CHUNK_WRITE_REPAIR. A malicious or buggy repair actor is therefore a write path into data it did not originate; the metadata server MUST validate repaired shards against the file's registered checksum before accepting them, and integrity against a malicious data server (as opposed to bit-flips) requires a cryptographic checksum_algorithm together with transport security. CHECKSUM_ALG_NONE and the CRC variants provide bit-flip detection only.

IANA Considerations introduced the "pNFS Layout Types Registry"; new layout type numbers in this registry need to be assigned by IANA. This document defines a new layout type number: LAYOUT4_FLEX_FILES_V2 (see ). Layout Type Assignments

Layout Type Name	Value	RFC	How	Minor Versions
LAYOUT4_FLEX_FILES_V2	0x6	RFCTBD10	L	1

also introduced the "NFSv4 Recallable Object Types Registry". This document defines new recallable objects for RCA4_TYPE_MASK_FF2_LAYOUT_MIN and RCA4_TYPE_MASK_FF2_LAYOUT_MAX (see ). Recallable Object Type Assignments

Recallable Object Type Name	Value	RFC	How	Minor Versions
RCA4_TYPE_MASK_FF2_LAYOUT_MIN	20	RFCTBD10	L	1
RCA4_TYPE_MASK_FF2_LAYOUT_MAX	21	RFCTBD10	L	1

This document also requests IANA to register a new bit in the "EXCHGID4FLAG*" flag space for the ExchangeID operation from The requested value is 0x00100000, outside the existing MASK_PNFS block (0x00070000); IANA MAY assign a different value at its discretion, in which case the numeric value in and its uses throughout the document are updated to match the assignment. EXCHGID4 Flag Assignments

Flag Name	Value	RFC	Reference
EXCHGID4_FLAG_USE_ERASURE_DS	0x00100000	RFCTBD10	, this doc

This document requests IANA to allocate two attribute numbers in the NFSv4 attribute-number registry (see ). NFSv4 Attribute Assignments

Attribute Number	Attribute Name	RFC	Reference
89	FATTR4_CODING_BLOCK_SIZE	RFCTBD10
90	FATTR4_CHUNKED_DATA_FILE	RFCTBD10

This document introduces the 'Flexible File Version 2 Layout Type Erasure Encoding Type Registry'. The registry uses a 32-bit value space partitioned into ranges based on the intended scope of the encoding type (see ). Erasure Encoding Type Value Ranges (32-bit space)

Range	Purpose	Allocation Policy
0x0000	Reserved (uninitialized)	--
0x0001-0x00FF	Standards Track	IETF Review
0x0100-0x0FFF	Experimental	Expert Review
0x1000-0x7FFF	Vendor (open)	First Come First Served
0x8000-0xFFFE	Private/proprietary	No registration required
0xFFFF	Reserved	--
0x00010000-0xFFFFFFFF	Reserved (upper range)	Reserved for future partition

The upper 16 bits of the 32-bit value space (0x00010000 through 0xFFFFFFFF) are reserved for future range extensions. A receiver that observes an ffv2_encoding_type4 value in the reserved region MUST treat it as an unsupported encoding type (NFS4ERR_ENCODING_NOT_SUPPORTED). Value 0x0000 is reserved as the uninitialized-field sentinel and MUST NOT be allocated to an encoding.

Standards Track (0x0000-0x00FF):: Enencoding types intended for broad interoperability. The specification MUST include a complete mathematical description sufficient for independent interoperable implementations (see ). Allocated by IETF Review.
Experimental (0x0100-0x0FFF):: Enencoding types under development or evaluation. An Internet-Draft is sufficient for allocation. The specification SHOULD include enough detail for interoperability testing. Allocated by Expert Review.
Vendor (open) (0x1000-0x7FFF):: Enencoding types with a published specification or patent reference. Interoperability is expected among implementations that license or implement the specification. The registration MUST include either a math specification or a patent reference. Allocated First Come First Served.
Private/proprietary (0x8000-0xFFFE):: Enencoding types for use within a single vendor's ecosystem. No IANA registration is required. Interoperability with other implementations is not expected; accidental codepoint collisions between independent vendors are possible and are managed operationally rather than by protocol mechanism. The encoding type name SHOULD include an organizational identifier (e.g., FFV2_ENCODING_ACME_FOOBAR). A client that encounters a value in this range from an unrecognized server SHOULD treat it as an unsupported encoding type (NFS4ERR_ENCODING_NOT_SUPPORTED).

This partitioning prevents contention for small numbers in the Standards Track range and provides a clear signal to clients about what level of interoperability to expect. This document defines seven encoding types: the flexible file v1 layout-compatible PASSTHROUGH (see ), the chunked MIRRORED (see ), and five chunked erasure encoding types (see ). Flexible File Version 2 Layout Type Encoding Type Assignments

Encoding Type Name	Value	RFC	How	Minor Versions
FFV2_ENCODING_PASSTHROUGH	1	RFCTBD10	L	2
FFV2_ENCODING_MOJETTE_SYSTEMATIC	2	RFCTBD10	L	2
FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC	3	RFCTBD10	L	2
FFV2_ENCODING_RS_VANDERMONDE	4	RFCTBD10	L	2
FFV2_ENCODING_REPLICATED	5	RFCTBD10	L	2
FFV2_ENCODING_XOR_PARITY	6	RFCTBD10	L	2
FFV2_ENCODING_LINUX_MD_RAID	7	RFCTBD10	L	2

Checksum Algorithm Registry This document introduces the "Flexible File Version 2 Layout Type Checksum Algorithm Registry". Values in this registry name the checksum_algorithm4 () carried in checksum4 on the wire and selected per-mirror via ffv2m_checksum_algorithm (). The registry uses a 32-bit value space. Registration policy is Specification Required ; the Designated Expert reviews each request for:

a complete and publicly available specification of the algorithm sufficient for independent interoperable implementations;
the exact length of the cs_value field for this algorithm (a single registered length per algorithm; variable-length variants register separately);
collision risk against existing registrations (the Expert MAY decline to register an algorithm whose output overlaps substantially with an existing registration).

Initial registrations are listed in . Initial Checksum Algorithm Registrations

Name	Value	cs_value bytes	Class	RFC
CHECKSUM_ALG_NONE	0	0	none	RFCTBD10
CHECKSUM_ALG_CRC32	1	4	bit-flip	RFCTBD10
CHECKSUM_ALG_CRC32C	2	4	bit-flip	RFCTBD10
CHECKSUM_ALG_FLETCHER4	3	32	bit-flip	RFCTBD10
CHECKSUM_ALG_SHA256	4	32	cryptographic	RFCTBD10
CHECKSUM_ALG_SHA512	5	64	cryptographic	RFCTBD10
CHECKSUM_ALG_BLAKE3	6	32	cryptographic	RFCTBD10

CHECKSUM_ALG_NONE (value 0) indicates that no protocol-level checksum is computed. The deployment relies on transport-layer integrity (RPC-over-TLS, RPCSEC_GSS_KRB5I) or storage-layer integrity instead; see . CHECKSUM_ALG_CRC32 (value 1) is the CRC-32 algorithm specified in Section 8.1.1.6.2 (the same CRC used in Ethernet Section 3.2.9, PNG Annex D, and zlib ). Concrete parameters, which two independent implementations MUST agree on to interoperate: generator polynomial 0x04C11DB7 (equivalently, the reflected form 0xEDB88320); initial register value 0xFFFFFFFF; final XOR value 0xFFFFFFFF; input reflected; output reflected. The 4-byte cs_value carries the CRC as a big-endian integer. Covered bytes follow the uniform coverage rule in (chunk header + chunk payload with the checksum field's bytes treated as zero, in wire-transmission order). Deployments SHOULD prefer CHECKSUM_ALG_CRC32C for new files since CRC32C is hardware-accelerated on every modern CPU. CHECKSUM_ALG_CRC32C (value 2) is the CRC-32 with the Castagnoli polynomial specified in and adopted by (SCTP), and also as the SSE4.2 / ARMv8 / RISC-V CRC-32C hardware-acceleration instructions. Concrete parameters: generator polynomial 0x1EDC6F41 (equivalently, the reflected form 0x82F63B78); initial register value 0xFFFFFFFF; final XOR value 0xFFFFFFFF; input reflected; output reflected. The 4-byte cs_value carries the CRC as a big-endian integer. Covered bytes follow the uniform coverage rule in . CHECKSUM_ALG_FLETCHER4 (value 3) is the ZFS Fletcher4 variant as documented in the OpenZFS on-disk format specification . Covered bytes follow the uniform coverage rule in ; this algorithm additionally requires the covered input (header + chunk with the checksum field's bytes treated as zero) to be a multiple of 4 bytes, since the input is processed as a sequence of little-endian 32-bit words. Implementations that need to checksum a chunk whose covered input is not a multiple of 4 bytes pad with zero bytes and register the padded variant separately. Concrete parameters: the four 64-bit accumulators A, B, C, D are updated per word wi as A += wi; B += A; C += B; D += C with 64-bit unsigned wrap-around; the 32-byte cs_value is the concatenation A || B || C || D with each accumulator serialized in big-endian byte order. Other Fletcher4 implementations (different word width, different endianness, truncated output) register separately. CHECKSUM_ALG_SHA256 (value 4) and CHECKSUM_ALG_SHA512 (value 5) are the SHA-256 and SHA-512 hash algorithms specified in , with output byte lengths 32 and 64 respectively. The cs_value carries the hash output in the byte order defined by Section 3.1 (most-significant word first, each word serialized big-endian). Covered bytes follow the uniform coverage rule in . CHECKSUM_ALG_BLAKE3 (value 6) is the BLAKE3 hash algorithm specified in at its standard 32-byte output length (BLAKE3 in its default mode, no keyed hash, no key-derivation context, no XOF output at other lengths). Extended-output BLAKE3, keyed BLAKE3, and the key-derivation mode register as separate algorithms. Covered bytes follow the uniform coverage rule in ; cs_value is the 32-byte hash output in the byte order defined by Section 2.4. A checksum4 whose cs_value length does not match the registered cs_value bytes for its cs_algorithm MUST be rejected with NFS4ERR_INVAL. The "Class" column in is informational and indicates the threat model the algorithm supports; see .

Proof-Profile Registry This document introduces the "Flexible File Version 2 Proof-Profile Registry". Values in this registry name the proof_profile_id4 () that a metadata server presents in CHUNK_ESCROW_TAKEOVER to identify which incarnation-lease proof format the accompanying ceta_proof_data carries. The registry uses a 32-bit value space. Registration policy is Specification Required ; the Designated Expert reviews each request for:

a complete and publicly available specification of the proof format sufficient for independent interoperable implementations (envelope, signature algorithm, payload field set, replay-cache semantics, and expiry handling);
compatibility with the strict evaluation ordering specified in (session replay -> presenter authorization -> profile support -> proof verification -> epoch compare-and-advance); a registration whose verification cannot be evaluated after presenter authorization MUST be declined;
a clear statement of the trust anchor the profile relies on (deployment-provisioned public key set, Kerberos KDC, etc.), and whether the profile requires wire negotiation of any parameter (a profile that requires wire negotiation not defined in this document or its cited references MUST be declined).

Initial registrations are listed in . Initial Proof-Profile Registrations

Name	Value	Reference
PROOF_PROFILE_UNSPECIFIED	0	Reserved sentinel; see
PROOF_PROFILE_HA_AUTHORITY_ED25519	1	"Mandatory-to-Implement Profile"

Additional profiles named in (ECDSA-P256 alg -7, RSASSA-PSS-SHA256 alg -37) MAY be added later per the Specification Required policy above. The value range 0xC0000000 to 0xFFFFFFFF is reserved for Private Use per ; deployments MAY assign values from this range for experimental, vendor-specific, or private profiles without IANA registration, but such values MUST NOT be presented on interoperability boundaries and this specification makes no compatibility guarantees for them.

XDR Description of the Flexible File Version 2 Layout Type This document contains the External Data Representation (XDR) description of the flexible file v2 layout. The XDR description is embedded in this document in a way that makes it simple for the reader to extract into a ready-to-compile form. The reader can feed this document into the shell script in to produce the machine-readable XDR description of the flexible file v2 layout.

extract.sh That is, if the above script is stored in a file called "extract.sh" and this document is in a file called "spec.txt", then the reader can run the script as in .

Example use of extract.sh flex_files2_prot.x ]]> The effect of the script is to remove leading blank space from each line, plus a sentinel sequence of "///". XDR descriptions with the sentinel sequence are embedded throughout the document. Note that the XDR code contained in this document depends on types from the NFSv4.2 nfs4_prot.x file (which itself builds on ). This includes both nfs types that end with a 4, such as offset4, length4, etc., as well as more generic types such as uint32_t and uint64_t. While the XDR can be appended to that from , the various code snippets belong in their respective areas of that XDR.

References Normative References Delta-Write Extension for the Flexible File Version 2 Layout Type Hammerspace The Flexible File Version 2 pNFS layout type defines a chunk-oriented data-server protocol in which every write is a full-chunk payload. For workloads that make small edits to files protected by an XOR- based erasure encoding, this forces client-side stripe fetch, re- encode, and transmit on every edit, with wire amplification of three to four orders of magnitude per byte edited. This document defines an optional extension, CHUNK_XOR_DELTA, that lets a client transmit a per-projection XOR delta directly to each data server holding a projection of the affected stripe; the data server applies the delta locally. The extension is restricted to XOR-linear systematic encodings and XOR-affine checksums, using the existing chunk state machine with no new commit protocol. The Kerberos Version 5 Generic Security Service Application Program Interface (GSS-API) Mechanism: Version 2 This document defines protocols, procedures, and conventions to be employed by peers implementing the Generic Security Service Application Program Interface (GSS-API) when using the Kerberos Version 5 mechanism. RFC 1964 is updated and incremental changes are proposed in response to recent developments such as the introduction of Kerberos cryptosystem framework. These changes support the inclusion of new cryptosystems, by defining new per-message tokens along with their encryption and checksum algorithms based on the cryptosystem profiles. [STANDARDS-TRACK] XDR: External Data Representation Standard This document describes the External Data Representation Standard (XDR) protocol as it is currently deployed and accepted. This document obsoletes RFC 1832. [STANDARDS-TRACK] RPC: Remote Procedure Call Protocol Specification Version 2 This document describes the Open Network Computing (ONC) Remote Procedure Call (RPC) version 2 protocol as it is currently deployed and accepted. This document obsoletes RFC 1831. [STANDARDS-TRACK] Network File System (NFS) Version 4 Minor Version 1 External Data Representation Standard (XDR) Description This document provides the External Data Representation Standard (XDR) description for Network File System version 4 (NFSv4) minor version 1. [STANDARDS-TRACK] Network File System (NFS) Version 4 Protocol The Network File System (NFS) version 4 protocol is a distributed file system protocol that builds on the heritage of NFS protocol version 2 (RFC 1094) and version 3 (RFC 1813). Unlike earlier versions, the NFS version 4 protocol supports traditional file access while integrating support for file locking and the MOUNT protocol. In addition, support for strong security (and its negotiation), COMPOUND operations, client caching, and internationalization has been added. Of course, attention has been applied to making NFS version 4 operate well in an Internet environment. This document, together with the companion External Data Representation (XDR) description document, RFC 7531, obsoletes RFC 3530 as the definition of the NFS version 4 protocol. Remote Procedure Call (RPC) Security Version 3 This document specifies version 3 of the Remote Procedure Call (RPC) security protocol (RPCSEC_GSS). This protocol provides support for multi-principal authentication of client hosts and user principals to a server (constructed by generic composition), security label assertions for multi-level security and type enforcement, structured privilege assertions, and channel bindings. This document updates RFC 5403. Network File System (NFS) Version 4 Minor Version 2 Protocol This document describes NFS version 4 minor version 2; it describes the protocol extensions made from NFS version 4 minor version 1. Major extensions introduced in NFS version 4 minor version 2 include the following: Server-Side Copy, Application Input/Output (I/O) Advise, Space Reservations, Sparse Files, Application Data Blocks, and Labeled NFS. Network File System (NFS) Version 4 Minor Version 2 External Data Representation Standard (XDR) Description This document provides the External Data Representation (XDR) description for NFS version 4 minor version 2. Guidelines for Writing an IANA Considerations Section in RFCs Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. Rules for NFSv4 Extensions and Minor Versions This document describes the rules relating to the extension of the NFSv4 family of protocols. It covers the creation of minor versions, the addition of optional features to existing minor versions, and the correction of flaws in features already published as Proposed Standards. The rules relating to the construction of minor versions and the interaction of minor version implementations that appear in this document supersede the minor versioning rules in RFC 5661 and other RFCs defining minor versions. Requirements for Parallel NFS (pNFS) Layout Types This document defines the requirements that individual Parallel NFS (pNFS) layout types need to meet in order to work within the pNFS framework as defined in RFC 5661. In so doing, this document aims to clearly distinguish between requirements for pNFS as a whole and those specifically directed to the pNFS file layout. The lack of a clear separation between the two sets of requirements has been troublesome for those specifying and evaluating new layout types. In this regard, this document updates RFC 5661. Parallel NFS (pNFS) Flexible File Layout Parallel NFS (pNFS) allows a separation between the metadata (onto a metadata server) and data (onto a storage device) for a file. The flexible file layout type is defined in this document as an extension to pNFS that allows the use of storage devices that require only a limited degree of interaction with the metadata server and use already-existing protocols. Client-side mirroring is also added to provide replication of files. Network File System (NFS) Version 4 Minor Version 1 Protocol This document describes the Network File System (NFS) version 4 minor version 1, including features retained from the base protocol (NFS version 4 minor version 0, which is specified in RFC 7530) and protocol extensions made subsequently. The later minor version has no dependencies on NFS version 4 minor version 0, and is considered a separate protocol. This document obsoletes RFC 5661. It substantially revises the treatment of features relating to multi-server namespace, superseding the description of those features appearing in RFC 5661. Towards Remote Procedure Call Encryption by Default This document describes a mechanism that, through the use of opportunistic Transport Layer Security (TLS), enables encryption of Remote Procedure Call (RPC) transactions while they are in transit. The proposed mechanism interoperates with Open Network Computing (ONC) RPC implementations that do not support it. This document updates RFC 5531. Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. CBOR Object Signing and Encryption (COSE): Structures and Process Concise Binary Object Representation (CBOR) is a data format designed for small code size and small message size. There is a need to be able to define basic security services for this data format. This document defines the CBOR Object Signing and Encryption (COSE) protocol. This specification describes how to create and process signatures, message authentication codes, and encryption using CBOR for serialization. This specification additionally describes how to represent cryptographic keys using CBOR. This document, along with RFC 9053, obsoletes RFC 8152. Concise Binary Object Representation (CBOR) The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. These design goals make it different from earlier binary serializations such as ASN.1 and MessagePack. This document obsoletes RFC 7049, providing editorial improvements, new details, and errata fixes while keeping full compatibility with the interchange format of RFC 7049. It does not create a new version of the format. CBOR Object Signing and Encryption (COSE): Initial Algorithms Concise Binary Object Representation (CBOR) is a data format designed for small code size and small message size. There is a need to be able to define basic security services for this data format. This document defines a set of algorithms that can be used with the CBOR Object Signing and Encryption (COSE) protocol (RFC 9052). This document, along with RFC 9052, obsoletes RFC 8152. Informative References A Tutorial on Reed-Solomon Coding for Fault-Tolerance in RAID-like System Private Enterprise Numbers IANA NFS Version 3 Protocol Specification This paper describes the NFS version 3 protocol. This paper is provided so that people can write compatible implementations. This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. ZLIB Compressed Data Format Specification version 3.3 This specification defines a lossless compressed data format. This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. PNG (Portable Network Graphics) Specification Version 1.0 This document describes PNG (Portable Network Graphics), an extensible file format for the lossless, portable, well-compressed storage of raster images. PNG provides a patent-free replacement for GIF and can also replace many common uses of TIFF. Indexed-color, grayscale, and truecolor images are supported, plus an optional alpha channel. Sample depths range from 1 to 16 bits. PNG is designed to work well in online viewing applications, such as the World Wide Web, so it is fully streamable with a progressive display option. PNG is robust, providing both full file integrity checking and simple detection of common transmission errors. Also, PNG can store gamma and chromaticity data for improved color matching on heterogeneous platforms. This specification defines the Internet Media Type image/png. Internet Small Computer Systems Interface (iSCSI) This document describes a transport protocol for Internet Small Computer Systems Interface (iSCSI) that works on top of TCP. The iSCSI protocol aims to be fully compliant with the standardized SCSI architecture model. SCSI is a popular family of protocols that enable systems to communicate with I/O devices, especially storage devices. SCSI protocols are request/response application protocols with a common standardized architecture model and basic command set, as well as standardized command sets for different device classes (disks, tapes, media-changers etc.). As system interconnects move from the classical bus structure to a network structure, SCSI has to be mapped to network transport protocols. IP networks now meet the performance requirements of fast system interconnects and as such are good candidates to "carry" SCSI. [STANDARDS-TRACK] Lightweight Directory Access Protocol (LDAP): Schema for User Applications This document is an integral part of the Lightweight Directory Access Protocol (LDAP) technical specification. It provides a technical specification of attribute types and object classes intended for use by LDAP directory clients for many directory services, such as White Pages. These objects are widely used as a basis for the schema in many LDAP directories. This document does not cover attributes used for the administration of directory servers, nor does it include directory objects defined for specific uses in other documents. [STANDARDS-TRACK] Stream Control Transmission Protocol This document obsoletes RFC 2960 and RFC 3309. It describes the Stream Control Transmission Protocol (SCTP). SCTP is designed to transport Public Switched Telephone Network (PSTN) signaling messages over IP networks, but is capable of broader applications. SCTP is a reliable transport protocol operating on top of a connectionless packet network such as IP. It offers the following services to its users: -- acknowledged error-free non-duplicated transfer of user data, -- data fragmentation to conform to discovered path MTU size, -- sequenced delivery of user messages within multiple streams, with an option for order-of-arrival delivery of individual user messages, -- optional bundling of multiple user messages into a single SCTP packet, and -- network-level fault tolerance through supporting of multi-homing at either or both ends of an association. The design of SCTP includes appropriate congestion avoidance behavior and resistance to flooding and masquerade attacks. [STANDARDS-TRACK] Network Time Protocol Version 4: Protocol and Algorithms Specification The Network Time Protocol (NTP) is widely used to synchronize computer clocks in the Internet. This document describes NTP version 4 (NTPv4), which is backwards compatible with NTP version 3 (NTPv3), described in RFC 1305, as well as previous versions of the protocol. NTPv4 includes a modified protocol header to accommodate the Internet Protocol version 6 address family. NTPv4 includes fundamental improvements in the mitigation and discipline algorithms that extend the potential accuracy to the tens of microseconds with modern workstations and fast LANs. It includes a dynamic server discovery scheme, so that in many cases, specific server configuration is not required. It corrects certain errors in the NTPv3 design and implementation and includes an optional extension mechanism. Improving Awareness of Running Code: The Implementation Status Section This document describes a simple process that allows authors of Internet-Drafts to record the status of known implementations by including an Implementation Status section. This will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. This process is not mandatory. Authors of Internet-Drafts are encouraged to consider using the process for their documents, and working groups are invited to think about applying the process to all of their protocol specifications. This document obsoletes RFC 6982, advancing it to a Best Current Practice. Secure Hash Standard (SHS) National Institute of Standards and Technology BLAKE3: one function, fast everywhere Error-correcting Procedures for DCEs Using Asynchronous-to-Synchronous Conversion International Telecommunication Union IEEE Standard for Ethernet IEEE OpenZFS On-Disk Format Specification, Section 2.2.4: Fletcher OpenZFS Multiple Description Coding Using Exact Discrete Radon Transform A Geometry Driven Reconstruction Algorithm for the Mojette Transform Questions of Uniqueness and Resolution in Reconstruction from Projections Linux kernel software RAID (md/raid6) -- lib/raid6 Linux kernel contributors dCache -- a distributed storage system dCache collaboration (DESY, Fermilab, NDGF) Proxy-Driven Server for Flexible Files Version 2 Hammerspace Parallel NFS (pNFS) with the Flexible Files Version 2 layout type supports client-side erasure coding and per-chunk repair between clients and data servers. This document extends that architecture with a proxy server role: a registered peer of the metadata server that polls the metadata server for work assignments and carries them out -- moving a file from one layout to another, reconstructing a whole file from surviving shards, or translating between encodings for clients that cannot participate in the file's native encoding (including NFSv3 clients). All proxy-server-to-metadata-server coordination is fore-channel: the metadata server returns work assignments inline in the response to a proxy-server-initiated PROXY_PROGRESS poll, and the proxy server reports completion via a fore-channel PROXY_DONE. No callback operations are required for the proxy server protocol.

Implementation Status This appendix records the implementation status of this specification at the time of writing. The purpose, per , is to help reviewers evaluate the protocol against running code and to document which parts have been validated end-to-end versus specified on paper. This appendix is reviewer-aid material and is removed from the final RFC.

reffs (metadata server, data server, and ec_demo client)

Organization:: Independent / open source.
License:: AGPL-3.0-or-later.
Source:: https://github.com/loghyr/reffs.
Implementation:: reffs is an NFSv4.2 server written in C that acts as both a metadata server and a data server in a flexible file v2 layout deployment. A separate binary implements the proxy server role defined in the proxy server draft. ec_demo is a client-side library with a demonstration driver that exercises the flexible file v2 layout data path over NFSv4.2.

Coverage:

CHUNK_WRITE, CHUNK_READ, CHUNK_FINALIZE, and CHUNK_COMMIT (the happy-path data-plane operations) are implemented end-to-end and have been exercised against multiple encoding families.
CHUNK_WRITE_REPAIR and CHUNK_REPAIRED (client-driven single-shard reconstruction with server-side layout-flag clearing) are implemented end-to-end and have been exercised across four file sizes, two encoding families, and one- and two-shard-loss patterns; end-to-end integrity verification passes on substantially all measured cells.
The chunk_guard4 CAS primitive, including the conflict-detection and deterministic-tiebreaker rules in , is implemented on both the client and the data server.
Per-chunk checksum integrity checking (see ) is implemented end-to-end.
Per-inode persistent storage of chunk state (PENDING / FINALIZED / COMMITTED) is implemented using write-temp / fdatasync / rename for crash safety.
Encoders for FFV2_ENCODING_REPLICATED, FFV2_ENCODING_PASSTHROUGH, FFV2_ENCODING_XOR_PARITY, FFV2_ENCODING_LINUX_MD_RAID, FFV2_ENCODING_RS_VANDERMONDE, FFV2_ENCODING_MOJETTE_SYSTEMATIC, and FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC are all implemented and cross-verified against the wire-compatibility relationships described in the per-encoding sections of this document.
The tight coupling control protocol (TRUST_STATEID, REVOKE_STATEID, BULK_REVOKE_STATEID) is specified but not yet implemented. Data servers currently advertise ffv2dv_coupling = FFV2_COUPLING_SYNTHETIC_UIDS, and synthetic AUTH_SYS credentials with fencing are used for access control.
The proxy server mediated repair callback CB_PROXY_REPAIR is specified but not yet implemented. Single-shard repair is currently client-driven via ec_demo.

Level of maturity:: Research-quality prototype. The implementation demonstrates the protocol and has produced the benchmark data summarized below. It is not production-ready.
Contact:: loghyr@gmail.com.
Last update:: August 2026.

Linux kernel flexible file v2 client

Organization:: Independent / open source. Out-of-tree kernel module built against a mainline Linux release candidate.
License:: GPL-2.0.
Source:: Topic branch tracked against a mainline Linux release candidate; contact the author for the current tree pointer.
Implementation:: A native pNFS layout driver at fs/nfs/flexfilesv2/ implementing the flexible file v2 layout for the Linux NFS client. Layout registration, XDR decode of ffv2_layout4, device-info discovery, striped and mirrored read and write, and the CHUNK operation wire path are implemented as a peer layout driver alongside the existing flexible file v1 driver.

Coverage:

CHUNK_WRITE, CHUNK_READ, CHUNK_FINALIZE, and CHUNK_COMMIT are implemented end-to-end and wire-verified against the reffs metadata server + data server on a 64-bit ARM Linux host at the 7.2-rc series.
Per-chunk checksum integrity, chunk_guard4 CAS, writeback-chain sequencing (CHUNK_WRITE -> CHUNK_FINALIZE -> CHUNK_COMMIT), and DENSE per-shard offset canonicalization are implemented.
Encoders for FFV2_ENCODING_REPLICATED, FFV2_ENCODING_XOR_PARITY, FFV2_ENCODING_LINUX_MD_RAID, and FFV2_ENCODING_RS_VANDERMONDE are present. FFV2_ENCODING_MOJETTE_SYSTEMATIC is scaffolded (returns -EOPNOTSUPP; native kernel implementation deferred).
NFSv3 and NFSv4.2 data-server dispatch are both implemented.
NFS4ERR_DELAY retry-with-backoff for concurrent writer contention on CHUNK_WRITE is not yet implemented; multi-writer workloads fall back to routing writes through the metadata server.
Client-side single-shard repair write-back is not yet implemented in the kernel client. Reconstruction is available via ec_demo against the same metadata server.

Level of maturity:: Developer preview. Wire-verified on the CHUNK operation happy path against the reffs server.
Contact:: loghyr@gmail.com.
Last update:: August 2026.

Interoperability and Benchmarks Two independent implementations of the client role now exist (the ec_demo userspace library and the Linux kernel layout driver above), and wire-compatibility between them on the CHUNK operation data path has been demonstrated against the reffs metadata server. In addition, the Mojette encoders in this document have been cross-verified against an independent Mojette implementation with byte-identical output for FFV2_ENCODING_MOJETTE_SYSTEMATIC and FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC. The benchmark suite is now organized to distinguish two costs that were previously conflated in a single-host Docker measurement:

Algorithm cost:: the CPU + memory-bandwidth cost of encoding and decoding, measured against pre-allocated RAM buffers with no I/O and no network. This is the encoder ceiling on a given host.
Transmit cost:: the end-to-end cost of a write or read on a real NFSv4.2 mount across three hosts on a LAN, including RPC round-trips, fsync commits, and network serialization.

The two-axis measurement makes explicit what the previous single-host measurement obscured: on a real network, encoder algorithm cost is a small fraction of end-to-end transmit cost at typical operating points.

Algorithm cost Algorithm cost has been measured at k=4, m=2, 64 KiB shards across the encoders defined in this document. Reference hosts spanned four CPU classes: Apple silicon (aarch64, NEON), Intel x86_64 (SSE+AVX2), AMD Zen 2 mobile (SSE+AVX2), and ten uniform AMD Zen 3 virtual machines (variance across the fleet under 5% on all encoders). Across these encoders and the four host classes, algorithm-cost spread on the same host reaches approximately two orders of magnitude at k=4, m=2 (the fastest SIMD-vectorized encoders approach the memory-bandwidth ceiling in the tens of GB/s; the slowest scalar encoders are compute-bound in the low hundreds of MB/s). Two hand-tuning passes on the Reed-Solomon Vandermonde encoder in this document, performed without any change to the wire format or test vectors, compounded to roughly 17x on aarch64 NEON and roughly 39x on x86_64 SSSE3 -- a precomputed field-multiplication table pass followed by a SIMD byte-shuffle field arithmetic pass (vqtbl1q_u8 / pshufb). A similar implementation-only optimization pass on the Mojette encoder wrapper yielded roughly 4.85x on x86_64 AVX2, again with byte-identical output. The generalization is that a naive per-encoder microbenchmark cell measures implementation quality on that host, not the encoding algorithm. Wide reported spreads across encoders in the literature reflect this in large part.

Wire compatibility across encoders Cross-encoder byte-identity holds among the GF(2^8) encoders defined in this document at low parity counts, verified by encoding on one implementation and decoding on another byte-for-byte:

At m = 1: FFV2_ENCODING_XOR_PARITY, the P row of FFV2_ENCODING_LINUX_MD_RAID, and the m=1 parity row of FFV2_ENCODING_RS_VANDERMONDE all emit byte-identical output for the same (k, data) input.
At m = 2: FFV2_ENCODING_LINUX_MD_RAID and FFV2_ENCODING_RS_VANDERMONDE emit byte-identical output for the same (k, data) input, provided RS_VANDERMONDE uses the hand-crafted P + Q parity rows this document defines (see ).
At m >= 3: the encoders diverge; RS_VANDERMONDE reverts to normalized-Vandermonde bottom rows and no cross-encoder byte-identity holds.

The practical consequence is that a receiver that supports any one of the m <= 2 members consumes bytes emitted by any of the others without re-encoding.

Transmit cost -- three-host real-network sweep A three-host LAN measurement distributes the roles: one client host (kernel NFSv4.2 mount), one proxy server host, and one host running both the metadata server and the data servers. Four wire-path variants are distinguished:

Variant	Client wire	Encoder location	Path hops
a	FFv1 -> 1 DS	none (baseline)	1
b	FFv1 striped	none (fan-out only)	1
c	FFv2 -> DSes	client	1
d	FFv2 -> PS -> DSes	proxy server	2

A 180-cell sweep across five encoders x four variants x three file sizes x three iterations verified end-to-end data integrity in every cell. Median write throughput at 1 MiB was:

Variant a (FFv1, single data server): approximately 3.2 to 5.1 MB/s across encoders.
Variant b (FFv1, striped): approximately 9.3 to 13.0 MB/s across encoders.
Variant c (FFv2, client-direct): approximately 10.0 to 13.5 MB/s across encoders.
Variant d (FFv2, via proxy server): approximately 1.1 to 2.2 MB/s across encoders.

The key finding: an approximately three-order-of-magnitude algorithm-cost spread across encoders collapses to approximately 1.15x wire spread at the client-direct FFv2 variant (variant c) at 1 MiB. End-to-end throughput on this topology is dominated by RPC round-trips, fsync commits, and network serialization; encoder algorithm cost is a rounding-error contributor at these operating points. The approximately seven-fold variant d penalty (1.1-2.2 MB/s vs 10.0-13.5 MB/s) is the extra client to proxy server hop, not the encoder. Decomposing a 1 MiB write on this topology into cost components:

Cost item	ms/MiB
Fastest SIMD encode (algorithm only)	0.05
Reed-Solomon Vandermonde SSSE3 encode	1.7
Reed-Solomon Vandermonde scalar encode (pre-opt)	66
Variant a: FFv1 to 1 DS (no EC, baseline)	215
Variant b: FFv1 striped (no EC, fan-out)	95
Variant c: FFv2 direct to DSes (client EC)	92
Variant d: FFv2 via PS (PS EC)	610-910

The variant c wire floor of approximately 92 ms/MiB is where end-to-end throughput lives; encoder algorithm cost accounts for well under 2 ms of that budget for every SIMD-tuned implementation measured.

Cost of fault tolerance -- single-shard repair Client-driven single-shard reconstruction, using the wire-level OP_CHUNK_WRITE_REPAIR + OP_CHUNK_REPAIRED operations this document defines, was benchmarked on a colocated topology across the following axes:

File sizes: 4 KB, 64 KB, 1 MB, 16 MB
Encoding families: Reed-Solomon Vandermonde and Mojette systematic (both at k=4, m=2)
Loss patterns: one shard missing, two shards missing
Five iterations per cell (80 cells total)

Substantially all cells passed end-to-end integrity verification. Median repair time at 1 MB, RS 4+2, one shard lost was approximately 80 ms; at 16 MB, approximately 990 ms. Mojette systematic at the same operating points was approximately 72 ms and 900 ms respectively. Repair cost decomposes as degraded-read cost + write-back cost per lost shard (not (k + m) writes -- CHUNK_WRITE_REPAIR is targeted). At file sizes >= 64 KB, repair cost is within one order of magnitude of a healthy write of the same size -- not the catastrophic overhead the phrase "erasure coding repair" might imply. Healthy no-loss reads on any systematic encoding pay zero decode cost (the shard is copied through unchanged).

Encoder-family trade-offs The wire-spread convergence does not mean encoder choice is irrelevant. It means encoder choice is decided by properties other than raw algorithm speed at typical operating points:

Fault tolerance and geometry:: FFV2_ENCODING_RS_VANDERMONDE and the Mojette family support arbitrary (k, m); FFV2_ENCODING_XOR_PARITY is m = 1 only; FFV2_ENCODING_LINUX_MD_RAID is m = 2 only.
Interoperability:: the m <= 2 byte-identical set described above lets a deployment mix implementations of different encoders at the same k and m without cross-encoding.
Reconstruction cost:: systematic encodings (FFV2_ENCODING_RS_VANDERMONDE, FFV2_ENCODING_MOJETTE_SYSTEMATIC, FFV2_ENCODING_XOR_PARITY, FFV2_ENCODING_LINUX_MD_RAID) short-circuit no-loss reads at wire speed; FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC transforms every shard on every read.
Wide-geometry scaling:: at k >= 8, m >= 4, the Mojette back-projection reconstruction cost scales with m (parity count) rather than k (data count), so its reconstruction overhead does not exhibit the O(k^3) growth Reed-Solomon matrix inversion incurs at wider geometries.
Implementation availability:: FFV2_ENCODING_XOR_PARITY has no external dependency; FFV2_ENCODING_LINUX_MD_RAID's reference construction is present in every Linux kernel at lib/raid6/ (). The remaining encoders in this document have reference implementations in the sources cited above.

A full benchmark report with per-cell tables, per-host medians, and per-variant transmit decompositions is available alongside the source code.

Architectural Implication: Cost of Fault Tolerance The headline question every storage audience asks of an erasure-coding protocol is: "what does it cost when something goes wrong?" At the systematic-encoding operating points measured (Mojette systematic at 4+2 and 8+2), the benchmark answer is essentially zero. Mojette systematic at 4+2 reconstructs a missing data shard with read-latency overhead within run-to-run noise of healthy operation. Mojette systematic at 8+2 holds at approximately +4%. This shifts the deployment conversation away from "is erasure coding cheap enough to enable" and toward "which encoding and geometry minimize the compromise." The compromise that remains is not the cost of fault tolerance; it is the cost of write-time encoding, which is bounded (under 60% at 1 MB, under 25% at 64 KB), and the cost of crash-safe durability via the chunk state machine (see ), which is +7% to +22% on writes and +2% to +10% on reads. Wire-format performance objections raised earlier in the working group's review of this work are addressed in : the per-RPC byte-shuffling cost of the original Mojette-specific projection header has been replaced with XDR-encoded chunk metadata (see ), so the remaining wire format cost is the XDR-encoded chunk header itself, which is identical for every encoding and is part of the +7% to +22% v2 write overhead measured above.

Design Rationale: Rejected Alternatives This appendix records design alternatives that were considered and rejected during the development of this specification. It is reviewer-aid material in this draft and is retained in the final RFC as design-history context for future implementers; the alternatives below are not part of the normative specification. The design of flexible file v2 layout went through several iterations between 2024 and 2026 that are recorded here for the benefit of future reviewers and implementers. Each alternative below was considered and rejected, with the specific concern that led to its rejection. Understanding why these approaches were rejected may help reviewers evaluate the current design against a fuller space of possibilities and may guide future extensions or replacements.

Proprietary Projection Header Inside Opaque Payload The earliest iteration placed a 16-byte Mojette-specific header at the start of the READ/WRITE opaque payload, interpreted in the endianness of the writer's host. The motivation was concrete: NFSv3 READ and WRITE arguments carry data as opaque data<> and provide no XDR room for per-write structured metadata such as encoding geometry, integrity, or write-ordering tiebreakers. An NFSv3 server cannot be extended; if a flexible file v2 layout deployment wanted an NFSv3 server to participate as a data server in an erasure-coded layout, the only place to put encoding metadata was inside that opaque payload, prepended to the data bytes. The data server stored the entire opaque blob without interpreting it; the reader peeled the 16-byte prefix off and acted on it. This was rejected because:

It embedded a specific erasure encoding type (Mojette) into the generic replication-method framework, preventing alternate codings from reusing the same wire format.
The header bytes were not XDR-aligned, which required every implementation to handle endianness explicitly rather than relying on XDR's natural byte order.
Carrying integrity and identification data inside an opaque disrespected the XDR self-description model that the rest of NFSv4 relies on. A generic NFSv3 inspector watching the wire could not tell those bytes apart from application data, which among other things made debugging, traffic analysis, and middlebox processing rely on out-of-band knowledge.

The endianness objection raised at IETF 120 (July 2024) was the surface complaint; the structural objection -- that smuggling structured fields through an opaque type bypasses XDR's self-description -- was the deeper reason the working group declined the approach. Once the design accepted that data servers in a flexible file v2 layout deployment would speak NFSv4.2 (with new ops in this document), the constraint that forced the smuggling disappeared: chunk metadata could be expressed as proper XDR fields in CHUNK_WRITE / CHUNK_READ / chunk_guard4, visible to every observer of the wire.

Per-Client Swap Files with metadata server MAPPING_RECALL One proposal split logical and physical chunk addressing: the metadata server maintained a mapping from logical offset to physical location, and the client appended new chunks to a per-client staging file on each data server before asking the metadata server to atomically remap the file to the new chunks. This was rejected because:

The MAPPING_RECALL operation required to atomically update the mapping would, in a multi-writer deployment, have to recall all outstanding read/write layouts on the file -- grinding the application to a halt during every remap.
Each client required its own staging file on every data server, producing N clients * M data servers staging files that had to be reconciled on client restart.
The approach was biased toward correctness at the expense of throughput, which inverted the expected workload mix where single writer cases dominate.

Server-Side Byte-Range Lock Manager per File Another proposal relied on byte range locks obtained by clients before writing, with the lock manager state spread across the data servers. This was rejected because:

A failed lock holder required a lock manager to arbitrate recovery, effectively reintroducing a centralized decision point for each chunk.
The lock recall path for HPC checkpoint workloads (many ranks writing disjoint regions) would have required thousands of locks per file, with recall storms on every phase transition.
The design did not specify how the lock manager itself would be replicated for high availability, deferring the hardest part of the problem.

The current design uses CHUNK_LOCK (see ) but only on the repair path, not on the normal write path.

Modified Two-Touch Paxos on Each Chunk A fully distributed-consensus proposal placed a lightweight (modified two-touch) Paxos round on each chunk write, reaching agreement among the data servers holding the mirror set. This was rejected because:

The constant-factor cost per write (two or three round trips, leader election overhead, majority quorum requirement) was unacceptable for workloads where single writer throughput dominates the deployment mix.
The approach demanded that data servers be peers in a consensus protocol, which is a substantially heavier requirement than being independent chunk stores.
A majority of (k + m) data servers must be reachable for any progress, which is a strictly stronger availability requirement than the k-of-(k + m) needed for erasure-coded reads.

Working-group feedback on this proposal was uniformly negative. The current design retains the option -- nothing in this specification prevents an implementation from running classical consensus internally among metadata server replicas (see ) -- but does not require it per write.

Automatic Commit of Empty Chunks An earlier version included a WRITE_BLOCK_FLAGS_COMMIT_IF_EMPTY flag (later renamed CHUNK_WRITE_FLAGS_ACTIVATE_IF_EMPTY) that automatically committed a write to a previously-empty chunk without a separate CHUNK_COMMIT round trip. The flag is retained in the current design but its scope was narrowed: it is performant in the exclusive-writer case but produces blocks that cannot be rolled back if a racing writer appears concurrently, requiring either hole-punching or an extension of CHUNK_ROLLBACK to work on committed blocks. The narrow scope is documented in the flag's definition; a broader version was rejected because it created rollback liabilities that were disproportionate to the single-RTT savings.

Global Clock or Wall-Clock-Based Generation Counter An early design used a wall-clock timestamp as the cg_gen_id. This was rejected because:

No global clock exists among the many clients of a multi-rack deployment. Clock skew can cause a newer write to appear to have an earlier timestamp than an older one.
Timestamps at millisecond or microsecond resolution are not fine-grained enough to disambiguate bursty writes from the same client.
Mixing client identity bits into the low-order bits of a timestamp (to make it unique) reduces effective timestamp resolution without providing a useful total ordering.

The current design uses a per-chunk monotonic counter scoped to the chunk on the data server, with cg_client_id as the disambiguator across clients. See .

Layout-Level Generation Counter An alternative raised at IETF 122 (March 2025) was adding a generation counter to the layout itself, transmitted to the data servers alongside each I/O, so that the metadata server could redirect writes to new data servers without issuing a full CB_LAYOUTRECALL storm across every holder of the file. This is a natural extension of the per-chunk cg_gen_id: where cg_gen_id disambiguates successive writes to the same chunk, a layout-level counter would disambiguate successive placements of the same data. This was rejected because:

The use case is already covered. CB_CHUNK_REPAIR (see ) and the proxy server mechanism together handle mid-layout remap without requiring a layout-level epoch on the wire. CB_CHUNK_REPAIR reaches the specific chunks that need redirection; the proxy server reaches the broader re-placement case; between them the full remap space is covered.
Adding a layout-level counter introduces a second, potentially-conflicting epoch alongside cg_gen_id. The CAS semantics on the data server would have to compose the two generations (per-chunk and per-layout), which multiplies the states the data server must reason about without strengthening any guarantee the protocol offers today.
The CB_LAYOUTRECALL storm that motivated the proposal is a worst-case cost that the current design pays only during a genuine data-server retirement or full re-placement. Partial remaps -- the common case -- already flow through CB_CHUNK_REPAIR + layout refresh on LAYOUTGET without disturbing other holders.

If a future revision determines that layout-level generation is needed, it can be added as a protocol extension: the on-wire change is additive rather than a replacement, because cg_gen_id's semantics are independent of any outer layout epoch.

Declustered RAID with Dynamic Parity Mapping An alternative raised at IETF 121 (November 2024) was borrowing from declustered RAID designs: the metadata server maintains, for every fixed-size region of each file, a mapping from logical address to the specific data servers that currently hold that region's data and parity shards; writes do not update chunks in place but instead produce a new parity stripe on a freshly allocated set of data servers, and the mapping is atomically swapped on the metadata server once the new stripe is durable. The attraction is that overwrite is replaced by remap, eliminating the write-hole problem entirely at the cost of moving consistency into the mapping table. This was rejected because:

The mapping load scales with the file's chunk count, not with the file count. A single large file with billions of chunks produces a billion-entry mapping that the metadata server must maintain with transactional semantics; the overhead is inverted from the usual "a few large files" regime that pNFS is designed for.
Remapping storms during rebalancing, data-server addition, or data-server failure require atomic updates to many mapping entries at once. Providing those updates with the reasonable-latency bounds required by HPC checkpoint workloads is an open research problem, not a specifiable protocol.
The approach reintroduces the metadata-server scale bottleneck that client-side erasure coding is designed to avoid: every write traverses the mapping table, and the mapping table is the hot-spot under concurrent writes.
The mapping table becomes the single point of failure that the rest of the flexible file v2 layout architecture works hard to avoid; replicating it with strong consistency requires a consensus protocol on the metadata server, which the current design deliberately does not require (see ).

The current design uses fixed per-file chunk placement decided at LAYOUTGET time plus chunk_guard4 CAS for writes, which localizes consistency decisions to the chunks being written rather than to a global mapping table.

Working Group Concern: Encoding on Every Client This appendix captures a working-group concern raised during the review of an earlier revision of this draft: the source of the concern, the question as the working group asked it, the authors' understanding of what was being asked, and how the current specification addresses it. This appendix is reviewer-aid material and is removed from the final RFC.

Source Christoph Hellwig, IETF 120, NFSv4 Working Group session, during the discussion of the original Flexible File Version 2 erasure-coding proposal.

The Question as Asked Christoph stated that he was "very scared of the implications of having every client be a full participant in a distributed storage system." He pointed out that any erasure-coding or replication protocol that runs at the client requires every client implementation to understand the encoding, and that encodings evolve over time as new algorithms appear in the storage research literature. He observed that the same problem appears with replication ("simple two-, three-, four-way replication"): a client power-failure event mid-write leaves the participating data servers in inconsistent states, and the recovery machinery (mirrored logs, write-ahead replay, partial-write detection) is "a bit of overkill for simple replication." David Black seconded the concern in the same session, stating that "it's better to have the data protection algorithm be inside the boundary of what you think the storage system is than outside."

What We Believe Is Being Asked Two coupled requirements:

Encoding correctness and encoding evolution must not be a per-client burden. An ecosystem in which every client must ship and update every supported encoding does not interoperate at scale: an organization cannot upgrade its storage system's encoding without coordinating an upgrade across every client.
The expensive recovery paths (partial writes, durable shard placement, mirrored logging) must not live at the client either. A protocol that exposes those paths to the client forces every client implementation to carry the failure-recovery machinery, which is precisely what RAID controllers and distributed storage systems put behind a service boundary so that hosts do not have to reason about it.

In short: the data-protection algorithm and its recovery story belong inside a storage boundary, not at the client.

How the Proxy Server Addresses This The proxy server role is the storage boundary that Christoph and David asked for. A proxy server is a peer of the metadata server and the data servers that:

speaks the encoding on behalf of clients that cannot;
receives whole-stripe operations from an encoding-ignorant client;
encodes (or decodes) using whatever the layout's demands;
drives the CHUNK operations to the participating data servers;
carries the partial-write / FINALIZE / COMMIT recovery machinery that the encoding requires.

Three properties follow:

A legacy NFSv4.2 (or even NFSv3) client gets erasure-coded durability without speaking erasure coding. The proxy server is where the encoding lives; the client does not have to be upgraded when the encoding is upgraded.
Encoding evolution is a server-side concern. Adding a new entry to requires updating the proxy servers and data servers, not every client in the deployment. This matches the operational pattern of every other distributed-storage protocol on the wire.
The recovery machinery (PENDING -> FINALIZED -> COMMITTED, the chunk-state machine, partial-write detection via ) executes on the proxy server, not the client. Clients see ordinary NFSv4.2 semantics; the proxy server is responsible for converting those semantics into the chunk state-machine the data servers implement.

An encoding-aware NFSv4.2 client is still permitted (and is the fast path: no proxy hop, no double bandwidth on the proxy's link). The proxy server is the answer for clients that either cannot speak the encoding or are too old to be upgraded. In Christoph's framing, the proxy server is the inside of the storage boundary; encoding-aware clients are implementations that have been admitted into that boundary by design. The proxy server does carry a data-plane cost: client bytes traverse the proxy on the way to the data servers, so the proxy's link sees roughly twice the bandwidth of a direct client-to-data server path, and the proxy server pays the encode/decode CPU. This is the price of admission for clients that do not speak the encoding; it is the same store-and-forward cost any storage gateway pays. It does not affect encoding-aware clients, which talk to the data servers directly.

Working Group Concern: Coherent Multi-data server Writes Without Recall Storms This appendix captures a working-group concern raised during the review of an earlier revision of this draft: the source of the concern, the question as the working group asked it, the authors' understanding of what was being asked, and how the current specification addresses it. This appendix is reviewer-aid material and is removed from the final RFC.

Source Christoph Hellwig, IETF 122, NFSv4 Working Group session, during the flexible file v2 layout erasure-coding discussion.

The Question as Asked Christoph observed that performing erasure coding across a set of data servers, where clients need a coherent view of the encoded data while writes are in flight, is "just really complicated, especially without recalling layouts." He continued: "maybe we need a more efficient network operation that doesn't recall layout but updates layouts in a different way, and that might reduce the overhead. Basically any scheme would require either a fair amount of intelligence on the data servers or some form of updating outstanding layouts to point to a new right-out-of-place location." He explicitly noted he was "leaning to updating the data servers to be smarter." The same conversation introduced the idea of a "generation counter that gets sent over the wire to the data servers, which means the data server now needs to look for a new location for the same existing layout."

What We Believe Is Being Asked Two coupled requirements:

The metadata server must be able to mutate where data lives -- replace a failing data server, redirect to a spare, rebalance, repair -- without serializing every layout-holding client through a CB_LAYOUTRECALL round-trip. A recall is global with respect to the layout: every client holding it must drain in-flight I/O and DELEGRETURN before the metadata server can mutate. In an erasure-coded workload with many concurrent clients, this turns a localized data server hiccup into a global stall.
The data servers must be smart enough to enforce per-client access on a finer grain than "the file is reachable from the network." Anonymous-stateid I/O combined with synthetic-uid fencing is a coarse instrument: fencing one client's access to a file affects every client's access to that file. The only way to selectively revoke is to teach the data server who is permitted, on which file, with which iomode -- which is the "smarter data server" Christoph was asking for.

How TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID Address This Sections , , and of this document define exactly the "smarter data server" the working group asked for. The mechanism:

At LAYOUTGET, the metadata server issues a real layout stateid and fans out TRUST_STATEID to each data server in the mirror set, registering (stateid.other, fh, clientid, iomode, expire) in a per-data server trust table. CHUNK_WRITE and CHUNK_READ on the data server now validate against the trust table; an unknown, expired, or revoked stateid yields NFS4ERR_BAD_STATEID.
When the metadata server needs to mutate the layout for a particular client -- because that client misbehaved, because a data server the layout points at is being drained, because the file is being repaired -- it issues REVOKE_STATEID to the affected data server. Other clients' trust entries on the same file are untouched.
When the metadata server needs to mutate at client-scope (lease expiry, client eviction), it issues BULK_REVOKE_STATEID, which removes every trust entry the named client has on the data server without affecting other clients.

The control-plane cost reshapes accordingly:

Layout mutation is no longer global. The metadata server reroutes data to a spare data server, rebuilds shards from surviving copies, and revokes only the trust entries that pointed at the failing location. The other clients holding the layout are not contacted.
The revoked client only learns of the mutation lazily, on its next CHUNK_WRITE or CHUNK_READ to the affected stripe. That operation returns NFS4ERR_BAD_STATEID; the client responds with LAYOUTERROR; the metadata server replies with a refreshed layout pointing at the new location; the client re-trusts and resumes. A client that never touches the affected stripe never pays the cost at all.
With warm spares known to the metadata server, the entire repair can complete before any client notices. The metadata server reconstructs onto a spare using server-to-server traffic, atomically swaps the layout slot in its in-memory state, and revokes only the trust entries on the now-evacuated data server. Reading clients see no interruption (any k of the surviving shards reconstructs); writing clients pay one round-trip to refresh the layout when they next write the affected stripe.

The combination of TRUST_STATEID and a warm-spare data server pool is the "more efficient network operation that updates layouts" Christoph asked for. It is not literally a layout update on the wire; it is a primitive that makes layout updates a local event the metadata server can resolve before the client has to pay a recall round-trip. The chunk state machine (PENDING -> FINALIZED -> COMMITTED) and address the orthogonal concern of partial-write recovery, ensuring that even when the metadata server reroutes mid-write the data servers can detect non-atomic stripes via per-chunk generation checks rather than via a global wall-clock or consensus protocol.

Combined Effect on the "Cluster Tax" The proxy server addresses the encoding-distribution cost; the trust stateid mechanism addresses the layout-mutation cost. Together, they confine the residual cluster overhead to:

the store-and-forward bandwidth on the proxy server link, paid only by clients that route through a proxy server rather than going data-server-direct; and
one LAYOUTERROR/LAYOUTGET round-trip per client per affected stripe, paid only by clients that actually try to use a stripe whose backing has changed.

Neither cost scales with the number of layout-holding clients, which is the property the working group asked for.

Acknowledgments The following from Hammerspace were instrumental in driving Flexible File Version 2 Layout Type: David Flynn, Trond Myklebust, Didier Feron, Jean-Pierre Monchanin, Pierre Evenou, and Brian Pawlowski. The Mojette Transform encoding type specification in -- including the algebra, the bin convention, the projection sizing, and the reconstruction algorithms -- was contributed by Pierre Evenou, drawing on the work of Nicolas Normand, Benoit Parrein, and the discrete geometry research group at the University of Nantes. Christoph Hellwig was instrumental in making sure the Flexible File Version 2 Layout Type was applicable to more than the Mojette Transformation. David Black clarified at IETF 124 that the consistency goal of flexible file v2 layout is RAID consistency across the shards of a stripe rather than POSIX write ordering across application writes; that framing is reflected in and in the Non-Goals of . The authors thank Dave Noveck, Chuck Lever, Tigran Mkrtchyan, Rick Macklem, Christoph Hellwig, and Sorin Faibish for their detailed review of earlier revisions of this draft. Their comments shaped the system model presentation, the chunk lifecycle and guard semantics, the trusted stateid design, and many smaller choices recorded throughout the document. Chris Inacio, Brian Pawlowski, Chuck Lever, Zahed Sarker, and Gorry Fairhurst guided this process.