Parallel NFS (pNFS) Flexible File Layout Version 2

6.2.1. Implementation Notes for Synthetic uids/gids

The selection method for the synthetic uids and gids to be used for fencing in loosely coupled storage devices is strictly an implementation issue. That is, an administrator might restrict a range of such ids available to the Lightweight Directory Access Protocol (LDAP) 'uid' field [RFC4519]. The administrator might also be able to choose an id that would never be used to grant access. Then, when the metadata server had a request to access a file, a SETATTR would be sent to the storage device to set the owner and group of the data file. The user and group might be selected in a round-robin fashion from the range of available ids.¶

Those ids would be sent back as ffv2ds_user and ffv2ds_group to the client, who would present them as the RPC credentials to the storage device. When the client is done accessing the file and the metadata server knows that no other client is accessing the file, it can reset the owner and group to restrict access to the data file.¶

When the metadata server wants to fence off a client, it changes the synthetic uid and/or gid to the restricted ids. Note that using a restricted id ensures that there is a change of owner and at least one id available that never gets allowed access.¶

Under an AUTH_SYS security model, synthetic uids and gids of 0 SHOULD be avoided. These typically either grant super access to files on a storage device or are mapped to an anonymous id. In the first case, even if the data file is fenced, the client might still be able to access the file. In the second case, multiple ids might be mapped to the anonymous ids.¶

6.2.2. Example of using Synthetic uids/gids

The user loghyr creates a file "ompha.c" on the metadata server, which then creates a corresponding data file on the storage device.¶

The metadata server entry may look like:¶

-rw-r--r--    1 loghyr  staff    1697 Dec  4 11:31 ompha.c

On the storage device, the file may be assigned some unpredictable synthetic uid/gid to deny access:¶

-rw-r-----    1 19452   28418    1697 Dec  4 11:31 data_ompha.c

When the file is opened on a client and accessed, the user will try to get a layout for the data file. Since the layout knows nothing about the user (and does not care), it does not matter whether the user loghyr or garbo opens the file. The client has to present an uid of 19452 to get write permission. If it presents any other value for the uid, then it must give a gid of 28418 to get read access.¶

Further, if the metadata server decides to fence the file, it SHOULD change the uid and/or gid such that these values neither match earlier values for that file nor match a predictable change based on an earlier fencing.¶

-rw-r-----    1 19453   28419    1697 Dec  4 11:31 data_ompha.c

The set of synthetic gids on the storage device SHOULD be selected such that there is no mapping in any of the name services used by the storage device, i.e., each group SHOULD have no members.¶

If the layout segment has an iomode of LAYOUTIOMODE4_READ, then the metadata server SHOULD return a synthetic uid that is not set on the storage device. Only the synthetic gid would be valid.¶

The client is thus solely responsible for enforcing file permissions in a loosely coupled model. To allow loghyr write access, it will send an RPC to the storage device with a credential of 1066:1067. To allow garbo read access, it will send an RPC to the storage device with a credential of 1067:1067. The value of the uid does not matter as long as it is not the synthetic uid granted when getting the layout.¶

While pushing the enforcement of permission checking onto the client may seem to weaken security, the client may already be responsible for enforcing permissions before modifications are sent to a server. With cached writes, the client is always responsible for tracking who is modifying a file and making sure to not coalesce requests from multiple users into one request.¶

6.3.1. Loosely Coupled Locking Model

When locking-related operations are requested, they are primarily dealt with by the metadata server, which generates the appropriate stateids. When an NFSv4 version is used as the data access protocol, the metadata server may make stateid-related requests of the storage devices. However, it is not required to do so, and the resulting stateids are known only to the metadata server and the storage device.¶

Given this basic structure, locking-related operations are handled as follows:¶

• OPENs are dealt with by the metadata server. Stateids are selected by the metadata server and associated with the client ID describing the client's connection to the metadata server. The metadata server may need to interact with the storage device to locate the file to be opened, but no locking-related functionality need be used on the storage device.¶

• OPEN_DOWNGRADE and CLOSE only require local execution on the metadata server.¶

• Advisory byte range locks can be implemented locally on the metadata server. As in the case of OPENs, the stateids associated with byte range locks are assigned by the metadata server and only used on the metadata server.¶

• Delegations are assigned by the metadata server that initiates recalls when conflicting OPENs are processed. No storage device involvement is required.¶

• TEST_STATEID and FREE_STATEID are processed locally on the metadata server, without storage device involvement.¶

All I/O operations to the storage device are done using the anonymous stateid. Thus, the storage device has no information about the openowner and lockowner responsible for issuing a particular I/O operation. As a result:¶

• Mandatory byte range locking cannot be supported because the storage device has no way of distinguishing I/O done on behalf of the lock owner from those done by others.¶

• Enforcement of share reservations is the responsibility of the client. Even though I/O is done using the anonymous stateid, the client MUST ensure that it has a valid stateid associated with the openowner.¶

In the event that a stateid is revoked, the metadata server is responsible for preventing client access, since it has no way of being sure that the client is aware that the stateid in question has been revoked.¶

As the client never receives a stateid generated by a storage device, there is no client lease on the storage device and no prospect of lease expiration, even when access is via NFSv4 protocols. Clients will have leases on the metadata server. In dealing with lease expiration, the metadata server may need to use fencing to prevent revoked stateids from being relied upon by a client unaware of the fact that they have been revoked.¶

6.3.2. Tightly Coupled Locking Model

When locking-related operations are requested, they are primarily dealt with by the metadata server, which generates the appropriate stateids. These stateids MUST be made known to the storage device using control protocol facilities. This document defines one such control protocol -- the TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID operations in Section 6.4 -- for deployments in which the storage devices are NFSv4.2 servers willing to implement the new operations. A storage device with its own established back-end control protocol that provides the equivalent functional capabilities is conformant under this specification without implementing the TRUST_STATEID family; see Section 6.4 for the conformance framing.¶

When using the TRUST_STATEID control protocol defined in Section 6.4, the metadata server and a storage device establish that they can use it via a two-part handshake, both parts of which MUST succeed before the metadata server may issue TRUST_STATEID against that storage device for production traffic:¶

Capability probe:

At control-session setup the metadata server sends a TRUST_STATEID against the anonymous stateid (see Section 6.4.1). A storage device that supports tight coupling MUST reject the probe with NFS4ERR_INVAL; a storage device that does not support tight coupling returns NFS4ERR_NOTSUPP and the metadata server falls back to loose coupling. The metadata server records result per storage device by setting the FFV2_COUPLING_TRUSTED_STATEID bit in ffv2dv_coupling on success (leaving the bit clear on NFS4ERR_NOTSUPP).¶

Control-session gating:

The metadata server presents EXCHGID4_FLAG_USE_PNFS_MDS at EXCHANGE_ID when it opens the control session to the storage device (see Section 6.4.2). The storage device MUST reject any incoming TRUST_STATEID, REVOKE_STATEID, or BULK_REVOKE_STATEID that does not arrive on such a session with NFS4ERR_PERM. This is the authorization mechanism that distinguishes the metadata server from ordinary pNFS clients, which connect with EXCHGID4_FLAG_USE_PNFS_DS or EXCHGID4_FLAG_USE_NON_PNFS and are therefore structurally unable to invoke these operations.¶

Given this basic structure, locking-related operations are handled as follows:¶

• OPENs are dealt with primarily on the metadata server. Stateids are selected by the metadata server and associated with the client ID describing the client's connection to the metadata server. The metadata server needs to interact with the storage device to locate the file to be opened and to make the storage device aware of the association between the metadata-server-chosen stateid and the client and openowner that it represents. OPEN_DOWNGRADE and CLOSE are executed initially on the metadata server, but the state change MUST be propagated to the storage device.¶

• Advisory byte range locks can be implemented locally on the metadata server. As in the case of OPENs, the stateids associated with byte range locks are assigned by the metadata server and are available for use on the metadata server. Because I/O operations are allowed to present lock stateids, the metadata server needs the ability to make the storage device aware of the association between the metadata-server-chosen stateid and the corresponding open stateid it is associated with.¶

• Mandatory byte range locks can be supported when both the metadata server and the storage devices have the appropriate support. As in the case of advisory byte range locks, these are assigned by the metadata server and are available for use on the metadata server. To enable mandatory lock enforcement on the storage device, the metadata server needs the ability to make the storage device aware of the association between the metadata-server-chosen stateid and the client, openowner, and lock (i.e., lockowner, byte range, and lock-type) that it represents. Because I/O operations are allowed to present lock stateids, this information needs to be propagated to all storage devices to which I/O might be directed rather than only to storage device that contain the locked region.¶

• Delegations are assigned by the metadata server that initiates recalls when conflicting OPENs are processed. Because I/O operations are allowed to present delegation stateids, the metadata server requires the ability:¶

  1. to make the storage device aware of the association between the metadata-server-chosen stateid and the filehandle and delegation type it represents¶

  2. to break such an association.¶

• TEST_STATEID is processed locally on the metadata server, without storage device involvement.¶

• FREE_STATEID is processed on the metadata server, but the metadata server requires the ability to propagate the request to the corresponding storage devices.¶

Scope of this document. The wire-level control-protocol operations this document defines -- TRUST_STATEID, REVOKE_STATEID, and BULK_REVOKE_STATEID (Section 6.4) -- carry only the association between a layout stateid, the ffv2m_client_id the metadata server assigned to the client, and (for TRUST_STATEID) the iomode, expiry, and principal. They do NOT carry openowner, lockowner, byte range, lock-type, the identity of an associated open stateid, or delegation-type.¶

The bullets above (OPEN state, advisory byte range lock state, mandatory byte range lock state, and delegation state) enumerate the associations the storage device would need in order to enforce POSIX-conformant OPEN, byte range locking, and delegation semantics against per-client identity rather than against the loose coupling synthetic uid/gid. These are inherited from the general tight coupling locking model in Section 2.3 of [RFC8435]. Trusted-stateid tight coupling as defined by this document satisfies these associations only for the layout stateid; a deployment that requires mandatory byte range locking, delegation recall, or the finer-grained open/lock stateid associations MUST use a back-end control protocol between the metadata server and the storage device that carries this state. Such a back-end control protocol is out of scope for this document.¶

A deployment that does not need those finer-grained associations -- for example, a flexible file v2 layout deployment whose per-file access-control decisions live entirely on the metadata server and whose storage devices see only chunk-level CAS on layout stateids -- is conformant with the trusted stateid tight coupling model using only the TRUST_STATEID family. That scope covers the layout stateid and, transitively via ffv2m_client_id, the writer identity carried on CHUNK operations.¶

Because the client will possess and use stateids valid on the storage device, there will be a client lease on the storage device, and the possibility of lease expiration does exist. The best approach for the storage device is to retain these locks as a courtesy. However, if it does not do so, control protocol facilities need to provide the means to synchronize lock state between the metadata server and storage device.¶

Clients will also have leases on the metadata server that are subject to expiration. In dealing with lease expiration, the metadata server would be expected to use control protocol facilities enabling it to invalidate revoked stateids on the storage device. In the event the client is not responsive, the metadata server may need to use fencing to prevent revoked stateids from being acted upon by the storage device.¶

6.4.1. Capability Discovery

A storage device indicates support for trusted stateid tight coupling implicitly, by processing TRUST_STATEID rather than returning NFS4ERR_NOTSUPP. (A storage device that supports a non-TRUST_STATEID form of tight coupling but not the trusted stateid variant defined here will return NFS4ERR_NOTSUPP on this probe; from this specification's perspective it is treated the same as a storage device that does not support tight coupling at all.) The metadata server probes each storage device during control-session setup:¶

SEQUENCE + PUTROOTFH + TRUST_STATEID(
    tsa_layout_stateid = ANONYMOUS_STATEID,
    tsa_client_id      = 0,
    tsa_iomode         = LAYOUTIOMODE4_READ,
    tsa_expire         = 0,
    tsa_principal      = "")

The anonymous stateid is used deliberately: a correctly implemented storage device MUST reject it (see Section 25.13), so the probe cannot accidentally register garbage in the trust table. The metadata server interprets the probe response as follows:¶

NFS4ERR_NOTSUPP:

trusted stateid tight coupling is not supported on this storage device. The metadata server leaves the FFV2_COUPLING_TRUSTED_STATEID bit clear in ffv2dv_coupling for this storage device. If ffv2dv_coupling has no other tight coupling bits set for this storage device, the metadata server falls back to the synthetic-uid model (anonymous stateid plus fencing).¶

NFS4ERR_INVAL:

trusted stateid tight coupling is supported. The anonymous stateid was correctly rejected. The metadata server sets the FFV2_COUPLING_TRUSTED_STATEID bit in ffv2dv_coupling for this storage device.¶

NFS4_OK:

the storage device accepted an anonymous stateid into its trust table. This is a storage device bug. The metadata server MAY treat the capability as confirmed to avoid downgrading to loose coupling, but it MUST immediately issue REVOKE_STATEID to remove the bogus entry.¶

The capability is recorded per storage device, not per file. Partial support across a mirror set is permitted: each ffv2_device_versions4 entry returned by GETDEVICEINFO carries its own ffv2dv_coupling value, set independently.¶

6.4.2. Control Session

The metadata server establishes an NFSv4.2 session to each tight coupling capable storage device at startup. On this session the metadata server acts as the storage device's client and presents EXCHGID4_FLAG_USE_PNFS_MDS in its EXCHANGE_ID args.¶

That is a repurposing of the flag, and implementers should understand it as one. Section 13.1 of [RFC8881] determines a server's pNFS role from the eir_flags it returns, not from what the client asks for, and treats the argument flags as role requests; read literally, EXCHGID4_FLAG_USE_PNFS_MDS in the args asks the peer to act as this client's metadata server, which is not what a metadata server wants from a storage device. The flag is used here precisely because it carries no other meaning on this path. Section 13.1 of [RFC8881] does name a flag for a client connecting to a data server, EXCHGID4_FLAG_USE_PNFS_DS, but a pNFS client presents that one too, so it cannot separate a metadata server from a client. EXCHGID4_FLAG_USE_PNFS_MDS is the one role request a client has no reason to make of a storage device, and that is what leaves it free to mark the control session.¶

Presenting it does not conflict with [RFC8881]. Section 13.1 of [RFC8881] permits a client to request zero or more of the three role flags even where the combination requested is contradictory, and constrains only the combinations a server may return in its eir_flags. A storage device that also serves a metadata server role, and that selects among its roles from the requested ones, should account for this document's use of the flag when composing its eir_flags.¶

The storage device MUST verify that any incoming TRUST_STATEID, REVOKE_STATEID, or BULK_REVOKE_STATEID compound arrives on a session whose owning client presented EXCHGID4_FLAG_USE_PNFS_MDS in its EXCHANGE_ID args. Requests that arrive on any other session MUST be rejected with NFS4ERR_PERM. This is the sole access control on these operations; a pNFS client connecting to the storage device does not present EXCHGID4_FLAG_USE_PNFS_MDS and therefore cannot invoke them.¶

The EXCHGID4_FLAG_USE_PNFS_MDS check replaces any path- or filehandle-level gating. TRUST_STATEID operates on a filehandle that may be any file on the storage device, and the metadata server is the sole authority that can legitimately speak this protocol.¶

Because the EXCHGID4_FLAG_USE_PNFS_MDS check relies on the owning client's self-declaration at EXCHANGE_ID time, the storage device cannot by itself distinguish a legitimate metadata server from any other host that sets the flag. The wire protocol provides no primitive that binds a request to a particular pNFS role; the flag is a hint whose operational meaning is only as strong as the deployment's authentication or isolation choice.¶

The deployment requirement is that only entities the deployment considers legitimate metadata servers can (a) establish a control session with EXCHGID4_FLAG_USE_PNFS_MDS against the storage device and (b) invoke TRUST_STATEID, REVOKE_STATEID, or BULK_REVOKE_STATEID on it. Deployments have historically satisfied this requirement using one or more of the following mechanisms; the wire protocol does not prescribe which is chosen:¶

• RPCSEC_GSS with a machine principal that the storage device has been configured to accept as a metadata server. The storage device validates the principal against a local policy list before accepting the flag.¶

• TLS ([RFC9289]) with a client certificate that the storage device has been configured to accept as a metadata server. The storage device validates the certificate against a local policy list before accepting the flag.¶

• Network-path isolation: the control-session path runs on a network segment (dedicated management VLAN, private link, or similar) that pNFS clients cannot reach, so only configured metadata servers can open a session at all.¶

• Operating-system-level filesystem access control on the storage device: the underlying export that backs the pNFS data files is configured so that only the metadata server's OS identity (typically root, or a dedicated privileged uid) can access it. A pNFS client that reaches the storage device but presents any other uid receives permission errors before it can invoke any operation, control-plane or otherwise.¶

A single host MAY legitimately act in multiple pNFS roles against the same storage device -- for example, an entity that is a metadata server for one export and a plain pNFS client for another. The wire protocol does not distinguish these roles at the operation level; the deployment is responsible for arranging that the storage device can tell them apart, typically by using distinct credentials or distinct sessions for the two roles (each session presents EXCHGID4_FLAG_USE_PNFS_MDS according to the role the entity is acting in on that session).¶

The security consequences of these choices -- what an unauthenticated attacker can invoke if none of the mechanisms above is deployed, and what an authenticated caller can still achieve by misrepresenting role -- are discussed in Section 28.7.¶

6.4.3. Flow at LAYOUTGET

For each new or refreshed layout segment, the metadata server:¶

1. chooses the layout stateid (as it would without tight coupling);¶

2. identifies the trusted stateid capable storage devices in the mirror set (those for which ffv2dv_coupling has the FFV2_COUPLING_TRUSTED_STATEID flag set);¶

3. fans out TRUST_STATEID to each such storage device, specifying the layout stateid, the layout iomode, a tsa_expire derived from the metadata server's lease (see Section 6.4.6), and the client's authenticated identity in tsa_principal;¶

4. waits for all fan-outs to complete (or reach their per-storage-device timeout) before returning the layout.¶

If every storage device in the mirror set rejects the TRUST_STATEID fan-out, the metadata server MUST NOT return the layout; instead it returns NFS4ERR_LAYOUTTRYLATER. If some storage devices accept and others reject, the metadata server MAY return a layout covering only the accepting storage devices, provided the accepting subset still meets the minimum servable coverage for the file's encoding: at least one replica for FFV2_ENCODING_PASSTHROUGH or FFV2_ENCODING_REPLICATED, or at least k of the k + m storage devices for an erasure-coded encoding at (k, m) parameters. If it does not, the metadata server MUST NOT return a partial layout and instead returns NFS4ERR_LAYOUTTRYLATER as in the all-reject case. A storage device that returns NFS4ERR_DELAY is retried until either success or the metadata server's LAYOUTGET-response budget is exhausted. If a storage device returns NFS4ERR_NOTSUPP at this time (having accepted the probe earlier), the metadata server MUST clear the FFV2_COUPLING_TRUSTED_STATEID flag in ffv2dv_coupling for this storage device. If no tight coupling flags remain set for this device, the metadata server falls back to the synthetic-uid model and re-issues the layout accordingly.¶

6.4.4. Principal Binding and the Kerberos Gap

The flexible file v1 layout has a known gap: a client authenticated to the metadata server with Kerberos has no way to present the same authenticated identity to the storage device, because flexible file v1 layouts carry only ffds_user / ffds_group (POSIX uid/gid for AUTH_SYS). A strict Kerberos deployment on the flexible file v1 layout must either allow AUTH_SYS from the metadata server's subnet or accept that the flexible file v1 layout's data path is not Kerberos-protected.¶

The tsa_principal field in TRUST_STATEID closes that gap. When a client authenticates to the metadata server as a Kerberos principal (e.g., alice@REALM), the metadata server passes that principal name to each storage device in tsa_principal. The storage device then enforces a two-part check on each CHUNK operation that presents the layout stateid:¶

a. the stateid is in the trust table and has not expired; and¶

b. the caller's authenticated identity (the RPCSEC_GSS display name on the CHUNK compound) matches tsa_principal.¶

Both conditions MUST hold. On principal mismatch the storage device MUST return NFS4ERR_ACCESS -- the semantics are "you do not have an authorized layout for this file", which matches the existing fencing error and avoids the confusion of NFS4ERR_WRONGSEC (which directs the client to re-authenticate with a different flavor) or NFS4ERR_BAD_STATEID (which directs the client to return the layout).¶

The metadata server MUST populate tsa_principal with the RPCSEC_GSS display name of the authenticated client when the client authenticated to the metadata server via RPCSEC_GSS. The metadata server MUST set tsa_principal to the empty string only for AUTH_SYS and TLS clients (for which there is no server-verified per-user identity). Setting tsa_principal to the empty string for an RPCSEC_GSS client disables the principal check on the storage device and silently re-opens the flexible file v1 layout Kerberos gap; it is a metadata server bug, not a protocol option.¶

If tsa_principal is the empty string, no principal check applies. This is the expected setting for AUTH_SYS and TLS clients:¶

• AUTH_SYS clients have no server-verified identity. The storage device's stateid check and the AUTH_SYS uid/gid on the data file together constitute the authorization. In a tightly coupled deployment the data file's owner/group need not match the metadata file's, since ffv2ds_user and ffv2ds_group are ignored (see Section 8.9).¶

• TLS clients have transport-layer authentication via mutual TLS ([RFC9289]). The TLS layer authenticates the client machine; the stateid check confirms the metadata server authorized that machine to access this file. The machine-level authentication is handled beneath the RPC layer and is not reflected in tsa_principal. Opportunistic TLS (STARTTLS without certificate verification) provides encryption but not authentication, and therefore has the same authorization properties as plain AUTH_SYS.¶

When a client's I/O is routed through a proxy server -- that is, the layout the metadata server returns to the client has FFV2_DS_FLAGS_PROXY set on the proxy's ffv2_data_server4 entry -- the storage device observes CHUNK operations arriving from the proxy server's address rather than from the client directly. The tsa_principal the metadata server populates in TRUST_STATEID is the principal the storage device will observe on those CHUNK operations, and [I-D.haynes-nfsv4-flexfiles-v2-proxy-server]'s credential-forwarding rules (in particular rule 1, "Credential pass-through") require the proxy server to forward the client's credentials verbatim on every CHUNK operation it issues on the client's behalf. Therefore:¶

• For an RPCSEC_GSS client whose I/O is proxied through a proxy server, the metadata server MUST set tsa_principal to the client's RPCSEC_GSS display name (identical to the non-proxied case). The storage device's principal check on CHUNK operations will match against the client's principal on the forwarded compound, not the proxy server's service identity.¶

• For an AUTH_SYS client whose I/O is proxied through a proxy server, the metadata server MUST set tsa_principal to the empty string (identical to the non-proxied case). The proxy server forwards the client's AUTH_SYS uid/gid; the storage device's stateid check plus the forwarded AUTH_SYS uid/gid constitute the authorization.¶

The metadata server MUST NOT set tsa_principal to the proxy server's own service principal. Doing so would require the proxy server to authenticate to the storage device as itself (bypassing credential forwarding) which is explicitly prohibited by rule 4 of [I-D.haynes-nfsv4-flexfiles-v2-proxy-server] ("proxy server service identity is for the control plane only").¶

6.4.5. Client-Detected Trust Gap

A window exists between a successful TRUST_STATEID fan-out and the client's first I/O to the storage device. A transient failure may cause the storage device to forget or reject the entry before the client's first CHUNK_WRITE arrives. The client cannot distinguish this case from legitimate revocation; both surface as NFS4ERR_BAD_STATEID on the storage device.¶

The recovery path:¶

1. The client sends LAYOUTERROR(layout_stateid, device_id, NFS4ERR_BAD_STATEID) to the metadata server.¶

2. The metadata server retries TRUST_STATEID against the reporting storage device. If the retry succeeds, the metadata server returns NFS4_OK for LAYOUTERROR. The client retries the original I/O.¶

3. If the retry fails -- the storage device is unreachable or returns a hard error -- the metadata server issues CB_LAYOUTRECALL for that device and the client returns the layout segment covering that storage device. The client is expected to re-request via LAYOUTGET.¶

This is the same LAYOUTERROR path used for NFS4ERR_ACCESS or NFS4ERR_PERM in the fencing model (see Section 6.2), with the metadata server's action being "retry TRUST_STATEID" instead of "rotate uid/gid".¶

6.4.6. Lease and Renewal

tsa_expire in a TRUST_STATEID request is a wall-clock expiry instant expressed as an nfstime4. The metadata server MUST set tsa_expire to the current wall-clock time plus the metadata server's client lease period.¶

Clock-synchronization assumption: the metadata server and each storage device MUST maintain wall-clock synchronization within one lease period, e.g., via NTP [RFC5905] or an equivalent mechanism. Under this assumption, a tsa_expire computed by the metadata server and evaluated by the storage device is interpreted consistently within the storage device's local clock. Deployments unable to guarantee sub-lease-period clock synchronization MUST either (a) shorten the effective TRUST_STATEID lease so it exceeds the worst-case skew by at least 2x, or (b) route I/O through the metadata server as the fallback path (no tight coupling control session, no TRUST_STATEID) so lease enforcement stays on the metadata server's clock alone. A storage device that detects sustained clock divergence from the metadata server (e.g., via periodic wall-clock exchange as part of its tight coupling control-session heartbeats) SHOULD log the divergence and MAY refuse further TRUST_STATEID entries with NFS4ERR_DELAY until the divergence is corrected.¶

The metadata server MUST re-issue TRUST_STATEID for an entry before tsa_expire while the corresponding layout is outstanding. The RECOMMENDED trigger is: when an entry is within half the lease period of its tsa_expire, re-issue TRUST_STATEID with a refreshed tsa_expire. Renewing on every SEQUENCE that keeps the layout stateid alive is correct but produces metadata-server-to-storage-device traffic proportional to the client's SEQUENCE rate, which is undesirable in steady state.¶

If an entry expires on the storage device before the metadata server renews it -- for example, because the metadata server is partitioned from the storage device for longer than the lease period -- the storage device MUST return NFS4ERR_BAD_STATEID to the client on the next CHUNK operation. The client returns the layout to the metadata server and re-requests. This is the same recovery path as the trust gap described above.¶

6.4.7. Storage Device Crash Recovery

A storage device MAY persist its trust table across restarts. An implementation that does so MUST also persist its server-instance identity, returning the same eir_server_owner.so_minor_id on EXCHANGE_ID after the restart (per Section 18.35 of [RFC8881]), so that clients and the metadata server observe the device as continuously available and the persisted trust entries remain valid against the layout stateids that were issued before the restart.¶

A storage device that does NOT persist its trust table empties the table on restart and MUST present a new server instance (incremented so_minor_id) so that clients detect the restart. The remainder of this section describes the recovery path for the volatile case.¶

The client detects a volatile storage device restart via NFS4ERR_BADSESSION or NFS4ERR_STALE_CLIENTID on its data server session. The client returns the affected layout segment to the metadata server via LAYOUTRETURN and re-requests via LAYOUTGET. The metadata server then fans out fresh TRUST_STATEID operations to the recovered storage device.¶

Planned storage device restarts (software upgrade, etc.) SHOULD drain in-flight CHUNK operations before shutting down.¶

6.4.8. Metadata Server Crash Recovery

A metadata server MAY persist all its trust-management state across restarts. An implementation that does so MUST also persist its server-instance identity, returning the same eir_server_owner.so_minor_id on EXCHANGE_ID after the restart (per Section 18.35 of [RFC8881]), so that storage devices observe the metadata server as continuously available and accept incoming TRUST_STATEID and REVOKE_STATEID operations against the existing trust entries without revalidation. No grace period is required.¶

A metadata server that presents a new server instance (incremented so_minor_id) on restart follows the recovery path in the remainder of this section.¶

When the metadata server restarts as a new instance, its control sessions to the storage devices are lost. Trust entries remain on the storage devices until tsa_expire, but the metadata server is no longer renewing them; the entries are effectively orphaned until the metadata server completes grace.¶

When the metadata server reconnects to a storage device with a new boot epoch -- that is, the EXCHANGE_ID returns a new server owner on the storage device's view of the metadata server -- the storage device SHOULD mark all trust entries established under the prior metadata-server epoch as pending-revalidation. While an entry is pending-revalidation:¶

• I/O that presents the entry's stateid MUST receive NFS4ERR_DELAY, not NFS4ERR_BAD_STATEID. NFS4ERR_DELAY tells the client to retry with the same stateid -- the metadata server is recovering and may yet revalidate the entry. NFS4ERR_BAD_STATEID would instead cause the client to return the layout immediately, producing a thundering herd against the metadata server during grace.¶

• An entry remains pending-revalidation until the metadata server either re-issues TRUST_STATEID for it (which transitions it back to trusted) or until the entry's tsa_expire elapses (which removes it).¶

The metadata server's recovery sequence is:¶

1. Reconnect to each storage device and establish a fresh control session.¶

2. Optionally issue BULK_REVOKE_STATEID with an all-zeros clientid to each storage device. This clears the prior trust table eagerly; skipping this step is correct, because orphan entries expire via tsa_expire.¶

3. Enter grace and accept RECLAIM operations from clients. For each reclaimed layout, fan out TRUST_STATEID to the relevant storage devices.¶

4. Exit grace. Clients that did not reclaim in time have their state revoked; the metadata server issues REVOKE_STATEID or BULK_REVOKE_STATEID on their behalf.¶

Metadata servers SHOULD persist the set of outstanding TRUST_STATEID entries (clientid, layout stateid, storage device address, tsa_expire) to stable storage. With this persistence the metadata server can re-issue TRUST_STATEID for all known entries immediately upon reconnecting to each storage device, before clients begin reclaiming. This shrinks the window during which the storage device returns NFS4ERR_DELAY for client I/O. Persistence is a latency optimization, not a correctness requirement: the re-layout path handles recovery in all cases.¶

6.4.9. Backward Compatibility

• NFSv3 storage devices are unchanged. They are always treated as loosely coupled; TRUST_STATEID does not exist on NFSv3 servers.¶

• NFSv4.2 storage devices for which the TRUST_STATEID probe returns NFS4ERR_NOTSUPP are treated as loosely coupled; fencing is the only revocation mechanism, the same as for NFSv3.¶

• NFSv4.2 storage devices for which the probe returns NFS4ERR_INVAL support tight coupling; the metadata server uses TRUST_STATEID at LAYOUTGET and REVOKE_STATEID or BULK_REVOKE_STATEID for revocation instead of fencing.¶

A single deployment MAY contain a mix of tight-coupled and loose-coupled storage devices; each is negotiated independently via the probe.¶

8.1.1. Heterogeneous Mirror Sets

A single flexible file v2 layout's ffv2l_mirrors array MAY carry mirror entries of different encoding types. The protocol does not require the entries to agree -- one mirror can be FFV2_ENCODING_PASSTHROUGH, another can be FFV2_ENCODING_RS_VANDERMONDE, both describing the same file's data range. This combination is the structural primitive for three operations that motivate keeping PASSTHROUGH in the layout type's vocabulary:¶

Assimilate:

A file that exists today as a plain copy on storage outside flexible file v2 layout -- no chunk envelope, no per-chunk CRC -- enters the namespace as a PASSTHROUGH mirror against the source bytes as they are. The metadata server then adds one or more encoded mirrors (MIRRORED, RS, Mojette) to the same layout and synchronizes them from the PASSTHROUGH source. Clients can read the file via the PASSTHROUGH mirror immediately; the encoded mirrors become available as they are populated. No "rewrite before serve" step is required.¶

Copy / migrate between encodings:

Changing a file from one encoding to another is "add a mirror in the target encoding to the same layout, let it sync from any healthy source mirror, retire the source mirror." PASSTHROUGH is the special case where one endpoint of that migration is "no encoding."¶

Repair across encodings:

When an encoded mirror has a chunk whose CRC fails and whose parity cannot reconstruct, a PASSTHROUGH mirror in the same layout is an authoritative source: CHUNK_READ a peer encoded mirror or read the PASSTHROUGH byte range, then CHUNK_WRITE the repaired chunk. The reverse is also true: a byte range on the PASSTHROUGH copy whose contents the metadata server suspects has drifted can be repaired by reconstructing from the verified-CRC chunks of an encoded peer. Two encodings of the same file are two independent recovery paths.¶

The metadata server is responsible for keeping the entries in a heterogeneous mirror set in sync; the protocol does not require client awareness of which encoding produced which mirror beyond what the layout already states.¶

The wire-level coordination that makes a heterogeneous mirror set safe to operate -- in particular, the rule that a client arriving during a transition sees a single layout naming the proxy server rather than two layouts naming the source and destination encodings, and the rule that the metadata server's commit of a transition is a single transaction -- is specified in the proxy server draft ([I-D.haynes-nfsv4-flexfiles-v2-proxy-server]), in the sections "Layout Shape During a Proxy Operation" and "Atomic commit on PROXY_DONE". This document specifies the per-mirror encoding naming primitive; the proxy server document specifies the transactional machinery that uses it.¶

The full description of each encoding type is deferred to its own section. The mapping between the enum values above and those sections is:¶

8.1.2. Encoding Type Interoperability

The data servers do not interpret erasure-coded data -- they store and return opaque chunks. The NFS wire protocol likewise does not depend on the encoding mathematics. However, a client that writes data using one encoding type MUST be able to read it back, and a different client implementation MUST be able to read data written by the first client if both claim to support the same encoding type.¶

This interoperability requirement means that each registered encoding type MUST fully specify the encoding and decoding mathematics such that two independent implementations produce byte-identical encoded output for the same input. The specification of a new encoding type MUST include one of the following:¶

1. A complete mathematical specification of the encoding and decoding algorithms, including all parameters (e.g., field polynomial, matrix construction, element size) sufficient for an independent implementation to produce interoperable results.¶

2. A reference to a published patent or pending patent application that contains the algorithm specification. Implementors can then evaluate the licensing terms and decide whether to support the encoding type.¶

3. A declaration that the encoding type is a proprietary implementation. In this case, the encoding type name SHOULD include an organizational prefix (e.g., FFV2_ENCODING_ACME_FOOBAR) to signal that interoperability is limited to implementations licensed by that organization.¶

Option 1 is RECOMMENDED for encoding types intended for broad interoperability. Options 2 and 3 allow vendors to register encoding types for use within their own ecosystems while preserving the encoding type namespace.¶

The rationale for this requirement is that erasure coding moves computation from the server to the client. If the client cannot determine how data was encoded, it cannot decode it. Unlike layout types (where the server controls the storage format), encoding types require client-side agreement on the mathematics.¶

8.2.1. ffv2_flags4

   /// const FFV2_FLAGS_NO_LAYOUTCOMMIT  = FF_FLAGS_NO_LAYOUTCOMMIT;
   /// const FFV2_FLAGS_NO_IO_THRU_MDS   = FF_FLAGS_NO_IO_THRU_MDS;
   /// const FFV2_FLAGS_NO_READ_IO       = FF_FLAGS_NO_READ_IO;
   /// const FFV2_FLAGS_WRITE_ONE_MIRROR =
   ///     FF_FLAGS_WRITE_ONE_MIRROR;
   /// const FFV2_FLAGS_ONLY_ONE_WRITER  = 0x00000010;
   ///
   /// typedef uint32_t            ffv2_flags4;

The ffv2_flags4 in Figure 10 is a bitmap that allows the metadata server to inform the client of particular conditions that may result from more or less tight coupling of the storage devices.¶

Each flag below describes both the semantics when set and the normative requirement it places on the client. When a flag is not set, the client MUST follow the default behavior described for its unset state.¶

FFV2_FLAGS_NO_LAYOUTCOMMIT:

When set, the client MAY omit the LAYOUTCOMMIT to the metadata server. When unset, the client MUST send LAYOUTCOMMIT per Section 18.42. of [RFC8881]¶

FFV2_FLAGS_NO_IO_THRU_MDS:

When set, the client MUST NOT proxy I/O operations through the metadata server, even after detecting a network disconnect to a storage device. When unset, the client MAY retry failed I/O via the metadata server.¶

FFV2_FLAGS_NO_READ_IO:

When set, the client MUST NOT issue READ against layouts of iomode LAYOUTIOMODE4_RW, and MUST instead request a separate layout of iomode LAYOUTIOMODE4_READ for any read I/O. When unset, the client MAY issue READ against either iomode.¶

FFV2_FLAGS_WRITE_ONE_MIRROR:

When set, the client MAY update only one mirror of each layout segment (see Section 11.1) and rely on the metadata server (or a proxy server acting on its behalf) or a peer data server to propagate the update to the remaining mirrors. When unset, the client MUST update all mirrors.¶

The metadata server MUST NOT set FFV2_FLAGS_WRITE_ONE_MIRROR on a layout whose ffv2l_mirrors carry more than one distinct ffv2m_encoding_type_data value unless a propagation actor is available that speaks every encoding present in the layout. Cross-encoding propagation requires the actor to decode through the source mirror's encoding transform and re-encode for each target mirror's transform, which the metadata server itself cannot do for chunked encodings (it does not hold the encoded shards); a proxy server is the entity that performs cross-encoding translation. On a mixed-encoding layout without a proxy server for the affected file, the metadata server MUST leave FFV2_FLAGS_WRITE_ONE_MIRROR unset and require the client to update all mirrors directly.¶

FFV2_FLAGS_ONLY_ONE_WRITER:

When set, the client is the exclusive writer for the layout and MAY issue CHUNK_WRITE without setting cwa_guard, retaining the ability to use CHUNK_ROLLBACK in the event of a write hole caused by overwriting. When unset, the client MUST set cwa_guard on every CHUNK_WRITE so that chunk_guard4 CAS can prevent collisions across concurrent writers.¶

8.11.1. Encoding Negotiation

Because the encoding type registry is expected to grow over time (new erasure encoding types are added, older ones fall out of favor, vendors register private codes; see Section 29), neither clients nor metadata servers are required to implement every registered encoding. The protocol negotiates encoding capabilities via ffv2_layouthint4:¶

Client-side advertisement:

A client that wishes to influence encoding selection SHOULD send the set of encodings it actually implements in ffv2lh_supported_types. A client MUST NOT claim support for an encoding it cannot encode or decode: a false advertisement produces silent data unavailability when the resulting layout is issued.¶

Metadata-server selection at file creation:

When the LAYOUTGET is against a newly-created file (the file has no committed data yet), the metadata server has real discretion. It SHOULD select an encoding from the client's ffv2lh_supported_types list when the server's policy permits. The server MAY override the hint when its policy dictates a specific encoding (for example, per-export objectives); in that case the server issues a layout with the policy-dictated encoding and the client MUST either honor it or fail its I/O with NFS4ERR_ENCODING_NOT_SUPPORTED.¶

Metadata-server selection for an existing file:

When the LAYOUTGET is against a file that already has committed data on the data servers, the file's encoding is fixed by the bytes already written; the metadata server does not choose an encoding at layout-issue time. Re-encoding an existing file is a separate operation (see the heterogeneous mirror set primitive in Section 8.1.1 and the migration paths in the proxy server draft), not a consequence of a LAYOUTGET hint. In this case ffv2lh_supported_types is not a selection input; it is an admissibility check. The metadata server issues a layout with the file's actual encoding and evaluates whether the client can consume it:¶

• If the file's encoding is in the client's ffv2lh_supported_types list, the metadata server issues the layout normally.¶

• If it is not, the metadata server takes one of the fallback actions enumerated in "Fallback when no overlap exists" below (return NFS4ERR_ENCODING_NOT_SUPPORTED, route I/O through the metadata server, or route through a translating proxy server).¶

Fallback when no overlap exists:

If the server's policy cannot be satisfied by any encoding the client supports, the metadata server has three options:¶

1. Return NFS4ERR_ENCODING_NOT_SUPPORTED on the LAYOUTGET. The client MAY retry with a different (possibly empty) ffv2lh_supported_types list to learn the server's encoding repertoire through the errors returned.¶

2. Fall back to I/O via the metadata server itself, so the client's reads and writes are satisfied by the metadata server translating to the underlying data server encoding on the client's behalf (see Section 6.2 for the metadata-server I/O fallback). This is correct but serializes all I/O for the encoding-ignorant client through a single actor.¶

3. Route the client through a translating proxy that understands both the file's native encoding and an encoding the client does support. The metadata server issues a layout with the proxy's data-server entry carrying FFV2_DS_FLAGS_PROXY and a coding_type the client does support (typically FFV2_ENCODING_REPLICATED for a minimal NFSv4.2 client, or FFV2_ENCODING_PASSTHROUGH / a flat NFSv3 view for an NFSv3 client). The proxy encodes and decodes on the fly against the real data servers. This preserves parallel I/O for the encoding-ignorant client that the metadata-server I/O fallback loses. The proxy registration, directive, and credential-forwarding rules are defined in the proxy server draft; this draft defines only the layout flag (FFV2_DS_FLAGS_PROXY in Section 8.4) that makes the proxy visible to the client.¶

Options (1), (2), and (3) are not mutually exclusive: a given deployment MAY implement any combination. A deployment that supports (3) covers all the clients that (1) and (2) would cover and additionally preserves parallel I/O for encoding-ignorant clients.¶

Runtime encoding change:

If a metadata server changes its encoding policy after layouts have been issued (for example, a deployment upgrade that retires an older encoding), the metadata server MUST recall the affected layouts via CB_LAYOUTRECALL and may re-issue new layouts with the new encoding. Clients that do not support the new encoding LAYOUTRETURN with NFS4ERR_ENCODING_NOT_SUPPORTED, and the server either grants a layout using a mutually-supported encoding or the client falls back to I/O via the metadata server.¶

This mechanism deliberately avoids a separate capability-bit handshake at EXCHANGE_ID. ffv2_layouthint4 already provides per-request negotiation; adding a session-level capability set would duplicate it and would complicate encoding upgrades without additional value, because a client that genuinely upgrades its encoding set at runtime can simply update the ffv2lh_supported_types on its next LAYOUTGET.¶

Note: In Figure 18 ffv2_encoding_type_data4 is an enumerated union with the payload of each arm being defined by the protection type. ffv2m_client_id tells the client which id to use when interacting with the data servers.¶

The ffv2_layout4 structure (see Figure 18) specifies a layout in that portion of the data file described in the current layout segment. It is either a single instance or a set of mirrored copies of that portion of the data file. When mirroring is in effect, it protects against loss of data in layout segments.¶

While not explicitly shown in Figure 18, each layout4 element returned in the logr_layout array of LAYOUTGET4res (see Section 18.43.2 of [RFC8881]) describes a layout segment. Hence, each ffv2_layout4 also describes a layout segment. It is possible that the file is concatenated from more than one layout segment. Each layout segment MAY represent different striping parameters.¶

The ffv2m_striping_unit_size field (inside each ffv2_mirror4) is the stripe unit size in use for that mirror. The stripe width W is given by the number of elements in ffv2s_data_servers within each ffv2_stripes4 (the count of data servers over which each stripe is spread). If ffv2m_striping is FFV2_STRIPING_NONE the mirror is unstriped and ffv2m_striping_unit_size MUST be 1 (matching the FFV2_STRIPING_NONE rule in Section 8.9 and Section 9); when ffv2m_striping is FFV2_STRIPING_SPARSE or FFV2_STRIPING_DENSE the field carries the stripe unit size in bytes with a minimum of 64. The mapping scheme (sparse or dense) is selected per mirror by ffv2m_striping and is detailed in Section 9.¶

Stripe unit size and stripe count MAY differ between mirrors in the same layout segment. In particular, mirrors of different encoding types (see Section 8.1.1) have stripe counts determined by their respective (ffv2dp_data, ffv2dp_parity) protection structures, and there is no requirement that those structures match across mirrors. Each mirror is self-consistent internally; cross-mirror coherence is at the byte level (every mirror represents the same file bytes), not at the stripe-geometry level.¶

The ffv2l_mirrors field represents an array of state information for each mirrored copy of the current layout segment. Each element is described by a ffv2_mirror4 type.¶

ffv2ds_deviceid provides the deviceid of the storage device holding the data file.¶

ffv2ds_file_info is an array of ffv2_file_info4 structures, each pairing a filehandle (ffv2fi_fh_vers) with a stateid (ffv2fi_stateid). There MUST be exactly as many elements in ffv2ds_file_info as there are in ffv2da_versions. Each element of the array corresponds to a particular combination of ffv2dv_version, ffv2dv_minorversion, and ffv2dv_coupling provided for the device. The array allows for server implementations that have different filehandles and stateids for different combinations of version, minor version, and coupling strength. See Section 8.14 for how to handle versioning issues between the client and storage devices.¶

For tight coupling, ffv2fi_stateid provides the stateid to be used by the client to access the file. The metadata server registers ffv2fi_stateid with each tight coupling capable storage device via TRUST_STATEID (see Section 6.4) before returning the layout; the storage device validates subsequent CHUNK operations against its trust table.¶

For loose coupling and an NFSv4 storage device (necessarily a PASSTHROUGH mirror per Section 7.1, since non-PASSTHROUGH encodings require tight coupling), the client MUST use the anonymous stateid to perform I/O on the storage device, because the metadata server stateid has no meaning to a storage device that is not participating in the control protocol. In this case the metadata server MUST set ffv2fi_stateid to the anonymous stateid.¶

For an NFSv3 storage device (ffv2dv_version = 3), the tight coupling model does not apply: Section 7.1 requires ffv2dv_coupling to equal FFV2_COUPLING_SYNTHETIC_UIDS whenever ffv2dv_version equals 3, because NFSv3 has no wire encoding for stateids. The corresponding ffv2fi_stateid element in the ffv2ds_file_info array MUST therefore be the anonymous stateid and is unused; an NFSv3 data server uses the synthetic-uid fencing model (see Section 6.2) rather than a stateid-based trust table.¶

This specification of the ffv2fi_stateid restricts both models for NFSv4.x storage protocols:¶

loosely couple

the stateid has to be an anonymous stateid¶

tightly couple

the stateid has to be a global stateid¶

By pairing each ffv2fi_fh_vers with its own ffv2fi_stateid inside ffv2_file_info4, the flexible file v2 layout addresses a limitation in the flexible file v1 layout where a single stateid was shared across all filehandles.¶

Whether the ffv2fi_stateid values across an ffv2_file_info4 array are distinct depends on each entry's coupling mode per the rules above. Loose-coupling and NFSv3 entries MUST carry the anonymous stateid; those entries are therefore byte-identical by mandate. Tight-coupling entries carry stateids the metadata server assigned and registered via TRUST_STATEID; the metadata server MAY assign these distinctly per filehandle version or MAY reuse the same stateid across entries.¶

The client MUST treat each (ffv2fi_fh_vers, ffv2fi_stateid) pair as an opaque, independent authorization unit. The client MUST NOT compare ffv2fi_stateid values across entries in the array and MUST NOT infer any relationship between two entries whose stateid values are byte-identical. When the client selects an entry to use for I/O, it presents that entry's stateid with that entry's filehandle; other entries in the array are unused for that I/O.¶

For loosely coupled storage devices, ffv2ds_user and ffv2ds_group provide the synthetic user and group to be used in the RPC credentials that the client presents to the storage device to access the data files. For tightly coupled storage devices, the user and group on the storage device will be the same as on the metadata server; that is, if ffv2dv_coupling has any tight coupling flag set (see Section 7.1), then the client MUST ignore both ffv2ds_user and ffv2ds_group.¶

The allowed values for both ffv2ds_user and ffv2ds_group are specified as owner and owner_group, respectively, in Section 5.9 of [RFC8881]. For NFSv3 compatibility, user and group strings that consist of decimal numeric values with no leading zeros can be given a special interpretation by clients and servers that choose to provide such support. The receiver may treat such a user or group string as representing the same user as would be represented by an NFSv3 uid or gid having the corresponding numeric value. Note that if using Kerberos for security, the expectation is that these values will be a name@domain string.¶

ffv2ds_efficiency describes the metadata server's evaluation as to the effectiveness of each mirror. Note that this is per layout and not per device as the metric may change due to perceived load, availability to the metadata server, etc. Higher values denote higher perceived utility. The way the client can select the best mirror to access is discussed in Section 11.1.1.¶

8.11.2. Error Codes from LAYOUTGET

[RFC8881] provides little guidance as to how the client is to proceed with a LAYOUTGET that returns an error of either NFS4ERR_LAYOUTTRYLATER, NFS4ERR_LAYOUTUNAVAILABLE, and NFS4ERR_DELAY. Within the context of this document:¶

NFS4ERR_LAYOUTUNAVAILABLE:

there is no layout available and the I/O is to go to the metadata server. Note that it is possible to have had a layout before a recall and not after.¶

NFS4ERR_LAYOUTTRYLATER:

there is some issue preventing the layout from being granted. If the client already has an appropriate layout, it should continue with I/O to the storage devices.¶

NFS4ERR_DELAY:

there is some issue preventing the layout from being granted. If the client already has an appropriate layout, it should not continue with I/O to the storage devices.¶

8.11.3. Client Interactions with FFV2_FLAGS_NO_IO_THRU_MDS

FFV2_FLAGS_NO_IO_THRU_MDS is normative: when the metadata server sets FFV2_FLAGS_NO_IO_THRU_MDS on a layout, the client MUST NOT proxy I/O for that layout through the metadata server, even after detecting a network disconnect to a storage device (Section 8.2.1). A client that cannot reach a storage device on which it holds a NO_IO_THRU_MDS layout MUST return the layout via LAYOUTRETURN and reacquire (via LAYOUTGET), at which point the metadata server chooses whether to grant a new layout with the flag cleared, grant a layout naming a different storage device, or fall back to metadata-server-terminated I/O via the encoding-negotiation path (Section 8.11.1) with the flag cleared.¶

The NO_IO_THRU_MDS flag is not advisory; it is an instruction the client MUST honor. When I/O through the metadata server is required (for example, via the encoding-negotiation fallback path in Section 8.11.1), the metadata server MUST clear NO_IO_THRU_MDS on the fallback layout it issues. A client MUST NOT interpret a set NO_IO_THRU_MDS flag as advisory or bypass it.¶

11.1.1. Selecting a Mirror

When the metadata server grants a layout to a client, it MAY let the client know how fast it expects each mirror to be once the request arrives at the storage devices via the ffv2ds_efficiency member. While the algorithms to calculate that value are left to the metadata server implementations, factors that could contribute to that calculation include speed of the storage device, physical memory available to the device, operating system version, current load, etc.¶

However, what should not be involved in that calculation is a perceived network distance between the client and the storage device. The client is better situated for making that determination based on past interaction with the storage device over the different available network interfaces between the two; that is, the metadata server might not know about a transient outage between the client and storage device because it has no presence on the given subnet.¶

As such, it is the client that decides which mirror to access for reading the file. The requirements for writing to mirrored layout segments are presented below.¶

11.1.2. Writing to Mirrors

11.1.3. Metadata Server Resilvering of the File

The metadata server may elect to create a new mirror of the layout segments at any time. This might be to resilver a copy on a storage device that was down for servicing, to provide a copy of the layout segments on storage with different storage performance characteristics, etc. As the client will not be aware of the new mirror and the metadata server will not be aware of updates that the client is making to the layout segments, the metadata server MUST recall the writable layout segment(s) that it is resilvering. If the client issues a LAYOUTGET for a writable layout segment that is in the process of being resilvered, then the metadata server can deny that request with an NFS4ERR_LAYOUTUNAVAILABLE.¶

The client's fallback while the layout is withheld follows the per-encoding rules in Section 10.1 and Section 10.2: for a mirror whose I/O reduces to FFV2_ENCODING_REPLICATED or FFV2_ENCODING_PASSTHROUGH the client MAY perform the I/O through the metadata server as an ordinary NFSv4.1+ READ or WRITE; for a mirror using any chunked encoding the metadata server itself cannot service that I/O (it does not hold the encoded shards), so the client's only fallback path is through a proxy server if one is available for the file. If no proxy server is available, the client MUST wait for the metadata server to complete resilvering and re-issue LAYOUTGET rather than attempt to route the I/O through the metadata server.¶

11.2.1. Encoding a Data Block

                 +-------------+
                 | data block  |
                 +-------+-----+
                         |
                         |
   +---------------------+-------------------------------+
   |            Erasure Encoding (Transform Forward)     |
   +---+----------------------+---------------------+----+
       |                      |                     |
       |                      |                     |
   +-------------------+  +-------------------+  +-------------------+
   | HEADER            |..| HEADER            |..| HEADER            |
   +-------------------+  +-------------------+  +-------------------+
   | owner:            |..| owner:            |..| owner:            |
   |   co_cohort_id:42 |..|   co_cohort_id:42 |..|   co_cohort_id:42 |
   |   co_client_id: 6 |..|   co_client_id: 6 |..|   co_client_id: 6 |
   |   co_id       : 1 |..|   co_id       : 1 |..|   co_id       : 1 |
   | guard:            |..| guard:            |..| guard:            |
   |   cg_gen_id   : 3 |..|   cg_gen_id   : 3 |..|   cg_gen_id   : 3 |
   |   cg_client_id: 6 |..|   cg_client_id: 6 |..|   cg_client_id: 6 |
   | payload_id    : 0 |..| payload_id    : M |..| payload_id    : 5 |
   | checksum      :   |..| checksum      :   |..| checksum      :   |
   +-------------------+  +-------------------+  +-------------------+
   | CHUNK             |..| CHUNK             |..| CHUNK             |
   +-------------------+  +-------------------+  +-------------------+
   | data: ....        |..| data: ....        |..| data: ....        |
   +-------------------+  +-------------------+  +-------------------+
      Data Server 1          Data Server N          Data Server 6

Each data block of the file resident in the client's cache of the file will be encoded into N different payloads to be sent to the data servers as shown in Figure 22. As CHUNK_WRITE (see Section 25.11) can encode multiple write_chunk4 into a single transaction, a more accurate description of a CHUNK_WRITE is in Figure 23.¶

  +------------------------------------+
  | CHUNK_WRITEargs                    |
  +------------------------------------+
  | cwa_stateid: 0                     |
  | cwa_offset: 1                      |
  | cwa_stable: FILE_SYNC4             |
  | cwa_cohort_id: 0x000000000000002a  |
  | cwa_client_id: 6                   |
  | cwa_co_ids:                        |
  |         [0]:  1                    |
  |         [1]:  2                    |
  |         [2]:  3                    |
  | cwa_payload_id: 0                  |
  | cwa_chunk_size  :  1048            |
  | cwa_checksums:                     |
  |         [0]:  0x32ef89             |
  |         [1]:  0x56fa89             |
  |         [2]:  0x7693af             |
  | cwa_chunks  :  ......              |
  +------------------------------------+

This describes a 3 block write of data from an offset of 1 block in the file. All three chunks share cwa_cohort_id and cwa_client_id, so the cohort identity is presented once; each chunk's writer-chosen opaque co_id appears in the co-indexed cwa_co_ids array (here the client chose 1, 2, 3, but any distinct-per-cohort uint32_t values would be equally valid per Section 24.3). The data server can construct the cohort record for the i'th chunk from cwa_chunks using cwa_payload_id, cwa_cohort_id + cwa_client_id + cwa_co_ids[i], and the i'th checksum from cwa_checksums. The cwa_chunks are sent together as a byte stream to increase performance.¶

Assuming that there were no issues, Figure 24 illustrates the results. The payload sequence id is implicit in the CHUNK_WRITEargs.¶

  +-------------------------------+
  | CHUNK_WRITEresok              |
  +-------------------------------+
  | cwr_count: 3                  |
  | cwr_committed: FILE_SYNC4     |
  | cwr_writeverf: 0xf1234abc     |
  | cwr_owners[0]:                |
  |        co_cohort_id: 0x2a     |
  |        co_client_id: 6        |
  |        co_id: 1               |
  | cwr_owners[1]:                |
  |        co_cohort_id: 0x2a     |
  |        co_client_id: 6        |
  |        co_id: 2               |
  | cwr_owners[2]:                |
  |        co_cohort_id: 0x2a     |
  |        co_client_id: 6        |
  |        co_id: 3               |
  +-------------------------------+

  +--------------------+
  | HEADER             |
  +--------------------+
  | guard:             |
  |   gen_id   : 7     |
  |   client_id: 6     |
  | payload_id : 0     |
  | crc32   : 0        |
  +--------------------+
  | CHUNK              |
  +--------------------+
  | data:  ....        |
  +--------------------+
        Data Server 1

  +------------------------------------+
  | CHUNK_WRITEargs                    |
  +------------------------------------+
  | cwa_stateid: 0                     |
  | cwa_offset: 1                      |
  | cwa_stable: FILE_SYNC4             |
  | cwa_cohort_id: 0x000000000000002b  |
  | cwa_client_id: 6                   |
  | cwa_co_ids:                        |
  |         [0]:  1                    |
  | cwa_payload_id: 0                  |
  | cwa_chunk_size  :  1048            |
  | cwa_checksums:                     |
  |         [0]:  0x21de8              |
  | cwa_chunks  :  ......              |
  +------------------------------------+

11.2.2. Decoding a Data Block

    Data Server 1          Data Server N          Data Server 6
  +----------------+     +----------------+     +----------------+
  | HEADER         | ... | HEADER         | ... | HEADER         |
  +----------------+     +----------------+     +----------------+
  | guard:         | ... | guard:         | ... | guard:         |
  |   gen_id   : 3 | ... |   gen_id   : 3 | ... |   gen_id   : 3 |
  |   client_id: 6 | ... |   client_id: 6 | ... |   client_id: 6 |
  | payload_id : 0 | ... | payload_id : M | ... | payload_id : 5 |
  | crc32   :      | ... | crc32   :      | ... | crc32   :      |
  +----------------+     +----------------+     +----------------+
  | CHUNK          | ... | CHUNK          | ... | CHUNK          |
  +----------------+     +----------------+     +----------------+
  | data: ....     | ... | data: ....     | ... | data: ....     |
  +---+------------+     +--+-------------+     +-+--------------+
      |                     |                     |
      |                     |                     |
  +---+---------------------+---------------------+-----+
  |            Erasure Decoding (Transform Reverse)     |
  +---------------------+-------------------------------+
                        |
                        |
                +-------+-----+
                | data block  |
                +-------------+

When reading chunks via a CHUNK_READ operation, the client will decode them into data blocks as shown in Figure 27.¶

At this time, the client could detect issues in the integrity of the data. The handling and repair are out of the scope of this document and MUST be addressed in the document describing each erasure encoding type.¶

  +------------------------------------+
  | CHUNK_READresok                    |
  +------------------------------------+
  | crr_eof: false                     |
  | crr_chunks[0]:                     |
  |        cr_checksum: 0x21de8        |
  |        cr_owner:                   |
  |            co_cohort_id: 0x2b      |
  |            co_client_id: 6         |
  |            co_id       : 1         |
  |        cr_chunk  :  ......         |
  +------------------------------------+

  +--------------------+
  | HEADER             |
  +--------------------+
  | guard:             |
  |   gen_id   : 7     |
  |   client_id: 6     |
  | payload_id  : 0    |
  | crc32    : 0       |
  +--------------------+
  | CHUNK              |
  +--------------------+
  | data:  ....        |
  +--------------------+
       Data Server 1

11.2.3. Write Modes

There are three writing modes for erasure coding, aligned with the three workload classes in Section 4. The mode is selected by the metadata server using FFV2_FLAGS_ONLY_ONE_WRITER in the ffv2l_flags in the ffv2_layout4 (see Figure 18) to inform the client whether it is the only writer to the file or not, and by the client's own understanding of its workload class when the flag is unset.¶

Single writer:

When FFV2_FLAGS_ONLY_ONE_WRITER is set, the client is the only writer to the file. CHUNK_WRITE with cwa_guard not set can be used to write chunks. There is no write contention, but write holes can occur as the client overwrites old data. The client does not need guarded writes, but it does need the ability to rollback writes. This mode corresponds to Use Case 1 (single writer, multiple readers) in Section 4.¶

Concurrent writers with occasional contention:

When FFV2_FLAGS_ONLY_ONE_WRITER is not set, the client is one of several possible concurrent writers. CHUNK_WRITE with cwa_guard set MUST be used to write chunks. Write holes can be caused by multiple clients writing to the same chunk, so the client needs guarded writes to prevent overwrites and also needs the ability to rollback writes. Racing writers that lose the chunk_guard4 CAS receive NFS4ERR_CHUNK_GUARDED and retry with a refreshed guard. This mode corresponds to Use Case 2 (multiple writers without sustained contention) in Section 4.¶

Concurrent writers on disjoint regions:

A specialization of the concurrent-writers mode above, targeting Use Case 3 (multiple writers, disjoint regions -- the HPC checkpoint pattern) in Section 4. The wire primitives are the same as the concurrent-writers mode (cwa_guard set, chunk_guard4 CAS), but the deployment relies on block alignment to keep per-chunk contention rare despite a high overall writer count. Contention that does occur is resolved via the deterministic tiebreaker rule defined in Section 24.2, so racing writers get a stable winner across the mirror set without additional round trips. Deployments that use an XOR-based erasure encoding and expect frequent small edits from this workload class MAY additionally use the delta-write protocol defined in [I-D.haynes-nfsv4-flexfiles-v2-delta-writes], which lets the client forward per-projection XOR deltas directly to each data server, avoiding client-side read-modify-write of the full stripe on the small-edit path.¶

In both modes, clients MUST NOT overwrite payloads which already contain non-atomicity. This directly follows from Section 11.2.7 and MUST be handled as discussed there. Once atomicity in the payload has been detected, the client can use those chunks as a basis for read/modify/update.¶

CHUNK_WRITE is a two-pass operation in cooperation with CHUNK_FINALIZE and CHUNK_ROLLBACK. It writes new bytes into the chunk and transitions the chunk to the PENDING state; the data server is responsible for retaining the prior COMMITTED content until the chunk reaches its next stable state. While a chunk is in PENDING or FINALIZED, a subsequent CHUNK_READ does NOT observe the new content (visibility rules of Section 12 apply: PENDING and FINALIZED chunks are not globally visible; CHUNK_READ returns only COMMITTED content).¶

Concurrent CHUNK_WRITE against a PENDING or FINALIZED chunk is regulated by chunk_guard4 (Section 24.2), not by an implicit lock. A racing writer whose guard check fails receives NFS4ERR_CHUNK_GUARDED; an explicit CHUNK_LOCK holder is signaled by NFS4ERR_CHUNK_LOCKED. No implicit chunk-write lock is acquired by CHUNK_WRITE. A client that requires exclusive access to a chunk MUST invoke CHUNK_LOCK explicitly (Section 25.6).¶

If the CHUNK_WRITE results in a atomic data block, then the client will send a CHUNK_FINALIZE in a subsequent compound to inform the data server that the chunk is finalized and can be overwritten by another CHUNK_WRITE.¶

If the CHUNK_WRITE results in an non-atomic data block, or if the data server returns NFS4ERR_CHUNK_LOCKED, the client reports the condition to the metadata server via LAYOUTERROR with an error code of NFS4ERR_PAYLOAD_NOT_ATOMIC.¶

11.2.4. Selecting the Repair Actor

The repair topology involves three actors communicating along distinct paths, as shown in Figure 30.¶

     +-------------+      (1)         +-----------------+
     |  Reporting  | ---------------> |                 |
     |  client     |   LAYOUTERROR    | Metadata Server |
     |  (detects   |                  |                 |
     |  error)     |                  |                 |
     +-------------+                  +--------+--------+
                                               |
                                               | (2b)
                                               | CB_CHUNK_REPAIR
                                               | (RACE or SCRUB)
                                               v
     +-------------+      (4)         +-----------------+
     |  Repair     | ---------------> |  Data Servers   |
     |  actor      |   CHUNK ops      |  (mirror set    |
     |  (selected  |                  |  for affected   |
     |  per (2a),  |                  |  ranges)        |
     |  adopts     |                  |                 |
     |  lock (3))  |                  |                 |
     +-------------+                  +-----------------+

     (1)   Reporting client LAYOUTERRORs to the metadata server.
     (2a)  Metadata server selects a repair actor (may be a
           client -- possibly the reporting client -- a data
           server under tight coupling, or a proxy server).
     (2b)  Metadata server escrows the chunk lock and issues
           CB_CHUNK_REPAIR to the selected repair actor.
     (3)   Repair actor adopts the lock and drives the repair.
     (4)   Repair actor issues CHUNK_LOCK_ADOPT, CHUNK_WRITE_REPAIR,
           CHUNK_FINALIZE, CHUNK_COMMIT, and CHUNK_REPAIRED against
           the mirror set.

The metadata server is the authority that selects the repair actor for a non-atomic payload. The candidate set is any client, any data server (in a tightly coupled deployment), or the proxy server (in a proxy server deployment); the selection is analogous to the way the metadata server assigns per-mirror priority via ffv2ds_efficiency (see Section 11.1.1): the protocol does not prescribe the selection algorithm, and each deployment MAY tune its policy.¶

Implementations MAY consider factors such as:¶

• Whether a client holds an active write layout on the affected payload (the client most likely to hold surviving shards in cache).¶

• Whether a client has previously reported atomic shards to the metadata server via LAYOUTSTATS or a prior LAYOUTERROR.¶

• Whether the layout exposes a data server carrying FFV2_DS_FLAGS_REPAIR as a target for reconstructed shards.¶

• Network proximity, observed latency, or recent client load -- the same class of information that informs ffv2ds_efficiency.¶

The selection algorithm is not normative. What is normative is that every client MUST be prepared to:¶

1. Receive a repair request for a payload that the client does not have an outstanding write layout on, and did not write; and¶

2. Continue its own workload after reporting NFS4ERR_PAYLOAD_NOT_ATOMIC without itself being selected to repair the payload it reported.¶

The metadata server signals the selected client via the CB_CHUNK_REPAIR callback (Section 26.1), which identifies the file, the affected ranges (each with its own triggering nfsstat4), and a wall-clock deadline. A client that receives CB_CHUNK_REPAIR for a file for which it does not already hold a layout MUST acquire a layout via LAYOUTGET before attempting the repair.¶

Operational expectations for CB_CHUNK_REPAIR: CB_CHUNK_REPAIR is an exceptional path, triggered only by concurrent writer races or data-server failures. It is not a steady-state operation and its frequency is a function of racing-writer and data-server-failure rates in the deployment rather than of normal client workload. Implementations SHOULD treat the CB_CHUNK_REPAIR handler as rare-path code and avoid over-optimizing it. Implementations SHOULD, however, provision enough client-side compute to handle a repair transaction without stalling their foreground I/O, because foreground throughput during repair is the externally observable cost of this callback.¶

11.2.5. Repair Protocol: Normative vs. Informative

The selection algorithm is non-normative and deployment-tunable. The externally-observable state transitions of the repair flow are normative. The line between the two is drawn at what another party on the wire -- the metadata server, another client, a reader -- can observe. What no other party can see (client-internal ordering, retry policy, whether to CHUNK_READ first to confirm the failure) is left to implementations.¶

The following requirements are normative. An implementation that violates any of these can leak inconsistency or write-holes into the cluster:¶

Final state flat:

Every shard in every range identified in a CB_CHUNK_REPAIR MUST reach either the COMMITTED state (repaired) or the EMPTY state (rolled back). No shard is left in PENDING or FINALIZED indefinitely.¶

Lock before write:

The repair actor MUST adopt the lock on every affected range via CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT (Section 25.6) before issuing any CHUNK_WRITE_REPAIR, CHUNK_ROLLBACK, or CHUNK_WRITE on a chunk in that range. The lock on the affected chunks is held continuously from the failure that triggered CB_CHUNK_REPAIR through the adoption; at no point is the range unlocked.¶

Clear the errored state:

On the reconstruction path, the repair actor MUST issue CHUNK_REPAIRED after CHUNK_COMMIT. Without it, readers continue to see holes regardless of on-disk state.¶

Release locks explicitly:

CHUNK_ROLLBACK does not release chunk locks. On the rollback path the client MUST issue CHUNK_UNLOCK on each affected chunk. A client that walks away without either completing CHUNK_REPAIRED or issuing CHUNK_UNLOCK holds the locks until lease expiry, blocking progress for other writers.¶

Deadline honored:

The client MUST drive every range to its final flat state before ccra_deadline, or MUST respond to the CB_CHUNK_REPAIR with NFS4ERR_DELAY (requesting an extension), NFS4ERR_ENCODING_NOT_SUPPORTED (declining), or NFS4ERR_PAYLOAD_LOST (declaring the data unrecoverable). A deadline that elapses without any of these leaves the metadata server free to re-select; the client MUST NOT continue repair-related CHUNK operations after the deadline without first re-verifying its layout and the chunk lock state.¶

Terminal return codes:

NFS4ERR_ENCODING_NOT_SUPPORTED MUST mean "decline; select another client." NFS4ERR_PAYLOAD_LOST MUST mean "the data is not recoverable; do not retry." The metadata server relies on these to decide whether to re-issue.¶

The following aspects are informative / implementation-defined:¶

• Choice between the reconstruction path (CHUNK_WRITE_REPAIR) and the rollback path (CHUNK_ROLLBACK) on a given range. The protocol MUST support both; the client MAY use either based on its local state and whether reconstruction is feasible from surviving shards.¶

• Ordering among multiple affected ranges in a single CB_CHUNK_REPAIR (parallel or serial).¶

• Whether to issue CHUNK_READ to confirm the failure mode before reconstructing.¶

• Retry policy on transient CHUNK_WRITE_REPAIR errors below the deadline cutoff.¶

• How the repair status is surfaced to local filesystem API callers.¶

11.2.6. Carrying Out the Repair

With the normative framing above, the reconstruction path is:¶

1. CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT on each affected range (Section 25.6).¶

2. CHUNK_WRITE_REPAIR with the reconstructed data for each non-atomic shard. The client's chunk_owner4 on this and all subsequent operations is the one it presented in the CHUNK_LOCK ADOPT above; prior owners' generation ids are now historical.¶

3. CHUNK_FINALIZE and CHUNK_COMMIT to persist the repaired shards.¶

4. CHUNK_REPAIRED to clear the errored state.¶

The rollback path, when reconstruction is not possible:¶

1. CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT on each affected range.¶

2. CHUNK_ROLLBACK on each affected shard to restore the previously committed content.¶

3. CHUNK_UNLOCK on each shard.¶

In both paths, the repair actor SHOULD target reconstructed shards according to the following fallback order: first, any data server in the layout carrying FFV2_DS_FLAGS_REPAIR; then the data server that reported the failure (the one carrying the failing shard at the range identified by ccr_offset and ccr_count in the CB_CHUNK_REPAIR argument). If neither is available, the client MUST return NFS4ERR_PAYLOAD_LOST on the CB_CHUNK_REPAIR response; the metadata server is then responsible for adding a new REPAIR-flagged data server to the layout (drawn from its out-of-band pool) and re-driving the repair.¶

11.2.7. Reading Chunks

The client reads chunks from the data file via CHUNK_READ. The number of chunks in the payload that need to be atomic depend on both the Erasure Encoding Type and the level of protection selected. If the client has enough atomic chunks in the payload, then it can proceed to use them to build a data block. If it does not have enough atomic chunks in the payload, then it can either decide to return a LAYOUTERROR of NFS4ERR_PAYLOAD_NOT_ATOMIC to the metadata server or it can retry the CHUNK_READ until there are enough atomic chunks in the payload.¶

As another client might be writing to the chunks as they are being read, it is entirely possible to read the chunks while they are not atomic. As such, it might even be the non-atomic chunks which contain the new data and a better action than building the data block is to retry the CHUNK_READ to see if new chunks are overwritten.¶

11.2.8. Whole File Repair

Whole-file repair is the case in which too many data servers have failed, or too many chunks have been lost, for the per-range repair flow defined in Section 11.2.4 to reconstruct the file in place. In this case the metadata server MUST either:¶

1. Construct a new layout backed by replacement data servers and drive the reconstruction via the proxy server mechanism (a designated data server acts as the source of truth for client I/O during the transition, pushing reconstructed content to the replacement data servers in the background). The proxy server mechanism also covers the non-repair cases where a file's layout must change while remaining available to clients -- policy-driven layout transitions, data server maintenance evacuation, administrative ingest, TLS coverage transition, and filehandle-backend migration.¶

2. If the metadata server has no proxy server capable data server available, or the surviving shards are insufficient to reconstruct any portion of the file, terminate the affected byte ranges with NFS4ERR_PAYLOAD_LOST (see Section 21.1.5).¶

Implementations that do not support the proxy server mechanism can still perform recovery for cases where per-range repair suffices, using CB_CHUNK_REPAIR and the repair client selection rules in Section 11.2.4. Such implementations will surface NFS4ERR_PAYLOAD_LOST on any failure that exceeds per-range repair's reach, including the multi-data-server failure scenarios the proxy server mechanism is intended to handle.¶

11.3.1. Steady-state heterogeneous mirrors

The transition-window patterns above (assimilation, migration, repair) are the most visible motivations for heterogeneous mirror sets, but they are not the only ones. A file's mirror set MAY be heterogeneous in steady state -- where no transition is in progress and no transition is planned -- when the deployment's storage pools have different encoding capabilities and the file is too large to fit in any single pool.¶

Consider an operator with three 100-TB storage pools. Pool A is a set of NFSv3-only data servers, which is a hard capability constraint: the chunked encodings require NFSv4.2's CHUNK operations, so an NFSv3-only pool can hold data only under an FFV2_ENCODING_PASSTHROUGH mirror. Pools B and C are NFSv4.2 data servers, which are encoding-agnostic at the wire level (CHUNK_READ and CHUNK_WRITE just move opaque chunk payloads; the encoding transform is client-side). The operator, however, has chosen to write Pool B under FFV2_ENCODING_RS_VANDERMONDE and Pool C under FFV2_ENCODING_MOJETTE_SYSTEMATIC -- a policy decision motivated by durability diversity (hedging against a correlated failure in any single encoder implementation) and by per-workload fit (each encoding's reconstruction cost and failure-tolerance profile suits a different tenant on that pool).¶

A 250-TB file cannot fit in any single pool. Striping the file across all three pools is forced by capacity arithmetic: 250 > 100. Because bytes in each pool were written under the pool's chosen encoding, the layout for this file MUST name the actual per-mirror ffv2_encoding_type4 so that clients decode each segment correctly: PASSTHROUGH for the Pool A segment, RS_VANDERMONDE for the Pool B segment, MOJETTE_SYSTEMATIC for the Pool C segment. The heterogeneity is not a transition window; it is the permanent consequence of striping across pools whose bytes were written under different operator-chosen encodings.¶

In this steady-state case, no proxy server mediated transition machinery is involved. The client receives a layout enumerating the mirrors at different ffv2m_coding values and routes chunk operations to the appropriate data server per segment (or, if the client cannot speak one of the encodings, requests proxy mediation per the proxy server draft's section "Encoding Translation for Encoding-Ignorant Clients" ([I-D.haynes-nfsv4-flexfiles-v2-proxy-server])). The layout machinery that supports this case is exactly the per-mirror encoding naming primitive described above; no additional protocol elements are required to express it.¶

The transient case (one file moving between encodings) and the steady-state case (one file permanently striped across heterogeneous pools) share a single wire primitive: an ffv2l_mirrors array that admits mixed ffv2_encoding_type4 values. Removing that primitive would foreclose both cases and would force one of three workarounds: (a) split the 250-TB file into three independently-named files (loses single-namespace semantics), (b) require every pool to implement every encoding (forces a single-vendor or single-implementation procurement story), or (c) require an always-on transcoding proxy in front of every read and write (re-introduces the centralized data-plane that the proxy server role is scoped to avoid in steady state). None of these alternatives address the steady-state case while preserving the transition-window flexibility this document specifies.¶

11.6.1. Overview

FFV2_ENCODING_XOR_PARITY is a single-parity, systematic, RAID-5-shape encoding: k data shards accompanied by one parity shard computed as the bytewise XOR of every data shard. Parameters: k in the range 1 to 254, m fixed at 1.¶

Unlike Reed-Solomon and Linux md/raid6, XOR_PARITY requires no finite-field arithmetic. The "encoding" is a plain XOR reduction across k shards, so the implementation footprint is trivial and the compute cost scales at memory bandwidth. This makes XOR_PARITY the simplest candidate encoding: any conformant implementation can support it without a GF(2^8) library.¶

11.6.2. Encoding

Given k data shards, each of shard_len bytes, encoding produces a single parity shard of shard_len bytes:¶

For each byte position j in [0, shard_len):
  parity[j] = data[0][j] XOR data[1][j] XOR ... XOR data[k-1][j]

All shards (data and parity) are the same size. No table lookups, no multiplications; a straight bitwise XOR reduction.¶

11.6.3. Recovery

XOR_PARITY tolerates the loss of exactly one shard (any one data shard or the parity shard).¶

• Missing data shard data[i]: reconstruct by XOR-ing all surviving shards (k-1 data shards plus the parity shard): data[i][j] = parity[j] XOR data[0][j] XOR ... (skip i) ... XOR data[k-1][j].¶

• Missing parity shard: recompute from the k intact data shards using the encoding step above.¶

Reconstruction cost is a single XOR reduction over shard_len bytes; memory bandwidth dominates. No matrix inversion is required.¶

Loss of two or more shards is unrecoverable under XOR_PARITY. Deployments requiring tolerance of two or more concurrent losses SHOULD use FFV2_ENCODING_LINUX_MD_RAID (m=2) or FFV2_ENCODING_RS_VANDERMONDE (configurable m).¶

11.6.4. Interoperability

XOR_PARITY produces a parity shard byte-identical to the P (first) parity row of Reed-Solomon Vandermonde encoding at m=1 and to the P row of Linux md/raid6 at any m >= 1. This follows from using the same primitive coefficients:¶

• RS Vandermonde at m=1 uses parity row [1, 1, ..., 1] in GF(2^8), which reduces to bitwise XOR (see Section 11.8).¶

• LINUX_MD_RAID's P row is defined as the bitwise XOR of every data shard, identical to XOR_PARITY's parity by construction.¶

A receiver capable of FFV2_ENCODING_RS_VANDERMONDE at m=1 or FFV2_ENCODING_LINUX_MD_RAID at m >= 1 therefore consumes an XOR_PARITY-encoded chunk without re-encoding, provided the k matches. Conversely, an XOR_PARITY receiver consumes the P row of any GF(2^8) family encoding at m >= 1 unchanged.¶

11.6.5. XOR_PARITY Interoperability Test Vectors

Concrete byte-level test vector with k = 3, m = 1, shard_len = 1:¶

The fourth row is byte-identical to the P entry of the same input in Table 8 -- the wire-compat property in action.¶

11.6.6. XOR_PARITY Shard Sizes

All XOR_PARITY shards (data and parity) are exactly shard_len bytes. chunk_size equals shard_len for every mirror in the layout. Total storage overhead is 1/k of payload.¶

11.7.1. Overview

FFV2_ENCODING_LINUX_MD_RAID is the Linux kernel md/raid6 P + Q double-parity encoding, evaluated in GF(2^8) with primitive polynomial x^8 + x^4 + x^3 + x^2 + 1 (encoded as 0x1d when the implicit degree-8 term is dropped, 0x11d when it is kept). Parameters: k in the range 2 to 253, m fixed at 2.¶

The encoding produces two parity shards. P is the bitwise XOR of every data shard (identical to FFV2_ENCODING_XOR_PARITY's parity, see Section 11.6). Q is a weighted sum sum(g^i * data_i) in GF(2^8) with g = 2. The construction is bit-for-bit compatible with Linux kernel lib/raid6 [LINUX-RAID6] and with FFV2_ENCODING_RS_VANDERMONDE at m<=2 (see Section 11.8).¶

The k=1 case (a single data shard with P and Q) is degenerate and MUST NOT be used with FFV2_ENCODING_LINUX_MD_RAID. Callers who need triple-mirror semantics MUST use FFV2_ENCODING_REPLICATED with N=3 instead.¶

11.7.2. LINUX_MD_RAID Galois Field Arithmetic

All LINUX_MD_RAID operations are performed over GF(2^8), the Galois field with 256 elements. Each element is represented as a byte.¶

Irreducible Polynomial:

The field is constructed using x^8 + x^4 + x^3 + x^2 + 1, encoded as 0x1d when the implicit x^8 term is dropped (Linux md convention), or 0x11d when it is kept (RS Vandermonde convention in Section 11.8). Both encodings refer to the same field; only the notation differs.¶

Primitive Element:

g = 2. Powers of g cycle through GF(2^8) \ {0} with period 255.¶

Addition:

Bitwise XOR.¶

Multiplication:

Via log/antilog tables in the reference implementation, or SIMD vector multiply in modern kernels. See the Linux kernel lib/raid6/{int.uc, sse2.c, altivec.uc, ...} sources [LINUX-RAID6] for the canonical implementation.¶

11.7.3. LINUX_MD_RAID Encoding

Given k data shards, each of shard_len bytes, encoding produces two parity shards P and Q, each shard_len bytes:¶

For each byte position j in [0, shard_len):
  P[j] = data[0][j] XOR data[1][j] XOR ... XOR data[k-1][j]
  Q[j] =         1 * data[0][j]
           XOR   g * data[1][j]
           XOR g^2 * data[2][j]
           XOR ...
           XOR g^(k-1) * data[k-1][j]

where multiplication is in GF(2^8) and g = 2. All shards (data, P, and Q) are the same size.¶

The P row is identical to FFV2_ENCODING_XOR_PARITY's parity by construction; the two encodings share the P computation. The Q row is the m=2 P + Q construction from Section 11.8, and produces byte-identical output to FFV2_ENCODING_RS_VANDERMONDE at m=2.¶

11.7.4. LINUX_MD_RAID Recovery

LINUX_MD_RAID tolerates up to two concurrent shard losses (any two of the k + 2 shards).¶

Single-shard failure:

If only P or only Q is missing, recompute from the k intact data shards using the encoding step above.¶

If exactly one data shard data[i] is missing but both P and Q are intact, prefer the P-only recovery (cheaper): XOR all surviving shards (k-1 data plus P) to recover data[i].¶

Dual-shard failure:

If two data shards are missing, use both P and Q to solve a 2x2 linear system in GF(2^8) per byte position. See the Linux kernel lib/raid6/recov.c for the canonical solver.¶

If one data shard and one parity shard are missing, first recover the data shard using the intact parity row (as in the single-shard case), then recompute the missing parity from the reconstructed data.¶

Loss of three or more shards is unrecoverable under LINUX_MD_RAID. Deployments requiring m >= 3 fault tolerance MUST use FFV2_ENCODING_RS_VANDERMONDE.¶

11.7.5. LINUX_MD_RAID Interoperability

FFV2_ENCODING_LINUX_MD_RAID is wire-compatible with FFV2_ENCODING_RS_VANDERMONDE at m <= 2 by construction: both encoders emit byte-identical P and Q for the same (k, data) input at m <= 2.¶

• At m = 1, LINUX_MD_RAID's P row equals RS_VANDERMONDE's m=1 parity and equals XOR_PARITY's parity (see Section 11.6).¶

• At m = 2, LINUX_MD_RAID's P + Q equals RS_VANDERMONDE's m=2 P + Q rows byte-for-byte.¶

RS_VANDERMONDE joined the m <= 2 wire-compat set as a wire format revision in this document: its m <= 2 parity rows are the hand-crafted P/Q construction rather than the normalized-Vandermonde bottom rows RS uses at m >= 3. See Section 11.8 for the RS construction and its interoperability parameters.¶

A client implementing either LINUX_MD_RAID or RS_VANDERMONDE at m <= 2 can consume the other without re-encoding, provided the (k, m) geometry matches.¶

11.7.6. LINUX_MD_RAID Interoperability Test Vectors

Concrete byte-level test vector with k = 3, m = 2, shard_len = 1:¶

Every row above is byte-identical to the k=3, m=2 test vector for FFV2_ENCODING_RS_VANDERMONDE (Table 8) at the same input. This is the wire-compat property in action: an implementation whose LINUX_MD_RAID output matches these bytes is producing the same wire encoding as RS_VANDERMONDE at m=2.¶

11.7.7. LINUX_MD_RAID Shard Sizes

All LINUX_MD_RAID shards (data, P, Q) are exactly shard_len bytes. chunk_size equals shard_len for every mirror in the layout. Total storage overhead is 2/k of payload.¶

11.8.1. Overview

Reed-Solomon (RS) codes are Maximum Distance Separable codes: for a (k + m, k) code, any k of the k + m encoded shards suffice to recover the original data. The code tolerates the simultaneous loss of up to m shards. [Plank97] is a tutorial treatment of RS coding in RAID-like systems and is the recommended background reading for implementers unfamiliar with the construction used here.¶

11.8.2. Galois Field Arithmetic

All RS operations are performed over GF(2^8), the Galois field with 256 elements. Each element is represented as a byte.¶

Irreducible Polynomial:

The field is constructed using the irreducible polynomial x^8 + x^4 + x^3 + x^2 + 1 (0x11d in hexadecimal). The primitive element (generator) is g = 2, which has multiplicative order 255.¶

Addition:

Addition in GF(2^8) is bitwise XOR.¶

Multiplication:

Multiplication uses log/antilog tables. For non-zero elements a and b: a * b = exp(log(a) + log(b)), where the exp table is doubled to 512 entries to avoid modular reduction on the index sum.¶

These are the classical constructions from Berlekamp (1968) and Peterson & Weldon (1972). The log/antilog table approach for GF(2^8) multiplication predates all known patents on SIMD-accelerated GF arithmetic. Implementors considering SIMD acceleration of GF(2^8) operations should be aware of US Patent 8,683,296 (StreamScale), which covers certain SIMD-based GF multiplication techniques.¶

11.8.3. Encoding Matrix

The systematic encoding matrix E has (k + m) rows and k columns. The top k rows are always the k x k identity, so data shards pass through unchanged. The bottom m parity rows are chosen as follows.¶

E[k][j] = 1    for j = 0, 1, ..., k-1

E[k][j]     = 1              for j = 0, 1, ..., k-1   (P row)
E[k+1][j]   = g^j            for j = 0, 1, ..., k-1   (Q row)

11.8.4. Encoding

Given k data shards, each of shard_len bytes, encoding produces m parity shards, each also shard_len bytes:¶

For each byte position j in [0, shard_len):
  For each parity shard i in [0, m):
    parity[i][j] = sum over s in [0, k) of P[i][s] * data[s][j]

where the sum and product are in GF(2^8). All shards (data and parity) are the same size.¶

11.8.5. Decoding

When one or more shards are lost (up to m), reconstruction proceeds by matrix inversion:¶

1. Select k available shards (from the k + m total).¶

2. Form a k x k sub-matrix S of the encoding matrix E by selecting the rows corresponding to the available shards.¶

3. Compute S_inv = S^(-1) using Gaussian elimination in GF(2^8).¶

4. Multiply S_inv by the vector of available shard data at each byte position to recover the original k data shards.¶

5. If any parity shards are also missing, regenerate them by re-encoding from the recovered data shards.¶

The reconstruction cost is dominated by the matrix inversion, which is O(k^3) in GF(2^8) multiplications.¶

11.8.6. RS Interoperability Requirements

For two implementations of FFV2_ENCODING_RS_VANDERMONDE to interoperate, they MUST agree on all of the following parameters. Any deviation produces a different encoding matrix and renders data unrecoverable by a different implementation.¶

Irreducible polynomial:

x^8 + x^4 + x^3 + x^2 + 1 (0x11d).¶

Primitive element:

g = 2.¶

Evaluation points:

shard i (i = 0, 1, ..., k+m-1) uses alpha_i = i + 1 in GF(2^8) (values 1 through k+m, all non-zero and distinct).¶

Vandermonde entries:

V[i][j] = alpha_i^j = (i+1)^j in GF(2^8) for i = 0..k+m-1, j = 0..k-1.¶

Matrix normalization:

E = V * T^(-1) where T is the top k x k sub-matrix (rows for shards 0..k-1).¶

Parameter bound:

k + m MUST NOT exceed 255.¶

These parameters fully determine the encoding matrix for any (k, m) configuration in the permitted range.¶

11.8.7. RS Interoperability Test Vectors

The following worked examples pin the encoding matrix and end-to-end encodings for two representative geometries: k=2 m=1 (exercises the m=1 all-ones parity row) and k=3 m=2 (exercises the m=2 P + Q parity rows). An implementation whose encoded output matches these tables is using the same GF(2^8) representation and the same parity-row construction as required by the interoperability parameters above.¶

E = [ [0x01, 0x00],    // identity block for data shard 0
      [0x00, 0x01],    // identity block for data shard 1
      [0x01, 0x01] ]   // parity row P = [1, 1]

E = [ [0x01, 0x00, 0x00],   // identity block for data shard 0
      [0x00, 0x01, 0x00],   // identity block for data shard 1
      [0x00, 0x00, 0x01],   // identity block for data shard 2
      [0x01, 0x01, 0x01],   // P row = [1, 1, 1]
      [0x01, 0x02, 0x04] ]  // Q row = [g^0, g^1, g^2] with g = 2

P[j] = data[0][j] XOR data[1][j] XOR data[2][j]
Q[j] = 1 * data[0][j] XOR 2 * data[1][j] XOR 4 * data[2][j]

11.8.8. RS Shard Sizes

All RS shards (data and parity) are exactly shard_len bytes. This simplifies the CHUNK operation protocol: chunk_size is exactly the shard size for all mirrors.¶

11.9.1. Overview

The Mojette Transform is an erasure coding technique based on discrete geometry rather than algebraic field operations. It computes 1D projections of a 2D grid along selected directions. Given enough projections, the original grid can be reconstructed exactly.¶

The transform operates on fixed-width words combined with bitwise XOR -- the additive group of (GF(2))^(W*8) where W is the element width in bytes. Encoders and decoders MUST use XOR; modular integer addition is not equivalent and is not interoperable. XOR has no carry chain, is its own inverse (so the residual subtraction in reconstruction is identical to the forward accumulation), and scales straightforwardly to wider SIMD lanes (NEON, SSE, AVX, AVX-512) without requiring multiplicative Galois field operations.¶

For interoperability, this specification pins the element width to W = 8 bytes (64 bits) for both FFV2_ENCODING_MOJETTE_SYSTEMATIC and FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC in this revision. All implementations MUST use W = 8; a future revision (or a distinct registered encoding type) may lift the fixed width, at which point it becomes a wire-visible parameter. Fixing W in this revision prevents an interoperability hazard: the same shard input XOR'd at W = 4 versus W = 8 produces different bin values that appear "close but wrong" to a mismatched decoder.¶

11.9.2. Grid Structure

Data is arranged as a P x Q grid of unsigned integer elements, where P is the number of columns and Q is the number of rows. For k data shards of S bytes each with W-byte elements:¶

P = S / W       (columns per row)
Q = k           (rows = data shards)

11.9.3. Directions

A direction is a pair of coprime integers (p_i, q_i). This specification pins all directions to q_i = 1 for both Mojette encoding types (systematic and non-systematic) in this revision. Non-unity q_i values may be introduced by a future distinct registered encoding type.¶

For n = k + m total shards (Mojette non-systematic) or m parity shards (Mojette systematic), the direction set is determined by the following mandatory rule on the shard count N (N = n for non-systematic, N = m for systematic).¶

For N even (N = 2t):

directions = { (p, 1) : p in {-t, -t+1, ..., -1, 1, 2, ..., t} } -- symmetric around zero, giving |directions| = 2t = N.¶

For N odd (N = 2t + 1):

directions = { (p, 1) : p in {-t, -t+1, ..., -1, 1, 2, ..., t, t+1} } -- asymmetric by including one additional positive magnitude so that |directions| = 2t + 1 = N.¶

Direction slots are then sorted by p_i ascending (most-negative p first, most-positive p last) to give the canonical direction order. The direction in slot i is used to compute the projection for shard slot (k + i) in systematic form, or for shard slot i in non-systematic form. Examples:¶

• N = 4 (even): p = {-2, -1, 1, 2}¶

• N = 6 (even): p = {-3, -2, -1, 1, 2, 3}¶

• N = 3 (odd): p = {-1, 1, 2}¶

• N = 5 (odd): p = {-2, -1, 1, 2, 3}¶

Two implementations that follow this algorithm on the same N generate identical direction sets in identical slot order and therefore identical shard layouts. Implementations MUST NOT diverge from this algorithm (e.g., by using a different tie-breaking rule for odd N) without registering a distinct encoding type.¶

11.9.4. Forward Transform (Encoding)

For each direction (p_i, q_i), the forward transform computes a 1D projection. Each bin XORs the grid elements that lie on a discrete line through the grid. This specification adopts the bin convention of [NORMAND]: a grid cell at (row, col) maps to bin¶

b = row * p + col * q - off

where off is chosen so that the smallest reachable bin index is 0 (off = min over all (row, col) in [0, Q) x [0, P) of row * p + col * q). All implementations MUST use this convention -- the alternative "transposed" convention b = col * p - row * q + off produces a different bin ordering and is not interoperable.¶

The full forward transform along direction (p, q) is then:¶

Projection(b, p, q) = XOR over all (row, col) where
                       row * p + col * q - off = b
                       of Grid[row][col]

The number of bins B in a projection is:¶

B(p, q, P, Q) = |p| * (Q - 1) + |q| * (P - 1) + 1

For q = 1, this simplifies to:¶

B = abs(p) * (Q - 1) + P

The byte size of the projection is B * W.¶

11.9.5. Katz Reconstruction Criterion

Reconstruction from a set of n projections is possible if and only if the Katz criterion [KATZ] holds over those n projections:¶

SUM(i=1..n) |q_i| >= Q    OR    SUM(i=1..n) |p_i| >= P

With q_i = 1 pinned for every direction (see the Directions subsection above), the q-sum simplifies to n >= Q.¶

For the non-systematic form, n = k + m and every direction counts toward the criterion. The criterion holds for the initial encoding (design-time check on the direction set) and must continue to hold after any losses; the surviving projections must satisfy Katz over the same grid dimensions (P, Q).¶

For the systematic form, the raw data rows act as "projections at direction (p=0, q=1)": row r is Grid[r] and contributes q = 1 toward the q-sum of the Katz criterion. For an arbitrary loss pattern with r data-row losses and s parity-projection losses (with r + s <= m), the surviving set comprises k - r data rows (each q_i = 1) and m - s parity projections. Decoding proceeds by:¶

1. Subtracting the contributions of the k - r surviving data rows from the m - s surviving parity projections (the "residual").¶

2. Applying the corner-peeling algorithm over the residual to recover the r missing rows.¶

Step 2 succeeds if and only if the Katz criterion holds over the m - s residual projections against the r x P unknown sub-grid. Substituting: the criterion reduces to m - s >= r (q-sum, since q_i = 1 for every parity direction and the unknown-grid Q is r) OR the analogous p-sum condition over the residual. Because the mandatory direction algorithm above generates m projections with distinct nonzero p values, the p-sum condition also holds for any loss pattern with r + s <= m; the systematic form therefore achieves Maximum-Distance-Separable-like recovery up to m combined data-row and parity-projection losses.¶

11.9.6. Inverse Transform (Decoding)

The choice of inverse algorithm is purely an implementer concern: all correct inverses produce byte-identical plaintext from the same shards and bin layout, so the choice has no on-the-wire impact. Two well-known algorithms apply.¶

The corner-peeling algorithm:¶

1. Count how many unknown elements contribute to each bin.¶

2. Find any bin with exactly one contributor (singleton).¶

3. Recover the element, XOR it back through all projections.¶

4. Repeat until all elements are recovered.¶

Corner peeling runs in O(n * P * Q) and is the simplest correct inverse. Implementations MAY instead use the geometry-driven inverse of [NORMAND], which precomputes a recurrence over the sorted projection slopes and walks each line once: it eliminates the inner singleton search and runs substantially faster on the parameter ranges typical of flexible file v2 layout deployments (high redundancy, wide stripes), with no change to the shards or to the reconstructed plaintext.¶

11.9.7. Systematic Mojette

In the systematic form (FFV2_ENCODING_MOJETTE_SYSTEMATIC), the first k shards are the original data rows and the remaining m shards are projections. Healthy reads require no decoding.¶

Reconstruction of missing data rows proceeds via the corner-peeling algorithm of [NORMAND]:¶

1. Load available parity projections.¶

2. Subtract contributions of present data rows (residual).¶

3. Corner-peel the residual to recover missing rows.¶

Reconstruction cost is O(m * k) -- a fundamental advantage over RS at wide geometries (k >= 8).¶

11.9.8. Non-Systematic Mojette

In the non-systematic form (FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC), all k + m shards are projections. Every read requires the full inverse transform. This provides constant performance regardless of failure count, but at higher baseline read cost than systematic.¶

11.9.9. Mojette Shard Sizes and Layout

Slot-to-direction mapping:

The canonical shard layout for Mojette is:¶

Systematic (FFV2_ENCODING_MOJETTE_SYSTEMATIC):

shard slots 0..k-1 carry the k data rows (row r in slot r); shard slots k..k+m-1 carry the m parity projections in the canonical direction order defined in the Directions subsection above (direction slot i occupies shard slot k + i).¶

Non-systematic (FFV2_ENCODING_MOJETTE_NON_SYSTEMATIC):

shard slots 0..k+m-1 carry the n = k + m parity projections in the canonical direction order (direction slot i occupies shard slot i).¶

Bin ordering within a projection:

Within a projection shard the bins are serialized in ascending bin-index order (bin 0 first, bin B-1 last), with no gap or header. Each bin value is W = 8 bytes wide; the W-byte element is serialized in big-endian byte order (the same order the data-row shards present their W-byte grid elements in).¶

Projection sizes:

Unlike Reed-Solomon, Mojette parity shard sizes vary by direction:¶

Chunk sizing for variable-length projections:

When a projection shard is written via CHUNK_WRITE / CHUNK_FINALIZE / CHUNK_COMMIT, the shard is divided into chunks by the following mapping. Let shard_bytes = B * W be the projection shard's total byte size (where B is the number of bins per the B formula above (Table 10 uses it) applied to the shard's direction (p, q) and the grid dimensions (P, Q)):¶

• num_chunks = ceil(shard_bytes / chunk_size)¶

• Chunk j (for j = 0..num_chunks-1) covers the shard byte range [j * chunk_size, min((j+1) * chunk_size, shard_bytes)).¶

• The final chunk (chunk num_chunks - 1) MAY be shorter than chunk_size if shard_bytes is not a multiple of chunk_size; all other chunks are exactly chunk_size bytes.¶

The chunk_size value is a per-mirror parameter and does not vary across the parity projections of a single file, even though the shard sizes vary. For a file with parity projections of sizes S_i = B_i * W, the number of chunks per shard is ceil(S_i / chunk_size) per shard; a reader that requests chunk offset >= S_i on shard i receives NFS4ERR_PAYLOAD_LOST (per Section 21.1.5) with a short read reporting the shard's true byte length.¶

13.2.1. Data-File Identification on the Data Server

The "MUST reject" rules in this section apply on the data server side only to files the data server has identified as chunked data files. A data server identifies a file by any of the following means, in decreasing order of authority:¶

fattr4_chunked_data_file = TRUE:

The metadata server has set this attribute (Section 23.2) when it allocated the file as a chunked data file. This is the authoritative per-file identification and is required for a data server that supports the attribute.¶

Live TRUST_STATEID entry for the file:

Under trusted-stateid tight coupling (Section 6.4), a live trust entry for the file registered via TRUST_STATEID identifies the file as under flexible file v2 layout management. This is a fallback for data servers that do not yet support fattr4_chunked_data_file, and it is per-client rather than per-file, but it is sufficient to trigger the "MUST reject" rules for that client's operations against the file.¶

Deployment namespace convention:

Deployments SHOULD export chunked data files in a namespace scope not shared with client-accessible file access, as one means of preventing a misconfigured or malicious client from reaching data files through a normal NFS mount. A data server MAY apply the "MUST reject" rules to every file in a namespace configured for flexible file v2 layout data-file service, without per-file classification. Note that a separate NFS export does not close direct filesystem access on the data server host itself; that is outside the scope of this specification.¶

If none of the above identifies the file, the data server cannot reliably classify it and the client-side "MUST NOT" rules in this section remain the primary defense. Client-side compliance is mandatory in all deployment modes regardless of what the data server can enforce.¶

13.2.2. Session and Identity Plumbing

Required for all protection modes:¶

• SEQUENCE, PUTFH, GETFH, PUTROOTFH ([RFC8881] Sections 18.46, 18.19, 18.8, 18.21).¶

• EXCHANGE_ID, CREATE_SESSION, DESTROY_SESSION, BIND_CONN_TO_SESSION, DESTROY_CLIENTID ([RFC8881] Sections 18.35, 18.36, 18.37, 18.34, 18.50).¶

• RECLAIM_COMPLETE (Section 18.51 of [RFC8881]).¶

• SECINFO, SECINFO_NO_NAME ([RFC8881] Sections 18.29, 18.45): discovery of acceptable security flavors on the data server.¶

These operations are baseline NFSv4.2 session plumbing and are supported on data files as on any NFSv4.2 file.¶

13.2.3. Stateid Model on the Data Server

The stateid presented on a CHUNK operation is a layout stateid returned by a prior LAYOUTGET against the metadata server (see Section 18.43 of [RFC8881]), NOT an open stateid, byte range lock stateid, or delegation stateid. A pNFS client does NOT issue OPEN against the data server. This is a meaningful departure from the stateid model in Section 18.32 of [RFC8881] (which states that the WRITE stateid "represents a value returned from a previous byte range LOCK or OPEN request or the stateid associated with a delegation"), and clients implementing Flexible File Version 2 MUST NOT carry over those expectations to the data path.¶

The three roles the RFC 8881 stateid plays on a regular NFSv4 server split apart in the Flexible File Version 2 data-server model:¶

Open and share-mode tracking:

Lives at the metadata server, established by OPEN (Section 18.16 of [RFC8881]) on the metadata-server filehandle. The metadata server's open stateid is NOT exposed to data servers; share-mode conflicts are resolved at the metadata server before LAYOUTGET grants a layout.¶

Byte-range lock tracking:

Does not apply at the data server. Locking on the data path is chunk-range rather than byte range, expressed via CHUNK_LOCK, and the lock holder is identified by chunk_owner4 (the (co_cohort_id, co_client_id, co_id) triple) rather than by a lock stateid. A client wanting byte range locks on a file MUST acquire them on the metadata-server filehandle, where standard Section 12 of [RFC8881] byte range locking applies.¶

I/O authorization on the data server:

The layout stateid carried on CHUNK operations. encodings that use CHUNK operations require tight coupling (Section 7.1); the metadata server registers each issued layout stateid with the data server via TRUST_STATEID together with the ffv2m_client_id assigned to the writer, and the data server validates subsequent CHUNK operation stateids against the trust table and the presented cwa_client_id against the trust-table's tsa_client_id. Loose coupling applies only to PASSTHROUGH mirrors, which use regular READ/WRITE authorized by the synthetic uid/gid the layout carries (see Section 6.2).¶

Because the layout stateid does authorization but does not identify a per-open or per-lock owner, a single client may present the same layout stateid on many CHUNK operations across many parallel writers within the client, without any of the open-owner ordering constraints Section 8.2.2 of [RFC8881] imposes on regular NFSv4 stateids. Chunk-level write ordering and contention are resolved by the per-chunk chunk_guard4 CAS (Section 24.2) and the chunk-range CHUNK_LOCK, not by stateid-owner sequencing.¶

13.2.4. GETATTR on a Data File

GETATTR MAY be issued by a client against a data file. The primary use case is repair: a repair actor selected by CB_CHUNK_REPAIR may need to query the per-server file size or allocation state when reconstructing a payload, and the proxy server described informally in Section 12.3 similarly benefits from attribute queries on surviving mirrors. Diagnostic use is also permitted.¶

Clients MUST NOT treat GETATTR values returned by a data server as authoritative for any file attribute (size, timestamps, owner, mode, ACL, and so on). The metadata server is the sole authority for file attributes. Values returned by a data server reflect the per-server data file instance only and MAY diverge from the metadata server's view, particularly during a write layout's lifetime or during a proxy server transition. A client that uses a data-server GETATTR result to determine the file's visible size will observe inconsistencies.¶

13.2.5. SETATTR on a Data File

Clients MUST NOT issue SETATTR against a data file. A data server that has identified the file as a chunked data file (see Section 13.2.1) MUST reject a client SETATTR with NFS4ERR_NOTSUPP.¶

Attribute changes on data files MUST be reconciled with the metadata server's view and cannot be applied unilaterally by a client. A client that wants to truncate, change the mode, change ownership, or otherwise modify attributes on a file MUST issue SETATTR to the metadata server for the file's metadata server handle; the metadata server fans the change out to the data files as a control-plane operation.¶

This rule explicitly covers truncate (SETATTR with size in the bitmap): a client MUST NOT truncate a data file directly. See Section 13.2.6 for how the metadata server handles truncate on chunked files. Similarly, a client MUST NOT issue DEALLOCATE against a data file; see the next subsection.¶

13.2.6. Metadata-Server-Driven Truncate on Chunked Files

A client that wants to truncate a chunked file MUST issue SETATTR(FATTR4_SIZE) to the metadata-server filehandle (see Section 13.2.5). The metadata server translates the logical truncate into per-shard size changes across the data servers in each mirror. For FFV2_ENCODING_REPLICATED, per-shard size equals the logical truncate size; for erasure-coded encodings the per-shard sizes are derived from the geometry parameters below.¶

Stripe-aligned truncate:

When the new size lies on a stripe boundary (including zero), no chunk re-encoding is required. The metadata server computes per-shard sizes from the encoding geometry it issued in the layout (k, m, and the projection parameters for Mojette; see Section 11.9) and issues per-data-server SETATTR(FATTR4_SIZE) with the computed per-shard size. Geometry parameters are sufficient arithmetic; no encoding implementation is required at the metadata server.¶

Non-stripe-aligned truncate:

When the new size falls within a stripe, the data shards covering the partial stripe must be truncated and the parity shards re-encoded from the truncated data. Because re-encoding requires running the erasure transform, the metadata server MUST delegate this case to an encoding-aware actor: either a proxy server for proxy-mediated truncate, or an encoding-aware client selected per Section 11.2.4 via CB_CHUNK_REPAIR with the affected partial-stripe chunks as the repair target. If neither path is available, the metadata server MUST return NFS4ERR_NOTSUPP to the originating SETATTR.¶

The metadata server knows encoding geometry from the layout but is not required to include an encoding implementation. The delegation rule above accommodates a metadata server that has geometry knowledge only.¶

13.2.7. PASSTHROUGH Data Files (FFV2_ENCODING_PASSTHROUGH)

For a mirror whose ffv2m_encoding_type_data is FFV2_ENCODING_PASSTHROUGH (see Section 11.4), client operations on the data file follow the same pattern as the File Layout Type in Section 13.6 of [RFC8881] and the Flexible File Version 1 Layout Type in [RFC8435]:¶

Required:¶

• READ (Section 18.22 of [RFC8881]).¶

• WRITE (Section 18.32 of [RFC8881]).¶

• COMMIT (Section 18.3 of [RFC8881]).¶

Optional (the client MAY send, and the data server MAY support):¶

• READ_PLUS (Section 15.10 of [RFC7862]): hole-aware reads.¶

• SEEK (Section 15.11 of [RFC7862]): hole and data detection.¶

• ALLOCATE (Section 15.1 of [RFC7862]): space reservation hint.¶

The client MUST NOT send:¶

• DEALLOCATE (Section 15.4 of [RFC7862]): hole punching is a metadata-server responsibility; the client issues DEALLOCATE on the metadata-server filehandle, and the metadata server fans out to the data servers as a control-plane operation.¶

13.2.8. Chunked Data Files

For a mirror whose ffv2m_encoding_type_data is any of the chunked encoding types defined in this document -- i.e., every FFV2ENCODING* value except FFV2_ENCODING_PASSTHROUGH (see Section 11.4) -- client operations use the CHUNK operations rather than READ / WRITE / COMMIT. This includes FFV2_ENCODING_REPLICATED, whose per-shard payload is the input bytes verbatim but is dispatched via the CHUNK operations to carry the per-chunk checksum and guard (Table 12 legend).¶

Required for all chunked clients:¶

• CHUNK_WRITE.¶

• CHUNK_READ.¶

• CHUNK_FINALIZE.¶

• CHUNK_COMMIT.¶

• CHUNK_HEADER_READ.¶

• CHUNK_LOCK and CHUNK_UNLOCK.¶

• CHUNK_ROLLBACK.¶

Required for clients that participate in repair:¶

• CHUNK_ERROR.¶

• CHUNK_REPAIRED.¶

• CHUNK_WRITE_REPAIR.¶

Clients MUST NOT send:¶

• READ, WRITE, COMMIT against a chunked data file. A data server MUST reject these with NFS4ERR_NOTSUPP and MAY log the client for operator attention; this case is almost always a client bug in which the client did not inspect the mirror's ffv2m_encoding_type_data before issuing I/O.¶

• READ_PLUS, SEEK, ALLOCATE, DEALLOCATE against a chunked data file. Chunk-level allocation is a metadata-server responsibility.¶

• SETATTR against a chunked data file (the general prohibition in Section 13.2.5 applies to all data files; truncate in particular is handled by the metadata server per Section 13.2.6).¶

13.2.9. Operations That MUST NOT Be Sent to a Data File

Clients MUST NOT send the following operations to a data server on a data file, regardless of protection mode. A data server that has identified the file as a chunked data file (see Section 13.2.1) MUST return NFS4ERR_NOTSUPP:¶

• OPEN, CLOSE, OPEN_DOWNGRADE, OPEN_CONFIRM ([RFC8881] Sections 18.16, 18.2, 18.18, 18.20). Opens occur on the metadata server; the stateid obtained there is used on the data path.¶

• LOCK, LOCKU, LOCKT, RELEASE_LOCKOWNER ([RFC8881] Sections 18.10, 18.11, 18.13, 18.24). Byte-range locks on data files are not supported; chunked files use CHUNK_LOCK, and PASSTHROUGH files rely on metadata-server coordination.¶

• DELEGPURGE, DELEGRETURN, WANT_DELEGATION ([RFC8881] Sections 18.5, 18.6 and Section 15.3 of [RFC7862]). Delegations are issued by the metadata server.¶

• Any operation whose purpose is to manipulate the file's namespace: RENAME, LINK, SYMLINK, CREATE (at the client's file-creation use, not the metadata server's own allocation of data files on the data server described above), REMOVE. Namespace operations belong on the metadata server.¶

• Any ACL-scoped SETATTR or GETATTR bit (FATTR4_ACL, FATTR4_DACL, FATTR4_SACL). Access control on data files is delegated to the metadata server.¶

• CLONE, COPY, COPY_NOTIFY, OFFLOAD_CANCEL, OFFLOAD_STATUS ([RFC7862] Sections 15.13, 15.2, 15.3, 15.8, 15.9). File-level data migration is a metadata-server responsibility.¶

• LAYOUTGET, LAYOUTCOMMIT, LAYOUTRETURN, LAYOUTSTATS, LAYOUTERROR, GETDEVICEINFO, GETDEVICELIST ([RFC8881] Sections 18.43, 18.42, 18.44, [RFC7862] Sections 15.7, 15.6, [RFC8881] Sections 18.40, 18.41). Layout operations belong on the metadata server.¶

• TRUST_STATEID, REVOKE_STATEID, BULK_REVOKE_STATEID (Section 25.13, Section 25.14, Section 25.15). These are metadata-server-to-data-server control-plane operations; a data server rejects them with NFS4ERR_PERM when received on a client session (see Section 6.4.2).¶

• NFSv4 READ, WRITE, and COMMIT ([RFC8881] Sections 18.22, 18.32, 18.3). Chunked data files use CHUNK_READ, CHUNK_WRITE, and CHUNK_COMMIT respectively; NFSv4 byte-range READ / WRITE / COMMIT bypasses the chunk envelope (chunk_guard4 CAS, per-chunk checksum, lifecycle state) that this document's consistency guarantees depend on. A data server that has identified the file as a chunked data file MUST reject NFSv4 READ / WRITE / COMMIT with NFS4ERR_NOTSUPP. PASSTHROUGH data files (Section 11.4) are non-chunked and continue to use NFSv4 READ / WRITE / COMMIT normally; their fattr4_chunked_data_file is FALSE (Section 23.2).¶

13.2.10. CHUNK Operations on Non-Chunked Data Files

The reciprocal rule: clients MUST NOT send CHUNK operations (CHUNK_READ, CHUNK_WRITE, CHUNK_FINALIZE, CHUNK_COMMIT, CHUNK_ROLLBACK, CHUNK_HEADER_READ, CHUNK_LOCK, CHUNK_UNLOCK, CHUNK_ERROR, CHUNK_REPAIRED, CHUNK_WRITE_REPAIR) to a data server on a file that is not a chunked data file. Non-chunked data files (PASSTHROUGH mirrors, and any file the metadata server has not marked as chunked) do not carry a chunk envelope and cannot honor CHUNK operation semantics. A data server that has identified the file as non-chunked (fattr4_chunked_data_file = FALSE) MUST reject CHUNK operations against that file with NFS4ERR_NOTSUPP. A data server that cannot identify the file under any of the paths in Section 13.2.1 relies on the client-side "MUST NOT" as the primary defense; a client with an active layout will not present a chunked-layout stateid against a non-chunked file, so standard stateid validation (Section 8.2 of [RFC8881]) catches the misdirected CHUNK operation with NFS4ERR_BAD_STATEID before the chunk-layer rules apply.¶

14.1.1. ffv2_ioerr4

   /// struct ffv2_ioerr4 {
   ///         offset4        ffv2ie_offset;
   ///         length4        ffv2ie_length;
   ///         stateid4       ffv2ie_stateid;
   ///         device_error4  ffv2ie_errors<>;
   /// };
   ///

Recall that [RFC7862] defines device_error4 as in Figure 36:¶

   struct device_error4 {
           deviceid4       de_deviceid;
           nfsstat4        de_status;
           nfs_opnum4      de_opnum;
   };

The ffv2_ioerr4 structure is used to return error indications for data files that generated errors during data transfers. These are hints to the metadata server that there are problems with that file. For each error, ffv2ie_errors.de_deviceid, ffv2ie_offset, and ffv2ie_length represent the storage device and byte range within the file in which the error occurred; ffv2ie_errors represents the operation and type of error. The use of device_error4 is described in Section 15.6 of [RFC7862].¶

Even though the storage device might be accessed via NFSv3 and reports back NFSv3 errors to the client, the client is responsible for mapping these to appropriate NFSv4 status codes as de_status. Likewise, the NFSv3 operations need to be mapped to equivalent NFSv4 operations.¶

14.2.1. ff_io_latency4

   /// struct ffv2_io_latency4 {
   ///         uint64_t       ffv2il_ops_requested;
   ///         uint64_t       ffv2il_bytes_requested;
   ///         uint64_t       ffv2il_ops_completed;
   ///         uint64_t       ffv2il_bytes_completed;
   ///         uint64_t       ffv2il_bytes_not_delivered;
   ///         nfstime4       ffv2il_total_busy_time;
   ///         nfstime4       ffv2il_aggregate_completion_time;
   /// };
   ///

Both operation counts and bytes transferred are kept in the ff_io_latency4 (see Figure 37). As seen in ff_layoutupdate4 (see Section 14.2.2), READ and WRITE operations are aggregated separately. READ operations are used for the ff_io_latency4 ffv2lu_read. Both WRITE and COMMIT operations are used for the ff_io_latency4 ffv2lu_write. "Requested" counters track what the client is attempting to do, and "completed" counters track what was done. There is no requirement that the client only report completed results that have matching requested results from the reported period.¶

ffv2il_bytes_not_delivered is used to track the aggregate number of bytes requested but not fulfilled due to error conditions. ffv2il_total_busy_time is the aggregate time spent with outstanding RPC calls. ffv2il_aggregate_completion_time is the sum of all round-trip times for completed RPC calls.¶

In Section 3.3.1 of [RFC8881], the nfstime4 is defined as the number of seconds and nanoseconds since midnight or zero hour January 1, 1970 Coordinated Universal Time (UTC). The use of nfstime4 in ff_io_latency4 is to store time since the start of the first I/O from the client after receiving the layout. In other words, these are to be decoded as duration and not as a date and time.¶

Note that LAYOUTSTATS are cumulative, i.e., not reset each time the operation is sent. If two LAYOUTSTATS operations for the same file and layout stateid originate from the same NFS client and are processed at the same time by the metadata server, then the one containing the larger values contains the most recent time series data.¶

14.2.2. ff_layoutupdate4

   /// const FFV2_LAYOUTSTATS_FLAGS_LOCAL  = 0x00000001;
   ///
   /// typedef uint32_t   ffv2_layoutstats_flags4;
   ///
   /// struct ffv2_layoutupdate4 {
   ///         netaddr4                ffv2lu_addr;
   ///         nfs_fh4                 ffv2lu_fhandle;
   ///         ffv2_io_latency4        ffv2lu_read;
   ///         ffv2_io_latency4        ffv2lu_write;
   ///         nfstime4                ffv2lu_duration;
   ///         ffv2_layoutstats_flags4 ffv2lu_flags;
   /// };
   ///

ffv2lu_addr differentiates which network address the client is connected to on the storage device. In the case of multipathing, ffv2lu_fhandle indicates which read-only copy was selected. ffv2lu_read and ffv2lu_write convey the latencies for both READ and WRITE operations, respectively. ffv2lu_duration is used to indicate the time period over which the statistics were collected. ffv2lu_flags is a 32-bit bitmask reporting per-report properties; when FFV2_LAYOUTSTATS_FLAGS_LOCAL is set, the I/O was serviced by the client's cache. This flag allows the client to inform the metadata server about "hot" access to a file it would not normally be allowed to report on. Unassigned bits are reserved for future revisions; receivers MUST ignore bits they do not recognize.¶

14.2.3. ff_iostats4

   /// struct ffv2_iostats4 {
   ///         offset4            ffv2is_offset;
   ///         length4            ffv2is_length;
   ///         stateid4           ffv2is_stateid;
   ///         io_info4           ffv2is_read;
   ///         io_info4           ffv2is_write;
   ///         deviceid4          ffv2is_deviceid;
   ///         ffv2_layoutupdate4 ffv2is_layoutupdate;
   /// };
   ///

[RFC7862] defines io_info4 as in Figure 39.¶

   struct io_info4 {
           uint64_t        ii_count;
           uint64_t        ii_bytes;
   };

With pNFS, data transfers are performed directly between the pNFS client and the storage devices. Therefore, the metadata server has no direct knowledge of the I/O operations being done and thus cannot create on its own statistical information about client I/O to optimize the data storage location. ff_iostats4 MAY be used by the client to report I/O statistics back to the metadata server upon returning the layout.¶

Since it is not feasible for the client to report every I/O that used the layout, the client MAY identify "hot" byte ranges for which to report I/O statistics. The definition and/or configuration mechanism of what is considered "hot" and the size of the reported byte range are out of the scope of this document. For client implementations, providing reasonable default values and an optional run-time management interface to control these parameters is suggested. For example, a client can define the default byte range resolution to be 1 MB in size and the thresholds for reporting to be 1 MB/second or 10 I/O operations per second.¶

For each byte range, ffv2is_offset and ffv2is_length represent the starting offset of the range and the range length in bytes. ffv2is_read.ii_count, ffv2is_read.ii_bytes, ffv2is_write.ii_count, and ffv2is_write.ii_bytes represent the number of contiguous READ and WRITE I/Os and the respective aggregate number of bytes transferred within the reported byte range.¶

The combination of ffv2is_deviceid and ffv2lu_addr uniquely identifies both the storage path and the network route to it. Finally, ffv2lu_fhandle allows the metadata server to differentiate between multiple read-only copies of the file on the same storage device.¶

21.1.1. NFS4ERR_ENCODING_NOT_SUPPORTED (Error Code 10097)

The client requested a ffv2_encoding_type4 which the metadata server does not support. I.e., if the client sends a layout_hint requesting an erasure encoding type that the metadata server does not support, this error code can be returned. The client might have to send the layout_hint several times to determine the overlapping set of supported erasure encoding types.¶

21.1.2. NFS4ERR_PAYLOAD_NOT_ATOMIC (Error Code 10098)

The client encountered a payload in which the blocks were non-atomic and stay non-atomic. As the client can not tell if another client is actively writing, it informs the metadata server of this error via LAYOUTERROR. The metadata server can then arrange for repair of the file.¶

21.1.3. NFS4ERR_CHUNK_LOCKED (Error Code 10099)

The client tried an operation on a chunk which resulted in the data server reporting that the chunk was locked. The client will then inform the metadata server of this error via LAYOUTERROR. The metadata server can then arrange for repair of the file.¶

21.1.4. NFS4ERR_CHUNK_GUARDED (Error Code 10100)

The client tried a guarded CHUNK_WRITE on a chunk which did not match the guard on the chunk in the data file. As such, the CHUNK_WRITE was rejected and the client should refresh the chunk it has cached.¶

21.1.5. NFS4ERR_PAYLOAD_LOST (Error Code 10101)

Returned by a repair actor on the CB_CHUNK_REPAIR response (ccrr_status) to indicate that the identified ranges cannot be repaired and the underlying data is no longer recoverable. Causes include: too few surviving shards to meet the reconstruction threshold (Katz criterion for Mojette, any k-of-(k + m) subset for Reed-Solomon Vandermonde), inability to roll back to a previously committed payload because that payload is also lost, or exhaustion of all FFV2_DS_FLAGS_REPAIR data servers available in the layout with no additional replacement reachable from the metadata server's out-of-band pool.¶

On receipt of NFS4ERR_PAYLOAD_LOST from a repair actor, the metadata server MUST NOT retry the repair by selecting a different client -- the payload is damaged. Subsequent client access to the affected ranges MUST fail: a CHUNK_READ (or an NFSv4 READ against a damaged PASSTHROUGH range) returns NFS4ERR_PAYLOAD_LOST; a CHUNK_WRITE that overwrites a damaged range in its entirety MAY succeed and produce a new atomic generation for that range (the write does not depend on the damaged prior content), but a partial-range write that would require reading the damaged bytes returns NFS4ERR_PAYLOAD_LOST as well. Client NFSv4 implementations MUST surface NFS4ERR_PAYLOAD_LOST to the calling application as the POSIX errno EIO; a client MAY additionally record a more specific diagnostic (e.g., Linux EMEDIUMTYPE) in an out-of-band log or attribute, but the application-visible errno is EIO for portability. The metadata server MAY additionally reflect the damage in an attribute-level indicator or operator-facing notification, but those mechanisms are outside the scope of this specification.¶

NFS4ERR_PAYLOAD_LOST is distinct from NFS4ERR_DELAY (transient; metadata server MAY extend the deadline or re-select) and from NFS4ERR_IO (per-operation failure; metadata server MAY retry or re-select). Only NFS4ERR_PAYLOAD_LOST is terminal.¶

21.1.6. NFS4ERR_LAYOUT_CHECKSUM_NOT_SUPPORTED (Error Code 10102)

Returned by the client on LAYOUTRETURN to indicate that the layout's ffv2m_checksum_algorithm (Section 8.9) names a checksum_algorithm4 (Section 24.7) that the client does not implement. The client returns the layout with this error code rather than attempting CHUNK operations it cannot validate.¶

On receipt, the metadata server MAY:¶

• issue a new layout for the same file naming a different checksum_algorithm4 that the client supports (if the file's policy permits any of the algorithms the client does support); or¶

• deny the layout request, in which case the client MUST either fall back to metadata-server-mediated I/O or report an I/O error to the application.¶

NFS4ERR_LAYOUT_CHECKSUM_NOT_SUPPORTED is distinct from NFS4ERR_BADLAYOUT (generic "this layout shape is unusable"): the explicit per-checksum-algorithm signal lets the metadata server discriminate "client can't read this layout because of the checksum algorithm" from "client can't read this layout for some other reason" and respond accordingly.¶

21.1.7. NFS4ERR_NO_PREDECESSOR (Error Code 10103)

Returned on a CHUNK_ROLLBACK against a COMMITTED chunk when the caller names a predecessor generation whose owner-to-index association is no longer recorded by the data server, and no other per-entry error covers the situation. This is the "no restorable predecessor" outcome of the restore case described under "Rollback of COMMITTED Chunks": the predecessor's payload+association pair was released some time earlier under the retention scope rule (Section 12.9) -- by lease expiry, by an earlier CHUNK_ROLLBACK delete case, or by any other terminal transition -- and the data server cannot restore what it no longer holds.¶

NFS4ERR_NO_PREDECESSOR is a per-entry status (a caller that names several generations in a single CHUNK_ROLLBACK MAY receive NFS4ERR_NO_PREDECESSOR on some slots and NFS4_OK or other per-entry errors on others). It is a data-plane result, NOT a control-plane failure: the caller reached the data server successfully, was authorized, and the operation evaluated normally -- the named predecessor simply does not exist to restore.¶

NFS4ERR_NO_PREDECESSOR is distinct from NFS4ERR_INVAL (the triple was invalidated by an explicit delete case and cannot be resurrected -- see "Deletion Atomicity and Invalidated Triples") and from NFS4ERR_PAYLOAD_LOST (terminal payload loss reported on CB_CHUNK_REPAIR). The data server distinguishes NFS4ERR_NO_PREDECESSOR from NFS4ERR_INVAL on the basis of what the data server can concretely observe about the presented triple, NOT on historical knowledge the data server may no longer retain:¶

NFS4ERR_INVAL:

returned in exactly the cases where the data server can concretely recognize the presented triple as invalid:¶

Structurally invalid:

the triple is malformed or truncated, or names a different file or a different data server.¶

Within-window release:

a prior explicit CHUNK_ROLLBACK delete case released the association and the current request falls within the same session slot's replay-cache window of that CHUNK_ROLLBACK, so the data server still holds the concrete invalidation context that identifies this triple as released-by-delete-case.¶

NFS4ERR_NO_PREDECESSOR:

covers every other case in which the data server holds no association for the presented triple, including:¶

Retention-scope release:

release under the retention scope rule (lease expiry, storage-pressure release).¶

Post-window release:

the eventual release of a triple invalidated by an earlier delete case whose replay-cache window has since elapsed.¶

No surviving record:

a well-formed triple for this file and this data server for which the data server holds no surviving record and no concrete invalidation context.¶

An implementation that does not retain a per-triple record after release cannot distinguish "the data server never recorded this triple" from "the data server recorded and later released this triple" outside a live invalidation context, and this specification does not require it to: both cases resolve to NFS4ERR_NO_PREDECESSOR under the "no surviving record" case above. The two errors are not interchangeable: NFS4ERR_INVAL is a caller-side signal that the client MUST NOT retry the same identity, while NFS4ERR_NO_PREDECESSOR is a data-plane signal that the caller MAY fall back to best-effort reconstruction via CHUNK_WRITE_REPAIR.¶

A client that receives NFS4ERR_NO_PREDECESSOR MAY fall back to reconstructing authoritative bytes from surviving shards and writing them via CHUNK_WRITE_REPAIR under a new owner triple; that fallback is best-effort and MAY itself terminate at NFS4ERR_PAYLOAD_LOST if no authoritative source exists. A client that requires the restored generation to retain the predecessor's original owner triple in cases where the retention scope (Section 12.9) would otherwise permit release MUST use the metadata-server escrow control plane (Section 25.16 through Section 25.19), which pins the predecessor's payload and its owner-to-index association jointly against that release rule for as long as the escrow-lock or a client-owned lock adopted from it remains in continuous custody (see Section 27).¶

21.1.8. NFS4ERR_NO_ADOPTABLE_LOCK (Error Code 10104)

Returned by CHUNK_LOCK when a repair actor attempts to adopt a metadata-server escrow lock (Section 24.2.4) and no such adoption is possible on the data server for the requested range. There are four state-level causes:¶

• no metadata-server escrow lock is installed on the data server for the requested range;¶

• a metadata-server escrow lock is installed but its escrow_id4 (Section 24.4) does not match the identity the repair actor presents;¶

• the range's escrow lock is currently under a reconciliation hold following a metadata-server incarnation change (see Section 25.19) and cannot be adopted until the hold clears; or¶

• the escrow was already adopted by a different repair actor whose adoption remains active.¶

Authorization-level failures -- the caller is not the metadata-server-designated repair actor for this escrow, or the caller lacks credentials for the escrow's scope -- surface as NFS4ERR_ACCESS rather than NFS4ERR_NO_ADOPTABLE_LOCK, so that no data-plane information about the current adopter is leaked to an unauthorized caller.¶

On receipt of NFS4ERR_NO_ADOPTABLE_LOCK, the repair client MUST report the outcome to the metadata server via CB_CHUNK_REPAIR's per-range status array (Section 26.1) and MUST NOT unilaterally acquire a fresh CHUNK_LOCK, retry the adoption, or invoke the discovery/fallback path of NFS4ERR_NO_PREDECESSOR (Section 21.1.7) -- the four causes above are all control-plane conditions the metadata server is best placed to resolve (by reissuing the escrow, waiting for the reconciliation hold to clear, or abandoning the repair).¶

21.1.9. NFS4ERR_STALE_ESCROW (Error Code 10105)

Returned by CHUNK_ESCROW_RELEASE when the escrow_id4 the metadata server presents does not match any escrow currently installed on the data server for the referenced range. Two causes lead to the same wire outcome:¶

• no escrow covers the referenced range at the data server (whether never installed, released earlier, or consumed by a repair adoption); or¶

• an escrow covers the range but its recorded escrow_id4 differs from the one presented (the presented identity is a stale reference to an earlier installation).¶

The data server MUST NOT alter any current lock or escrow state as a side effect of returning NFS4ERR_STALE_ESCROW: the response reports "the identity you presented is not what I hold" without changing what is held. In particular, if the range carries a metadata-server escrow lock whose escrow_id4 differs from the presented identity, that lock survives the call unchanged; and if the range carries a client-owned lock that was previously adopted from an escrow whose identity matches, that adopted lock is not affected by a release of the presented (now-stale) identity.¶

21.1.10. NFS4ERR_STALE_MDS_EPOCH (Error Code 10106)

Returned by any of the CHUNK_ESCROW operations (Section 25.16, Section 25.17, Section 25.18) when the requesting metadata server presents an epoch value the data server no longer accepts because a newer metadata-server incarnation has completed a CHUNK_ESCROW_TAKEOVER. The metadata server that receives NFS4ERR_STALE_MDS_EPOCH has been fenced from continued escrow operations on this data server; it MUST NOT retry the operation under the same epoch and MUST obtain a fresh incarnation-lease token and reissue via CHUNK_ESCROW_TAKEOVER before resuming. CHUNK_ESCROW_TAKEOVER itself is not subject to this rejection -- it is the recovery path out of an expired epoch and carries its own compare-and-advance semantics per Section 25.19.¶

NFS4ERR_STALE_MDS_EPOCH is distinct from NFS4ERR_ACCESS (credential-level failure) and from NFS4ERR_STALE_ESCROW (identity mismatch on a specific escrow): the stale-epoch response fences the presenter's entire escrow-control session, not a single per-escrow operation.¶

21.1.11. NFS4ERR_PARTIAL (Error Code 10107)

Returned as the top-level status of a CB_CHUNK_REPAIR response (Section 26.1) when the repair actor evaluated every named range but at least one range's per-range status is not NFS4_OK. The per-range status array (ccrr_range_status) is authoritative for the outcome of each range; NFS4ERR_PARTIAL is the top-level signal that the metadata server MUST consume the per-range array rather than treating the response as uniformly successful or uniformly failed.¶

NFS4ERR_PARTIAL is distinct from operation-wide errors (decode failures, authorization failures, session state failures) that fail every range at once: those return the operation-wide error at the top level with an EMPTY per-range array. NFS4ERR_PARTIAL requires a co-indexed array with one entry per named range and MAY carry any mix of NFS4_OK and per-range failure codes.¶

24.2.1. Metadata-Server Assignment Rules for cg_client_id

To uphold the uniqueness contract, the metadata server MUST follow these rules when assigning cg_client_id (that is, when populating ffv2m_client_id at layout-grant time):¶

• Two clients holding concurrent write layouts on the same file MUST receive distinct cg_client_id values. A client that holds only a read layout need not be assigned a distinct value.¶

• The reserved sentinel CHUNK_GUARD_CLIENT_ID_NONE (0x00000000) MUST NOT be assigned to any client. Reserving 0 prevents an uninitialized cg_client_id field from passing as a real client and ensures the deterministic tiebreaker (numerically lowest wins) does not encode an implicit priority via assignment of 0.¶

• The reserved sentinel CHUNK_GUARD_CLIENT_ID_MDS (0xFFFFFFFF) MUST NOT be assigned to any client.¶

• A cg_client_id MAY be reused by the metadata server after the prior holder's layout has been fully returned (via LAYOUTRETURN or revocation). The metadata server SHOULD avoid reusing a cg_client_id within a single lease period to simplify diagnosis of stale writes.¶

• cg_client_id values do not persist across metadata-server restart. Clients reclaiming layouts during the grace period receive freshly assigned values; the protocol does not rely on any pre-restart assignment surviving.¶

24.2.2. Data-Server Collision Handling

A (cg_gen_id, cg_client_id) pair that the uniqueness contract would otherwise render unique can nonetheless collide if a client and the metadata server disagree about which cg_client_id the client currently holds, or if a client presents a spoofed cg_client_id. The data server enforces the contract locally:¶

• If the data server receives a CHUNK_WRITE whose chunk_guard4 has the same (cg_gen_id, cg_client_id) as a chunk already in PENDING, FINALIZED, or COMMITTED state AND the presented payload differs from the retained payload, the data server MUST reject the write with NFS4ERR_CHUNK_GUARDED and SHOULD report the collision to the metadata server via LAYOUTERROR. This situation is a protocol violation on one side of the conversation; the metadata server resolves it by revoking the offending client's layout and selecting a repair actor under Section 11.2.4.¶

• If a client presents CHUNK_GUARD_CLIENT_ID_MDS as cg_client_id in any client-originated operation, the data server MUST reject the operation with NFS4ERR_INVAL (see Section 24.2.4).¶

• A cg_client_id that does not match the tsa_client_id recorded for the layout stateid under which the CAS operation is issued MUST be rejected with NFS4ERR_BAD_STATEID. This (stateid, client_id) binding is registered by the metadata server via TRUST_STATEID; an unmatched cg_client_id is treated as a stale-layout condition, see Section 6.4.¶

24.2.3. Reserved cg_client_id Value: CHUNK_GUARD_CLIENT_ID_NONE

The value CHUNK_GUARD_CLIENT_ID_NONE (0x00000000) is reserved. It does not denote any client. Reserving 0 prevents an uninitialized cg_client_id field from passing as a real client and ensures the deterministic tiebreaker (numerically lowest wins, see Section 24.2) does not encode an implicit priority via assignment of 0.¶

Clients MUST NOT present CHUNK_GUARD_CLIENT_ID_NONE as the cg_client_id of any client-originated chunk_guard4 or chunk_owner4. A data server that receives such a value from a client MUST reject the operation with NFS4ERR_INVAL.¶

24.2.4. CHUNK_GUARD_CLIENT_ID_MDS and the Metadata-Server Escrow Lock

The value CHUNK_GUARD_CLIENT_ID_MDS (0xFFFFFFFF) is reserved. It denotes that the chunk lock is held by the metadata server itself, in escrow during a repair coordination sequence (see Section 11.2.4). The data server produces a chunk_guard4 with this cg_client_id when the metadata server revokes the prior holder's stateid while that holder still holds chunk locks; the locks MUST NOT be dropped and are transferred to the metadata-server escrow owner instead.¶

The metadata server does not originate CHUNK_LOCK or CHUNK_WRITE traffic on its own session. Clients MUST NOT present CHUNK_GUARD_CLIENT_ID_MDS as the cg_client_id of any client-originated chunk_guard4 or chunk_owner4. A data server that receives such a value from a client MUST reject the operation with NFS4ERR_INVAL.¶

The metadata-server escrow owner is released only by a CHUNK_LOCK from the client selected via CB_CHUNK_REPAIR, carrying CHUNK_LOCK_FLAGS_ADOPT. See Section 25.6.¶

Each metadata-server escrow lock carries an escrow_id4 (Section 24.4) the metadata server chose at CHUNK_ESCROW_INSTALL (Section 25.16) time and presents on every subsequent operation that refers to that lock (CHUNK_ESCROW_RELEASE, CB_CHUNK_REPAIR, CHUNK_LOCK adoption via cla_adopt). When a repair client adopts the metadata-server escrow lock (CHUNK_LOCK with CHUNK_LOCK_FLAGS_ADOPT), the data server MUST retain the adopted escrow_id4 as durable custody metadata on the resulting client-owned lock, for as long as a subsequent revocation-transfer can occur that would convert the client-owned lock back to a metadata-server escrow lock (for example, when the client's lease later expires while the lock is still held). If such a revocation-transfer occurs, the resulting new metadata-server escrow lock MUST be installed with the SAME escrow_id4 that was preserved as custody metadata, not a fresh identity: this preserves the full durable key (file, escrow_id, data-server set) across the entire custody chain so that a metadata server's tuple records can re-associate with the reappeared escrow across an incarnation change (see Section 25.18 discovery).¶

24.6.1. Mandatory-to-Implement Profile

A conforming implementation MUST support at least one proof profile so that two independent implementations can verify each other's takeover proofs at the wire level without prior out-of-band negotiation. This specification designates the following profile as mandatory-to-implement.¶

The mandatory profile carries a signed token whose signer is the incarnation-lease authority (NOT the metadata server itself). The envelope is a COSE_Sign1 structure (Section 4.2 of [RFC9052]) over a deterministic CBOR payload (Section 4.2 of [RFC8949]) with the following normative choices:¶

Signature algorithm (mandatory-to-implement):

Ed25519 (algorithm identifier -8 per [RFC9053]). A conforming signer MUST use Ed25519; a conforming verifier MUST accept Ed25519. Other signature algorithms (e.g., ECDSA-P256 with identifier -7, RSASSA-PSS-SHA256 with identifier -37) MAY be registered as additional profiles per the IANA Considerations.¶

kid header parameter:

OPTIONAL. A deployment with a single trust anchor MAY omit it; a deployment supporting key rotation or multiple trust anchors SHOULD include it so a data server can select the correct verification key. Absent kid, the data server attempts verification against each configured trust anchor and accepts on the first match.¶

Payload map fields:

the signed CBOR payload is a map with integer-keyed fields (per [RFC9053] convention for compact wire size). The mandatory-profile keys are: - 1 = principal (CBOR text string): the metadata-server principal that holds this incarnation. Comparison is byte-for-byte against the RPCSEC_GSS authenticated name of the presenter (Section 5 of [RFC7861]), with no Unicode normalization or case folding; a deployment MUST provision the authority to sign tokens whose principal field is exactly the RPC-authenticated name the data server will observe. - 2 = epoch (CBOR uint): the metadata-server epoch value being claimed. On presentation in CHUNK_ESCROW_TAKEOVER the data server MUST verify that this field equals ceta_new_epoch in the operation arguments; any mismatch is a signature-and-payload check failure at step 4 of "Presentation and Verification" below and returns NFS4ERR_ACCESS. - 3 = scope (CBOR text string): identifier of the data server or data-server set the token is valid for. Comparison is byte-for-byte against the data-server-side value the deployment provisions at trust-anchor setup (see "Trust Anchor Provisioning" below); a data server accepts a token whose scope field exactly matches any scope identifier it has been provisioned to serve, and rejects any other value. The scope namespace is deployment-local -- this specification neither defines a format nor constrains the character set beyond requiring UTF-8 CBOR text. - 4 = issued_at (CBOR tag-1 epoch-based date/time per [RFC8949] Section 3.4.2): the instant the authority signed the token. A conforming issuer MUST encode this as a CBOR unsigned integer number of seconds since the POSIX epoch; a verifier MUST reject a token whose issued_at is not an unsigned integer under tag 1 (fractional or negative values are rejected). - 5 = expires_at (CBOR tag-1 epoch-based date/time per [RFC8949] Section 3.4.2): the instant the token ceases to be admissible. Same encoding constraints as issued_at. - 6 = token_id (CBOR byte string, 16 bytes): a nonce for replay detection.¶

24.6.2. Presentation and Verification

The metadata server presents the profile identifier and the proof bytes together in the takeover arguments (see Section 25.19). The data server MUST evaluate the takeover in a fixed order so that no failure discloses state that a prior check would have denied:¶

1. session replay-cache lookup: retransmission of a prior request in the current session slot returns the cached response;¶

2. presenter authorization: RPCSEC_GSS presenter authentication and credential check (NFS4ERR_ACCESS if the caller lacks metadata-server role);¶

3. profile support: unknown proof_profile_id4 returns NFS4ERR_NOTSUPP;¶

4. proof verification: the profile's signature and payload checks are applied (NFS4ERR_ACCESS on any failure -- bad signature, mismatched principal, mismatched epoch, mismatched scope, token past expires_at, or token_id already in the data server's replay cache);¶

5. epoch compare-and-advance: NFS4ERR_STALE_MDS_EPOCH on mismatch, otherwise the epoch and epoch_expires_at are advanced atomically per Section 25.19.¶

The strict ordering ensures an unauthenticated caller learns nothing about which profiles the data server supports or which epoch it currently holds.¶

Replay-cache scoping (both layers): the NFSv4.1 session replay cache in step 1 is scoped to the session slot and its lifetime is bounded by session liveness. The token_id replay cache in step 4 is scoped to the (proof_profile_id4, verified issuer) pair and MUST be sized and expired coherently with the token's own expires_at, so that a valid token cannot be replayed after its natural expiry and a recently-observed token cannot be inadvertently retired while still admissible. A data server MAY persist the token_id replay cache across restart. Loss of a cache entry (eviction, non-persisted restart) does not permanently defeat lost-response recovery: the byte-identical uncertain-completion recovery path in Section 24.6.3 covers the cache-miss form and returns the same postcondition-equivalent NFS4_OK when the operation had already completed.¶

24.6.3. Uncertain-Completion Recovery for TAKEOVER

CHUNK_ESCROW_TAKEOVER is the recovery path a metadata server uses after an incarnation change; the compare-and-advance semantics make the successful case observable to the data server, but the metadata server MAY lose the response to a successful TAKEOVER (network drop, RPC retransmission timeout, session loss). The bare strict-ordering rules above would rebuff a byte-identical reissue: either the token_id replay-cache check at step 4 rejects with NFS4ERR_ACCESS (the cache still holds the entry) or the epoch compare-and-advance at step 5 rejects with NFS4ERR_STALE_MDS_EPOCH (the ordinary path, because the reissue's ceta_expected_prior_epoch names the pre-advance epoch that the earlier successful TAKEOVER has already moved past). In either case the metadata server has no way to distinguish "the prior TAKEOVER succeeded and the response was lost" from "the proof is invalid."¶

To close that recovery gap, a data server MUST accept a byte-identical CHUNK_ESCROW_TAKEOVER reissue as postcondition-equivalent success when step 4's signature and payload checks succeed (signature verifies against the deployment-provisioned trust anchor; the signed principal, scope, and epoch match; the token is within its expires_at window; all per Section 24.6 "Payload map fields"), with the ordinary step-4 "token_id already in the replay cache" rejection overridden as stated below, and both of the following hold:¶

• the reissue's ceta_new_epoch equals the data server's currently-recorded metadata-server epoch (the takeover the token authorized has already completed); and¶

• the reissue's ceta_expected_prior_epoch is strictly less than the reissue's ceta_new_epoch (the token names a genuine advance, not a no-op).¶

The recovery rule takes two forms distinguished by the state of the token_id replay cache at the data server:¶

Cache-hit form (ordinary lost-response case):

the token_id is present in the token_id replay cache. The data server recognizes the cached byte-identical decision and returns the cached NFS4_OK result; step 5 is not re-executed. Cache-miss form (applies after eviction or non-persisted restart):¶

the token_id is not present in the replay cache. The data server treats the presentation as a fresh byte-identical proof under the two epoch predicates above and does NOT execute step 5 (the epoch is already at the post-advance state).¶

Under these predicates the data server returns NFS4_OK without side effect: the epoch and epoch_expires_at are already at the post-advance state, no state changes, and the second observation is idempotent. The rule is safe against a fresh presentation of an already-used token by a different party -- a different party would not present the same proof bytes without stealing the signer's key material, the token was issued to a specific principal matched by step 4's principal check, expiry admissibility windows for successor principals do not overlap (Section 24.6 "Time-Related Bounds"), and a subsequent successful TAKEOVER by any party advances the current epoch past ceta_new_epoch and disqualifies the reissue on the first predicate above.¶

When any predicate fails the data server returns NFS4ERR_ACCESS or NFS4ERR_STALE_MDS_EPOCH per the ordinary strict-ordering rule and the metadata server MUST obtain a fresh incarnation-lease token from the authority. A fresh token has a new token_id and does not collide with the replay cache; the fresh takeover uses the ordinary advance form.¶

24.6.4. Trust Anchor Provisioning

The verification trust anchor (the public key or key set the authority's signatures verify against) is provisioned at each data server at deployment time. Key distribution mechanisms (X.509 chain, JWK set fetch, raw public key push) are deployment concerns outside the scope of this specification; rotation is likewise deployment-local. The wire format is fully specified so that two independent implementations sharing the same trust anchor can interoperate; the trust-anchor bootstrap itself is not a wire-negotiated step.¶

24.6.5. Time-Related Bounds

The token's issued_at MUST NOT be more than skew_tolerance in the future when the data server evaluates it (deployment-configured; recommend NTP-consistent, roughly 10 s). The token's expires_at is compared strictly: the token becomes inadmissible at the first instant now >= expires_at. No skew tolerance is added to the right edge; the incarnation-lease authority MUST NOT issue a successor token to a different metadata-server principal until at least prior.expires_at + max_ds_skew_tolerance, where max_ds_skew_tolerance is a deployment-configured bound that MUST equal or exceed the largest skew_tolerance any data server in the scope is allowed to use. This ordering ensures that no data server admits the prior holder at any instant after the successor's token becomes admissible.¶

25.2.1. ARGUMENTS

   /// struct CHUNK_COMMIT4args {
   ///     /* CURRENT_FH: file */
   ///     stateid4        cca_stateid;
   ///     offset4         cca_offset;
   ///     count4          cca_count;
   ///     chunk_owner4    cca_chunks<>;
   /// };

25.2.2. RESULTS

   /// struct CHUNK_COMMIT4resok {
   ///     verifier4       ccr_writeverf;
   ///     nfsstat4        ccr_status<>;
   /// };

   /// union CHUNK_COMMIT4res switch (nfsstat4 ccr_status) {
   ///     case NFS4_OK:
   ///         CHUNK_COMMIT4resok   ccr_resok4;
   ///     default:
   ///         void;
   /// };

25.2.3. DESCRIPTION

The CHUNK_COMMIT operation is based upon the NFSv4.1 COMMIT operation (see Section 18.3 of [RFC8881]) and similarly commits previously written data to stable storage on the regular file identified by the current filehandle, with the difference that CHUNK_COMMIT operates on the chunk coordinate system used by Flexible File Version 2 layouts rather than on the byte coordinate system, and that CHUNK_COMMIT advances each named chunk through the chunk state machine from FINALIZED to COMMITTED (Figure 32) rather than acting on a byte range without a state-machine context.¶

The client provides cca_offset and cca_count to bound the chunk range, and cca_chunks to name the specific (chunk_owner4) generations within that range to commit:¶

cca_offset:

starting chunk index in the file (not a byte offset).¶

cca_count:

number of chunks the range covers, starting at cca_offset. When cca_count is zero, the client MUST also supply an empty cca_chunks array, and the data server returns NFS4_OK with an empty ccr_status array. A cca_offset beyond the data server's highest chunk with a non-zero cca_count is not itself an error at the operation level: the data server evaluates each cca_chunks entry per its normal per-entry rules and returns the co-indexed ccr_status.¶

cca_chunks:

an array of chunk_owner4 entries (Figure 54) naming the specific generations to commit. For each entry the data server looks up the chunk index it associated with the complete owner triple when the client wrote it via CHUNK_WRITE or CHUNK_WRITE_REPAIR. That recorded chunk index MUST lie in [cca_offset, cca_offset + cca_count); if the triple does not match any recorded owner association on this data server for this file, or the recorded chunk index lies outside the requested range, the entry is rejected with NFS4ERR_INVAL in the corresponding ccr_status slot. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear as the co_client_id of any cca_chunks entry; see Section 24.2.3 and Section 24.2.4.¶

cca_offset and cca_count are NOT redundant with cca_chunks: the owner triples in cca_chunks name specific generations (which the data server correlates via its recorded owner-to-index association), while cca_offset and cca_count bound the intended chunk-index scope of the operation. A chunk index MAY have multiple persisted generations at the moment CHUNK_COMMIT arrives -- an older COMMITTED generation retained for the rollback invariant (Section 12.8) alongside a newer FINALIZED successor. cca_chunks selects which owner triple to advance to COMMITTED at each affected index; cca_offset and cca_count let the data server reject malformed requests that name generations whose recorded chunk index lies outside the intended commit window.¶

The CHUNK_COMMIT result reports the outcome per chunk in the same order as cca_chunks:¶

ccr_writeverf:

a verifier identifying the data server's incarnation at the time the commit completed. A client compares ccr_writeverf to the cwr_writeverf returned by the prior CHUNK_WRITE to detect a data server restart that lost UNSTABLE4 writes between the write and the commit; on a mismatch the client MUST re-issue the CHUNK_WRITE before any committed bytes are considered durable. ccr_writeverf changes on every data server restart that loses uncommitted state.¶

ccr_status:

per-chunk commit status, one entry per cca_chunks entry, co-indexed. NFS4_OK indicates that the named chunk is COMMITTED on return. Other per-entry failure codes are described in "Interaction with CHUNK_FINALIZE" and "Interaction with a Locked Chunk" below. The top-level CHUNK_COMMIT status is NFS4_OK as long as the data server could evaluate each cca_chunks entry; per-chunk failures are reported in ccr_status rather than by failing the whole operation. The top-level status returns a non-OK code only when the request could not be evaluated at all.¶

Like CHUNK_READ and CHUNK_WRITE, CHUNK_COMMIT carries an explicit layout stateid in cca_stateid. The data server authorizes CHUNK_COMMIT by validating cca_stateid against the file identified by the current filehandle: cca_stateid MUST be the layout stateid the metadata server issued to the caller for the current filehandle, or the special anonymous stateid (see below). Under trusted stateid tight coupling (Section 25.13), the data server rejects CHUNK_COMMIT with NFS4ERR_BAD_STATEID unless cca_stateid is present in the data server's trust table for the current filehandle. The explicit field ensures that a CHUNK_COMMIT in its own standalone compound (typical for recovery and pipelined lifecycle operations) carries the authorization the data server needs without depending on any prior operation's implicit state. Passing the special anonymous stateid is permitted only when the underlying security regime authorizes an unattributed writer (that is, when tight coupling is not in force and the deployment's access-control policy permits it).¶

If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per Section 13.2).¶

Compound A:  SEQUENCE PUTFH CHUNK_WRITE(a)
Compound B:  SEQUENCE PUTFH CHUNK_WRITE(b) CHUNK_FINALIZE(a)
Compound C:  SEQUENCE PUTFH CHUNK_WRITE(c) CHUNK_FINALIZE(b)
                              CHUNK_COMMIT(a)
Compound D:  SEQUENCE PUTFH CHUNK_WRITE(d) CHUNK_FINALIZE(c)
                              CHUNK_COMMIT(b)
...

25.2.4. RESPONSE CODES

NFS4_OK:

every named chunk transitioned to COMMITTED.¶

NFS4ERR_ACCESS:

the layout stateid or credentials are not permitted to commit on this file.¶

NFS4ERR_BADXDR:

arguments could not be decoded.¶

NFS4ERR_BAD_STATEID:

no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See Section 25.¶

NFS4ERR_DELAY:

the data server is temporarily unable to process the request.¶

NFS4ERR_FHEXPIRED:

the current filehandle has expired.¶

NFS4ERR_INVAL:

arguments named chunks outside the file's mirror set or in a non-atomic state.¶

NFS4ERR_IO:

an I/O error occurred while persisting the commit.¶

NFS4ERR_NOTSUPP:

the data server does not implement CHUNK_COMMIT.¶

NFS4ERR_SERVERFAULT:

the data server failed while processing the request.¶

NFS4ERR_STALE:

the current filehandle no longer identifies a valid file.¶

25.3.1. ARGUMENTS

   /// struct CHUNK_ERROR4args {
   ///     /* CURRENT_FH: file */
   ///     stateid4        cea_stateid;
   ///     offset4         cea_offset;
   ///     count4          cea_count;
   ///     nfsstat4        cea_error;
   ///     chunk_owner4    cea_owner;
   /// };

25.3.2. RESULTS

   /// struct CHUNK_ERROR4res {
   ///     nfsstat4        cer_status;
   /// };

25.3.3. DESCRIPTION

CHUNK_ERROR allows a client that has detected corruption or inconsistency in a chunk to report the condition to the data server, so that the data server can mark the affected chunks as errored. Errored chunks are excluded from subsequent CHUNK_READ responses until they are repaired via CHUNK_WRITE_REPAIR and the repair is confirmed via CHUNK_REPAIRED.¶

CHUNK_ERROR has no direct analog in [RFC8881]. The closest parallel is LAYOUTERROR (Section 15.6 of [RFC7862]), which reports layout-level errors to the metadata server. CHUNK_ERROR is the data-path counterpart: it reports a chunk-level integrity finding directly to the data server so that the corrupted chunks are quarantined before the metadata server has had time to coordinate repair. A client SHOULD issue CHUNK_ERROR to the data server holding the bad chunks before issuing LAYOUTERROR to the metadata server.¶

The client provides:¶

cea_stateid:

the layout stateid the metadata server granted for this file. Under trusted stateid tight coupling (Section 25.13), this stateid MUST be in the data server's trust table; otherwise the data server rejects the operation with NFS4ERR_BAD_STATEID.¶

cea_offset:

starting chunk index of the affected range (not a byte offset).¶

cea_count:

number of chunks the affected range covers, starting at cea_offset.¶

cea_error:

the nfsstat4 error code that describes the integrity finding. Typical values include NFS4ERR_PAYLOAD_NOT_ATOMIC (the chunk's persisted checksum or guard did not match the value the client expected), NFS4ERR_IO (the client's CHUNK_READ returned an I/O error from this data server), and NFS4ERR_INVAL (the chunk's chunk_owner4 did not match the expected generation across mirrors). The data server MAY record the supplied error code in operator logs but does not otherwise interpret it; the chunk-level effect (mark errored) is the same for any cea_error value.¶

cea_owner:

the chunk_owner4 (Figure 54) the client read when it observed the error, so the data server can record which owner triple generation was reported as corrupted. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear in cea_owner; see Section 24.2.3 and Section 24.2.4.¶

CHUNK_ERROR returns a single top-level status in cer_status; there is no per-chunk status array because the data server either accepts the report for the whole range or returns a top-level error. Once a CHUNK_ERROR has been accepted, the affected chunks transition into the errored state described in Section 12.6; subsequent CHUNK_READ operations against those chunks return NFS4ERR_PAYLOAD_NOT_ATOMIC in the per-chunk cr_status slot until a successful CHUNK_REPAIRED sequence clears the errored flag.¶

If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per Section 13.2).¶

25.3.4. RESPONSE CODES

NFS4_OK:

the client's chunk error report has been recorded.¶

NFS4ERR_ACCESS:

the layout stateid or credentials are not permitted to report errors on this file.¶

NFS4ERR_BADXDR:

arguments could not be decoded.¶

NFS4ERR_BAD_STATEID:

no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See Section 25.¶

NFS4ERR_INVAL:

the reported chunk range or error code was not recognized.¶

NFS4ERR_NOTSUPP:

the data server does not implement CHUNK_ERROR.¶

NFS4ERR_SERVERFAULT:

the data server failed while processing the request.¶

25.4.1. ARGUMENTS

   /// struct CHUNK_FINALIZE4args {
   ///     /* CURRENT_FH: file */
   ///     stateid4        cfa_stateid;
   ///     offset4         cfa_offset;
   ///     count4          cfa_count;
   ///     chunk_owner4    cfa_chunks<>;
   /// };

25.4.2. RESULTS

   /// struct CHUNK_FINALIZE4resok {
   ///     verifier4       cfr_writeverf;
   ///     nfsstat4        cfr_status<>;
   /// };

   /// union CHUNK_FINALIZE4res switch (nfsstat4 cfr_status) {
   ///     case NFS4_OK:
   ///         CHUNK_FINALIZE4resok   cfr_resok4;
   ///     default:
   ///         void;
   /// };

25.4.3. DESCRIPTION

CHUNK_FINALIZE transitions chunks from the PENDING state (set by CHUNK_WRITE, see Section 25.11) to the FINALIZED state in the chunk state machine (Figure 32). A FINALIZED chunk is visible on the owning stateid for reads (Section 12.8) and is eligible for CHUNK_COMMIT; the FINALIZED transition is the writer's signal that it will issue no further CHUNK_WRITEs for the named owner triple generation of each chunk.¶

CHUNK_FINALIZE has no direct analog in [RFC8881]: the COMMIT operation in Section 18.3 of [RFC8881] combines the "no more writes" signal and the "make durable and globally visible" step into one operation; the Flexible File Version 2 chunk lifecycle separates them so a writer in multiple-writer mode can validate the per-chunk acceptance status reported by CHUNK_WRITE before committing any chunk to durable storage (see "Pipelining Considerations" in Section 25.2).¶

The client provides cfa_offset and cfa_count to bound the chunk range, and cfa_chunks to name the specific (chunk_owner4) generations within that range to finalize:¶

cfa_offset:

starting chunk index in the file (not a byte offset).¶

cfa_count:

number of chunks the range covers, starting at cfa_offset. When cfa_count is zero, the client MUST also supply an empty cfa_chunks array, and the data server returns NFS4_OK with an empty cfr_status array. A cfa_offset beyond the data server's highest chunk with a non-zero cfa_count is not itself an error at the operation level: the data server evaluates each cfa_chunks entry per its normal per-entry rules and returns the co-indexed cfr_status.¶

cfa_chunks:

an array of chunk_owner4 entries (Figure 54) naming the specific generations to finalize. For each entry the data server looks up the chunk index it associated with the complete owner triple when the client wrote it via CHUNK_WRITE or CHUNK_WRITE_REPAIR. That recorded chunk index MUST lie in [cfa_offset, cfa_offset + cfa_count); if the triple does not match any recorded owner association on this data server for this file, or the recorded chunk index lies outside the requested range, the entry is rejected with NFS4ERR_INVAL in the corresponding cfr_status slot. The reserved sentinels CHUNK_GUARD_CLIENT_ID_NONE and CHUNK_GUARD_CLIENT_ID_MDS MUST NOT appear as the co_client_id of any cfa_chunks entry; see Section 24.2.3 and Section 24.2.4.¶

The CHUNK_FINALIZE result reports the outcome per chunk in the same order as cfa_chunks:¶

cfr_writeverf:

a verifier identifying the data server's incarnation at the time the finalization completed. Semantics match cwr_writeverf in CHUNK_WRITE: a client that observes a different writeverf on a subsequent CHUNK_COMMIT MUST re-issue the CHUNK_WRITE before treating any of the finalized chunks as durable.¶

cfr_status:

per-chunk finalization status, one entry per cfa_chunks entry, co-indexed. NFS4_OK indicates that the named chunk is FINALIZED on return. Other per-entry failure cases:¶

• NFS4ERR_INVAL -- the named generation is not in the PENDING state at its recorded chunk index (the chunk is EMPTY, FINALIZED at a different generation, or COMMITTED), or the triple does not match any recorded owner association, or the recorded chunk index lies outside [cfa_offset, cfa_offset + cfa_count).¶

• NFS4ERR_CHUNK_GUARDED -- the chunk is PENDING but the current PENDING owner is a different (co_cohort_id, co_client_id) than the one named in the cfa_chunks entry. A client that sees this has lost a race with another writer; see Section 24.2.¶

• NFS4ERR_CHUNK_LOCKED -- the chunk is locked by a CHUNK_LOCK held by a different stateid; the finalize is rejected.¶

The top-level CHUNK_FINALIZE status is NFS4_OK as long as the data server could evaluate each cfa_chunks entry; per-chunk failures are reported in cfr_status rather than by failing the whole operation. The top-level status returns a non-OK code only when the request could not be evaluated at all.¶

CHUNK_FINALIZE serves as the CRC validation checkpoint for the chunk lifecycle. The data server SHOULD have validated each chunk's checksum against the value supplied in cwa_checksums at CHUNK_WRITE time; the FINALIZE transition persists the chunk metadata (CRC, owner, state) to stable storage so it survives a data server restart. An implementation MAY defer some metadata persistence to CHUNK_COMMIT instead of CHUNK_FINALIZE; in that case the FINALIZED state is recovered by replay of the data server's local journal on restart.¶

A chunk that has been FINALIZED but not yet COMMITTED MAY be rolled back via CHUNK_ROLLBACK, which returns the chunk to the EMPTY state (or to the prior COMMITTED generation, if one exists).¶

Like CHUNK_COMMIT, CHUNK_FINALIZE carries an explicit layout stateid in cfa_stateid. The data server authorizes CHUNK_FINALIZE by validating cfa_stateid against the file identified by the current filehandle: cfa_stateid MUST be the layout stateid the metadata server issued to the caller for the current filehandle, or the special anonymous stateid (see below). Under trusted stateid tight coupling (Section 25.13), the data server rejects CHUNK_FINALIZE with NFS4ERR_BAD_STATEID unless cfa_stateid is present in the data server's trust table for the current filehandle. The explicit field ensures that a CHUNK_FINALIZE in its own standalone compound (typical for pipelined and recovery cases) carries the authorization the data server needs. Passing the special anonymous stateid is permitted only when the underlying security regime authorizes an unattributed writer.¶

If the current filehandle is not an ordinary file, an error MUST be returned (NFS4ERR_ISDIR / NFS4ERR_SYMLINK / NFS4ERR_WRONG_TYPE for non-regular; NFS4ERR_NOTSUPP if the file is a regular file but not a chunked data file, per Section 13.2).¶

25.4.4. RESPONSE CODES

NFS4_OK:

every named chunk transitioned from PENDING to FINALIZED.¶

NFS4ERR_ACCESS:

the layout stateid or credentials are not permitted to finalize on this file.¶

NFS4ERR_BADXDR:

arguments could not be decoded.¶

NFS4ERR_BAD_STATEID:

no active layout stateid for this file (or, in trusted stateid tight coupling, the stateid is not in the trust table). See Section 25.¶

NFS4ERR_DELAY:

the data server is temporarily unable to process the request.¶

NFS4ERR_FHEXPIRED:

the current filehandle has expired.¶

NFS4ERR_INVAL:

arguments named chunks not in PENDING or outside the file's mirror set.¶

NFS4ERR_IO:

an I/O error occurred while persisting the transition.¶

NFS4ERR_NOTSUPP:

the data server does not implement CHUNK_FINALIZE.¶

NFS4ERR_SERVERFAULT:

the data server failed while processing the request.¶

NFS4ERR_STALE:

the current filehandle no longer identifies a valid file.¶

25.5.1. ARGUMENTS

   /// struct CHUNK_HEADER_READ4args {
   ///     /* CURRENT_FH: file */
   ///     stateid4    chra_stateid;
   ///     offset4     chra_offset;
   ///     count4      chra_count;
   /// };

25.5.2. RESULTS

   /// /* Upper bound on both chra_count and each of the
   ///  * four co-indexed response arrays.  Bounds argument
   ///  * range width and response-array width together so
   ///  * that a caller cannot request an unbounded scan
   ///  * and a data server cannot construct an unbounded
   ///  * response.
   ///  */
   /// const CHUNK_HEADER_READ_MAX4 = 1024;
   ///
   /// struct retained_predecessor4 {
   ///     chunk_owner4  rp_owner;
   /// };
   ///
   /// enum retained_generation_disposition4 {
   ///     RETAINED_GENERATION_DISPOSITION_ABSENT   = 0,
   ///     RETAINED_GENERATION_DISPOSITION_PRESENT  = 1,
   ///     RETAINED_GENERATION_DISPOSITION_ERRORED  = 2
   /// };
   ///
   /// union optional_retained4
   ///     switch (retained_generation_disposition4
   ///             disposition) {
   /// case RETAINED_GENERATION_DISPOSITION_ABSENT:
   ///     void;
   /// case RETAINED_GENERATION_DISPOSITION_PRESENT:
   ///     retained_predecessor4  restorable;
   /// case RETAINED_GENERATION_DISPOSITION_ERRORED:
   ///     retained_predecessor4  errored;
   /// };
   ///
   /// struct CHUNK_HEADER_READ4resok {
   ///     bool                chrr_eof;
   ///     nfsstat4
   ///         chrr_status<CHUNK_HEADER_READ_MAX4>;
   ///     chunk_state_flags4
   ///         chrr_locked<CHUNK_HEADER_READ_MAX4>;
   ///     chunk_owner4
   ///         chrr_owners<CHUNK_HEADER_READ_MAX4>;
   ///     chunk_guard4
   ///         chrr_guards<CHUNK_HEADER_READ_MAX4>;
   ///     optional_retained4
   ///         chrr_predecessors<CHUNK_HEADER_READ_MAX4>;
   /// };

   /// union CHUNK_HEADER_READ4res switch (nfsstat4 chrr_status) {
   ///     case NFS4_OK:
   ///         CHUNK_HEADER_READ4resok     chrr_resok4;
   ///     default:
   ///         void;
   /// };

25.5.3. DESCRIPTION

CHUNK_HEADER_READ returns the per-chunk metadata (chunk_owner4, chunk_guard4, lock state, and per-chunk status) for a range of chunks in the target data file without returning the chunk payloads. The operation enables clients and repair coordinators to inspect chunk lifecycle and ownership cheaply, without the data-transfer cost of CHUNK_READ. CHUNK_HEADER_READ has no direct analog in [RFC8881]; it is the chunk-protocol counterpart of a stat-like fast probe and exists because chunks are first-class state-bearing objects whose ownership, lock state, and lifecycle status are not recoverable from a byte-offset query.¶

The client provides:¶

chra_stateid:

the layout stateid the metadata server granted for this file. Under trusted stateid tight coupling (Section 25.13), this stateid MUST be in the data server's trust table; otherwise the data server rejects the operation with NFS4ERR_BAD_STATEID.¶

chra_offset:

starting chunk index of the range to inspect (not a byte offset).¶

chra_count:

number of chunks the inspection range covers, starting at chra_offset. chra_count MUST NOT exceed CHUNK_HEADER_READ_MAX4; a request that exceeds the bound is rejected with NFS4ERR_INVAL. The bound protects the data server against arbitrarily large response construction.¶

The CHUNK_HEADER_READ result returns five co-indexed arrays, one entry per chunk in the requested range in chunk-offset order from chra_offset:¶

chrr_eof:

TRUE if the requested range extended at or past the data server's last chunk for this file. Same per-data-server semantics as crr_eof in CHUNK_READ.¶

chrr_status:

per-chunk lifecycle state encoded as an nfsstat4 (see "Per-Chunk Status Encoding" below).¶

chrr_locked:

per-chunk chunk_state_flags4 (Section 24.5). CHUNK_STATE_FLAGS_LOCKED set indicates the chunk currently has a CHUNK_LOCK held by some chunk_owner4. Lock state is reported orthogonally to chrr_status so that a locked chunk still surfaces its lifecycle state and chunk_owner4 to the inspector.¶

chrr_owners:

per-chunk chunk_owner4 (Figure 54). For a chunk whose chrr_status is NFS4_OK the field is the COMMITTED generation's owner. For NFS4ERR_PAYLOAD_NOT_ATOMIC the field is the writer of the in-progress (PENDING or FINALIZED) generation. For NFS4ERR_NOENT (EMPTY chunk) the chunk_owner4 is unspecified.¶

chrr_guards:

per-chunk chunk_guard4 (Figure 53) -- the (cg_gen_id, cg_client_id) pair the data server holds as the chunk's current guard at CHUNK_HEADER_READ time. A caller that intends to update the chunk in multiple writer mode uses the corresponding chrr_guards entry as the expected prior value for the guard CAS (Section 11.2.6.4), avoiding the payload cost of CHUNK_READ. For a chunk whose chrr_status is NFS4ERR_NOENT (EMPTY) the chunk_guard4 is the all-zeros pair (cg_gen_id = 0 and cg_client_id = CHUNK_GUARD_CLIENT_ID_NONE, see Section 24.2.3). chrr_guards is a read-time observation; a concurrent writer MAY advance the guard between the CHUNK_HEADER_READ response and the subsequent CHUNK_WRITE, in which case the CAS returns NFS4ERR_CHUNK_GUARDED and the caller re-observes the guard per the rollback-and-retry flow in Section 24.2.¶

chrr_predecessors:

per-chunk immediate-predecessor disposition, one optional_retained4 entry per chunk in the returned range, co-indexed with chrr_status, chrr_locked, and chrr_owners. Each entry names the read-time state of the single most recent retained predecessor of the current generation the data server holds for that chunk index under the retention scope rule (Section 12.9); the entry's discriminant is exactly one of:¶

RETAINED_GENERATION_DISPOSITION_ABSENT:

the data server holds no retained predecessor at that index (either the current chrr_owners generation is the only one, or the chunk is EMPTY). The arm carries no owner triple.¶

RETAINED_GENERATION_DISPOSITION_PRESENT:

the data server retains an immediate predecessor whose owner triple is carried in the restorable arm and whose payload is in the AVAILABLE read-time state (Section 12.7). A CHUNK_ROLLBACK naming this owner triple satisfies "Rollback of COMMITTED Chunks" case (a) (Section 25.9).¶

RETAINED_GENERATION_DISPOSITION_ERRORED:

the data server retains the immediate predecessor's owner triple (in the errored arm) but its payload is in the ERRORED read-time state and cannot be restored by CHUNK_ROLLBACK. A CHUNK_ROLLBACK naming this owner triple MUST return NFS4ERR_NO_PREDECESSOR (Section 21.1.7); the client falls back to best-effort reconstruction via CHUNK_WRITE_REPAIR with an authoritative source of its own choosing. The owner triple is disclosed so a caller can coordinate reconstruction from surviving shards.¶

The list is informational and MAY change between successive CHUNK_HEADER_READ calls -- the data server MAY release a predecessor between calls under the retention scope rule. A caller that observes a PRESENT disposition and issues CHUNK_ROLLBACK before that release remains guaranteed by the composed rollback guarantee (Section 27) when it holds a qualifying lock or escrow; without such a lock, the retention scope MAY release the predecessor at any time and a subsequent CHUNK_ROLLBACK MAY return NFS4ERR_NO_PREDECESSOR even though a previous CHUNK_HEADER_READ observed PRESENT.¶

Cardinality and short responses:

All five response arrays are the same length; the data server MUST NOT sparsify or truncate one array independently of the others.¶

The response array length N MAY be smaller than chra_count when the requested range extends past the data server's last chunk (chrr_eof = TRUE, N = the number of chunks the data server holds within the requested range) or when the fully-populated response would exceed the session-negotiated ca_maxresponsesize (Section 18.36.3 of [RFC8881]). In the response-size case the data server returns a short response with chrr_eof = FALSE containing as many entries N as fit under ca_maxresponsesize minus COMPOUND/RPC overhead; the client resumes at chra_offset + N. If even the minimum useful response (a single entry) will not fit, the data server returns NFS4ERR_REP_TOO_BIG per [RFC8881]; the client MUST NOT retry with a smaller chra_count and instead uses a session or COMPOUND with more available response budget.¶

The operation has several uses:¶

Whole-file repair scan:

A repair actor selected via CB_CHUNK_REPAIR walks the affected chunk range and uses the per-chunk chunk_owner4 returned by each mirror's data server to identify which chunks carry an atomic stripe (all k data shards share the same chunk_guard4) and which require reconstruction. CHUNK_HEADER_READ is the discovery primitive that drives the per-chunk decisions described in Section 11.2.6.5; without it, a repair client would have to issue CHUNK_READ to retrieve the full payload of every chunk merely to inspect its guard.¶

Client-side recovery from partial writes:

After a network disruption or client restart, a writer that holds the file's layout MAY issue CHUNK_HEADER_READ to learn which of its prior CHUNK_WRITEs reached the data server. Chunks whose chunk_owner4 reports the writer's own (co_cohort_id, co_client_id) pair are PENDING or FINALIZED and recoverable; chunks absent from the response or carrying another writer's owner are not. The writer can then re-issue CHUNK_WRITE for the missing chunks or CHUNK_ROLLBACK for the abandoned ones without reading payloads it has already committed locally.¶

Read-side atomicity check:

Before issuing a multi-chunk CHUNK_READ in multiple-writer mode, a client MAY issue CHUNK_HEADER_READ to verify that the chunks in the target range share a common (co_cohort_id, co_client_id) pair in chrr_owners (the cohort-atomicity property in Section 12.8) and MAY additionally inspect chrr_guards as a cheaper generation-level corroboration. If the cohort pairs diverge, the client knows the read will not be atomic and can wait for a writer to commit, retry, or report NFS4ERR_PAYLOAD_NOT_ATOMIC via LAYOUTERROR. This is a hint rather than a guarantee: a concurrent writer MAY advance a chunk's state between the CHUNK_HEADER_READ response and the subsequent CHUNK_READ.¶

Predecessor-guided rollback discovery:

A caller preparing a CHUNK_ROLLBACK against a COMMITTED chunk inspects the corresponding chrr_predecessors entry to decide whether CHUNK_ROLLBACK will succeed.¶

PRESENT:

name the disclosed owner triple in the cra_chunks entry of the subsequent CHUNK_ROLLBACK. "Rollback of COMMITTED Chunks" case (a) will succeed subject to the composed rollback guarantee's continuous-custody condition (Section 27).¶

ERRORED:

do NOT issue CHUNK_ROLLBACK against the disclosed owner triple. The data server MUST return NFS4ERR_NO_PREDECESSOR for that owner; use CHUNK_WRITE_REPAIR directly with a reconstructed authoritative source. The disclosed owner triple lets the caller coordinate reconstruction from other sources.¶

ABSENT:

no restorable predecessor exists. Skip CHUNK_ROLLBACK; use CHUNK_WRITE_REPAIR if reconstruction is possible, or defer to a guaranteed-pinning mechanism when the caller requires the original owner triple be preserved.¶

As with the atomicity check, a subsequent lifecycle event MAY change a chunk's disposition between the CHUNK_HEADER_READ response and the CHUNK_ROLLBACK (a PRESENT observation MAY become ABSENT if the retention scope releases the predecessor and the caller does not hold a qualifying lock or escrow).¶

Lock probe before write:

A client MAY issue CHUNK_HEADER_READ and inspect the chrr_locked array to discover whether any chunk in the target range is currently held by a CHUNK_LOCK before attempting CHUNK_WRITE, avoiding the round-trip cost of receiving NFS4ERR_CHUNK_LOCKED. As above, this is a hint; a lock MAY be acquired between the header read and the write.¶