PipeStream: A Recursive Entity Streaming Protocol for Distributed Processing over QUIC

1.1. Problem Statement

Distributed processing pipelines face significant challenges when handling large, complex work units that require multiple stages of transformation, analysis, and enrichment. Traditional batch processing approaches require entire inputs to be loaded into memory, processed sequentially, and transmitted in their entirety between processing stages. This methodology introduces substantial latency, excessive memory consumption, and poor utilization of distributed computing resources.¶

Modern distributed processing workflows increasingly demand the ability to:¶

• Process inputs incrementally as data becomes available¶

• Distribute processing load across heterogeneous worker nodes¶

• Maintain consistency guarantees across parallel processing paths¶

• Handle inputs of arbitrary size without memory constraints¶

• Support recursive decomposition where constituent parts may themselves be decomposed¶

• Scale from single work units to collections of millions of inputs¶

Current approaches based on batch processing and store-and-forward architectures are inefficient for large inputs and fail to exploit the inherent parallelism available in distributed processing environments. Furthermore, existing streaming protocols do not provide the consistency semantics required for hierarchical processing where the integrity of the rehydrated output depends on the successful processing of all constituent parts.¶

1.2. Applicability

Although PipeStream was originally motivated by distributed document processing pipelines, its recursive scatter-gather design is domain-neutral. The protocol is applicable to any workload that can be modeled as hierarchical decomposition of a root entity into sub-entities, parallel processing of those sub-entities, and deterministic reassembly of results. Example domains include:¶

• Distributed document processing and content enrichment pipelines¶

• Hierarchical video transcoding and rendering (scene decomposition)¶

• Federated computation pipelines such as distributed machine learning¶

• Genomic sequencing and assembly workflows¶

Throughout this document, "document" is used as the canonical example of a root entity, but all protocol mechanisms apply equally to any decomposable work unit.¶

1.3. PipeStream Overview

PipeStream addresses these challenges by defining a streaming protocol that enables incremental processing with strong consistency guarantees. The protocol is built upon QUIC [RFC9000] transport, leveraging its native support for multiplexed streams, low-latency connection establishment, and reliable delivery semantics.¶

The fundamental innovation of PipeStream is its treatment of inputs as recursive compositions of entities. A root entity MAY be decomposed into multiple sub-entities, each of which MAY itself be further decomposed, creating a tree structure of processing tasks. This recursive decomposition enables fine-grained parallelism while the protocol's control stream mechanism ensures that all branches of the decomposition tree are tracked and synchronized.¶

PipeStream employs a dual-stream design:¶

1. Data Stream: Carries entity payloads through the processing pipeline. Entities flow through this stream with minimal buffering, enabling low-latency incremental processing.¶

2. Control Stream: Carries control information tracking the status of entity decomposition and rehydration. The control stream ensures that all parts of a dehydrated entity are accounted for before rehydration proceeds.¶

1.4. Design Philosophy

PipeStream implements a recursive scatter-gather pattern [scatter-gather] over QUIC streams. An input entity is "dehydrated" (scattered) at the source into constituent sub-entities, these sub-entities are transmitted and processed in parallel across distributed pipeline stages, and finally the results are "rehydrated" (gathered) at the destination to reconstitute the complete processed output. The checkpoint blocking mechanism (Section 9.3) provides barrier synchronization semantics analogous to the barrier pattern in parallel computing.¶

This approach provides several advantages:¶

• Incremental Processing: Processing nodes MAY begin work on early entities before the complete input has been transmitted.¶

• Parallelism: Independent entities MAY be processed concurrently across multiple worker nodes.¶

• Memory Efficiency: No single node is required to hold the complete input in memory.¶

• Fault Isolation: Failures in processing individual entities can be detected, reported, and potentially retried without affecting other entities.¶

• Consistency: The checkpoint blocking mechanism ensures that rehydration operations proceed only when all constituent parts have been successfully processed.¶

1.5. Protocol Layering

PipeStream is organized into three protocol layers to accommodate varying deployment requirements:¶

Implementations MUST support Layer 0. Support for Layers 1 and 2 is OPTIONAL and negotiated during connection establishment.¶

1.6. Scope

This document specifies the PipeStream protocol including message formats, state machines, error handling, and the interaction between data and control streams. The document defines the four standard data layers but does not mandate specific processing semantics, which are left to application-layer specifications.¶

2.1. Protocol Entities

Entity

The fundamental unit of data flowing through a PipeStream pipeline. An Entity represents either a complete document or a constituent part of a decomposed document. Each Entity possesses a unique identifier within its processing scope and carries payload data in one of the four defined Layer formats. Entities are immutable once created; transformations produce new Entities rather than modifying existing ones.¶

Document

A logical unit of content or work submitted to a PipeStream pipeline for processing. A Document enters the pipeline as a single root Entity and MAY be decomposed into multiple Entities during processing. The Document is considered complete when its root Entity (or the rehydrated result of its decomposition) exits the pipeline.¶

Scope

A hierarchical namespace for Entity IDs. Each scope maintains its own Entity ID space, cursor, and Assembly Manifest. Scopes enable collections to contain documents, documents to contain parts, and parts to contain jobs, each with independent ID management. (Protocol Layer 1)¶

2.2. Dehydration and Rehydration

Scatter-Gather

The distributed processing pattern implemented by PipeStream. A single input is "scattered" (dehydrated) into multiple parts for parallel processing, and the results are "gathered" (rehydrated) back into a single output. PipeStream extends classical scatter-gather with recursive nesting: any scattered part may itself be scattered further.¶

Dehydrate (Scatter)

The operation of decomposing a document or Entity into multiple constituent Entities for parallel or distributed processing. When an Entity is dehydrated, the originating node MUST create an Assembly Manifest entry recording the identifiers of all resulting sub-entities. The dehydration operation is recursive; a sub-entity produced by dehydration MAY itself be dehydrated, creating a tree of decomposition. Dehydration transitions data from a solid state (a single stored record) to a fluid state (multiple in-flight entities).¶

Rehydrate (Gather)

The operation of reassembling multiple Entities back into a single composite Entity or Document. A rehydrate operation MUST NOT proceed until all constituent Entities listed in the corresponding Assembly Manifest entry have been received and processed (or handled according to the Completion Policy). Rehydration transitions data from a fluid state back to a solid state.¶

Solid State

A document or Entity that exists as a complete, stored record -- either at rest in storage or as a single root Entity entering or exiting a pipeline. Contrast with "fluid state".¶

Fluid State

A document that has been decomposed into multiple in-flight Entities being processed in parallel across distributed nodes. A document is in the fluid state between dehydration and rehydration. Contrast with "solid state".¶

2.3. Consistency Mechanisms

Checkpoint

A synchronization point in the processing pipeline where all in-flight Entities MUST reach a consistent state before processing may continue. A checkpoint is considered "satisfied" when all Assembly Manifest entries created before the checkpoint have been resolved.¶

Barrier

A synchronization point scoped to a specific subtree. Unlike checkpoints which are global, barriers block only entities dependent on a specific parent's descendants. (Protocol Layer 1)¶

Control Stream

The control stream that tracks Entity completion status throughout the processing pipeline. The Control Stream is transmitted on a dedicated QUIC stream parallel to the data streams.¶

Assembly Manifest

A data structure within the Control Stream that tracks the relationship between a composite Entity and its constituent sub-entities produced by dehydration.¶

Cursor

A pointer to the lowest unresolved Entity ID within a scope. Entity IDs behind the cursor are considered resolved and MAY be recycled. The cursor enables efficient ID space management without global coordination.¶

2.4. Resilience Mechanisms (Protocol Layer 2)

Yield

A temporary pause in Entity processing, typically due to external dependencies (API calls, rate limiting, human approval). A yielded Entity carries a continuation token enabling resumption without reprocessing.¶

Claim Check

A detached reference to a deferred Entity that can be queried or resumed independently, potentially in a different session. Claim checks enable asynchronous processing patterns and retry queues.¶

Completion Policy

A configuration specifying how to handle partial failures during dehydration. Policies include STRICT (all must succeed), LENIENT (continue with partial results), BEST_EFFORT (complete with whatever succeeds), and QUORUM (require minimum success ratio).¶

2.5. Data Representation

Data Layer

One of four defined representations for Entity payload data:¶

1. BlobBag: Raw binary data with minimal metadata¶

2. SemanticLayer: Annotated content with structural and semantic metadata¶

3. ParsedData: Structured information extracted from document content¶

4. CustomEntity: Application-specific extension Layer¶

2.6. Additional Terms

Pipeline

A configured sequence of processing stages through which Entities flow.¶

Processor

A node in the mesh that performs operations on entities (e.g., transformation, dehydration, or rehydration).¶

Sink

A terminal stage in a pipeline where rehydrated documents are persisted or delivered to an external system.¶

Stage

A single processing step within a Pipeline.¶

Scope Digest

A cryptographic summary (Merkle root) of all Entity statuses within a completed scope, propagated to parent scopes for efficient verification. (Protocol Layer 1)¶

3.1. Layer 0: Core Protocol

Layer 0 provides the fundamental streaming capabilities:¶

• Unified Control Frame (UCF) header (1-octet type)¶

• Status frame (16-octet base bit-packed frame)¶

• Entity frame (header + payload)¶

• Status codes: PENDING, PROCESSING, COMPLETE, FAILED, CHECKPOINT¶

• Assembly Manifest for parent-child tracking¶

• Cursor-based Entity ID recycling¶

• Single-level dehydrate/rehydrate¶

• Checkpoint blocking¶

All implementations MUST support Layer 0.¶

3.2. Layer 1: Recursive Extension

Layer 1 adds hierarchical processing capabilities:¶

• Scoped Entity ID namespaces (collection -> document -> part -> job)¶

• Explicit Depth tracking in status frames¶

• SCOPE_DIGEST for Merkle-based subtree completion¶

• BARRIER for subtree-scoped synchronization¶

• Nested dehydration with depth tracking¶

Layer 1 is OPTIONAL. Implementations advertise Layer 1 support during capability negotiation.¶

3.3. Layer 2: Resilience Extension

Layer 2 adds fault tolerance and async processing:¶

• YIELDED status with continuation tokens¶

• DEFERRED status with claim checks¶

• RETRYING, SKIPPED, ABANDONED statuses¶

• Completion policies (STRICT, LENIENT, BEST_EFFORT, QUORUM)¶

• Claim check extensions and deferred processing tokens¶

• Stopping point validation¶

Layer 2 is OPTIONAL and requires Layer 1. Implementations advertise Layer 2 support during capability negotiation.¶

3.4. Capability Negotiation

PipeStream uses a two-tier negotiation model. The ALPN identifier (Section 11.1) identifies the base PipeStream transport mapping, while the capabilities structure handles dynamic resource limits and optional layer support that may vary based on endpoint configuration or real-time load.¶

During CONNECT, endpoints exchange supported capabilities using the capabilities structure. This message MUST be encoded using the default CBOR format for both the client's initiation and the server's response (Section 3.5).¶

serialization-format = &(
  cbor: 0,                         ; Default (IETF native)
  protobuf: 1,
)

capabilities = {
  layer0-core: bool,               ; Always true
  layer1-recursive: bool,          ; Scoped IDs, digests
  layer2-resilience: bool,         ; Yield, claim checks
  ? max-scope-depth: uint .le 7,   ; Default: 7 (8 levels, 0-7)
  ? max-entities-per-scope: uint,  ; Default: 4,294,967,294
  ? max-window-size: uint,         ; Default: 2,147,483,648 (2^31)
                                   ; (Max in-flight entities)
  ? serialization-format: serialization-format, ; Default: CBOR
  ? keepalive-timeout-ms: uint,    ; Default: 30000 (30s)
}

Peers negotiate down to common capabilities. If Layer 2 is requested but Layer 1 is not supported, Layer 2 MUST be disabled.¶

4.1. Design Goals

4.2. Architecture Summary

PipeStream uses a dual-stream architecture within a single QUIC connection between a Client (Producer) and Server (Consumer):¶

4.3. Connection Lifecycle

A PipeStream connection follows this lifecycle:¶

1. Establishment: Client initiates QUIC connection with ALPN identifier "pipestream/1"¶

2. Capability Exchange: Client and server exchange supported protocol layers and limits¶

3. Control Stream Initialization: Client opens Stream 0 as bidirectional Control Stream¶

4. Entity Streaming: Entities are transmitted per Sections 5 and 6¶

5. Termination: Connection closes via GOAWAY-initiated graceful shutdown (Section 6.5) or QUIC CONNECTION_CLOSE¶

5.1. Control Stream (Stream 0)

The Control Stream provides the control plane for PipeStream operations.¶

5.2. Entity Streams (Streams 2+)

Entity Streams carry the actual entity data.¶

5.3. Transport Error Mapping

PipeStream error signaling on Stream 0 and QUIC transport signals are complementary. Endpoints SHOULD bridge them so peers receive both transport-level and protocol-level context. An implementation MAY omit bridging when operating as a simple pass-through proxy that does not inspect entity status.¶

1. If an Entity Stream is aborted with RESET_STREAM or STOP_SENDING, the endpoint SHOULD emit a corresponding terminal status (FAILED, ABANDONED, or policy-driven equivalent) for that entity on Stream 0. The endpoint MAY omit the status frame only if the connection itself is being closed immediately.¶

2. If PipeStream determines a terminal entity error first (for example, checksum failure or invalid frame), the endpoint SHOULD abort the affected Entity Stream with an appropriate QUIC error and emit the corresponding PipeStream status/error context on Stream 0. Aborting the stream MAY be deferred if the entity payload is still needed for diagnostic purposes.¶

3. If Stream 0 is reset or becomes unusable, endpoints SHOULD treat this as a control-plane failure and close the connection with PIPESTREAM_CONTROL_RESET (0x03). An endpoint that can recover control-plane state through an application-layer mechanism MAY attempt reconnection before closing.¶

4. On QUIC connection termination (CONNECTION_CLOSE), entities without a previously observed terminal status MUST be treated as failed by local policy.¶

6.1. Control Stream Framing (Stream 0)

To support mixed content (bit-packed frames and serialized messages) on the Control Stream, PipeStream uses a Unified Control Frame (UCF) header.¶

6.2. Status Frames (Layer 0)

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Type (0x50)  |Ver(4) |Stat(4)|E|C|D(3) |   Flags (11 bits)   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Entity ID (32 bits)                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Scope ID (32 bits)                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Reserved (32 bits)                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  New Cursor Value (32 bits)                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6.3. Scope Digest Frame (0x54)

When Protocol Layer 1 is negotiated, a scope completion is summarized:¶

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Type (0x54)  |  Flags (8)      |        Reserved (16)        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Scope ID (32 bits)                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                   Entities Processed (64 bits)                |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                   Entities Succeeded (64 bits)                |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                    Entities Failed (64 bits)                  |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                    Entities Deferred (64 bits)                |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                    Merkle Root (256 bits)                     |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Flags (8 bits):

Reserved for future use. MUST be zero when sent and MUST be ignored by receivers.¶

Scope ID (32 bits):

Identifier of the scope being summarized.¶

Entities Processed (64 bits):

The total number of entities that were processed within the scope.¶

Entities Succeeded (64 bits):

The number of entities that reached a terminal success state.¶

Entities Failed (64 bits):

The number of entities that reached a terminal failure state.¶

Entities Deferred (64 bits):

The number of entities that were deferred via claim checks.¶

Merkle Root (256 bits):

The SHA-256 Merkle root covering all entity statuses in the scope (see Section 9.4).¶

The SCOPE_DIGEST frame is 72 octets total. The Scope ID MUST match the 32-bit identifier defined in Section 6.2.1.¶

6.4. Barrier Frame (0x55)

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Type (0x55)  |S|  Reserved (23 bits)                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Scope ID (32 bits)                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Parent Entity ID (32 bits)                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

S (1 bit):

Status (0 = waiting, 1 = released).¶

Reserved (23 bits):

Reserved for future use. MUST be zero when sent and MUST be ignored by receivers.¶

Scope ID (32 bits):

Identifier for the scope to which this barrier applies.¶

Parent Entity ID (32 bits):

The identifier of the parent entity whose sub-tree is blocked by this barrier.¶

6.5. GOAWAY Frame (0x56)

The GOAWAY frame signals that the sender will not accept new entities beyond a specified Entity ID. It enables graceful shutdown: in-flight entities with IDs at or below the Last Entity ID are processed to completion, while the peer refrains from opening new Entity Streams.¶

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Type (0x56)  |      Reserved (24 bits)                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                   Last Entity ID (32 bits)                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Reserved (24 bits):

Reserved for future use. MUST be zero when sent and MUST be ignored by receivers.¶

Last Entity ID (32 bits):

The highest Entity ID that the sender will process. Entities with IDs greater than this value (per the circular ordering defined in Section 9.1) MUST NOT be sent by the peer after receiving this frame.¶

6.6. Yield and Claim Check Extensions (Layer 2)

When E=1 in a status frame, an Extension Header MUST follow.¶

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Extension Length (32 bits)                                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Yield Reason  |           Token Length (24 bits)              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                  Yield Token (variable)                       |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                    Claim Check ID (64 bits)                   |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                Expiry Timestamp (64 bits, Unix micros)        |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

6.7. Variable-Length Serialized Messages (0x80-0xFF)

Messages in this range are preceded by a 4-octet length field. The message body is encoded using the serialization format negotiated during capability exchange (Section 3.5). If no format was negotiated, CBOR [RFC8949] is the default.¶

6.8. Entity Frames

Entity frames carry the actual document entity data on Entity Streams.¶

   +---------------------------+
   |    Header Length (4)      |   4 octets, big-endian uint32
   +---------------------------+
   |                           |
   |    Header (serialized)    |   Variable length
   |                           |
   +---------------------------+
   |                           |
   |    Payload                |   Variable length (per header)
   |                           |
   +---------------------------+

entity-header = {
  entity-id: uint,               ; 32-bit on wire (STATUS frame)
  ? parent-id: uint,             ; 32-bit scope-local
  ? scope-id: uint,              ; 32-bit (Section 6.2.1)
  layer: uint .le 3,             ; Data layer 0-3
  ? content-type: tstr,          ; MIME type
  payload-length: uint,          ; 32-bit (UCF header)
  ? checksum: bstr .size 32,     ; SHA-256; SHOULD be present
  ? metadata: { * tstr => tstr },
  ? chunk-info: chunk-info,
  ? completion-policy: completion-policy, ; Layer 2
}

chunk-info = {
  total-chunks: uint,
  chunk-index: uint,
  chunk-offset: uint,
}

yield-token = {
  reason: yield-reason,            ; See Appendix C for enum values
  ? continuation-state: bstr,
  ? validation: stopping-point-validation,
}

claim-check = {
  claim-id: uint,
  entity-id: uint,
  ? scope-id: uint,
  expiry-timestamp: uint,        ; Unix epoch microseconds
  ? validation: stopping-point-validation,
}

entity-status = uint .size 1 ; Values 0x0-0xC per Section 6.2.2

stopping-point-validation = {
  ? state-checksum: bstr,        ; Hash of processing state
  ? bytes-processed: uint,       ; Progress marker
  ? children-complete: uint,
  ? children-total: uint,
  ? is-resumable: bool,
  ? checkpoint-ref: tstr,
}

7.1. Core Fields

Every PipeStream entity is represented as a PipeDoc message:¶

7.2. Four Data Layers

Each PipeDoc carries entity payload in one of four data layers:¶

7.3. Cloud-Agnostic Storage Reference

file-storage-reference = {
  provider: tstr,                ; Storage provider identifier
  bucket: tstr,                  ; Bucket/container name
  key: tstr,                     ; Object key/path
  ? region: tstr,                ; Optional region hint
  ? attrs: { * tstr => tstr },   ; Provider-specific attributes
  ? encryption: encryption-metadata,
}

encryption-metadata = {
  algorithm: tstr,               ; "AES-256-GCM", "AES-256-CBC"
  ? key-provider: tstr,          ; "aws-kms", "azure-keyvault",
                                 ; "gcp-kms", "vault"
  ? key-id: tstr,                ; Key ARN/URI/ID
  ? wrapped-key: bstr,           ; Client-side encrypted DEK
  ? iv: bstr,                    ; Initialization vector
  ? context: { * tstr => tstr }, ; Encryption context
}

8.1. Overview

A PipeStream session proceeds through four sequential actions:¶

                +---------------------------------------------+
                |           PipeStream Action Flow            |
                +---------------------------------------------+
                                     |
                                     v
                +---------------------------------------------+
                |                  CONNECT                    |
                |    (Session + Capability Negotiation)       |
                +---------------------------------------------+
                                     |
                                     v
                +---------------------------------------------+
                |                   PARSE                     |
                |        (Dehydration: 1:N possible)         |
                +---------------------------------------------+
                                     |
                       +-------------+-------------+
                       v             v             v
                +-----------+ +-----------+ +-----------+
                |  PROCESS  | |  PROCESS  | |  PROCESS  |
                |   (1:1)   | |   (1:1)   | |   (N:1)   |
                +-----------+ +-----------+ +-----------+
                       |             |             |
                       +-------------+-------------+
                                     |
                                     v
                +---------------------------------------------+
                |                   SINK                      |
                |          (Terminal Consumption)             |
                +---------------------------------------------+

8.2. CONNECT Action

The CONNECT action establishes the session with capability negotiation.¶

8.3. PARSE Action

The PARSE action performs dehydration with optional completion policy:¶

completion-policy = {
  ? mode: completion-mode,
  ? max-retries: uint,           ; Default: 3
  ? retry-delay-ms: uint,        ; Default: 1000
  ? timeout-ms: uint,            ; Default: 300000 (5 min)
  ? min-success-ratio: float32,  ; For QUORUM mode
  ? on-timeout: failure-action,
  ? on-failure: failure-action,
}

completion-mode = &(
  unspecified: 0,                ; Default; treat as STRICT
  strict: 1,                    ; All children MUST complete
  lenient: 2,                   ; Continue with partial results
  best-effort: 3,               ; Complete with whatever succeeds
  quorum: 4,                    ; Need min-success-ratio
)

failure-action = &(
  unspecified: 0,                ; Default; treat as FAIL
  fail: 1,                      ; Propagate failure up
  skip: 2,                      ; Skip, continue with siblings
  retry: 3,                     ; Retry up to max-retries
  defer: 4,                     ; Create claim check, continue
)

8.4. PROCESS Action

8.5. SINK Action

9.1. Entity ID Lifecycle and Cursor

Entity IDs are managed using a cursor-based circular recycling scheme within the 32-bit ID space. The ID space is divided into three logical regions relative to the current cursor and last_assigned pointers:¶

The window size is computed as (last_assigned - cursor) mod 0xFFFFFFFD. If window_size >= max_window, the sender MUST apply backpressure and stop assigning new IDs until the cursor advances.¶

Rules:¶

1. new_id = (last_assigned + 1) % 0xFFFFFFFD¶

2. If new_id == 0, new_id = 1 (skip reserved NULL_ENTITY)¶

3. If (new_id - cursor) % 0xFFFFFFFD >= max_window -> STOP, apply backpressure¶

4. On reaching a terminal state (COMPLETE, SKIPPED, ABANDONED, or FAILED with no remaining retries): mark resolved; if entity_id == cursor, advance cursor past all contiguous resolved IDs¶

5. IDs behind cursor are implicitly recyclable¶

An entity in the FAILED state that may still transition to RETRYING (Section 6.2.2a) MUST NOT be marked resolved. Only when retries are exhausted or no completion policy permits retries does FAILED become terminal for cursor purposes.¶

9.2. Assembly Manifest

The Assembly Manifest is a local data structure maintained by each endpoint to track the parent-child relationships created during dehydration. It is not transmitted on the wire; rather, each endpoint constructs its own manifest from the parent-id fields in received EntityHeaders and from status updates observed on the Control Stream. The CDDL below defines the logical structure of each entry; implementations MAY use any internal representation that preserves the required semantics.¶

Each Assembly Manifest entry tracks:¶

assembly-manifest-entry = {
  parent-id: uint,
  ? scope-id: uint,              ; Layer 1
  children-ids: [* uint],
  ? children-status: [* entity-status],
  ? policy: completion-policy,   ; Layer 2
  ? created-at: uint,
  ? state: resolution-state,
}

resolution-state = &(
  unspecified: 0,
  active: 1,
  resolved: 2,
  partial: 3,                   ; Some children failed/skipped
  failed: 4,
)

9.3. Checkpoint Blocking

A checkpoint is satisfied when:¶

1. All entities in the checkpoint scope with IDs less than checkpoint_entity_id (considering circular wrap) have reached terminal state.¶

2. All Assembly Manifest entries within the checkpoint scope have been resolved.¶

3. All nested checkpoints within the checkpoint scope have been satisfied.¶

CheckpointFrame (Section 6.6 / Appendix C) carries both:¶

checkpoint-frame = {
  checkpoint-id: tstr,
  sequence-number: uint,
  checkpoint-entity-id: uint,
  ? scope-id: uint,
  ? flags: uint,
  ? timeout-ms: uint,
}

• checkpoint_id: an opaque identifier for logging and correlation.¶

• checkpoint_entity_id: the numeric ordering key used for barrier evaluation.¶

Implementations MUST use checkpoint_entity_id (not checkpoint_id) when evaluating Condition 1.¶

For circular comparison in Condition 1, implementations MUST use the same modulo ordering as cursor management. Define MAX = 0xFFFFFFFD and:¶

is_before(a, b) = ((b - a + MAX) % MAX) < (MAX / 2)¶

An entity ID a is considered "less than checkpoint_entity_id b" iff is_before(a, b) is true.¶

9.4. Scope ID Allocation (Layer 1)

When Layer 1 is negotiated, Scope IDs are 32-bit unsigned integers assigned by the endpoint that initiates the dehydration. The allocation scheme is as follows:¶

1. Scope ID 0 is the root scope and MUST NOT be used for child scopes.¶

2. The dehydrating endpoint assigns a unique Scope ID to each new child scope created during dehydration. The Scope ID MUST be unique within the connection for the lifetime of that scope (i.e., until the scope's SCOPE_DIGEST frame has been emitted and acknowledged).¶

3. Scope IDs MAY be allocated sequentially or randomly; the protocol does not require any particular ordering. Sequential allocation is RECOMMENDED for simplicity and debuggability.¶

4. Once a scope has been closed (its SCOPE_DIGEST has been sent), the Scope ID MAY be reused for a new scope. Implementations MUST ensure that no in-flight status frames reference a recycled Scope ID; this is guaranteed if the implementation waits until all entities within the scope have reached terminal state before recycling.¶

9.5. Scope Digest Propagation (Layer 1)

When a scope completes, the endpoint MUST compute a Scope Digest and propagate it to the parent scope via a SCOPE_DIGEST frame (Section 6.3).¶

The Merkle root in the Scope Digest is computed as follows:¶

1. For each entity in the scope, ordered by Entity ID (ascending), construct a 5-octet leaf value by concatenating:¶

   • The 4-octet big-endian Entity ID.¶

   • A 1-octet status field where the lower 4 bits contain the Stat code (Section 6.2.2) and the upper 4 bits are zero.¶

2. Compute SHA-256 over each 5-octet leaf to produce leaf hashes.¶

3. Build a binary Merkle tree by repeatedly hashing pairs of sibling nodes: SHA-256(left || right). If the number of nodes at any level is odd, the last node is promoted to the next level without hashing.¶

4. The root of this tree is the merkle_root value in the SCOPE_DIGEST frame.¶

This construction is deterministic: any two implementations processing the same set of entity statuses MUST produce the same Merkle root.¶

9.6. Rehydration Readiness Tracking

Implementations MUST track Assembly Manifest resolution order using a mechanism that provides O(1) insertion and amortized O(log n) minimum extraction. The tracking mechanism MUST support efficient decrease-key operations to handle out-of-order status updates.¶

Implementations MAY use a Fibonacci heap or similar priority queue to satisfy these complexity requirements.¶

9.7. Stopping Point Validation (Layer 2)

When yielding or deferring, include validation:¶

stopping-point-validation = {
  ? state-checksum: bstr,        ; Hash of processing state
  ? bytes-processed: uint,       ; Progress marker
  ? children-complete: uint,
  ? children-total: uint,
  ? is-resumable: bool,
  ? checkpoint-ref: tstr,
}

10.1. Transport Security

PipeStream inherits security from QUIC [RFC9000] and TLS 1.3 [RFC8446]. All connections MUST use TLS 1.3 or later. Implementations MUST NOT provide mechanisms to disable encryption.¶

10.2. Entity Payload Integrity

Each Entity SHOULD include a SHA-256 [FIPS-180-4] checksum in its EntityHeader (the checksum field defined in Section 6.7.2). The checksum is OPTIONAL in the wire format to accommodate zero-length entities, streamed entities whose final length is unknown at header-emission time, and scenarios where application-layer integrity mechanisms provide equivalent guarantees. When a checksum is present, it MUST be exactly 32 octets containing the SHA-256 digest computed over the raw payload bytes (the octet sequence following the EntityHeader on the Entity Stream). The checksum does not cover the EntityHeader itself.¶

For chunked entities (where chunk-info is present in the EntityHeader), each chunk MAY carry its own per-chunk checksum. The checksum in the first chunk's EntityHeader, if present, MUST cover only that chunk's payload bytes. An implementation that requires whole-entity integrity verification MUST either compute a rolling digest across all chunks or require the sender to transmit a final summary entity containing the whole-payload checksum.¶

To support true streaming of large entities, implementations MAY begin processing an entity payload before the complete payload has been received and verified. However, the final rehydration or terminal SINK operation MUST NOT be committed until the complete payload checksum has been verified.¶

If a checksum verification fails, the implementation MUST:¶

1. Reject the entity with PIPESTREAM_INTEGRITY_ERROR (0x04).¶

2. Discard any partial results or temporary state associated with the entity.¶

3. Propagate the failure according to the Completion Policy (Section 8.3).¶

Implementations that require immediate consistency SHOULD buffer the entire entity and verify the checksum before initiating processing.¶

10.3. Resource Exhaustion

Implementations MUST enforce all resource limits listed above. Exceeding any limit MUST result in the corresponding error code (see Section 11.4). Implementations SHOULD allow operators to configure stricter limits than the defaults shown here.¶

To prevent memory-exhaustion attacks, implementations MUST NOT pre-allocate memory for variable-length payloads based solely on the 32-bit Length field in the UCF header (Section 6.1.1). Memory MUST be allocated incrementally as octets are received, or capped at a smaller initial buffer until the message type and context are verified.¶

10.4. Amplification Attacks

A single dehydration operation can produce an arbitrary number of child entities from a small input, creating a potential amplification vector. To mitigate this:¶

1. Implementations MUST enforce the max_entities_per_scope limit negotiated during capability exchange (Section 3.4). Any dehydration that would exceed this limit MUST be rejected.¶

2. Implementations MUST enforce the max_scope_depth limit. A dehydration chain deeper than this limit MUST be rejected with PIPESTREAM_DEPTH_EXCEEDED (0x07).¶

3. Implementations SHOULD enforce a configurable ratio between input entity size and total child entity count. A recommended default is no more than 1,000 children per megabyte of parent payload.¶

4. The backpressure mechanism (Section 9.1) provides a natural throttle: when the in-flight window fills, no new Entity IDs can be assigned until existing entities complete and the cursor advances. Implementations MUST NOT bypass backpressure for dehydration-generated entities.¶

10.5. Privacy Considerations

PipeStream entity headers and control stream frames carry metadata that may reveal information about the entities being processed, even when payloads are encrypted at the application layer:¶

1. Entity structure leakage: The number of child entities produced by dehydration, the scope depth, and the Entity ID assignment pattern may reveal the structure of the input being processed (e.g., an entity that dehydrates into 50 children is likely a multi-part input). Implementations that require structural privacy SHOULD pad dehydration counts or use fixed decomposition granularity. Deployments that do not handle privacy-sensitive data MAY omit this padding.¶

2. Metadata in headers: The content_type, metadata map, and payload_length fields in EntityHeader (Section 6.7) are transmitted in cleartext within the QUIC-encrypted stream. Implementations that require metadata confidentiality beyond transport encryption SHOULD encrypt EntityHeader fields at the application layer and use an opaque content_type such as application/octet-stream. This overhead is unnecessary when the deployment operates within a trusted network.¶

3. Traffic analysis: The timing and size of status frames on the Control Stream may correlate with processing patterns. Implementations operating in privacy-sensitive environments SHOULD send status frames at fixed intervals with padding to obscure processing timing. Deployments in trusted environments MAY omit traffic padding to reduce bandwidth overhead.¶

4. Identifiers: The doc_id field in PipeDoc (Section 7.1) and filenames in BlobBag entries are application-layer data but may be logged by intermediate processing nodes. Implementations SHOULD provide mechanisms to redact or pseudonymize identifiers at pipeline boundaries. This recommendation may be relaxed when all nodes in the pipeline are operated by the same administrative entity.¶

10.6. Replay and Token Reuse

10.7. Encryption Key Management

When using FileStorageReference with encryption:¶

1. Key IDs MUST reference keys in approved providers.¶

2. Wrapped keys MUST use approved envelope encryption.¶

3. Key rotation MUST be supported via key_id versioning.¶

4. Implementations MUST NOT log key material.¶

5. Implementations MUST NOT include unwrapped data encryption keys in EntityHeader metadata or Control Stream frames.¶

11.1. ALPN Identifier Registration

This document registers the following ALPN [RFC7301] protocol identifier:¶

Protocol:

PipeStream Version 1¶

Identification Sequence:

0x70 0x69 0x70 0x65 0x73 0x74 0x72 0x65 0x61 0x6D 0x2F 0x31 ("pipestream/1")¶

Specification:

This document¶

11.2. PipeStream Frame Type Registry

IANA is requested to create the "PipeStream Frame Types" registry. Values are categorized into Fixed (type-sized, no length prefix) frames in 0x50-0x7F and Variable (4-octet length prefix) frames in 0x80-0xFF. Values 0xC0-0xFF are reserved for private use.¶

11.3. PipeStream Status Code Registry

IANA is requested to create the "PipeStream Status Codes" registry. Status codes are 4-bit values (0x0-0xF). Values 0xD-0xE are reserved for Expert Review. Value 0xF is reserved for private use.¶

11.4. PipeStream Error Code Registry

IANA is requested to create the "PipeStream Error Codes" registry. Values in the range 0x00-0x3F are assigned by Expert Review. Values in the range 0x40-0xFF are reserved for private use.¶

PipeStream error codes are used as QUIC application error codes in CONNECTION_CLOSE and RESET_STREAM frames. When terminating a connection or aborting a stream due to a protocol-level error, the endpoint MUST use the corresponding PipeStream error code value as the QUIC Application Error Code.¶

11.5. PipeStream Serialization Format Registry

IANA is requested to create the "PipeStream Serialization Formats" registry. New entries require Expert Review [RFC8126].¶

11.6. URI Scheme Registration

This section registers the "pipestream" URI scheme per [RFC7595]. The URI scheme identifies application context for detached or resumable resources (for example, Layer 2 yield/claim-check flows). PipeStream Layer 0 streaming semantics do not depend on this URI scheme.¶

Scheme name:

pipestream¶

Status:

Permanent¶

Applications/protocols that use this scheme:

PipeStream protocol (this document)¶

Contact:

Kristian Rickert (kristian.rickert@pipestream.ai)¶

Change controller:

IETF¶

pipestream-URI = "pipestream://" authority "/" session-id
                 [ "/" scope-path ] [ "/" entity-ref ]

session-id     = 1*( ALPHA / DIGIT / "-" )
scope-path     = scope-id *( "." scope-id )
scope-id       = 1*DIGIT
entity-ref     = 1*( ALPHA / DIGIT )
authority      = <authority, see RFC 3986, Section 3.2>

12.1. Normative References

[FIPS-180-4]

National Institute of Standards and Technology, "Secure Hash Standard (SHS)", FIPS PUB 180-4, August 2015.

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC5234]

Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/RFC5234, January 2008, <https://www.rfc-editor.org/rfc/rfc5234>.

[RFC7301]

Friedl, S., Popov, A., Langley, A., and E. Stephan, "Transport Layer Security (TLS) Application-Layer Protocol Negotiation Extension", RFC 7301, DOI 10.17487/RFC7301, July 2014, <https://www.rfc-editor.org/rfc/rfc7301>.

[RFC7595]

Thaler, D., Ed., Hansen, T., and T. Hardie, "Guidelines and Registration Procedures for URI Schemes", BCP 35, RFC 7595, DOI 10.17487/RFC7595, June 2015, <https://www.rfc-editor.org/rfc/rfc7595>.

[RFC8126]

Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, <https://www.rfc-editor.org/rfc/rfc8126>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

[RFC8446]

Rescorla, E., "The Transport Layer Security (TLS) Protocol Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018, <https://www.rfc-editor.org/rfc/rfc8446>.

[RFC8610]

Birkholz, H., Vigano, C., and C. Bormann, "Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610, June 2019, <https://www.rfc-editor.org/rfc/rfc8610>.

[RFC8949]

Bormann, C. and P. Hoffman, "Concise Binary Object Representation (CBOR)", STD 94, RFC 8949, DOI 10.17487/RFC8949, December 2020, <https://www.rfc-editor.org/rfc/rfc8949>.

[RFC9000]

Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based Multiplexed and Secure Transport", RFC 9000, DOI 10.17487/RFC9000, May 2021, <https://www.rfc-editor.org/rfc/rfc9000>.

12.2. Informative References

[MOQT]

Curley, L., Pugin, K., Nandakumar, S., Vasiliev, V., and I. Swett, "Media over QUIC Transport", Work in Progress, Internet-Draft, draft-ietf-moq-transport-16, 2025, <https://datatracker.ietf.org/doc/draft-ietf-moq-transport/>.

[RFC7574]

Bakker, A., Petrocco, R., and V. Grishchenko, "Peer-to-Peer Streaming Peer Protocol (PPSPP)", RFC 7574, DOI 10.17487/RFC7574, July 2015, <https://www.rfc-editor.org/rfc/rfc7574>.

[RFC7696]

Housley, R., "Guidelines for Cryptographic Algorithm Agility and Selecting Mandatory-to-Implement Algorithms", BCP 201, RFC 7696, DOI 10.17487/RFC7696, November 2015, <https://www.rfc-editor.org/rfc/rfc7696>.

[RFC9114]

Bishop, M., Ed., "HTTP/3", RFC 9114, DOI 10.17487/RFC9114, June 2022, <https://www.rfc-editor.org/rfc/rfc9114>.

[RFC9250]

Huitema, C., Dickinson, S., and A. Mankin, "DNS over Dedicated QUIC Connections", RFC 9250, DOI 10.17487/RFC9250, May 2022, <https://www.rfc-editor.org/rfc/rfc9250>.

[RFC9260]

Stewart, R., Tüxen, M., and K. Nielsen, "Stream Control Transmission Protocol", RFC 9260, DOI 10.17487/RFC9260, June 2022, <https://www.rfc-editor.org/rfc/rfc9260>.

[RFC9297]

Schinazi, D. and L. Pardue, "HTTP Datagrams and the Capsule Protocol", RFC 9297, DOI 10.17487/RFC9297, August 2022, <https://www.rfc-editor.org/rfc/rfc9297>.

[RFC9308]

Kühlewind, M. and B. Trammell, "Applicability of the QUIC Transport Protocol", RFC 9308, DOI 10.17487/RFC9308, September 2022, <https://www.rfc-editor.org/rfc/rfc9308>.

[scatter-gather]

Lea, D., "The Scatter-Gather Design Pattern", DOI 10.1007/978-1-4612-1260-6, 1996, <https://doi.org/10.1007/978-1-4612-1260-6>.

B.1. QUIC as Application Transport

RFC 9308 [RFC9308] provides guidance for designers of application protocol mappings to QUIC. PipeStream follows this guidance: stream semantics are mapped to the protocol's data model (Section 4), flow control operates at both the stream and connection level, and an ALPN token is registered for protocol identification (Section 11).¶

The precedent for application protocols that bypass HTTP and map directly onto QUIC is well established. DNS over Dedicated QUIC Connections [RFC9250] adopts a direct mapping on the grounds that HTTP framing introduces unnecessary overhead when the application has its own message semantics. The Media over QUIC Transport protocol [MOQT] similarly defines its own framing and control messages over QUIC streams, with HTTP/3 as an optional encapsulation rather than a requirement.¶

PipeStream's requirements align with these precedents. The protocol's dual-stream architecture (Section 4), bit-packed control frames (Section 6), and recursive entity lifecycle (Section 9) have no counterpart in HTTP semantics, and mapping them onto request-response pairs would add complexity without benefit.¶

B.2. Efficiency Considerations

The decision to implement PipeStream as a standalone protocol over QUIC is driven by the following efficiency requirements:¶

B.3. HTTP/3

HTTP/3 [RFC9114] provides multiplexed request-response exchanges over QUIC. Its stream model binds each client-initiated request to a server response on the same stream. PipeStream requires bidirectional, peer-initiated entity streams where either endpoint may open new streams to transmit sub-entities arising from recursive decomposition. The request-response constraint precludes this.¶

PipeStream also requires a persistent control stream carrying compact, fixed-size status frames at high frequency. HTTP/3 does define unidirectional control streams, but their framing is specific to HTTP semantics (SETTINGS, GOAWAY, MAX_PUSH_ID) and cannot be repurposed for application-level status coordination without introducing a parallel signaling mechanism that duplicates much of what QUIC already provides.¶

B.4. gRPC

gRPC defines a remote procedure call framework over HTTP/2, with experimental support for HTTP/3. Bidirectional streaming in gRPC is scoped to a single RPC method: one request stream and one response stream per call. PipeStream requires an arbitrary number of concurrent entity streams with independent flow control, plus a dedicated control stream, all within a single connection. Achieving this over gRPC would require either multiplexing all entities onto a single bidirectional RPC (sacrificing per-stream flow control and head-of-line independence) or opening a separate RPC per entity (sacrificing session-level coordination and incurring per-call overhead).¶

gRPC further mandates a 5-octet length-prefixed framing envelope for every message. PipeStream's fixed-size control frames are bit-packed at the wire level with zero serialization overhead, which is material at the status update frequencies the protocol is designed to sustain.¶

B.5. WebTransport

WebTransport [RFC9297] provides bidirectional streams and unreliable datagrams over HTTP/3, and is the closest existing protocol to the transport abstraction PipeStream requires. However, several properties make it unsuitable as a substrate:¶

WebTransport sessions are established via an HTTP/3 CONNECT request, inheriting the client-server asymmetry of HTTP. In PipeStream, both endpoints participate symmetrically in capability negotiation and may initiate entity streams; the protocol does not distinguish a "client" role from a "server" role after the handshake.¶

WebTransport is designed for environments constrained by the web security model (origin-based isolation, CORS). PipeStream targets server-to-server processing pipelines where these constraints are inapplicable.¶

WebTransport provides raw byte streams with no built-in coordination semantics. PipeStream would need to implement its own framing, status state machine, checkpoint barriers, and scope hierarchy on top of WebTransport streams. At that point, the HTTP/3 session layer introduces an additional round trip during establishment and per-stream framing overhead, with no corresponding benefit.¶

B.6. Summary

PipeStream occupies a design point not addressed by existing protocols: a QUIC-native application protocol combining multiplexed entity streaming, recursive decomposition with hierarchical scopes, Merkle-based integrity propagation, and barrier-synchronized reassembly. While existing protocols like SCTP [RFC9260] and PPSPP [RFC7574] address subsets of these requirements, none provide the integrated lifecycle and coordination semantics that PipeStream defines.¶

D.1. Protocol-Level Messages

// Copyright 2026 PipeStream AI
//
// PipeStream Protocol - IETF draft protocol for recursive
// entity streaming over QUIC. Defines the wire-format
// messages for Layers 0-2 of the PipeStream architecture:
// core streaming, recursive scoping, and resilience.
//
// Edition 2023 is used for closed enums (critical for wire-protocol
// safety) and implicit field presence (distinguishing "not set" from
// zero values). In this edition, all fields have explicit presence
// by default, making the 'optional' keyword unnecessary.

edition = "2023";

package pipestream.protocol.v1;

import "google/protobuf/any.proto";

// All enums in this file are CLOSED. Unknown enum values received on
// the wire MUST be rejected. This is essential because status codes
// are encoded as 4-bit values in the status frame wire format;
// accepting unknown values could cause undefined behavior in state
// machines and cursor advancement.
option features.enum_type = CLOSED;

// SerializationFormat specifies the encoding for variable-
// length control messages (0x80-0xFF) and entity headers.
enum SerializationFormat {
  SERIALIZATION_FORMAT_CBOR = 0;
  SERIALIZATION_FORMAT_PROTOBUF = 1;
}

// Capabilities describes the feature set supported by a PipeStream
// endpoint. Exchanged during the CONNECT handshake so that both
// sides can negotiate which protocol layers and resource limits
// apply to the session.
message Capabilities {
  // Whether the endpoint supports Layer 0 (core entity streaming).
  bool layer0_core = 1;

  // Whether the endpoint supports Layer 1 (recursive scoping and
  // dehydration).
  bool layer1_recursive = 2;

  // Whether the endpoint supports Layer 2 (resilience, yield, and
  // claim-check). Requires Layer 1 support; if layer1_recursive is
  // false, this MUST be false.
  bool layer2_resilience = 3;

  // Maximum nesting depth allowed for recursive scopes.
  // Default is 7 (8 levels: 0-7).
  uint32 max_scope_depth = 4;

  // Maximum number of entities permitted within a single scope.
  uint32 max_entities_per_scope = 5;

  // Maximum flow-control window size, in number of in-flight
  // entities. Default is 2,147,483,648 (2^31).
  uint32 max_window_size = 6;

  // Serialization format (default: CBOR).
  SerializationFormat serialization_format = 7;
}

// EntityHeader is sent at the beginning of each entity stream to
// describe the payload that follows. It carries identity, lineage,
// content metadata, chunking information, and the completion policy
// that governs how partial failures of this entity's children are
// handled.
message EntityHeader {
  // Scope-local entity identifier.
  uint32 entity_id = 1;

  // Identifier of the parent entity.
  uint32 parent_id = 2;

  // Identifier of the scope.
  uint32 scope_id = 3;

  // Data layer (0-3).
  uint32 layer = 4;

  // MIME content type.
  string content_type = 5;

  // Length in bytes of the complete entity payload, before any
  // chunking.
  uint64 payload_length = 6;

  // SHA-256 integrity checksum.
  bytes checksum = 7;

  // Arbitrary metadata.
  map<string, string> metadata = 8;

  // Chunking information.
  ChunkInfo chunk_info = 9;

  // Resilience completion policy.
  CompletionPolicy completion_policy = 10;
}

// ChunkInfo describes how a payload is divided into chunks.
message ChunkInfo {
  // Total number of chunks.
  uint32 total_chunks = 1;

  // Zero-based chunk index.
  uint32 chunk_index = 2;

  // Byte offset within the complete payload.
  uint64 chunk_offset = 3;
}

// CompletionPolicy controls Layer 2 resilience behavior.
message CompletionPolicy {
  // Mode for evaluating completion.
  CompletionMode mode = 1;

  // Maximum retry attempts.
  uint32 max_retries = 2;

  // Delay between retries in milliseconds.
  uint32 retry_delay_ms = 3;

  // Maximum wait time in milliseconds.
  uint32 timeout_ms = 4;

  // Minimum success ratio for QUORUM mode.
  float min_success_ratio = 5;

  // Action on timeout.
  FailureAction on_timeout = 6;

  // Action on failure.
  FailureAction on_failure = 7;
}

// CompletionMode specifies completion evaluation strategies.
enum CompletionMode {
  COMPLETION_MODE_UNSPECIFIED = 0;
  COMPLETION_MODE_STRICT = 1;
  COMPLETION_MODE_LENIENT = 2;
  COMPLETION_MODE_BEST_EFFORT = 3;
  COMPLETION_MODE_QUORUM = 4;
}

// FailureAction specifies error handling behaviors.
enum FailureAction {
  FAILURE_ACTION_UNSPECIFIED = 0;
  FAILURE_ACTION_FAIL = 1;
  FAILURE_ACTION_SKIP = 2;
  FAILURE_ACTION_RETRY = 3;
  FAILURE_ACTION_DEFER = 4;
}

// EntityStatus represents the lifecycle state of an entity.
enum EntityStatus {
  ENTITY_STATUS_UNSPECIFIED = 0;
  ENTITY_STATUS_PENDING = 1;
  ENTITY_STATUS_PROCESSING = 2;
  ENTITY_STATUS_COMPLETE = 3;
  ENTITY_STATUS_FAILED = 4;
  ENTITY_STATUS_CHECKPOINT = 5;
  ENTITY_STATUS_DEHYDRATING = 6;
  ENTITY_STATUS_REHYDRATING = 7;
  ENTITY_STATUS_YIELDED = 8;
  ENTITY_STATUS_DEFERRED = 9;
  ENTITY_STATUS_RETRYING = 10;
  ENTITY_STATUS_SKIPPED = 11;
  ENTITY_STATUS_ABANDONED = 12;
}

// StatusFrame represents a status transition.
message StatusFrame {
  // Identifier of the entity.
  uint32 entity_id = 1;

  // Identifier of the scope.
  uint32 scope_id = 2;

  // Current status.
  EntityStatus status = 3;

  // Optional extension data.
  google.protobuf.Any extended_data = 4;
}

// CheckpointFrame (Type 0x81)
message CheckpointFrame {
  // Unique checkpoint identifier.
  string checkpoint_id = 1;

  // Monotonic sequence number.
  uint64 sequence_number = 2;

  // Numeric ordering key for barrier evaluation.
  uint32 checkpoint_entity_id = 3;

  // Scope to which this checkpoint applies.
  uint32 scope_id = 4;

  // Checkpoint flags.
  uint32 flags = 5;

  // Maximum wait time in milliseconds.
  uint32 timeout_ms = 6;
}

// AssemblyManifestEntry tracks parent-child relationships.
message AssemblyManifestEntry {
  // Identifier of the parent entity.
  uint32 parent_id = 1;

  // Scope identifier.
  uint32 scope_id = 2;

  // Ordered child identifiers.
  repeated uint32 children_ids = 3;

  // Current status of each child.
  repeated EntityStatus children_status = 4;

  // Governing completion policy.
  CompletionPolicy policy = 5;

  // Creation timestamp (Unix epoch microseconds).
  uint64 created_at = 6;

  // Current resolution state.
  ResolutionState state = 7;
}

// ResolutionState tracks Assembly Manifest completion.
enum ResolutionState {
  RESOLUTION_STATE_UNSPECIFIED = 0;
  RESOLUTION_STATE_ACTIVE = 1;
  RESOLUTION_STATE_RESOLVED = 2;
  RESOLUTION_STATE_PARTIAL = 3;
  RESOLUTION_STATE_FAILED = 4;
}

// YieldToken captures continuation state for paused entities.
message YieldToken {
  // Reason for yielding.
  YieldReason reason = 1;

  // Opaque continuation state.
  bytes continuation_state = 2;

  // Validation data for resumption.
  StoppingPointValidation validation = 3;
}

// YieldReason describes why processing was yielded.
enum YieldReason {
  YIELD_REASON_UNSPECIFIED = 0;
  YIELD_REASON_EXTERNAL_CALL = 1;
  YIELD_REASON_RATE_LIMITED = 2;
  YIELD_REASON_AWAITING_SIBLING = 3;
  YIELD_REASON_AWAITING_APPROVAL = 4;
  YIELD_REASON_RESOURCE_BUSY = 5;
}

// ClaimCheck is a Layer 2 deferred-processing reference.
message ClaimCheck {
  // Unique claim identifier.
  uint64 claim_id = 1;

  // Identifier of the deferred entity.
  uint32 entity_id = 2;

  // Scope identifier.
  uint32 scope_id = 3;

  // Expiry timestamp (Unix epoch microseconds).
  uint64 expiry_timestamp = 4;

  // Validation data.
  StoppingPointValidation validation = 5;
}

// StoppingPointValidation captures a snapshot of progress.
message StoppingPointValidation {
  // Hash of internal state.
  bytes state_checksum = 1;

  // Payload bytes consumed.
  uint64 bytes_processed = 2;

  // Completed child count.
  uint32 children_complete = 3;

  // Total expected child count.
  uint32 children_total = 4;

  // Resumption capability flag.
  bool is_resumable = 5;

  // Last passed checkpoint reference.
  string checkpoint_ref = 6;
}

// ScopeDigest is a Layer 1 summary of a completed scope.
message ScopeDigest {
  // Identifier of the scope.
  uint32 scope_id = 1;

  // Total processed count.
  uint64 entities_processed = 2;

  // Total succeeded count.
  uint64 entities_succeeded = 3;

  // Total failed count.
  uint64 entities_failed = 4;

  // Total deferred count.
  uint64 entities_deferred = 5;

  // Merkle root hash.
  bytes merkle_root = 6;
}

// PipeDoc represents the top-level document envelope for an entity.
message PipeDoc {
  // Unique document identifier.
  string doc_id = 1;

  // Identifier of the entity.
  uint32 entity_id = 2;

  // Ownership and access context.
  OwnershipContext ownership = 3;
}

// OwnershipContext defines multi-tenancy and access control for
// entities.
message OwnershipContext {
  // Entity owner identifier.
  string owner_id = 1;

  // Group identifier.
  string group_id = 2;

  // List of access scopes.
  repeated string scopes = 3;
}

// FileStorageReference provides a location for external data.
message FileStorageReference {
  // Storage provider identifier.
  string provider = 1;

  // Bucket or container name.
  string bucket = 2;

  // Object key or path.
  string key = 3;

  // Optional region hint.
  string region = 4;

  // Provider-specific attributes.
  map<string, string> attrs = 5;

  // Encryption metadata.
  EncryptionMetadata encryption = 6;
}

// EncryptionMetadata defines encryption parameters.
message EncryptionMetadata {
  // Encryption algorithm.
  string algorithm = 1;

  // Key provider identifier.
  string key_provider = 2;

  // Encryption key identifier.
  string key_id = 3;

  // Optional wrapped DEK.
  bytes wrapped_key = 4;

  // Initialization vector.
  bytes iv = 5;

  // Additional encryption context.
  map<string, string> context = 6;
}