]> Vivoh, Inc.

erik@vivoh.com

Media Over QUIC This document updates the MOQT Streaming Format (MSF) by defining a new optional feature for the streaming format. It specifies the syntax and semantics for adding Neural Video Codec (NVC) packaged media to MSF. NVC codecs use learned neural network transforms for video compression, and their bitstreams require a distinct packaging model from traditional block-based codecs. NMSF maps neural keyframes (Intra) and delta frames (Inter) onto MoQ Groups and Objects, enabling real-time neural video streaming over any standard MoQ relay. About This Document This note is to be removed before publishing as an RFC. Source for this draft and an issue tracker can be found at .

Introduction Neural Video Codecs (NVCs) represent a new class of video compression that uses learned neural network transforms instead of block-based motion estimation and discrete cosine transforms. NVCs such as DCVC-RT [DCVC-RT], SSF, FVC, and RLVC produce compressed bitstreams that differ fundamentally from traditional codecs:

• No container format. NVC bitstreams consist of entropy-coded latent tensors, not fMP4 boxes, LOC containers, or NAL units.
• Stateful decoding. The decoder maintains a learned context buffer (analogous to a decoded picture buffer) that must be initialized from a full neural keyframe before delta frames can be decoded.
• Variable-rate representations. Compressed frame sizes vary significantly based on scene complexity, as the codec allocates bits adaptively in a learned feature space.

The existing MSF packaging types -- LOC and the timeline types -- do not accommodate these bitstreams. The CMSF extension adds CMAF packaging for traditional block-based codecs. Neither is suitable for NVC data, which has no container structure and requires a different model for keyframe semantics and decoder state management. This document defines NMSF, an MSF extension that adds NVC packaging. NMSF follows the same extension pattern established by CMSF: it registers a new "packaging" value, defines the Object payload format, and specifies Group-level requirements for decoder random access. A single MoQ Broadcast MAY contain tracks using any combination of packaging types. For example, an NVC video track may coexist with a LOC or CMAF audio track in the same catalog.

MSF Extension All of the specifications, requirements, and terminology defined in apply to implementations of this extension unless explicitly noted otherwise in this document.

NVC Packaging

Object Wire Format Each MoQ Object payload for a track with "nvc" packaging consists of a fixed 17-byte header followed by a variable-length compressed bitstream:

frame_type:

1 byte. The type of neural video frame contained in this Object. See .

frame_number:

4 bytes, unsigned 32-bit integer, big-endian. The absolute sequence number of this frame within the stream, starting from zero.

width:

4 bytes, unsigned 32-bit integer, big-endian. The frame width in pixels.

height:

4 bytes, unsigned 32-bit integer, big-endian. The frame height in pixels.

payload_len:

4 bytes, unsigned 32-bit integer, big-endian. The byte length of the payload field that follows.

payload:

Variable length. The NVC compressed bitstream for this frame. The internal structure is codec-specific and opaque to the packaging layer. See .

Frame Types The frame_type field identifies the role of the frame in the neural codec's prediction structure. The following values are defined:

Intra frames are analogous to SAP Type 1 access points in CMAF. They enable random access by fully initializing the decoder's context buffer without dependence on any prior frame. Inter frames are analogous to non-SAP frames (P-frames). They depend on the decoder's context buffer, which contains the reconstructed output of the immediately preceding frame. Unlike traditional codecs with multi-frame reference picture buffers, current NVCs maintain a single context buffer containing learned features from the previous reconstructed frame. The reserved range (0x02-0xFF) accommodates future NVCs that may introduce bidirectional prediction, hierarchical quality layers, or multi-reference architectures.

Object Packaging The payload of each Object is subject to the following requirements:

• MUST contain exactly one NVC frame in the wire format defined in .
• MUST NOT span multiple frames. Each frame is carried in a separate Object.
• Objects within a Group MUST be sequentially ordered by frame_number. Out-of-order processing causes encoder-decoder context buffer divergence.

Group Packaging Each MOQT Group:

• MUST begin with an Object containing an Intra frame (frame_type = 0x00).
• MUST contain one contiguous Group of Pictures (GOP): one Intra frame followed by zero or more Inter frames.
• The Group boundary aligns with the publisher's neural GOP boundary. Typical GOP sizes are 30-120 frames (1-4 seconds at 30 fps).

This structure ensures that a subscriber joining mid-stream or recovering from loss can begin decoding from the next Group boundary. The Intra frame at the start of each Group fully initializes the decoder context buffer, enabling immediate playback without waiting for a future keyframe.

Payload Format The payload field of each Object contains the NVC's compressed bitstream. Its internal structure is codec-specific and opaque to the NMSF packaging layer. For NVCs using hyperprior-based entropy coding (such as DCVC-RT with entropy models derived from ), the following payload sub-format is RECOMMENDED:

• hyper_bitstream_len (4 bytes, uint32 big-endian): Length of the entropy-coded hyperprior side information.
• hyper_bitstream (variable): Entropy-coded hyperprior z.
• latent_bitstream_len (4 bytes, uint32 big-endian): Length of the entropy-coded latent tensor.
• latent_bitstream (variable): Entropy-coded latent y.
• latent_height (4 bytes, uint32 big-endian): Spatial height of the latent tensor.
• latent_width (4 bytes, uint32 big-endian): Spatial width.
• num_channels (4 bytes, uint32 big-endian): Channel count.

Catalog Description

NVC Packaging Type This specification extends the allowed packaging values defined in to include one new entry, as defined in Table 2 below:

Every Track entry in an MSF catalog carrying NVC-packaged media data MUST declare a "packaging" type value of "nvc".

NVC-specific Catalog Fields This specification adds the following track-level catalog fields for tracks with "nvc" packaging:

The standard MSF track fields "name", "packaging", "isLive", "width", "height", and "framerate" retain their definitions from and are REQUIRED for NVC tracks.

NVC Metadata Object The optional "nvc" object within a track catalog entry carries codec-specific metadata that a subscriber may need to configure its decoder:

Subscribers that do not recognize the "codec" value or cannot satisfy the metadata requirements SHOULD NOT subscribe to the track.

Decoder Requirements This section specifies the behavior required of an NMSF decoder. These requirements ensure that encoder and decoder context buffers remain synchronized, preventing visual artifacts caused by state drift.

Context Buffer Management The decoder MUST maintain a context buffer containing the reconstructed output of the most recently decoded frame. This buffer is used as input to the synthesis transform when decoding Inter frames. The context buffer is uninitialized when the decoder starts. The decoder MUST NOT attempt to decode Inter frames until it has successfully decoded an Intra frame. After decoding each frame (Intra or Inter), the decoder MUST replace its context buffer with the newly reconstructed output. Failure to update the context buffer causes progressive drift between encoder and decoder state, manifesting as visual artifacts commonly described as "ghosting" or "smearing."

Intra Frame Handling When the decoder receives an Intra frame (frame_type = 0x00):

1. Discard any existing context buffer.
2. Decode the frame using only the payload data (no context).
3. Store the reconstructed output as the new context buffer.
4. Output the reconstructed frame for display.

Inter Frame Handling When the decoder receives an Inter frame (frame_type = 0x01):

1. Verify that a context buffer exists (i.e., an Intra frame has been previously decoded). If not, discard the frame.
2. Decode the frame using the payload AND the current context buffer.
3. Replace the context buffer with the newly reconstructed output.
4. Output the reconstructed frame for display.

Stream Join and Recovery When a subscriber joins a stream mid-session or recovers from packet loss:

1. Wait for the start of the next MoQ Group.
2. The first Object in the Group is an Intra frame.
3. Decode the Intra frame to initialize the context buffer.
4. Continue decoding subsequent Inter frames normally.

Objects received before the first Intra frame MUST be discarded.

Encoder Context Synchronization The encoder MUST maintain its own context buffer that mirrors the decoder's state. After encoding each frame, the encoder MUST run the synthesis (decoding) transform on the quantized latent representation to produce a reconstructed frame, and use that reconstructed frame as the context for encoding the next frame. This "encode-decode loop" ensures that the encoder's context buffer contains exactly the same data that a decoder would produce from the transmitted bitstream, preventing encoder-decoder state drift.

Codec Registration The "codec" field in the catalog identifies which NVC is used. The following identifiers are defined by this document:

New NVC codecs are compatible with NMSF packaging if they produce distinct Intra and Inter frame types and use a single sequential context buffer. Registration requires choosing a unique codec identifier string and documenting the payload sub-format (). No changes to the wire format header () are required for new codecs.

Catalog Examples The following section provides non-normative JSON examples of catalogs compliant with this draft.

NVC video with LOC audio This example shows a catalog for a live broadcast with one DCVC-RT neural video track and one Opus audio track using MSF's native LOC packaging.

NVC video with CMAF audio This example shows a catalog mixing NVC packaging (video) with CMSF's CMAF packaging (audio) in a single broadcast. This demonstrates interoperability between NMSF and CMSF extensions.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Security Considerations NMSF relies on the security properties of MoQ Transport , which provides confidentiality and integrity via QUIC's TLS 1.3 encryption. NMSF does not add its own integrity or authentication mechanisms. The "payload_len" field permits payloads up to 4 GiB. Decoders SHOULD enforce a maximum payload size appropriate for their deployment environment (e.g., 100 MiB for 4K video) and reject Objects exceeding that limit to mitigate resource exhaustion. A malicious publisher could craft Intra frames that cause the decoder's context buffer to enter a state producing misleading visual output on subsequent Inter frames. This is analogous to reference picture manipulation in traditional codecs and is mitigated by the same trust model: subscribers SHOULD only connect to authenticated and authorized publishers. Neural video codec model weights are typically large (tens to hundreds of megabytes) and are NOT transmitted via MoQ. Both publisher and subscriber must have compatible model weights pre-installed. The "nvc" catalog metadata () enables version negotiation, but the secure distribution of model weights is outside the scope of this document.

IANA Considerations This document requests registration of a new packaging type value "nvc" in the MSF packaging registry defined by . This document requests creation of an "NVC Codec Identifiers" registry with the initial values defined in Table 5 of . New entries require Specification Required registration policy.

References Normative References Akamai This document specifies the MOQT Streaming Format, designed to operate on Media Over QUIC Transport. Akamai This document updates [MSF] by defining a new optional feature for the streaming format. It specifies the syntax and semantics for adding CMAF-packaged media [CMAF] to MSF. Cisco Google Google Meta This document defines the core behavior for Media over QUIC Transport (MOQT), a media transport protocol designed to operate over QUIC and WebTransport, which have similar functionality. MOQT allows a producer of media to publish data and have it consumed via subscription by a multiplicity of endpoints. It supports intermediate content distribution networks and is designed for high scale and low latency distribution. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References Microsoft Research InterDigital ITU-R

Acknowledgments The author would like to thank Will Law for the MSF and CMSF specifications which established the extension pattern that NMSF follows, and the MoQ working group for the transport protocol that makes this work possible.