web-stream: A General Purpose Message Framing over Byte-Stream Oriented Wire Protocols

Background There have been several attempts to improve message-based streaming communication on the Web. The server-sent events (SSE) realized message-based streaming communication in the server-to-client direction. It introduced a new Web API and a special text-based framing format while using HTTP as the wire protocol. WebSockets realized native client-server full-duplex message-based communication for the Web by introducing both a new Web API and a new wire protocol (the WebSocket Protocol). While widely deployed, the wire protocol is incompatible with the broad HTTP ecosystem . Consequently, intermediaries and frameworks have to be upgraded to understand the wire protocol to support WebSockets. This Internet-Draft addresses the aforementioned problem space with a solution fully compliant with the standard HTTP semantics, informally referred to as "Web in Strict HTTP" (WiSH).

Introduction web-stream offers a general-purpose message framing over a byte-stream. The framing is compatible with the well-adopted and security-proven base framing of the WebSocket Protocol . The underlying communication protocols providing the standard HTTP semantics can be HTTP/1.1 , HTTP/2 , HTTP/3 . The web-stream framing can also be used over any other byte-stream oriented protocols (over TCP or QUIC ). Transport-level features such as multiplexing and session priority are provided by the underlying byte-stream protocol . HTTP implementations may not allow earlier 2xx responses or full-duplex communication , particularly when HTTP/1.1 is used as the underlying byte-stream protocol. In such cases, parallel HTTP requests can be used to support full-duplex messaging between the client and server. Separately, when HTTP body streaming is unsupported by the Web client or blocked by intermediaries, multiple short-lived HTTP requests can be used to send messages between the client and server. web-stream strictly complies with the HTTP semantics. Consequently, while web-stream utilizes the base framing of the WebSocket Protocol, web-stream drops the opening handshake and the closing handshake of the WebSocket Protocol.

Conformance Requirements and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in . Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and abort these steps") are to be interpreted with the meaning of the key word ("MUST", "SHOULD", "MAY", etc.) used in introducing the algorithm. Conformance requirements phrased as algorithms or specific steps can be implemented in any manner, so long as the end result is equivalent. In particular, the algorithms defined in this specification are intended to be easy to understand and are not intended to be performant.

Web-Stream over HTTP This section describes how to use web-stream over HTTP . To use web-stream with an HTTP request or response, the content part (i.e. body part) MUST be a sequence of zero or more web-stream frames, as defined in . The Content-Type header for an HTTP request or response using web-stream MUST be set to the web-stream media type, application/web-stream, which includes an optional message parameter. Example: Content-Type: application/web-stream; message="application/json" The message parameter specifies the content type of the web-stream message payload. The standard HTTP semantics should be followed, particularly regarding the choice of HTTP method and headers. Certain HTTP semantics, such as Cache-Control, may not be applicable when the HTTP body is streamed to the application. These limitations are intrinsic to streaming, not web-stream specifically. It's also possible to use web-stream without application-level streaming, such as sending batched messages over HTTP.

Framing The web-stream framing is designed to be compatible with the base framing protocol of the WebSocket Protocol . A high-level overview of the framing is provided in the figure below.

Payload length The payload length and extended payload length fields are used to determine the length of the Payload Data field of the frame, following the exact mechanism defined in the WebSocket Protocol.

Fragmentation The FIN bit and the continuation frame opcode enable a fragmentation mechanism, which is identical to that of the WebSocket Protocol. One or more continuous frames constitute a web-stream message. The FIN bit indicates that the frame is the final frame of a web-stream message. For the second and following frames of a web-stream message, the opcode field MUST be set to the continuation frame opcode. The mechanism allows for sending portions of a web-stream message with a large payload before the total size of the message is determined.

Payload Compression web-stream defines the CMP bit as the second most significant bit of the first octet of a frame. The CMP bit in the first frame of a message indicates whether that message's payload is compressed. When the message payload is compressed, the CMP bit of the first frame MUST be set to 1; otherwise, it MUST be set to 0. The CMP bit MUST always be set to 0 for all non-first frames. See for details of the payload compression.

Opcode For the "opcode" field, web-stream uses the following opcodes: %x0 denotes a continuation frame %x1 denotes the first frame of a text message %x2 denotes the first frame of a binary message %x3 denotes the first frame of a metadata message %x9 denotes a ping %xA denotes a pong Any values not listed above are reserved. The opcodes for continuation frames, text messages, and binary messages are identical to those specified in the WebSocket Protocol. web-stream introduces the metadata message opcode (%x3), which is used for exchanging in-stream metadata between the client and server. When the opcode field of the first frame is set to %x3, the message contains metadata. The application-level protocol determines the encoding of the metadata message payload. Support for metadata is optional; web-stream implementations that do not support or consume metadata messages MAY discard them. We don't expect the WebSocket Protocol to introduce any new opcodes in future. The WebSocket Protocol's connection close frame opcode (%x8) is ignored for web-stream. The ping (%x9) and pong (%xA) control frames are retained from the WebSocket Protocol. web-stream endpoints MAY alternatively rely on keep-alive mechanisms provided by the underlying transport.

Compression over HTTP

General Guidelines about Compression For short-lived streaming, the content codings (for HTTP body) MAY be applied in the underlying HTTP layer for web-stream as specified in Section 8.4. Content-Encoding of . HTTP compression is transparent to the web-stream framing and is generally efficient for short-lived streaming. For long-lived streaming, Per-message Compression for WebSocket (PMCE) may provide better performance. Implementations may port PMCE to web-stream as described in the following section. web-stream compression and HTTP compression SHOULD NOT be enabled simultaneously to avoid redundant compression.

web-stream Compression defined a general framework on how to implement message-oriented compression for using its extension framework. This is named Per-message Compression for WebSocket (PMCE). The web-stream compression is an adaptation (port) of PMCE for web-stream.

Negotiation Enabling the web-stream compression requires the participating endpoints to perform the extension negotiation defined in through HTTP request and response headers. The name of the header MUST be Web-Stream-Extensions instead of Sec-WebSocket-Extensions. Example negotiation: The client includes the following header in HTTP request to offer use of a compression that is equivalent to the permessage-deflate PMCE for the WebSocket Protocol. Web-Stream-Extensions: permessage-deflate Some parameters can be added to the permessage-deflate entry as follows: Web-Stream-Extensions: permessage-deflate; client_max_window_bits=10 The server includes the following header in HTTP response to accept use of the compression as follows: Web-Stream-Extensions: permessage-deflate

Framing web-stream compression uses the second significant bit of the first octet of a frame to indicate whether the message payload is compressed. This bit is referred to as the "Per-Message Compressed" bit in the PMCE specification. In the web-stream framing, this bit is named the CMP bit. When the CMP bit is set in the first frame of a message, the message payload MUST be compressed.

Compression Methods other than DEFLATE While DEFLATE is the only compression algorithm supported by the WebSocket Protocol, web-stream may support additional compression algorithms, such as Zstandard. A separate Internet-Draft will be published in the future to specify these web-stream compression methods.

UTF-8 Vallidation The WebSocket Protocol requires the endpoints to _Fail the WebSocket Connection_ when they encounter an invalid UTF-8 byte sequence in a text message. Consequently, server frameworks typically check for UTF-8 validity. While the contents of web-stream text messages also MUST be a valid UTF-8 sequence, web-stream endpoints are not REQUIRED to perform UTF-8 validation. This design choice provides greater flexibility to server implementations, which may, for example, choose to perform UTF-8 validation within a JSON parser.

Acknowledgements The authors gratefully acknowledge the feedback received on the original version of this draft from the IETF BiDirectional or Server-Initiated HTTP (hybi) Working Group and the IETF HTTP Working Group during 2016-2017.