Streaming protocol specification

Server streams data to clients. There is only one server and there can be zero or more connected clients.

Messages from Server to Client (ClientBound)

Each server streaming session must start with ClientBound{start} and stop with ClientBound{stop} messages. Interleaved ClientBound{video} and ClientBound{overlay} are allowed in between:

-> ClientBound{start={width=<width>, height=<height>}}

-> ClientBound{video} or ClientBound{overlay}

-> ClientBound{stop}

Each video message contains one or more H264 encoded NAL units. Partial NAL units are not allowed. The first NAL unit must be SPS (Sequence Parameter Set), the second one is IDR. Concatenation of all video messages should produce a valid H264 bitstream. All subsequent SPS NAL units must contain the same information as the first one.

Overlay messages are allowed at any time. You can think that there are two logical streams during the session: video stream and stream of overlays. Each overlay message contains SVG image which is drawn on top of the video.

Messages from Client to Server (ServerBound)

Client can control the server by sending ServerBound{stream_control} messages. There are only two allowed: ServerBound{stream_control={enabled=true}} to start streaming and ServerBound{stream_control={enabled=false}} to stop streaming. Server could ignore these messages if it is already in the requested state.

<- ServerBound{stream_control={enabled=true}}


<- ServerBound{stream_control={enabled=false}}

Example

From the Server's point of view ordered by absolute time:

<- ServerBound{stream_control={enabled=true}}
-> ClientBound{start={width=720, height=480}}
-> ClientBound{overlay={svg=<DATA>}
-> ClientBound{video={<SPS>}}  # First NAL unit in video stream
-> ClientBound{overlay={svg=<DATA>}
-> ClientBound{video={<IDR>}}  # Second NAL unit in video stream
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{overlay={svg=<DATA>}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{overlay={svg=<DATA>}
-> ClientBound{video={<SPS>}}  # The same as the first SPS NAL unit.
-> ClientBound{video={<IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{overlay={svg=<DATA>}
<- ServerBound{stream_control={enabled=false}}
-> ClientBound{video={<IDR>}}
-> ClientBound{video={<NON-IDR>}}
-> ClientBound{stop}

References

Protocol Buffers H264 Specification SVG Specification