docs/design/part-buffering.txt - mtk-gstreamer-debian - Git at Google

 Buffering
 ---------

 This document outlines the buffering policy used in the GStreamer
 core that can be used by plugins and applications.

 The purpose of buffering is to accumulate enough data in a pipeline so that
 playback can occur smoothly and without interruptions. It is typically done
 when reading from a (slow) non-live network source but can also be used for
 live sources.

 We want to be able to implement the following features:

  - buffering up to a specific amount of data, in memory, before starting playback
    so that network fluctuations are minimized.
  - download of the network file to a local disk with fast seeking in the
    downloaded data. This is similar to the quicktime/youtube players.
  - caching of semi-live streams to a local, on disk, ringbuffer with seeking in
    the cached area. This is similar to tivo-like timeshifting.
  - progress report about the buffering operations
  - the possibility for the application to do more complex buffering

 Some use cases:

  * Stream buffering:

    +---------+     +---------+     +-------+
    | httpsrc |     | buffer  |     | demux |
    |        src - sink      src - sink     ....
    +---------+     +---------+     +-------+

   In this case we are reading from a slow network source into a buffer element
   (such as queue2).

   The buffer element has a low and high watermark expressed in bytes. The
   buffer uses the watermarks as follows:

    - The buffer element will post BUFFERING messages until the high watermark
      is hit. This instructs the application to keep the pipeline PAUSED, which
      will eventually block the srcpad from pushing while data is prerolled in
      the sinks.
    - When the high watermark is hit, a BUFFERING message with 100% will be
      posted, which instructs the application to continue playback.
    - When the low watermark is hit during playback, the queue will start posting
      BUFFERING messages again, making the application PAUSE the pipeline again
      until the high watermark is hit again. This is called the rebuffering
      stage.
    - During playback, the queue level will fluctuate between the high and
      low watermarks as a way to compensate for network irregularities.

   This buffering method is usable when the demuxer operates in push mode.
   Seeking in the stream requires the seek to happen in the network source.
   It is mostly desirable when the total duration of the file is not known, such
   as in live streaming or when efficient seeking is not possible/required.

  * Incremental download

    +---------+     +---------+     +-------+
    | httpsrc |     | buffer  |     | demux |
    |        src - sink      src - sink     ....
    +---------+     +----|----+     +-------+
                         V
                        file

   In this case, we know the server is streaming a fixed length file to the
   client. The application can choose to download the file to disk. The buffer
   element will provide a push or pull based srcpad to the demuxer to navigate in
   the downloaded file.

   This mode is only suitable when the client can determine the length of the
   file on the server.

   In this case, buffering messages will be emitted as usual when the requested
   range is not within the downloaded area + buffersize. The buffering message
   will also contain an indication that incremental download is being performed.
   This flag can be used to let the application control the buffering in a more
   intelligent way, using the BUFFERING query, for example.

   The application can use the BUFFERING query to get the estimated download time
   and match this time to the current/remaining playback time to control when
   playback should start to have a non-interrupted playback experience.


  * Timeshifting

    +---------+     +---------+     +-------+
    | httpsrc |     | buffer  |     | demux |
    |        src - sink      src - sink     ....
    +---------+     +----|----+     +-------+
                         V
                     file-ringbuffer

   In this mode, a fixed size ringbuffer is kept to download the server content.
   This allows for seeking in the buffered data. Depending on the size of the
   buffer one can seek further back in time.

   This mode is suitable for all live streams.

   As with the incremental download mode, buffering messages are emitted along
   with an indication that timeshifting download is in progress.


  * Live buffering

   In live pipelines we usually introduce some latency between the capture and
   the playback elements. This latency can be introduced by a queue (such as a
   jitterbuffer) or by other means (in the audiosink).

   Buffering messages can be emitted in those live pipelines as well and serve as
   an indication to the user of the latency buffering. The application usually
   does not react to these buffering messages with a state change.


 Messages
 ~~~~~~~~

 A GST_MESSAGE_BUFFERING must be posted on the bus when playback temporarily
 stops to buffer and when buffering finishes. When the percentage field in the
 BUFFERING message is 100, buffering is done. Values less than 100 mean that
 buffering is in progress.

 The BUFFERING message should be intercepted and acted upon by the application.
 The message contains at least one field that is sufficient for basic
 functionality:

   "buffer-percent", G_TYPE_INT, between 0 and 100

 Several more clever ways of dealing with the buffering messages can be used when
 in incremental or timeshifting download mode. For this purpose additional fields
 are added to the buffering message:

   "buffering-mode", GST_TYPE_BUFFERING_MODE,
                    enum { "stream", "download", "timeshift", "live" }
       - Buffering mode in use. See above for an explanation of the
         different alternatives. This field can be used to let the
 	application have more control over the buffering process.

   "avg-in-rate", G_TYPE_INT
       - Average input buffering speed in bytes/second. -1 is unknown.
         This is the average number of bytes per second that is received on the
 	buffering element input (sink) pads. It is a measurement of the network
 	speed in most cases.

   "avg-out-rate", G_TYPE_INT
       - Average consumption speed in bytes/second. -1 is unknown.
         This is the average number of bytes per second that is consumed by the
 	downstream element of the buffering element.

   "buffering-left", G_TYPE_INT64
       - Estimated time that buffering will take in milliseconds. -1 is unknown.
         This is measured based on the avg-in-rate and the filled level of the
         queue. The application can use this hint to update the GUI about the
         estimated remaining time that buffering will take.

 Application
 ~~~~~~~~~~~

 While data is buffered the pipeline should remain in the PAUSED state. It is
 also possible that more data should be buffered while the pipeline is PLAYING,
 in which case the pipeline should be PAUSED until the buffering finishes.

 BUFFERING messages can be posted while the pipeline is prerolling. The
 application should not set the pipeline to PLAYING before a BUFFERING message
 with a 100 percent value is received, which might only happen after the pipeline
 prerolls.

 An exception is made for live pipelines. The application may not change
 the state of a live pipeline when a buffering message is received. Usually these
 buffering messages contain the "buffering-mode" = "live".

 The buffering message can also instruct the application to switch to a
 periodical BUFFERING query instead, so it can more precisely control the
 buffering process. The application can, for example, choose not to act on the
 BUFFERING complete message (buffer-percent = 100) to resume playback but use
 the estimated download time instead, resuming playback when it has determined
 that it should be able to provide uninterrupted playback.


 Buffering Query
 ~~~~~~~~~~~~~~~

 In addition to the BUFFERING messages posted by the buffering elements, we want
 to be able to query the same information from the application. We also want to
 be able to present the user with information about the downloaded range in the
 file so that the GUI can react on it.

 In addition to all the fields present in the buffering message, the BUFFERING
 query contains the following field, which indicates the available downloaded
 range in a specific format and the estimated time to complete:

   "busy", G_TYPE_BOOLEAN
     - if buffering was busy. This flag allows the application to pause the
       pipeline by using the query only.

   "format", GST_TYPE_FORMAT
     - the format of the "start" and "stop" values below

   "start", G_TYPE_INT64, -1 unknown
     - the start position of the available data. If there are multiple ranges,
       this field contains the start position of the currently downloading
       range.

   "stop", G_TYPE_INT64, -1 unknown
     - the stop position of the available data. If there are multiple ranges,
       this field contains the stop position of the currently downloading
       range.

   "estimated-total", G_TYPE_INT64
     - gives the estimated download time in milliseconds. -1 unknown.

     When the size of the downloaded file is known, this value will contain
     the latest estimate of the remaining download time of the currently
     downloading range. This value is usually only filled for the "download"
     buffering mode. The application can use this information to estimate the
     amount of remaining time to download till the end of the file.

   "buffering-ranges", G_TYPE_ARRAY of GstQueryBufferingRange
     - contains optionally the downloaded areas in the format given above. One
       of the ranges contains the same start/stop position as above.

       typedef struct
       {
         gint64 start;
         gint64 stop;
       } GstQueryBufferingRange;


 For the "download" and "timeshift" buffering-modes, the start and stop positions
 specify the ranges where efficient seeking in the downloaded media is possible.
 Seeking outside of these ranges might be slow or not at all possible.

 For the "stream" and "live" mode the start and stop values describe the oldest
 and newest item (expressed in "format") in the buffer.


 Defaults
 ~~~~~~~~

 Some defaults for common elements:

 A GstBaseSrc with random access replies to the BUFFERING query with:

   "buffer-percent" = 100
   "buffering-mode" = "stream"
   "avg-in-rate" = -1
   "avg-out-rate" = -1
   "buffering-left" = 0
   "format" = GST_FORMAT_BYTES
   "start" = 0
   "stop" = the total filesize
   "estimated-total" = 0
   "buffering-ranges" = NULL

 A GstBaseSrc in push mode replies to the BUFFERING query with:

   "buffer-percent" = 100
   "buffering-mode" = "stream"
   "avg-in-rate" = -1
   "avg-out-rate" = -1
   "buffering-left" = 0
   "format" = a valid GST_TYPE_FORMAT
   "start" = current position
   "stop" = current position
   "estimated-total" = -1
   "buffering-ranges" = NULL


 Buffering strategies
 ~~~~~~~~~~~~~~~~~~~~

  Buffering strategies are specific implementations based on the buffering
  message and query described above.

  Most strategies have to balance buffering time versus maximal playback
  experience.

  * simple buffering

    NON-live pipelines are kept in the paused state while buffering messages with
    a percent < 100% are received.

    This buffering strategy relies on the buffer size and low/high watermarks of
    the element. It can work with a fixed size buffer in memory or on disk.

    The size of the buffer is usually expressed in a fixed amount of time units
    and the estimated bitrate of the upstream source is used to convert this time
    to bytes.

    All GStreamer applications must implement this strategy. Failure to do so
    will result in starvation at the sink.

  * no-rebuffer strategy

    This strategy tries to buffer as much data as possible so that playback can
    continue without any further rebuffering.

    This strategy is initially similar to simple buffering, the difference is in
    deciding on the condition to continue playback. When a 100% buffering message
    has been received, the application will not yet start the playback but it will
    start a periodic buffering query, which will return the estimated amount of
    buffering time left. When the estimated time left is less than the remaining
    playback time, playback can continue.

    This strategy requires a unlimited buffer size in memory or on disk, such as
    provided by elements that implement the incremental download buffering mode.

    Usually, the application can choose to start playback even before the
    remaining buffer time elapsed in order to more quickly start the playback at
    the expense of a possible rebuffering phase.

  * Incremental rebuffering

    The application implements the simple buffering strategy but with each
    rebuffering phase, it increases the size of the buffer.

    This strategy has quick, fixed time startup times but incrementally longer
    rebuffering times if the network is slower than the media bitrate.
	Buffering
	---------

	This document outlines the buffering policy used in the GStreamer
	core that can be used by plugins and applications.

	The purpose of buffering is to accumulate enough data in a pipeline so that
	playback can occur smoothly and without interruptions. It is typically done
	when reading from a (slow) non-live network source but can also be used for
	live sources.

	We want to be able to implement the following features:

	- buffering up to a specific amount of data, in memory, before starting playback
	so that network fluctuations are minimized.
	- download of the network file to a local disk with fast seeking in the
	downloaded data. This is similar to the quicktime/youtube players.
	- caching of semi-live streams to a local, on disk, ringbuffer with seeking in
	the cached area. This is similar to tivo-like timeshifting.
	- progress report about the buffering operations
	- the possibility for the application to do more complex buffering

	Some use cases:

	* Stream buffering:

	+---------+ +---------+ +-------+
	\| httpsrc \| \| buffer \| \| demux \|
	\| src - sink src - sink ....
	+---------+ +---------+ +-------+

	In this case we are reading from a slow network source into a buffer element
	(such as queue2).

	The buffer element has a low and high watermark expressed in bytes. The
	buffer uses the watermarks as follows:

	- The buffer element will post BUFFERING messages until the high watermark
	is hit. This instructs the application to keep the pipeline PAUSED, which
	will eventually block the srcpad from pushing while data is prerolled in
	the sinks.
	- When the high watermark is hit, a BUFFERING message with 100% will be
	posted, which instructs the application to continue playback.
	- When the low watermark is hit during playback, the queue will start posting
	BUFFERING messages again, making the application PAUSE the pipeline again
	until the high watermark is hit again. This is called the rebuffering
	stage.
	- During playback, the queue level will fluctuate between the high and
	low watermarks as a way to compensate for network irregularities.

	This buffering method is usable when the demuxer operates in push mode.
	Seeking in the stream requires the seek to happen in the network source.
	It is mostly desirable when the total duration of the file is not known, such
	as in live streaming or when efficient seeking is not possible/required.

	* Incremental download

	+---------+ +---------+ +-------+
	\| httpsrc \| \| buffer \| \| demux \|
	\| src - sink src - sink ....
	+---------+ +----\|----+ +-------+
	V
	file

	In this case, we know the server is streaming a fixed length file to the
	client. The application can choose to download the file to disk. The buffer
	element will provide a push or pull based srcpad to the demuxer to navigate in
	the downloaded file.

	This mode is only suitable when the client can determine the length of the
	file on the server.

	In this case, buffering messages will be emitted as usual when the requested
	range is not within the downloaded area + buffersize. The buffering message
	will also contain an indication that incremental download is being performed.
	This flag can be used to let the application control the buffering in a more
	intelligent way, using the BUFFERING query, for example.

	The application can use the BUFFERING query to get the estimated download time
	and match this time to the current/remaining playback time to control when
	playback should start to have a non-interrupted playback experience.


	* Timeshifting

	+---------+ +---------+ +-------+
	\| httpsrc \| \| buffer \| \| demux \|
	\| src - sink src - sink ....
	+---------+ +----\|----+ +-------+
	V
	file-ringbuffer

	In this mode, a fixed size ringbuffer is kept to download the server content.
	This allows for seeking in the buffered data. Depending on the size of the
	buffer one can seek further back in time.

	This mode is suitable for all live streams.

	As with the incremental download mode, buffering messages are emitted along
	with an indication that timeshifting download is in progress.


	* Live buffering

	In live pipelines we usually introduce some latency between the capture and
	the playback elements. This latency can be introduced by a queue (such as a
	jitterbuffer) or by other means (in the audiosink).

	Buffering messages can be emitted in those live pipelines as well and serve as
	an indication to the user of the latency buffering. The application usually
	does not react to these buffering messages with a state change.


	Messages
	~~~~~~~~

	A GST_MESSAGE_BUFFERING must be posted on the bus when playback temporarily
	stops to buffer and when buffering finishes. When the percentage field in the
	BUFFERING message is 100, buffering is done. Values less than 100 mean that
	buffering is in progress.

	The BUFFERING message should be intercepted and acted upon by the application.
	The message contains at least one field that is sufficient for basic
	functionality:

	"buffer-percent", G_TYPE_INT, between 0 and 100

	Several more clever ways of dealing with the buffering messages can be used when
	in incremental or timeshifting download mode. For this purpose additional fields
	are added to the buffering message:

	"buffering-mode", GST_TYPE_BUFFERING_MODE,
	enum { "stream", "download", "timeshift", "live" }
	- Buffering mode in use. See above for an explanation of the
	different alternatives. This field can be used to let the
	application have more control over the buffering process.

	"avg-in-rate", G_TYPE_INT
	- Average input buffering speed in bytes/second. -1 is unknown.
	This is the average number of bytes per second that is received on the
	buffering element input (sink) pads. It is a measurement of the network
	speed in most cases.

	"avg-out-rate", G_TYPE_INT
	- Average consumption speed in bytes/second. -1 is unknown.
	This is the average number of bytes per second that is consumed by the
	downstream element of the buffering element.

	"buffering-left", G_TYPE_INT64
	- Estimated time that buffering will take in milliseconds. -1 is unknown.
	This is measured based on the avg-in-rate and the filled level of the
	queue. The application can use this hint to update the GUI about the
	estimated remaining time that buffering will take.

	Application
	~~~~~~~~~~~

	While data is buffered the pipeline should remain in the PAUSED state. It is
	also possible that more data should be buffered while the pipeline is PLAYING,
	in which case the pipeline should be PAUSED until the buffering finishes.

	BUFFERING messages can be posted while the pipeline is prerolling. The
	application should not set the pipeline to PLAYING before a BUFFERING message
	with a 100 percent value is received, which might only happen after the pipeline
	prerolls.

	An exception is made for live pipelines. The application may not change
	the state of a live pipeline when a buffering message is received. Usually these
	buffering messages contain the "buffering-mode" = "live".

	The buffering message can also instruct the application to switch to a
	periodical BUFFERING query instead, so it can more precisely control the
	buffering process. The application can, for example, choose not to act on the
	BUFFERING complete message (buffer-percent = 100) to resume playback but use
	the estimated download time instead, resuming playback when it has determined
	that it should be able to provide uninterrupted playback.


	Buffering Query
	~~~~~~~~~~~~~~~

	In addition to the BUFFERING messages posted by the buffering elements, we want
	to be able to query the same information from the application. We also want to
	be able to present the user with information about the downloaded range in the
	file so that the GUI can react on it.

	In addition to all the fields present in the buffering message, the BUFFERING
	query contains the following field, which indicates the available downloaded
	range in a specific format and the estimated time to complete:

	"busy", G_TYPE_BOOLEAN
	- if buffering was busy. This flag allows the application to pause the
	pipeline by using the query only.

	"format", GST_TYPE_FORMAT
	- the format of the "start" and "stop" values below

	"start", G_TYPE_INT64, -1 unknown
	- the start position of the available data. If there are multiple ranges,
	this field contains the start position of the currently downloading
	range.

	"stop", G_TYPE_INT64, -1 unknown
	- the stop position of the available data. If there are multiple ranges,
	this field contains the stop position of the currently downloading
	range.

	"estimated-total", G_TYPE_INT64
	- gives the estimated download time in milliseconds. -1 unknown.

	When the size of the downloaded file is known, this value will contain
	the latest estimate of the remaining download time of the currently
	downloading range. This value is usually only filled for the "download"
	buffering mode. The application can use this information to estimate the
	amount of remaining time to download till the end of the file.

	"buffering-ranges", G_TYPE_ARRAY of GstQueryBufferingRange
	- contains optionally the downloaded areas in the format given above. One
	of the ranges contains the same start/stop position as above.

	typedef struct
	{
	gint64 start;
	gint64 stop;
	} GstQueryBufferingRange;


	For the "download" and "timeshift" buffering-modes, the start and stop positions
	specify the ranges where efficient seeking in the downloaded media is possible.
	Seeking outside of these ranges might be slow or not at all possible.

	For the "stream" and "live" mode the start and stop values describe the oldest
	and newest item (expressed in "format") in the buffer.


	Defaults
	~~~~~~~~

	Some defaults for common elements:

	A GstBaseSrc with random access replies to the BUFFERING query with:

	"buffer-percent" = 100
	"buffering-mode" = "stream"
	"avg-in-rate" = -1
	"avg-out-rate" = -1
	"buffering-left" = 0
	"format" = GST_FORMAT_BYTES
	"start" = 0
	"stop" = the total filesize
	"estimated-total" = 0
	"buffering-ranges" = NULL

	A GstBaseSrc in push mode replies to the BUFFERING query with:

	"buffer-percent" = 100
	"buffering-mode" = "stream"
	"avg-in-rate" = -1
	"avg-out-rate" = -1
	"buffering-left" = 0
	"format" = a valid GST_TYPE_FORMAT
	"start" = current position
	"stop" = current position
	"estimated-total" = -1
	"buffering-ranges" = NULL


	Buffering strategies
	~~~~~~~~~~~~~~~~~~~~

	Buffering strategies are specific implementations based on the buffering
	message and query described above.

	Most strategies have to balance buffering time versus maximal playback
	experience.

	* simple buffering

	NON-live pipelines are kept in the paused state while buffering messages with
	a percent < 100% are received.

	This buffering strategy relies on the buffer size and low/high watermarks of
	the element. It can work with a fixed size buffer in memory or on disk.

	The size of the buffer is usually expressed in a fixed amount of time units
	and the estimated bitrate of the upstream source is used to convert this time
	to bytes.

	All GStreamer applications must implement this strategy. Failure to do so
	will result in starvation at the sink.

	* no-rebuffer strategy

	This strategy tries to buffer as much data as possible so that playback can
	continue without any further rebuffering.

	This strategy is initially similar to simple buffering, the difference is in
	deciding on the condition to continue playback. When a 100% buffering message
	has been received, the application will not yet start the playback but it will
	start a periodic buffering query, which will return the estimated amount of
	buffering time left. When the estimated time left is less than the remaining
	playback time, playback can continue.

	This strategy requires a unlimited buffer size in memory or on disk, such as
	provided by elements that implement the incremental download buffering mode.

	Usually, the application can choose to start playback even before the
	remaining buffer time elapsed in order to more quickly start the playback at
	the expense of a possible rebuffering phase.

	* Incremental rebuffering

	The application implements the simple buffering strategy but with each
	rebuffering phase, it increases the size of the buffer.

	This strategy has quick, fixed time startup times but incrementally longer
	rebuffering times if the network is slower than the media bitrate.