| <chapter id="chapter-buffering"> |
| <title>Buffering</title> |
| <para> |
| The purpose of buffering is to accumulate enough data in a pipeline so that |
| playback can occur smoothly and without interruptions. It is typically done |
| when reading from a (slow) and non-live network source but can also be |
| used for live sources. |
| </para> |
| <para> |
| &GStreamer; provides support for the following use cases: |
| <itemizedlist> |
| <listitem> |
| <para> |
| Buffering up to a specific amount of data, in memory, before starting |
| playback so that network fluctuations are minimized. |
| See <xref linkend="section-buffering-stream"/>. |
| </para> |
| </listitem> |
| <listitem> |
| <para> |
| Download of the network file to a local disk with fast seeking in the |
| downloaded data. This is similar to the quicktime/youtube players. |
| See <xref linkend="section-buffering-download"/>. |
| </para> |
| </listitem> |
| <listitem> |
| <para> |
| Caching of (semi)-live streams to a local, on disk, ringbuffer with |
| seeking in the cached area. This is similar to tivo-like timeshifting. |
| See <xref linkend="section-buffering-timeshift"/>. |
| </para> |
| </listitem> |
| </itemizedlist> |
| </para> |
| <para> |
| &GStreamer; can provide the application with progress reports about the |
| current buffering state as well as let the application decide on how |
| to buffer and when the buffering stops. |
| </para> |
| <para> |
| In the most simple case, the application has to listen for BUFFERING |
| messages on the bus. If the percent indicator inside the BUFFERING message |
| is smaller than 100, the pipeline is buffering. When a message is |
| received with 100 percent, buffering is complete. In the buffering state, |
| the application should keep the pipeline in the PAUSED state. When buffering |
| completes, it can put the pipeline (back) in the PLAYING state. |
| </para> |
| <para> |
| What follows is an example of how the message handler could deal with |
| the BUFFERING messages. We will see more advanced methods in |
| <xref linkend="section-buffering-strategies"/>. |
| </para> |
| <programlisting> |
| <![CDATA[ |
| [...] |
| |
| switch (GST_MESSAGE_TYPE (message)) { |
| case GST_MESSAGE_BUFFERING:{ |
| gint percent; |
| |
| /* no state management needed for live pipelines */ |
| if (is_live) |
| break; |
| |
| gst_message_parse_buffering (message, &percent); |
| |
| if (percent == 100) { |
| /* a 100% message means buffering is done */ |
| buffering = FALSE; |
| /* if the desired state is playing, go back */ |
| if (target_state == GST_STATE_PLAYING) { |
| gst_element_set_state (pipeline, GST_STATE_PLAYING); |
| } |
| } else { |
| /* buffering busy */ |
| if (!buffering && target_state == GST_STATE_PLAYING) { |
| /* we were not buffering but PLAYING, PAUSE the pipeline. */ |
| gst_element_set_state (pipeline, GST_STATE_PAUSED); |
| } |
| buffering = TRUE; |
| } |
| break; |
| case ... |
| |
| [...] |
| ]]> |
| </programlisting> |
| |
| <sect1 id="section-buffering-stream"> |
| <title>Stream buffering </title> |
| <programlisting> |
| +---------+ +---------+ +-------+ |
| | httpsrc | | buffer | | demux | |
| | src - sink src - sink .... |
| +---------+ +---------+ +-------+ |
| </programlisting> |
| <para> |
| In this case we are reading from a slow network source into a buffer |
| element (such as queue2). |
| </para> |
| <para> |
| The buffer element has a low and high watermark expressed in bytes. The |
| buffer uses the watermarks as follows: |
| </para> |
| <itemizedlist> |
| <listitem> |
| <para> |
| The buffer element will post BUFFERING messages until the high |
| watermark is hit. This instructs the application to keep the pipeline |
| PAUSED, which will eventually block the srcpad from pushing while |
| data is prerolled in the sinks. |
| </para> |
| </listitem> |
| <listitem> |
| <para> |
| When the high watermark is hit, a BUFFERING message with 100% will be |
| posted, which instructs the application to continue playback. |
| </para> |
| </listitem> |
| <listitem> |
| <para> |
| When during playback, the low watermark is hit, the queue will start |
| posting BUFFERING messages again, making the application PAUSE the |
| pipeline again until the high watermark is hit again. This is called |
| the rebuffering stage. |
| </para> |
| </listitem> |
| <listitem> |
| <para> |
| During playback, the queue level will fluctuate between the high and |
| the low watermark as a way to compensate for network irregularities. |
| </para> |
| </listitem> |
| </itemizedlist> |
| <para> |
| This buffering method is usable when the demuxer operates in push mode. |
| Seeking in the stream requires the seek to happen in the network source. |
| It is mostly desirable when the total duration of the file is not known, |
| such as in live streaming or when efficient seeking is not |
| possible/required. |
| </para> |
| <para> |
| The problem is configuring a good low and high watermark. Here are some |
| ideas: |
| </para> |
| <itemizedlist> |
| <listitem> |
| <para> |
| It is possible to measure the network bandwidth and configure the |
| low/high watermarks in such a way that buffering takes a fixed |
| amount of time. |
| </para> |
| <para> |
| The queue2 element in &GStreamer; core has the max-size-time property |
| that, together with the use-rate-estimate property, does exactly |
| that. Also the playbin buffer-duration property uses the rate estimate |
| to scale the amount of data that is buffered. |
| </para> |
| </listitem> |
| <listitem> |
| <para> |
| Based on the codec bitrate, it is also possible to set the watermarks |
| in such a way that a fixed amount of data is buffered before playback |
| starts. Normally, the buffering element doesn't know about the |
| bitrate of the stream but it can get this with a query. |
| </para> |
| </listitem> |
| <listitem> |
| <para> |
| Start with a fixed amount of bytes, measure the time between |
| rebuffering and increase the queue size until the time between |
| rebuffering is within the application's chosen limits. |
| </para> |
| </listitem> |
| </itemizedlist> |
| <para> |
| The buffering element can be inserted anywhere in the pipeline. You could, |
| for example, insert the buffering element before a decoder. This would |
| make it possible to set the low/high watermarks based on time. |
| </para> |
| <para> |
| The buffering flag on playbin, performs buffering on the parsed data. |
| Another advantage of doing the buffering at a later stage is that you can |
| let the demuxer operate in pull mode. When reading data from a slow |
| network drive (with filesrc) this can be an interesting way to buffer. |
| </para> |
| </sect1> |
| |
| <sect1 id="section-buffering-download"> |
| <title>Download buffering </title> |
| <programlisting> |
| +---------+ +---------+ +-------+ |
| | httpsrc | | buffer | | demux | |
| | src - sink src - sink .... |
| +---------+ +----|----+ +-------+ |
| V |
| file |
| </programlisting> |
| <para> |
| If we know the server is streaming a fixed length file to the client, |
| the application can choose to download the entire file on disk. The |
| buffer element will provide a push or pull based srcpad to the demuxer |
| to navigate in the downloaded file. |
| </para> |
| <para> |
| This mode is only suitable when the client can determine the length of |
| the file on the server. |
| </para> |
| <para> |
| In this case, buffering messages will be emitted as usual when the |
| requested range is not within the downloaded area + buffersize. The |
| buffering message will also contain an indication that incremental |
| download is being performed. This flag can be used to let the application |
| control the buffering in a more intelligent way, using the BUFFERING |
| query, for example. See <xref linkend="section-buffering-strategies"/>. |
| </para> |
| </sect1> |
| |
| <sect1 id="section-buffering-timeshift"> |
| <title>Timeshift buffering </title> |
| <programlisting> |
| +---------+ +---------+ +-------+ |
| | httpsrc | | buffer | | demux | |
| | src - sink src - sink .... |
| +---------+ +----|----+ +-------+ |
| V |
| file-ringbuffer |
| </programlisting> |
| <para> |
| In this mode, a fixed size ringbuffer is kept to download the server |
| content. This allows for seeking in the buffered data. Depending on the |
| size of the ringbuffer one can seek further back in time. |
| </para> |
| <para> |
| This mode is suitable for all live streams. As with the incremental |
| download mode, buffering messages are emitted along with an indication |
| that timeshifting download is in progress. |
| </para> |
| </sect1> |
| |
| <sect1 id="section-buffering-live"> |
| <title>Live buffering </title> |
| <para> |
| In live pipelines we usually introduce some fixed latency between the |
| capture and the playback elements. This latency can be introduced by |
| a queue (such as a jitterbuffer) or by other means (in the audiosink). |
| </para> |
| <para> |
| Buffering messages can be emitted in those live pipelines as well and |
| serve as an indication to the user of the latency buffering. The |
| application usually does not react to these buffering messages with a |
| state change. |
| </para> |
| </sect1> |
| |
| <sect1 id="section-buffering-strategies"> |
| <title>Buffering strategies </title> |
| <para> |
| What follows are some ideas for implementing different buffering |
| strategies based on the buffering messages and buffering query. |
| </para> |
| |
| <sect2 id="section-buffering-norebuffer"> |
| <title>No-rebuffer strategy </title> |
| <para> |
| We would like to buffer enough data in the pipeline so that playback |
| continues without interruptions. What we need to know to implement |
| this is know the total remaining playback time in the file and the |
| total remaining download time. If the buffering time is less than the |
| playback time, we can start playback without interruptions. |
| </para> |
| <para> |
| We have all this information available with the DURATION, POSITION and |
| BUFFERING queries. We need to periodically execute the buffering query |
| to get the current buffering status. We also need to have a large |
| enough buffer to hold the complete file, worst case. It is best to |
| use this buffering strategy with download buffering (see |
| <xref linkend="section-buffering-download"/>). |
| </para> |
| <para> |
| This is what the code would look like: |
| </para> |
| <programlisting> |
| <!-- example-begin norebuffer.c a --> |
| <![CDATA[ |
| #include <gst/gst.h> |
| |
| GstState target_state; |
| static gboolean is_live; |
| static gboolean is_buffering; |
| |
| static gboolean |
| buffer_timeout (gpointer data) |
| { |
| GstElement *pipeline = data; |
| GstQuery *query; |
| gboolean busy; |
| gint percent; |
| gint64 estimated_total; |
| gint64 position, duration; |
| guint64 play_left; |
| |
| query = gst_query_new_buffering (GST_FORMAT_TIME); |
| |
| if (!gst_element_query (pipeline, query)) |
| return TRUE; |
| |
| gst_query_parse_buffering_percent (query, &busy, &percent); |
| gst_query_parse_buffering_range (query, NULL, NULL, NULL, &estimated_total); |
| |
| if (estimated_total == -1) |
| estimated_total = 0; |
| |
| /* calculate the remaining playback time */ |
| if (!gst_element_query_position (pipeline, GST_FORMAT_TIME, &position)) |
| position = -1; |
| if (!gst_element_query_duration (pipeline, GST_FORMAT_TIME, &duration)) |
| duration = -1; |
| |
| if (duration != -1 && position != -1) |
| play_left = GST_TIME_AS_MSECONDS (duration - position); |
| else |
| play_left = 0; |
| |
| g_message ("play_left %" G_GUINT64_FORMAT", estimated_total %" G_GUINT64_FORMAT |
| ", percent %d", play_left, estimated_total, percent); |
| |
| /* we are buffering or the estimated download time is bigger than the |
| * remaining playback time. We keep buffering. */ |
| is_buffering = (busy || estimated_total * 1.1 > play_left); |
| |
| if (!is_buffering) |
| gst_element_set_state (pipeline, target_state); |
| |
| return is_buffering; |
| } |
| |
| static void |
| on_message_buffering (GstBus *bus, GstMessage *message, gpointer user_data) |
| { |
| GstElement *pipeline = user_data; |
| gint percent; |
| |
| /* no state management needed for live pipelines */ |
| if (is_live) |
| return; |
| |
| gst_message_parse_buffering (message, &percent); |
| |
| if (percent < 100) { |
| /* buffering busy */ |
| if (!is_buffering) { |
| is_buffering = TRUE; |
| if (target_state == GST_STATE_PLAYING) { |
| /* we were not buffering but PLAYING, PAUSE the pipeline. */ |
| gst_element_set_state (pipeline, GST_STATE_PAUSED); |
| } |
| } |
| } |
| } |
| |
| static void |
| on_message_async_done (GstBus *bus, GstMessage *message, gpointer user_data) |
| { |
| GstElement *pipeline = user_data; |
| |
| if (!is_buffering) |
| gst_element_set_state (pipeline, target_state); |
| else |
| g_timeout_add (500, buffer_timeout, pipeline); |
| } |
| |
| gint |
| main (gint argc, |
| gchar *argv[]) |
| { |
| GstElement *pipeline; |
| GMainLoop *loop; |
| GstBus *bus; |
| GstStateChangeReturn ret; |
| |
| /* init GStreamer */ |
| gst_init (&argc, &argv); |
| loop = g_main_loop_new (NULL, FALSE); |
| |
| /* make sure we have a URI */ |
| if (argc != 2) { |
| g_print ("Usage: %s <URI>\n", argv[0]); |
| return -1; |
| } |
| |
| /* set up */ |
| pipeline = gst_element_factory_make ("playbin", "pipeline"); |
| g_object_set (G_OBJECT (pipeline), "uri", argv[1], NULL); |
| g_object_set (G_OBJECT (pipeline), "flags", 0x697 , NULL); |
| |
| bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline)); |
| gst_bus_add_signal_watch (bus); |
| |
| g_signal_connect (bus, "message::buffering", |
| (GCallback) on_message_buffering, pipeline); |
| g_signal_connect (bus, "message::async-done", |
| (GCallback) on_message_async_done, pipeline); |
| gst_object_unref (bus); |
| |
| is_buffering = FALSE; |
| target_state = GST_STATE_PLAYING; |
| ret = gst_element_set_state (pipeline, GST_STATE_PAUSED); |
| |
| switch (ret) { |
| case GST_STATE_CHANGE_SUCCESS: |
| is_live = FALSE; |
| break; |
| |
| case GST_STATE_CHANGE_FAILURE: |
| g_warning ("failed to PAUSE"); |
| return -1; |
| |
| case GST_STATE_CHANGE_NO_PREROLL: |
| is_live = TRUE; |
| break; |
| |
| default: |
| break; |
| } |
| |
| /* now run */ |
| g_main_loop_run (loop); |
| |
| /* also clean up */ |
| gst_element_set_state (pipeline, GST_STATE_NULL); |
| gst_object_unref (GST_OBJECT (pipeline)); |
| g_main_loop_unref (loop); |
| |
| return 0; |
| } |
| ]]> |
| <!-- example-end norebuffer.c a --> |
| </programlisting> |
| <para> |
| See how we set the pipeline to the PAUSED state first. We will receive |
| buffering messages during the preroll state when buffering is needed. |
| When we are prerolled (on_message_async_done) we see if buffering is |
| going on, if not, we start playback. If buffering was going on, we start |
| a timeout to poll the buffering state. If the estimated time to download |
| is less than the remaining playback time, we start playback. |
| </para> |
| </sect2> |
| </sect1> |
| </chapter> |
