docs/pwg/advanced-qos.xml

<chapter id="chapter-advanced-qos">
  <title>Quality Of Service (QoS)</title>

  <para>
    Quality of Service in &GStreamer; is about measuring and adjusting
    the real-time performance of a pipeline. The real-time performance is
    always measured relative to the pipeline clock and typically happens in
    the sinks when they synchronize buffers against the clock.
  </para>
  <para>
    When buffers arrive late in the sink, i.e. when their running-time is
    smaller than that of the clock, we say that the pipeline is having a
    quality of service problem. These are a few possible reasons:
  </para>
  <itemizedlist>
    <listitem>
      <para>
        High CPU load, there is not enough CPU power to handle the stream,
        causing buffers to arrive late in the sink.
      </para>
    </listitem>
    <listitem>
      <para>
        Network problems
      </para>
    </listitem>
    <listitem>
      <para>
        Other resource problems such as disk load, memory bottlenecks etc
      </para>
    </listitem>
  </itemizedlist>
  <para>
    The measurements result in QOS events that aim to adjust the datarate
    in one or more upstream elements. Two types of adjustments can be
    made:
  </para>
  <itemizedlist>
    <listitem>
      <para>
        Short time "emergency" corrections based on latest observation in
        the sinks.
      </para>
      <para>
        Long term rate corrections based on trends observed in the sinks.
      </para>
    </listitem>
  </itemizedlist>
  <para>
    It is also possible for the application to artificially introduce delay
    between synchronized buffers, this is called throttling. It can be used
    to limit or reduce the framerate, for example.
  </para>

  <sect1 id="section-measuring">
    <title>Measuring QoS</title>
    <para>
      Elements that synchronize buffers on the pipeline clock will usually
      measure the current QoS. They will also need to keep some statistics
      in order to generate the QOS event.
    </para>
    <para>
      For each buffer that arrives in the sink, the element needs to calculate
      how late or how early it was. This is called the jitter. Negative jitter
      values mean that the buffer was early, positive values mean that the
      buffer was late. the jitter value gives an indication of how early/late
      a buffer was.
    </para>
    <para>
      A synchronizing element will also need to calculate how much time
      elapsed between receiving two consecutive buffers. We call this the
      processing time because that is the amount of time it takes for the
      upstream element to produce/process the buffer. We can compare this
      processing time to the duration of the buffer to have a measurement
      of how fast upstream can produce data, called the proportion.
      If, for example, upstream can produce a buffer in 0.5 seconds of 1
      second long, it is operating at twice the required speed. If, on the
      other hand, it takes 2 seconds to produce a buffer with 1 seconds worth
      of data, upstream is producing buffers too slow and we won't be able to
      keep synchronization. Usually, a running average is kept of the
      proportion.
    </para>
    <para>
      A synchronizing element also needs to measure its own performance in
      order to figure out if the performance problem is upstream of itself.
    </para>
    <para>
      These measurements are used to construct a QOS event that is sent
      upstream. Note that a QoS event is sent for each buffer that arrives
      in the sink.
    </para>
  </sect1>

  <sect1 id="section-handling">
    <title>Handling QoS</title>
    <para>
      An element will have to install an event function on its source pads
      in order to receive QOS events. Usually, the element will need to
      store the value of the QOS event and use them in the data processing
      function. The element will need to use a lock to protect these QoS
      values as shown in the example below. Also make sure to pass the
      QoS event upstream.
    </para>
    <programlisting>
<![CDATA[
    [...]

    case GST_EVENT_QOS:
    {
      GstQOSType type;
      gdouble proportion;
      GstClockTimeDiff diff;
      GstClockTime timestamp;

      gst_event_parse_qos (event, &type, &proportion, &diff, &timestamp);

      GST_OBJECT_LOCK (decoder);
      priv->qos_proportion = proportion;
      priv->qos_timestamp = timestamp;
      priv->qos_diff = diff;
      GST_OBJECT_UNLOCK (decoder);

      res = gst_pad_push_event (decoder->sinkpad, event);
      break;
    }

    [...]
]]>
    </programlisting>
    <para>
      With the QoS values, there are two types of corrections that an element
      can do:
    </para>

    <sect2 id="section-handling-short">
      <title>Short term correction</title>
      <para>
        The timestamp and the jitter value in the QOS event can be used to
        perform a short term correction. If the jitter is positive, the
        previous buffer arrived late and we can be sure that a buffer with
        a timestamp &lt; timestamp + jitter is also going to be late. We
        can thus drop all buffers with a timestamp less than timestamp +
        jitter.
      </para>
      <para>
        If the buffer duration is known, a better estimation for the next
        likely timestamp as: timestamp + 2 * jitter + duration.
      </para>
      <para>
        A possible algorithm typically looks like this:
      </para>
      <programlisting>
<![CDATA[
  [...]

  GST_OBJECT_LOCK (dec);
  qos_proportion = priv->qos_proportion;
  qos_timestamp = priv->qos_timestamp;
  qos_diff = priv->qos_diff;
  GST_OBJECT_UNLOCK (dec);

  /* calculate the earliest valid timestamp */
  if (G_LIKELY (GST_CLOCK_TIME_IS_VALID (qos_timestamp))) {
    if (G_UNLIKELY (qos_diff > 0)) {
      earliest_time = qos_timestamp + 2 * qos_diff + frame_duration;
    } else {
      earliest_time = qos_timestamp + qos_diff;
    }
  } else {
    earliest_time = GST_CLOCK_TIME_NONE;
  }

  /* compare earliest_time to running-time of next buffer */
  if (earliest_time > timestamp)
    goto drop_buffer;

  [...]
]]>
      </programlisting>
    </sect2>

    <sect2 id="section-handling-long">
      <title>Long term correction</title>
      <para>
        Long term corrections are a bit more difficult to perform. They
        rely on the value of the proportion in the QOS event. Elements should
        reduce the amount of resources they consume by the proportion
        field in the QoS message.
      </para>
      <para>
        Here are some possible strategies to achieve this:
      </para>
      <itemizedlist>
        <listitem>
          <para>
            Permanently dropping frames or reducing the CPU or bandwidth
            requirements of the element. Some decoders might be able to
            skip decoding of B frames.
          </para>
        </listitem>
        <listitem>
          <para>
            Switch to lower quality processing or reduce the algorithmic
            complexity. Care should be taken that this doesn't introduce
            disturbing visual or audible glitches.
          </para>
        </listitem>
        <listitem>
          <para>
            Switch to a lower quality source to reduce network bandwidth.
          </para>
        </listitem>
        <listitem>
          <para>
            Assign more CPU cycles to critical parts of the pipeline. This
            could, for example, be done by increasing the thread priority.
          </para>
        </listitem>
      </itemizedlist>
      <para>
        In all cases, elements should be prepared to go back to their normal
        processing rate when the proportion member in the QOS event approaches
        the ideal proportion of 1.0 again.
      </para>
    </sect2>
  </sect1>

  <sect1 id="section-throttle">
    <title>Throttling</title>
    <para>
      Elements synchronizing to the clock should expose a property to configure
      them in throttle mode. In throttle mode, the time distance between buffers
      is kept to a configurable throttle interval. This means that effectively
      the buffer rate is limited to 1 buffer per throttle interval. This can be
      used to limit the framerate, for example.
    </para>
    <para>
      When an element is configured in throttling mode (this is usually only
      implemented on sinks) it should produce QoS events upstream with the jitter
      field set to the throttle interval. This should instruct upstream elements to
      skip or drop the remaining buffers in the configured throttle interval.
    </para>
    <para>
      The proportion field is set to the desired slowdown needed to get the
      desired throttle interval. Implementations can use the QoS Throttle type,
      the proportion and the jitter member to tune their implementations.
    </para>
    <para>
      The default sink base class, has the <quote>throttle-time</quote>
      property for this feature. You can test this with: 
      <command>gst-launch-1.0 videotestsrc !
        xvimagesink throttle-time=500000000</command>
    </para>
  </sect1>

  <sect1 id="section-messages">
    <title>QoS Messages</title>
    <para>
      In addition to the QOS events that are sent between elements in the
      pipeline, there are also QOS messages posted on the pipeline bus to
      inform the application of QoS decisions. The QOS message contains
      the timestamps of when something was dropped along with the amount
      of dropped vs processed items.  Elements must post a QOS
      message under these conditions:
    </para>
    <itemizedlist>
      <listitem>
        <para>
          The element dropped a buffer because of QoS reasons.
        </para>
      </listitem>
      <listitem>
        <para>
          An element changes its processing strategy because of QoS reasons
          (quality). This could include a decoder that decides to drop every
          B frame to increase its processing speed or an effect element
          switching to a lower quality algorithm.
        </para>
      </listitem>
    </itemizedlist>
  </sect1>
  
</chapter>