docs/pwg/advanced-negotiation.xml - mtk-gstreamer-debian - Git at Google

 <chapter id="chapter-negotiation" xreflabel="Caps negotiation">
   <title>Caps negotiation</title>
   <para>
     Caps negotiation is the act of finding a media format (GstCaps) between
     elements that they can handle. This process in &GStreamer; can in most
     cases find an optimal solution for the complete pipeline. In this section
     we explain how this works.
   </para>

   <sect1 id="section-nego-basics">
     <title>Caps negotiation basics</title>
     <para>
       In &GStreamer;, negotiation of the media format always follows the
       following simple rules:
     </para>
     <itemizedlist>
       <listitem>
         <para>
           A downstream element suggest a format on its sinkpad and places the
           suggestion in the result of the CAPS query performed on the sinkpad.
           See also <xref linkend="section-nego-getcaps"/>.
         </para>
       </listitem>
       <listitem>
         <para>
           An upstream element decides on a format. It sends the selected media
           format downstream on its source pad with a CAPS event. Downstream
           elements reconfigure themselves to handle the media type in the CAPS
           event on the sinkpad.
         </para>
       </listitem>
       <listitem>
         <para>
           A downstream element can inform upstream that it would like to
           suggest a new format by sending a RECONFIGURE event upstream. The
           RECONFIGURE event simply instructs an upstream element to restart
           the negotiation phase. Because the element that sent out the
           RECONFIGURE event is now suggesting another format, the format
           in the pipeline might change.
         </para>
       </listitem>
     </itemizedlist>
     <para>
       In addition to the CAPS and RECONFIGURE event and the CAPS query, there
       is an ACCEPT_CAPS query to quickly check if a certain caps can
       be accepted by an element.
     </para>
     <para>
       All negotiation follows these simple rules. Let's take a look at some
       typical uses cases and how negotiation happens.
     </para>
   </sect1>

   <sect1 id="section-nego-usecases">
     <title>Caps negotiation use cases</title>
     <para>
       In what follows we will look at some use cases for push-mode scheduling.
       The pull-mode scheduling negotiation phase is discussed in
       <xref linkend="section-nego-pullmode"/> and is actually similar as we
       will see.
     </para>
     <para>
       Since the sink pads only suggest formats and the source pads need to
       decide, the most complicated work is done in the source pads.
       We can identify 3 caps negotiation use cases for the source pads:
     </para>
     <itemizedlist>
       <listitem>
         <para>
           Fixed negotiation. An element can output one format only.
           See <xref linkend="section-nego-fixed"/>.
         </para>
       </listitem>
       <listitem>
         <para>
           Transform negotiation. There is a (fixed) transform between the
           input and output format of the element, usually based on some
           element property. The caps that the element will produce depend
           on the upstream caps and the caps that the element can accept
           depend on the downstream caps.
           See <xref linkend="section-nego-transform"/>.
         </para>
       </listitem>
       <listitem>
         <para>
           Dynamic negotiation. An element can output many formats.
           See <xref linkend="section-nego-dynamic"/>.
         </para>
       </listitem>
     </itemizedlist>

     <sect2 id="section-nego-fixed">
       <title>Fixed negotiation</title>
       <para>
         In this case, the source pad can only produce a fixed format. Usually
         this format is encoded inside the media. No downstream element can
         ask for a different format, the only way that the source pad will
         renegotiate is when the element decides to change the caps itself.
       </para>
       <para>
         Elements that could implement fixed caps (on their source pads) are,
         in general, all elements that are not renegotiable. Examples include:
       </para>
       <itemizedlist>
         <listitem>
           <para>
             A typefinder, since the type found is part of the actual data stream
             and can thus not be re-negotiated. The typefinder will look at the
             stream of bytes, figure out the type, send a CAPS event with the
             caps and then push buffers of the type.
           </para>
         </listitem>
         <listitem>
           <para>
             Pretty much all demuxers, since the contained elementary data
             streams are defined in the file headers, and thus not
             renegotiable.
           </para>
         </listitem>
         <listitem>
           <para>
             Some decoders, where the format is embedded in the data stream
             and not part of the peercaps <emphasis>and</emphasis> where the
             decoder itself is not reconfigurable, too.
           </para>
         </listitem>
         <listitem>
           <para>
             Some sources that produce a fixed format.
           </para>
         </listitem>
       </itemizedlist>
       <para>
         <function>gst_pad_use_fixed_caps()</function> is used on the source
         pad with fixed caps. As long as the pad is not negotiated, the default
         CAPS query will return the caps presented in the padtemplate. As soon
         as the pad is negotiated, the CAPS query will return the negotiated
         caps (and nothing else). These are the relevant code snippets for fixed
         caps source pads.
       </para>
       <programlisting>
 <![CDATA[
 [..]
   pad = gst_pad_new_from_static_template (..);
   gst_pad_use_fixed_caps (pad);
 [..]
 ]]>
       </programlisting>
       <para>
         The fixed caps can then be set on the pad by calling
         <function>gst_pad_set_caps ()</function>.
       </para>
       <programlisting>
 <![CDATA[
 [..]
     caps = gst_caps_new_simple ("audio/x-raw",
         "format", G_TYPE_STRING, GST_AUDIO_NE(F32),
         "rate", G_TYPE_INT, <samplerate>,
         "channels", G_TYPE_INT, <num-channels>, NULL);
     if (!gst_pad_set_caps (pad, caps)) {
       GST_ELEMENT_ERROR (element, CORE, NEGOTIATION, (NULL),
           ("Some debug information here"));
       return GST_FLOW_ERROR;
     }
 [..]
 ]]>
       </programlisting>
       <para>
         These types of elements also don't have a relation between the input
         format and the output format, the input caps simply don't contain the
         information needed to produce the output caps.
       </para>
       <para>
         All other elements that need to be configured for the format should
         implement full caps negotiation, which will be explained in the next
         few sections.
       </para>
     </sect2>

     <sect2 id="section-nego-transform">
       <title>Transform negotiation</title>
       <para>
         In this negotiation technique, there is a fixed transform between
         the element input caps and the output caps. This transformation
         could be parameterized by element properties but not by the
         content of the stream (see <xref linkend="section-nego-fixed"/>
         for that use-case).
       </para>
       <para>
         The caps that the element can accept depend on the (fixed
         transformation) downstream caps. The caps that the element can
         produce depend on the (fixed transformation of) the upstream
         caps.
       </para>
       <para>
         This type of element can usually set caps on its source pad from
         the <function>_event()</function> function on the sink pad when
         it received the CAPS event. This means that the caps transform
         function transforms a fixed caps into another fixed caps.
         Examples of elements include:
       </para>
       <itemizedlist>
         <listitem>
           <para>
             Videobox. It adds configurable border around a video frame
             depending on object properties.
           </para>
         </listitem>
         <listitem>
           <para>
             Identity elements. All elements that don't change the format
             of the data, only the content. Video and audio effects are an
             example. Other examples include elements that inspect the
             stream.
           </para>
         </listitem>
         <listitem>
           <para>
             Some decoders and encoders, where the output format is defined
             by input format, like mulawdec and mulawenc. These decoders
             usually have no headers that define the content of the stream.
             They are usually more like conversion elements.
           </para>
         </listitem>
       </itemizedlist>
       <para>
         Below is an example of a negotiation steps of a typical transform
         element. In the sink pad CAPS event handler, we compute the caps
         for the source pad and set those.
       </para>
       <programlisting>
 <![CDATA[
   [...]

 static gboolean
 gst_my_filter_setcaps (GstMyFilter *filter,
 		       GstCaps *caps)
 {
   GstStructure *structure;
   int rate, channels;
   gboolean ret;
   GstCaps *outcaps;

   structure = gst_caps_get_structure (caps, 0);
   ret = gst_structure_get_int (structure, "rate", &rate);
   ret = ret && gst_structure_get_int (structure, "channels", &channels);
   if (!ret)
     return FALSE;

   outcaps = gst_caps_new_simple ("audio/x-raw",
       "format", G_TYPE_STRING, GST_AUDIO_NE(S16),
       "rate", G_TYPE_INT, rate,
       "channels", G_TYPE_INT, channels, NULL);
   ret = gst_pad_set_caps (filter->srcpad, outcaps);
   gst_caps_unref (outcaps);

   return ret;
 }

 static gboolean
 gst_my_filter_sink_event (GstPad    *pad,
 		          GstObject *parent,
 		          GstEvent  *event)
 {
   gboolean ret;
   GstMyFilter *filter = GST_MY_FILTER (parent);

   switch (GST_EVENT_TYPE (event)) {
     case GST_EVENT_CAPS:
     {
       GstCaps *caps;

       gst_event_parse_caps (event, &caps);
       ret = gst_my_filter_setcaps (filter, caps);
       break;
     }
     default:
       ret = gst_pad_event_default (pad, parent, event);
       break;
   }
   return ret;
 }

   [...]
 ]]>
       </programlisting>
     </sect2>

     <sect2 id="section-nego-dynamic">
       <title>Dynamic negotiation</title>
       <para>
         A last negotiation method is the most complex and powerful dynamic
         negotiation.
       </para>
       <para>
         Like with the transform negotiation in
         <xref linkend="section-nego-transform"/>, dynamic negotiation will
         perform a transformation on the downstream/upstream caps. Unlike the
         transform negotiation, this transform will convert fixed caps to
         unfixed caps. This means that the sink pad input caps can be converted
         into unfixed (multiple) formats. The source pad will have to choose a
         format from all the possibilities. It would usually like to choose a
         format that requires the least amount of effort to produce but it does
         not have to be. The selection of the format should also depend on the
         caps that can be accepted downstream (see a QUERY_CAPS function in
         <xref linkend="section-nego-getcaps"/>).
       </para>
       <para>
         A typical flow goes like this:
       </para>
       <itemizedlist>
         <listitem>
           <para>
             Caps are received on the sink pad of the element.
           </para>
         </listitem>
         <listitem>
           <para>
             If the element prefers to operate in passthrough mode, check
             if downstream accepts the caps with the ACCEPT_CAPS query. If it
             does, we can complete negotiation and we can operate in
             passthrough mode.
           </para>
         </listitem>
         <listitem>
           <para>
             Calculate the possible caps for the source pad.
           </para>
         </listitem>
         <listitem>
           <para>
             Query the downstream peer pad for the list of possible
             caps.
           </para>
         </listitem>
         <listitem>
           <para>
             Select from the downstream list the first caps that you can
             transform to and set this as the output caps. You might have to
             fixate the caps to some reasonable defaults to construct
             fixed caps.
           </para>
         </listitem>
       </itemizedlist>
       <para>
         Examples of this type of elements include:
       </para>
       <itemizedlist>
         <listitem>
           <para>
             Converter elements such as videoconvert, audioconvert, audioresample,
             videoscale, ...
           </para>
         </listitem>
         <listitem>
           <para>
             Source elements such as audiotestsrc, videotestsrc, v4l2src,
             pulsesrc, ...
           </para>
         </listitem>
       </itemizedlist>
       <para>
         Let's look at the example of an element that can convert between
         samplerates, so where input and output samplerate don't have to be
         the same:
       </para>
       <programlisting>
 <![CDATA[
 static gboolean
 gst_my_filter_setcaps (GstMyFilter *filter,
 		       GstCaps *caps)
 {
   if (gst_pad_set_caps (filter->srcpad, caps)) {
     filter->passthrough = TRUE;
   } else {
     GstCaps *othercaps, *newcaps;
     GstStructure *s = gst_caps_get_structure (caps, 0), *others;

     /* no passthrough, setup internal conversion */
     gst_structure_get_int (s, "channels", &filter->channels);
     othercaps = gst_pad_get_allowed_caps (filter->srcpad);
     others = gst_caps_get_structure (othercaps, 0);
     gst_structure_set (others,
       "channels", G_TYPE_INT, filter->channels, NULL);

     /* now, the samplerate value can optionally have multiple values, so
      * we "fixate" it, which means that one fixed value is chosen */
     newcaps = gst_caps_copy_nth (othercaps, 0);
     gst_caps_unref (othercaps);
     gst_pad_fixate_caps (filter->srcpad, newcaps);
     if (!gst_pad_set_caps (filter->srcpad, newcaps))
       return FALSE;

     /* we are now set up, configure internally */
     filter->passthrough = FALSE;
     gst_structure_get_int (s, "rate", &filter->from_samplerate);
     others = gst_caps_get_structure (newcaps, 0);
     gst_structure_get_int (others, "rate", &filter->to_samplerate);
   }

   return TRUE;
 }

 static gboolean
 gst_my_filter_sink_event (GstPad    *pad,
 		          GstObject *parent,
 		          GstEvent  *event)
 {
   gboolean ret;
   GstMyFilter *filter = GST_MY_FILTER (parent);

   switch (GST_EVENT_TYPE (event)) {
     case GST_EVENT_CAPS:
     {
       GstCaps *caps;

       gst_event_parse_caps (event, &caps);
       ret = gst_my_filter_setcaps (filter, caps);
       break;
     }
     default:
       ret = gst_pad_event_default (pad, parent, event);
       break;
   }
   return ret;
 }

 static GstFlowReturn
 gst_my_filter_chain (GstPad    *pad,
 		     GstObject *parent,
 		     GstBuffer *buf)
 {
   GstMyFilter *filter = GST_MY_FILTER (parent);
   GstBuffer *out;

   /* push on if in passthrough mode */
   if (filter->passthrough)
     return gst_pad_push (filter->srcpad, buf);

   /* convert, push */
   out = gst_my_filter_convert (filter, buf);
   gst_buffer_unref (buf);

   return gst_pad_push (filter->srcpad, out);
 }
 ]]>
       </programlisting>
     </sect2>
   </sect1>

   <sect1 id="section-nego-upstream" xreflabel="Upstream caps (re)negotiation">
     <title>Upstream caps (re)negotiation</title>
     <para>
       Upstream negotiation's primary use is to renegotiate (part of) an
       already-negotiated pipeline to a new format. Some practical examples
       include to select a different video size because the size of the video
       window changed, and the video output itself is not capable of rescaling,
       or because the audio channel configuration changed.
     </para>
     <para>
       Upstream caps renegotiation is requested by sending a GST_EVENT_RECONFIGURE
       event upstream. The idea is that it will instruct the upstream element
       to reconfigure its caps by doing a new query for the allowed caps and then
       choosing a new caps. The element that sends out the RECONFIGURE event
       would influence the selection of the new caps by returning the new
       preferred caps from its GST_QUERY_CAPS query function. The RECONFIGURE
       event will set the GST_PAD_FLAG_NEED_RECONFIGURE on all pads that it
       travels over.
     </para>
     <para>
       It is important to note here that different elements actually have
       different responsibilities here:
     </para>
     <itemizedlist>
       <listitem>
         <para>
           Elements that want to propose a new format upstream need to first
           check if the new caps are acceptable upstream with an ACCEPT_CAPS
           query. Then they would send a RECONFIGURE event and be prepared to
           answer the CAPS query with the new preferred format. It should be
           noted that when there is no upstream element that can (or wants)
           to renegotiate, the element needs to deal with the currently
           configured format.
         </para>
       </listitem>
       <listitem>
         <para>
           Elements that operate in transform negotiation according to
           <xref linkend="section-nego-transform"/> pass the RECONFIGURE
           event upstream. Because these elements simply do a fixed transform
           based on the upstream caps, they need to send the event upstream
           so that it can select a new format.
         </para>
       </listitem>
       <listitem>
         <para>
           Elements that operate in fixed negotiation
           (<xref linkend="section-nego-fixed"/>) drop the RECONFIGURE event.
           These elements can't reconfigure and their output caps don't depend
           on the upstream caps so the event can be dropped.
         </para>
       </listitem>
       <listitem>
         <para>
           Elements that can be reconfigured on the source pad (source pads
           implementing dynamic negotiation in
           <xref linkend="section-nego-dynamic"/>) should check its
           NEED_RECONFIGURE flag with
           <function>gst_pad_check_reconfigure ()</function> and it should
           start renegotiation when the function returns TRUE.
         </para>
       </listitem>
     </itemizedlist>
   </sect1>

   <sect1 id="section-nego-getcaps" xreflabel="Implementing a CAPS query function">
     <title>Implementing a CAPS query function</title>
     <para>
       A <function>_query ()</function>-function with the GST_QUERY_CAPS query
       type is called when a peer element would like to know which formats
       this pad supports, and in what order of preference. The return value
       should be all formats that this elements supports, taking into account
       limitations of peer elements further downstream or upstream, sorted by
       order of preference, highest preference first.
     </para>
     <para>
     </para>
     <programlisting>
 <![CDATA[
 static gboolean
 gst_my_filter_query (GstPad *pad, GstObject * parent, GstQuery * query)
 {
   gboolean ret;
   GstMyFilter *filter = GST_MY_FILTER (parent);

   switch (GST_QUERY_TYPE (query)) {
     case GST_QUERY_CAPS
     {
       GstPad *otherpad;
       GstCaps *temp, *caps, *filt, *tcaps;
       gint i;

       otherpad = (pad == filter->srcpad) ? filter->sinkpad :
                                            filter->srcpad;
       caps = gst_pad_get_allowed_caps (otherpad);

       gst_query_parse_caps (query, &filt);

       /* We support *any* samplerate, indifferent from the samplerate
        * supported by the linked elements on both sides. */
       for (i = 0; i < gst_caps_get_size (caps); i++) {
         GstStructure *structure = gst_caps_get_structure (caps, i);

         gst_structure_remove_field (structure, "rate");
       }

       /* make sure we only return results that intersect our
        * padtemplate */
       tcaps = gst_pad_get_pad_template_caps (pad);
       if (tcaps) {
         temp = gst_caps_intersect (caps, tcaps);
         gst_caps_unref (caps);
         gst_caps_unref (tcaps);
         caps = temp;
       }
       /* filter against the query filter when needed */
       if (filt) {
         temp = gst_caps_intersect (caps, filt);
         gst_caps_unref (caps);
         caps = temp;
       }
       gst_query_set_caps_result (query, caps);
       gst_caps_unref (caps);
       ret = TRUE;
       break;
     }
     default:
       ret = gst_pad_query_default (pad, parent, query);
       break;
   }
   return ret;
 }
 ]]>
     </programlisting>
   </sect1>

   <sect1 id="section-nego-pullmode">
     <title>Pull-mode Caps negotiation</title>
     <para>
       WRITEME, the mechanism of pull-mode negotiation is not yet fully
       understood.
     </para>

     <para>
       Using all the knowledge you've acquired by reading this chapter, you
       should be able to write an element that does correct caps negotiation.
       If in doubt, look at other elements of the same type in our git
       repository to get an idea of how they do what you want to do.
     </para>
   </sect1>
 </chapter>
	<chapter id="chapter-negotiation" xreflabel="Caps negotiation">
	<title>Caps negotiation</title>
	<para>
	Caps negotiation is the act of finding a media format (GstCaps) between
	elements that they can handle. This process in &GStreamer; can in most
	cases find an optimal solution for the complete pipeline. In this section
	we explain how this works.
	</para>

	<sect1 id="section-nego-basics">
	<title>Caps negotiation basics</title>
	<para>
	In &GStreamer;, negotiation of the media format always follows the
	following simple rules:
	</para>
	<itemizedlist>
	<listitem>
	<para>
	A downstream element suggest a format on its sinkpad and places the
	suggestion in the result of the CAPS query performed on the sinkpad.
	See also <xref linkend="section-nego-getcaps"/>.
	</para>
	</listitem>
	<listitem>
	<para>
	An upstream element decides on a format. It sends the selected media
	format downstream on its source pad with a CAPS event. Downstream
	elements reconfigure themselves to handle the media type in the CAPS
	event on the sinkpad.
	</para>
	</listitem>
	<listitem>
	<para>
	A downstream element can inform upstream that it would like to
	suggest a new format by sending a RECONFIGURE event upstream. The
	RECONFIGURE event simply instructs an upstream element to restart
	the negotiation phase. Because the element that sent out the
	RECONFIGURE event is now suggesting another format, the format
	in the pipeline might change.
	</para>
	</listitem>
	</itemizedlist>
	<para>
	In addition to the CAPS and RECONFIGURE event and the CAPS query, there
	is an ACCEPT_CAPS query to quickly check if a certain caps can
	be accepted by an element.
	</para>
	<para>
	All negotiation follows these simple rules. Let's take a look at some
	typical uses cases and how negotiation happens.
	</para>
	</sect1>

	<sect1 id="section-nego-usecases">
	<title>Caps negotiation use cases</title>
	<para>
	In what follows we will look at some use cases for push-mode scheduling.
	The pull-mode scheduling negotiation phase is discussed in
	<xref linkend="section-nego-pullmode"/> and is actually similar as we
	will see.
	</para>
	<para>
	Since the sink pads only suggest formats and the source pads need to
	decide, the most complicated work is done in the source pads.
	We can identify 3 caps negotiation use cases for the source pads:
	</para>
	<itemizedlist>
	<listitem>
	<para>
	Fixed negotiation. An element can output one format only.
	See <xref linkend="section-nego-fixed"/>.
	</para>
	</listitem>
	<listitem>
	<para>
	Transform negotiation. There is a (fixed) transform between the
	input and output format of the element, usually based on some
	element property. The caps that the element will produce depend
	on the upstream caps and the caps that the element can accept
	depend on the downstream caps.
	See <xref linkend="section-nego-transform"/>.
	</para>
	</listitem>
	<listitem>
	<para>
	Dynamic negotiation. An element can output many formats.
	See <xref linkend="section-nego-dynamic"/>.
	</para>
	</listitem>
	</itemizedlist>

	<sect2 id="section-nego-fixed">
	<title>Fixed negotiation</title>
	<para>
	In this case, the source pad can only produce a fixed format. Usually
	this format is encoded inside the media. No downstream element can
	ask for a different format, the only way that the source pad will
	renegotiate is when the element decides to change the caps itself.
	</para>
	<para>
	Elements that could implement fixed caps (on their source pads) are,
	in general, all elements that are not renegotiable. Examples include:
	</para>
	<itemizedlist>
	<listitem>
	<para>
	A typefinder, since the type found is part of the actual data stream
	and can thus not be re-negotiated. The typefinder will look at the
	stream of bytes, figure out the type, send a CAPS event with the
	caps and then push buffers of the type.
	</para>
	</listitem>
	<listitem>
	<para>
	Pretty much all demuxers, since the contained elementary data
	streams are defined in the file headers, and thus not
	renegotiable.
	</para>
	</listitem>
	<listitem>
	<para>
	Some decoders, where the format is embedded in the data stream
	and not part of the peercaps <emphasis>and</emphasis> where the
	decoder itself is not reconfigurable, too.
	</para>
	</listitem>
	<listitem>
	<para>
	Some sources that produce a fixed format.
	</para>
	</listitem>
	</itemizedlist>
	<para>
	<function>gst_pad_use_fixed_caps()</function> is used on the source
	pad with fixed caps. As long as the pad is not negotiated, the default
	CAPS query will return the caps presented in the padtemplate. As soon
	as the pad is negotiated, the CAPS query will return the negotiated
	caps (and nothing else). These are the relevant code snippets for fixed
	caps source pads.
	</para>
	<programlisting>
	<![CDATA[
	[..]
	pad = gst_pad_new_from_static_template (..);
	gst_pad_use_fixed_caps (pad);
	[..]
	]]>
	</programlisting>
	<para>
	The fixed caps can then be set on the pad by calling
	<function>gst_pad_set_caps ()</function>.
	</para>
	<programlisting>
	<![CDATA[
	[..]
	caps = gst_caps_new_simple ("audio/x-raw",
	"format", G_TYPE_STRING, GST_AUDIO_NE(F32),
	"rate", G_TYPE_INT, <samplerate>,
	"channels", G_TYPE_INT, <num-channels>, NULL);
	if (!gst_pad_set_caps (pad, caps)) {
	GST_ELEMENT_ERROR (element, CORE, NEGOTIATION, (NULL),
	("Some debug information here"));
	return GST_FLOW_ERROR;
	}
	[..]
	]]>
	</programlisting>
	<para>
	These types of elements also don't have a relation between the input
	format and the output format, the input caps simply don't contain the
	information needed to produce the output caps.
	</para>
	<para>
	All other elements that need to be configured for the format should
	implement full caps negotiation, which will be explained in the next
	few sections.
	</para>
	</sect2>

	<sect2 id="section-nego-transform">
	<title>Transform negotiation</title>
	<para>
	In this negotiation technique, there is a fixed transform between
	the element input caps and the output caps. This transformation
	could be parameterized by element properties but not by the
	content of the stream (see <xref linkend="section-nego-fixed"/>
	for that use-case).
	</para>
	<para>
	The caps that the element can accept depend on the (fixed
	transformation) downstream caps. The caps that the element can
	produce depend on the (fixed transformation of) the upstream
	caps.
	</para>
	<para>
	This type of element can usually set caps on its source pad from
	the <function>_event()</function> function on the sink pad when
	it received the CAPS event. This means that the caps transform
	function transforms a fixed caps into another fixed caps.
	Examples of elements include:
	</para>
	<itemizedlist>
	<listitem>
	<para>
	Videobox. It adds configurable border around a video frame
	depending on object properties.
	</para>
	</listitem>
	<listitem>
	<para>
	Identity elements. All elements that don't change the format
	of the data, only the content. Video and audio effects are an
	example. Other examples include elements that inspect the
	stream.
	</para>
	</listitem>
	<listitem>
	<para>
	Some decoders and encoders, where the output format is defined
	by input format, like mulawdec and mulawenc. These decoders
	usually have no headers that define the content of the stream.
	They are usually more like conversion elements.
	</para>
	</listitem>
	</itemizedlist>
	<para>
	Below is an example of a negotiation steps of a typical transform
	element. In the sink pad CAPS event handler, we compute the caps
	for the source pad and set those.
	</para>
	<programlisting>
	<![CDATA[
	[...]

	static gboolean
	gst_my_filter_setcaps (GstMyFilter *filter,
	GstCaps *caps)
	{
	GstStructure *structure;
	int rate, channels;
	gboolean ret;
	GstCaps *outcaps;

	structure = gst_caps_get_structure (caps, 0);
	ret = gst_structure_get_int (structure, "rate", &rate);
	ret = ret && gst_structure_get_int (structure, "channels", &channels);
	if (!ret)
	return FALSE;

	outcaps = gst_caps_new_simple ("audio/x-raw",
	"format", G_TYPE_STRING, GST_AUDIO_NE(S16),
	"rate", G_TYPE_INT, rate,
	"channels", G_TYPE_INT, channels, NULL);
	ret = gst_pad_set_caps (filter->srcpad, outcaps);
	gst_caps_unref (outcaps);

	return ret;
	}

	static gboolean
	gst_my_filter_sink_event (GstPad *pad,
	GstObject *parent,
	GstEvent *event)
	{
	gboolean ret;
	GstMyFilter *filter = GST_MY_FILTER (parent);

	switch (GST_EVENT_TYPE (event)) {
	case GST_EVENT_CAPS:
	{
	GstCaps *caps;

	gst_event_parse_caps (event, &caps);
	ret = gst_my_filter_setcaps (filter, caps);
	break;
	}
	default:
	ret = gst_pad_event_default (pad, parent, event);
	break;
	}
	return ret;
	}

	[...]
	]]>
	</programlisting>
	</sect2>

	<sect2 id="section-nego-dynamic">
	<title>Dynamic negotiation</title>
	<para>
	A last negotiation method is the most complex and powerful dynamic
	negotiation.
	</para>
	<para>
	Like with the transform negotiation in
	<xref linkend="section-nego-transform"/>, dynamic negotiation will
	perform a transformation on the downstream/upstream caps. Unlike the
	transform negotiation, this transform will convert fixed caps to
	unfixed caps. This means that the sink pad input caps can be converted
	into unfixed (multiple) formats. The source pad will have to choose a
	format from all the possibilities. It would usually like to choose a
	format that requires the least amount of effort to produce but it does
	not have to be. The selection of the format should also depend on the
	caps that can be accepted downstream (see a QUERY_CAPS function in
	<xref linkend="section-nego-getcaps"/>).
	</para>
	<para>
	A typical flow goes like this:
	</para>
	<itemizedlist>
	<listitem>
	<para>
	Caps are received on the sink pad of the element.
	</para>
	</listitem>
	<listitem>
	<para>
	If the element prefers to operate in passthrough mode, check
	if downstream accepts the caps with the ACCEPT_CAPS query. If it
	does, we can complete negotiation and we can operate in
	passthrough mode.
	</para>
	</listitem>
	<listitem>
	<para>
	Calculate the possible caps for the source pad.
	</para>
	</listitem>
	<listitem>
	<para>
	Query the downstream peer pad for the list of possible
	caps.
	</para>
	</listitem>
	<listitem>
	<para>
	Select from the downstream list the first caps that you can
	transform to and set this as the output caps. You might have to
	fixate the caps to some reasonable defaults to construct
	fixed caps.
	</para>
	</listitem>
	</itemizedlist>
	<para>
	Examples of this type of elements include:
	</para>
	<itemizedlist>
	<listitem>
	<para>
	Converter elements such as videoconvert, audioconvert, audioresample,
	videoscale, ...
	</para>
	</listitem>
	<listitem>
	<para>
	Source elements such as audiotestsrc, videotestsrc, v4l2src,
	pulsesrc, ...
	</para>
	</listitem>
	</itemizedlist>
	<para>
	Let's look at the example of an element that can convert between
	samplerates, so where input and output samplerate don't have to be
	the same:
	</para>
	<programlisting>
	<![CDATA[
	static gboolean
	gst_my_filter_setcaps (GstMyFilter *filter,
	GstCaps *caps)
	{
	if (gst_pad_set_caps (filter->srcpad, caps)) {
	filter->passthrough = TRUE;
	} else {
	GstCaps othercaps, newcaps;
	GstStructure s = gst_caps_get_structure (caps, 0), others;

	/* no passthrough, setup internal conversion */
	gst_structure_get_int (s, "channels", &filter->channels);
	othercaps = gst_pad_get_allowed_caps (filter->srcpad);
	others = gst_caps_get_structure (othercaps, 0);
	gst_structure_set (others,
	"channels", G_TYPE_INT, filter->channels, NULL);

	/* now, the samplerate value can optionally have multiple values, so
	* we "fixate" it, which means that one fixed value is chosen */
	newcaps = gst_caps_copy_nth (othercaps, 0);
	gst_caps_unref (othercaps);
	gst_pad_fixate_caps (filter->srcpad, newcaps);
	if (!gst_pad_set_caps (filter->srcpad, newcaps))
	return FALSE;

	/* we are now set up, configure internally */
	filter->passthrough = FALSE;
	gst_structure_get_int (s, "rate", &filter->from_samplerate);
	others = gst_caps_get_structure (newcaps, 0);
	gst_structure_get_int (others, "rate", &filter->to_samplerate);
	}

	return TRUE;
	}

	static gboolean
	gst_my_filter_sink_event (GstPad *pad,
	GstObject *parent,
	GstEvent *event)
	{
	gboolean ret;
	GstMyFilter *filter = GST_MY_FILTER (parent);

	switch (GST_EVENT_TYPE (event)) {
	case GST_EVENT_CAPS:
	{
	GstCaps *caps;

	gst_event_parse_caps (event, &caps);
	ret = gst_my_filter_setcaps (filter, caps);
	break;
	}
	default:
	ret = gst_pad_event_default (pad, parent, event);
	break;
	}
	return ret;
	}

	static GstFlowReturn
	gst_my_filter_chain (GstPad *pad,
	GstObject *parent,
	GstBuffer *buf)
	{
	GstMyFilter *filter = GST_MY_FILTER (parent);
	GstBuffer *out;

	/* push on if in passthrough mode */
	if (filter->passthrough)
	return gst_pad_push (filter->srcpad, buf);

	/* convert, push */
	out = gst_my_filter_convert (filter, buf);
	gst_buffer_unref (buf);

	return gst_pad_push (filter->srcpad, out);
	}
	]]>
	</programlisting>
	</sect2>
	</sect1>

	<sect1 id="section-nego-upstream" xreflabel="Upstream caps (re)negotiation">
	<title>Upstream caps (re)negotiation</title>
	<para>
	Upstream negotiation's primary use is to renegotiate (part of) an
	already-negotiated pipeline to a new format. Some practical examples
	include to select a different video size because the size of the video
	window changed, and the video output itself is not capable of rescaling,
	or because the audio channel configuration changed.
	</para>
	<para>
	Upstream caps renegotiation is requested by sending a GST_EVENT_RECONFIGURE
	event upstream. The idea is that it will instruct the upstream element
	to reconfigure its caps by doing a new query for the allowed caps and then
	choosing a new caps. The element that sends out the RECONFIGURE event
	would influence the selection of the new caps by returning the new
	preferred caps from its GST_QUERY_CAPS query function. The RECONFIGURE
	event will set the GST_PAD_FLAG_NEED_RECONFIGURE on all pads that it
	travels over.
	</para>
	<para>
	It is important to note here that different elements actually have
	different responsibilities here:
	</para>
	<itemizedlist>
	<listitem>
	<para>
	Elements that want to propose a new format upstream need to first
	check if the new caps are acceptable upstream with an ACCEPT_CAPS
	query. Then they would send a RECONFIGURE event and be prepared to
	answer the CAPS query with the new preferred format. It should be
	noted that when there is no upstream element that can (or wants)
	to renegotiate, the element needs to deal with the currently
	configured format.
	</para>
	</listitem>
	<listitem>
	<para>
	Elements that operate in transform negotiation according to
	<xref linkend="section-nego-transform"/> pass the RECONFIGURE
	event upstream. Because these elements simply do a fixed transform
	based on the upstream caps, they need to send the event upstream
	so that it can select a new format.
	</para>
	</listitem>
	<listitem>
	<para>
	Elements that operate in fixed negotiation
	(<xref linkend="section-nego-fixed"/>) drop the RECONFIGURE event.
	These elements can't reconfigure and their output caps don't depend
	on the upstream caps so the event can be dropped.
	</para>
	</listitem>
	<listitem>
	<para>
	Elements that can be reconfigured on the source pad (source pads
	implementing dynamic negotiation in
	<xref linkend="section-nego-dynamic"/>) should check its
	NEED_RECONFIGURE flag with
	<function>gst_pad_check_reconfigure ()</function> and it should
	start renegotiation when the function returns TRUE.
	</para>
	</listitem>
	</itemizedlist>
	</sect1>

	<sect1 id="section-nego-getcaps" xreflabel="Implementing a CAPS query function">
	<title>Implementing a CAPS query function</title>
	<para>
	A <function>_query ()</function>-function with the GST_QUERY_CAPS query
	type is called when a peer element would like to know which formats
	this pad supports, and in what order of preference. The return value
	should be all formats that this elements supports, taking into account
	limitations of peer elements further downstream or upstream, sorted by
	order of preference, highest preference first.
	</para>
	<para>
	</para>
	<programlisting>
	<![CDATA[
	static gboolean
	gst_my_filter_query (GstPad pad, GstObject parent, GstQuery * query)
	{
	gboolean ret;
	GstMyFilter *filter = GST_MY_FILTER (parent);

	switch (GST_QUERY_TYPE (query)) {
	case GST_QUERY_CAPS
	{
	GstPad *otherpad;
	GstCaps temp, caps, filt, tcaps;
	gint i;

	otherpad = (pad == filter->srcpad) ? filter->sinkpad :
	filter->srcpad;
	caps = gst_pad_get_allowed_caps (otherpad);

	gst_query_parse_caps (query, &filt);

	/* We support any samplerate, indifferent from the samplerate
	* supported by the linked elements on both sides. */
	for (i = 0; i < gst_caps_get_size (caps); i++) {
	GstStructure *structure = gst_caps_get_structure (caps, i);

	gst_structure_remove_field (structure, "rate");
	}

	/* make sure we only return results that intersect our
	* padtemplate */
	tcaps = gst_pad_get_pad_template_caps (pad);
	if (tcaps) {
	temp = gst_caps_intersect (caps, tcaps);
	gst_caps_unref (caps);
	gst_caps_unref (tcaps);
	caps = temp;
	}
	/* filter against the query filter when needed */
	if (filt) {
	temp = gst_caps_intersect (caps, filt);
	gst_caps_unref (caps);
	caps = temp;
	}
	gst_query_set_caps_result (query, caps);
	gst_caps_unref (caps);
	ret = TRUE;
	break;
	}
	default:
	ret = gst_pad_query_default (pad, parent, query);
	break;
	}
	return ret;
	}
	]]>
	</programlisting>
	</sect1>

	<sect1 id="section-nego-pullmode">
	<title>Pull-mode Caps negotiation</title>
	<para>
	WRITEME, the mechanism of pull-mode negotiation is not yet fully
	understood.
	</para>

	<para>
	Using all the knowledge you've acquired by reading this chapter, you
	should be able to write an element that does correct caps negotiation.
	If in doubt, look at other elements of the same type in our git
	repository to get an idea of how they do what you want to do.
	</para>
	</sect1>
	</chapter>