docs/random/negotiation

Some notes on pad negotiation


A "pad link" is a connection between two pads.  It can be in one of
two states, "negotiated" or "not negotiated".  Pad links are created
by gst_pad_link().

A "pad link" is created when two pads are linked using gst_pad_link().
When initially created, the link only specifies a src pad, a sink pad,
and optionally a filter caps provided by the application.

In order to pass data through a link, the peer pads must decide on
what data format to use.  This is called negotiation.  Pads
describe acceptable data formats by using a combination of  pad
template caps and (optionally) a pad->getcaps function.


Negotiation can happen in one of two forms, directed or undirected.
Directed negotiation happens when one element has decided (usually
during negotiation on another pad) to ask for a specific format on
a pad.  This happens when a pad calls gst_pad_try_set_caps().
Undirected negotiation happens when the core decides to negotiate
a link, either due to a state change or a specific application
request.

Steps in undirected negotiation (core view):

 - core checks that both pad's parent elements are in the READY or
   higher state.

 - core calls gst_pad_get_caps() on each pad, intersects the two caps,
   and intersects again with the filter caps.  If the intersection is
   empty, then the pads have no formats in common, and the link fails.

 - If the intersection caps is not fixed, there are multiple possible
   formats that the link could use.  If this is the case, fixate
   functions are called until the caps are fixed.  The fixate functions
   are called by priority -- src application fixate function, sink
   application fixate function, src and sink fixate functions, and
   the default core fixate function.  The application fixate functions
   are implemented by the "fixate" signal on each pad.  The core
   loops through the fixate functions until a fixed caps is decided
   on.

 - Each pad may have a pad_link function, which is called with the
   fixed caps.  The pad_link function has the option of accepting,
   rejecting, or delaying the negotiation.

 - If both pads accept the caps, the link is then negotiated.

Steps in directed negotiation (gst_pad_try_set_caps):

 - the originator is the pad that gst_pad_try_set_caps() is called
   on.

 - the elements owning both pads are assumed to be in a non-NULL state

 - the caps argument of try_set_caps() must be fixed.
 
 - gst_pad_get_caps() is called on the peer pad, and intersected with
   the originating pad's pad template caps and the filter caps.  The
   caps argument is checked to be a subset of the intersection.  (It's
   important that this intersection uses the pad _template_ caps.)

 - Fixate functions don't need to be called, since the caps are
   already fixed.

 - The peer's pad_link function is called.

 - If the peer's pad_link function accepts the caps, the link is then
   negotiated.
 
 - If the peer's pad_link function refuses the caps, and the link had
   already been negotiated, the peer's pad_link function is called
   with the caps of the old negotiation.

 - Note: the originator's pad_link function is _not_ called.  The
   originator must take appropriate alternative steps.


Notes about renegotiation:

 - same as negotiation.  Note that get_caps() functions should always
   ignore the currently negotiated caps of a link.

 - if renegotiation fails, the previous negotiation is still in effect.
   If the renegotiation fails in the last pad_link step, the pad_link
   functions are called with the previously negotiated caps.


Notes for sources and sinks:

 - sources and sinks that talk to hardware may not be able to fully
   describe their available formats, and thus need to rely on pad_link
   functions to test a particular format.  FIXME: currently, the core
   completely fails negotiation if a pad_link function refuses a caps,
   instead of attempting with an alternate caps.

   Example: Assume osssink advertises rate=(int)[8000,48000], but
   the device cannot actually handle rate=44100 (unknown to osssink).
   Assume that the pad_link function is called with rate=44100 --
   ideally, the pad_link function should return GST_PAD_LINK_DELAYED,
   and future calls to getcaps should return {[8000,44099],[44101,
   48000]}.  I don't know how to make this easy and/or work well.


Notes for decoders/demuxers:

 - Decoders will typically negotiate a sink pad, receive some data,
   determine the output format, and call try_set_caps() with the given
   format.  If the output format is non-fixed, gst_pad_renegotiate()
   may be used instead, in order to have the fixate functions choose
   the optimal format.  Note that this requires communication so that
   the pad's getcaps function returns the correct caps.

Notes for converters:

 - Converters change one or more properties of the format of a data
   stream.  A typical converter's getcaps function will call
   gst_pad_get_allowed_caps() for the opposite pad in the element,
   change one or more fields in the caps, and return the result.

 - getcaps function:

   - call gst_pad_get_allowed_caps() on the other pad in the element

   - for each possible format ("A") in the allowed caps, determine all
     the formats ("B") that your converter could convert the original
     format (A) to.  The union of all these formats (all the B's) is
     the caps that should be returned.  (This is how to do it
     _theoretically_, but an optimal implementation will probably be 
     quite different.)
     For example, a simple way to do this for an element that can convert
     a given field of the caps is to remove the field(s) from the structure,
     then intersect with the pad template.

   - As an example, videoscale can convert any sized video to any other
     sized video.  Its getcaps function iterates over each structure in
     the caps, and replaces the width and height with the range
     [1,MAXINT].
 
 - pad_link function:
  
   - the "otherpad" is the opposite pad in the element.

   - extract fields from the caps that are relevant for your converter
     handling the format.  Store these in _local_ variables.  (E.g,
     things like video size, sample rate, etc.)

   - If it's possible to pass buffers through without modifying them
     (passthrough), you should call gst_try_set_caps() with the caps
     that was specified as the parameter to the pad_link function.  If
     this is successful, save the local variables to the element
     structure, perform whatever other setup is necessary for your
     element, and return GST_PAD_LINK_OK.

   - Otherwise, you're not using passthrough, and may need to
     change the caps on the otherpad to match the given format.

   - If the otherpad is not negotiated (!gst_pad_is_negotiated()),
     you shouldn't attempt to set a format on it.  It will eventually
     be negotiated.  Save the local variables to the element structure,
     perform whatever other setup is necessary, and return
     GST_PAD_LINK_OK.

   - At this point, the other pad is already negotiated, but won't
     accept the passthrough format, so you should combine the existing
     negotiated caps on the otherpad and the caps that was the pad link
     argument.  This can either be done using existing information in the
     element that was saved during a previous pad_link call, or you can
     get the information from the negotiated caps
     (gst_pad_get_negotiated_caps()).

     As an example, consider the videoscale element.  Assume that
     videoscale.src has already negotiated "video/x-raw-yuv,
     format=(fourcc)I420, width=320, height=240", and that the sink
     pad's link function is called with "video/x-raw-yuv,
     format=(fourcc)YUY2, width=640, height=480".  Since it's the
     videoscale element, we can have different width and height
     fields on the pads, but the format must be the same.  So we'll
     use the existing negotiated size (640x480), and the new format,
     and call gst_pad_try_set_caps() with "video/x-raw-yuv,
     format=(fourcc)I420, width=640, height=480".

     This may seem overkill, but most of the time, you'll end up
     calling try_set_caps() with the same caps that are currently
     negotiated -- try_set_caps() just returns GST_PAD_LINK_OK in
     this case.

   - If gst_pad_try_set_caps() returns GST_PAD_LINK_OK, save the
     local variables to the element structure.  In any case, return
     the return value of gst_pad_try_set_caps().


Notes for filters:

 - Filters can almost always use gst_pad_proxy_getcaps() as the
   getcaps function.  This just returns gst_pad_get_allowed_caps()
   on the otherpad.
 
 - You may be able to use gst_pad_proxy_pad_link() as the pad link
   function, but only if you don't need to extract parameters from
   the caps.


Notes for encoders/muxers:

 - Encoders and muxers should roughly work like converters.  Many
   converters are symmetric; encoders and muxers obvious are not,
   thus it may make the code clearer to have separate src and sink
   getcaps and pad_link functions.

 - Encoders and muxers should handle multiple negotiations until
   the first buffer has been passed.  After this point, it's unlikely
   that additional negotiations will happen in well-constructed
   pipelines, but it may be wise to "lock" the caps after the
   muxer has committed to a format.  (FIXME: it's still unclear to
   me when the caps should get "unlocked".  Obviously at EOS or
   PAUSED->READY transitions.  Any others?)

 - Locking caps can be done by adding (near the top) of the getcaps
   function:

   if (my_element->lock_caps) {
     return gst_pad_get_negotiated_caps (pad);
   }


Explicit caps:

 - There's a hack in the core to make the code for decoder elements
   a lot simpler.  This hack can be used only for src pads of elements
   that get their srcpad capabilities directly from the data stream,
   i.e., decoders, demuxers, and typefind.  This hack overrides the
   pad's getcaps() and pad_link() function, so that they work correctly
   in all decoder states.

 - To enable this hack on a pad, call gst_pad_use_explicit_caps().

 - To indicate that a decoder has found the format of the stream, call
   gst_pad_set_explicit_caps(pad,caps) with the caps of the stream.
   This caps must be fixed.

 - To indicate that a decoder has lost the format of the stream, i.e.,
   there's been a NEW_MEDIA event, call gst_pad_set_explicit_caps(pad,
   NULL).

 - If the explicit caps are set, the getcaps function will return that
   caps, and the pad_link function will return GST_PAD_LINK_OK.  If
   the explicit caps are not set, the getcaps function returns the pad
   template caps, and the pad_link function returns GST_PAD_LINK_DELAYED.



Other junk:

 - negotiation can happen at any time

 - negotiation can happen multiple times/often happens multiple times

 - initial negotiation can lead to strange caps

 - directed negotiation can happen in either direction (src to sink or
   sink to src)

 - Other considerations ignored, every pad should have a getcaps function.

 - If a pad's getcaps function returns the same caps in every
   circumstance, the getcaps function can be omitted.
   
 - If you use gst_pad_use_explicit_caps(), the getcaps function must
   be ommitted.

 - fixate functions are a method for applications to exert influence
   on how a format is chosen from a caps.  It's also used as a hack to
   allow elements to do the same.  Element fixate functions are _not_
   intended to give good results for applications -- they're intended
   to give non-disgusting results in gst-launch.  Don't attempt to
   make them do more than they're capable of.

 - Fixate functions should not be implemented on anything except source
   and sink elements.