docs/random/wtay/negotiation3 - imx-gstreamer - Git at Google

 Caps Negotiation
 ================


 Definitions
 -----------

 - GstCaps is a structure that holds a mimetype and a set of properties. The properties are
   either fixed or non-fixed, they contain ranges or lists.

 - filter caps: caps set by the application to constrain a link to a specific GstCaps.

 - allowed caps: caps calculated when connecting elements, these are the types that are
   possible on this connection. If no types are possible, the link is refused.

 - pad template caps: caps put on padtemplates to describe the possible media types of
   the pad.

 - pad caps: the caps of the pad as it is currently negotiated.


 General overview
 ----------------

 Caps negotiation is the process of two pads agreeing on a common fixed GstCaps structure
 that describes the media type that will be exchanged between those pads.

 We define that caps negotiation only happens in the PLAYING state, when there is actual
 dataflow and elements thus know what kind of data they are dealing with, or when
 linking elements.

 Caps negotiation is first performed when linking elements. If the elements can agree
 on a media type at link time then negotiation will not have to be performed at runtime.
 Usually this is not possible, so the pads remain unnegotiated until at runtime.

 Caps negotiation is initiated by an element that wants to send data on a non-negotiated
 pad, thus the source pad. The core only provides a policy and convenience methods to aid
 the element in performing the negotiation. The core does not keep track of what pads
 are negotiated or not, this is a task of the pads.

 Caps negotiation is normally started by the source element right before it sends out
 data (case 1).
 Caps negotiation can also be redone by a sink element that wants to receive another
 format from a downstream element (case 2).

 There are two functions handling the negotiation, the link function and the do_link
 function. The link function is always called first and must eventually call do_link on
 the peer pad.  The link function must figure out a compatible format for the connection
 and then call the do_link function on the peer pad. The do_link function can only
 accept or refuse the provided caps.

 For autopluggers it is important to know when the pad is ready to start the negotiation.
 It is also inportant to know when the negotiation failed and it must be possible to
 restart the negotiation with another element. This functionality will be provided
 with signals.

 Pad functions
 -------------

 !
 ! const GstCaps* gst_pad_iterate_caps (GstPad *pad, gint position);
 !
 Returns the nth possible caps that describes the media type that can flow over
 this pad. This function should not return static caps but caps that are
 dependent of the media type and the peer connections of the element.

 This function can be called at any time so that an autoplugger can know the
 exact types of the pads at any time.

 !
 ! gboolean gst_pad_accept_caps (GstPad *pad, GstCaps *caps);
 !
 Checks that a given caps is acceptable for this pad. Returns FALSE if the
 pad cannot handle the caps.

 !
 ! gboolean gst_pad_link (GstPad *pad, GstPad *peer, GstCaps *caps);
 !
 Tells the pad to start negotiation with the given filtercaps. The
 caps need not be fixed and serves as a filter for performing the negotiation
 with the peerpad.

 !
 ! gboolean gst_pad_do_link (GstPad *pad, GstPad *peer, GstCaps *caps);
 !
 Configures the pad to accept data with the given caps.  The caps must be fixed.

 !
 ! const GstCaps* gst_pad_get_negotiated_caps (GstPad *pad);
 !
 Get the negotiated caps of a pad or NULL if the pad is not negotiated.


 Linking Elements
 ----------------

 When linking elements with the gst_pad_link function, the core will call
 the link function on the srcpad. If that pad returns GST_PAD_LINK_OK,
 the link is established, else the link fails.

 Since the link function of the pad will call the do_link function of
 the peerpad, the caps will be set on both pads.

 It is not required to decide on a caps at link time, a plugin can choose to
 dynamically renegotiate at runtime.

 When a link fails, the core emits a signal link_failed, which the application
 can catch to try to establish a new link with another element, for example.

 !
 ! def gst_pad_link_filtered (pad1, pad2, filtercaps):
 !
 !   ... get srcpad and sinkpad ...
 !   srcpad  = (pad1 == GST_PAD_SRC ? pad1 : pad2);
 !   sinkpad = (pad1 == GST_PAD_SINK ? pad1 : pad2);
 !
 !   ... more checks to see if the pads are of different types and
 !   ... that they live in the same scheduler etc...
 !
 !   res = gst_pad_link (srcpad, sinkpad, filtercaps)
 !   if (res == GST_PAD_LINK_OK)
 !     ... perform other setup ...
 !   else
 !     res = signal_emit (srcpad, "link_failed")
 !
 !   return res
 !

 Pad link is just a convenience function that passes a NULL filtercaps:
 !
 ! def gst_pad_link (pad1, pad2):
 !   gst_pad_link_filtered (pad1, pad2, NULL)
 !


 Dynamic renegotiation
 ---------------------

 Dynamic renegotiation happens at runtime when the element knows the exact media
 type it is handling.

 The element that wants to redecide on the data type of a connection just calls
 gst_pad_relink on one of it's pads. The core will call unlink and link on the
 pads again as if this were a new connection. Since the iterate_caps function
 returns other values while linking, the new link will renegotiate to a new
 format or the link will fail.

 Prototype of the relink function:
 !
 ! def gst_pad_relink_filtered (pad, filtercaps):
 !   gst_pad_unlink (pad, peerpad)
 !   gst_pad_link_filtered (pad, peerpad, filtercaps)
 !

 Again the relink function is a convenience function for not having to pass
 filtercaps.
 !
 ! def gst_pad_relink (pad):
 !   gst_pad_relink_filtered (pad, NULL)
 !

 The core, however, should optimize the relink function so that it does a minimal
 amount of work, like not informing the scheduler about the unlink/link call in
 case of success.

 Error recovery
 --------------

 When a pad is ready to negotiate and it has no peer pad, it fires the "need-link"
 signal. Autopluggers can use this signal to start plugging elements to the pad.

 When a link fails because of a renegotiation, the "link-failed" signal is fired
 so that autopluggers can try other elements.


 Default implementations
 -----------------------

 !
 ! gst_pad_src_link_func (pad, peerpad, filtercaps)
 ! {
 !   srcpad->negotiated_caps = NULL;
 !   gboolean possible = FALSE;
 !
 !   for (i=0; peercaps = gst_pad_iterate_caps (peerpad, i); i++)
 !   {
 !     for (j=0; srccaps = gst_pad_iterate_caps (srcpad, j); j++)
 !     {
 !       test = gst_caps_intersect (srccaps, peercaps, filtercaps);
 !       if (test == GST_CAPS_EMPTY)
 !         continue;
 !
 !       /* non empty caps, something is possible */
 !       possible = TRUE;
 !
 !       if (!gst_caps_is_fixed (caps))
 !         continue;
 !
 !       if (gst_pad_accepts_caps (peerpad, test))
 !       {
 !         if (gst_pad_do_link (peerpad, srcpad, test))
 !         {
 !           srcpad->negotiated_caps = test;
 !           goto done;
 !         }
 !         else
 !         {
 !           /* weird, it accepted but did not want to link */
 !         }
 !       }
 !       else
 !       {
 !         /* caps are not accepted by peer, try next */
 !       }
 !     }
 !   }
 ! done:
 !   if (!possible)
 !     return GST_PAD_LINK_FAILED;
 !   else
 !     return GST_PAD_LINK_OK;
 ! }
 !

 gst_pad_iterate_caps returns caps in the following order:

  1) prefered caps (if any)
  2) negotiated caps (if any)
  3) profile caps (if any, filtered against caps of current media type)
  4) padtemplate caps (filtered against caps of current media type)

 1 = a caps describing the media type that would result in optimal
     processing of the current media type
 2 = a caps that it is currently handling
 3 = a caps describing the user configured default values
 4 = a caps describing all other possibilities that this pad can
     produce or consume given the current media type if any.


 generic flow 1:

  Linkage

          src                         sink
           |                           |
           | <--- link(sink, A)        |
 check if  |                           |
 A is ok   |                           |
 or filter |                           |
 against   |                           |
 allowed   |                           |
 src caps  |                           |
           |  iterate_caps             |
 if A not  +-------------------------> |
 fixed     |                           |
           | <-------------------------+
 filter    |                           |
 against A |                           |
 to get A' |                           |
           |  can_accept(A')           |
 if A' fix +-------------------------> |
           |                           | check if A' is ok
           |        yes                |
           | <-------------------------+
           |                           |
 if A' not |                           |
 fixed,    |                           |
 A'=null   |                           |
           |  do_link(src, A')         |
           +-------------------------> |
           |        ok                 | store src, A'
           | <-------------------------+
 store     |                           |
 sink, A'  |                           |


 Unlink

          src                         sink
           |                           |
           | <--- unlink()             |
           |                           |
           |  unlink()                 |
           +-------------------------> |
 unref     |                           | unref src, format
 sink, fmt |                           |


   Dynamic negotiation of B

          src                         sink
           |  can_accept(B)            |
           +-------------------------> |
           |                           | check if B is ok
           |        yes                |
           | <-------------------------+
           |                           |
           |       do_link(B)          |
           +-------------------------> |
           |        ok                 | store B
           | <-------------------------+
 store     |                           |
 B         |                           |


 TODO
 -----

 Write test objects to simulate different behaviour.


 tests (in python):

    vts -> colorspace -> ximagsink

   relink with different colorspaces while running


    vts -> videoscale -> ximagesink

   relink to different sizes while running, scaling vts or ximagesink.

    sinesrc -> audiorate -> audiosink

   where audiosink only has one possible fixed caps.
	Caps Negotiation
	================


	Definitions
	-----------

	- GstCaps is a structure that holds a mimetype and a set of properties. The properties are
	either fixed or non-fixed, they contain ranges or lists.

	- filter caps: caps set by the application to constrain a link to a specific GstCaps.

	- allowed caps: caps calculated when connecting elements, these are the types that are
	possible on this connection. If no types are possible, the link is refused.

	- pad template caps: caps put on padtemplates to describe the possible media types of
	the pad.

	- pad caps: the caps of the pad as it is currently negotiated.


	General overview
	----------------

	Caps negotiation is the process of two pads agreeing on a common fixed GstCaps structure
	that describes the media type that will be exchanged between those pads.

	We define that caps negotiation only happens in the PLAYING state, when there is actual
	dataflow and elements thus know what kind of data they are dealing with, or when
	linking elements.

	Caps negotiation is first performed when linking elements. If the elements can agree
	on a media type at link time then negotiation will not have to be performed at runtime.
	Usually this is not possible, so the pads remain unnegotiated until at runtime.

	Caps negotiation is initiated by an element that wants to send data on a non-negotiated
	pad, thus the source pad. The core only provides a policy and convenience methods to aid
	the element in performing the negotiation. The core does not keep track of what pads
	are negotiated or not, this is a task of the pads.

	Caps negotiation is normally started by the source element right before it sends out
	data (case 1).
	Caps negotiation can also be redone by a sink element that wants to receive another
	format from a downstream element (case 2).

	There are two functions handling the negotiation, the link function and the do_link
	function. The link function is always called first and must eventually call do_link on
	the peer pad. The link function must figure out a compatible format for the connection
	and then call the do_link function on the peer pad. The do_link function can only
	accept or refuse the provided caps.

	For autopluggers it is important to know when the pad is ready to start the negotiation.
	It is also inportant to know when the negotiation failed and it must be possible to
	restart the negotiation with another element. This functionality will be provided
	with signals.

	Pad functions
	-------------

	!
	! const GstCaps* gst_pad_iterate_caps (GstPad *pad, gint position);
	!
	Returns the nth possible caps that describes the media type that can flow over
	this pad. This function should not return static caps but caps that are
	dependent of the media type and the peer connections of the element.

	This function can be called at any time so that an autoplugger can know the
	exact types of the pads at any time.

	!
	! gboolean gst_pad_accept_caps (GstPad pad, GstCaps caps);
	!
	Checks that a given caps is acceptable for this pad. Returns FALSE if the
	pad cannot handle the caps.

	!
	! gboolean gst_pad_link (GstPad pad, GstPad peer, GstCaps *caps);
	!
	Tells the pad to start negotiation with the given filtercaps. The
	caps need not be fixed and serves as a filter for performing the negotiation
	with the peerpad.

	!
	! gboolean gst_pad_do_link (GstPad pad, GstPad peer, GstCaps *caps);
	!
	Configures the pad to accept data with the given caps. The caps must be fixed.

	!
	! const GstCaps* gst_pad_get_negotiated_caps (GstPad *pad);
	!
	Get the negotiated caps of a pad or NULL if the pad is not negotiated.


	Linking Elements
	----------------

	When linking elements with the gst_pad_link function, the core will call
	the link function on the srcpad. If that pad returns GST_PAD_LINK_OK,
	the link is established, else the link fails.

	Since the link function of the pad will call the do_link function of
	the peerpad, the caps will be set on both pads.

	It is not required to decide on a caps at link time, a plugin can choose to
	dynamically renegotiate at runtime.

	When a link fails, the core emits a signal link_failed, which the application
	can catch to try to establish a new link with another element, for example.

	!
	! def gst_pad_link_filtered (pad1, pad2, filtercaps):
	!
	! ... get srcpad and sinkpad ...
	! srcpad = (pad1 == GST_PAD_SRC ? pad1 : pad2);
	! sinkpad = (pad1 == GST_PAD_SINK ? pad1 : pad2);
	!
	! ... more checks to see if the pads are of different types and
	! ... that they live in the same scheduler etc...
	!
	! res = gst_pad_link (srcpad, sinkpad, filtercaps)
	! if (res == GST_PAD_LINK_OK)
	! ... perform other setup ...
	! else
	! res = signal_emit (srcpad, "link_failed")
	!
	! return res
	!

	Pad link is just a convenience function that passes a NULL filtercaps:
	!
	! def gst_pad_link (pad1, pad2):
	! gst_pad_link_filtered (pad1, pad2, NULL)
	!


	Dynamic renegotiation
	---------------------

	Dynamic renegotiation happens at runtime when the element knows the exact media
	type it is handling.

	The element that wants to redecide on the data type of a connection just calls
	gst_pad_relink on one of it's pads. The core will call unlink and link on the
	pads again as if this were a new connection. Since the iterate_caps function
	returns other values while linking, the new link will renegotiate to a new
	format or the link will fail.

	Prototype of the relink function:
	!
	! def gst_pad_relink_filtered (pad, filtercaps):
	! gst_pad_unlink (pad, peerpad)
	! gst_pad_link_filtered (pad, peerpad, filtercaps)
	!

	Again the relink function is a convenience function for not having to pass
	filtercaps.
	!
	! def gst_pad_relink (pad):
	! gst_pad_relink_filtered (pad, NULL)
	!

	The core, however, should optimize the relink function so that it does a minimal
	amount of work, like not informing the scheduler about the unlink/link call in
	case of success.

	Error recovery
	--------------

	When a pad is ready to negotiate and it has no peer pad, it fires the "need-link"
	signal. Autopluggers can use this signal to start plugging elements to the pad.

	When a link fails because of a renegotiation, the "link-failed" signal is fired
	so that autopluggers can try other elements.


	Default implementations
	-----------------------

	!
	! gst_pad_src_link_func (pad, peerpad, filtercaps)
	! {
	! srcpad->negotiated_caps = NULL;
	! gboolean possible = FALSE;
	!
	! for (i=0; peercaps = gst_pad_iterate_caps (peerpad, i); i++)
	! {
	! for (j=0; srccaps = gst_pad_iterate_caps (srcpad, j); j++)
	! {
	! test = gst_caps_intersect (srccaps, peercaps, filtercaps);
	! if (test == GST_CAPS_EMPTY)
	! continue;
	!
	! /* non empty caps, something is possible */
	! possible = TRUE;
	!
	! if (!gst_caps_is_fixed (caps))
	! continue;
	!
	! if (gst_pad_accepts_caps (peerpad, test))
	! {
	! if (gst_pad_do_link (peerpad, srcpad, test))
	! {
	! srcpad->negotiated_caps = test;
	! goto done;
	! }
	! else
	! {
	! /* weird, it accepted but did not want to link */
	! }
	! }
	! else
	! {
	! /* caps are not accepted by peer, try next */
	! }
	! }
	! }
	! done:
	! if (!possible)
	! return GST_PAD_LINK_FAILED;
	! else
	! return GST_PAD_LINK_OK;
	! }
	!

	gst_pad_iterate_caps returns caps in the following order:

	1) prefered caps (if any)
	2) negotiated caps (if any)
	3) profile caps (if any, filtered against caps of current media type)
	4) padtemplate caps (filtered against caps of current media type)

	1 = a caps describing the media type that would result in optimal
	processing of the current media type
	2 = a caps that it is currently handling
	3 = a caps describing the user configured default values
	4 = a caps describing all other possibilities that this pad can
	produce or consume given the current media type if any.


	generic flow 1:

	Linkage

	src sink
	\| \|
	\| <--- link(sink, A) \|
	check if \| \|
	A is ok \| \|
	or filter \| \|
	against \| \|
	allowed \| \|
	src caps \| \|
	\| iterate_caps \|
	if A not +-------------------------> \|
	fixed \| \|
	\| <-------------------------+
	filter \| \|
	against A \| \|
	to get A' \| \|
	\| can_accept(A') \|
	if A' fix +-------------------------> \|
	\| \| check if A' is ok
	\| yes \|
	\| <-------------------------+
	\| \|
	if A' not \| \|
	fixed, \| \|
	A'=null \| \|
	\| do_link(src, A') \|
	+-------------------------> \|
	\| ok \| store src, A'
	\| <-------------------------+
	store \| \|
	sink, A' \| \|



	Unlink

	src sink
	\| \|
	\| <--- unlink() \|
	\| \|
	\| unlink() \|
	+-------------------------> \|
	unref \| \| unref src, format
	sink, fmt \| \|


	Dynamic negotiation of B

	src sink
	\| can_accept(B) \|
	+-------------------------> \|
	\| \| check if B is ok
	\| yes \|
	\| <-------------------------+
	\| \|
	\| do_link(B) \|
	+-------------------------> \|
	\| ok \| store B
	\| <-------------------------+
	store \| \|
	B \| \|



	TODO
	-----

	Write test objects to simulate different behaviour.


	tests (in python):

	vts -> colorspace -> ximagsink

	relink with different colorspaces while running


	vts -> videoscale -> ximagesink

	relink to different sizes while running, scaling vts or ximagesink.

	sinesrc -> audiorate -> audiosink

	where audiosink only has one possible fixed caps.