docs/random/omega/caps2 - imx-gstreamer - Git at Google

 The elementfactory for a given element will contain some information about the capabilities of element's
 pads or potential pads.  An indication will be provided as to whether the pad always exists, always
 exists once data is present, or *might* exist once data is present (the latter case is for things like
 the MPEG system parsers, where an audio stream might or might not exist).


 First, an entirely normal example:
 ----------------------------------

 (-----------)        (----------)        (-------------)
 ! disksrc   !        ! mpg123   !        ! audiosink   !
 !         src        sink     src        sink          !
 !           !        !          !        !             !
 (-----------)        (----------)        (-------------)

 We start with only the disksrc.  The typefind filter is attached to the disksrc, and via typefind magic
 the properties of the disksrc are found to be:

   disksrc->src->caps = {
     "audio/mp3",
     "layer",   GST_CAPS_INT (3),
     "bitrate", GST_CAPS_INT (128),
     NULL
   };

 A look through the plugin registry shows that we have an element called mpg123 that has the following
 caps:

   static GstCapsFactory mpg123_sink_caps = {
     "audio/mp3",
     "layer",   GST_CAPS_INT_RANGE (1, 3),
     "bitrate", GST_CAPS_INT_RANGE (8, 320),
     NULL
   };

 The caps of the disksrc fit within those parameters, so we instantiate an mpg123 and attach it to the
 disksrc.  The connection succeeds negotiation and as a result the mpg123 specifies its output caps as:

   mpg123->src->caps = {
     "audio/raw",
     "format",   GST_CAPS_BITFIELD (S16),
     "depth",    GST_CAPS_INT (16),
     "rate",     GST_CAPS_INT (44100),
     "channels", GST_CAPS_INT (2),
     NULL
   };

 Again from the plugin registry we find an element audiosink that has appropriate caps:

   static GstCapsFactory audiosink_src_caps = {
     "audio/raw",
     "format",   GST_CAPS_BITFIELD (S16,....),
     "depth",    GST_CAPS_INT (16),
     "rate",     GST_CAPS_INT_RANGE (4000, 96000),
     "channels", GST_CAPS_INT_RANGE (1, 2),
     NULL
   };

 A copy of the audiosink is instantiated and attached, negotiation goes smoothly, and we're done.  No
 dataflow has occured, no failure found, etc.  An ideal autoplug.


 Now, a slightly more convoluted example:
 ----------------------------------------

 Start with the same graph:

 (-----------)        (----------)        (-------------)
 ! disksrc   !        ! mpg123   !        ! audiosink   !
 !         src        sink     src        sink          !
 !           !        !          !        !             !
 (-----------)        (----------)        (-------------)

 Run typefind on the disksrc's output, get the same output caps:

   disksrc->src->caps = {
     "audio/mp3",
     "layer",   GST_CAPS_INT (3),
     "bitrate", GST_CAPS_INT (128),
     NULL
   };

 Find and attach mpg123, get the following output caps this time:

   mpg123->src->caps = {
     "audio/raw",
     "format",   GST_CAPS_BITFIELD (S16),
     "depth",    GST_CAPS_INT (16),
     "rate",     GST_CAPS_INT (44100),
     "channels", GST_CAPS_INT (1),
     NULL
   };

 Note that this time we have a mono output.  A look into the audiosink caps shows that we have a match.
 So we instantiate a copy.  Oops.  We now find that the caps for the input pad on our audiosink have
 changed:

   mpg123->src->caps = {
     "audio/raw",
     "format",   GST_CAPS_BITFIELD (S16,...),
     "depth",    GST_CAPS_INT (16),
     "rate",     GST_CAPS_INT (11025, 48000),
     "channels", GST_CAPS_INT (2),
     NULL
   };

 Whoops.  It seems that the sound card we've got in this machine (FIXME how on earth to deal with
 multiple sound cards???) doesn't support mono output *at all*.  This is a problem.  We now find that we
 hae no options as far as directly matching the mpg123 to the audiosink.

 A look through our (ficticious) plugin registry shows at least one element that at least has audio/raw
 on both input and ouput (since both mpg123 and audiosink have open pads with this mime type).  A closerlook shows that its caps are:

   static GstCapsFactory mono2stereo_sink_caps = {
     "audio/raw",
     "channels", GST_CAPS_INT (1),
     NULL
   };
   static GstCapsFactory mono2stereo_src_caps = {
     "audio/raw",
     "channels", GST_CAPS_INT (2),
     NULL
   };

 Wow, that's a perfect match.  Instantiate, attach to mpg123, no problems.  Attach to audiosink, no
 problems.  Done.  When we start up the pipeline, we should get absolutely no callbacks from pads saying
 "help me, I've fallen and..., er, I don't like this buffer!".


 A really messy case:
 --------------------

 Start with a disksrc, typefind it, get the following:

   disksrc->src->caps = {
     "audio/mp3",
     "layer",   GST_CAPS_INT (3),
     "bitrate", GST_CAPS_INT (128),
     NULL
   };

 Look through the plugin registry, find mpg123.  Instantiate it, attach it.  It spits out audio
 parameters as usual:

   mpg123->src->caps = {
     "audio/raw",
     "format",   GST_CAPS_BITFIELD (S16),
     "depth",    GST_CAPS_INT (16),
     "rate",     GST_CAPS_INT (44100),
     "channels", GST_CAPS_INT (2),
     NULL
   };

 Now we instantiate an audiosink plugin.  This time, we're sunk:

   mpg123->src->caps = {
     "audio/raw",
     "format",   GST_CAPS_BITFIELD (S8,U8),
     "depth",    GST_CAPS_INT (8),
     "rate",     GST_CAPS_INT_RANGE (11025, 22050),
     "channels", GST_CAPS_INT (1),
     NULL
   };

 ACK!  It's one of those Disney Sound Source things.  We've got a problem here that isn't obviously
 solvable.  However, there happens to be another mp3 decoder sitting around.  It's got the same
 properties as mpg123, but a lower merit value.  Let's instantiate one and attach it.  We get the
 following output pad caps:

   mp3decoder->src->caps = {
     "audio/raw",
     "format",   GST_CAPS_BITFIELD (S8,S16),
     "depth",    GST_CAPS_INT_RANGE (8,16),
     "rate",     GST_CAPS_INT_RANGE (8000, 44100),
     "channels", GST_CAPS_INT (1,2),
     NULL
   };

 Well, that matches the audiosink.  We try attaching it, and during negotiation the mp3decoder finds
 sufficient common ground with the castrated audiosink and sets its output pad to match the best of the
 options: S8 at 22050 KHz.


 Next to impossible scenario: DVD
 --------------------------------

 Start with a dvdsrc.  It's output pad caps are:

   static GstCapsFactory dvdsrc_src_caps = {
     "video/mpeg",
     "mpegversion",  GST_CAPS_INT (2),
     "systemstream", GST_CAPS_BOOLEAN (TRUE),
     NULL
   };

 The type would be classified as incomplete via some mechanism.  This might cause the autoplug code to go
 and run the typefind function.  It would flesh the type out to the following:

   dvdsrc->src->caps = {
     "video/mpeg",
     "mpegversion",  GST_CAPS_INT (2),
     "systemstream", GST_CAPS_BOOLEAN (TRUE),
     "videostreams", GST_CAPS_INT (1),
     "audiostreams", GST_CAPS_INT (3),
     "bitrate",      GST_CAPS_INT (40960),
     NULL,
   };

 Wow, that helped a lot.  A check through the plugin registry shows that the mpeg2parse will match those
 properties:

   static GstCapsFactory mpeg2parse_sink_caps = {
     "video/mpeg",
     "mpegversion",  GST_CAPS_INT (2),
     "systemstream", GST_CAPS_BOOLEAN (TRUE),
     NULL
   };

 (In retrospect, it may not be necessary to run typefind if there's match this good right away.  Only run
 typefind when there's no exact match.)
 Since there are no output pads yet, we have to actually push data through the pipeline.  The moment a
 buffer or two get to the mpeg2parse element, it promptly goes and creates an output pad, probably of the
 following caps:

   mpeg2parse_video_src_caps = {
     "video/mpeg",
     "mpegversion",  GST_CAPS_RANGE (1,2),
     "systemstream", GST_CAPS_BOOLEAN (FALSE),
     NULL
   };

 This seems to be a task for typefind again.  But since data is flowing, we have to be careful with the
 buffers.  (This is the case in any typefind maneuver, but moreso when one really can't rewind the
 source without consequences)  The autoplug system attaches a special pseudo-element to mpeg2parse's new
 output pad, and attaches the typefind element to the end of that.  The pseudo-element takes the buffer,
 stores it, and passes a copy off to the attached element, in this case typefind.  This repeats until
 typefind has determined the type, at which point the typefind is removed, and the newly found element is
 attached instead.

 The pseudo-element is 'rewound' and the stored buffers flow out and into the newly attached element.
 When the cache of buffers is gone, a signal fires and the autoplug system removes the pseudo-element and
 reconnects the pipeline.

 In this case, the typefind function will find the following:

   mpeg2parse_video_src_caps = {
     "video/mpeg",
     "mpegversion",  GST_CAPS_INT (2),
     "systemstream", GST_CAPS_BOOLEAN (FALSE),
     "bitrate",      GST_CAPS_INT (36864),
     "width",        GST_CAPS_INT (720),
     "height",       GST_CAPS_INT (480),
     "framerate",    GST_CAPS_FLOAT (29.97002997),
     "chromaformat", GST_CAPS_INT (1),		[GST_CAPS_STRING ("4:2:0") ?]
     NULL
   };

 Back to the plugin registry, we find our only choice is mpeg2dec, which has input caps of:

   static GstCapsFactory mpeg2dec_sink_caps = {
     "video/mpeg",
     "mpegversion",  GST_CAPS_RANGE (1,2),
     "systemstream", GST_CAPS_BOOLEAN (FALSE),
     NULL
   };

 Once again it just so happens that we really didn't need to do the typefind at all.  But it can't hurt
 unless the typefind is slow and painful, which we can guess won't be the case since the choices are
 rather limited by the fact that there's already a MIME type attached, meaning we can drastically reduce
 the number of typefind functions we try (down to one, actually).

 However, since we *have* run the typefind, upon attachment of the input pad of mpeg2dec, the output pad
 looks like the following:

   mpeg2dec_src_caps = {
     "video/raw",
     "fourcc",    GST_CAPS_LIST (
 			GST_CAPS_FOURCC ("YV12"),	[identical...]
 			GST_CAPS_FOURCC ("IYUV"),
 			GST_CAPS_FOURCC ("I420"),
                  ),
     "width",     GST_CAPS_INT (720),
     "height",    GST_CAPS_INT (480),
     "framerate", GST_CAPS_FLOAT (29.97002997),
     NULL
   };

 Currently only videosink supports the output of video/raw.  It claims a list of FOURCCs but nothing
 more:

   static GstCapsFactory videosink_sink_caps = {
     "video/raw",
     "fourcc", GST_CAP_LIST ( GST_CAPS_FOURCC ("YV12"),
 		GST_CAPS_FOURCC ("IYUV"), GST_CAPS_FOURCC ("I420"),
 		GST_CAPS_FOURCC ("YUY2"), GST_CAPS_FOURCC ("UYVY"),
 		[ etc... ],
               ),
     NULL
   };

 When instantiated, we potentially have the same problem as with the audiosink: we don't necessarily know
 which hardware output to use.  Somehow we have to solve the problem of setting some element arguments
 before we can get useful information out of them as to the properties.  In this case anyway, if the
 videosink were to find only one output possibility, it would trim the list of FOURCCs it can deal with
 to what the hardware can handle, as well as add further properties:

   videosink_sink_caps = {
     "video/raw",
     "fourcc", GST_CAPS_LIST (GST_CAPS_FOURCC ("YV12"),
 			GST_CAPS_FOURCC ("YUY2"),
               ),
     "width",  GST_CAPS_INT_RANGE (4,1020),
     "height", GST_CAPS_INT_RANGE (4,1020),
     NULL
   };

 We can now connect the mpeg2dec output to the videosink, and we now have displaying video.

 . . . .
	The elementfactory for a given element will contain some information about the capabilities of element's
	pads or potential pads. An indication will be provided as to whether the pad always exists, always
	exists once data is present, or might exist once data is present (the latter case is for things like
	the MPEG system parsers, where an audio stream might or might not exist).


	First, an entirely normal example:
	----------------------------------

	(-----------) (----------) (-------------)
	! disksrc ! ! mpg123 ! ! audiosink !
	! src sink src sink !
	! ! ! ! ! !
	(-----------) (----------) (-------------)

	We start with only the disksrc. The typefind filter is attached to the disksrc, and via typefind magic
	the properties of the disksrc are found to be:

	disksrc->src->caps = {
	"audio/mp3",
	"layer", GST_CAPS_INT (3),
	"bitrate", GST_CAPS_INT (128),
	NULL
	};

	A look through the plugin registry shows that we have an element called mpg123 that has the following
	caps:

	static GstCapsFactory mpg123_sink_caps = {
	"audio/mp3",
	"layer", GST_CAPS_INT_RANGE (1, 3),
	"bitrate", GST_CAPS_INT_RANGE (8, 320),
	NULL
	};

	The caps of the disksrc fit within those parameters, so we instantiate an mpg123 and attach it to the
	disksrc. The connection succeeds negotiation and as a result the mpg123 specifies its output caps as:

	mpg123->src->caps = {
	"audio/raw",
	"format", GST_CAPS_BITFIELD (S16),
	"depth", GST_CAPS_INT (16),
	"rate", GST_CAPS_INT (44100),
	"channels", GST_CAPS_INT (2),
	NULL
	};

	Again from the plugin registry we find an element audiosink that has appropriate caps:

	static GstCapsFactory audiosink_src_caps = {
	"audio/raw",
	"format", GST_CAPS_BITFIELD (S16,....),
	"depth", GST_CAPS_INT (16),
	"rate", GST_CAPS_INT_RANGE (4000, 96000),
	"channels", GST_CAPS_INT_RANGE (1, 2),
	NULL
	};

	A copy of the audiosink is instantiated and attached, negotiation goes smoothly, and we're done. No
	dataflow has occured, no failure found, etc. An ideal autoplug.


	Now, a slightly more convoluted example:
	----------------------------------------

	Start with the same graph:

	(-----------) (----------) (-------------)
	! disksrc ! ! mpg123 ! ! audiosink !
	! src sink src sink !
	! ! ! ! ! !
	(-----------) (----------) (-------------)

	Run typefind on the disksrc's output, get the same output caps:

	disksrc->src->caps = {
	"audio/mp3",
	"layer", GST_CAPS_INT (3),
	"bitrate", GST_CAPS_INT (128),
	NULL
	};

	Find and attach mpg123, get the following output caps this time:

	mpg123->src->caps = {
	"audio/raw",
	"format", GST_CAPS_BITFIELD (S16),
	"depth", GST_CAPS_INT (16),
	"rate", GST_CAPS_INT (44100),
	"channels", GST_CAPS_INT (1),
	NULL
	};

	Note that this time we have a mono output. A look into the audiosink caps shows that we have a match.
	So we instantiate a copy. Oops. We now find that the caps for the input pad on our audiosink have
	changed:

	mpg123->src->caps = {
	"audio/raw",
	"format", GST_CAPS_BITFIELD (S16,...),
	"depth", GST_CAPS_INT (16),
	"rate", GST_CAPS_INT (11025, 48000),
	"channels", GST_CAPS_INT (2),
	NULL
	};

	Whoops. It seems that the sound card we've got in this machine (FIXME how on earth to deal with
	multiple sound cards???) doesn't support mono output at all. This is a problem. We now find that we
	hae no options as far as directly matching the mpg123 to the audiosink.

	A look through our (ficticious) plugin registry shows at least one element that at least has audio/raw
	on both input and ouput (since both mpg123 and audiosink have open pads with this mime type). A closerlook shows that its caps are:

	static GstCapsFactory mono2stereo_sink_caps = {
	"audio/raw",
	"channels", GST_CAPS_INT (1),
	NULL
	};
	static GstCapsFactory mono2stereo_src_caps = {
	"audio/raw",
	"channels", GST_CAPS_INT (2),
	NULL
	};

	Wow, that's a perfect match. Instantiate, attach to mpg123, no problems. Attach to audiosink, no
	problems. Done. When we start up the pipeline, we should get absolutely no callbacks from pads saying
	"help me, I've fallen and..., er, I don't like this buffer!".


	A really messy case:
	--------------------

	Start with a disksrc, typefind it, get the following:

	disksrc->src->caps = {
	"audio/mp3",
	"layer", GST_CAPS_INT (3),
	"bitrate", GST_CAPS_INT (128),
	NULL
	};

	Look through the plugin registry, find mpg123. Instantiate it, attach it. It spits out audio
	parameters as usual:

	mpg123->src->caps = {
	"audio/raw",
	"format", GST_CAPS_BITFIELD (S16),
	"depth", GST_CAPS_INT (16),
	"rate", GST_CAPS_INT (44100),
	"channels", GST_CAPS_INT (2),
	NULL
	};

	Now we instantiate an audiosink plugin. This time, we're sunk:

	mpg123->src->caps = {
	"audio/raw",
	"format", GST_CAPS_BITFIELD (S8,U8),
	"depth", GST_CAPS_INT (8),
	"rate", GST_CAPS_INT_RANGE (11025, 22050),
	"channels", GST_CAPS_INT (1),
	NULL
	};

	ACK! It's one of those Disney Sound Source things. We've got a problem here that isn't obviously
	solvable. However, there happens to be another mp3 decoder sitting around. It's got the same
	properties as mpg123, but a lower merit value. Let's instantiate one and attach it. We get the
	following output pad caps:

	mp3decoder->src->caps = {
	"audio/raw",
	"format", GST_CAPS_BITFIELD (S8,S16),
	"depth", GST_CAPS_INT_RANGE (8,16),
	"rate", GST_CAPS_INT_RANGE (8000, 44100),
	"channels", GST_CAPS_INT (1,2),
	NULL
	};

	Well, that matches the audiosink. We try attaching it, and during negotiation the mp3decoder finds
	sufficient common ground with the castrated audiosink and sets its output pad to match the best of the
	options: S8 at 22050 KHz.



	Next to impossible scenario: DVD
	--------------------------------

	Start with a dvdsrc. It's output pad caps are:

	static GstCapsFactory dvdsrc_src_caps = {
	"video/mpeg",
	"mpegversion", GST_CAPS_INT (2),
	"systemstream", GST_CAPS_BOOLEAN (TRUE),
	NULL
	};

	The type would be classified as incomplete via some mechanism. This might cause the autoplug code to go
	and run the typefind function. It would flesh the type out to the following:

	dvdsrc->src->caps = {
	"video/mpeg",
	"mpegversion", GST_CAPS_INT (2),
	"systemstream", GST_CAPS_BOOLEAN (TRUE),
	"videostreams", GST_CAPS_INT (1),
	"audiostreams", GST_CAPS_INT (3),
	"bitrate", GST_CAPS_INT (40960),
	NULL,
	};

	Wow, that helped a lot. A check through the plugin registry shows that the mpeg2parse will match those
	properties:

	static GstCapsFactory mpeg2parse_sink_caps = {
	"video/mpeg",
	"mpegversion", GST_CAPS_INT (2),
	"systemstream", GST_CAPS_BOOLEAN (TRUE),
	NULL
	};

	(In retrospect, it may not be necessary to run typefind if there's match this good right away. Only run
	typefind when there's no exact match.)
	Since there are no output pads yet, we have to actually push data through the pipeline. The moment a
	buffer or two get to the mpeg2parse element, it promptly goes and creates an output pad, probably of the
	following caps:

	mpeg2parse_video_src_caps = {
	"video/mpeg",
	"mpegversion", GST_CAPS_RANGE (1,2),
	"systemstream", GST_CAPS_BOOLEAN (FALSE),
	NULL
	};

	This seems to be a task for typefind again. But since data is flowing, we have to be careful with the
	buffers. (This is the case in any typefind maneuver, but moreso when one really can't rewind the
	source without consequences) The autoplug system attaches a special pseudo-element to mpeg2parse's new
	output pad, and attaches the typefind element to the end of that. The pseudo-element takes the buffer,
	stores it, and passes a copy off to the attached element, in this case typefind. This repeats until
	typefind has determined the type, at which point the typefind is removed, and the newly found element is
	attached instead.

	The pseudo-element is 'rewound' and the stored buffers flow out and into the newly attached element.
	When the cache of buffers is gone, a signal fires and the autoplug system removes the pseudo-element and
	reconnects the pipeline.

	In this case, the typefind function will find the following:

	mpeg2parse_video_src_caps = {
	"video/mpeg",
	"mpegversion", GST_CAPS_INT (2),
	"systemstream", GST_CAPS_BOOLEAN (FALSE),
	"bitrate", GST_CAPS_INT (36864),
	"width", GST_CAPS_INT (720),
	"height", GST_CAPS_INT (480),
	"framerate", GST_CAPS_FLOAT (29.97002997),
	"chromaformat", GST_CAPS_INT (1), [GST_CAPS_STRING ("4:2:0") ?]
	NULL
	};

	Back to the plugin registry, we find our only choice is mpeg2dec, which has input caps of:

	static GstCapsFactory mpeg2dec_sink_caps = {
	"video/mpeg",
	"mpegversion", GST_CAPS_RANGE (1,2),
	"systemstream", GST_CAPS_BOOLEAN (FALSE),
	NULL
	};

	Once again it just so happens that we really didn't need to do the typefind at all. But it can't hurt
	unless the typefind is slow and painful, which we can guess won't be the case since the choices are
	rather limited by the fact that there's already a MIME type attached, meaning we can drastically reduce
	the number of typefind functions we try (down to one, actually).

	However, since we have run the typefind, upon attachment of the input pad of mpeg2dec, the output pad
	looks like the following:

	mpeg2dec_src_caps = {
	"video/raw",
	"fourcc", GST_CAPS_LIST (
	GST_CAPS_FOURCC ("YV12"), [identical...]
	GST_CAPS_FOURCC ("IYUV"),
	GST_CAPS_FOURCC ("I420"),
	),
	"width", GST_CAPS_INT (720),
	"height", GST_CAPS_INT (480),
	"framerate", GST_CAPS_FLOAT (29.97002997),
	NULL
	};

	Currently only videosink supports the output of video/raw. It claims a list of FOURCCs but nothing
	more:

	static GstCapsFactory videosink_sink_caps = {
	"video/raw",
	"fourcc", GST_CAP_LIST ( GST_CAPS_FOURCC ("YV12"),
	GST_CAPS_FOURCC ("IYUV"), GST_CAPS_FOURCC ("I420"),
	GST_CAPS_FOURCC ("YUY2"), GST_CAPS_FOURCC ("UYVY"),
	[ etc... ],
	),
	NULL
	};

	When instantiated, we potentially have the same problem as with the audiosink: we don't necessarily know
	which hardware output to use. Somehow we have to solve the problem of setting some element arguments
	before we can get useful information out of them as to the properties. In this case anyway, if the
	videosink were to find only one output possibility, it would trim the list of FOURCCs it can deal with
	to what the hardware can handle, as well as add further properties:

	videosink_sink_caps = {
	"video/raw",
	"fourcc", GST_CAPS_LIST (GST_CAPS_FOURCC ("YV12"),
	GST_CAPS_FOURCC ("YUY2"),
	),
	"width", GST_CAPS_INT_RANGE (4,1020),
	"height", GST_CAPS_INT_RANGE (4,1020),
	NULL
	};

	We can now connect the mpeg2dec output to the videosink, and we now have displaying video.

	. . . .