| Transform elements |
| ------------------ |
| |
| Transform elements transform input buffers to output buffers based |
| on the sink and source caps. |
| |
| An important requirement for a transform is that the ouput caps are completely |
| defined by the input caps and vice versa. This means that a typical decoder |
| element can NOT be implemented with a transform element, this is because the |
| output caps like width and height of the decompessed video frame, for example, |
| are endcoded in the stream and thus not defined by the input caps. |
| |
| Typical transform elements include: |
| |
| - audio convertors (audioconvert, audioresample,...) |
| - video convertors (colorspace, videoscale, ...) |
| - filters (capfilter, volume, colorbalance, ...) |
| |
| The implementation of the transform element has to take care of |
| the following things: |
| |
| - efficient negotiation both up and downstream |
| - efficient buffer alloc and other buffer management |
| |
| Some transform elements can operate in different modes: |
| |
| - passthrough (no changes are done on the input buffers) |
| - in-place (changes made directly to the incomming buffers without requiring a |
| copy or new buffer allocation) |
| - metadata changes only |
| |
| Depending on the mode of operation the buffer allocation strategy might change. |
| |
| The transform element should at any point be able to renegotiate sink and src |
| caps as well as change the operation mode. |
| |
| In addition, the transform element will typically take care of the following |
| things as well: |
| |
| - flushing, seeking |
| - state changes |
| - timestamping, this is typically done by copying the input timestamps to the |
| output buffers but subclasses should be able to override this. |
| - QoS, avoiding calls to the subclass transform function |
| - handle scheduling issues such as push and pull based operation. |
| |
| In the next sections, we will describe the behaviour of the transform element in |
| each of the above use cases. We focus mostly on the buffer allocation strategies |
| and caps negotiation. |
| |
| Processing |
| ---------- |
| |
| A transform has 2 main processing functions: |
| |
| - transform(): |
| |
| Transform the input buffer to the output buffer. The output buffer is |
| guaranteed to be writable and different from the input buffer. |
| |
| - transform_ip(): |
| |
| Transform the input buffer in-place. The input buffer is writable and of |
| bigger or equal size than the output buffer. |
| |
| A transform can operate in the following modes: |
| |
| - passthrough: |
| |
| The element will not make changes to the buffers, buffers are pushed straight |
| through, caps on both sides need to be the same. The element can optionally |
| implement a transform_ip() function to take a look at the data, the buffer |
| does not have to be writable. |
| |
| - in-place: |
| |
| Changes can be made to the input buffer directly to obtain the output buffer. |
| The transform must implement a transform_ip() function. |
| |
| - copy-transform |
| |
| The transform is performed by copying and transforming the input buffer to a |
| new output buffer. The transform must implement a transform() function. |
| |
| When no transform() function is provided, only in-place and passthrough |
| operation is allowed, this means that source and destination caps must be equal |
| or that the source buffer size is bigger or equal than the destination buffer. |
| |
| When no transform_ip() function is provided, only passthrough and |
| copy-transforms are supported. Providing this function is an optimisation that |
| can avoid a buffer copy. |
| |
| When no functions are provided, we can only process in passthrough mode. |
| |
| |
| Negotiation |
| ----------- |
| |
| Typical (re)negotiation of the transform element in push mode always goes from |
| sink to src, this means triggers the following sequence: |
| |
| - the sinkpad receives a buffer with new caps, this triggers the setcaps |
| function on the sinkpad before handing the buffer to transform. |
| - the transform function figures out what it can convert these caps to. |
| - try to see if we can configure the caps unmodified on the peer. We need to |
| do this because we prefer to not do anything. |
| - the transform configures itself to transform from the new sink caps to the |
| target src caps |
| - the transform processes and sets the output caps on the src pad |
| |
| We call this downstream negotiation (DN) and it goes roughly like this: |
| |
| sinkpad transform srcpad |
| setcaps() | | | |
| ------------>| find_transform() | | |
| |------------------->| | |
| | | setcaps() | |
| | |--------------------->| |
| | <configure caps> <-| | |
| |
| |
| These steps configure the element for a transformation from the input caps to |
| the output caps. |
| |
| The transform has 3 function to perform the negotiation: |
| |
| - transform_caps(): |
| |
| Transform the caps on a certain pad to all the possible supported caps on |
| the other pad. The input caps are guaranteed to be a simple caps with just |
| one structure. The caps do not have to be fixed. |
| |
| - fixate_caps(): |
| |
| Given a caps on one pad, fixate the caps on the other pad. The target caps |
| are writable. |
| |
| - set_caps(): |
| |
| Configure the transform for a transformation between src caps and dest |
| caps. Both caps are guaranteed to be fixed caps. |
| |
| If no transform_caps() is defined, we can only perform the identity transform, |
| by default. |
| |
| If no set_caps() is defined, we don't care about caps. In that case we also |
| assume nothing is going to write to the buffer and we don't enforce a writable |
| buffer for the transform_ip function, when present. |
| |
| One common function that we need for the transform element is to find the best |
| transform from one format (src) to another (dest). Since the function is |
| bidirectional, we will use the src->dest negotiation. Some requirements of this |
| function are: |
| |
| - has a fixed src caps |
| - finds a fixed dest caps that the transform element can transform to |
| - the dest caps are compatible and can be accepted by peer elements |
| - the transform function prefers to make src caps == dest caps |
| - the transform function can optionally fixate dest caps. |
| |
| The find_transform() function goes like this: |
| |
| - start from src aps, these caps are fixed. |
| - check if the caps are acceptable for us as src caps. This is usually |
| enforced by the padtemplate of the element. |
| - calculate all caps we can transform too with transform_caps() |
| - if the original caps are a subset of the transforms, try to see if the |
| the caps are acceptable for the peer. If this is possible, we can |
| perform passthrough and make src == dest. This is performed by simply |
| calling gst_pad_peer_accept_caps(). |
| - if the caps are not fixed, we need to fixate it, start by taking the peer |
| caps and intersect with them. |
| - for each of the transformed caps retrieved with transform_caps(): |
| - try to fixate the caps with fixate_caps() |
| - if the caps are fixated, check if the peer accepts them with |
| _peer_accept_caps(), if the peer accepts, we have found a dest caps. |
| - if we run out of caps, we fail to find a transform. |
| - if we found a destination caps, configure the transform with set_caps(). |
| |
| After this negotiation process, the transform element is usually in a steady |
| state. We can identify these steady states: |
| |
| - src and sink pads both have the same caps. Note that when the caps are equal |
| on both pads, the input and output buffers automatically have the same size. |
| The element can operate on the buffers in the following ways: (Same caps, SC) |
| |
| - passthrough: buffers are inspected but no metadata or buffer data |
| is changed. The input buffers don't need to be writable. The input |
| buffer is simply pushed out again without modifications. (SCP) |
| |
| sinkpad transform srcpad |
| chain() | | | |
| ------------>| handle_buffer() | | |
| |------------------->| pad_push() | |
| | |--------------------->| |
| | | | |
| |
| - in-place: buffers are modified in-place, this means that the input |
| buffer is modified to produce a new output buffer. This requires the |
| input buffer to be writable. If the input buffer is not writable, a new |
| buffer has to be allocated with pad-alloc. (SCI) |
| |
| sinkpad transform srcpad |
| chain() | | | |
| ------------>| handle_buffer() | | |
| |------------------->| | |
| | | [!writable] | |
| | | pad-alloc() | |
| | |--------------------->| |
| | [caps-changed] .-| [caps-changed] | |
| | <reconfigure> | | setcaps() | |
| | '>|--------------------->| |
| | .-| | |
| | <transform_ip> | | | |
| | '>| | |
| | | pad_push() | |
| | |--------------------->| |
| | | | |
| |
| - copy transform: a new output buffer is allocated with pad-alloc and data |
| from the input buffer is transformed into the output buffer. (SCC) |
| |
| sinkpad transform srcpad |
| chain() | | | |
| ------------>| handle_buffer() | | |
| |------------------->| | |
| | | pad_alloc() | |
| | |--------------------->| |
| | [caps-changed] .-| [caps-changed] | |
| | <reconfigure> | | setcaps() | |
| | '>|--------------------->| |
| | .-| | |
| | <transform> | | | |
| | '>| | |
| | | pad_push() | |
| | |--------------------->| |
| | | | |
| |
| - src and sink pads have different caps. The element can operate on the |
| buffers in the following way: (Different Caps, DC) |
| |
| - in-place: input buffers are modified in-place. This means that the input |
| buffer has a size that is larger or equal to the output size. The input |
| buffer will be resized to the size of the output buffer. If the input |
| buffer is not writable or the output size is bigger than the input size, |
| we need to pad-alloc a new buffer. (DCI) |
| |
| sinkpad transform srcpad |
| chain() | | | |
| ------------>| handle_buffer() | | |
| |------------------->| | |
| | | [!writable || !size] | |
| | | pad-alloc | |
| | |--------------------->| |
| | [caps-changed] .-| [caps-changed] | |
| | <reconfigure> | | setcaps() | |
| | '>|--------------------->| |
| | .-| | |
| | <transform_ip> | | | |
| | '>| | |
| | | pad_push() | |
| | |--------------------->| |
| | | | |
| |
| - copy transform: a new output buffer is allocated and the data from the |
| input buffer is transformed into the output buffer. The flow is exactly |
| the same as the case with the same-caps negotiation. (DCC) |
| |
| We can immeditatly observe that the copy transform states will need to |
| allocate a buffer from a downstream element using pad-alloc. When the transform |
| element is receiving a non-writable buffer in the in-place state, it will also |
| need to perform a pad-alloc. There is no reason why the passthrough state would |
| perform a pad-alloc. This is important because upstream re-negotiation can only |
| happen when the transform uses pad-alloc for all outgoing buffers. |
| |
| This steady state changes when one of the following actions occur: |
| |
| - the sink pad receives new caps, this triggers the above downstream |
| renegotation process, see above for the flow. |
| - the src pad is instructed to produce new caps because of new caps from |
| pad-alloc, this only happens when the transform calls pad-alloc on the |
| srcpad in order to produce a new output buffer. |
| - the transform element wants to renegotiate (because of changed properties, |
| for example). This essentially clears the current steady state and |
| triggers the downstream and upstream renegotiation process. |
| |
| Parallel to the downstream negotiation process there is an upstream negotiation |
| process. The handling and proxy of buffer-alloc is the most comple part of the |
| transform element. This upstream negotiation process has 3 cases: (UN) |
| |
| - upstream calls the buffer-alloc function of the transform sinkpad and this |
| call is proxied downstream (UNP) |
| - upstream calls the buffer-alloc function of the transform sinkpad, the |
| transform does not proxy the call but returns a buffer itself (UNU) |
| - the transform calls the pad-alloc function downstream to allocate a new |
| output buffer (but not because of a proxied buffer-alloc) (UNA) |
| |
| The case where the pad-alloc is called because an output buffer must be |
| generated in the chain function is handled above in the copy-transform and the |
| in-place transform when the input buffer is not writable or the input buffer |
| size is smaller than the output size. |
| |
| We are left with the last case (proxy an incomming pad-alloc or not). We have 2 |
| possibilities here: |
| |
| - pad-alloc is called with the same caps as are currently being handled by |
| the transform on the sinkcaps. Note that this will only be true when the |
| transform element is completely negotiated because of data processing, see |
| above. Then the element is not yet negotiated, we proceed with the case |
| where sinkcaps are different from thos in the buffer-alloc. |
| |
| * If the transform is using copy-transform, we don't need to proxy because |
| we will call pad-alloc when generating an output buffer. |
| |
| sinkpad transform srcpad |
| buffer_alloc() | | | |
| --------------->| | | |
| | | | |
| |-. [same caps && | | |
| return default | | copy-trans] | | |
| <------------|<' | | |
| | | | |
| |
| * If the transform is using in-place and insize < outsize, we proxy |
| the pad-alloc with the srccaps. If the caps are unmodified, we proxy |
| the buffer after changing the caps and size. |
| |
| sinkpad transform srcpad |
| buffer_alloc() | | | |
| --------------->| | | |
| | [same caps && | | |
| | in-place] | | |
| |------------------->| pad_alloc() | |
| | |--------------------->| |
| | [caps unchanged] | | |
| return | adjust_buffer | | |
| <----------------------------------| | |
| | | | |
| | | | |
| |
| * If the transform is using in-place and insize < outsize, we proxy |
| the pad-alloc with the srccaps. If the caps are modified find the best |
| transform from these new caps and return a buffer of this size/caps |
| instead. |
| |
| sinkpad transform srcpad |
| buffer_alloc() | | | |
| --------------->| | | |
| | [same caps && | | |
| | in-place] | pad-alloc() | |
| |------------------------------------------>| |
| | [caps changed] .-| | |
| | find_transform() | | | |
| return | '>| | |
| <----------------------------------| | |
| | | | |
| |
| * If the transform is using in-place and insize >= outsize, we cannot proxy |
| the pad-alloc because the resulting buffer would be too small to return |
| anyway. |
| |
| * If the transform is using passthrough, we can proxy the pad-alloc to the |
| source pad. If the caps change, find the best transform and return a |
| buffer of those caps and size instead. |
| |
| sinkpad transform srcpad |
| buffer_alloc() | | | |
| --------------->| [same caps && | | |
| | passtrough] | pad-alloc() | |
| |------------------------------------------>| |
| | [caps changed] .-| | |
| | find_transform() | | | |
| return | '>| | |
| <----------------------------------| | |
| | | | |
| |
| - pad-alloc is called with different caps than are currently being handled by |
| the transform on the sinkcaps we have to try to negotiate a new |
| configuration for the transform element. |
| |
| * we perform the standard way to finding a best transform using |
| find_transform() and we call the pad-alloc function with these caps. |
| If we get different caps from pad-alloc, we find the best format to |
| transform these to and return those caps instead. |
| |
| |
| sinkpad transform srcpad |
| buffer_alloc() | | | |
| --------------->| | | |
| | find_transform() | | |
| |------------------->| | |
| | | pad-alloc() | |
| | |--------------------->| |
| return | [caps unchanged] | | |
| <----------------------------------| | |
| | | | |
| | [caps changed] .-| | |
| | find_transform() | | | |
| return | '>| | |
| <----------------------------------| | |
| | | | |
| |
| In order to perform passthrough buffer-alloc or pad-alloc, we need to be able |
| to get the size of the output buffer after the transform. |
| |
| For passthrough buffer-alloc, this is trivial: the input size equals the output |
| size. |
| |
| For the copy transform or the in-place transform we need additional function to |
| retrieve the size. There are two functions: |
| |
| - transform_size() |
| |
| Given a caps and a size on one pad, and a caps on the other pad, calculate |
| the size of the other buffer. This function is able to perform all size |
| transforms and is the prefered method of transforming a size. |
| |
| - get_unit_size() |
| |
| When the input size and output size are always a multiple of eachother |
| (audio conversion, ..) we can define a more simple get_unit_size() function. |
| The transform will use this function to get the same amount of units in the |
| source and destination buffers. |
| |
| For performance reasons, the mapping between caps and size is kept in a cache. |
| |
| |
| Issues |
| ------ |
| |
| passthrough and in-place transforms (with writable buffers) never need to |
| perform a pad-alloc on the srcpad. This means that if upstream negotiation |
| happens, the transform element will never know about it. |
| |
| The transform element will keep therefore track of the allocation pattern of |
| the peer elements. We can see the following cases: |
| |
| - upstream peer calls buffer-alloc on the sinkpad of the transform. In some |
| cases (see above) this call gets proxied or not. |
| |
| - upstream peer does never call buffer-alloc. |
| |
| We will keeps state about this allocation pattern and perform the following in |
| each case respectively: |
| |
| - Upstream calls buffer-alloc: In passthrough and (some) in-place we proxy |
| this call onto the downstream element. If the caps are changed, we mark |
| a flag that we will require a new pad-alloc for the output of the next |
| output buffer. |
| |
| - upstream peer does not call buffer-alloc: We always perform a pad-alloc |
| when processing buffers. We can further optimize by only looking at the |
| returned caps instead of doing a full, needless buffer copy. |
| |
| |
| Use cases |
| --------- |
| |
| videotestsrc ! ximagesink |
| |
| - resizing happens because videotestsrc performs pad-alloc. |
| |
| videotestsrc peer-alloc=0 ! ximagesink |
| |
| - resizing cannot happen because videotestsrc never performs pad-alloc. |
| |
| videotestsrc ! videoscale ! ximagesink |
| |
| - videoscale is initially configured in passthrough mode, pad-alloc from |
| videotestsrc is proxied through videoscale. |
| - pad-alloc will renegotiate a new size in videotestsrc. |
| |
| videotestsrc peer-alloc=0 ! videoscale ! ximagesink |
| |
| - videoscale is initially configured in passthrough mode. |
| - videoscale performs pad-alloc because no buffer-alloc is called on the |
| sinkpad |
| - resizing the videosink makes videoscale perform the scaling. |
| |
| Problematic |
| ----------- |
| |
| filesrc location=~/media/moveyourfeet.mov ! decodebin ! |
| ffmpegcolorspace ! videoscale ! ffmpegcolorspace ! ximagesink -v |
