| Element types | |
| ------------- | |
| SOURCES | |
| ------- | |
| * never chain-based | |
| * have no sinkpads | |
| 1) get based src | |
| (----------) | |
| ! fakesrc ! | |
| ! src- (get based) | |
| (----------) | |
| * no sinkpads | |
| * srcpad(s) that are get-based | |
| 2) loop based src | |
| (----------) | |
| ! fakesrc ! | |
| ! src- (loop based) | |
| (----------) | |
| * no sinkpads | |
| * element is loop-based | |
| * data is pushed on the srcpad(s) | |
| FILTERS | |
| ------- | |
| 3) chain based filter | |
| (----------) | |
| ! identity ! | |
| -sink src- | |
| (----------) | |
| * sinkpad(s) have a chain function | |
| * srcpad(s) push data | |
| 4) loop-based filter | |
| (----------) | |
| ! identity ! | |
| -sink src- | |
| (----------) | |
| * element is loop-based | |
| * data is pushed on the srcpads | |
| * data is pulled from sinkpad(s) | |
| SINKS | |
| ----- | |
| 5) chain based sink | |
| (----------) | |
| ! fakesink ! | |
| -sink ! | |
| (----------) | |
| * sinkpad(s) have a chain function | |
| * no srcpads | |
| 6) loop-based sink | |
| (----------) | |
| ! fakesink ! | |
| -sink ! | |
| (----------) | |
| * element is loop-based | |
| * data is pulled from sinkpad(s) | |
| DECOUPLED | |
| --------- | |
| 7) decoupled element | |
| (----------) | |
| ! queue ! | |
| -sink src- | |
| (----------) | |
| * sinkpad(s) have chain function | |
| * srcpad(s) have get function | |
| * never loop-based | |
| * always acts like a chain-based sink for upstream elements | |
| * always acts like a get-based src for downstream elements | |
| * are not added to a group, but marked as an entry point in | |
| case it acts as a src element | |
| Connection types | |
| ---------------- | |
| 1) get based src | |
| 2) loop based src | |
| 3) chain based filter | |
| 4) loop-based filter | |
| 5) chain based sink | |
| 6) loop-based sink | |
| 7) decoupled element | |
| ! 1 ! 2 ! 3 ! 4 ! 5 ! 6 ! 7 ! | |
| ---+-----+-----+-----+-----+-----+-----+-----+ | |
| 1 ! X ! X ! A ! C ! A ! C ! A ! | |
| ! ! ! ! ! ! ! ! | |
| 2 ! X ! X ! B ! F ! B ! F ! B ! | |
| ! ! ! ! ! ! ! ! | |
| 3 ! X ! X ! D ! E ! D ! E ! D ! | |
| ! ! ! ! ! ! ! ! | |
| 4 ! X ! X ! B ! F ! B ! F ! B ! | |
| ! ! ! ! ! ! ! ! | |
| 5 ! X ! X ! X ! X ! X ! X ! X ! | |
| ! ! ! ! ! ! ! ! | |
| 6 ! X ! X ! X ! X ! X ! X ! X ! | |
| ! ! ! ! ! ! ! ! | |
| 7 ! X ! X ! A ! C ! A ! C ! X ! | |
| ! ! ! ! ! ! ! ! | |
| A) | |
| src -> sink | |
| src -> filter | |
| src -> decoupled | |
| decoupled -> sink | |
| decoupled -> filter | |
| - get based source | |
| - chain based sink | |
| * one group | |
| * src at start of group and entry point | |
| * _get from src, push to sink | |
| (-group1---------------) | |
| ! ! | |
| ! *fakesrc -> fakesink ! | |
| (----------------------) | |
| B) | |
| src -> sink | |
| src -> filter | |
| src -> decoupled | |
| filter -> sink | |
| filter -> filter | |
| filter -> decoupled | |
| - loop based source/filter | |
| - chain based sink/filter/decoupled | |
| * one group | |
| * src/filter at start of group and entry point | |
| * loop on src, chainhandler set to chain function | |
| (-group1----------------) | |
| ! ! | |
| ! %*fakesrc -> fakesink ! | |
| (-----------------------) | |
| C) | |
| src -> sink | |
| src -> filter | |
| decoupled -> sink | |
| decoupled -> filter | |
| - get based source/decoupled | |
| - loop based sink/filter | |
| * one group | |
| * loop based element is entry point | |
| * loop on sink/filter, gethandler set to getfunction | |
| (-group1----------------) | |
| ! ! | |
| ! fakesrc -> %*fakesink ! | |
| (-----------------------) | |
| D) | |
| filter -> filter | |
| filter -> sink | |
| filter -> decoupled | |
| - chain based filter | |
| - chain based filter/sink/decoupled | |
| * one group is created to hold the two elements | |
| * no entry point | |
| * chainhandler set to peer chainfunction | |
| (-group1----------------) | |
| ! ! | |
| ! identity -> identity ! | |
| (-----------------------) | |
| E) | |
| filter -> filter | |
| filter -> sink | |
| - chain based filter | |
| - loop based filter/sink | |
| * two groups | |
| * group is created for src element if needed | |
| * chainhandler of loop based element set to loop wrapper, control is | |
| handed to the peer group | |
| * gethandler of loop based element set to get wrapper | |
| (-group1---) (-group2------) | |
| ! ! ! ! | |
| ! identity ---> %*identity ! | |
| (----------) (-------------) | |
| F) | |
| src -> filter | |
| src -> sink | |
| filter -> filter | |
| filter -> sink | |
| - loop based filter/src | |
| - loop based filter/sink | |
| * two groups | |
| * two entry points | |
| * chainhandler set to loop wrapper | |
| * gethandler set to get wrapper | |
| (-group1-----) (-group2------) | |
| ! ! ! ! | |
| ! %*identity ---> %*identity ! | |
| (------------) (-------------) | |
| Grouping | |
| -------- | |
| * a group has at most one loop based element | |
| * elements in a group are sorted, src elements first (not mandatory) | |
| * a group has one cothread | |
| * a group is created immediatly for loop based elements, all other elements | |
| are added to a group when a pad connection is made | |
| * get-based plugins are put in the same group as a peer loop based element | |
| * chain based elements are put in the same group as sink peer elements | |
| * entry point in the group is: | |
| - loopbased element | |
| - first src element if no loopbased element exists in the group | |
| Result: you end up with a group of connected elements with either: | |
| - a loop based plugin as the entry point | |
| - a get based plugin as the entry point | |
| Scheduling the group is a matter of starting the cothread and calling | |
| the loop function or doing a _get/_push on a srcpad. | |
| other examples of groups: | |
| ------------------------- | |
| % = loop based | |
| * = entry point of group | |
| . | |
| (-group1---------------) | |
| ! ! | |
| ! *fakesrc -> fakesink ! | |
| (----------------------) | |
| . | |
| (-group1---------------------------------------) | |
| ! ! | |
| ! *fakesrc -> identity -> identity -> fakesink ! | |
| (----------------------------------------------) | |
| . | |
| (-group1-----------------------------------------) | |
| ! ! | |
| ! fakesrc -> %*identity -> identity -> fakesink ! | |
| (------------------------------------------------) | |
| . | |
| (-group1---------------) (-group2-----------------) | |
| ! ! ! ! | |
| ! *fakesrc -> identity --> *%identity -> fakesink ! | |
| (----------------------) (------------------------) | |
| . | |
| (-group1------------------------------------) | |
| ! ! | |
| ! *fakesrc -> tee --> identity -> fakesink ! | |
| ! --> identity -> fakesink ! | |
| (-------------------------------------------) | |
| . | |
| (-group1------------------------------------) | |
| ! ! | |
| ! *fakesrc -> tee --> identity -> fakesink ! | |
| (--------------!----------------------------) | |
| v | |
| (-group2-----------------) | |
| ! ! | |
| ! *%identity -> fakesink ! | |
| (------------------------) | |
| . | |
| (-group1----------) (-group2-----------------) | |
| ! ! ! ! | |
| ! *fakesrc -> tee --> *%identity -> fakesink ! | |
| (--------------!--) (------------------------) | |
| v | |
| (-group3-----------------) | |
| ! ! | |
| ! *%identity -> fakesink ! | |
| (------------------------) | |
| . | |
| (-group1-----------------------) (-group2----------------------) | |
| ! ! ! ! | |
| ! filesrc -> *%mpegdemux --> queue* -> mpeg2dec -> xvideosink ! | |
| ! ! (-----------------------------) | |
| ! ! (-group3----------------------) | |
| ! ! ! ! | |
| ! --> queue* -> mad -> osssink ! | |
| (------------------------------) (-----------------------------) | |
| Chaining | |
| -------- | |
| * groups that are connected end up in the same chain | |
| * a group always belongs to a chain | |
| * updating the chain is only needed when two groups are | |
| connected with a connection of type E/F. for other | |
| connection types, the group itself is updated. | |
| * a chain is scheduled by scheduling a random group in the chain. | |
| Wrapper functions | |
| ----------------- | |
| iterating without cothreads | |
| --------------------------- | |
| A cothread for each group is the easiest way to schedule most of | |
| the pipelines. Some pipelines are however schedulable without | |
| any cothreads. | |
| Each group is schedulable without cothreads, one can call the | |
| group schedule function and be done with it. Problems arise | |
| one the group boundaries of connected elements, which are always | |
| of type E and F (chain->loop, loop->loop) | |
| We always have a producer group and a provider group in this case. | |
| chain->loop | |
| ----------- | |
| Scheduler algorithm | |
| ------------------- | |
| 1: select (random?) group in chain | |
| 2: schedule group | |
| 3: on E/F connections, the get/chain wrapper is called | |
| - get wrapper puts the peer element on the runqueue and | |
| recursively invokes the scheduler. | |
| - chain wrapper puts the buffer in the bufpen and puts | |
| the peer element in the runqueue | |
| 4: when the group is scheduled, take group from the runqueue | |
| and goto 2: | |
| 5: no more groups on the runqueue, iteration ends | |
| NOTES: | |
| - We need a GList instead of a single bufpen to hold buffers | |
| for multi-out elements. | |
| - We probably need to set a limit on the maximum number of | |
| recursions and size of the bufpen list. | |
| - elements run non-preemptively, a group is done scheduling when all | |
| elements in that group complete their chain/loop function. | |
| - can we only have a stack overflow when there is a loop in the | |
| pipeline? I think so. | |
| - putting groups twice on the runqueue is not a good idea, we | |
| need to check a flag or something, maybe give the group a | |
| higher priority? | |
| - what about starvation? We'll probably have to put the group | |
| at the end of the runqueue. | |
| - multi-out elements can introduce significant latency. consider: | |
| (-group1----------) (-group2-----------------) | |
| ! ! ! ! | |
| ! *filesrc -> mad --> *%identity -> osssink ! | |
| (-----------------) (------------------------) | |
| mad produces N output buffers, they are queued in the bufpen, when | |
| group1 is done, group2 is scheduled to empty the bufpen queue. | |
| The time it takes for the first buffer to arrive at osssink equals | |
| the time needed to decode the N-1 other buffers. | |
| Of course, given the right buffersize for filesrc, N can be reduced | |
| to 1. mad can also suggest a buffersize to filesrc with the | |
| BUFFER_SIZE event. | |
| Ungrouping | |
| ---------- | |
| Ungrouping might be the hardest part. especially in the case where | |
| an element is still running (state changes and pipeline modifications | |
| in element callbacks). | |
| Ungrouping involves two tasks: | |
| - breaking up groups. This can happen when a pad connection is broken | |
| so that the group contains two clusters of unconnected elements. | |
| - breaking up chains. This happens when a pad connection is broken | |
| so that the chains contains two clusters of unconnected groups. | |
| case1 | |
| ----- | |
| The most simple case is where two elements are disconnected and one | |
| of the elements does not belong to a group. In this case, we don't need | |
| to do anything at all. | |
| case2 | |
| ----- | |
| if the elements are part of different groups, we need to check if the | |
| chain needs to be broken. A chain is broken if it contains two sets | |
| of unconnected groups. | |
| To test this case, we will create a new chain and recursively move | |
| one of the groups with all of its connected groups to it. If the | |
| origial chain is empty after this operation, there was still a connection | |
| and the new chain replaces the old one, else we end up with two chains. | |
| case3 | |
| ----- | |
| When the elements are part of the same group we check if both elements | |
| still have a connection to some other element in that group. The elements | |
| without connections are removed from the group. | |
| It is possible that when an element still has a connection with some other | |
| element in the group, the group has to be split up anyway. This can happen | |
| in fakesrc ! indentity ! identity ! fakesink when we break the connection | |
| between the two identity elements. We have to be careful here in the cothread | |
| case that we don't take away the running cothread from under the elements. | |
| In the non-cothread case we can just move the elements to another new group. | |
| Interrupt with cothreadless optimal scheduler | |
| --------------------------------------------- | |
| Interrupts are usually performed when a blocking _get based source or | |
| decoupled element is unlocked for a state change. The idea of the | |
| interrupt scheduler call is to return to the main execution stack frame | |
| ASAP so that the state change can take place. For cothread based | |
| implementations of the scheduler this is not a problem as one can jump | |
| to the main cothread context without problems. For non cothread based | |
| schedulers we need to follow this scheme: | |
| get <-> chain based connection. | |
| - enter get group scheduler | |
| - call _get function on source, this can block internally | |
| - app performs state change, element is told to unlock itself | |
| - lock inside the get based function unlocks and _get function | |
| return GST_EVENT_INTERRUPT | |
| - group scheduler notices the event, unrefs it and jumps out of the | |
| get_group_scheduler function, a scheduler INTERRUPTED flag is set. | |
| get <-> loop based connection | |
| - enter loop group scheduler | |
| - loop based element does pull on sinkpad | |
| - _get blocks | |
| - app performs state change, _get is unlocked and returns | |
| GST_EVENT_INTERRUPT. | |
| - loop based function receives INTERRUPT event and exits its loop | |
| ASAP using gst_element_interrupt. | |
| loop/chain <-> loop based connection | |
| - when returning from the recursive call to the scheduler, the state | |
| of the scheduler is checked, if it was interrupted, a | |
| GST_EVENT_INTERRUPTED event is returned to the loop based element. | |
| - the loop based element exits its loop ASAP. | |
| This technique will unwind the stack of scheduled groups ASAP and returns | |
| to the main execution stack frame where the iterate() function can return | |
| and the state change can take place. | |
| Another alternative could be implemented when the _push and _pull functions | |
| would return a result code... | |