| Fixing Threading |
| ---------------- |
| |
| 1) Observations |
| |
| The following observations are made when considering the current (17/11/2004) |
| problems in gstreamer. |
| |
| - Bin state changes. |
| Currently the state of a bin is determined by the highest state of the |
| children, This is in particular a problem for GstThread because a thread |
| should start/stop spinning at any time depending on the state of a child. |
| |
| ex 1: |
| |
| +-------------------------------------+ |
| | GstThread | |
| | +--------+ +---------+ +------+ | |
| | | src | | decoder | | sink | | |
| | | src-sink src-sink | | |
| | +--------+ +---------+ +------+ | |
| +-------------------------------------+ |
| |
| When performing the state change on the GstThread to PLAYING, one of the |
| children (at random) will go to PLAYING first, this will trigger a method |
| in GstThread that will start spinning the thread. Some elements are not yet |
| in the PLAYING state when the scheduler starts iterating elements. This |
| is not a clean way to start the data passing. |
| |
| State changes also trigger negotiation and scheduling (in the other thread) |
| can do too. This creates races in negotiation. |
| |
| - ERROR and EOS conditions triggering a state change |
| |
| A typical problem is also that since scheduling starts while the state change |
| happens, it is possible that the elements go to EOS or ERROR before the |
| state change completes. Currently this makes the elements go to PAUSED again, |
| creating races with the state change in progress. This also gives the |
| impression to the core that the state change failed. |
| |
| - no locking whatsoever |
| |
| When an element does a state change, it is possible for another thread to |
| perform a conflicting state change. |
| |
| - negotiation is not designed to work over multithread boundaries. |
| |
| negotiation over a queue is not possible. There is no method or policy of |
| discovering a media type and then commiting it. It is also not possible to |
| tie the negotiated media to the relevant buffer. |
| |
| ex1: |
| it Should be possible to queue the old and the new formats in a queue. |
| The element connected to the sinkpad of the queue should be able to |
| find out that the new format will be accepted by the element connected |
| on the srcpad of the queue, even if that element is streaming the old |
| format. |
| |
| +------------------------------+ |
| | GstQueue | |
| | +++++++++++++++++++++++++ | |
| -sink |B|B|B|B|B|B|A|A|A|A|A|A| src- |
| | +++++++++++++++++++++++++ | |
| +------------------------------+ |
| +----------+ +----------+ |
| buffers in buffers in |
| new format old format |
| |
| - element properties are not threadsafe |
| |
| When setting an element property while streaming, the element does no |
| locking whatsoever to guarantee its internal consistency. |
| |
| - No control over streaming. |
| |
| When some GstThread is iterating and you want to reconnect a pad, there |
| is no way to block the pad, perform the actions and then unblock it |
| again. This leads to thread problems where a pad is negotiation at the |
| same time that it is passing data. |
| |
| This is currently solved by PAUSING the pipeline or performing the actions |
| in the same threadcontext as the iterate loop. |
| |
| - race conditions in synchronizing the clocks and spinning up the pipeline. |
| Currently the clock is started as soon as the pipeline is set to playing. |
| Because some time elaspes before the elements are negotiated, autoplugged |
| and streaming, the first frame/sample almost always arrives late at the |
| sinks. Hacks exist to adjust the element base time to compensate for the |
| delay but this clearly is not clean. |
| |
| - race conditions when performing seeks in the pipeline. Since the elements |
| have no control over the streaming threads, they cannot block them or |
| resync them to the new seek position. It is also hard to synchronize them |
| with the clock. |
| |
| - race conditions when sending tags and error messages up the pipeline |
| hierarchy. These races are either caused by glib refcounting problems and |
| by not properly locking. |
| |
| - more as changes are implemented and testcases are written |
| |
| 2) possible solutions |
| |
| - not allowing threading at all |
| |
| Run the complete pipeline in a single thread. Complications to solve include |
| handling of blocking operations like source elements blocking in kernel |
| space, sink elements blocking on the clock or kernel space, etc.. In practice, |
| all operations should be made non-blocking because a blocking element can |
| cause the rest of the pipeline to block as well and cause it to miss a deadline. |
| A non-blocking model needs cooperation from the kernel (with callbacks) or |
| requires the use of a polling mechanism, both of which are either impractical |
| or too CPU intensive and in general not achievable for a general purpose |
| Multimedia framework. For this reason we will not go further with this |
| solution. |
| |
| - Allow threading. |
| |
| To make this work, We propose the following changes: |
| |
| - Remove GstThread, it does not add anything useful in a sense that you cannot |
| arbitrarily place the thread element, it needs decoupled elements around the |
| borders. |
| |
| - Simplify the state changes of bins elements. A bin or element never changes |
| state automatically on EOS and ERROR. |
| |
| - Introduce the concept of the application and the streaming thread. All data |
| passing is done in the streaming thread. This also means that all operations |
| either are performed in the application thread or streaming thread and that |
| they should be protected against competing operations in other threads. |
| This would define a policy for adding appropriate locking. |
| |
| - Move the creation of threads into source and loop-based elements. This will |
| make it possible for the elements in control of the threads to perform the |
| locking when needed. One particular instance is for example the state changes, |
| by creating the threads in the element, it is possible to sync the streaming |
| and the application thread (which does the state change). |
| |
| - Remove negotiation from state changes. This will remove the conflict between |
| streaming and negotiating elements. |
| |
| - add locks around pad operations like negotiation, streaming, linking, etc. This |
| will remove races between these conflicting operations. This will also make it |
| possible to un/block dataflow. |
| |
| - add locks around bin operations like add/removing elements. |
| |
| - add locks around element operations like state changes and property changes. |
| |
| - add a 2-phase directed negotiation process. The source pad queries and figures |
| out what the sinkpad can take in the first phase. In the second phase it sends |
| the new format change as an event to the peer element. This event can be |
| interleaved with the buffers and can travel over queues inbetween the buffers. |
| Need to rethink this wrt bufferpools (see DShow and old bufferpool implementation) |
| |
| - add a preroll phase that will be used to spin up the pipeline and align frames/samples |
| in the sinks. This phase will happen in the PAUSED state. This also means that |
| dataflow will happen in the PAUSED state. Sinks will not sink samples in the PAUSED |
| state but will complete their state change asynchronously. This will allow |
| us to have perfect synchronisation with the clock. |
| |
| - a two phase seek policy. First the event travels upstream, putting all elements in |
| the seeking phase and making them synchronize to the new position. In the |
| second phase the DISCONT event signals the end of the seek and all filters can |
| continue with the new position. |
| |
| - Error messages, EOS, tags and other events in the pipeline should be sent to a |
| mainloop. The app then has an in-thread mechanism for getting information about |
| the pipeline. It should also be possible to get the messages directly from the |
| elements itself, like signals. The application programmer has to know that |
| working these events come from another thread and should handle them accordingly. |
| |
| - Add return values to push/pull so that errors upstream or downstream can be noted |
| by other elements so that they can disable themselves or propagate the error. |
| |
| |
| 3) detailed explanation |
| |
| a) Pipeline construction |
| |
| Pipeline construction includes: |
| |
| - adding/removing elements to the bin |
| - finding elements in a bin by name and interface |
| - setting the clock on a pipeline. |
| - setting properties on objects |
| - linking/unlinking pads |
| |
| These operations should take the object lock to make sure it can be |
| executed from different threads. |
| |
| When connecting pads to other pads from elements inside another bin, |
| we require that the bin has a ghostpad for the pad. This is needed so |
| that the bin looks like a self-contained element. |
| |
| not allowed: |
| +---------------------+ |
| | GstBin | |
| +---------+ | +--------+ | |
| | element | | | src | | |
| sink src------sink src- ... | |
| +---------+ | +--------+ | |
| +---------------------+ |
| |
| allowed: |
| +-----------------------+ |
| | GstBin | |
| | +--------+ | |
| +---------+ | | src | | |
| | element | | sink src- ... | |
| sink src---sink/ +--------+ | |
| +---------+ +-----------------------+ |
| |
| This requirement is important when we need to sort the elements in the |
| bin to perfrom the state change. |
| |
| |
| testcases: |
| |
| - create a bin, add/remove elements from it |
| - add/remove from different threads and check the bin integrity. |
| |
| b) state changes |
| |
| An element can be in one of the four states NULL, READY, PAUSED, PLAYING. |
| |
| NULL: starting state of the element |
| READY: element is ready to start running. |
| PAUSED: element is streaming data, has opened devices etc. |
| PLAYING: element is streaming data and clock is running |
| |
| Note that data starts streaming even in the PAUSED state. The only difference |
| between the PAUSED and PLAYING state is that the clock is running in the |
| PLAYING state. This mostly has an effect on the renderers which will block on |
| the first sample they receive when in PAUSED mode. The transition from |
| READY->PAUSED is called the preroll state. During that transition, media is |
| queued in the pipeline and autoplugging is done. |
| |
| Elements are put in a new state using the _set_state function. This function |
| can return the following return values: |
| |
| typedef enum { |
| GST_STATE_FAILURE = 0, |
| GST_STATE_PARTIAL = 1, |
| GST_STATE_SUCCESS = 2, |
| GST_STATE_ASYNC = 3 |
| } GstElementStateReturn; |
| |
| GST_STATE_FAILURE is returned when the element failed to go to the |
| required state. When dealing with a bin, this is returned when one |
| of the elements failed to go to the required state. The other elements |
| in the bin might have changed their states succesfully. This return |
| value means that the element did _not_ change state, for bins this |
| means that not all children have changed their state. |
| |
| GST_STATE_PARTIAL is returned when some elements in a bin where in the |
| locked state and therefore did not change their state. Note that the |
| state of the bin will be changed regardless of this PARTIAL return value. |
| |
| GST_STATE_SUCCES is returned when all the elements successfully changed their |
| states. |
| |
| GST_STATE_ASYNC is returned when an element is going to report the success |
| or failure of a state change later. |
| |
| The state of a bin is not related to the state of its children but only to |
| the last state change directly performed on the bin or on a parent bin. This |
| means that changing the state of an element inside the bin does not affect |
| the state of the bin. |
| |
| Setting the state on a bin that is already in the correct state will |
| perform the requested state change on the children. |
| |
| Elements are not allowed to change their own state. For bins, it is allowed |
| to change the state of its children. This means that the application |
| can only know about the states of the elements it has explicitly set. |
| |
| There is a difference in the way a pipeline and a bin handles the state |
| change of its children: |
| |
| - a bin returns GST_STATE_ASYNC when one of its children returns an |
| ASYNC reply. |
| |
| - a pipeline never returns GST_STATE_ASYNC but returns from the state |
| change function after all ASYNC elements completed the state change. |
| This is done by polling the ASYNC elements until they return their |
| final state. |
| |
| The state change function must be fast an cannot block. If a blocking behaviour |
| is unavoidable, the state change function must perform an async state change. |
| Sink elements therefore always use async state changes since they need to |
| wait before the first buffer arrives at the sink. |
| |
| A bin has to change the state of its children elements from the sink to the |
| source elements. This makes sure that a sink element is always ready to |
| receive data from a source element in the case of a READY->PAUSED state change. |
| In the case of a PAUSED->READY state, the sink element will be set to READY |
| first so that the source element will receive an error when it tries to push |
| data to this element so that it will shut down as well. |
| |
| For loop based elements we have to be careful since they can pull a buffer |
| from the peer element before it has been put in the right state. |
| The state of a loop based element is therefore only changed after the source |
| element has been put in the new state. |
| |
| c) Element state change functions |
| |
| The core will call the change_state function of an element with the element |
| lock held. The element is responsible for starting any streaming tasks/threads |
| and making sure that it synchronizes them to the state change function if |
| needed. |
| |
| This means that no other thread is allowed to change the state of the element |
| at that time and for bins, it is not possible to add/remove elements. |
| |
| When an element is busy doing the ASYNC state change, it is possible that another |
| state change happens. The elements should be prepared for this. |
| |
| An element can receive a state change for the same state it is in. This |
| is not a problem, some elements (like bins) use this to resynchronize their |
| children. Other elements should ignore this state change and return SUCCESS. |
| |
| When performing a state change on an element that returns ASYNC on one of |
| the state changes, ASYNC is returned and you can only proceed to the next |
| state change change when this ASYNC state change completed. Use the |
| gst_element_get_state function to know when the state change completed. |
| An example of this behaviour is setting a videosink to PLAYING, it will |
| return ASYNC in the state change from READY->PAUSED. You can only set |
| it to PLAYING when this state change completes. |
| |
| Bins will perform the state change code listed in d). |
| |
| For performing the state change, two variables are used: the current state |
| of the element and the pending state. When the element is not performing a |
| state change, the pending state == None. The state change variables are |
| protected by the element lock. The pending state != None as long as the |
| state change is performed or when an ASYNC state change is running. |
| |
| The core provides the following function for applications and bins to |
| get the current state of an element: |
| |
| bool gst_element_get_state(&state, &pending, timeout); |
| |
| This function will block while the state change function is running inside |
| the element because it grabs the element lock. |
| When the element did not perform an async state change, this function returns |
| TRUE immediatly with the state updated to reflect the current state of the |
| element and pending set to None. |
| When the element performed an async state change, this function will block |
| for the value of timeout and will return TRUE if the element completed the |
| async state change within that timeout, otherwise it returns FALSE, with |
| the current and pending state filled in. |
| |
| The algorithm is like this: |
| |
| bool gst_element_get_state(elem, &state, &pending, timeout) |
| { |
| g_mutex_lock (ELEMENT_LOCK); |
| if (elem->pending != none) { |
| if (!g_mutex_cond_wait(STATE, ELEMENT_LOCK, timeout) { |
| /* timeout triggered */ |
| *state = elem->state; |
| *pending = elem->pending; |
| ret = FALSE; |
| } |
| } |
| if (elem->pending == none) { |
| *state = elem->state; |
| *pending = none; |
| ret = TRUE; |
| } |
| g_mutex_unlock (ELEMENT_LOCK); |
| |
| return ret; |
| } |
| |
| For plugins the following function is provided to commit the pending state, |
| the ELEMENT_LOCK should be held when calling this function: |
| |
| gst_element_commit_state(element) |
| { |
| if (pending != none) { |
| state = pending; |
| pending = none; |
| } |
| g_cond_broadcast (STATE); |
| } |
| |
| For bins the gst_element_get_state() works slightly different. It will run |
| the function on all of its children, as soon as one of the children returns |
| FALSE, the method returns FALSE with the state set to the current bin state |
| and the pending set to pending state. |
| |
| For bins with elements that did an ASYNC state change, the _commit_state() |
| is only executed when actively calling _get_state(). The reason for this is |
| that when a child of the bin commits its state, this is not automatically |
| reported to the bin. This is not a problem since the _get_state() function |
| is the only way to get the current and pending state of the bin and is always |
| consistent. |
| |
| d) bin state change algorithm |
| |
| In order to perform the sink to source state change a bin must be able to sort |
| the elements. To make this easier we require that elements are connected to |
| bins using ghostpads on the bin. |
| |
| The algoritm goes like this: |
| |
| d = [ ] # list of delayed elements |
| p = [ ] # list of pending async elements |
| q = [ elements without srcpads ] # sinks |
| while q not empty do |
| e = dequeue q |
| s = [ all elements connected to e on the sinkpads ] |
| q = q append s |
| if e is entry point |
| d = d append e |
| else |
| r = state change e |
| if r is ASYNC |
| p = p append e |
| done |
| while d not empty do |
| e = dequeue d |
| r = state change e |
| if r is ASYNC |
| p = p append e |
| done |
| # a bin would return ASYNC here if p is not empty |
| |
| # this last part is only performed by a pipeline |
| while p not empty do |
| e = peek p |
| if state completed e |
| dequeue e from p |
| done |
| |
| The algorithm first tries to find the sink elements, ie. ones without |
| sinkpads. Then it changes the state of each sink elements and queues |
| the elements connected to the sinkpads. |
| |
| The entry points (loopbased and getbased elements) are delayed as we |
| first need to change the state of the other elements before we can activate |
| the entry points in the pipeline. |
| |
| The pipeline will poll the async children before returning. |
| |
| e) The GstTask infrastructure |
| |
| A new component: GstTask is added to the core. A task is created by |
| an instance of the abstract GstScheduler class. |
| |
| Each schedulable element (when added to a pipeline) is handed a |
| reference to a GstScheduler. It can use this object to create |
| a GstTask, which is basically a managed wrapper around a threading |
| library like GThread. It should be possible to write a GstScheduler |
| instance that uses other means of scheduling, like one that does not |
| use threads but implements task switching based on mutex locking. |
| |
| When source and loopbased elements want to create the streaming thread |
| they create an instance of a GstTask, which they pass a pointer to |
| a loop-function. This function will be called as soon as the element |
| performs GstTask.start(). The element can stop and uses mutexes to |
| pause the GstTask from, for example, the state change function or the |
| event functions. |
| |
| The GstTasks implement the streaming threads. |
| |
| f) the preroll phase |
| |
| Element start the streaming threads in the READY->PAUSED state. Since |
| the elements that start the threads are put in the PAUSED state last, |
| after their connected elements, they will be able to deliver data to |
| their peers without problems. |
| |
| Sink elements like audio and videosinks will return an async state change |
| reply and will only commit the state change after receiving the first |
| buffer. This will implement the preroll phase. |
| |
| The following pseudo code shows an algorithm for commiting the state |
| change in the streaming method. |
| |
| GST_OBJECT_LOCK (element); |
| /* if we are going to PAUSED, we can commit the state change */ |
| if (GST_STATE_TRANSITION (element) == GST_STATE_READY_TO_PAUSED) { |
| gst_element_commit_state (element); |
| } |
| /* if we are paused we need to wait for playing to continue */ |
| if (GST_STATE (element) == GST_STATE_PAUSED) { |
| |
| /* here we wait for the next state change */ |
| do { |
| g_cond_wait (element->state_cond, GST_OBJECT_GET_LOCK (element)); |
| } while (GST_STATE (element) == GST_STATE_PAUSED); |
| |
| /* check if we got playing */ |
| if (GST_STATE (element) != GST_STATE_PLAYING) { |
| /* not playing, we can't accept the buffer */ |
| GST_OBJECT_UNLOCK (element); |
| gst_buffer_unref (buf); |
| return GST_FLOW_WRONG_STATE; |
| } |
| } |
| GST_OBJECT_UNLOCK (element); |
| |
| |
| |
| g) return values for push/pull |
| |
| To recover from pipeline errors in a more elegant manner than just |
| shutting down the pipeline, we need more finegrained error messages |
| in the data transport. The plugins should be able to know what goes |
| wrong when interacting with their outside environment. This means |
| that gst_pad_push/gst_pad_pull and gst_event_send should return a |
| result code. |
| |
| Possible return values include: |
| |
| - GST_OK |
| - GST_ERROR |
| - GST_NOT_CONNECTED |
| - GST_NOT_NEGOTIATED |
| - GST_WRONG_STATE |
| - GST_UNEXPECTED |
| - GST_NOT_SUPPORTED |
| |
| GST_OK |
| Data transport was successful |
| |
| GST_ERROR |
| An error occured during transport, such as a fatal decoding error, |
| the pad should not be used again. |
| |
| GST_NOT_CONNECTED |
| The pad was not connected |
| |
| GST_NOT_NEGOTIATED |
| The peer does not know what datatype is going over the pipeline. |
| |
| GST_WRONG_STATE |
| The peer pad is not in the correct state. |
| |
| GST_UNEXPECTED |
| The peer pad did not expect the data because it was flushing or |
| received an eos. |
| |
| GST_NOT_SUPPORTED |
| The operation is not supported. |
| |
| The signatures of the functions will become: |
| |
| GstFlowReturn gst_pad_push (GstPad *pad, GstBuffer *buffer); |
| GstFlowReturn gst_pad_pull (GstPad *pad, GstBuffer **buffer); |
| |
| GstResult gst_pad_push_event (GstPad *pad, GstEvent *event); |
| |
| - push_event will send the event to the connected pad. |
| |
| For sending events from the application: |
| |
| GstResult gst_pad_send_event (GstPad *pad, GstEvent *event); |
| |
| h) Negotiation |
| |
| Implement a simple two phase negotiation. First the source queries the |
| sink if it accepts a certain format, then it sends the new format |
| as an event. Sink pads can also trigger a state change by requesting |
| a renegotiation. |
| |
| i) Mainloop integration/GstBus |
| |
| All error, warning and EOS messages from the plugins are sent to an event |
| queue. The pipeline reads the messages from the queue and will either |
| handle them or forward them to the main event queue that is read by the |
| application. |
| |
| Specific pipelines can be written that deal with negotiation messages and |
| errors in the pipeline intelligently. The basic pipeline will stop the |
| pipeline when an error occurs. |
| |
| Whenever an element posts a message on the event queue, a signal is also |
| fired that can be catched by the application. When dealing with those |
| signals the application has to be aware that they come from the streaming |
| threads and need to make sure they use proper locking to protect their |
| own data structures. |
| |
| The messages will be implemented using a GstBus object that allows |
| plugins to post messages and allows the application to read messages either |
| synchronous or asynchronous. It is also possible to integrate the bus in |
| the mainloop. |
| |
| The messages will derive from GstData to make them a lightweight refcounted |
| object. Need to figure out how we can extend this method to encapsulate |
| generic signals in messages too. |
| |
| This decouples the streaming thread from the application thread and should |
| avoid race conditions and pipeline stalling due to application interaction. |
| |
| It is still possible to receive the messages in the streaming thread context |
| if an application wants to. When doing this, special care has to be taken |
| when performing state changes. |
| |
| j) EOS |
| |
| When an element goes to EOS, it sends the EOS event to the peer plugin |
| and stops sending data on that pad. The peer element that received an EOS |
| event on a pad can refuse any buffers on that pad. |
| |
| All elements without source pads must post the EOS message on the message |
| queue. When the pipeline receives an EOS event from all sinks, it will |
| post the EOS message on the application message queue so that the application |
| knows the pipeline is in EOS. Elements without any connected sourcepads |
| should also post the EOS message. This makes sure that all "dead-ends" |
| signalled the EOS. |
| |
| No state change happens when elements go to EOS but the elements with the |
| GstTask will stop their tasks and so stop producing data. |
| |
| An application can issue a seek operation which makes all tasks running |
| again so that they can start streaming from the new location. |
| |
| |
| |
| A) threads and lowlatency |
| |
| People often think it is a sin to use threads in low latency applications. This is true |
| when using the data has to pass thread boundaries but false when it doesn't. Since |
| source and loop based elements create a thread, it is possible to construct a pipeline |
| where data passing has to cross thread boundaries, consider this case: |
| |
| +-----------------------------------+ |
| | +--------+ +--------+ | |
| | |element1| |element2| | |
| | .. -sink src-sink src- .. | |
| | +--------+ +--------+ | |
| +-----------------------------------+ |
| |
| The two elements are loop base and thus create a thread to drive the pipeline. At the |
| border between the two elements there is a mutex to pass the data between the two |
| threads. When using these kinds of element in a pipeline, low-latency will not be |
| possible. For low-latency apps, don't use these constructs! |
| |
| Note that in a typical pipeline with one get-based element and two chain-based |
| elements (decoder/sink) there is only one thread, no data is crossing thread |
| boundaries and thus this pipeline can be low-latency. Also note that while this |
| pipeline is streaming no interaction or locking is done between it and the main |
| application. |
| |
| +-------------------------------------+ |
| | +--------+ +---------+ +------+ | |
| | | src | | decoder | | sink | | |
| | | src-sink src-sink | | |
| | +--------+ +---------+ +------+ | |
| +-------------------------------------+ |
| |
| |
| B) howto make non-threaded pipelines |
| |
| For low latency it is required to not have datapassing cross any thread |
| borders. Here are some pointers for making sure this requirement is met: |
| |
| - never connect a loop or chain based element to a loop based element, this |
| will create a new thread for the sink loop element. |
| |
| - do not use queues or any other decoupled element, as they implicitly |
| create a thread boundary. |
| |
| - At least one thread will be created for any source element (either in the |
| connected loop-based element or in the source itself) unless the source |
| elements are connected to the same loop based element. |
| |
| - when designing sinks, make them non-blocking, use the async clock callbacks |
| to schedule media rendering in the same thread (if any) as the clock. Sinks that |
| provide the clock can be made blocking. |
| |
