docs/design/part-states.txt - imx-gstreamer - Git at Google

 States
 ------

 Both elements and pads can be in different states. The states of the pads are
 linked to the state of the element so the design of the states is mainly
 focused around the element states.

 An element can be in 4 states. NULL, READY, PAUSED and PLAYING. When an element
 is initially instantiated, it is in the NULL state.


 State definitions
 ~~~~~~~~~~~~~~~~~

  - NULL:    This is the initial state of an element.
  - READY:   The element should be prepared to go to PAUSED.
  - PAUSED:  The element should be ready to accept and process data. Sink
             elements however only accept one buffer and then block.
  - PLAYING: The same as PAUSED except for live sources and sinks. Sinks accept
             and rendering data. Live sources produce data.

 We call the sequence NULL->PLAYING an upwards state change and PLAYING->NULL
 a downwards state change.


 State transitions
 ~~~~~~~~~~~~~~~~~

 the following state changes are possible:

  NULL -> READY
    - The element must check if the resources it needs are available.
      Device sinks and -sources typically try to probe the device to constrain
      their caps.
    - The element opens the device, this is needed if the previous step requires
      the device to be opened.

  READY -> PAUSED
    - The element pads are activated in order to receive data in PAUSED.
      Streaming threads are started.
    - Some elements might need to return ASYNC and complete the state change
      when they have enough information. It is a requirement for sinks to
      return ASYNC and complete the state change when they receive the first
      buffer or EOS event (preroll). Sinks also block the dataflow when in PAUSED.
    - A pipeline resets the running_time to 0.
    - Live sources return NO_PREROLL and don't generate data.

  PAUSED -> PLAYING
    - Most elements ignore this state change.
    - The pipeline selects a clock and distributes this to all the children
      before setting them to PLAYING. This means that it is only allowed to
      synchronize on the clock in the PLAYING state.
    - The pipeline uses the clock and the running_time to calculate the base_time.
      The base_time is distributed to all children when performing the state
      change.
    - Sink elements stop blocking on the preroll buffer or event and start
      rendering the data.
    - Sinks can post the EOS message in the PLAYING state. It is not allowed to
      post EOS when not in the PLAYING state.
    - While streaming in PAUSED or PLAYING elements can create and remove
      sometimes pads.
    - Live sources start generating data and return SUCCESS.

  PLAYING -> PAUSED
    - Most elements ignore this state change.
    - The pipeline calculates the running_time based on the last selected clock
      and the base_time. It stores this information to continue playback when
      going back to the PLAYING state.
    - Sinks unblock any clock wait calls.
    - When a sink does not have a pending buffer to play, it returns ASYNC from
      this state change and completes the state change when it receives a new
      buffer or an EOS event.
    - Any queued EOS messages are removed since they will be reposted when going
      back to the PLAYING state. The EOS messages are queued in GstBins.
    - Live sources stop generating data and return NO_PREROLL.

  PAUSED -> READY
    - Sinks unblock any waits in the preroll.
    - Elements unblock any waits on devices
    - Chain or get_range functions return WRONG_STATE.
    - The element pads are deactivated so that streaming becomes impossible and
      all streaming threads are stopped.
    - The sink forgets all negotiated formats
    - Elements remove all sometimes pads

  READY -> NULL
    - Elements close devices
    - Elements reset any internal state.


 State variables
 ~~~~~~~~~~~~~~~

 An element has 4 state variables that are protected with the object LOCK:

   - STATE
   - STATE_NEXT
   - STATE_PENDING
   - STATE_RETURN

 The STATE always reflects the current state of the element.
 The STATE_NEXT reflects the next state the element will go to.
 The STATE_PENDING always reflects the required state of the element.
 The STATE_RETURN reflects the last return value of a state change.

 The STATE_NEXT and STATE_PENDING can be VOID_PENDING if the element is in
 the right state.

 An element has a special lock to protect against concurrent invocations of
 _set_state(), called the STATE_LOCK.


 Setting state on elements
 ~~~~~~~~~~~~~~~~~~~~~~~~~

 The state of an element can be changed with _element_set_state(). When changing
 the state of an element all intermediate states will also be set on the element
 until the final desired state is set.

 The _set_state() function can return 3 possible values:

   GST_STATE_FAILURE: The state change failed for some reason. The plugin should
                      have posted an error message on the bus with information.

   GST_STATE_SUCCESS: The state change is completed successfully.

   GST_STATE_ASYNC:   The state change will complete later on. This can happen
                      when the element needs a long time to perform the state
 		     change or for sinks that need to receive the first buffer
 		     before they can complete the state change (preroll).

   GST_STATE_NO_PREROLL: The state change is completed successfully but the element
                      will not be able to produce data in the PAUSED state.

 In the case of an ASYNC state change, it is possible to proceed to the next
 state before the current state change completed, however, the element will only
 get to this next state before completing the previous ASYNC state change.
 After receiving an ASYNC return value, you can use _element_get_state() to poll
 the status of the element. If the polling returns SUCCESS, the element completed
 the state change to the last requested state with _set_state().

 When setting the state of an element, the STATE_PENDING is set to the required
 state. Then the state change function of the element is called and the result of
 that function is used to update the STATE and STATE_RETURN fields, STATE_NEXT,
 STATE_PENDING and STATE_RETURN fields. If the function returned ASYNC, this result
 is immediately returned to the caller.


 Getting state of elements
 ~~~~~~~~~~~~~~~~~~~~~~~~~

 The _get_state() function takes 3 arguments, two pointers that will hold the
 current and pending state and one GstClockTime that holds a timeout value. The
 function returns a GstElementStateReturn.

  - If the element returned SUCCESS to the previous _set_state() function, this
    function will return the last state set on the element and VOID_PENDING in
    the pending state value. The function returns GST_STATE_SUCCESS.

  - If the element returned NO_PREROLL to the previous _set_state() function, this
    function will return the last state set on the element and VOID_PENDING in
    the pending state value. The function returns GST_STATE_NO_PREROLL.

  - If the element returned FAILURE to the previous _set_state() call, this
    funciton will return FAILURE with the state set to the current state of
    the element and the pending state set to the value used in the last call
    of _set_state().

  - If the element returned ASYNC to the previous _set_state() call, this function
    will wait for the element to complete its state change up to the amount of time
    specified in the GstClockTime.

    * If the element does not complete the state change in the specified amount of
      time, this function will return ASYNC with the state set to the current state
      and the pending state set to the pending state.

    * If the element completes the state change within the specified timeout, this
      function returns the updated state and VOID_PENDING as the pending state.

    * If the element aborts the ASYNC state change due to an error within the
      specified timeout, this function returns FAILURE with the state set to last
      successfull state and pending set to the last attempt. The element should
      also post an error message on the bus with more information about the problem.


 States in GstBin
 ~~~~~~~~~~~~~~~~

 A GstBin manages the state of its children. It does this by propagating the state
 changes performed on it to all of its children.  The _set_state() function on a
 bin will call the _set_state() function on all of its children, that are
 not already in the target state or in a change state to the target state.

 The children are iterated from the sink elements to the source elements. This makes
 sure that when changing the state of an element, the downstream elements are in
 the correct state to process the eventual buffers. In the case of a downwards
 state change, the sink elements will shut down first which makes the upstream
 elements shut down as well since the _push() function returns a GST_FLOW_WRONG_STATE
 error.

 If all the children return SUCCESS, the function returns SUCCESS as well.

 If one of the children returns FAILURE, the function returns FAILURE as well. In
 this state it is possible that some elements successfuly changed state. The
 application can check which elements have a changed state, which were in error
 and which were not affected by iterating the elements and calling _get_state()
 on the elements.

 If after calling the state function on all children, one of the children returned
 ASYNC, the function returns ASYNC as well.

 If after calling the state function on all children, one of the children returned
 NO_PREROLL, the function returns NO_PREROLL as well.

 If both NO_PREROLL and ASYNC children are present, NO_PREROLL is returned.

 The current state of the bin can be retrieved with _get_state().

 If the bin is performing an ASYNC state change, it will automatically update its
 current state fields when it receives state messages from the children.


 Implementing states in elements
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 READY
 ^^^^^


 upward state change
 ~~~~~~~~~~~~~~~~~~~

 Upward state changes always return ASYNC either if the STATE_PENDING is
 reached or not.

 Element:

   A -> B => SUCCESS
     - commit state

   A -> B => ASYNC
     - no commit state
     - element commits state ASYNC

   A -> B while ASYNC
     - update STATE_PENDING state
     - no commit state
     - no change_state called on element

 Bin:

   A->B: all elements SUCCESS
     - commit state

   A->B: some elements ASYNC
     - no commit state
     - listen for commit messages on bus
     - for each commit message, poll elements, this happens in another
       thread.
     - if no ASYNC elements, commit state, continue state change
       to STATE_PENDING

 downward state change
 ~~~~~~~~~~~~~~~~~~~~~

 Downward state changes only return ASYNC if the final state is ASYNC.
 This is to make sure that it's not needed to wait for an element to
 complete the preroll or other ASYNC state changes when one only wants to
 shut down an element.

 Element:

   A -> B => SUCCESS
     - commit state

   A -> B => ASYNC not final state
     - commit state on behalf of element

   A -> B => ASYNC final state
     - element will commit ASYNC

 Bin:

   A -> B -> SUCCESS
     - commit state

   A -> B -> ASYNC not final state
     - commit state on behalf of element, continue state change

   A -> B => ASYNC final state
     - no commit state
     - listen for commit messages on bus
     - for each commit message, poll elements
     - if no ASYNC elements, commit state


 Locking overview (element)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~

 * Element commiting SUCCESS

  - STATE_LOCK is taken in set_state
  - change state is called if SUCCESS, commit state is called
  - commit state calls change_state to next state change.
  - if final state is reached, stack unwinds and result is returned to
    set_state and caller.


  set_state(element)       change_state (element)   commit_state

       |                         |                       |
       |                         |                       |
   STATE_LOCK                    |                       |
       |                         |                       |
       |------------------------>|                       |
       |                         |                       |
       |                         |                       |
       |                         | (do state change)     |
       |                         |                       |
       |                         |                       |
       |                         | if SUCCESS            |
       |                         |---------------------->|
       |                         |                       | post message
       |                         |                       |
       |                         |<----------------------| if (!final) change_state (next)
       |                         |                       | else SIGNAL
       |                         |                       |
       |                         |                       |
       |                         |                       |
       |<------------------------|                       |
       |     SUCCESS
       |
   STATE_UNLOCK
       |
     SUCCESS


 * Element commiting ASYNC

  - STATE_LOCK is taken in set_state
  - change state is called and returns ASYNC
  - ASYNC returned to the caller.
  - element takes LOCK in streaming thread.
  - element calls commit_state in streaming thread.
  - commit state calls change_state to next state change.


  set_state(element)       change_state (element)     stream_thread      commit_state (element)

       |                         |                          |                  |
       |                         |                          |                  |
   STATE_LOCK                    |                          |                  |
       |                         |                          |                  |
       |------------------------>|                          |                  |
       |                         |                          |                  |
       |                         |                          |                  |
       |                         | (start_task)             |                  |
       |                         |                          |                  |
       |                         |                     STREAM_LOCK             |
       |                         |                          |...               |
       |<------------------------|                          |                  |
       |     ASYNC                                     STREAM_UNLOCK           |
   STATE_UNLOCK                                             |                  |
       |                .....sync........               STATE_LOCK             |
     ASYNC                                                  |----------------->|
                                                            |                  |
                                                            |                  |---> post_message()
                                                            |                  |---> if (!final) change_state (next)
                                                            |                  |     else SIGNAL
                                                            |<-----------------|
                                                        STATE_UNLOCK
                                                            |
                                                       STREAM_LOCK
                                                            | ...
                                                       STREAM_UNLOCK

 Remarks
 ~~~~~~~

 set_state cannot be called from multiple threads at the same time. The STATE_LOCK
 prevents this.

 state variables are protected with the LOCK.

 calling set_state while gst_state is called should unlock the get_state with
 an error. The cookie will do that.


  set_state(element)

   STATE_LOCK

   LOCK
   update current, next, pending state
   cookie++
   UNLOCK

   change_state

   STATE_UNLOCK
	States
	------

	Both elements and pads can be in different states. The states of the pads are
	linked to the state of the element so the design of the states is mainly
	focused around the element states.

	An element can be in 4 states. NULL, READY, PAUSED and PLAYING. When an element
	is initially instantiated, it is in the NULL state.


	State definitions
	~~~~~~~~~~~~~~~~~

	- NULL: This is the initial state of an element.
	- READY: The element should be prepared to go to PAUSED.
	- PAUSED: The element should be ready to accept and process data. Sink
	elements however only accept one buffer and then block.
	- PLAYING: The same as PAUSED except for live sources and sinks. Sinks accept
	and rendering data. Live sources produce data.

	We call the sequence NULL->PLAYING an upwards state change and PLAYING->NULL
	a downwards state change.


	State transitions
	~~~~~~~~~~~~~~~~~

	the following state changes are possible:

	NULL -> READY
	- The element must check if the resources it needs are available.
	Device sinks and -sources typically try to probe the device to constrain
	their caps.
	- The element opens the device, this is needed if the previous step requires
	the device to be opened.

	READY -> PAUSED
	- The element pads are activated in order to receive data in PAUSED.
	Streaming threads are started.
	- Some elements might need to return ASYNC and complete the state change
	when they have enough information. It is a requirement for sinks to
	return ASYNC and complete the state change when they receive the first
	buffer or EOS event (preroll). Sinks also block the dataflow when in PAUSED.
	- A pipeline resets the running_time to 0.
	- Live sources return NO_PREROLL and don't generate data.

	PAUSED -> PLAYING
	- Most elements ignore this state change.
	- The pipeline selects a clock and distributes this to all the children
	before setting them to PLAYING. This means that it is only allowed to
	synchronize on the clock in the PLAYING state.
	- The pipeline uses the clock and the running_time to calculate the base_time.
	The base_time is distributed to all children when performing the state
	change.
	- Sink elements stop blocking on the preroll buffer or event and start
	rendering the data.
	- Sinks can post the EOS message in the PLAYING state. It is not allowed to
	post EOS when not in the PLAYING state.
	- While streaming in PAUSED or PLAYING elements can create and remove
	sometimes pads.
	- Live sources start generating data and return SUCCESS.

	PLAYING -> PAUSED
	- Most elements ignore this state change.
	- The pipeline calculates the running_time based on the last selected clock
	and the base_time. It stores this information to continue playback when
	going back to the PLAYING state.
	- Sinks unblock any clock wait calls.
	- When a sink does not have a pending buffer to play, it returns ASYNC from
	this state change and completes the state change when it receives a new
	buffer or an EOS event.
	- Any queued EOS messages are removed since they will be reposted when going
	back to the PLAYING state. The EOS messages are queued in GstBins.
	- Live sources stop generating data and return NO_PREROLL.

	PAUSED -> READY
	- Sinks unblock any waits in the preroll.
	- Elements unblock any waits on devices
	- Chain or get_range functions return WRONG_STATE.
	- The element pads are deactivated so that streaming becomes impossible and
	all streaming threads are stopped.
	- The sink forgets all negotiated formats
	- Elements remove all sometimes pads

	READY -> NULL
	- Elements close devices
	- Elements reset any internal state.


	State variables
	~~~~~~~~~~~~~~~

	An element has 4 state variables that are protected with the object LOCK:

	- STATE
	- STATE_NEXT
	- STATE_PENDING
	- STATE_RETURN

	The STATE always reflects the current state of the element.
	The STATE_NEXT reflects the next state the element will go to.
	The STATE_PENDING always reflects the required state of the element.
	The STATE_RETURN reflects the last return value of a state change.

	The STATE_NEXT and STATE_PENDING can be VOID_PENDING if the element is in
	the right state.

	An element has a special lock to protect against concurrent invocations of
	_set_state(), called the STATE_LOCK.


	Setting state on elements
	~~~~~~~~~~~~~~~~~~~~~~~~~

	The state of an element can be changed with _element_set_state(). When changing
	the state of an element all intermediate states will also be set on the element
	until the final desired state is set.

	The _set_state() function can return 3 possible values:

	GST_STATE_FAILURE: The state change failed for some reason. The plugin should
	have posted an error message on the bus with information.

	GST_STATE_SUCCESS: The state change is completed successfully.

	GST_STATE_ASYNC: The state change will complete later on. This can happen
	when the element needs a long time to perform the state
	change or for sinks that need to receive the first buffer
	before they can complete the state change (preroll).

	GST_STATE_NO_PREROLL: The state change is completed successfully but the element
	will not be able to produce data in the PAUSED state.

	In the case of an ASYNC state change, it is possible to proceed to the next
	state before the current state change completed, however, the element will only
	get to this next state before completing the previous ASYNC state change.
	After receiving an ASYNC return value, you can use _element_get_state() to poll
	the status of the element. If the polling returns SUCCESS, the element completed
	the state change to the last requested state with _set_state().

	When setting the state of an element, the STATE_PENDING is set to the required
	state. Then the state change function of the element is called and the result of
	that function is used to update the STATE and STATE_RETURN fields, STATE_NEXT,
	STATE_PENDING and STATE_RETURN fields. If the function returned ASYNC, this result
	is immediately returned to the caller.


	Getting state of elements
	~~~~~~~~~~~~~~~~~~~~~~~~~

	The _get_state() function takes 3 arguments, two pointers that will hold the
	current and pending state and one GstClockTime that holds a timeout value. The
	function returns a GstElementStateReturn.

	- If the element returned SUCCESS to the previous _set_state() function, this
	function will return the last state set on the element and VOID_PENDING in
	the pending state value. The function returns GST_STATE_SUCCESS.

	- If the element returned NO_PREROLL to the previous _set_state() function, this
	function will return the last state set on the element and VOID_PENDING in
	the pending state value. The function returns GST_STATE_NO_PREROLL.

	- If the element returned FAILURE to the previous _set_state() call, this
	funciton will return FAILURE with the state set to the current state of
	the element and the pending state set to the value used in the last call
	of _set_state().

	- If the element returned ASYNC to the previous _set_state() call, this function
	will wait for the element to complete its state change up to the amount of time
	specified in the GstClockTime.

	* If the element does not complete the state change in the specified amount of
	time, this function will return ASYNC with the state set to the current state
	and the pending state set to the pending state.

	* If the element completes the state change within the specified timeout, this
	function returns the updated state and VOID_PENDING as the pending state.

	* If the element aborts the ASYNC state change due to an error within the
	specified timeout, this function returns FAILURE with the state set to last
	successfull state and pending set to the last attempt. The element should
	also post an error message on the bus with more information about the problem.


	States in GstBin
	~~~~~~~~~~~~~~~~

	A GstBin manages the state of its children. It does this by propagating the state
	changes performed on it to all of its children. The _set_state() function on a
	bin will call the _set_state() function on all of its children, that are
	not already in the target state or in a change state to the target state.

	The children are iterated from the sink elements to the source elements. This makes
	sure that when changing the state of an element, the downstream elements are in
	the correct state to process the eventual buffers. In the case of a downwards
	state change, the sink elements will shut down first which makes the upstream
	elements shut down as well since the _push() function returns a GST_FLOW_WRONG_STATE
	error.

	If all the children return SUCCESS, the function returns SUCCESS as well.

	If one of the children returns FAILURE, the function returns FAILURE as well. In
	this state it is possible that some elements successfuly changed state. The
	application can check which elements have a changed state, which were in error
	and which were not affected by iterating the elements and calling _get_state()
	on the elements.

	If after calling the state function on all children, one of the children returned
	ASYNC, the function returns ASYNC as well.

	If after calling the state function on all children, one of the children returned
	NO_PREROLL, the function returns NO_PREROLL as well.

	If both NO_PREROLL and ASYNC children are present, NO_PREROLL is returned.

	The current state of the bin can be retrieved with _get_state().

	If the bin is performing an ASYNC state change, it will automatically update its
	current state fields when it receives state messages from the children.


	Implementing states in elements
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	READY
	^^^^^



	upward state change
	~~~~~~~~~~~~~~~~~~~

	Upward state changes always return ASYNC either if the STATE_PENDING is
	reached or not.

	Element:

	A -> B => SUCCESS
	- commit state

	A -> B => ASYNC
	- no commit state
	- element commits state ASYNC

	A -> B while ASYNC
	- update STATE_PENDING state
	- no commit state
	- no change_state called on element

	Bin:

	A->B: all elements SUCCESS
	- commit state

	A->B: some elements ASYNC
	- no commit state
	- listen for commit messages on bus
	- for each commit message, poll elements, this happens in another
	thread.
	- if no ASYNC elements, commit state, continue state change
	to STATE_PENDING

	downward state change
	~~~~~~~~~~~~~~~~~~~~~

	Downward state changes only return ASYNC if the final state is ASYNC.
	This is to make sure that it's not needed to wait for an element to
	complete the preroll or other ASYNC state changes when one only wants to
	shut down an element.

	Element:

	A -> B => SUCCESS
	- commit state

	A -> B => ASYNC not final state
	- commit state on behalf of element

	A -> B => ASYNC final state
	- element will commit ASYNC

	Bin:

	A -> B -> SUCCESS
	- commit state

	A -> B -> ASYNC not final state
	- commit state on behalf of element, continue state change

	A -> B => ASYNC final state
	- no commit state
	- listen for commit messages on bus
	- for each commit message, poll elements
	- if no ASYNC elements, commit state


	Locking overview (element)
	~~~~~~~~~~~~~~~~~~~~~~~~~~

	* Element commiting SUCCESS

	- STATE_LOCK is taken in set_state
	- change state is called if SUCCESS, commit state is called
	- commit state calls change_state to next state change.
	- if final state is reached, stack unwinds and result is returned to
	set_state and caller.


	set_state(element) change_state (element) commit_state

	\| \| \|
	\| \| \|
	STATE_LOCK \| \|
	\| \| \|
	\|------------------------>\| \|
	\| \| \|
	\| \| \|
	\| \| (do state change) \|
	\| \| \|
	\| \| \|
	\| \| if SUCCESS \|
	\| \|---------------------->\|
	\| \| \| post message
	\| \| \|
	\| \|<----------------------\| if (!final) change_state (next)
	\| \| \| else SIGNAL
	\| \| \|
	\| \| \|
	\| \| \|
	\|<------------------------\| \|
	\| SUCCESS
	\|
	STATE_UNLOCK
	\|
	SUCCESS



	* Element commiting ASYNC

	- STATE_LOCK is taken in set_state
	- change state is called and returns ASYNC
	- ASYNC returned to the caller.
	- element takes LOCK in streaming thread.
	- element calls commit_state in streaming thread.
	- commit state calls change_state to next state change.


	set_state(element) change_state (element) stream_thread commit_state (element)

	\| \| \| \|
	\| \| \| \|
	STATE_LOCK \| \| \|
	\| \| \| \|
	\|------------------------>\| \| \|
	\| \| \| \|
	\| \| \| \|
	\| \| (start_task) \| \|
	\| \| \| \|
	\| \| STREAM_LOCK \|
	\| \| \|... \|
	\|<------------------------\| \| \|
	\| ASYNC STREAM_UNLOCK \|
	STATE_UNLOCK \| \|
	\| .....sync........ STATE_LOCK \|
	ASYNC \|----------------->\|
	\| \|
	\| \|---> post_message()
	\| \|---> if (!final) change_state (next)
	\| \| else SIGNAL
	\|<-----------------\|
	STATE_UNLOCK
	\|
	STREAM_LOCK
	\| ...
	STREAM_UNLOCK

	Remarks
	~~~~~~~

	set_state cannot be called from multiple threads at the same time. The STATE_LOCK
	prevents this.

	state variables are protected with the LOCK.

	calling set_state while gst_state is called should unlock the get_state with
	an error. The cookie will do that.


	set_state(element)

	STATE_LOCK

	LOCK
	update current, next, pending state
	cookie++
	UNLOCK

	change_state

	STATE_UNLOCK