[gst-devel] Request for ideas: decoupled elements

[in7y118 at public.uni-hamburg.de]

(This is mainly a mail to the core devs, as I'm a bit out of ideas - so you 
others, please don't get all suicidal if you don't understand everything)

I don't know how I got to thinking about it, but decoupled elements are not 
designed cleanly inside GStreamer - and therefore cause a lot of issues. For 
everyone that is still with me after decoupled: This means elements that are in 
different schedulers. The most prominent example is the "queue" element. 
Another example is the pipefilter element. The problem is that GStreamer 
doesn't handle the "one element belongs to multiple schedulers" fact and that 
causes some issues.

Imagine this simple pipeline:
{ filesrc } ! { queue ! filesink }   (example 1)
This should be the same in functionality as:
{ filesrc ! queue } ! { filesink }   (example 2)
or even:
{ filesrc } ! queue ! { filesink }   (example 3)
though I wouldn't try that last one...

What this really means is that the first thread - or better: the scheduler of 
the first thread - controls the filesrc and the sinkpad of the queue while the 
second thread controls the filesink and the srcpad of the queue.

_But_who_controls_the_queue_ ?

To illustrate some of the issues:
- When unblocking elements, the current code explicitly unblocks all connected 
elements. It has to do this because the scheduler might be blocking inside an 
element that doesn't belong to him (imagine the filesrc thread in example 1 
blocking in the queue).
- Signal emissions can be started from totally unexpected threads. The EOS 
signal in a queue always gets emitted from the scheduler that controls the 
srcpad. (This would mean that the EOS signal of the filesrc thread in example 2 
gets emitted in the context of the filesink thread, how's that for assumptions 
about signal handlers?)
- State changes lead to interesting races. If you set the whole pipeline to 
PLAYING in example 1, the filesrc thread may already be pushing data to the 
queue while the queue has not yet changed state. (This btw is the cause for 
the "pushing data into inactive pads is not allowed" g_warning's you get 
sometimes).
- What scheduler sets the clock on decoupled elements?

I think what we need is a way to make an element belong to multiple schedulers 
somehow, probably via subclassing to GstDecoupledElement. But this is currently 
not at all possible because every element provides loads of stuff to the 
schedulers (like 2 FLAGS entries, an element->scheduler_priv field and whatnot).
Another approach would be to make decoupled elements many elements (queuesrc 
and queuesink), that get put inside different bins and schedulers and need to 
be connected somehow. But this is ugly from a user standpoint, isn't it?
{filesrc ! queuesink partner=queuesrc0 } { queuesrc0 partner=queuesink0 ! 
filesink } looks evil, too.

Any ideas or opinions on if, when or how we do want this solved?
It might need quite a bit of refactoring inside the core.


Benjamin