[gst-devel] 0.9 proposals

[David Schleef]

On Tue, Dec 14, 2004 at 05:48:23PM +0100, Thomas Vander Stichele wrote:
> Hi,
> 
> > No, Wim proposes to make lots of internal GStreamer function calls
> > threadsafe by adding locking around ie gst_pad_(un)link. This requires
> > lots of work - especially deadlock prevention - inside the core.
> > My proposal is to make the whole API not threadsafe and require that each
> > function be called in its context. That's a lot easier.
> 
> Ok, I've tried to piece together from some IRC logs and some mails what
> you *do* propose wrt. threads in general and I can't seem to piece
> together a coherent pictures.  Could you please explain exactly what you
> want to do, where there are threads and where there aren't, and so on ?
> 
> A mail higher up in the thread has you saying
> 
> >>>
> David and I agreed on the fact that we want to get rid of threads. Do
> not
> support threads in Gstreamer anymore, it only causes races. This of
> course
> only works with a non-blocking API. But if you want a threadsafe model
> you're in for quite a ride, especially with dynamic pipelines.
> <<<
> 
> which Colin said he assumed meant "the GstThread element".  Is that the
> case ?

Essentially, yes.  AFAICT, Benjamin (in this context) wants to remove
the concept of pipelines having explicit threads in them, i.e.,
gst-launch lines with {}'s.  I've advocated this change for a long
time, but for a different reason -- explicitly stating threading
is not necessary, since a scheduler like opt can figure out correct
threading without the hints.

> Also, I assume everyone agrees that for all approaches, there could be
> more than one thread involved ? Even if that thread is private to the
> element ?

Yes.  It's quite clear that at the bare minimum, we need to allow
elements to start up threads and execute code.  There should be
a GstElement subclass or a helper library for this.

> I am not sure either how you can for example queue a network stream
> inside the pipeline without using threads somehow.

This is actually dirt simple.  'socketsrc ! queue ! sink', socketsrc 
select()s on its socket (or, more likely, requests the core to
perform the same action) and passes buffers to queue.  queue accepts
any buffers and saves them.  When requested, queue pushes a buffer.
sink (assuming it works like osssink, say) waits for the output device
to be ready (again, using select() functionality in the core), pulls
a buffer, and writes it to the device.

Note that this is all done without any waiting done while in the
stack frame of any of the elements.  That means interruptions,
events, state changes, property changes, object desctruction, etc.,
all happen when the element is not executing, automagically.

> I'm getting various different interpretations from different mails/irc
> snippets/other people's interpretation, and it would be nice to see a
> coherent picture of exactly what it is you want to do wrt. thread-
> safety.  It's impossible to evaluate if your ideas can work for actual
> applications without having a good picture of what your ideas are.
> 
> Specifics that are important is "how do apps react to signals/events
> from the pipeline", "how can apps manipulate the pipeline", "which part
> of gstreamer uses threads and for what reason", "how do they not leak",
> and so on.

- How do apps react to signals/events from the pipeline: exactly like
gtk -- you create an object (a pipeline) from the main thread, call
it from the main thread, manipulate it from the main, and get signals
from the object (or its subobjects) in the main thread, marshalled
from the main loop in the main thread.  This is the very simple case;
in a single threaded app, gstreamer should look and act like a single
threaded library, and not ever make the app think about threading or
locking.

Advanced usage: replace 'main thread' above with 'thread B', and it
works exactly the same.  Key point is that a particular gst object
is tied to a particular application thread.  (Marlin and rhythmbox
use this.)  Independent pipelines do not affect each other in any
way.

Really advanced usage: (don't know if I really want to support this)
As long as the application serializes all accesses to a particular
GstPipeline and its children, gstreamer should not care which thread
it is called from.  It's unclear, in this case, to which thread
signals should be sent, which is why I don't want to support it.  I
also don't know any application that needs this, and can't think of
a sane design that would.

Usage I'm not supporting: libgstreamer is completely reentrant, and
you can access any object from any thread with no concern about
concurrency, and gstreamer will handle all serialization and locking.

- How can apps manipulate the pipeline: exactly like gtk, see above.

- Which part of gstreamer uses threads and for what reason: In the
cleanest, simplest case, _no part_ of gstreamer needs to use threads.
Obviously, a limited number of elements must be written with a helper
thread, since they can't properly work with a device and/or library
without blocking.  [Note that this is nearly identical in spirit to
0.8, except that the core forces lots of elements to be blocking since
it doesn't provide appropriate API to allow them to be non-blocking,
and it complicates helper threads by tying them into the scheduler.

So, a simple application will run in the main thread, call gtk and
gstreamer from the main thread, gstreamer schedules all elements in
the main thread, and since every element is non-blocking, no portion
is CPU starved (e.g., gtk, gst, or any of the elements.)  This model
is really simple and requires nothing special in terms of threading
or locking in gstreamer -- signals are emitted and marshalled with
no special thread-changing code, and you can even call gui code
directly from a GstElement.

There are a few difficulties with the above model.  First, you can't
take advantage of multiple processors.  Second, certain elements
would have trouble providing glitch-free output, e.g., ximagesink,
since it depends on precise timing of code execution to display 
an image without timing glitches.  Arguably, you could have
ximagesink create a helper thread, which is what I would recommend
if we were to strictly stick to the model in the previous paragraph.
However, I think it's generally wise to separate gstreamer execution
from GUI glitches.

To fix this, the core starts up a thread for each CPU (roughly) for
the purpose of executing element code.  The core also might use the
main thread for executing element code, particularly at the request
of the element, which may want to run GUI code or manipulate the
pipeline (which, remember, can only be done from the main thread).
Most elements, however, are completely happy executing in another
thread.

There are a number of complications executing element code in a
non-main thread: 1, signals from the elements need to be sent to
the main thread to be marshalled, 2, the main thread needs to make
sure that none of the execution threads are using the element when
it wants to negotiate pads, link/unlink, change properties, etc.,
and 3, the API generally used by elements (which doesn't overlap
significantly with the application API) needs to be thread-safe,
since it can be called from a non-main thread.

- How do they not leak?  As you can see from the above paragraphs,
non-main threads are very limited in what they can actually do,
including manipulate objects, so leaking is not a problem.



dave...