[gst-devel] 0.9 proposals

[Wim Taymans]

On Tue, 2004-12-14 at 17:17 -0800, David Schleef wrote:
> On Tue, Dec 14, 2004 at 05:48:23PM +0100, Thomas Vander Stichele wrote:
> > Hi,
> > 
> > > No, Wim proposes to make lots of internal GStreamer function calls
> > > threadsafe by adding locking around ie gst_pad_(un)link. This requires
> > > lots of work - especially deadlock prevention - inside the core.
> > > My proposal is to make the whole API not threadsafe and require that each
> > > function be called in its context. That's a lot easier.
> > 
> > Ok, I've tried to piece together from some IRC logs and some mails what
> > you *do* propose wrt. threads in general and I can't seem to piece
> > together a coherent pictures.  Could you please explain exactly what you
> > want to do, where there are threads and where there aren't, and so on ?
> > 
> > A mail higher up in the thread has you saying
> > 
> > >>>
> > David and I agreed on the fact that we want to get rid of threads. Do
> > not
> > support threads in Gstreamer anymore, it only causes races. This of
> > course
> > only works with a non-blocking API. But if you want a threadsafe model
> > you're in for quite a ride, especially with dynamic pipelines.
> > <<<
> > 
> > which Colin said he assumed meant "the GstThread element".  Is that the
> > case ?
> 
> Essentially, yes.  AFAICT, Benjamin (in this context) wants to remove
> the concept of pipelines having explicit threads in them, i.e.,
> gst-launch lines with {}'s.  I've advocated this change for a long
> time, but for a different reason -- explicitly stating threading
> is not necessary, since a scheduler like opt can figure out correct
> threading without the hints.

Correct. I want to get rid of it too.

> 
> > Also, I assume everyone agrees that for all approaches, there could be
> > more than one thread involved ? Even if that thread is private to the
> > element ?
> 
> Yes.  It's quite clear that at the bare minimum, we need to allow
> elements to start up threads and execute code.  There should be
> a GstElement subclass or a helper library for this.

Is this element allowed to make gst core method calls? 

Can it fire signals at will or does it have to call special methods that
marshal the signals to the main thread?  

> 
> > I am not sure either how you can for example queue a network stream
> > inside the pipeline without using threads somehow.
> 
> This is actually dirt simple.  'socketsrc ! queue ! sink', socketsrc 
> select()s on its socket (or, more likely, requests the core to
> perform the same action) and passes buffers to queue.  queue accepts
> any buffers and saves them.  When requested, queue pushes a buffer.
> sink (assuming it works like osssink, say) waits for the output device
> to be ready (again, using select() functionality in the core), pulls
> a buffer, and writes it to the device.

> 
> Note that this is all done without any waiting done while in the
> stack frame of any of the elements.  That means interruptions,
> events, state changes, property changes, object desctruction, etc.,
> all happen when the element is not executing, automagically.
> 
> > I'm getting various different interpretations from different mails/irc
> > snippets/other people's interpretation, and it would be nice to see a
> > coherent picture of exactly what it is you want to do wrt. thread-
> > safety.  It's impossible to evaluate if your ideas can work for actual
> > applications without having a good picture of what your ideas are.
> > 
> > Specifics that are important is "how do apps react to signals/events
> > from the pipeline", "how can apps manipulate the pipeline", "which part
> > of gstreamer uses threads and for what reason", "how do they not leak",
> > and so on.
> 
> - How do apps react to signals/events from the pipeline: exactly like
> gtk -- you create an object (a pipeline) from the main thread, call
> it from the main thread, manipulate it from the main, and get signals
> from the object (or its subobjects) in the main thread, marshalled
> from the main loop in the main thread.  This is the very simple case;
> in a single threaded app, gstreamer should look and act like a single
> threaded library, and not ever make the app think about threading or
> locking.
> 
> Advanced usage: replace 'main thread' above with 'thread B', and it
> works exactly the same.  Key point is that a particular gst object
> is tied to a particular application thread.  (Marlin and rhythmbox
> use this.)  Independent pipelines do not affect each other in any
> way.
> 
> Really advanced usage: (don't know if I really want to support this)
> As long as the application serializes all accesses to a particular
> GstPipeline and its children, gstreamer should not care which thread
> it is called from.  It's unclear, in this case, to which thread
> signals should be sent, which is why I don't want to support it.  I
> also don't know any application that needs this, and can't think of
> a sane design that would.
> 
> Usage I'm not supporting: libgstreamer is completely reentrant, and
> you can access any object from any thread with no concern about
> concurrency, and gstreamer will handle all serialization and locking.
> 
> - How can apps manipulate the pipeline: exactly like gtk, see above.
> 
> - Which part of gstreamer uses threads and for what reason: In the
> cleanest, simplest case, _no part_ of gstreamer needs to use threads.
> Obviously, a limited number of elements must be written with a helper
> thread, since they can't properly work with a device and/or library
> without blocking.  [Note that this is nearly identical in spirit to
> 0.8, except that the core forces lots of elements to be blocking since
> it doesn't provide appropriate API to allow them to be non-blocking,
> and it complicates helper threads by tying them into the scheduler.
> 
> So, a simple application will run in the main thread, call gtk and
> gstreamer from the main thread, gstreamer schedules all elements in
> the main thread, and since every element is non-blocking, no portion
> is CPU starved (e.g., gtk, gst, or any of the elements.)  This model
> is really simple and requires nothing special in terms of threading
> or locking in gstreamer -- signals are emitted and marshalled with
> no special thread-changing code, and you can even call gui code
> directly from a GstElement.
> 
> There are a few difficulties with the above model.  First, you can't
> take advantage of multiple processors.  Second, certain elements
> would have trouble providing glitch-free output, e.g., ximagesink,
> since it depends on precise timing of code execution to display 
> an image without timing glitches.  Arguably, you could have
> ximagesink create a helper thread, which is what I would recommend
> if we were to strictly stick to the model in the previous paragraph.
> However, I think it's generally wise to separate gstreamer execution
> from GUI glitches.
> 
> To fix this, the core starts up a thread for each CPU (roughly) for
> the purpose of executing element code.  The core also might use the
> main thread for executing element code, particularly at the request
> of the element, which may want to run GUI code or manipulate the
> pipeline (which, remember, can only be done from the main thread).
> Most elements, however, are completely happy executing in another
> thread.
> 
> There are a number of complications executing element code in a
> non-main thread: 1, signals from the elements need to be sent to
> the main thread to be marshalled, 2, the main thread needs to make
> sure that none of the execution threads are using the element when
> it wants to negotiate pads, link/unlink, change properties, etc.,
> and 3, the API generally used by elements (which doesn't overlap
> significantly with the application API) needs to be thread-safe,
> since it can be called from a non-main thread.
> 
> - How do they not leak?  As you can see from the above paragraphs,
> non-main threads are very limited in what they can actually do,
> including manipulate objects, so leaking is not a problem.
> 

Thanks for clarifications, I see what you mean now. This is indeed very
well doable. I'm still struggling with the drawbacks though:

- data passing is quite heavy as each push/pull/event/signal/poll needs
to go through a scheduler. People are asking that Gstreamer should add
less overhead (current is at about 3%, in a last benchmark encoding a
theora video), I know there are many causes for this but scheduling is
currently using up quite some CPU and even that is currently supposed to
be quite fast with opt...
- latency of events can get high as the event can only be scheduled when
the current event has been handled (you mention the videosink).
- (re)writing the elements as state machines.
- marshalling of signals/methods over thread boundaries. You basically
need to check in each method call if the target object(s) run in the
same current thread and marshall if not.

Have you considered using cothreads for this kind of scheduling as a way
of implementing more efficient scheduling and plugin development?

Wim.  


> 
> 
> dave...
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now. 
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
-- 
Wim Taymans <wim at fluendo.com>