[gst-devel] Re: MuCoS and media streaming

[Erik Walthinsen]

On Thu, 11 May 2000, David Olofson wrote:

> I assume this can be taken care of by requiring plugins to give the
> host enough information to set up adequate buffering at net building
> time.
There is no specific mechanism for this, but the GtkObject system allows
arbitrary parameters, so you could define some standard set of values
necessary to fully specify an element.

> Another problem is that this model results in a kind of "dynamic"
> scheduling scheme for the plugin callback functions, as the host can
> never know beforehand how many buffers it needs to throw at a plugin
> to get enough buffers for the plugins down the line.
The neat thing is that the default implementation of a container is by no
means the only possible implementation.  My current design is aiming at
the more buffered streaming-media case, but I've intended from the
beginning to deal with much more bounded cases.  I just haven't given it
any thought yet.

A very cool setup would be to eventually have the ability to take a thread
full of elements doing some operation, and compile the whole mess down
into DSP code.  That would make the container opaque until it's brought
back up into the host CPU, but it could enable extremely high performance
if properly architected.

> The third problem is closely related to the previous one; how does
> the host/engine/scheduler find out how much input buffering is needed
> for a full duplex system? (That is, a system that inputs data,
> processes it, and then outputs it with the lowest possible latency,
> without dropouts.) Basically, plugins cannot be alloved to behave
> like that in full duplex, real time systems when placed in chains
> that originate from real time input streams, and end in real time
> outputs. BUT, plugins of this kind still are still needed in such
> systems! For example, a sample player needs to be able to work
> something like this, as it basically converts input streams at
> various rates into output streams at fixed rates. Other examples
> would be a sprite animator for a game engine, or a "varispeed PIP
> viewport" for a video editing system.
How an element deals with sizing the buffers it deals with is up to the
element.  The architecture only deals with things at a buffer granularity,
so the application and the elements are responsible for working out
latency.  GStreamer really is not much more than a dataflow architecture
with lots of neat features, it leaves all the hard decisions up to the
application, where they belong.  If you want to do realtime processing,
with the current crop of elements, you simply set the read size on the
audio input to whatever's appropriate, say 64 samples.  It reads 64
samples, put them into a buffer, and hands it off to whoever's next, and
repeats.

> Rather than making all plugins input driven, making plugins like the
> ones in the examples above (variable rate inputs, fixed rate outputs)
> output or request driven seems to make life a lot easier for the
> host/engine/scheduler. It avoids recursive style scheduling
> algorithms, and allows entire processing nets to run one cycle at a
> time with one call per plugin, using a fixed execution order -
> simpler and faster, although perhaps not as sexy as the input driven
> model.
I don't remember when I wrote up the first email, but I've done a lot of
work in the generic scheduling case.  In the current model, cothreads are
used extensively when necessary to allow elements to have arbitrary
input/output patterns.  Any complex element (takes more than 1 buffer in
at a time to cause some output) must be written as a loop function, which
is basically

while (1) {
  buf = gst_pad_pull(sink);
  ...process...
  gst_pad_push(src,newbuf);
}

If you have a complex input, i.e. several pulls per iteration, each time
you pull a buffer it simply switches to the cothread of the peer element,
causing it to do the same if necessary, eventually ending in the buffer
being generated and switching back to the element in question.

> How does your system deal with these issues?
As of now, it's basically as above.  But the core idea is that the
container can do things any way you want.  One thing that's in progress is
redoing the scheduling plan generation.  If it determines that there's no
need to use cothread (i.e. zero complex elements), it will fall back to
the chain method.  A more application specific implementation could read
various constants from the elements as mentioned above, and make very
specific scheduling decisions based on these parameters.  This would be
optimal for realtime.

Also, the (p)threading model is a function of the containers.  The generic
Bin is what provides the element scheduling, and the Thread simply wraps
this with code that spawns a new thread and starts the iterate() function
in a loop.  If you want to use this on a non-pthread system, you just
replace the Thread, and it will still inherit the cothreading from the
Bin.  If you want to replace the whole mess of functionality, you just
override the methods.

> This reminds of the MuCoS client/server model, where plugins become
> clients and run in their own threads, communicating data via shared
> memory whenever possible.
The key in GStreamer being that cothreads mean you can keep a single
process context (LWP) for a whole set of related elements.  This keeps
your inter-thread buffering at a minimum, as well as locking.  A whole
slew of threads means a lot of buffering between them, or condition
variables to lock-step the threads together.  This way it's implicit.

> As for the above issues I mentioned, this model still has the same
> problems when dealing with real time systems, as it simply results in
> too much overhead and too many chances to get scheduling latencies if
> the control is allowed to jump back and forth between clients for
> "recursive" buffer requests. That is, the same solutions apply
> (output driven when the outputs are fixed rate), although the
> scheduling is done by the OS kernel and the clients sleeping/waking
> up on the event queues that connect them.
Again, you should be able to write a new, more specialized scheduling
container to fix any problems, if it's possible to fix them.  In most
cases, though, you're going to have to start out with an application that
simply doesn't construct or allow the construction of pipelines that could
cause these problems.

> As I still don't have much running code (haven't had time to do
> anything interesting for a long while), I'd be interested to hear
> about your experience with the plugin scheduling performance, latency
> issues etc. And of course, I'm always interested in cool design ideas!
> :-)
I've got some other stuff to do in the short term (check out
libdv.sourceforge.net for neat video stuff ;-), but I'm trying to squeeze
in time for gstreamer when I can.  I haven't done any audio stuff yet, but
I'm ordering a Hammerfall card to replace the STUDI/O that just isn't
gonna work, and then we'll install the processing machine somewhere where
we really can get 16 tracks of audio in at once (Creamware A16 in new
sound booth with new board, dual Celeron in old booth with touchscreen
LCD).  At that point I'll be wanting to do quite a bit of audio stuff,
though it'll be entirely experimental until I'm a lot more confident with
it.

The biggest thing I'm missing right now is experience with ALSA sufficient
to write a multi-channel audio element for the ADAT card.  Once I get the
Hammerfall installed and running, I'll play with Ardour and see what I can
learn about the multi-channel API from there, then I can start really
doing some experiments.

TTYL,
    Omega

         Erik Walthinsen <omega at cse.ogi.edu> - Staff Programmer @ OGI
        Quasar project - http://www.cse.ogi.edu/DISC/projects/quasar/
   Video4Linux Two drivers and stuff - http://www.cse.ogi.edu/~omega/v4l2/
        __
       /  \             SEUL: Simple End-User Linux - http://www.seul.org/
      |    | M E G A           Helping Linux become THE choice
      _\  /_                          for the home or office user