[gst-devel] Stuff that needs doing...

[David I. Lehn]

* Erik Walthinsen <omega at temple-baptist.com> [20010305 16:58]:
> 4) Event system
> This is the big one.  I realized a week ago that there are more things
> that may be of interest to a plugin than just what comes in on its pads.
> Specifically, consider an mp3parse element and a file with an id3v2 tag.
> The tag is at the end of the file.  In order for the parser to seek to the
> end, read the tag, and seek back, it has to know that it's got a new file.
> How does it do that?  It could guess when it gets an offset 0 buffer, but
> what if you start out in the middle?
> 
> The solution is some kind of event system that encompasses EOS, empty,
> flush, seeking, new-file notification, and everything else.  This system
> would have to be carefully constructed to follow the dataflow left to
> right, and have the "right" behavoir when going right to left (seek).
> Some have suggested attaching these events to the buffers, but that
> introduces the overhead of checking each and every buffer at every point
> in the pipeline, which we've intentionally avoided so far.
> 
> In general, a plugin would have a standard handler that would simply pass
> events through.  Various plugins would either override that handler or use
> some other mechanism to register for interesting events, and handle them
> that way.
> 
> We need to discuss the format of these events, which events we're going to
> start with, and how the plugins would interact with them.  This is
> currently my second-highest priority, behind figuring out a first pass at
> the cross-process[or] stuff.
> 

[ long response of rambling random thoughts, enjoy ;) ]

When I look at this I see EOS, empty, flush, seeking, etc being a subset
of the desire to combine both data, control, and application level
data/control flow.  Erik, you mentioned (in IRC at least) that you
wanted to have control flow as a seperate layer above what gstreamer
currently offers.  For example MIDI or similar sort of things.  It looks
like you're solving it in gstreamer now though.

We want to have objects that react to events at the right time.  Is just
handling events and passing them on enough?  For things like EOS you
don't really want to fire that event on the last part of the pipeline
until the data from the first part has propagated down.  Seems like you
would have to match up timestamps somehow on control and data streams.
Not sure how to do this.

What events does the gstreamer core -need- to handle?  EOS for sure.
What are the semantics of an EOS message though?  Seems there are
different levels of EOS:

  final EOS: no more data, we're done, release resources

  flush: still processing same type of stream just can't expect next
  data to be next in sequence to last data.  ie, during seek or the next
  similar stream.  It's kind of like EOS IMHO.

empty event?  what is that?

seek is just a flush and start sending new data.  or is it more complex
than that?  Seems with a nice pipeline approach you don't have to
actually wait for a flush before sending the new data.

new-file notification?  you mean new stream notification?  well, that
sounds like a flush too.  Seems like there are different "EOS" events:

  stream done:  pipeline going down.  deallocate on your way out.
  Serious End Of Stream.

  new stream:  new stream coming in, last one done, reset yourself.

  immediate flush: empty buffers out now.  don't send any currently held
  data to output pads.

  soft flush: finish what your doing, send anything you can to output
  pads then prepare for new stream.

  discontinuous: next data will not be next in sequence to last.  still
  same stream though, so ID3 tags or timing info is similar to what we
  have now.  Could be used during seek.  Already a buffer flag for this.

Hmm... not sure about all that...

How do handle such things?

Well... ideally we want data and control to be temporally interleaved
going down the same pipeline structure we have now.  Or at least that's
what I'd like. ;)

Right now (I think) it's mostly a data buffer with flags.  Check flags
all the time for each new buffer.  Kind of messy for the plugins to deal
with.

Some applications will want just control streams.  Well... line blurs
between data and control for some things like MIDI.  I'm not sure what
you would call it but someone mentioned filters processing MIDI streams.

So... now you want elements that can handle streams with various things
coming down them in temporal order.  Next someone is going to want to
have >1 input pad with this stuff on it.  Starting to get messy.
Interleaved timestamped data and control on multiple inputs.

I'm not sure what the best way to deal with such things is.

But... I'll suggest something complex anyway. ;)

I'll also start talking about stuff I don't understand that may have
been previously thought of and thrown out... beware.

How about generalizing everything to event streams?  Ie, low level you
are just passing around timestamped events with (very) few possible
flags (if any).

A data buffer then just becomes a DataEvent.  A flush is just a
FlushEvent.  These all can come through the same pad.  Lowest level you
could do something similar to what is done now.  Have a chain or loop
function that processes -all- events.  All the details left to the
unfortunate user.

At a higher level start providing more common functionality.  A basic
element will filter the input event stream for a pad to sub event
processors.  So a element will start up and set functions for data
processing and (for instance) eos processing.

So now the data processing is handled in one loop and the eos processing
in another.  Yeah, you still need the loop based data processing to
check a "done" flag.  The flag would be set in the the eos event loop.
Provide an API and some code to do the event splitting.  And also
provide some policy that events will be dispatched to sub chain/loop
handlers in their temporal order.

This isn't -that- much more code than what is available now.  Just some
naming changes and a good efficient (the hard part) event
filter/dispatcher.  It's just a fancy wrapper that automatically creates
more complex sub structures that make assumptions on the events they
will be getting.

The question is of course if the event filter/dispatcher can be made
efficient.  I haven't tried it so I'd rather not speculate on its
performance.  Though a basic test with null data processing would give a
metric on maximum possible streaming event througput.

Passing events like EOS onward would have to be handled by each plugin.
Perhaps the common case would be to have a default handler that takes
unknown events and broadcasts them to all outputs... I dunno.  (This is
the sort of thing flags may be needed for... broadcast vs an error for
unhandled events)  Could have default handlers for certain types.  Which
sort of implies that all elements will have to have event handling
tables.

Passing events backwards?  When does this need to be done that isn't
applicaiton specific?

Maybe have an event hierarchy?  Could have handlers for general case or
handlers for more specific cases of that event?  ie, something to handle
the different EOS types mentioned above in some easy way.

Syncing multiple input streams?  Perhaps some sort of
EventDispatcherGroup between input event dispatchers that ensures events
are sent out in temporal order across inputs.

Event loops due to default behavior?  Don't do that.  ;)  Does the rest
of gstreamer deal with just acyclic graphs?

Anyway... this is drastic for sure.  Maybe too complex.  Probably
enables a whole slew of new application domains if it were done
properly.  It also tries to build on the "make the simple case easy and
the complicated case possible" idea.  I've probably forgotten many
things here... async programming tends to turn your brain to mush. ;)

Any comments?

-dave
-- 
David I. Lehn <dlehn at vt.edu>  | http://www.lehn.org/~dlehn/
Computer Engineering Graduate @ Virginia Tech in sunny Blacksburg, VA