[gst-devel] GValue event networks in GStreamer

[Josh Green]

On Sun, 2004-07-25 at 11:51, Andy Wingo wrote:
> Hey Josh,
> 
> As I read your mail, I'm realizing that you won't get any really good
> comments until people start to make apps based on such a framework. For
> that reason, it's not appropriate for GStreamer core IMO (but I'm not
> the one to talk to). At some point, I think you should set up a
> repository somewhere (use arch :-) so that people can include the code
> into their source trees, pound on it a bit, then maybe if it stabilizes
> put it in core.
> 
> That is, unless Benjamin thinks it's useful for any nefarious plan he
> might have. Until then, no plugin using this framework would be in
> gst-plugins. It would be just for connections internal to an app. (It
> would be interesting to see if it is feasible to have a library common
> to DSP apps, though.)
> 

This sounds like good advice. I'll start thinking about setting up a
repository at some point. Have to get the basics implemented before it
makes sense, though.

> On Thu, 2004-07-22 at 11:08 -0600, Josh Green wrote:
> > struct _GstValueEvent {
> >   GValue                *value; /* the GValue of this value event */
> 
> Just wondering: Why not have the GValue inline in the event, instead of
> as a pointer? The benefit is it would be only one allocation. But
> perhaps many events could share one GValue, so that wouldn't make sense.
> Dunno, thoughts?
> 

Actually, it was supposed to be inline. You identified the first bug :) 
It wouldn't be that useful to be able to share a GValue, since its
usually a fairly trivial operation to copy one.

> >   /* the origin event (NULL if is origin or origin of event chain) */
> >   GstValueEvent         *origin;
> >  
> >   /* active event propagation count */
> >   guint                  active_count;
> 
> I read your explanation, but I still don't understand this. Can you give
> some use cases?
> 

An event may continue to exist (some one holds a refcount) even though
it is no longer being propagated between pads. This is where the
active_count field comes into play. Its like a reference count, but for
any current (or future) event propagation. After an event is created,
the active_count is incremented before event propagation begins. If the
event encounters an element that would like to re-send the event at a
later time (say a queue or delay) it will also increment the
active_count to indicate that an additional subsystem holds a right to
send the event. Once an event's active_count reaches 0, it can be
removed from all pad active event lists (which is used to detect loops),
and may not be resent. So as long as an event has the chance of being
propagated its active_count will be greater than 0. This extends the
thoroughness of the loop detection, since a loop could be created
through a queue as well. Queues can be useful for batch processing
events at certain intervals (an example usage in Swami is for the GUI,
which has its own queue to process all incoming events to GUI controls
in the GUI thread).

> > The next stage is integrating this new GstData type with GstPad and
> > GstCaps. This is where I'm still a bit ignorant in regards to the
> > workings of GStreamer.
> 
> You need to say that it's a GstValueEvent, the type of the GValue, and
> some type-specific constraints. The first will certainly be encoded in
> the MIME type. Then the other things go as structure properties.
> 
> application/x-gvalue, type=gint, max=4, min=2, default=3
> 
> `type' could be a number as well; implementation would have to flesh out
> which is best. (In this case, you'd have to call g_type_from_name to get
> the type.)
> 

There are times when a pad would want to indicate that it will accept
any event type (a wildcard so to speak). I suppose this would also be
stored in the structure properties in some way. Its good to know that
its up to the implementation, though. However the value type is stored,
retrieving it should be quick, since it will be used during event
propagation to convert event values and such. Storing the value type as
a GType would of course be faster.

> > Control types
> > -------------
> > There are a couple of different kinds of controls. Value controls, which
> > have a specific value that can be queried at any time (a GUI spin button
> > for example), and event controls which emit/receive events but don't
> > have a specific "set" value (a MIDI sink or source for example).
> 
> I think the GUI needs to be separated from the processing. Thus it
> doesn't make sense to have an element that has a GUI. Perhaps a better
> solution would be that, instead of querying a graphical element, that
> the graphical element _pushes_ a value to the processing network.
> 

When you say that the GUI would be separate from the processing, is this
because it needs to be processed in its own thread? I was using an event
queue to handle this situation, would that be appropriate for GStreamer?

> That is, have a "plug" element that has an object property plug::value,
> and the GUI updates the property. The plug element is the one that
> generates the GstValueEvent.
> 
> Consider also the idea of having a default value. For example, the phase
> offset input of an oscillator may be left unconnected, in which case the
> offset would be 0. Or it might be set to a constant value (which
> shouldn't require an extra element: this implies that pads have values).
> Or it might be varied at control rate (via GstValueEvents). Or maybe it
> can be varied at audio rate (in the modular synth style, whereby another
> oscillator can be hooked to the phase-offset control).
> 
> This is what I would want, anyway.
> 

Thats how the SwamiControl works (the existing code that I'm basing this
system on), its analogous to a GstPad, but may "contain" its own value
(if its a value control) leaving the implementation up to the control,
or be valueless like in the case of a purely event driven sink/source.
So in the case of a value control, a GstPad would have its own value
that could be queried.

> > Event I/O
> > ---------
> > A GstPad needs to be able to send and receive these events. I assume
> > gst_pad_push and gst_pad_pull could be used with this new GstData type.
> > Therefore there also needs to be a couple callbacks in a GstPad to
> > retrieve a value from a pad or send a value to a pad
> 
> Why? Have you actually coded this? I think the standard get / chain /
> loop functions are powerful enough. (Of course, you access them via
> gst_pad_pull/push.) The only thing they don't do is allow for a pad to
> have a value; that has to be implemented in a subclass of GstRealPad.
> 

I could try and use the existing callback functions, but I see a couple
of things that might not be possible using these callbacks.

For value controls its nice to indicate what event value type the GstPad
will accept and have all incoming values be converted to this type (so a
floating point control would receive automatically converted incoming
events of type int for example). This was the reason for the additional
parameter in the GstPadSetValueEventFunc that I described before:

/* for pushing a value to a pad, @value is possibly converted,
   @event is the original received event */
typedef void (*GstPadSetValueEventFunc)(GstPad *pad,
                                        GstValueEvent *event,
                                        const GValue *value);

I suppose this functionality is not required, but it is nice to have the
value auto-converted for the incoming event callback function. Perhaps
there is another way to implement this with existing callbacks.

The other issue is that when querying a value from a pad, it would be
nice to just be able to get the value without creating an GstValueEvent,
which is shown in the other function definition:

/* for pulling a value from a pad */
typedef void (*GstPadGetValueEventFunc)(GstPad *pad, GValue *value);

If these two issues could be resolved and still use the existing
callbacks, then I have no problem using them.

> > Capability Flags
> > ----------------
> > There are a couple of flags that could also be useful in the Caps of a
> > GstValueEvent pad. A "no convert" flag to indicate that even though
> > there is a set GParamSpec, incoming events should not be converted;
> 
> Use case?
> 

I currently use this flag to give an input type "suggestion" via the
parameter spec (structure properties in the case of the GStreamer
implementation) and indicate that the event handler will process the
type itself which saves some time converting the value if its not
necessary.

> That's all my thoughts for now :) My attention span is spotty, so I
> can't promise to be on the ball with comments. Good luck, though!
> 
> Cheers,

I really appreciate the well informed feedback, it has helped to put me
on the right track. I'm looking forward to working on this task and also
the task of adding support for instrument files and wavetable synthesis
to GStreamer using Swami/libInstPatch. Cheers!
	Josh Green


Swami - Sampled Waveforms And Musical Instruments
http://swami.sourceforge.net