[gst-devel] Muxing subtitles to matroska [Was: Re: dvdsrc status update, sundry questions]

Michal Benes michal.benes at itonis.tv
Mon Dec 18 12:04:35 CET 2006

Hi Mark,

good to hear, you are still around :-) Perhaps there is time to do
something about subtitle handling in muxers. 

First of all, I do not think there is a point writing CollectPads2
unless there is a consensus among core developers what it should do.

As for BaseMuxer, I know there are "intricate differences in a muxer's
taste". Still I think that if we could design a base class that
factorize out common functionality of existing muxers it is worth doing
it. I know that BaseMuxer could be used in matroskamux and ffmux because
it was me who ported these two to 0.10. It seems to me that avimux is
quite simple too. I still do not quite understand oggmux - I have to
make time to take closer look at the code (I would be thankful if you
could point me to oggmux specialities).

I have been thinking about BaseMuxer as base class rather than component
because I think it fits well with other Base* classes in GStreamer. But
if it turns out that the component approach is better I do not see any
problem. In any case it would be the muxer itself (i.e. nont BaseMuxer)
handling _request_pad, _release_pad, _setcaps, etc...

Let me write shortly the rough idea

I assume that the patch
http://bugzilla.gnome.org/attachment.cgi?id=64824&action=view from
http://bugzilla.gnome.org/show_bug.cgi?id=340060 is applied.
The only other API addition to CollectPads I would need are two

void gst_collect_pads_set_waiting (GstCollectPads *pads, GstPad *pad,
gboolean waiting)
gboolean gst_collect_pads_get_waiting (GstCollectPads *pads, GstPad

which would control if the collectpads is waiting for that pad (any
ideas for a better name are welcomed)

Now, the NEWSEGMENT event handler would do this:

if (newsegment start time is in future) {
	set waiting mode on the pad
} else {
	unset waiting mode on the pad
note that CollectPads should automatically call
gst_collect_pads_is_collected if the waiting mode has changed

Pads collected function would do this:

peek oldest buffer
if (there is a non-waiting pad whose newsegment start time is older than
the oldest buffer) {
	set this pad to waiting state and return
} else {
	we have found the oldest buffer, send it to the muxer
(of course, muxer could register custom _compare function for handling
special buffers).

I think set/get_waiting is an acceptable addition to CollectPads API,
though not so trivial one. One has to take care about proper locking as
set_waiting can be called asynchronously and we need to recheck
is_collected condition after each change.

I will try to propose the BaseMuxer class after New Year. 


Mark Nauwelaerts píše v So 16. 12. 2006 v 20:04 +0100: 
> Michal Benes wrote:
> >>> Well, just hoping that these hacks around collectpads can shed their "hacking"
> >>> existence some day, and that the functionality they provide can make it cleanly
> >>> into upstream, as much re-usable as possible ... :-)
> >>>
> >> Probably by considering CollectPads to have been an experimenting and
> >> creating a CollectPads2 to replace it, as we've done with decodebin2
> > 
> > I have been thinking about this last night and I think that the real
> > problem here is that CollectPads is used in very different situations
> > (muxer ver. adder), therefore it is very difficult to implement correct
> > NEWSEGMENT handling (I do not know what is correct anyway).
> > 
> > So it would be probably better to implement new BaseMuxer class (that
> > uses CollectPads internally). This class would automatically find the
> > oldest buffer handling all the NEWSEGMENT stuff. I have quick checked
> > oggmux, avimux ffmux and matroskamux. It seems that they are interested
> > only in the oldest buffer so that they they could all benefit from the
> > BaseMuxer class (I am not quite sure about oggmux as I do not understand
> > it completely).
> > 
> > BaseMuxer would probably need some api addition to CollectPads to
> > disable/enable waiting for a pad (alternatively BaseMuxer could use
> > filler buffer hack internally).
> I believe the most important (and tricky) of the above lies in the "some api
> additions to CollectPads" part.  This is currently at the core of
> things/problems regarding e.g. muxers not having enough say in what is going on.
>  E.g. though CollectPads does publicly expose a segment for inspection, but this
> does not provide enough "thread" responsivity for your filler trick.
> This part of it brings us back to CollectPads2 etc
> Beyond that, avimux, matroskamux etc are typically interested in the "oldest
> buffer", but there may be intricate differences in a muxer's taste (e.g. how to
> deal with GST_CLOCK_TIME_NONE buffers, typically carrying headers ?).  Other's
> taste may be even more exotic, e.g. oggmux.  This type of "finer control" would
> be lost versus the gain of factorizing out a "typical/simple" _collected
> function.  So the tripping point seems to be how much other trickery this
> class/BaseMuxer would have to (and can) take care of beyond _collected, and
> therefore, again, how easy (or how hacking) CollectPads(2) makes this (and it
> not being a point of hiding the hacking in a BaseMuxer ;-) ).
> Also, if BaseMuxer is to be truly a base class rather than a component to use
> like CollectPads (re-use by inheritance versus composition), then it raises
> other questions such as "who" will handle _request_pad, _release_pad, _setcaps
> etc.  If that can be adressed, it would require quite a re-write for existing
> muxers, but may have the benefit that they are more aligned.
> As for "some api additions to CollectPads", I think the following would be a
> start, at least based on current (unaddressed) use-cases:
> - give some more (direct interactive) access to events; not necessarily all of
> them, depends on how much wanting to prevent the mux element to "shoot itself in
> the foot"
> - (as you say) request CollectPads to not wait around for some pad; the element
> would then typically have a _peek to see if there is something interesting
> (would have to be careful about detecting eos in these cases).  To keep
> buffers/streams in increasing time order, it should also be able to request
> CollectPads to start waiting around again for some activity on the pad in
> question (when nearing announced NEWSEGMENT time), so that other pads do not
> overshoot.  So, a sort of gst_collect_pads_wait (collect, pad, TRUE/FALSE), to
> be called from within event handling or _collected function.
> However, the plans/direction for CollectPads are not clear to me,
> considering that a suggestion like the first item above was already given in a
> patch in http://bugzilla.gnome.org/show_bug.cgi?id=340060, but apparently to no
> avail ... (maybe the problem there is in some details ?)
> Regards,
> Mark.

More information about the gstreamer-devel mailing list