[gst-devel] Muxing subtitles to matroska [Was: Re: dvdsrc status update, sundry questions]

[Mark Nauwelaerts]

Michal Benes wrote:
> Hi Mark,
> 
> good to hear, you are still around :-) Perhaps there is time to do
> something about subtitle handling in muxers. 
> 
> First of all, I do not think there is a point writing CollectPads2
> unless there is a consensus among core developers what it should do.

Indeed, it seems we can squeeze some more "experimenting" out of CollectPads (as
in your suggestions below), so we may get a better idea what CollectPads2 should
do.  Until then, it does serve a number of elements as they are (of course give
or take the shortcomings presently discussed).

API additions and BaseMuxer are a promising outline, looks like a plan ;-)
So, it remains (a.o.) for a plan to get e.g. dvd subtitles into a mkv (in some
form or another), i.e. not necessarily the evidently required bit in matroskamux
 itself which needs the sparse handling in the first place. I'll probably have a
bit of a look at that (some fair time after New Year), if I don't come to regret
that, or if there's already some hack out there doing so.

Regards,
Mark.

> As for BaseMuxer, I know there are "intricate differences in a muxer's
> taste". Still I think that if we could design a base class that
> factorize out common functionality of existing muxers it is worth doing
> it. I know that BaseMuxer could be used in matroskamux and ffmux because
> it was me who ported these two to 0.10. It seems to me that avimux is
> quite simple too. I still do not quite understand oggmux - I have to
> make time to take closer look at the code (I would be thankful if you
> could point me to oggmux specialities).
> 
> I have been thinking about BaseMuxer as base class rather than component
> because I think it fits well with other Base* classes in GStreamer. But
> if it turns out that the component approach is better I do not see any
> problem. In any case it would be the muxer itself (i.e. nont BaseMuxer)
> handling _request_pad, _release_pad, _setcaps, etc...
> 
> Let me write shortly the rough idea
> 
> I assume that the patch
> http://bugzilla.gnome.org/attachment.cgi?id=64824&action=view from
> http://bugzilla.gnome.org/show_bug.cgi?id=340060 is applied.
> The only other API addition to CollectPads I would need are two
> functions
> 
> void gst_collect_pads_set_waiting (GstCollectPads *pads, GstPad *pad,
> gboolean waiting)
> gboolean gst_collect_pads_get_waiting (GstCollectPads *pads, GstPad
> *pad)
> 
> which would control if the collectpads is waiting for that pad (any
> ideas for a better name are welcomed)
> 
> Now, the NEWSEGMENT event handler would do this:
> 
> if (newsegment start time is in future) {
> 	set waiting mode on the pad
> } else {
> 	unset waiting mode on the pad
> }
> note that CollectPads should automatically call
> gst_collect_pads_is_collected if the waiting mode has changed
> 
> Pads collected function would do this:
> 
> peek oldest buffer
> if (there is a non-waiting pad whose newsegment start time is older than
> the oldest buffer) {
> 	set this pad to waiting state and return
> } else {
> 	we have found the oldest buffer, send it to the muxer
> }
> (of course, muxer could register custom _compare function for handling
> special buffers).
> 
> I think set/get_waiting is an acceptable addition to CollectPads API,
> though not so trivial one. One has to take care about proper locking as
> set_waiting can be called asynchronously and we need to recheck
> is_collected condition after each change.
> 
> I will try to propose the BaseMuxer class after New Year. 
> 
> 	Michal
> 
> Mark Nauwelaerts píše v So 16. 12. 2006 v 20:04 +0100: 
>> Michal Benes wrote:
>>>>> Well, just hoping that these hacks around collectpads can shed their "hacking"
>>>>> existence some day, and that the functionality they provide can make it cleanly
>>>>> into upstream, as much re-usable as possible ... :-)
>>>>>
>>>> Probably by considering CollectPads to have been an experimenting and
>>>> creating a CollectPads2 to replace it, as we've done with decodebin2
>>> I have been thinking about this last night and I think that the real
>>> problem here is that CollectPads is used in very different situations
>>> (muxer ver. adder), therefore it is very difficult to implement correct
>>> NEWSEGMENT handling (I do not know what is correct anyway).
>>>
>>> So it would be probably better to implement new BaseMuxer class (that
>>> uses CollectPads internally). This class would automatically find the
>>> oldest buffer handling all the NEWSEGMENT stuff. I have quick checked
>>> oggmux, avimux ffmux and matroskamux. It seems that they are interested
>>> only in the oldest buffer so that they they could all benefit from the
>>> BaseMuxer class (I am not quite sure about oggmux as I do not understand
>>> it completely).
>>>
>>> BaseMuxer would probably need some api addition to CollectPads to
>>> disable/enable waiting for a pad (alternatively BaseMuxer could use
>>> filler buffer hack internally).
>> I believe the most important (and tricky) of the above lies in the "some api
>> additions to CollectPads" part.  This is currently at the core of
>> things/problems regarding e.g. muxers not having enough say in what is going on.
>>  E.g. though CollectPads does publicly expose a segment for inspection, but this
>> does not provide enough "thread" responsivity for your filler trick.
>> This part of it brings us back to CollectPads2 etc
>>
>> Beyond that, avimux, matroskamux etc are typically interested in the "oldest
>> buffer", but there may be intricate differences in a muxer's taste (e.g. how to
>> deal with GST_CLOCK_TIME_NONE buffers, typically carrying headers ?).  Other's
>> taste may be even more exotic, e.g. oggmux.  This type of "finer control" would
>> be lost versus the gain of factorizing out a "typical/simple" _collected
>> function.  So the tripping point seems to be how much other trickery this
>> class/BaseMuxer would have to (and can) take care of beyond _collected, and
>> therefore, again, how easy (or how hacking) CollectPads(2) makes this (and it
>> not being a point of hiding the hacking in a BaseMuxer ;-) ).
>> Also, if BaseMuxer is to be truly a base class rather than a component to use
>> like CollectPads (re-use by inheritance versus composition), then it raises
>> other questions such as "who" will handle _request_pad, _release_pad, _setcaps
>> etc.  If that can be adressed, it would require quite a re-write for existing
>> muxers, but may have the benefit that they are more aligned.
>>
>> As for "some api additions to CollectPads", I think the following would be a
>> start, at least based on current (unaddressed) use-cases:
>> - give some more (direct interactive) access to events; not necessarily all of
>> them, depends on how much wanting to prevent the mux element to "shoot itself in
>> the foot"
>> - (as you say) request CollectPads to not wait around for some pad; the element
>> would then typically have a _peek to see if there is something interesting
>> (would have to be careful about detecting eos in these cases).  To keep
>> buffers/streams in increasing time order, it should also be able to request
>> CollectPads to start waiting around again for some activity on the pad in
>> question (when nearing announced NEWSEGMENT time), so that other pads do not
>> overshoot.  So, a sort of gst_collect_pads_wait (collect, pad, TRUE/FALSE), to
>> be called from within event handling or _collected function.
>>
>> However, the plans/direction for CollectPads are not clear to me,
>> considering that a suggestion like the first item above was already given in a
>> patch in http://bugzilla.gnome.org/show_bug.cgi?id=340060, but apparently to no
>> avail ... (maybe the problem there is in some details ?)
>>
>> Regards,
>> Mark.
> 
> 
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> 
>