[Bug 754230] qtdemux: support sparse time ranges in qtdemux without needing a seek for MSE

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=754230

--- Comment #7 from Enrique Ocaña González <eocanha at igalia.com> ---
Sure!

I talked to both of you (Nicolas and Sebastian) and commented that it would be
a pity for Samsung and Igalia to have to resort to ChunkDemuxer (Chromium's
standalone demuxer) when GStreamer can actually demux the appended raw data for
us, or at least it should.

I explained to Nicolas that the MSE API[1] in general and our implementation in
WebKit[2] in particular splits the concept of appending data to get "buffered
ranges" from the concept of playing, pausing, stopping and seeking the
previously buffered data. Raw data is appended to a SourceBuffer at any time
and the MSE layer has no idea of what's been appended until it processes
(demuxes) the data, after which "buffered ranges" are created. The player can
then playback what's buffered, or even portions not yet buffered (so it first
waits for the needed buffered ranges to appear and then continues). Same for
seeks.

As I said in previous posts, there's no problem when we append data linearly.
The first append usually contains the "MSE init segment" (headers describing
the data that will follow) and no real samples, but it's very important to let
qtdemux configure itself properly, create src pads, etc. The second append
usually contains the samples. Our append pipeline captures those samples with
an appsink and return them to the upper MSE layer so it can track the "buffered
ranges".

However, when the JavaScript code in the webpage seeks to a future/past
position out of the buffered ranges, the most probable (but not guaranteed)
consequence is that the raw data appended since that moment belongs to some
distant position compared to what was appended the last time. Qtdemux doesn't
know it and just keeps generating timestamps (PTS) contiguous to the last ones.

It was suggested that some sort of heuristic rule could be used, so that when a
seek is performed in the playback pipeline, the qtdemux(es) in the append
pipeline(s) are reset. There's no guarantee that such heuristic would work, and
in any case it would be very weak and error prone. So, by now, we have to
assume that we have no hints about what's being appended and that the
demuxer(s) must be able to produce absolute time information on their own.

To try to understand what was happening during seeks, I saved the raw data to
disk as it was appended and analyzed it offline using MP4Box.js[3]. What I
found was that the TFDT atom contained the absolute position of the "MSE
segment" being appended. I considered that this would be a good hint and coded
the patch I linked in the description of this bug[4]. The patch enforces
honoring the TFDT atom when it can help us detect a gap.

The side effect of this patch is that it breaks the current qtdemux behaviour
(ie: breaks some tests), so it's apparently not desirable to have it upstream.
After talking to Nicolas, he thought that one posible solution would be to add
a new boolean property to qtdemux ("prioritizeTfdt"? any better name?) that
would default to false (so the tests keep passing). If set to true, it would
honor the TFDT atom and make the MSE append use case work.

Sebastian commented that in any case the pastebin patch should be improved,
because it only worked for "forward discontinuities/jumps" due to the way the
comparison is being done. It should be fixed to work also for "backward jumps".

Please, let me know if I forgot or can clarify something more.

[1] https://w3c.github.io/media-source/#sourcebuffer-segment-parser-loop
[2] Slides 14-16 in
http://www.slideshare.net/philn2/gstreamer-support-in-webkit-whats-new
[3] http://download.tsi.telecom-paristech.fr/gpac/mp4box.js/filereader.html
[4] http://pastebin.com/612D3kyU

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.