[Bug 796382] flvmux: (too) easily produces backwards flv-timestamps

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=796382

--- Comment #10 from Håvard Graff (hgr) <havard.graff at gmail.com> ---
(In reply to Nicolas Dufresne (ndufresne) from comment #9)
> Review of attachment 372387 [details] [review]:
> 
> This is not really a fix. I think we get in this situation because the mixer
> let data go, on timeout, and the some "late" data comes in, and get pushed.
> If we want to use that model, late is late, data need to be dropped then.
> When you add latency, you move the timeout later, allowing a certain delay
> between a/v. So, weigher we follow the paradigm, and massively drop, or just
> disable timeout mode, but hacking the timestamp after, I don't agree.

I agree. This causes more problems than it solves. Not great stuff indeed.

Now, picture the following scenario:

We have requested an audio and a video pad on flvmux. However, due to some
various application/device delays, the audio stream starts flowing 2 seconds
before the video can get going. What should the muxer do now?

If we have queues in front of the flvmuxer it won't start doing anything with
the audio until the video-caps arrives, and now we will get into huge
muxing-trouble (with todays implementation) with the timestamp jumping back and
forth between the buffer audio-stream (starting from dts 0) and the video
stream (starting from dts 2000 (ms)).

There are a few options available here:

1. Drop all the audio leading up the the point where the video starts.
This will ensure sync from that point onward but at the cost of some missing
audio.

2. Start processing the audio only first for 2 seconds, effectively introducing
a 2 second delay on the stream.

Obviously option 1 would work better for "live" scenarios, and 2 for recording
type scenarios.

Our use-case calls for 1, but I am not sure how this is best implemented.

As for 2 I think this is supposed to work, but testing shows that a
video-buffer might be nominated in gst_flv_mux_find_best_pad if a audio-buffer
does not reach the flv-sinkpad in time (racy).

I will write a simple test demonstrating this and then we can discuss the best
way forward.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.