Gap between audio and video using multipartdemux

[Tim Müller]

On Sun, 2013-12-22 at 00:31 -0800, Alexander Ushakov wrote:

Hi Alexander,

> Hello. We are trying to record video from Sony SNC EP-580 IP camera using
> HTTP protocol. This model can send audio and video streams in one http
> stream (details here:
> http://pro.sony.com.cn/productinfo/vs/downloads/G5%20Camera%20CGI%20manual.pdf),
> so we use this pipeline to demux, convert and record video and audio
> together:
> gst-launch-1.0 souphttpsrc location="http://xx.xx.xx.xx/h264?audioin=on"
> is-live=true ! queue leaky=2 max-size-time=0 max-size-buffers=0
> max-size-bytes=0 ! multipartdemux name=demux

Not sure there's a point in making *this* queue leaky or unlimited in
size. Shouldn't be needed.

> demux.src_0 ! queue leaky=2 ! decodebin ! videoconvert ! x264enc bitrate=800
> ! video/x-h264, profile=baseline ! h264parse ! mux.
> demux.src_1 ! queue leaky=2 ! decodebin ! audioconvert ! audioresample !
> audio/x-raw, rate=44100, channels=2 ! faac ! mux.
> mp4mux name=mux ! filesink location="/tmp/test.mp4"
> 
> We use the latest version of gstreamer: 1.2.1 (we made only one minor change
> in multipartdemux sources - added non-standard Sony mime types)

You need to figure out where the problem comes from. As a starting
point, try muxing raw audio/video with qtmux (video downscaled maybe
and/or jpeg encoded), to eliminate the encoders as source of problems.
All this without any queues being set as leaky or unlimited. If I'm not
mistaken, the pipeline will lock up after a while then if the process
persists.

If it's not the encoders that leaves decoders or qtmux.

Maybe have a look at the collectpads/qtmux debug log to see what
timestamps the muxer is seeing.

If there are no input timestamps, the decoders will have to make stuff
up, maybe there's a problem with how they do that (e.g. if they start
from 0, what's the framerate etc.)

Cheers
 -Tim

> When the pipeline starts working, everything goes fine at first, but after
> some time a delay between audio and video streams appears in the recording.
> This gap keeps increasing, until it reaches 15 seconds and sets on this
> value. So the audio stream is about 15 seconds ahead the video stream.
> If I don't set "leaky" parameter to queue, the pipeline crashes with queue
> buffer full after 2 hours of recording.
> 
> According to the documentation, all branches of gstreamer pipeline (audio
> and video) must work as fast as they can (because there is no timestamp
> generated in souphttpsrc or multipartdemux). So I can't understand why there
> is an increasing delay between video and audio streams, although CPU load is
> less than 50%?
> 
> PS. Adding sync=false and/or async=false parameters to filesink doesn't
> solve the problem
> Adding do-timestamp=true to souphttpsrc doesn't solve the problem and makes
> the recording buggy - all video and audio come in spurts: 5 seconds freeze,
> then 1 second fast playback of the previous 5 seconds and so on. I think it
> is because souphttpsrc with do-timestamp=true creates timestamps for too big
> chunks of data.

-- 
Tim Müller, Centricular Ltd - http://www.centricular.com