Trying to keep non-transcoded audio and transcoded video in sync

[Nirbheek Chauhan]

Hi,

On Mon, Feb 14, 2022 at 4:45 PM Matthew Shapiro via gstreamer-devel
<gstreamer-devel at lists.freedesktop.org> wrote:
>
> I have a custom written RTMP server written in Rust, and I'm trying to implement gstreamer to provide dynamic transcoding pipelines.  I wrote all the RTMP code by hand so I have some media knowledge, but I'm far, far from an expert and I'm confused about some timing issues I'm getting.
>
> I have a pre-encoded FLV of big buck bunny and I am using ffmpeg to push the video (without transcoding) into my RTMP server, and I'm using ffplay to act as an RTMP client.  When I do zero transcoding the audio and video are perfectly synced, but once I pass the packets through the x264enc for encoding, audio is now several seconds before the corresponding video.  My understanding is that RTMP clients/video players would use the RTMP Timestamp values to keep both audio and video in sync, but either the timestamps that I'm getting from gstreamer are incorrect or I'm misunderstanding something.
>
> For transcoding I'm using the following pipeline for my proof of concept:
>
> -------------------
> appsrc name=input ! decodebin ! videoscale ! video/x-raw,width=800,height=600 ! x264enc speed-preset=veryfast name=encoder ! h264parse name=parser ! appsink name=output
> -------------------
>

The issue is likely something to do with timestamps being wrong when
pushing video into the pipeline (or getting it out of the pipeline),
or synchronization breaking due to some other reason on the sender
side (probably in the muxer).

The simplest way to fix this would be to let gstreamer do the muxing
too, with a pipeline like:

appsrc name=vinput ! queue ! decodebin ! videoscale !
video/x-raw,width=800,height=600 ! x264enc speed-preset=veryfast
name=encoder ! h264parse name=parser ! mux.video
appsrc name=ainput ! queue ! mux.audio
flvmux name=mux ! appsink name=voutput

Here flvmux will ensure that synchronization is taking place
correctly, as long as the audio and video buffers pushed into the
appsrc are timestamped correctly.

Cheers,
Nirbheek