Trying to keep non-transcoded audio and transcoded video in sync

Tue Feb 15 18:59:04 UTC 2022

So I tried to do a bit more digging on this, not because I have an issue with your pipeline (I will probably go that route in the end) but I suspect there's something else off with my element clocks that's the true problem.  Specifically, when I play video through my pipeline VLC is showing a start time of 1000:00:00.

Through debugging I have verified the pts and dts values I am setting look correct (by looking at the results of the buffer.pts() and buffer.dts() values.  For example, I'm getting these for the first 3 h264 buffers I am passing in:

-------------------
INPUT - dts Some(0:00:00.000000000) pts Some(0:00:00.083000000)
INPUT - dts Some(0:00:00.041000000) pts Some(0:00:00.250000000)
INPUT - dts Some(0:00:00.083000000) pts Some(0:00:00.166000000)
--------------------

(The Some() just means it's not a null value, if you aren't familiar with rust)

I then added debugging of the values coming out of the appsink, which gives me

---------------------
dts 999:59:59.917000000 pts 1000:00:00.000000000
Running time: Some(0:00:00.000138500)
Start time: Some(0:00:00.000000000)
Clock time: Some(25:59:28.529101100)

dts 999:59:59.959000000 pts 1000:00:00.167000000
Running time: Some(0:00:00.047566400)
Start time: Some(0:00:00.000000000)
Clock time: Some(25:59:28.576529600)

dts 1000:00:00.000000000 pts 1000:00:00.083000000
Running time: Some(0:00:00.092902000)
Start time: Some(0:00:00.000000000)
Clock time: Some(25:59:28.621872200)
------------------------

We can see that the dts and pts values are all being set up so that presentation starts at 1000:00:00.  The pipeline's start time is 00:00:00, the current running time seems to be the correct.    Clock time seems weird (doesn't match my system time but based on docs it can be pulling this from a device somewhere, and probably doesn't matter?).

So, why would my pipelines be starting from a 1000 pts?  I can't find much in the docs to help debug this further

For reference, my current iteration of my test code is using the following code to create a pipeline of appsrc ! decodebin ! videoscale ! capsfilter ! x264enc ! h264parse ! appsink

---------------
        let appsrc = ElementFactory::make("appsrc", None).unwrap();
        let decoder = ElementFactory::make("decodebin", None).unwrap();
        let scale = ElementFactory::make("videoscale", None).unwrap();
        let capsfilter = ElementFactory::make("capsfilter", None).unwrap();
        let encoder = ElementFactory::make("x264enc", None).unwrap();
        let output_parser = ElementFactory::make("h264parse", None).unwrap();
        let appsink = ElementFactory::make("appsink", None).unwrap();

        let pipeline = Pipeline::new(None);
        pipeline.add_many(
            &[&appsrc, &decoder, &scale, &capsfilter, &encoder, &output_parser, &appsink]
        ).unwrap();

        appsrc.link(&decoder).unwrap();
        Element::link_many(&[&scale, &capsfilter, &encoder, &output_parser, &appsink]).unwrap();

        decoder.connect_pad_added(move |src, src_pad| {
            src.link_pads(Some(&src_pad.name()), &scale.clone(), None).unwrap();
        });

        let caps = Caps::builder("video/x-raw")
            .field("width", 800)
            .field("height", 600)
            .build();

        capsfilter.set_property("caps", caps);
        encoder.set_property_from_str("speed-preset", "veryfast");
-------------

I appreciate any insights you (or anyone) could give!
________________________________
From: Nirbheek Chauhan <nirbheek.chauhan at gmail.com>
Sent: Tuesday, February 15, 2022 8:52 AM
To: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org>
Cc: Matthew Shapiro <me at mshapiro.net>
Subject: Re: Trying to keep non-transcoded audio and transcoded video in sync

Hi,

On Mon, Feb 14, 2022 at 4:45 PM Matthew Shapiro via gstreamer-devel
<gstreamer-devel at lists.freedesktop.org> wrote:
>
> I have a custom written RTMP server written in Rust, and I'm trying to implement gstreamer to provide dynamic transcoding pipelines.  I wrote all the RTMP code by hand so I have some media knowledge, but I'm far, far from an expert and I'm confused about some timing issues I'm getting.
>
> I have a pre-encoded FLV of big buck bunny and I am using ffmpeg to push the video (without transcoding) into my RTMP server, and I'm using ffplay to act as an RTMP client.  When I do zero transcoding the audio and video are perfectly synced, but once I pass the packets through the x264enc for encoding, audio is now several seconds before the corresponding video.  My understanding is that RTMP clients/video players would use the RTMP Timestamp values to keep both audio and video in sync, but either the timestamps that I'm getting from gstreamer are incorrect or I'm misunderstanding something.
>
> For transcoding I'm using the following pipeline for my proof of concept:
>
> -------------------
> appsrc name=input ! decodebin ! videoscale ! video/x-raw,width=800,height=600 ! x264enc speed-preset=veryfast name=encoder ! h264parse name=parser ! appsink name=output
> -------------------
>

The issue is likely something to do with timestamps being wrong when
pushing video into the pipeline (or getting it out of the pipeline),
or synchronization breaking due to some other reason on the sender
side (probably in the muxer).

The simplest way to fix this would be to let gstreamer do the muxing
too, with a pipeline like:

appsrc name=vinput ! queue ! decodebin ! videoscale !
video/x-raw,width=800,height=600 ! x264enc speed-preset=veryfast
name=encoder ! h264parse name=parser ! mux.video
appsrc name=ainput ! queue ! mux.audio
flvmux name=mux ! appsink name=voutput

Here flvmux will ensure that synchronization is taking place
correctly, as long as the audio and video buffers pushed into the
appsrc are timestamped correctly.

Cheers,
Nirbheek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20220215/3fd00bce/attachment-0001.htm>