<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
So I tried to do a bit more digging on this, not because I have an issue with your pipeline (I will probably go that route in the end) but I suspect there's something else off with my element clocks that's the true problem. Specifically, when I play video
through my pipeline VLC is showing a start time of 1000:00:00.<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Through debugging I have verified the pts and dts values I am setting look correct (by looking at the results of the
<code>buffer.pts()</code> and <code>buffer.dts()</code> values. For example, I'm getting these for the first 3 h264 buffers I am passing in:<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
-------------------</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
INPUT - dts Some(0:00:00.000000000) pts Some(0:00:00.083000000)
<div>INPUT - dts Some(0:00:00.041000000) pts Some(0:00:00.250000000)</div>
INPUT - dts Some(0:00:00.083000000) pts Some(0:00:00.166000000)</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
--------------------</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
(The <code>Some()</code> just means it's not a null value, if you aren't familiar with rust)</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I then added debugging of the values coming out of the appsink, which gives me</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
---------------------</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
dts 999:59:59.917000000 pts 1000:00:00.000000000
<div>Running time: Some(0:00:00.000138500)</div>
<div>Start time: Some(0:00:00.000000000)</div>
Clock time: Some(25:59:28.529101100)</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
dts 999:59:59.959000000 pts 1000:00:00.167000000
<div>Running time: Some(0:00:00.047566400)</div>
<div>Start time: Some(0:00:00.000000000)</div>
Clock time: Some(25:59:28.576529600)</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
dts 1000:00:00.000000000 pts 1000:00:00.083000000
<div>Running time: Some(0:00:00.092902000)</div>
<div>Start time: Some(0:00:00.000000000)</div>
Clock time: Some(25:59:28.621872200)</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
------------------------</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
We can see that the dts and pts values are all being set up so that presentation starts at 1000:00:00. The pipeline's start time is 00:00:00, the current running time seems to be the correct. Clock time seems weird (doesn't match my system time but based
on docs it can be pulling this from a device somewhere, and probably doesn't matter?).</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
So, why would my pipelines be starting from a 1000 pts? I can't find much in the docs to help debug this further</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
For reference, my current iteration of my test code is using the following code to create a pipeline of
<code>appsrc ! decodebin ! videoscale ! capsfilter ! x264enc ! h264parse ! appsink</code></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
---------------</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
let appsrc = ElementFactory::make("appsrc", None).unwrap();
<div> let decoder = ElementFactory::make("decodebin", None).unwrap();</div>
<div> let scale = ElementFactory::make("videoscale", None).unwrap();</div>
<div> let capsfilter = ElementFactory::make("capsfilter", None).unwrap();</div>
<div> let encoder = ElementFactory::make("x264enc", None).unwrap();</div>
<div> let output_parser = ElementFactory::make("h264parse", None).unwrap();</div>
<div> let appsink = ElementFactory::make("appsink", None).unwrap();</div>
<div><br>
</div>
<div> let pipeline = Pipeline::new(None);</div>
<div> pipeline.add_many(</div>
<div> &[&appsrc, &decoder, &scale, &capsfilter, &encoder, &output_parser, &appsink]</div>
<div> ).unwrap();</div>
<div><br>
</div>
<div> appsrc.link(&decoder).unwrap();</div>
<div> Element::link_many(&[&scale, &capsfilter, &encoder, &output_parser, &appsink]).unwrap();</div>
<div><br>
</div>
<div> decoder.connect_pad_added(move |src, src_pad| {</div>
<div> src.link_pads(Some(&src_pad.name()), &scale.clone(), None).unwrap();</div>
<div> });</div>
<div><br>
</div>
<div> let caps = Caps::builder("video/x-raw")</div>
<div> .field("width", 800)</div>
<div> .field("height", 600)</div>
<div> .build();</div>
<div><br>
</div>
<div> capsfilter.set_property("caps", caps);</div>
encoder.set_property_from_str("speed-preset", "veryfast");</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
-------------</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I appreciate any insights you (or anyone) could give!<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Nirbheek Chauhan <nirbheek.chauhan@gmail.com><br>
<b>Sent:</b> Tuesday, February 15, 2022 8:52 AM<br>
<b>To:</b> Discussion of the development of and with GStreamer <gstreamer-devel@lists.freedesktop.org><br>
<b>Cc:</b> Matthew Shapiro <me@mshapiro.net><br>
<b>Subject:</b> Re: Trying to keep non-transcoded audio and transcoded video in sync</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hi,<br>
<br>
On Mon, Feb 14, 2022 at 4:45 PM Matthew Shapiro via gstreamer-devel<br>
<gstreamer-devel@lists.freedesktop.org> wrote:<br>
><br>
> I have a custom written RTMP server written in Rust, and I'm trying to implement gstreamer to provide dynamic transcoding pipelines. I wrote all the RTMP code by hand so I have some media knowledge, but I'm far, far from an expert and I'm confused about
some timing issues I'm getting.<br>
><br>
> I have a pre-encoded FLV of big buck bunny and I am using ffmpeg to push the video (without transcoding) into my RTMP server, and I'm using ffplay to act as an RTMP client. When I do zero transcoding the audio and video are perfectly synced, but once I pass
the packets through the x264enc for encoding, audio is now several seconds before the corresponding video. My understanding is that RTMP clients/video players would use the RTMP Timestamp values to keep both audio and video in sync, but either the timestamps
that I'm getting from gstreamer are incorrect or I'm misunderstanding something.<br>
><br>
> For transcoding I'm using the following pipeline for my proof of concept:<br>
><br>
> -------------------<br>
> appsrc name=input ! decodebin ! videoscale ! video/x-raw,width=800,height=600 ! x264enc speed-preset=veryfast name=encoder ! h264parse name=parser ! appsink name=output<br>
> -------------------<br>
><br>
<br>
The issue is likely something to do with timestamps being wrong when<br>
pushing video into the pipeline (or getting it out of the pipeline),<br>
or synchronization breaking due to some other reason on the sender<br>
side (probably in the muxer).<br>
<br>
The simplest way to fix this would be to let gstreamer do the muxing<br>
too, with a pipeline like:<br>
<br>
appsrc name=vinput ! queue ! decodebin ! videoscale !<br>
video/x-raw,width=800,height=600 ! x264enc speed-preset=veryfast<br>
name=encoder ! h264parse name=parser ! mux.video<br>
appsrc name=ainput ! queue ! mux.audio<br>
flvmux name=mux ! appsink name=voutput<br>
<br>
Here flvmux will ensure that synchronization is taking place<br>
correctly, as long as the audio and video buffers pushed into the<br>
appsrc are timestamped correctly.<br>
<br>
Cheers,<br>
Nirbheek<br>
</div>
</span></font></div>
</body>
</html>