NLE composition inconsistent duration
Rodrigo Santos
rsantos at sequence.film
Wed Oct 12 23:02:53 UTC 2022
I created a pastebin having a complete Rust application that reproduces this error: https://pastebin.com/3pNDyM4e (you only have to change <URI1> and <URI2> before running it).
We're basically creating a nlecomposition that has 2 nleurisources. The first one starts at 0s and has 5s of duration, and the second one starts at 5s and has 10s of duration. We're trying to render this pipeline using autovideosink. The first video plays without any problem but we are seeing a LOT of frames being dropped when the second video starts to play. If instead of using an autovideosink we write the output to a file, the resulting video file plays in any player without problems.
This is the type of QoS message we're getting as soon as the second video starts (no QoS message is posted while playing the first video):
QoS: Qos(Message { ptr: 0x1b77e336810, type: "qos", seqnum: 1347, src: Some("vp8dec0"), structure: Some(GstMessageQOS, live=(boolean)false, running-time=(guint64)9999000000, stream-time=(guint64)9999000000, timestamp=(guint64)9999000000, duration=(guint64)18446744073709551615, jitter=(gint64)569225500, proportion=(double)0.6517624653823002, quality=(int)1000000, format=(GstFormat)buffers, processed=(guint64)6, dropped=(guint64)124;) })
Also this is the output we're seeing on the console when enabling debug messages (there are plenty of messages similar to that one):
0:00:05.014678000 23072 00000223CD18A240 WARN videodecoder gstvideodecoder.c:3675:gst_video_decoder_clip_and_push_buf:<avdec_h264-0> Dropping frame due to QoS. start:0:00:00.100000000 deadline:0:00:00.066666667 earliest_time:0:00:00.245362000
Any ideas why we cannot play this pipeline smoothly using autovideosink? We tested it on Windows and Mac and got the same result on both OSs. Also, I'm using GStreamer 1.20.2
Rodrigo Santos
---- On Wed, 12 Oct 2022 15:41:03 -0300 Thibault Saunier <thibault at saunier.eu> wrote ---
Hi,
On Wed, 2022-10-12 at 09:20 -0700, Lucas McGartland via gstreamer-devel
wrote:
> Thibault,
>
> Managed to get the pipeline properly writing to a file by setting the
> proper caps for the framerate.
>
> However, we're unable to output to autovideosink due to a QoS event.
> videodecoder
> gstvideodecoder.c:3675:gst_video_decoder_clip_and_push_buf:<avdec_h26
> 4-1> Dropping frame due to QoS. start:0:00:00.000000000
> deadline:0:00:00.000000000 earliest_time:0:00:03.034568335
> All the frames get dropped because they are arriving "late"
>
> How would we go about updating the latency for this?
That sounds weird, it shouldn't have anything to do with latency as it
is not a live pipeline. I am not sure what is happenning here, would
need some investigation.
> And regarding GES, we have some specific requirements that need finer
> grained control of the timeline state than what GES allows.
OOc, what do you think is not doable with GES that you need? And why do
you think we could not enhance GES so it can solve that?
Regards,
- Thibault
> Thanks,
> Luke
>
>
>
> ---- On Tue, 11 Oct 2022 09:58:48 -0700 Thibault Saunier
> <mailto:thibault at saunier.eu> wrote ---
>
> > Hi Rodrigo,
> >
> >
> > That sounds weird I just tested your program and it just works
> > here,
> > ooc, what version of Gst are you using?
> >
> > By the way, you should also commit the composition, emitting the
> > 'commit' signal (but for this simple case it will be done
> > automatically).
> >
> > Also, have you checked the GStreamer Editing Services? It is
> > simpler to
> > use and mighthelp with your use case.
> >
> > Regards,
> >
> > - Thibault
> >
> > On Tue, 2022-10-11 at 11:53 -0300, Rodrigo Santos via gstreamer-
> > devel
> > wrote:
> > > Hi,
> > >
> > > I have the following code in Rust that creates a nlecomposition
> > > and
> > > adds to it two nleurisource elements. The duration of both
> > > sources is
> > > set to 10s, and the start of the second source is set to 10s. The
> > > output is being stored in a file.
> > >
> > > My expectation is that the video file would have a total of 20s
> > > of
> > > duration, but its actual duration is 33s: the first video has a
> > > duration of 22s while the second video has 11s of duration. Is
> > > this a
> > > bug or am I missing something?
> > >
> > > The weird thing is if I don't change the uri when I create the
> > > second
> > > source (both sources having the same value for the uri property)
> > > the
> > > resulting video file will have 20s of duration as expected (each
> > > video plays for 10s).
> > >
> > > fn add_to_bin(bin: &Element, element: &Element) {
> > > let bin = bin.dynamic_cast_ref::<Bin>().unwrap();
> > > bin.add_many(&[element]).unwrap();
> > > }
> > >
> > > fn create_nle_src (uri: &str, start: u64, inpoint: u64, duration:
> > > i64, priority: u32) -> Element {
> > > let nlesource = ElementFactory::make("nleurisource",
> > > None).unwrap();
> > >
> > > nlesource.set_property("uri", uri);
> > > nlesource.set_property("start", start);
> > > nlesource.set_property("inpoint", inpoint);
> > > nlesource.set_property("duration", duration);
> > > nlesource.set_property("priority", priority);
> > >
> > > nlesource
> > > }
> > >
> > > fn main() {
> > > <....>
> > > let comp = ElementFactory::make("nlecomposition", Some("test-
> > > composition")).unwrap();
> > > let caps = "video/x-raw(ANY)";
> > > let caps = Caps::from_str(caps).expect("Could not create caps
> > > from str");
> > > comp.set_property("caps", caps);
> > >
> > > let uri = "file:///C:/Users/rodsantos/Downloads/video.mp4";
> > > //big
> > > buck bunny video
> > > let inpoint = ClockTime::from_seconds(0).nseconds();
> > > let duration: i64 =
> > > ClockTime::from_seconds(10).nseconds().try_into().expect("Cast
> > > failed");
> > > let mut start: u64 = 0;
> > > let nlesource= create_nle_src(uri, start, inpoint, duration,
> > > 1);
> > >
> > > add_to_bin(&comp, &nlesource);
> > >
> > > start += duration as u64;
> > >
> > > let uri =
> > > "file:///C:/Users/rodsantos/Downloads/sintel_trailer-
> > > 480p.webm"; //sintel trailer video
> > > let duration: i64 =
> > > ClockTime::from_seconds(10).nseconds().try_into().expect("Cast
> > > failed");
> > > let nlesource = create_nle_src(uri, start, inpoint, duration,
> > > 1);
> > > add_to_bin(&comp, &nlesource);
> > >
> > > comp.emit_by_name::<bool>("commit", &[&true]);
> > >
> > > let videoconvert = ElementFactory::make("videoconvert",
> > > None).unwrap();
> > > let compositor = ElementFactory::make("compositor",
> > > None).unwrap();
> > > let encoder = ElementFactory::make("x264enc", None).unwrap();
> > > let mux = ElementFactory::make("avimux", None).unwrap();
> > > let filesink = ElementFactory::make("filesink",
> > > None).unwrap();
> > >
> > > let pipeline = Pipeline::new(None);
> > > pipeline.add_many(&[
> > > &comp,
> > > &videoconvert,
> > > &compositor,
> > > &encoder,
> > > &mux,
> > > &filesink,
> > > ]).unwrap();
> > >
> > > Element::link_many(&[
> > > &comp,
> > > &videoconvert,
> > > &compositor,
> > > &encoder,
> > > &mux,
> > > &filesink
> > > ]).unwrap();
> > >
> > > filesink.set_property("location", "test.avi");
> > >
> > > pipeline.set_state(State::Playing).expect("Could not set
> > > pipeline
> > > to Playing state");
> > >
> > > <....>
> > > }
> > >
> > >
> > > Rodrigo Santos
> > >
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20221012/413a8453/attachment.htm>
More information about the gstreamer-devel
mailing list