NLE composition inconsistent duration
amindfv at mailbox.org
amindfv at mailbox.org
Fri Oct 14 05:05:35 UTC 2022
On Wed, Oct 12, 2022 at 08:02:53PM -0300, Rodrigo Santos via gstreamer-devel wrote:
> I created a pastebin having a complete Rust application that reproduces this error: https://pastebin.com/3pNDyM4e (you only have to change <URI1> and <URI2> before running it).
>
>
>
> We're basically creating a nlecomposition that has 2 nleurisources. The first one starts at 0s and has 5s of duration, and the second one starts at 5s and has 10s of duration. We're trying to render this pipeline using autovideosink. The first video plays without any problem but we are seeing a LOT of frames being dropped when the second video starts to play. If instead of using an autovideosink we write the output to a file, the resulting video file plays in any player without problems.
>
>
>
> This is the type of QoS message we're getting as soon as the second video starts (no QoS message is posted while playing the first video):
>
> QoS: Qos(Message { ptr: 0x1b77e336810, type: "qos", seqnum: 1347, src: Some("vp8dec0"), structure: Some(GstMessageQOS, live=(boolean)false, running-time=(guint64)9999000000, stream-time=(guint64)9999000000, timestamp=(guint64)9999000000, duration=(guint64)18446744073709551615, jitter=(gint64)569225500, proportion=(double)0.6517624653823002, quality=(int)1000000, format=(GstFormat)buffers, processed=(guint64)6, dropped=(guint64)124;) })
>
To bisect the problem, do you see any QoS messages when using autovideosink and simple segment seeking instead of nle{urisource,composition}s?
I wonder if this could be related in any way to an issue I've encountered, where clocks seem not to agree. E.g. QoS messages occur because according to one part of the system you're at 5.0 seconds (in the overall timeline), while other parts believe you're at 0.0 seconds (in the individual clip), thus creating an appearance of 5 seconds of latency? The fact that the first clip has no QoS issues and the second experiences major failure makes me wonder about that part of the code.
(Another couple shots in the dark: if you make the second clip really long, does the playback ever "recover," and stop giving QoSs? What if you add a third and fourth clip after the first two?)
Cheers,
Tom
>
>
> Also this is the output we're seeing on the console when enabling debug messages (there are plenty of messages similar to that one):
>
>
>
> 0:00:05.014678000 23072 00000223CD18A240 WARN videodecoder gstvideodecoder.c:3675:gst_video_decoder_clip_and_push_buf:<avdec_h264-0> Dropping frame due to QoS. start:0:00:00.100000000 deadline:0:00:00.066666667 earliest_time:0:00:00.245362000
>
>
>
> Any ideas why we cannot play this pipeline smoothly using autovideosink? We tested it on Windows and Mac and got the same result on both OSs. Also, I'm using GStreamer 1.20.2
>
>
>
>
>
> Rodrigo Santos
>
>
>
>
>
>
>
>
> ---- On Wed, 12 Oct 2022 15:41:03 -0300 Thibault Saunier <thibault at saunier.eu> wrote ---
>
>
>
> Hi,
>
>
>
> On Wed, 2022-10-12 at 09:20 -0700, Lucas McGartland via gstreamer-devel
> wrote:
> > Thibault,
> >
> > Managed to get the pipeline properly writing to a file by setting the
> > proper caps for the framerate.
> >
> > However, we're unable to output to autovideosink due to a QoS event.
> > videodecoder
> > gstvideodecoder.c:3675:gst_video_decoder_clip_and_push_buf:<avdec_h26
> > 4-1> Dropping frame due to QoS. start:0:00:00.000000000
> > deadline:0:00:00.000000000 earliest_time:0:00:03.034568335
> > All the frames get dropped because they are arriving "late"
> >
> > How would we go about updating the latency for this?
>
> That sounds weird, it shouldn't have anything to do with latency as it
> is not a live pipeline. I am not sure what is happenning here, would
> need some investigation.
>
> > And regarding GES, we have some specific requirements that need finer
> > grained control of the timeline state than what GES allows.
>
>
> OOc, what do you think is not doable with GES that you need? And why do
> you think we could not enhance GES so it can solve that?
>
> Regards,
>
> - Thibault
>
>
> > Thanks,
> > Luke
> >
> >
> >
> > ---- On Tue, 11 Oct 2022 09:58:48 -0700 Thibault Saunier
> > <mailto:thibault at saunier.eu> wrote ---
> >
> > > Hi Rodrigo,
> > >
> > >
> > > That sounds weird I just tested your program and it just works
> > > here,
> > > ooc, what version of Gst are you using?
> > >
> > > By the way, you should also commit the composition, emitting the
> > > 'commit' signal (but for this simple case it will be done
> > > automatically).
> > >
> > > Also, have you checked the GStreamer Editing Services? It is
> > > simpler to
> > > use and mighthelp with your use case.
> > >
> > > Regards,
> > >
> > > - Thibault
> > >
> > > On Tue, 2022-10-11 at 11:53 -0300, Rodrigo Santos via gstreamer-
> > > devel
> > > wrote:
> > > > Hi,
> > > >
> > > > I have the following code in Rust that creates a nlecomposition
> > > > and
> > > > adds to it two nleurisource elements. The duration of both
> > > > sources is
> > > > set to 10s, and the start of the second source is set to 10s. The
> > > > output is being stored in a file.
> > > >
> > > > My expectation is that the video file would have a total of 20s
> > > > of
> > > > duration, but its actual duration is 33s: the first video has a
> > > > duration of 22s while the second video has 11s of duration. Is
> > > > this a
> > > > bug or am I missing something?
> > > >
> > > > The weird thing is if I don't change the uri when I create the
> > > > second
> > > > source (both sources having the same value for the uri property)
> > > > the
> > > > resulting video file will have 20s of duration as expected (each
> > > > video plays for 10s).
> > > >
> > > > fn add_to_bin(bin: &Element, element: &Element) {
> > > > let bin = bin.dynamic_cast_ref::<Bin>().unwrap();
> > > > bin.add_many(&[element]).unwrap();
> > > > }
> > > >
> > > > fn create_nle_src (uri: &str, start: u64, inpoint: u64, duration:
> > > > i64, priority: u32) -> Element {
> > > > let nlesource = ElementFactory::make("nleurisource",
> > > > None).unwrap();
> > > >
> > > > nlesource.set_property("uri", uri);
> > > > nlesource.set_property("start", start);
> > > > nlesource.set_property("inpoint", inpoint);
> > > > nlesource.set_property("duration", duration);
> > > > nlesource.set_property("priority", priority);
> > > >
> > > > nlesource
> > > > }
> > > >
> > > > fn main() {
> > > > <....>
> > > > let comp = ElementFactory::make("nlecomposition", Some("test-
> > > > composition")).unwrap();
> > > > let caps = "video/x-raw(ANY)";
> > > > let caps = Caps::from_str(caps).expect("Could not create caps
> > > > from str");
> > > > comp.set_property("caps", caps);
> > > >
> > > > let uri = "file:///C:/Users/rodsantos/Downloads/video.mp4";
> > > > //big
> > > > buck bunny video
> > > > let inpoint = ClockTime::from_seconds(0).nseconds();
> > > > let duration: i64 =
> > > > ClockTime::from_seconds(10).nseconds().try_into().expect("Cast
> > > > failed");
> > > > let mut start: u64 = 0;
> > > > let nlesource= create_nle_src(uri, start, inpoint, duration,
> > > > 1);
> > > >
> > > > add_to_bin(&comp, &nlesource);
> > > >
> > > > start += duration as u64;
> > > >
> > > > let uri =
> > > > "file:///C:/Users/rodsantos/Downloads/sintel_trailer-
> > > > 480p.webm"; //sintel trailer video
> > > > let duration: i64 =
> > > > ClockTime::from_seconds(10).nseconds().try_into().expect("Cast
> > > > failed");
> > > > let nlesource = create_nle_src(uri, start, inpoint, duration,
> > > > 1);
> > > > add_to_bin(&comp, &nlesource);
> > > >
> > > > comp.emit_by_name::<bool>("commit", &[&true]);
> > > >
> > > > let videoconvert = ElementFactory::make("videoconvert",
> > > > None).unwrap();
> > > > let compositor = ElementFactory::make("compositor",
> > > > None).unwrap();
> > > > let encoder = ElementFactory::make("x264enc", None).unwrap();
> > > > let mux = ElementFactory::make("avimux", None).unwrap();
> > > > let filesink = ElementFactory::make("filesink",
> > > > None).unwrap();
> > > >
> > > > let pipeline = Pipeline::new(None);
> > > > pipeline.add_many(&[
> > > > &comp,
> > > > &videoconvert,
> > > > &compositor,
> > > > &encoder,
> > > > &mux,
> > > > &filesink,
> > > > ]).unwrap();
> > > >
> > > > Element::link_many(&[
> > > > &comp,
> > > > &videoconvert,
> > > > &compositor,
> > > > &encoder,
> > > > &mux,
> > > > &filesink
> > > > ]).unwrap();
> > > >
> > > > filesink.set_property("location", "test.avi");
> > > >
> > > > pipeline.set_state(State::Playing).expect("Could not set
> > > > pipeline
> > > > to Playing state");
> > > >
> > > > <....>
> > > > }
> > > >
> > > >
> > > > Rodrigo Santos
> > > >
> > > >
> > >
> >
> >
More information about the gstreamer-devel
mailing list