NLE composition inconsistent duration

[Thibault Saunier]

Hi,



On Wed, 2022-10-12 at 09:20 -0700, Lucas McGartland via gstreamer-devel
wrote:
> Thibault,
> 
> Managed to get the pipeline properly writing to a file by setting the
> proper caps for the framerate.
> 
> However, we're unable to output to autovideosink due to a QoS event.
> videodecoder
> gstvideodecoder.c:3675:gst_video_decoder_clip_and_push_buf:<avdec_h26
> 4-1> Dropping frame due to QoS. start:0:00:00.000000000
> deadline:0:00:00.000000000 earliest_time:0:00:03.034568335
> All the frames get dropped because they are arriving "late"
>
> How would we go about updating the latency for this?

That sounds weird, it shouldn't have anything to do with latency as it
is not a live pipeline. I am not sure what is happenning here, would
need some investigation.

> And regarding GES, we have some specific requirements that need finer
> grained control of the timeline state than what GES allows.


OOc, what do you think is not doable with GES that you need? And why do
you think we could not enhance GES so it can solve that?

Regards,

- Thibault


> Thanks,
> Luke
> 
> 
> 
> ---- On Tue, 11 Oct 2022 09:58:48 -0700 Thibault Saunier
> <thibault at saunier.eu> wrote ---
> 
> > Hi Rodrigo, 
> >  
> >  
> > That sounds weird I just tested your program and it just works
> > here, 
> > ooc, what version of Gst are you using? 
> >  
> > By the way, you should also commit the composition, emitting the 
> > 'commit' signal (but for this simple case it will be done 
> > automatically). 
> >  
> > Also, have you checked the GStreamer Editing Services? It is
> > simpler to 
> > use and mighthelp with your use case. 
> >  
> > Regards, 
> >  
> > - Thibault 
> >  
> > On Tue, 2022-10-11 at 11:53 -0300, Rodrigo Santos via gstreamer-
> > devel 
> > wrote: 
> > > Hi, 
> > > 
> > > I have the following code in Rust that creates a nlecomposition
> > > and 
> > > adds to it two nleurisource elements. The duration of both
> > > sources is 
> > > set to 10s, and the start of the second source is set to 10s. The
> > > output is being stored in a file. 
> > > 
> > > My expectation is that the video file would have a total of 20s
> > > of 
> > > duration, but its actual duration is 33s: the first video has a 
> > > duration of 22s while the second video has 11s of duration. Is
> > > this a 
> > > bug or am I missing something? 
> > > 
> > > The weird thing is if I don't change the uri when I create the
> > > second 
> > > source (both sources having the same value for the uri property)
> > > the 
> > > resulting video file will have 20s of duration as expected (each 
> > > video plays for 10s). 
> > > 
> > > fn add_to_bin(bin: &Element, element: &Element) { 
> > >     let bin = bin.dynamic_cast_ref::<Bin>().unwrap(); 
> > >     bin.add_many(&[element]).unwrap(); 
> > > } 
> > > 
> > > fn create_nle_src (uri: &str, start: u64, inpoint: u64, duration:
> > > i64, priority: u32) -> Element { 
> > >     let nlesource = ElementFactory::make("nleurisource", 
> > > None).unwrap(); 
> > > 
> > >     nlesource.set_property("uri", uri); 
> > >     nlesource.set_property("start", start); 
> > >     nlesource.set_property("inpoint", inpoint); 
> > >     nlesource.set_property("duration", duration); 
> > >     nlesource.set_property("priority", priority); 
> > > 
> > >     nlesource 
> > > } 
> > > 
> > > fn main() { 
> > >     <....> 
> > >     let comp = ElementFactory::make("nlecomposition", Some("test-
> > > composition")).unwrap(); 
> > >     let caps = "video/x-raw(ANY)"; 
> > >     let caps = Caps::from_str(caps).expect("Could not create caps
> > > from str"); 
> > >     comp.set_property("caps", caps); 
> > > 
> > >     let uri = "file:///C:/Users/rodsantos/Downloads/video.mp4";
> > > //big 
> > > buck bunny video 
> > >     let inpoint = ClockTime::from_seconds(0).nseconds(); 
> > >     let duration: i64 = 
> > > ClockTime::from_seconds(10).nseconds().try_into().expect("Cast 
> > > failed"); 
> > >     let mut start: u64 = 0; 
> > >     let nlesource= create_nle_src(uri, start, inpoint, duration,
> > > 1); 
> > > 
> > >     add_to_bin(&comp, &nlesource); 
> > > 
> > >     start += duration as u64; 
> > > 
> > >     let uri =
> > > "file:///C:/Users/rodsantos/Downloads/sintel_trailer- 
> > > 480p.webm"; //sintel trailer video 
> > >     let duration: i64 = 
> > > ClockTime::from_seconds(10).nseconds().try_into().expect("Cast 
> > > failed"); 
> > >     let nlesource = create_nle_src(uri, start, inpoint, duration,
> > > 1); 
> > >     add_to_bin(&comp, &nlesource); 
> > > 
> > >     comp.emit_by_name::<bool>("commit", &[&true]); 
> > > 
> > >     let videoconvert = ElementFactory::make("videoconvert", 
> > > None).unwrap(); 
> > >     let compositor = ElementFactory::make("compositor", 
> > > None).unwrap(); 
> > >     let encoder = ElementFactory::make("x264enc", None).unwrap();
> > >     let mux = ElementFactory::make("avimux", None).unwrap(); 
> > >     let filesink = ElementFactory::make("filesink",
> > > None).unwrap(); 
> > > 
> > >     let pipeline = Pipeline::new(None); 
> > >     pipeline.add_many(&[ 
> > >         &comp, 
> > >         &videoconvert, 
> > >         &compositor, 
> > >         &encoder, 
> > >         &mux, 
> > >         &filesink, 
> > >     ]).unwrap(); 
> > > 
> > >     Element::link_many(&[ 
> > >         &comp, 
> > >         &videoconvert, 
> > >         &compositor, 
> > >         &encoder, 
> > >         &mux, 
> > >         &filesink 
> > >     ]).unwrap(); 
> > > 
> > >     filesink.set_property("location", "test.avi"); 
> > > 
> > >     pipeline.set_state(State::Playing).expect("Could not set
> > > pipeline 
> > > to Playing state"); 
> > >     
> > >    <....> 
> > > } 
> > > 
> > > 
> > > Rodrigo Santos 
> > > 
> > > 
> >  
> 
>