<div dir="ltr">I have never used the videoaggregator, but I did something similar (Picture in Picture) using GES (a.k.a. Gstreamer Editing Services). Basically I followed the GES examples, but I also needed help from the mailing list.<div> <div><a href="https://lists.freedesktop.org/archives/gstreamer-devel/2018-March/067402.html">https://lists.freedesktop.org/archives/gstreamer-devel/2018-March/067402.html</a></div><div> </div><div><a href="https://lists.freedesktop.org/archives/gstreamer-devel/2018-March/067349.html">https://lists.freedesktop.org/archives/gstreamer-devel/2018-March/067349.html</a> </div><div> </div><div>The nice thing about GES is that it allows you to define a timeline and most of the hard work is abstracted away.</div><div> </div></div></div><div class="gmail_extra"> <div class="gmail_quote">On Mon, Apr 16, 2018 at 2:33 AM, Weber, Holger <<a href="mailto:Holger.Weber@telerob.com" target="_blank">Holger.Weber@telerob.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div lang="DE" link="#0563C1" vlink="#954F72"> <div class="m_4088612774358111026WordSection1"> Hi, I try to mix different videos into one image. Therefore I use the compositor element. As source I use three different video files (same format). This is a test setup and the file sources should be replaced by udpsrc as soon as the test is running. At the beginning is create the pipeline as shown here: bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) \ -> compositor -> capsfilter -> d3dvideosink bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) / I add all elements and after that I start the pipline. The output screen shows up and the two videos are mixed into one (ok). Some time later I want to add a third video stream, while the pipeline is still live. So the pipeline I want to have should look like below: bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) \ -----> compositor -> capsfilter -> d3dvideosink bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) / / bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) / I add the third stream as follows: 1) start a timer cb for doing the work 2) cb is called 3) create the new stream bin with all elements 4) add the bin to the pipeline 5) request a new sink pad (sink_2) from the compositor and link it to my bin 6) set GST_STATE_PLAYING for my bin After adding the third stream I get an internal data stream error and the playback terminates. I see that it has something to do with caps negotiation but I have no idea how to solve that. I'm also not sure if the order of adding the stream is right. Below is the output of the video videoaggregator: 0:00:04.340065238 5372 d806120 DEBUG aggregator gstaggregator.c:1370:gst_aggregator_request_new_pad:<mixer> Adding pad sink_2 Linking stream to mixer 0:00:04.340164542 5372 d806120 DEBUG videoaggregator gstvideoaggregator.c:962:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2> Get caps with filter: (NULL) 0:00:04.340271020 5372 d806120 DEBUG videoaggregator gstvideoaggregator.c:1009:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2> Returning caps: video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)mixed, format=(string){ Y444, Y42B, YUY2, UYVY, YVYU, I420, YV12, NV12, NV21, Y41B, RGB, BGR, xRGB, xBGR, RGBx, BGRx } End Of Timeout CB 0:00:04.368477511 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.220236, diff -0:00:00.034084034, timestamp 0:00:04.160000000 0:00:04.368641004 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.368675741 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.368707080 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.368750502 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.368836969 5372 d999240 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.400371842 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.220695, diff -0:00:00.028304133, timestamp 0:00:04.200000000 0:00:04.400502486 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.400539111 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.400604433 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.400636149 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.448112148 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.220049, diff -0:00:00.036412067, timestamp 0:00:04.240000000 0:00:04.448252608 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.448325859 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.448353800 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.448382496 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.491757956 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.220017, diff -0:00:00.028644199, timestamp 0:00:04.280000000 0:00:04.492040387 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.492075502 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.492123455 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.492152907 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.522669442 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.221226, diff -0:00:00.024879829, timestamp 0:00:04.320000000 0:00:04.522871826 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.522912605 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.522937903 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.522967354 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.559194437 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.221642, diff -0:00:00.034092945, timestamp 0:00:04.360000000 0:00:04.559336785 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.559370012 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.559416832 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.559484042 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.611193109 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.220725, diff -0:00:00.037583432, timestamp 0:00:04.400000000 0:00:04.611332059 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.611364531 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.611409463 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.611455906 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.641116841 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.221058, diff -0:00:00.025611945, timestamp 0:00:04.440000000 0:00:04.641256547 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.641374352 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.641400405 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.641459308 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.696486941 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.220465, diff -0:00:00.035588154, timestamp 0:00:04.480000000 0:00:04.696580582 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.696612676 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.696635708 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.696681773 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 0:00:04.732660029 5372 d999270 DEBUG videoaggregator gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer> Updating QoS: proportion 0.222089, diff -0:00:00.020380036, timestamp 0:00:04.520000000 0:00:04.732872230 5372 d999270 WARN aggregator gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer> Latency query failed 0:00:04.732935664 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0> return of event push is 1 0:00:04.732962095 5372 d999270 DEBUG aggregator gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not sending event to inactive pad 0:00:04.732990413 5372 d999270 DEBUG aggregator gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1> return of event push is 1 Dynamic pad created, linking demuxer/decoder 0:00:04.819389050 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:962:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2> Get caps with filter: (NULL) 0:00:04.819510254 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1009:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2> Returning caps: video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)mixed, format=(string){ Y444, Y42B, YUY2, UYVY, YVYU, I420, YV12, NV12, NV21, Y41B, RGB, BGR, xRGB, xBGR, RGBx, BGRx } 0:00:04.879588421 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:962:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2> Get caps with filter: video/x-raw 0:00:04.879594839 5372 db15400 DEBUG videoaggregator gstvideoaggregator.c:1841:gst_video_aggregator_sink_event:<mixer:sink_2> Got stream-start event on pad mixer:sink_2 0:00:04.879772303 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1009:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2> Returning caps: video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)mixed, format=(string){ Y444, Y42B, YUY2, UYVY, YVYU, I420, YV12, NV12, NV21, Y41B, RGB, BGR, xRGB, xBGR, RGBx, BGRx } 0:00:04.879798356 5372 db15400 DEBUG aggregator gstaggregator.c:1202:gst_aggregator_default_sink_event:<mixer:sink_2> Eating event: stream-start event: 1cc29c28, time 99:99:99.999999999, seq-num 390, GstEventStreamStart, stream-id=(string)6feac829e54d6d117e303c312fbdde0a9d5adbc93378c4cd4cd4b77a408402a4:1/0000012c, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, stream=(GstStream)"\(GstStream\)\ stream12", group-id=(uint)2; 0:00:04.999976212 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:962:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2> Get caps with filter: video/x-raw 0:00:05.000110254 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1009:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2> Returning caps: video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ], interlace-mode=(string)mixed, format=(string){ Y444, Y42B, YUY2, UYVY, YVYU, I420, YV12, NV12, NV21, Y41B, RGB, BGR, xRGB, xBGR, RGBx, BGRx } 0:00:05.003960455 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004057871 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> not accepted caps video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004126968 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004186626 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> not accepted caps video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004470945 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004551370 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> not accepted caps video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004594792 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004655205 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> not accepted caps video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004737896 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004807748 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> not accepted caps video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004887040 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 0:00:05.004957648 5372 db155b0 DEBUG videoaggregator gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2> not accepted caps video/x-raw, format=(string)I420, width=(int)720, height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11, chroma-site=(string)mpeg2, colorimetry=(string)bt601, framerate=(fraction)25/1 Regards, </div> </div> _______________________________________________ gstreamer-devel mailing list <a href="mailto:gstreamer-devel@lists.freedesktop.org">gstreamer-devel@lists.freedesktop.org</a> <a href="https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a> </blockquote></div> </div>