Can't add video stream to video compositor
David Ing
ding at panopto.com
Mon Apr 16 15:13:27 UTC 2018
I have never used the videoaggregator, but I did something similar (Picture
in Picture) using GES (a.k.a. Gstreamer Editing Services). Basically I
followed the GES examples, but I also needed help from the mailing list.
https://lists.freedesktop.org/archives/gstreamer-devel/2018-March/067402.html
https://lists.freedesktop.org/archives/gstreamer-devel/2018-March/067349.html
The nice thing about GES is that it allows you to define a timeline and
most of the hard work is abstracted away.
On Mon, Apr 16, 2018 at 2:33 AM, Weber, Holger <Holger.Weber at telerob.com>
wrote:
> Hi,
>
>
>
> I try to mix different videos into one image. Therefore I use the
> compositor element.
>
> As source I use three different video files (same format). This is a test
> setup and the file sources should
>
> be replaced by udpsrc as soon as the test is running.
>
>
>
> At the beginning is create the pipeline as shown here:
>
>
>
> bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) \
>
>
> -> compositor ->
> capsfilter -> d3dvideosink
>
> bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) /
>
>
>
> I add all elements and after that I start the pipline. The output screen
> shows up and the two videos are mixed into one (ok).
>
>
>
> Some time later I want to add a third video stream, while the pipeline is
> still live. So the pipeline I want to have should look like
>
> below:
>
>
>
> bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) \
>
>
> ----->
> compositor -> capsfilter -> d3dvideosink
>
> bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) / /
>
> bin(Filesrc -> tsdemux -> h264parse -> avdec_h264 -> queue) /
>
>
>
> I add the third stream as follows:
>
> 1) start a timer cb for doing the work
>
> 2) cb is called
>
> 3) create the new stream bin with all elements
>
> 4) add the bin to the pipeline
>
> 5) request a new sink pad (sink_2) from the compositor and link it to my
> bin
>
> 6) set GST_STATE_PLAYING for my bin
>
>
>
> After adding the third stream I get an internal data stream error and the
> playback terminates. I see that it has something to do with caps
> negotiation but I have no idea how to solve that.
>
> I'm also not sure if the order of adding the stream is right.
>
>
>
> Below is the output of the video videoaggregator:
>
>
>
>
>
>
>
> 0:00:04.340065238 5372 d806120 DEBUG aggregator
> gstaggregator.c:1370:gst_aggregator_request_new_pad:<mixer> Adding pad
> sink_2
>
> Linking stream to mixer
>
> 0:00:04.340164542 5372 d806120 DEBUG videoaggregator
> gstvideoaggregator.c:962:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2>
> Get caps with filter: (NULL)
>
> 0:00:04.340271020 5372 d806120 DEBUG videoaggregator
> gstvideoaggregator.c:1009:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2>
> Returning caps: video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1,
> 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ],
> interlace-mode=(string)mixed, format=(string){ Y444, Y42B, YUY2, UYVY,
> YVYU, I420, YV12, NV12, NV21, Y41B, RGB, BGR, xRGB, xBGR, RGBx, BGRx }
>
> End Of Timeout CB
>
> 0:00:04.368477511 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.220236, diff -0:00:00.034084034, timestamp
> 0:00:04.160000000
>
> 0:00:04.368641004 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.368675741 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.368707080 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.368750502 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.368836969 5372 d999240 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.400371842 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.220695, diff -0:00:00.028304133, timestamp
> 0:00:04.200000000
>
> 0:00:04.400502486 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.400539111 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.400604433 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.400636149 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.448112148 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.220049, diff -0:00:00.036412067, timestamp
> 0:00:04.240000000
>
> 0:00:04.448252608 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.448325859 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.448353800 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.448382496 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.491757956 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.220017, diff -0:00:00.028644199, timestamp
> 0:00:04.280000000
>
> 0:00:04.492040387 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.492075502 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.492123455 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.492152907 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.522669442 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.221226, diff -0:00:00.024879829, timestamp
> 0:00:04.320000000
>
> 0:00:04.522871826 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.522912605 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.522937903 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.522967354 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.559194437 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.221642, diff -0:00:00.034092945, timestamp
> 0:00:04.360000000
>
> 0:00:04.559336785 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.559370012 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.559416832 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.559484042 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.611193109 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.220725, diff -0:00:00.037583432, timestamp
> 0:00:04.400000000
>
> 0:00:04.611332059 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.611364531 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.611409463 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.611455906 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.641116841 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.221058, diff -0:00:00.025611945, timestamp
> 0:00:04.440000000
>
> 0:00:04.641256547 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.641374352 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.641400405 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.641459308 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.696486941 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.220465, diff -0:00:00.035588154, timestamp
> 0:00:04.480000000
>
> 0:00:04.696580582 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.696612676 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.696635708 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.696681773 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> 0:00:04.732660029 5372 d999270 DEBUG videoaggregator
> gstvideoaggregator.c:1023:gst_video_aggregator_update_qos:<mixer>
> Updating QoS: proportion 0.222089, diff -0:00:00.020380036, timestamp
> 0:00:04.520000000
>
> 0:00:04.732872230 5372 d999270 WARN aggregator
> gstaggregator.c:1393:gst_aggregator_query_latency_unlocked:<mixer>
> Latency query failed
>
> 0:00:04.732935664 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_0>
> return of event push is 1
>
> 0:00:04.732962095 5372 d999270 DEBUG aggregator
> gstaggregator.c:1571:gst_aggregator_event_forward_func:<mixer:sink_2> not
> sending event to inactive pad
>
> 0:00:04.732990413 5372 d999270 DEBUG aggregator
> gstaggregator.c:1575:gst_aggregator_event_forward_func:<mixer:sink_1>
> return of event push is 1
>
> Dynamic pad created, linking demuxer/decoder
>
> 0:00:04.819389050 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:962:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2>
> Get caps with filter: (NULL)
>
> 0:00:04.819510254 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1009:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2>
> Returning caps: video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1,
> 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ],
> interlace-mode=(string)mixed, format=(string){ Y444, Y42B, YUY2, UYVY,
> YVYU, I420, YV12, NV12, NV21, Y41B, RGB, BGR, xRGB, xBGR, RGBx, BGRx }
>
> 0:00:04.879588421 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:962:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2>
> Get caps with filter: video/x-raw
>
> 0:00:04.879594839 5372 db15400 DEBUG videoaggregator
> gstvideoaggregator.c:1841:gst_video_aggregator_sink_event:<mixer:sink_2>
> Got stream-start event on pad mixer:sink_2
>
> 0:00:04.879772303 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1009:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2>
> Returning caps: video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1,
> 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ],
> interlace-mode=(string)mixed, format=(string){ Y444, Y42B, YUY2, UYVY,
> YVYU, I420, YV12, NV12, NV21, Y41B, RGB, BGR, xRGB, xBGR, RGBx, BGRx }
>
> 0:00:04.879798356 5372 db15400 DEBUG aggregator
> gstaggregator.c:1202:gst_aggregator_default_sink_event:<mixer:sink_2>
> Eating event: stream-start event: 1cc29c28, time 99:99:99.999999999,
> seq-num 390, GstEventStreamStart, stream-id=(string)
> 6feac829e54d6d117e303c312fbdde0a9d5adbc93378c4cd4cd4b77a408402a4:1/0000012c,
> flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, stream=(GstStream)"\(GstStream\)\
> stream12", group-id=(uint)2;
>
> 0:00:04.999976212 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:962:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2>
> Get caps with filter: video/x-raw
>
> 0:00:05.000110254 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1009:gst_video_aggregator_pad_sink_getcaps:<mixer:sink_2>
> Returning caps: video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1,
> 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ],
> interlace-mode=(string)mixed, format=(string){ Y444, Y42B, YUY2, UYVY,
> YVYU, I420, YV12, NV12, NV21, Y41B, RGB, BGR, xRGB, xBGR, RGBx, BGRx }
>
> 0:00:05.003960455 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> video/x-raw, format=(string)I420, width=(int)720, height=(int)576,
> interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004057871 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> not accepted caps video/x-raw, format=(string)I420, width=(int)720,
> height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004126968 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> video/x-raw, format=(string)I420, width=(int)720, height=(int)576,
> interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004186626 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> not accepted caps video/x-raw, format=(string)I420, width=(int)720,
> height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004470945 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> video/x-raw, format=(string)I420, width=(int)720, height=(int)576,
> interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004551370 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> not accepted caps video/x-raw, format=(string)I420, width=(int)720,
> height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004594792 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> video/x-raw, format=(string)I420, width=(int)720, height=(int)576,
> interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004655205 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> not accepted caps video/x-raw, format=(string)I420, width=(int)720,
> height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004737896 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> video/x-raw, format=(string)I420, width=(int)720, height=(int)576,
> interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004807748 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> not accepted caps video/x-raw, format=(string)I420, width=(int)720,
> height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004887040 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:1990:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> video/x-raw, format=(string)I420, width=(int)720, height=(int)576,
> interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
> 0:00:05.004957648 5372 db155b0 DEBUG videoaggregator
> gstvideoaggregator.c:2020:gst_video_aggregator_pad_sink_acceptcaps:<mixer:sink_2>
> not accepted caps video/x-raw, format=(string)I420, width=(int)720,
> height=(int)576, interlace-mode=(string)mixed, pixel-aspect-ratio=(fraction)12/11,
> chroma-site=(string)mpeg2, colorimetry=(string)bt601,
> framerate=(fraction)25/1
>
>
>
> Regards,
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180416/b0362240/attachment-0001.html>
More information about the gstreamer-devel
mailing list