Overlay on a live source

Thu Oct 23 06:50:19 PDT 2014

I have compiled the latest source from git (as of two evenings ago) with the new timeout property on compositor. The example pipeline I provided previously now appears to work much better when the videomixer is switched out for the compositor. I have now continued on to experiment with a udpsrc in place of tstsrc0 by using the following pipelines:

gst-launch-1.0 -v udpsrc port=3000 ! \
                  tsdemux ! h264parse ! avdec_h264 ! \
                  videoscale add-borders=true ! video/x-raw,width=1280,height=720 ! \
                  compositor name=mix timeout=100000000 ! \
                  videoconvert ! \
                  ximagesink \
                  videotestsrc name=tstsrc1 pattern=ball background-color=0 ! \
                  video/x-raw,format=ARGB,width=1280,height=720,framerate=10/1 ! \
                  mix.

gst-launch-1.0 -v videotestsrc ! video/x-raw,width=300,height=200,framerate=5/1 ! \
                  x264enc ! mpegtsmux ! \
                  udpsink host=127.0.0.1 port=3000

When running just the udpsrc pipeline, the ximagesink does create a window before data is received (which is different from the previous behavior with videomixer), however nothing is displayed in the window until I start the udpsink pipeline and data is received by the udpsrc. Interestingly, I can then stop the udpsink pipeline, change the parameters (such as width and height), and start it again, and the udpsrc pipeline will adjust accordingly and continue running.

It seems like compositor should be able to handle streams that contain dynamic pads and honor the timeout before the stream is dynamically linked. I can see how not knowing the size and framerate of the incoming stream could complicate this, but maybe this can be overcome by determining the output caps of the compositor by the non-live input streams. In my particular case, I'm ensuring that this will work by using a videoscale between the udpsrc and the compositor. For now, I'm addressing the issue by using a videoswitch to switch between the udpsrc/decoder/compositor and a videotestsrc.

Thomas

-----Original Message-----
From: gstreamer-devel [mailto:gstreamer-devel-bounces at lists.freedesktop.org] On Behalf Of Matthew Waters
Sent: Sunday, October 19, 2014 5:22 PM
To: gstreamer-devel at lists.freedesktop.org
Subject: Re: RE: Overlay on a live source

Hi,

On 18/10/14 20:33, Moore, Thomas (FtWorth) wrote:
> I tried videorate without success. The following bug appears to confirm that videorate does not duplicate frames of a live source:
>
> https://bugzilla.gnome.org/show_bug.cgi?idr0104
>
> Thomas
> ________________________________________
> From: gstreamer-devel [gstreamer-devel-bounces at lists.freedesktop.org] 
> on behalf of Paul Barber [paul.barber at oncology.ox.ac.uk]
> Sent: Saturday, October 18, 2014 2:41 AM
> To: Discussion of the development of and with GStreamer
> Subject: Re: Overlay on a live source
>
> Hi,
> I have a similar problem but have not really solved it either. Can you use a videorate element to maintain a constant fast frame rate (by duplicating frames) from a source that may stall or run at a reduced rate for some reason?
> Cheers,
> Paul
>
> On 17 October 2014 22:20:02 BST, "Moore, Thomas (FtWorth)" <Thomas.Moore2 at ATK.COM> wrote:
> Hello,
>
>
> I have an interesting problem that I hope you can help me with. I'm working on an application that will overlay custom graphics onto a live video stream in real time. Ultimately, I'm planning on using a udpsrc and the appropriate decoders to process the video stream and cairooverlay to generate the custom graphics (we already have the cairo part done from a previous project, so that part is easy). I've been experimenting with several different combinations of plugins to achieve the desired effect, but I'm running into two problems.
>
>
> First, the pipeline isn't constructed until the udpsrc receives data. I think this can be overcome by using a videotestsrc in place of the udpsrc (and decoders) and a videoswitch to swap to the video stream once data is detected via a filter pad.
>
>
> Second, if the video stream stops (or pauses), then so does the output of the pipeline. At first, I just had the cairooverlay on top of the uspsrc/decoder chain, and it makes sense that the pipeline will stop if the streaming video stops. My next thought was to use a video mixer to mix the cairooverlay atop a transparent videotestsrc with the live video stream. This approach sort of works, but with a live stream as an input to the video mixer, the pipeline still seems to stop if the video stream stops. Here is an example pipeline that demonstrates the issue:
>
>
> gst-launch-1.0 -v videotestsrc name=tsrc0 is-live=true ! \
>                   video/x-raw,width80,heightr0,framerate=1/1 ! \
>                   videomixer name=x ! \
>                   videoconvert ! \
>                   ximagesink syncúlse \
>                   videotestsrc name=tsrc1 patternºll background-color=0 ! \
>                   video/x-raw,format=GB,width80,heightr0,framerate/1 ! \
>                   mix.

That's correct, videomixer will stop producing data if it does not receive input frames from any pad.  There is recent work on master [1] to fix this using GstAggregator of which the compositor element is based on.  Play around with the timeout property on compositor which should be at least the length of time for one buffer to appear on a single compositor sink pad.

[1]
http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/commit/?id=a41bc98b6e7436e5bb4a465b55674f47a98a6f5f

> You'll notice that tstsrc0, which is the underlay, is running at 1 fps and tstsrc1, the overlay, is running at 10 fps. If is-live is set to false on tstsrc0, then the pipeline runs as expected with the ball moving at 10 fps and the underlay updating at 1 fps. However, when is-live is set to true (which requires the syncúlse on the sink to prevent timing errors), the ball moves for about half a second and then stops for about half a second and then repeats.
>
>
> For these tests, I originally started with GStreamer 0.10 as installed with openSUSE 12.1, but I have since updated to openSUSE 13.1 and am running the following versions:
>
>
> gstreamer 1.4.3-3.2
> gstreamer-plugins-base 1.4.3-2.4
> gstreamer-plugins-good 1.4.3-2.1
> gstreamer-plugins-bad 1.4.3-5.7
> gstreamer-plugins-bad-orig-addon 1.4.3-5.7 gstreamer-plugins-ugly 
> 1.4.3-2.2 gstreamer-plugins-ugly-orig-addon 1.4.3-2.2 
> gstreamer-plugins-libav 1.4.3-1.1
>
>
> Any ideas as to how I can get this working would be greatly appreciated.
>
>
> Thanks!
>
>
> Thomas
>
>
>
>
>
>
>
>
>
>
>
>
>
> ________________________________
>
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.