[gst-devel] Compositing and GStreamer
Timothy Braun
braunsquared at gmail.com
Mon Nov 29 03:06:09 CET 2010
Hi Edward,
Thanks for the quick reply. It has been most informative. A couple
questions if I may. For the video rescale/capsfilter part, what is this
doing and how should it look in the pipeline? The input videos are all
240x240 px with a goal output frame size of 720x480px.
For a very raw test, I am using this to build my pipeline:
GstPipeline *pipeline = GST_PIPELINE(gst_pipeline_new("pipeline"));
GstCaps *caps = gst_caps_from_string("video/x-raw-yuv;video/x-raw-rgb");
// Create our composition
GstElement *comp = gst_element_factory_make("gnlcomposition",
"composition");
g_object_set(G_OBJECT(comp), "caps", caps, NULL);
GstElement *src1 = gst_element_factory_make("gnlfilesource", "source1");
g_object_set(G_OBJECT(src1), "location", "loop1.mp4",
"start", 0,
"duration", 30 * GST_SECOND,
"media-start", 0,
"media-duration", 30 * GST_SECOND,
"priority", 1,
"caps", caps,
NULL);
GstElement *src2 = gst_element_factory_make("gnlfilesource", "source2");
g_object_set(G_OBJECT(src2), "location", "loop2.mp4",
"start", 0,
"duration", 30 * GST_SECOND,
"media-start", 0,
"media-duration", 30 * GST_SECOND,
"priority", 2,
"caps", caps,
NULL);
GstElement *oper = gst_element_factory_make("gnloperation", NULL);
g_object_set(G_OBJECT(oper), "caps", caps, "expandable", TRUE, NULL);
GstElement *mixer = gst_element_factory_make("videomixer", NULL);
gst_bin_add(GST_BIN(oper), mixer);
// listen for the input priority change signal
// to update the video mixer pad
g_object_connect(oper, "signal::input-priority-changed",
onPriorityChange, mixer, NULL );
// add the sources to the composition
gst_bin_add(GST_BIN(comp), src1);
gst_bin_add(GST_BIN(comp), src2);
gst_bin_add(GST_BIN(comp), oper);
// build the output stream
GstElement *color = gst_element_factory_make("ffmpegcolorspace",
"colorspace");
GstElement *identity = gst_element_factory_make("identity", "ident");
g_object_set(identity, "single-segment", TRUE, NULL);
GstElement *enc = gst_element_factory_make("ffenc_mpeg1video",
"encoder");
GstElement *mux = gst_element_factory_make("ffmux_mpeg", "mux");
GstElement *queue = gst_element_factory_make("queue", "queue");
GstElement *sink = gst_element_factory_make("filesink", "sink");
g_object_set(sink, "location", "output.mpg", NULL);
gst_bin_add_many(GST_BIN(pipeline), comp, color, identity, enc, mux,
queue, sink, NULL);
/*
g_object_connect (comp, "signal::pad-added",
onPad, mixer, NULL);
*/
gst_element_link_many(mixer, queue, color, identity, enc, mux, sink,
NULL);
It seems to link up fine, but fails to stream as it gets stuck in the paused
state. I've tried adding queues all around, but no luck. Here is the debug
output:
** Message: Creating run loop...
** Message: Building pipeline...
** Message: Attaching to bus...
** Message: Setting state to PLAYING...
0:00:00.141504000 [334m91821[00m 0x100609d30 [33;01mWARN [00m
[00;01;34m gnlsource
gnlsource.c:545:gnl_source_change_state:<source2>[00m Couldn't find a valid
source pad
0:00:00.162296000 [334m91821[00m 0x100609d30 [33;01mWARN [00m
[00;01;35m GST_SCHEDULING
gstpad.c:4692:gst_pad_get_range:<source:src>[00m getrange failed unexpected
0:00:00.191513000 [334m91821[00m 0x100609d30 [33;01mWARN [00m
[00;01;34m gnlsource
gnlsource.c:545:gnl_source_change_state:<source1>[00m Couldn't find a valid
source pad
0:00:00.199956000 [334m91821[00m 0x100686b10 [33;01mWARN [00m
[00m qtdemux qtdemux.c:5801:qtdemux_parse_trak:<qtdemux0>[00m
unknown version 00000000
0:00:00.200693000 [334m91821[00m 0x100609d30 [33;01mWARN [00m
[00;01;35m GST_SCHEDULING
gstpad.c:4692:gst_pad_get_range:<source:src>[00m getrange failed unexpected
0:00:00.210835000 [334m91821[00m 0x10064f030 [33;01mWARN [00m
[00m qtdemux qtdemux.c:5801:qtdemux_parse_trak:<qtdemux1>[00m
unknown version 00000000
0:00:00.244706000 [334m91821[00m 0x101879f60 [33;01mWARN [00m
[00;01;34m gnlsource
gnlsource.c:221:element_pad_added_cb:<source2>[00m We already have
(pending) ghost-ed a valid source pad (ghostpad:'':'', pendingblock:1
0:00:00.244852000 [334m91821[00m 0x1006501f0 [33;01mWARN [00m
[00;01;34m gnlsource
gnlsource.c:221:element_pad_added_cb:<source1>[00m We already have
(pending) ghost-ed a valid source pad (ghostpad:'':'', pendingblock:1
** Message: Priority Changed:
composition::gnloperation0->gnloperation0::sink_1(videomixer0::sink_1) - 2
** Message: Priority Changed:
composition::gnloperation0->gnloperation0::sink_0(videomixer0::sink_0) - 1
I've been battling with this for a bit now, can't seem to make any headway.
Any input is, again, much appreciated.
Best,
Tim
On Sun, Nov 28, 2010 at 3:58 AM, Edward Hervey <bilboed at gmail.com> wrote:
> Hi,
>
> On Sat, 2010-11-27 at 13:50 -0500, Timothy Braun wrote:
> > Kapil,
> > Thanks for the suggestion, but with multifilesrc I would have to
> > have the files static with incrementing named files. A single box in
> > the 3x2 grid may contain the same file multiple times so I'm afraid it
> > won't be the best solution.
> >
> > I guess, ultimately, there's multiple ways to attack this one from
> > what I've been able to find. Here are the two that I've been looking
> > at, I'm just not sure which is the better solution at this point:
> > * A single gnonlin composition with a 4 gnlsources similar in
> > setup as the gst-launch text I have below.
>
> Using one composition would actually be the 'proper' way.
>
> > * 6 gnonlin compositions, each feeding to a single videomixer
> > which combines them into the final frame.
> > * This path I'm currently investigating. I have a test
> > written in C, but I'm having some difficulties with
> > pad linkage as I still don't have a complete
> > understanding of when certain things will exist and
> > how to get them.
> > * Here's currently whats happening:
> > * Create a new pipeline
> > * Create a videomixer
> > * Create 6 gnonlin compositions each with a
> > pad-added signal callback to connect
> > gnlcomposition pad to videomixer.
> > * ... (this is were it's going wrong)
> > * In the pad-added callback I have:
> > * static void onPad(GstElement *comp, GstPad
> > *pad, GstElement *sink) {
> > GstPad *v = gst_element_get_pad(sink,
> > "sink");
> > gst_pad_link(pad, v);
> > gst_object_unref(v);
> > }
> > * gst_element_get_pad is not returning a pad
> > from the video mixer (sink) which leads me to
> > believe that I'm either not asking in the
> > right manner or the pad doesn't exist. (I'm
> > aware that gst_element_get_pad is deprecated,
> > I'm just looking to test at the moment)
> > * I noticed in one of the repositories under a
> > unit test, the videomixer was attached as a
> > gnloperation? Is this the better path to
> > take?
> >
> > This all leads me to a couple more questions as well:
> > * A video mixer pad has xpos and ypos properties. This would
> > let me shift the video around without needing a video box
> > which I believe may be more efficient?
>
> Yes, it will be more efficient.
>
> > * If I use the xpos and ypos properties, is the video mixer
> > smart enough to change the frame size appropriately or will it
> > simply crop the frame to the size of the largest input frame?
> > * If so, would it be better to add a videobox to do the
> > adjustments for me, or feed in a solid color
> > background of the required output size?
>
> No, it won't change the size, but what you could do is mix the
> original sizes with original offsets and then downconvert the video
> later.
>
> Example for one 3x2 segment:
>
> Create a gnloperation with a videomixer in it with a gnl priority of
> 0.
> Create a gnlfilesource for each clip with increasing priorities (1->6)
> going from left-right and then top to bottom:
> 1 2 3
> 4 5 6
>
> Connect to the gnloperation 'input-priority-changed' signal. When your
> callback is called, you will know which priority is being connected to
> which gnloperation ghostpad. You can get the videomixer sink pad by
> using the gst_ghost_pad_get_target() method and then setting the proper
> xpos/ypos property on that pad based on the priority of the feed being
> provided.
>
> Set 'video/x-raw-yuv;video/x-raw-rgb' as the caps property on all your
> sources.
>
> Set duration and media-duration of *all* gnlobjects to the same
> duration.
> If you want to add another segment of 3x2 clips, you'll need to re-add
> all those 7 objects with a modified 'start' property.
>
> First connect your composition to an imagesink to make sure the result
> is what you want. When it is, insert a videoscale element followed with
> a capsfilter with your target resolution.
>
> Hope this helps.
>
> >
> > Thanks again for the time. I know there's a lot of questions above,
> > but any help of any kind is greatly appreciated.
> >
> > All the best,
> > Tim
> >
> >
> > On Fri, Nov 26, 2010 at 1:04 AM, Kapil Agrawal <kapil.agl at gmail.com>
> > wrote:
> > Just a quick clue that might help, try using multifilesrc ?
> >
> >
> > On Thu, Nov 25, 2010 at 9:47 PM, Timothy Braun
> > <braunsquared at gmail.com> wrote:
> >
> >
> > Hello Everyone,
> > I'm fairly new to G-Streamer so any input you can
> > provide is much appreciated. I'm working on a project
> > where we need to generate a 2 minute video which is a
> > composite of a total of 24 input videos. The output
> > video will have 4 different 30 second sections, each
> > containing a 3x2 grid of the smaller input videos.
> > The input videos are all naturally at 240x240 with the
> > goal of having a final output frame size of 720x480.
> >
> > Using gst-launch, I've been able to construct a
> > sample 30 second clip using a combination of inputs,
> > videoboxes and a videomixer. Here is what I've come
> > up with so far:
> >
> > videomixer name=mix ! ffmpegcolorspace !
> > ffenc_mpeg1video ! ffmux_mpeg name=mux ! queue !
> > filesink location=output.mpg
> > adder name=adder ! audioconvert ! ffenc_mp2 ! mux.
> > filesrc location=loop1.mp4 ! decodebin name=decode1
> > decode1. ! videobox border-alpha=0 top=-240 left=0 !
> > queue ! mix.
> > decode1. ! adder.
> > filesrc location=loop2.mp4 ! decodebin name=decode2
> > decode2. ! videobox border-alpha=0 top=-240
> > left=-240 ! queue ! mix.
> > decode2. ! adder.
> > filesrc location=loop3.mp4 ! decodebin name=decode3
> > decode3. ! videobox border-alpha=0 top=-240
> > left=-480 ! queue ! mix.
> > decode3. ! adder.
> > filesrc location=loop4.mp4 ! decodebin name=decode4
> > decode4. ! videobox border-alpha=0 top=0 left=0 !
> > queue ! mix.
> > decode4. ! adder.
> > filesrc location=loop5.mp4 ! decodebin name=decode5
> > decode5. ! videobox border-alpha=0 top=0 left=-240 !
> > queue ! mix.
> > decode5. ! adder.
> > filesrc location=loop6.mp4 ! decodebin name=decode6
> > decode6. ! videobox border-alpha=0 top=0 left=-480 !
> > queue ! mix.
> > decode6. ! adder.
> >
> > Now I need to do this 4 times, each time with a
> > potentially different video in each box. I've started
> > looking into C interfaces as there's other pieces of
> > the puzzle which need to be tied into this, and I am
> > trying to determine the best way to tackle this. I
> > originally was looking at Gnonlin, but the
> > documentation is lacking in regards to how
> > gnloperations work. I also recently stumbled upon the
> > GES library by Edward Hervey, this looks promising as
> > well, but I haven't been able to spend much time on
> > it.
> >
> > If I go the Gnonlin route, I believe I would need 6
> > compositions, one for each box. At the 30 second
> > marker, I would swap the filesource to a new one using
> > dynamic pads and listening for messages on the
> > pipeline bus. Am I far off on this? Any suggestions?
> >
> > As for the GES library, it looks very promising and
> > powerful from the little I read on it. Would this be
> > the smarter route to take? If so, does anyone have
> > any suggestions for how the pipeline would be
> > structured?
> >
> > Thank you in advance for your time on this and I
> > truly appreciate any information you are willing to
> > share with me.
> >
> > Happy Thanksgiving,
> > Tim
> >
> >
> >
> ------------------------------------------------------------------------------
> > Increase Visibility of Your 3D Game App & Earn a
> > Chance To Win $500!
> > Tap into the largest installed PC base & get more eyes
> > on your game by
> > optimizing for Intel(R) Graphics Technology. Get
> > started today with the
> > Intel(R) Software Partner Program. Five $500 cash
> > prizes are up for grabs.
> > http://p.sf.net/sfu/intelisp-dev2dev
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.sourceforge.net
> >
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >
> >
> >
> >
> > --
> > www.mediamagictechnologies.com (Gstreamer, ffmpeg, Red5,
> > Streaming)
> > twitter handle: @gst_kaps
> > http://www.linkedin.com/in/kapilagrawal
> >
> >
> ------------------------------------------------------------------------------
> > Increase Visibility of Your 3D Game App & Earn a Chance To Win
> > $500!
> > Tap into the largest installed PC base & get more eyes on your
> > game by
> > optimizing for Intel(R) Graphics Technology. Get started today
> > with the
> > Intel(R) Software Partner Program. Five $500 cash prizes are
> > up for grabs.
> > http://p.sf.net/sfu/intelisp-dev2dev
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >
> >
> >
> ------------------------------------------------------------------------------
> > Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> > Tap into the largest installed PC base & get more eyes on your game by
> > optimizing for Intel(R) Graphics Technology. Get started today with the
> > Intel(R) Software Partner Program. Five $500 cash prizes are up for
> grabs.
> > http://p.sf.net/sfu/intelisp-dev2dev
> > _______________________________________________ gstreamer-devel mailing
> list gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20101128/b413b3a6/attachment.htm>
More information about the gstreamer-devel
mailing list