Zero-copy "decodebin ! glimagesink" equivalent to playbin on Android

[Matthew Waters]

On 09/02/16 11:05, Grégoire Gentil wrote:
> Hello again,
>
> Regarding my question, I have been told the following:
>
> "decodebin ! glimagesink doesn't use the zerocopy as you would see if
> you inspected the caps on amcvideodec's src pad (no memory:GLMemory)
> due to https://bugzilla.gnome.org/show_bug.cgi?id=742924
>
> One has to write the equivalent code in playbin in your application
> that uses the autoplug-query signal on (uri)decodebin."
>
>
>
> So I have opened a bug: https://bugzilla.gnome.org/show_bug.cgi?id=761738
>
> In the mean time, I'm trying to make it work in an Android application.
>
> As suggested, first I clearly see the pads difference between playbin
> and decodebin:
>
> PLAYBIN PIPELINE
> Element: amcvideodec-omxqcomvideodecoderavc0
> Src: video/x-raw, format=(string)RGBA, width=(int)1280,
> height=(int)1440, interlace-mode=(string)progressive,
> pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)sRGB,
> framerate=(fraction)105/4, texture-target=(string)external-oes;
>
> Element: glimagesinkbin0
> Sink: video/x-raw, format=(string)RGBA, width=(int)1280,
> height=(int)1440, interlace-mode=(string)progressive,
> pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)sRGB,
> framerate=(fraction)105/4, texture-target=(string)external-oes;
>
>
> DECODEBIN PIPELINE
> Element: amcvideodec-omxqcomvideodecoderavc0
> Src: video/x-raw, format=(string)NV12, width=(int)1280,
> height=(int)1440, interlace-mode=(string)progressive,
> pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2,
> colorimetry=(string)bt709, framerate=(fraction)105/4;
>
> Element: glimagesinkbin0
> Sink: video/x-raw, format=(string)NV12, width=(int)1280,
> height=(int)1440, interlace-mode=(string)progressive,
> pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2,
> colorimetry=(string)bt709, framerate=(fraction)105/4;
>
>
> I understand that the key here is
> "texture-target=(string)external-oes;" as well as the context passed
> between the two plugins.
>
>
>
> Problem #1: I'm unsure if only amcvideodec-omxqcomvideodecoderavc0 and
> glimagesink should be taken care of, or also all the intermediate
> invisible gl plugins (glcolorbalance0 | glcolorconvertelement0 |
> gluploadelement0).

The only required thing to do is too make sure that amcvideodec is
talking to glimagesink correctly.  You don't have to worry about what's
contained within the bin.  That's one of the purposes of bins ;)

> So, I have added the following code in my Android application:
>
> data->decodebin = gst_bin_get_by_name(GST_BIN(data->scd[id].pipeline),
> "decodebin0");
> data->video_sink =
> gst_bin_get_by_name(GST_BIN(data->scd[id].pipeline), "glimagesinkbin0");
> g_signal_connect(G_OBJECT(d), "autoplug-query",
> G_CALLBACK(autoplug_query_cb), data);
>
>
> the callback is definitely called and I have then tried to do in the
> callback:
>
>
> static gboolean autoplug_query_cb(GstElement *bin, GstPad *pad,
> GstElement *element, GstQuery *query, CustomData *data) {
>
> ///////////////////////////
> //Begin debug info
> GST_DEBUG("Element:%s Pad:%s Type-Query:%s\n",
> gst_element_get_name(element), gst_pad_get_name(pad),
> gst_query_type_get_name(GST_QUERY_TYPE(query)));
>
> if (GST_QUERY_TYPE(query) == GST_QUERY_ACCEPT_CAPS) {
> GstCaps *filt;
> gst_query_parse_accept_caps(query, &filt);
> GstStructure *structure = gst_caps_get_structure(filt, 0);
> GST_DEBUG("accept-caps: %s\n", gst_structure_to_string(structure));
> }

Should use gst_caps_to_string() to get the caps features printed as well
(which are also different here).

> if (GST_QUERY_TYPE(query) == GST_QUERY_CAPS) {
> GstCaps *filt;
> gst_query_parse_caps(query, &filt);
> GstStructure *structure = gst_caps_get_structure(filt, 0);
> GST_DEBUG("caps: %s\n", gst_structure_to_string(structure));
> }
> //End debug info
> ///////////////////////////
>
>
>
> if (GST_QUERY_TYPE(query) == GST_QUERY_CONTEXT) {
> GstPad *sinkpad = gst_element_get_static_pad(data->scd[0].video_sink,
> "sink");
> if (sinkpad) {
> gst_pad_query(sinkpad, query);
> gst_object_unref(sinkpad);
> }
> }
>
> if (GST_QUERY_TYPE(query) == GST_QUERY_CAPS) {
> GstCaps *filter;
> gst_query_parse_caps(query, &filter);
> GstPad *sinkpad = gst_element_get_static_pad(data->scd[0].video_sink,
> "sink");
> if (sinkpad) {
> GstCaps *sinkcaps = gst_pad_query_caps(sinkpad, filter);
> gst_caps_unref(sinkcaps);

Uhm, you're querying downstream then not passing that on?  amcvideodec
doesn't receive any caps then and goes into the sysmem path.

Another thing to watch, autoplug-query is called for every query from
every element at the end of every the decode chain inside decodebin. 
e.g. you'll get queries from the demuxer and the audio chain as well so
you need to choose the sink appropriately.

> gst_object_unref(sinkpad);
> }
> }
>
> return FALSE;
> }
>
>
>
> Problem #2: I'm not sure what I'm doing here. :-( I believe that I
> should force amcvideodec-omxqcomvideodecoderavc0 in
> "texture-target=(string)external-oes;" and I need to pass the context.
> How should I do that exactly?

If everything is working,  then it will be picked automatically.  The
fact that it isn't, indicates a problem with your code.  Look at
playbin's implementation as well as the debug logs to determine what
exactly is happening (It's all there somewhere ;)).

Cheers
-Matt

> Grégoire
>
>
>
>
> -------- Forwarded Message --------
> Subject: Zero-copy "decodebin ! glimagesink" equivalent to playbin on
> Android
> Date: Fri, 5 Feb 2016 17:56:54 -0800
> From: Grégoire Gentil <gregoire at gentil.com>
> Reply-To: gregoire at gentil.com
> Organization: Gregoire Gentil
> To: gstreamer-android at lists.freedesktop.org
>
> Hello,
>
> On Android platform, with latest gstreamer master 1.7.2, it's possible
> to have zero copy with a playbin pipeline. I would like to implement a
> more complex pipeline on the left side of the decoder.
>
> More precisely, instead of doing:
>
> playbin uri=...
>
> I'm trying:
>
> filesrc location=... ! decodebin ! glimagesink
>
> It works but performance is extremely poor meaning that there is no zero
> copy between the hardware acceleration plugin
> amcvideodec-omxqcomvideodecoderavc0 and glimagesink.
>
> The two pipelines gives the same kind of elements so I'm thinking that
> an option/flag/setting should be passed to one of the elements for
> zero-copy. Is that the case or am I missing anything else?
>
> Grégoire
>
>
>
>
>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/gstreamer-android/attachments/20160209/fb475ea7/attachment-0001.sig>