Zero-copy "decodebin ! glimagesink" equivalent to playbin on Android

[Grégoire Gentil]

Hello again,

Regarding my question, I have been told the following:

"decodebin ! glimagesink doesn't use the zerocopy as you would see if 
you inspected the caps on amcvideodec's src pad (no memory:GLMemory) due 
to https://bugzilla.gnome.org/show_bug.cgi?id=742924

One has to write the equivalent code in playbin in your application that 
uses the autoplug-query signal on (uri)decodebin."



So I have opened a bug: https://bugzilla.gnome.org/show_bug.cgi?id=761738

In the mean time, I'm trying to make it work in an Android application.

As suggested, first I clearly see the pads difference between playbin 
and decodebin:

PLAYBIN PIPELINE
Element: amcvideodec-omxqcomvideodecoderavc0
Src: video/x-raw, format=(string)RGBA, width=(int)1280, 
height=(int)1440, interlace-mode=(string)progressive, 
pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)sRGB, 
framerate=(fraction)105/4, texture-target=(string)external-oes;

Element: glimagesinkbin0
Sink: video/x-raw, format=(string)RGBA, width=(int)1280, 
height=(int)1440, interlace-mode=(string)progressive, 
pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)sRGB, 
framerate=(fraction)105/4, texture-target=(string)external-oes;


DECODEBIN PIPELINE
Element: amcvideodec-omxqcomvideodecoderavc0
Src: video/x-raw, format=(string)NV12, width=(int)1280, 
height=(int)1440, interlace-mode=(string)progressive, 
pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, 
colorimetry=(string)bt709, framerate=(fraction)105/4;

Element: glimagesinkbin0
Sink: video/x-raw, format=(string)NV12, width=(int)1280, 
height=(int)1440, interlace-mode=(string)progressive, 
pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)mpeg2, 
colorimetry=(string)bt709, framerate=(fraction)105/4;


I understand that the key here is "texture-target=(string)external-oes;" 
as well as the context passed between the two plugins.



Problem #1: I'm unsure if only amcvideodec-omxqcomvideodecoderavc0 and 
glimagesink should be taken care of, or also all the intermediate 
invisible gl plugins (glcolorbalance0 | glcolorconvertelement0 | 
gluploadelement0).




So, I have added the following code in my Android application:

data->decodebin = gst_bin_get_by_name(GST_BIN(data->scd[id].pipeline), 
"decodebin0");
data->video_sink = gst_bin_get_by_name(GST_BIN(data->scd[id].pipeline), 
"glimagesinkbin0");
g_signal_connect(G_OBJECT(d), "autoplug-query", 
G_CALLBACK(autoplug_query_cb), data);


the callback is definitely called and I have then tried to do in the 
callback:


static gboolean autoplug_query_cb(GstElement *bin, GstPad *pad, 
GstElement *element, GstQuery *query, CustomData *data) {

///////////////////////////
//Begin debug info
GST_DEBUG("Element:%s Pad:%s Type-Query:%s\n", 
gst_element_get_name(element), gst_pad_get_name(pad), 
gst_query_type_get_name(GST_QUERY_TYPE(query)));

if (GST_QUERY_TYPE(query) == GST_QUERY_ACCEPT_CAPS) {
GstCaps *filt;
gst_query_parse_accept_caps(query, &filt);
GstStructure *structure = gst_caps_get_structure(filt, 0);
GST_DEBUG("accept-caps: %s\n", gst_structure_to_string(structure));
}

if (GST_QUERY_TYPE(query) == GST_QUERY_CAPS) {
GstCaps *filt;
gst_query_parse_caps(query, &filt);
GstStructure *structure = gst_caps_get_structure(filt, 0);
GST_DEBUG("caps: %s\n", gst_structure_to_string(structure));
}
//End debug info
///////////////////////////



if (GST_QUERY_TYPE(query) == GST_QUERY_CONTEXT) {
GstPad *sinkpad = gst_element_get_static_pad(data->scd[0].video_sink, 
"sink");
if (sinkpad) {
gst_pad_query(sinkpad, query);
gst_object_unref(sinkpad);
}
}

if (GST_QUERY_TYPE(query) == GST_QUERY_CAPS) {
GstCaps *filter;
gst_query_parse_caps(query, &filter);
GstPad *sinkpad = gst_element_get_static_pad(data->scd[0].video_sink, 
"sink");
if (sinkpad) {
GstCaps *sinkcaps = gst_pad_query_caps(sinkpad, filter);
gst_caps_unref(sinkcaps);
gst_object_unref(sinkpad);
}
}

return FALSE;
}



Problem #2: I'm not sure what I'm doing here. :-( I believe that I 
should force amcvideodec-omxqcomvideodecoderavc0 in 
"texture-target=(string)external-oes;" and I need to pass the context. 
How should I do that exactly?

Grégoire




-------- Forwarded Message --------
Subject: Zero-copy "decodebin ! glimagesink" equivalent to playbin on 
Android
Date: Fri, 5 Feb 2016 17:56:54 -0800
From: Grégoire Gentil <gregoire at gentil.com>
Reply-To: gregoire at gentil.com
Organization: Gregoire Gentil
To: gstreamer-android at lists.freedesktop.org

Hello,

On Android platform, with latest gstreamer master 1.7.2, it's possible
to have zero copy with a playbin pipeline. I would like to implement a
more complex pipeline on the left side of the decoder.

More precisely, instead of doing:

playbin uri=...

I'm trying:

filesrc location=... ! decodebin ! glimagesink

It works but performance is extremely poor meaning that there is no zero
copy between the hardware acceleration plugin
amcvideodec-omxqcomvideodecoderavc0 and glimagesink.

The two pipelines gives the same kind of elements so I'm thinking that
an option/flag/setting should be passed to one of the elements for
zero-copy. Is that the case or am I missing anything else?

Grégoire