Optimized surface sharing between gstreamer(-vaapi) and a openGL-application on INTEL ATOM GPUs?

Tue Feb 6 09:44:01 UTC 2018

On Mon, 05 Feb 2018 at 18:23, Sebastian Stelmasik wrote:
> Hi Victor,
> thanks for Your fast answer! Regarding Your questions:
> 
> - we are using vaapisink
> - gstreamer version 1.6.3, but we have the same issues with current
> gstreamer 1.12.4
> 
> Here are our basic functions:

As far as I understand these functions you are basically using the
GLUploadTextureMeta, through the handoff signal in vaapisink. And furthermore
you are using the GstVaapi API, which was dropped when gstreamer-vaapi were
merged as part of the GStreamer project, at 1.8

It is a surprise to me that in AMD, the GLUploadTextureMeta is actually zero
copy, since for Intel it has always implied some memcopy.

So you are basically facing several potential problems:

1\ you are using a old version of gstreamer and gstreamer-vaapi
2\ you are using a deprecated API for handling VA-API
3\ you are using a soon-to-be deprecated buffer meta
4\ you are relying on GLX, which has no future, as far as I understand

What I propose to you is:

1\ upgrade to the latest gstreamer, gstreamer-vaapi release
2\ remove the old GstVaapi code
3\ use EGL

For that you should change your code to use a pipeline like this

playbin video-sink="glupload ! appsink"

and you code will use the appsink API to fetch the decoded video frames with
GLMemory, which it will contain a texture that you can render.

By using EGL vaapi will export dmabuf-based buffers, those buffers will be
imported as EGLImages, wrapped the texture, and thus no memcopy are required.

vmjl

> 
> ----- video_load ---
> 
> /int video_load(lua_State *L, const char *path, const char *name) { Window
> rootwin; XWindowAttributes wattr; GLint *attr; int n_fbconfig_attrs; const
> gchar *platform; memset(&video, 0, sizeof(video_t)); video.x11_display =
> glXGetCurrentDisplay(); screen = DefaultScreen (video.x11_display);
> //video.x11_window = (Window) RootWindow (video.x11_display, screen);
> video.x11_window = (Window) glXGetCurrentDrawable(); XSynchronize
> (video.x11_display, True); platform = "glx"; char dir[256], uri[256] =
> "file:///"; getcwd(dir, sizeof(uri)); strcat(uri, dir); strcat(uri, "/");
> strcat(uri, path); if (!gst_video_init) { src = gst_element_factory_make
> ("playbin", NULL); sink = gst_element_factory_make ("vaapisink", NULL);
> g_object_set (sink, "signal-handoffs", TRUE, NULL); handoff_id =
> g_signal_connect (G_OBJECT (sink), "handoff", G_CALLBACK (on_client_draw),
> &video); //g_object_set (video.sink, "display", 1, NULL); //1 is x11
> g_object_set (G_OBJECT(src), "video-sink", sink, NULL); //bus =
> gst_pipeline_get_bus (GST_PIPELINE (pipeline)); //gst_bus_add_signal_watch
> (video.bus); //gst_bus_enable_sync_message_emission (bus);
> //g_signal_connect (video.bus, "message", G_CALLBACK (sync_bus_call),
> &video); //g_signal_connect (bus, "sync-message", G_CALLBACK
> (sync_bus_call), &video); //gst_object_unref (bus); GstPad *sinkpad =
> gst_element_get_static_pad (sink, "sink"); gst_pad_add_probe (sinkpad,
> GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM, (GstPadProbeCallback)
> playout_item_pad_probe_event, &video, NULL); gst_object_unref (sinkpad); }
> g_object_set (src, "uri", uri, NULL); video.eos = FALSE; if
> (!video_open(&video, path)) return luaL_error(L, "cannot open video %s",
> path); glGenTextures(1, &video.tex); glBindTexture(GL_TEXTURE_2D,
> video.tex); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER,
> GL_LINEAR_MIPMAP_LINEAR); glTexParameteri(GL_TEXTURE_2D,
> GL_TEXTURE_MAG_FILTER, GL_LINEAR_MIPMAP_LINEAR);
> glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
> glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP); glTexImage2D(
> GL_TEXTURE_2D, 0, GL_BGRA, video.width, video.height, 0, GL_BGRA,
> GL_UNSIGNED_BYTE, NULL ); /* Initialize FBConfig attributes */ static GLint
> fbconfig_attrs[] = { //values from MPV GLX_BIND_TO_TEXTURE_RGBA_EXT, True,
> GLX_DRAWABLE_TYPE, GLX_PIXMAP_BIT, GLX_BIND_TO_TEXTURE_TARGETS_EXT,
> GLX_TEXTURE_2D_BIT_EXT, GLX_Y_INVERTED_EXT, True, GLX_DOUBLEBUFFER, False,
> GLX_RED_SIZE, 8, GLX_GREEN_SIZE, 8, GLX_BLUE_SIZE, 8, GLX_ALPHA_SIZE, 0,
> None }; video.fb_config = glXChooseFBConfig (video.x11_display, screen,
> fbconfig_attrs, &n_fbconfig_attrs); if (!video.fb_config) { fprintf (stderr,
> "FBCONFIG failed\n"); exit(0); } video.pixmap = XCreatePixmap
> (video.x11_display, video.x11_window, video.width, video.height,
> 32);//wattr.depth); int pixmap_attrs[10] = { //Values from MPV
> GLX_TEXTURE_TARGET_EXT, GLX_TEXTURE_2D_EXT, GLX_TEXTURE_FORMAT_EXT,
> GLX_TEXTURE_FORMAT_RGB_EXT, GLX_MIPMAP_TEXTURE_EXT, False, None, };
> video.glx_pixmap = glXCreatePixmap (video.x11_display, video.fb_config[0],
> video.pixmap, pixmap_attrs ); video.surface_present = 0; glx_bind_tex_image
> = (PFNGLXBINDTEXIMAGEEXTPROC) glXGetProcAddress ("glXBindTexImageEXT");
> glx_release_tex_image = (PFNGLXRELEASETEXIMAGEEXTPROC) glXGetProcAddress
> ("glXReleaseTexImageEXT"); g_return_val_if_fail (GST_IS_VIDEO_OVERLAY
> (sink), NULL); video_overlay = GST_VIDEO_OVERLAY (sink);
> gst_video_overlay_set_window_handle (video_overlay, (guintptr)
> video.x11_window); if (gst_video_init) { //gint64 position;
> //gst_element_query_position (src, GST_FORMAT_TIME, &position); handoff_id =
> g_signal_connect (G_OBJECT (sink), "handoff", G_CALLBACK (on_client_draw),
> &video); gst_element_set_state (src, GST_STATE_PLAYING); /*if
> (!gst_element_seek_simple (src, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH,
> position)) { fprintf (stderr,"Seek failed set pos 0 manually\n");
> gst_element_seek_simple (src, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH,0); }*/ }
> else { //g_return_val_if_fail (GST_IS_VIDEO_OVERLAY (sink), NULL);
> //video_overlay = GST_VIDEO_OVERLAY (sink);
> //gst_video_overlay_set_window_handle (video_overlay, (guintptr)
> video.x11_window); gst_element_set_state (GST_ELEMENT (src),
> GST_STATE_PLAYING); gst_video_init = 1; } /*g_return_val_if_fail
> (GST_IS_VIDEO_OVERLAY (video.sink), NULL); video.video_overlay =
> GST_VIDEO_OVERLAY (video.sink); gst_video_overlay_set_window_handle
> (video.video_overlay, (guintptr) video.x11_window); gst_element_set_state
> (GST_ELEMENT (video.pipeline), GST_STATE_PLAYING);*/ /* we save the current
> video in a global variable */ check_video = push_video(L); *check_video =
> video; return 1; }/
> 
> ----- on client draw ---
> 
> /static gboolean on_client_draw (GstElement * glsink, GstBuffer *buf,
> gpointer data) { video_t *video = (video_t*) data; GstVideoInfo v_info;
> GstCaps *caps; GstVaapiSurface *surface; GstVaapiSurfaceProxy *proxy;
> GstVaapiDisplay *vaapi_display; VAStatus status; GstVaapiVideoMeta *meta =
> NULL; if (video->eos) return FALSE; meta = gst_buffer_get_vaapi_video_meta
> (buf); if (!meta) fprintf(stderr, "could not find vaapi meta in buffer!\n");
> if (video->buf) { video->old_buf = video->buf; } video->buf = buf;
> gst_buffer_ref (video->buf); proxy = gst_vaapi_video_meta_get_surface_proxy
> (meta); surface = gst_vaapi_surface_proxy_get_surface (proxy); vaapi_display
> = gst_vaapi_video_meta_get_display(meta); /* fprintf(stderr, "%s video %p
> buffer %p video tex: %u width %d height %d screen %d %d, display: %p %p \n",
> __func__, video, video->buf, video->tex, video->width, video->height,
> screen, gst_vaapi_display_x11_get_screen (vaapi_display),
> video->x11_display, gst_vaapi_display_x11_get_display (vaapi_display) ); */
> status = vaPutSurface ( GST_VAAPI_DISPLAY_VADISPLAY (vaapi_display),
> GST_VAAPI_OBJECT_ID (surface), video->pixmap, 0, 0, video->width,
> video->height, 0, 0, video->width, video->height, NULL, 0, VA_FRAME_PICTURE
> | VA_SRC_BT709); //VA_SRC_BT709 if (!video_vaapi_check_status (status,
> "vaPutSurface()")) { fprintf(stderr, "could not put VAAPI surface!\n");
> return FALSE; } if (!gst_vaapi_surface_sync (surface)) { fprintf(stderr,
> "could not sync VAAPI surface!\n"); } video->surface_present = 1; if
> (check_video) { check_video->buf = video->buf; check_video->surface_present
> = video->surface_present; } if (video->old_buf) { gst_buffer_unref
> (video->old_buf); } return TRUE; }/
> 
> Any optimization welcome!
> 
> Thanks in advance!
> 
> Sebastian
> 
> Am 05.02.2018 um 16:09 schrieb Víctor Jáquez:
> > Hi,
> > 
> > On Mon, 05 Feb 2018 at 13:39, Sebastian Stelmasik wrote:
> > > Hi there!
> > > 
> > > We are developing a Digital Signage application on basis of
> > > gstreamer(-vaapi) and info-beamer (https://info-beamer.com/opensource).
> > > Until now we were using (cheap) AMD devices where we use glx-pixmaps for
> > > surface sharing between gstreamer and info-beamer which works quite perfect
> > > (no mem copy at all).
> > > 
> > > Now we have to switch to INTEL devices as there are no cheap (and compatible
> > > with the former open souce driver) AMD devices and now we are expiriencing
> > > framedrop issues in surface sharing between gstreamer(-vaapi) and
> > > info-beamer on INTEL ATOM Z8300 devices (which should be a lot faster than
> > > the AMD E-350 we have been using before). As I said before we use
> > > glx-pixmaps for surface sharing which works perfect on AMD-GPUs. But now
> > > there seems to be a lot of in mem copy which couses a lot of framedrops.
> > > 
> > > Is there a optimized way/pipeline for INTEL ATOM GPUs?
> > Which video sink are you using in the application?
> > 
> > Which version of gstreamer?
> > 
> > I guess, in your AMD box, you are in a GLX environment and the sink negotiates
> > GLUploadTextureMeta, which isn't absolutely mem copy free, and it is going to be
> > deprecated soon.
> > 
> > A possibility could be to switch to EGL and the sink could import dmabuf, using,
> > perhaps, glupload, and no mem copy is involved.
> > 
> > The other option, would be, of course, embedded vaapisink, negotiating the
> > memory:VASurface feature.
> > 
> > vmjl
> > 
> > 
> > > Or any other issue we didn't think about while switching from AMD to INTEL
> > > GPUs?
> > > 
> > > Best regards and thanks in advance!
> > > 
> > > Sebastian Stelmasik
> > > _______________________________________________
> > > gstreamer-devel mailing list
> > > gstreamer-devel at lists.freedesktop.org
> > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel