Sending GLMemory buffers into VAAPI plugins

[Matt Fischer]

Hi, I hope this is the right place for VAAPI questions, I didn't see a
dedicated mailing list for it.  Please redirect me if I missed it. :)

I'm trying to set up a GStreamer configuration where I produce GL buffers,
and then send them off to a VAAPI encoder.  Currently there's no direct way
to do that without a software download in between, so I've been trying to
figure out how to add that support.  I've hacked something together that
seems to work, based largely on the code which is already present for
importing dmabuf buffers.  However there are still several pieces of it
that aren't right, and I'm not exactly sure how to proceed, so I was hoping
somebody here could give me some guidance.

I've attached a copy of my patch as it currently stands.  Essentially I've
just added a new block in gst_vaapi_plugin_base_get_input_buffer() which
can detect GLMemory buffers, and which uses gst_vaapi_texture_new_wrapped()
to get the texture into a GstVaapiSurfaceProxy.  Here are the parts I still
haven't completely worked out:

1. Right now I have to fake the chroma_type so that
is_chroma_type_supported() in gstvaapiencoder.c doesn't reject the
surface.  Is this check actually necessary when creating a VASurface the
way I'm doing?  The colors in the encoded video seem to be right, even
though it's an ARGB texture.

2. I had to punch a hole through GstVaapiTexture to get at the
GstVaapiSurface that is created in the EGL backend.  That doesn't
generalize to the other texture backends...is there a better way for me to
turn a GstVaapiTexture into a GstVaapiSurface that would be more portable?

3. I'm creating a new GstVaapiTexture every time a buffer comes in, which
obviously isn't ideal.  I'd like to cache these things somehow...maybe a
new meta on the incoming GLMemory buffer or something like that?

4. I had to add code to gstvaapipluginbase.c to ensure that we have a GL
context before creating the VAAPI display.  I'm not exactly sure how this
plays with the existing code that tried to get a GL context during
decide_allocation for GLTextureUploadMeta.  Would that still be necessary,
or could it just use the same mechanism that I added, which queries for the
context like other GL plugins do?

I'd appreciate any feedback you can provide.

Thanks,
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20170104/cd57a31b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-TEMP-GL-VAAPI.patch
Type: application/octet-stream
Size: 8631 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20170104/cd57a31b/attachment-0001.obj>