[Bug 743345] New: glimagesink: Add support for dmabuf

Thu Jan 22 04:54:29 PST 2015

https://bugzilla.gnome.org/show_bug.cgi?id=743345
  GStreamer | gst-plugins-bad | unspecified

           Summary: glimagesink: Add support for dmabuf
    Classification: Platform
           Product: GStreamer
           Version: unspecified
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: enhancement
          Priority: Normal
         Component: gst-plugins-bad
        AssignedTo: gstreamer-bugs at lists.freedesktop.org
        ReportedBy: lubosz at gmail.com
         QAContact: gstreamer-bugs at lists.freedesktop.org
     GNOME version: ---

A summary of the work George Kiagiadakis and I have been doing on dmabuf
support in the GL plugins.
Hopefully the remaining problems can be resolved and dmabuf support can land
upstream soon.

The current branch can be found here
http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/log/

I am not uploading the patches to bugzilla yet, since I want to wait for the
discussion.

== Platform Support ==

Since dmabuf is a Linux feature, it won't be available on other platforms. 
Currently my tests were limited to the Intel driver, but I will soon test on
Mali and NVIDIA / nouveau.

dmabuf support in the GL plugins can be achieved conveniently with an EGL
extension.
https://www.khronos.org/registry/egl/extensions/EXT/EGL_EXT_image_dma_buf_import.txt

An EGL context is a requirement for the extension, but
EGL_EXT_image_dma_buf_import is not necessarily an OpenGL ES only feature.
But currently the Intel i965 driver does not implement it for Desktop GL.
http://cgit.freedesktop.org/mesa/mesa/tree/src/mesa/drivers/dri/i965/intel_tex_image.c#n387

To achieve the dma_buf_import on ES, a GL_OES_EGL_image_external texture target
is needed.
https://www.khronos.org/registry/gles/extensions/OES/OES_EGL_image_external.txt

== Patch 1 # glupload: make texture target parameterizable ==

To use a different texture target during upload, I extended the glmemory calls 
http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/commit/?id=c26e149557954a2e9b2203a87b6cba6bac6b184d

The best solution for this problem is being discussed in another issue
https://bugzilla.gnome.org/show_bug.cgi?id=743242

== Patch 2 # glupload: implement dmabuf upload

This patch was originally authored by George Kiagiadakis. I ported it onto the
current glupload.

The dma_buf_upload_accept method checks for gst_is_dmabuf_memory and does not
require dmabuf caps features.
This is needed, since v4l2 elements do not expose this caps feature, but the
allocated memory type.

http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/commit/?id=21678d1e6b8799c4d7514bb00b35f03060f2099b

== Patch 3 # glimagesink: implement dmabuf display

This patch was originally authored by George Kiagiadakis.
The glsink needs to use the different texture target for displaying the dmabuf.
The target needs also to be specified in the shader. To recognize a dmabuf,
gst_is_dmabuf_memory is used.

http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/commit/?id=2b1955714168a6bc46a36549ed7425ea01ba3e1a

== Patch 4 # dmabuf: add drm format support without conversion

Using a dmabuf without conversion requires to know when a drm format is
supported by the driver.
According to the spec this can not be queried, but only recognized by trial and
error:

    6. How can an application query which formats the EGL implementation
    supports?

    PROPOSAL: Don't provide a query mechanism but instead add an error
condition
    that EGL_BAD_MATCH is raised if the EGL implementation doesn't support that
    particular format.

Several fourcc formats are specified in the drm header:
http://cgit.freedesktop.org/mesa/drm/tree/include/drm/drm_fourcc.h

To avoid inclusion of this header I am currently redefining these in a switch
to convert from GStreamer to drm format.
The "known working formats" problem could be solved by testing all drm formats
and checking for EGL errors.
The question is if this should be done every time during negotiation of the
pipeline.

This supported drm format detection is also needed in the glimagesink, since it
is not communicated in another way.
This could be solved however with caps feature and formats (as seen in the
later described dma caps features patch).

I am also omitting gst_video_frame_map for dmabufs, since it seems to be an
unneeded copy.
Additionally gst_video_frame_map needs to have a RGBA formatted source buffer
from glcolorconvert.

http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/commit/?id=c207652604376c8095b2708d745ecb15586889be

== Patch 5 # glupload: add GST_GL_DRM_FOURCC_OVERRIDE option

I used this patch to test supported drm formats and correct conversion.

For example:
GST_GL_DRM_FOURCC_OVERRIDE=XB24

This was the script I was using for the test
https://gist.github.com/lubosz/43ba48cdb74b6dd77d33

http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/commit/?id=76c278bd752454e61e03fc7919b35289e6965021

== Patch 6 # glimagesink: use memory:dmabuf caps features

These caps features are currently only negotiated from the inteldmabufupload
element.
Making the uplaod rely on these features would require the v4l2 elements to use
them.

http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/commit/?id=cc89ca45b172919ccdd9c68e237e6be3abe3ec06

== Patch 7 # WIP: glconvert: enable conversion for dmabuf

This shows how glconvert would needed to be modified to support conversion for
formats drm does not support.
All the shaders would require to be modified. In multiplane cases, more than
the sampler type would needed to be changed.

After a new texture is created during conversion, the sink needs to display a
GL_TEXTURE_2D target.
The question is if this should be done at all, and how this could be solved in
a proper way.

http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/commit/?id=4bd2fad504c42cc9076c427cf0c63ff21c1f36bf

== Multiplane format support ==

I was not able yet to run multiplane support, since the Intel driver does not
support it.

Following EGL error is reported when using I420:
libEGL debug: EGL user error 0x3004 (EGL_BAD_ATTRIBUTE) in too many plane
attributes

I hopefully will be able to test this on the Mali driver.

== inteldmabufupload ==

For testing reasons George Kiagiadakis implemented an element to create dmabufs
in a GStreamer pipeline.
It uses memory:dmabuf caps features and was not accepted upstream

https://bugzilla.gnome.org/show_bug.cgi?id=732281

I rebased it on current master for convenience and testing

http://cgit.collabora.com/git/user/lubosz/gst-plugins-bad.git/commit/?id=0d0836444d80024ffa3c2d2f2388df0a15bc366d

Example pipelines for displaying the dmabuf in glimagesink is 

 $ GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 videotestsrc !
inteldmabufupload ! glimagesink
 $ GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 videotestsrc !
inteldmabufupload ! video/x-raw\(memory:dmabuf\),format=RGBx ! glimagesink

== vivid module ==

Since I do not own dmabuf supported video hardware, I relied on the vivid
kernel module to emulate one.
This module allows generation of a virtual video device with dmabuf
capabilities suitable for v4l2src.

http://git.linuxtv.org/cgit.cgi/media_tree.git/tree/Documentation/video4linux/vivid.txt

The module can be found on the Linux media branch and has not yet been merged
into upstream kernel 3.18.

git://linuxtv.org/media_tree.git

The module can easily be loaded with modprobe and creates a /dev/videoX device.

 $ sudo modprobe vivid n_devs=1 node_types=0x1

An example pipeline for displaying the vivid dmabuf in glimagesink is 

 $ GST_GL_PLATFORM=egl GST_GL_API=gles2 gst-launch-1.0 v4l2src
device=/dev/video1 io-mode=4 ! video/x-raw,format=BGRA ! glimagesink

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.