[Mesa-dev] [RFC 0/2] mesa/st: support for YUV EGLImages

[Rob Clark]

So, android and blob GLES drivers were left unchecked for too long, and
now we are stuck with this annoying OES_EGL_image_external extension and
the expectation that the driver can import multi-planar YUV buffers (via,
for example, EGL_EXT_image_dma_buf_import), despite the fact that nearly
all hardware out there needs this lowered to multiple samplers (one per
plane) and colorspace conversion in the shader.  It would be nice to
ignore this (mis)feature, except that by now it is required by android.

This patchset adds a TGSI lowering pass to handle 2 or 3 planar YUV.
And associated logic in mesa/st to append additional sampler view/state
to handle the additional planes.  There is no change needed in the
individual gallium drivers (provided you support R8 and R8G8).  The
extra logic and shader variants only kick in when the shader uses
samplerExternalOES.

I think I got the _NEW_xyz state handling correct, but that is worth
a double check from someone who knows that better.

So far, I've only wired this up in fragment shader stage.  I'm not
entirely sure who in their right mind would be using a YUV image in
(for example) a geometry shader, but I didn't see anything in the
OES_EGL_image_external spec that lets us get out of adding variants
for all the other shader stages, other than that the spec is written
against GLES 1.1 or GLES 2.0, which doesn't have these extra stages.
I would be interested in opinions about whether we can weasel our
way out of compute/tess/geom support and stick to just VS and FS.

I've got some simple test code, which uses gbm to create dmabuf's
so it should run on any driver:

 https://github.com/robclark/kmscube/commits/yuv-cube

You can find the latest versions of the patches here:

 https://github.com/freedreno/mesa/commits/wip-yuv

Rough list of things TODO:

 + Make sure something validates that user is not requiring too
   many texture slots (ie. using texObj->RequiredTextureImageUnits)
   *before* getting to the point of asserting in atom update path
   which appends the extra U/V sampler slots

 + NIR lowering.. currently nir_lower_tex does have support for
   lowering yuv (used by i965), but it utilizes a new tex_src_plane
   arg, instead of just using additional sampler slots.  Maybe just
   update nir_lower_tex to support both ways?

 + Additional shader stages (at least VS)

 + single plane (interleaved) YUV formats?  And seems like android
   wants YV12 (which is I420 with the 2nd and 3rd planes swapped).
   In general I don't want to go too crazy and add every YUV format
   under the sun, but try and limit it to the must-have formats.

 + Few spots are a bit fugly, so probably some room for a bit of
   cleanup.  And maybe I could split up the patchset better.

Anyways, figured I should send this out in it's current state,
before I spend too much longer on it, to get some feedback.


Rob Clark (2):
  mesa/st: add lowering pass for YUV samplers
  mesa/st: support lowering multi-planar YUV

 src/gallium/auxiliary/util/u_inlines.h      |   4 +-
 src/gallium/include/pipe/p_state.h          |   9 +
 src/gallium/include/state_tracker/st_api.h  |   3 +
 src/gallium/state_trackers/dri/dri2.c       | 111 +++++--
 src/gallium/state_trackers/dri/dri_screen.c |  11 +
 src/mesa/Makefile.sources                   |   2 +
 src/mesa/main/mtypes.h                      |  16 +
 src/mesa/program/ir_to_mesa.cpp             |   1 +
 src/mesa/state_tracker/st_atom_sampler.c    |  52 +++-
 src/mesa/state_tracker/st_atom_shader.c     |  27 ++
 src/mesa/state_tracker/st_atom_texture.c    |  71 ++++-
 src/mesa/state_tracker/st_cb_eglimage.c     |  18 ++
 src/mesa/state_tracker/st_context.c         |  10 +-
 src/mesa/state_tracker/st_glsl_to_nir.cpp   |   1 +
 src/mesa/state_tracker/st_glsl_to_tgsi.cpp  |   4 +
 src/mesa/state_tracker/st_manager.c         |   1 +
 src/mesa/state_tracker/st_program.c         |  18 ++
 src/mesa/state_tracker/st_program.h         |   3 +
 src/mesa/state_tracker/st_tgsi_lower_yuv.c  | 441 ++++++++++++++++++++++++++++
 src/mesa/state_tracker/st_tgsi_lower_yuv.h  |  34 +++
 20 files changed, 807 insertions(+), 30 deletions(-)
 create mode 100644 src/mesa/state_tracker/st_tgsi_lower_yuv.c
 create mode 100644 src/mesa/state_tracker/st_tgsi_lower_yuv.h

-- 
2.7.4