[Mesa-dev] [RFC 0/2] mesa/st: support for YUV EGLImages

[Marek Olšák]

On Fri, Sep 2, 2016 at 9:45 PM, Rob Clark <robdclark at gmail.com> wrote:
> So, android and blob GLES drivers were left unchecked for too long, and
> now we are stuck with this annoying OES_EGL_image_external extension and
> the expectation that the driver can import multi-planar YUV buffers (via,
> for example, EGL_EXT_image_dma_buf_import), despite the fact that nearly
> all hardware out there needs this lowered to multiple samplers (one per
> plane) and colorspace conversion in the shader.  It would be nice to
> ignore this (mis)feature, except that by now it is required by android.
>
> This patchset adds a TGSI lowering pass to handle 2 or 3 planar YUV.
> And associated logic in mesa/st to append additional sampler view/state
> to handle the additional planes.  There is no change needed in the
> individual gallium drivers (provided you support R8 and R8G8).  The
> extra logic and shader variants only kick in when the shader uses
> samplerExternalOES.
>
> I think I got the _NEW_xyz state handling correct, but that is worth
> a double check from someone who knows that better.
>
> So far, I've only wired this up in fragment shader stage.  I'm not
> entirely sure who in their right mind would be using a YUV image in
> (for example) a geometry shader, but I didn't see anything in the
> OES_EGL_image_external spec that lets us get out of adding variants
> for all the other shader stages, other than that the spec is written
> against GLES 1.1 or GLES 2.0, which doesn't have these extra stages.
> I would be interested in opinions about whether we can weasel our
> way out of compute/tess/geom support and stick to just VS and FS.

I suggest handling YUV in FS only and ignoring all other stages.

>
> I've got some simple test code, which uses gbm to create dmabuf's
> so it should run on any driver:
>
>  https://github.com/robclark/kmscube/commits/yuv-cube
>
> You can find the latest versions of the patches here:
>
>  https://github.com/freedreno/mesa/commits/wip-yuv
>
> Rough list of things TODO:
>
>  + Make sure something validates that user is not requiring too
>    many texture slots (ie. using texObj->RequiredTextureImageUnits)
>    *before* getting to the point of asserting in atom update path
>    which appends the extra U/V sampler slots
>
>  + NIR lowering.. currently nir_lower_tex does have support for
>    lowering yuv (used by i965), but it utilizes a new tex_src_plane
>    arg, instead of just using additional sampler slots.  Maybe just
>    update nir_lower_tex to support both ways?
>
>  + Additional shader stages (at least VS)

Please ignore VS unless there is a known use case.

>
>  + single plane (interleaved) YUV formats?  And seems like android
>    wants YV12 (which is I420 with the 2nd and 3rd planes swapped).
>    In general I don't want to go too crazy and add every YUV format
>    under the sun, but try and limit it to the must-have formats.

Have you considered using PIPE_FORMAT_R8G8_R8B8_UNORM and the BGR version?

Marek