[Mesa-dev] [RFC 0/2] mesa/st: support for YUV EGLImages

[Marek Olšák]

On Mon, Sep 5, 2016 at 6:32 PM, Rob Clark <robdclark at gmail.com> wrote:
> On Mon, Sep 5, 2016 at 12:12 PM, Marek Olšák <maraeo at gmail.com> wrote:
>> On Fri, Sep 2, 2016 at 9:45 PM, Rob Clark <robdclark at gmail.com> wrote:
>>> So, android and blob GLES drivers were left unchecked for too long, and
>>> now we are stuck with this annoying OES_EGL_image_external extension and
>>> the expectation that the driver can import multi-planar YUV buffers (via,
>>> for example, EGL_EXT_image_dma_buf_import), despite the fact that nearly
>>> all hardware out there needs this lowered to multiple samplers (one per
>>> plane) and colorspace conversion in the shader.  It would be nice to
>>> ignore this (mis)feature, except that by now it is required by android.
>>>
>>> This patchset adds a TGSI lowering pass to handle 2 or 3 planar YUV.
>>> And associated logic in mesa/st to append additional sampler view/state
>>> to handle the additional planes.  There is no change needed in the
>>> individual gallium drivers (provided you support R8 and R8G8).  The
>>> extra logic and shader variants only kick in when the shader uses
>>> samplerExternalOES.
>>>
>>> I think I got the _NEW_xyz state handling correct, but that is worth
>>> a double check from someone who knows that better.
>>>
>>> So far, I've only wired this up in fragment shader stage.  I'm not
>>> entirely sure who in their right mind would be using a YUV image in
>>> (for example) a geometry shader, but I didn't see anything in the
>>> OES_EGL_image_external spec that lets us get out of adding variants
>>> for all the other shader stages, other than that the spec is written
>>> against GLES 1.1 or GLES 2.0, which doesn't have these extra stages.
>>> I would be interested in opinions about whether we can weasel our
>>> way out of compute/tess/geom support and stick to just VS and FS.
>>
>> I suggest handling YUV in FS only and ignoring all other stages.
>
> Compute stage might be be reasonable.  I agree the other stages do not
> seem too sensible and I doubt they would get much use outside of some
> yet-to-be-written piglit tests.
>
> I do have the other stages wired up now (locally).. it isn't all that
> difficult to do.  Probably the harder part is writing piglit tests for
> them.  I'd be curious what i965 does, I suppose that would seem like
> enough of a precedent.  I certainly don't mind dropping the support
> for other stages, or maybe stashing off on a branch on github in case
> someone eventually comes up with a valid use-case.
>
> But since piglit only tests FS (and has some problems at that, see RFC
> patch I sent to piglit list), I guess use in other stages is at least
> untested, if supported in i965.
>
>>>
>>> I've got some simple test code, which uses gbm to create dmabuf's
>>> so it should run on any driver:
>>>
>>>  https://github.com/robclark/kmscube/commits/yuv-cube
>>>
>>> You can find the latest versions of the patches here:
>>>
>>>  https://github.com/freedreno/mesa/commits/wip-yuv
>>>
>>> Rough list of things TODO:
>>>
>>>  + Make sure something validates that user is not requiring too
>>>    many texture slots (ie. using texObj->RequiredTextureImageUnits)
>>>    *before* getting to the point of asserting in atom update path
>>>    which appends the extra U/V sampler slots
>>>
>>>  + NIR lowering.. currently nir_lower_tex does have support for
>>>    lowering yuv (used by i965), but it utilizes a new tex_src_plane
>>>    arg, instead of just using additional sampler slots.  Maybe just
>>>    update nir_lower_tex to support both ways?
>>>
>>>  + Additional shader stages (at least VS)
>>
>> Please ignore VS unless there is a known use case.
>>
>>>
>>>  + single plane (interleaved) YUV formats?  And seems like android
>>>    wants YV12 (which is I420 with the 2nd and 3rd planes swapped).
>>>    In general I don't want to go too crazy and add every YUV format
>>>    under the sun, but try and limit it to the must-have formats.
>>
>> Have you considered using PIPE_FORMAT_R8G8_R8B8_UNORM and the BGR version?
>
> Hmm, no.. (I also don't support it in freedreno, so not entirely sure
> what that format is.)
>
> How did you expect that to help?

It would help if the texture were not layered.

Marek