[Mesa-dev] [RFC 0/2] mesa/st: support for YUV EGLImages

[Rob Clark]

On Mon, Sep 5, 2016 at 12:12 PM, Marek Olšák <maraeo at gmail.com> wrote:
> On Fri, Sep 2, 2016 at 9:45 PM, Rob Clark <robdclark at gmail.com> wrote:
>> So, android and blob GLES drivers were left unchecked for too long, and
>> now we are stuck with this annoying OES_EGL_image_external extension and
>> the expectation that the driver can import multi-planar YUV buffers (via,
>> for example, EGL_EXT_image_dma_buf_import), despite the fact that nearly
>> all hardware out there needs this lowered to multiple samplers (one per
>> plane) and colorspace conversion in the shader.  It would be nice to
>> ignore this (mis)feature, except that by now it is required by android.
>>
>> This patchset adds a TGSI lowering pass to handle 2 or 3 planar YUV.
>> And associated logic in mesa/st to append additional sampler view/state
>> to handle the additional planes.  There is no change needed in the
>> individual gallium drivers (provided you support R8 and R8G8).  The
>> extra logic and shader variants only kick in when the shader uses
>> samplerExternalOES.
>>
>> I think I got the _NEW_xyz state handling correct, but that is worth
>> a double check from someone who knows that better.
>>
>> So far, I've only wired this up in fragment shader stage.  I'm not
>> entirely sure who in their right mind would be using a YUV image in
>> (for example) a geometry shader, but I didn't see anything in the
>> OES_EGL_image_external spec that lets us get out of adding variants
>> for all the other shader stages, other than that the spec is written
>> against GLES 1.1 or GLES 2.0, which doesn't have these extra stages.
>> I would be interested in opinions about whether we can weasel our
>> way out of compute/tess/geom support and stick to just VS and FS.
>
> I suggest handling YUV in FS only and ignoring all other stages.

Compute stage might be be reasonable.  I agree the other stages do not
seem too sensible and I doubt they would get much use outside of some
yet-to-be-written piglit tests.

I do have the other stages wired up now (locally).. it isn't all that
difficult to do.  Probably the harder part is writing piglit tests for
them.  I'd be curious what i965 does, I suppose that would seem like
enough of a precedent.  I certainly don't mind dropping the support
for other stages, or maybe stashing off on a branch on github in case
someone eventually comes up with a valid use-case.

But since piglit only tests FS (and has some problems at that, see RFC
patch I sent to piglit list), I guess use in other stages is at least
untested, if supported in i965.

>>
>> I've got some simple test code, which uses gbm to create dmabuf's
>> so it should run on any driver:
>>
>>  https://github.com/robclark/kmscube/commits/yuv-cube
>>
>> You can find the latest versions of the patches here:
>>
>>  https://github.com/freedreno/mesa/commits/wip-yuv
>>
>> Rough list of things TODO:
>>
>>  + Make sure something validates that user is not requiring too
>>    many texture slots (ie. using texObj->RequiredTextureImageUnits)
>>    *before* getting to the point of asserting in atom update path
>>    which appends the extra U/V sampler slots
>>
>>  + NIR lowering.. currently nir_lower_tex does have support for
>>    lowering yuv (used by i965), but it utilizes a new tex_src_plane
>>    arg, instead of just using additional sampler slots.  Maybe just
>>    update nir_lower_tex to support both ways?
>>
>>  + Additional shader stages (at least VS)
>
> Please ignore VS unless there is a known use case.
>
>>
>>  + single plane (interleaved) YUV formats?  And seems like android
>>    wants YV12 (which is I420 with the 2nd and 3rd planes swapped).
>>    In general I don't want to go too crazy and add every YUV format
>>    under the sun, but try and limit it to the must-have formats.
>
> Have you considered using PIPE_FORMAT_R8G8_R8B8_UNORM and the BGR version?

Hmm, no.. (I also don't support it in freedreno, so not entirely sure
what that format is.)

How did you expect that to help?

BR,
-R

> Marek