[Mesa-stable] [PATCH v2] st/mesa: fix the layer of VDPAU surface samplers

[Ilia Mirkin]

On Fri, Nov 4, 2016 at 8:28 AM, Nicolai Hähnle <nhaehnle at gmail.com> wrote:
> From: Nicolai Hähnle <nicolai.haehnle at amd.com>
>
> A (latent) bug in VDPAU interop was exposed by commit
> e5cc84dd43be066c1dd418e32f5ad258e31a150a.
>
> Before that commit, the st_vdpau code created samplers with
> first_layer == last_layer == 1 that the general texture handling code
> would immediately delete and re-create, because the layer does not match
> the information in the GL texture object.
>
> This was correct behavior at least in the DMABUF case, because the imported
> resource is supposed to have the correct offset already applied.  In the
> non-DMABUF case, this was just plain wrong but apparently nobody noticed.
>
> After that commit, the state tracker assumes that an existing sampler is
> correct at all times.  Existing samplers are supposed to be deleted when
> they may become invalid, and they will be created on-demand.  This meant
> that the sampler with first_layer == last_layer == 1 stuck around, leading
> to rendering artefacts (on radeonsi), command stream failures (on r600), and
> assertions (in debug builds everywhere).
>
> This patch fixes the problem by simply not creating a sampler at all in
> st_vdpau_map_surface.  We rely on the generic texture code to do the right
> thing, adding the layer_override to make the non-DMABUF case work.
>
> v2: add the layer_override
>
> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=98512
> Cc: 13.0 <mesa-stable at lists.freedesktop.org>
> Cc: Christian König <deathsimple at vodafone.de>
> Cc: Ilia Mirkin <imirkin at alum.mit.edu>
> --
> This fixes the problem on radeon, but it would be good to get some testing
> on Nouveau as well. According to Christian, testing nouveau with Kodi should
> do the trick, probably mpv --hwdec=vdpau --vo=opengl should also work. Errors
> may only be noticeable as bad (de-)interlacing.

VDPAU on nouveau is in a pretty broken state. The actual decoding is
[mostly] fine, of course, just the later layers. The presentation bit
of the API currently tries to render (and blend) to a GART surface
which is slow. The GL interop's only known (to me) users do vpdau and
GL stuff from separate threads, which destroys nouveau.

Put another way, don't worry about nouveau here.

Cheers,

  -ilia