[BUG][AMD][VDPAU] RX6600 GPU locks up when decoding HEVC

[Chris Rankin]

Thanks, I have discovered that my PC needs for VLC to use hardware
decoding when playing 4K HEVC streams, but that VLC won't support
VA-API again until VLC 4 is released.

As an aside, one of these 4K HEVC streams correctly uses VDPAU if I use:

$ mpv --vo=gpu --hwdec=vdpau --hwdec-image-format=yuv420p

but uses software decoding if I remove the
'--hwdec-image-format=yuv420p' parameter because:

[vd] Opening decoder hevc
[vd] Looking at hwdec hevc-vdpau...
[vd] Trying hardware decoding via hevc-vdpau.
[vd] Selected codec: hevc (HEVC (High Efficiency Video Coding))
[vd] Pixel formats supported by decoder: vaapi vdpau cuda yuv420p10le
[vd] Codec profile: Main 10 (0x2)
[ffmpeg] AVHWFramesContext: Unsupported sw format: yuv420p10le
Failed to allocate hw frames.
[vd] Requesting pixfmt 'yuv420p10le' from decoder.
[vd] Attempting next decoding method after failure of hevc-vdpau.
[vd] Skipping previously attempted hwdec: hevc-vdpau
[vd] Using software decoding.

Does this look like another Mesa / VDPAU bug (i.e. should I raise a
new issue for it?), or is it actually a problem with mpv?

Cheers,
Chris

On Thu, 15 Feb 2024 at 16:01, Liu, Leo <Leo.Liu at amd.com> wrote:
>
> [AMD Official Use Only - General]
>
> This is being looked.
>
> > -----Original Message-----
> > From: mesa-dev <mesa-dev-bounces at lists.freedesktop.org> On Behalf Of
> > Chris Rankin
> > Sent: Thursday, February 15, 2024 10:38 AM
> > To: mesa-dev at lists.freedesktop.org
> > Subject: [BUG][AMD][VDPAU] RX6600 GPU locks up when decoding HEVC
> >
> > I have discovered that my RX6600 cannot use VDPAU to decode HEVC, although
> > VA-API works OK. Unfortunately, VLC 3.x only supports VDPAU these days for
> > hardware decoding.
> >
> > I have raised this issue:
> > https://gitlab.freedesktop.org/mesa/mesa/-/issues/10599
> >
> > Mesa 23.3.5 just used to fail, but both Mesa 24.0.1 and 24.1.0-devel actually
> > lock up the GPU and force me to reboot my PC.
> >
> > According to vdpauinfo:
> >
> > name                        level macbs width height
> > ----------------------------------------------------
> > ...
> > HEVC_MAIN                      186 139264  8192  4352
> > HEVC_MAIN_10                   186 139264  8192  4352
> > HEVC_MAIN_STILL                --- not supported ---
> > HEVC_MAIN_12                   --- not supported ---
> > HEVC_MAIN_444                  --- not supported ---
> > HEVC_MAIN_444_10               --- not supported ---
> > HEVC_MAIN_444_12               --- not supported ---
> > ...
> >
> > Cheers,
> > Chris