[Mesa-dev] nouveau hardware decoding and vaDeriveImage

[Philipp Kerling]

Hi,

I recently noticed that hardware video decoding with the nouveau driver
and libva does not play nicely with Wayland and decided to dig a bit.

To use libva in conjunction with Wayland in EGL applications, you
usually export the surface dmabuf via vaDeriveImage &
vaAcquireBufferHandle and then import it in EGL via eglCreateImageKHR &
EGL_LINUX_DMA_BUF_EXT.
However, this does not work with nouveau.

The first problem I could identify is that nouveau uses NV12 as buffer
format and mesa vaDeriveImage cannot handle that (see FourCCs handled
in vlVaDeriveImage). It seems to be easy to add though (just have to
set num_planes/offset etc. correctly), and I did so locally. Also,
nouveau always sets the interlaced flag on the buffer, which
vlVaDeriveImage also errors out on. Not sure why it does that and what
to do with that currently, I just removed the check locally.

Then I hit another problem, which is that NV12 uses two planes and thus
has an offset into the buffer for the second plane, but nouveau cannot
handle offsets in eglCreateImageKHR (see nouveau_screen_bo_from_handle
which has a check for this).
Is there an easy/obvious way to add handling for the offset parameter
in nouveau_screen.c? This might be all that is currently breaking hwdec
on nouveau+Wayland, but I couldn't test it of course, so there might
still be other problems lurking.

Best regards,
Philipp Kerling