[Mesa-dev] nouveau hardware decoding and vaDeriveImage

[Ilia Mirkin]

[adding Julien, who did the nouveau va-api integration.]

On Tue, Aug 15, 2017 at 9:59 AM, Philipp Kerling <pkerling at casix.org> wrote:
> Hi,
>
> I recently noticed that hardware video decoding with the nouveau driver
> and libva does not play nicely with Wayland and decided to dig a bit.
>
> To use libva in conjunction with Wayland in EGL applications, you
> usually export the surface dmabuf via vaDeriveImage &
> vaAcquireBufferHandle and then import it in EGL via eglCreateImageKHR &
> EGL_LINUX_DMA_BUF_EXT.
> However, this does not work with nouveau.
>
> The first problem I could identify is that nouveau uses NV12 as buffer
> format and mesa vaDeriveImage cannot handle that (see FourCCs handled
> in vlVaDeriveImage). It seems to be easy to add though (just have to
> set num_planes/offset etc. correctly), and I did so locally. Also,
> nouveau always sets the interlaced flag on the buffer, which
> vlVaDeriveImage also errors out on. Not sure why it does that and what
> to do with that currently, I just removed the check locally.

The hw decoder produces an interlaced NV12 image -- i.e. the Y plane
is 2 images and the UV plane is 2 images, below one another. This is
in firmware written in falcon (VP3+) or xtensa (VP2) microcode by
NVIDIA. There is no open-source implementation, and thus would be
tricky to change. It's possible that this firmware supports other
output formats as well ... but we're not presently aware of it.
Presumably the 10-bit decoding is done *somehow*.

>
> Then I hit another problem, which is that NV12 uses two planes and thus
> has an offset into the buffer for the second plane, but nouveau cannot
> handle offsets in eglCreateImageKHR (see nouveau_screen_bo_from_handle
> which has a check for this).

The VP2-era hardware decoder (G84..G200, minus G98) wants a single
output buffer - you enter it in as a single base address and then an
offset to the UV plane. It's a 32-bit offset but a 40-bit VA space.
Later decoder generations (VP3+) have a separate address for both Y
and UV planes. Actually looking over the VP2 logic (which I'm sad to
say I wrote many years ago), it *might* be possible to have separate
buffers for the final output image (in the second VP step).

> Is there an easy/obvious way to add handling for the offset parameter
> in nouveau_screen.c? This might be all that is currently breaking hwdec
> on nouveau+Wayland, but I couldn't test it of course, so there might
> still be other problems lurking.

Well, strictly speaking this should be easy - while nouveau_bo has no
offset, nv04_resource does - specifically things tend to use
nv04_resource->address. I believe this offset stuff was added
recently, and I don't know much about EGL or Wayland or ... well a lot
of the new hotness. So e.g. in nv50_miptree_from_handle you could set
the mt->base.address = bo->offset + whandle->offset; I wonder if the
tiling stuff will be an issue -- the decoder can get *very* picking
about tiling -- somehow you'll have to ensure that the images are
allocated with proper tiling flags set -- see how nvc0_miptree_create
calls nvc0_miptree_init_layout_video, which sets tile_mode to 0x10.
That's not by coincidence.

BTW, note that you can't use a decoder in one thread and GL commands
in another thread. This will cause death in nouveau. Also note that
there are known (but unfixed) artifacts when decoding some H.264
videos.

Feel free to join us in #nouveau on irc.freenode.net.

Cheers,

  -ilia