NVDEC and 10-bit HEVC decode

[Samuel Hurst]

Hi all,

I find myself experimenting with hardware accelerated HEVC decoding 
under GStreamer, and I'm trying to use the nvdec plugin from plugins-bad.

When playing 8-bit HEVC content, playback is fine. However, when I try 
playing back 10-bit HEVC content, it comes out as a garbled green mess 
as shown in the screenshots in this Dropbox folder:

https://www.dropbox.com/sh/07wzec9lcj689tl/AADo8JuZDNXqCfkYktCcZmEJa?dl=0

I don't see anything in the log that immediately strikes me as why it's 
not decoding properly (also in the Dropbox link). There's the WARN line 
where it's moaning about CUDA_ERROR_INVALID_HANDLE but it does this for 
the 8-bit streams as well and they work fine.

The public DASH MPDs I'm using to test with are available here:

8-bit: 
http://testassets.dashif.org/#testvector/details/586fb3879ae9045678eacd3c

10-bit: 
http://testassets.dashif.org/#testvector/details/586fb3879ae9045678eacd3f

If anyone has any ideas about what's going wrong, I'd love to hear from you.

For reference, I've tried running on the NVidia 390.48, 390.59 and 
396.26 drivers, as well as on CUDA 9.1 and 9.2 and it's broken across 
the board. I'm using a GT 1030 GPU on a Fedora 28 install, but it's also 
broken on Debian 9. It works fine when I use the libav software decoder, 
but that makes my CPU cry.

Best Regards,
Sam