[Libva] H264 decoder erroneous output, some bytes off-by-one

[Sam Jansen]

Hi all,

I've been working on H264 encode, decode, and JPEG decode VA-API programs
recently, using the Intel va-driver on Sandy Bridge and Bay Trail.

This has gone well, but I've hit a problem with the H264 decoder in some
situations where I believe it is producing incorrect output. In some cases
chroma, and less often luma, data does not agree with the reference decoder
[1], often in Intra frames (but sometimes P frames too), always baseline
profile. I have an intra-encoded test stream where the 389th frame of a
stream encoded from the standard "paris (cif)" test stream is incorrect on
both Sandy Bridge and Bay Trail. Many other test streams pass fine, leaving
me to believe our decoder implementation is largely correct.

I've also implemented a mode where I am able to get the current encoded
picture out of the VA-API encoder. I can then compare this, byte-for-byte,
against a decode of the frame just encoded. I use this with out software
codecs to ensure our encode and decode match exactly. Using this technique
I can easily compare our software codec, and the VA-API H264 codecs in any
combination. I've found the VA-API encode always agrees with our decoder,
but the VA-API decoder does not always agree with the encode - either our
software encoder or the VA-API encoder (this testing just on Bay Trail).

I'd like to get to the bottom of this. The first step is removing our
decode implementation from the test, and using one you believe to be good.
What tools do you use for regression testing? Is this perhaps ffmpeg
invoked in some way? Is your test setup open source, such that I can modify
it to include my new test stream?

I'd be happy to provide test a stream or two here, but before I do, I'd
like to prove the problem isn't introduced in my software.

Regards,
Sam

PS: here's an extract of some of the chroma bytes where it has gone right.
On the top, some bytes of reference decoder output, on the bottom, bytes of
VA-API H264 decoder output. Note the 0x8f83 becomes 0x908f, then 0x8e8e
becomes 0x8d8d, the 0x8e8e becomes 0x8d8e.

9460 0024f30: 9894 8d8c 8d8f 9090 8f8e 8e8e 8e8e 8e8e ................

9460 0024f30: 9894 8d8c 8d8f 9090 908f 8d8d 8d8e 8e8e ................


PPS: resend, I had initially sent from the wrong address

[1]: http://iphome.hhi.de/suehring/tml/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/libva/attachments/20140610/24e7883b/attachment.html>