[Libva] [PATCH] decoder: h264: improve AVC_REF_IDX_STATE for MVC.

[Gwenole Beauchesne]

Each Reference List Entry has Bit 6 set to one if the reference
picture is to be used as a long-term reference picture. However,
the H.264 standard, and subsequently the VA-API specs, makes it
possible to mark the picture as "used for short-term reference",
as "used for long-term reference", or even none of those flags.

This means we have to handle a minimum of 3 states. This doesn't
fit the range of a single bit. Let's examine how this could be
fixed from known practices.

There are cases where the picture is added to RefPicListX[] even
if it is not marked as "used for short-term reference" or "used
for long-term reference": MVC with inter-view reference components
or inter-view only reference components [H.8.4]. Ultimately, this
has an incidence on the value of colZeroFlag (8.4.1.2.2). Since
there is no way to program that, and that it depends on the picture
to be marked as "used for short-term reference" or not, then it
looks reasonable to imply Bit 6 (LongTermPicFlag) as a picture
that is *not* "used for short-term reference", i.e. thus including
genuine long-term reference pictures, and those that are neither
long-term reference nor short-term reference pictures.

In practice, this fixes MVCNV-2.264.

Regression tested on Broadwell with GStreamer/vaapi and FFmpeg/vaapi
for H.264 AVC cases. All tests (168/168) still PASS and produce
bitexact accurate output.

Note: a driver solution is needed as (i) we cannot expect all codec
layers to provide a workaround, (ii) it is perfectly allowed to
provide a RefPicListX entry without any reference flag set, and (iii)
there are actually hardware decoders that allow colocation info to be
programmed.

Signed-off-by: Gwenole Beauchesne <gwenole.beauchesne at intel.com>
---
 src/i965_decoder_utils.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/i965_decoder_utils.c b/src/i965_decoder_utils.c
index db8fe17..c72f6bc 100644
--- a/src/i965_decoder_utils.c
+++ b/src/i965_decoder_utils.c
@@ -348,8 +348,10 @@ avc_get_first_mb_bit_offset_with_epb(
 static inline uint8_t
 get_ref_idx_state_1(const VAPictureH264 *va_pic, unsigned int frame_store_id)
 {
+    const unsigned int ref_flags = VA_PICTURE_H264_SHORT_TERM_REFERENCE |
+        VA_PICTURE_H264_LONG_TERM_REFERENCE;
     const unsigned int is_long_term =
-        !!(va_pic->flags & VA_PICTURE_H264_LONG_TERM_REFERENCE);
+        ((va_pic->flags & ref_flags) != VA_PICTURE_H264_SHORT_TERM_REFERENCE);
     const unsigned int is_top_field =
         !!(va_pic->flags & VA_PICTURE_H264_TOP_FIELD);
     const unsigned int is_bottom_field =
-- 
1.8.3.2