[Mesa-dev] [PATCH] vl/rbsp: add a check for emulation prevention three byte

[Christian König]

Am 08.08.2016 um 22:10 schrieb Leo Liu:
> This is the case when the "00 00 03" is very close to the beginning of
> nal unit header

I see where the problem is, but the fix is incorrect.

You always search for the emulation prevention three byte even when the 
previous fill has done so already. So it could happen in theory that you 
revert the escaping twice, e.g. remove valid bits.

You need to add this extra check to the end of vl_rbsp_init() after we 
searched for the end of the NAL unit.

Regards,
Christian.

>
> Signed-off-by: Leo Liu <leo.liu at amd.com>
> ---
>   src/gallium/auxiliary/vl/vl_rbsp.h | 10 +++++++++-
>   1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/src/gallium/auxiliary/vl/vl_rbsp.h b/src/gallium/auxiliary/vl/vl_rbsp.h
> index 7867238..c134d31 100644
> --- a/src/gallium/auxiliary/vl/vl_rbsp.h
> +++ b/src/gallium/auxiliary/vl/vl_rbsp.h
> @@ -77,8 +77,16 @@ static inline void vl_rbsp_fillbits(struct vl_rbsp *rbsp)
>      unsigned i, bits;
>   
>      /* abort if we still have enough bits */
> -   if (valid >= 32)
> +   if (valid >= 32) {
> +      /* search for the emulation prevention three byte */
> +      for (i = 24; i <= valid; i += 8) {
> +         if ((vl_vlc_peekbits(&rbsp->nal, i) & 0xffffff) == 0x3) {
> +            vl_vlc_removebits(&rbsp->nal, i - 8, 8);
> +            i += 8;
> +         }
> +      }
>         return;
> +   }
>   
>      vl_vlc_fillbits(&rbsp->nal);
>