[Bug 777506] vaapidecoder_h264: Recursive loop and segfault after a few seconds

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=777506

--- Comment #16 from Hyunjun Ko <zzoon at igalia.com> ---
(In reply to Víctor Manuel Jáquez Leal from comment #15)
> Review of attachment 345008 [details] [review]:
> 
> ::: gst-libs/gst/vaapi/gstvaapidecoder_h264.c
> @@ +3172,3 @@
> +      priv->frame_num -= MaxFrameNum;
> +
> +    if (slice_hdr->frame_num - priv->frame_num - 1 > sps->num_ref_frames)
> 
> is that -1 correct? it wasn't before, nor in the libav code.
I found if (slice_hdr->frame_num - priv->frame_num - 1) equals
sps->num_ref_frames, it doesn't need to reset prev_ref_frame_num. But without
-1, it's working likewise too, though it reset prev_ref_frame_num.

> 
> @@ +3176,3 @@
> +
> +    if (priv->frame_num < 0)
> +      priv->frame_num += MaxFrameNum;
> 
> in the libav this conditional is nested in the above if... is this correct?
This is a bit tricky, but I think it's correct.
And I don't think it's nested, since priv->frame_num is reset already.

Let's say, slice hdr frame is 1, sps->num ref frames is 1.
Then prev frame is -1, so it should be reset to assuming previous value which
is positive.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.