[Bug 110702] segfault in radeonsi HEVC hardware decoding with yuv420p10le

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=110702

--- Comment #11 from Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer at amd.com> ---
(In reply to Christian König from comment #10)
> (In reply to Pierre-Eric Pelloux-Prayer from comment #9)
> > The driver fails when trying to allocate a buffer for this video with a
> > ENOMEM error (the requested size is 3 GB).
> 
> Well that strongly sounds like we miscalculated the necessary size somewhere.

Indeed.
The size is computed by the `calc_ctx_size_h265_main10()` function.

I'm not familiar enough with hevc to fix it though (but the calculation seems
to overflow because context_buffer_size_per_ctb_row is 1GB and is multiplied by
max_references (= 23) and the result is stored in an unsigned int).

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20190619/e2b6e1f1/attachment-0001.html>