[Bug 767712] xvimage: Missing sanity checking for allocation sizes for various video formats

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=767712

--- Comment #25 from Duncan Palmer <dunk.palmer at gmail.com> ---
Created attachment 330629
  --> https://bugzilla.gnome.org/attachment.cgi?id=330629&action=edit
Error out on sanity check failure, and add checking for RGB pixel formats

This patch causes gst_xvimage_allocator_alloc() to call on g_set_error() on
buffer size sanity check failure, rather than just logging a warning. I've also
added sanity checking for RGB pixel formats (note that these can't be derived
from im_format using the FOURCC notation).

I pass a *GstVideoFormatInfo into gst_xvimage_allocator_alloc() to facilitate
the extra sanity check. If the function is called from
gst_xvimage_memory_copy(), then that pointer will be NULL, and the function
won't do sanity checking for RGB pixel formats. I think this is likely ok, as
the bug as it is occurs on initial allocation, and this saves us from having to
store extra info in the GstXvImageMemory struct solely for sanity checking.

I've testing this with the YUV pixel formats supported by xv on my machine, but
there are no working RGB formats on any machine I have access to. So, please
take a look at my expected_size calculations for RGB and let me know if you see
any issues.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.