[Spice-devel] [spice-gtk 1/4] mjpeg: properly abort decompression in error path

[Christophe Fergeau]

spice-gtk jpeg decompression code honors libjpeg's recommended size
for its output buffer to improve performance. However, when this
recommended size is too big, it just gives up on the decompression
process by returning early from the function. But since
jpeg_start_decompress has been called to compute this recommended
size, the decompression must be aborted before returning, otherwise
libjpeg will get in an inconsistent state and will abort next time
we try to use it.
This commit also moves the check that the recommended size isn't
too big out of the decompression loop because it shouldn't changed
during decompression.
---
 gtk/channel-display-mjpeg.c |   10 ++++++++--
 1 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/gtk/channel-display-mjpeg.c b/gtk/channel-display-mjpeg.c
index b9ff108..dee0ef9 100644
--- a/gtk/channel-display-mjpeg.c
+++ b/gtk/channel-display-mjpeg.c
@@ -95,9 +95,15 @@ void stream_mjpeg_data(display_stream *st)
 #endif
     // TODO: in theory should check cinfo.output_height match with our height
     jpeg_start_decompress(&st->mjpeg_cinfo);
+    /* rec_outbuf_height is the recommended size of the output buffer we
+     * pass to libjpeg for optimum performance
+     */
+    if (st->mjpeg_cinfo.rec_outbuf_height > G_N_ELEMENTS(lines)) {
+        jpeg_abort_decompress(&st->mjpeg_cinfo);
+        g_return_if_reached();
+    }
+
     for (i = 0; i < height; ) {
-        // according to header
-        g_return_if_fail(st->mjpeg_cinfo.rec_outbuf_height <= 4);
         for (j = 0; j < st->mjpeg_cinfo.rec_outbuf_height; j++) {
             lines[j] = dest;
 #ifdef JCS_EXTENSIONS
-- 
1.7.6