[Pixman] pixman hangs in affine-test

[Søren Sandmann]

Søren Sandmann <sandmann at cs.au.dk> writes:

> Good catch. This branch:
>
>     http://cgit.freedesktop.org/~sandmann/pixman/log/?h=infinite-loop
>
> adds a test case. I haven't tried tracking down the bug at all.

The infinite loop is caused by an overflow in this expression from
FAST_BILINEAR_MAINLOOP_INT in pixman-inlines.h:

    pixman_int_to_fixed (vx + (width - 1) * unit_x) + 1; 

where unit_x is 0x200017bd and width - 1 is 5 so the multiplication
result doesn't fit in a signed 32 bit integer. A simple fix is to just
cast unit_x to int64_t, which is what the following patch does.

However, I don't know this code very well and I'm not convinced that
just casting is the right fix, so review and better suggestions are
welcome.


Soren