[PATCH weston] compositor: Fix rendering with shm transformed buffers

[Ander Conselvan de Oliveira]

On 11/28/2012 06:56 PM, Rob Bradford wrote:
> Hi Ander, good catch..
>
> On 28 November 2012 15:10, Ander Conselvan de Oliveira
> <ander.conselvan.de.oliveira at intel.com> wrote:
>> The implementation of buffer transformation didn't handle transformed
>> shm buffers properly. The partial texture upload was broken since the
>> damage is in surface coordinates that don't necessarily match the
>> buffer's coordinates. It also wouldn't handle the buffer stride
>> properly, resulting in incorrect rendering if it didn't match the
>> buffer's width.
>>
>
>> +WL_EXPORT pixman_box32_t
>> +weston_surface_to_buffer_rect(struct weston_surface *surface,
>> +                             pixman_box32_t rect)
>> +{
>> +       float x1, x2, y1, y2;
>> +
>> +       pixman_box32_t ret;
>> +
>> +       weston_surface_to_buffer_float(surface, rect.x1, rect.y1, &x1, &y1);
>> +       weston_surface_to_buffer_float(surface, rect.x2, rect.y2, &x2, &y2);
>> +
>> +       if (x1 <= x2) {
>> +               ret.x1 = x1;
>> +               ret.x2 = x2;
>> +       } else {
>> +               ret.x1 = x2;
>> +               ret.x2 = x1;
>> +       }
>> +
>> +       if (y1 <= y2) {
>> +               ret.y1 = y1;
>> +               ret.y2 = y2;
>> +       } else {
>> +               ret.y1 = y2;
>> +               ret.y2 = y1;
>> +       }
>
> But since we pass these values into glTexImage2D - I think we might
> need to floor the lower x/y bound and ceil the upper x/y bound?

gl_renderer_flush_damage() shouldn't pass a rectangle that falls outside 
the surface to this function (and the gl-renderer has the assumption 
that the texture size matches the surface size). The problem this last 
snippet solves is to fix up the result rectangle so that one can use x1, 
y1, x2 - x1, y2 - y1 as a x, y, width, height rectangle.

Say you have a 500x250 surface, but its buffer is rotated 90 degrees, so 
that it is 250x500. If the surface damage is the rectangle defined by 
the points (10, 20) and (40, 50), we would convert both coordinates and 
have (230,10) and (200,40). This does define a rectangle in the buffer, 
but if we calculate width as x2 - x1, we would get a negative value.

Ander