[PATCH] randr: check rotated virtual size limits correctly

[Robert Hooker]

On Thu, May 12, 2011 at 2:05 AM, Jeremy Huddleston <jeremyhu at apple.com> wrote:
> That looks like it does the right thing... (still... get a Tested-by:)
>
> Reviewed-by: Jeremy Huddleston <jeremyhu at apple.com>
>
> On May 11, 2011, at 7:18 AM, Aaron Plattner wrote:
>
>> Commit d1107918d4626268803b54033a07405122278e7f introduced checks to
>> the RandR path that cause RRSetScreenConfig requests to fail if the
>> size is too large.  Unfortunately, when RandR 1.1 rotation is enabled
>> it compares the rotated screen dimensions to the unrotated limits,
>> which causes 90- and 270-degree rotation to fail unless your screen
>> happens to be square:
>>
>>  X Error of failed request:  BadValue (integer parameter out of range for operation)
>>    Major opcode of failed request:  153 (RANDR)
>>    Minor opcode of failed request:  2 (RRSetScreenConfig)
>>    Value in failed request:  0x780
>>    Serial number of failed request:  14
>>    Current serial number in output stream:  14
>>
>> Fix this by moving the check above the code that swaps the dimensions
>> based on the rotation.
>>
>> Signed-off-by: Aaron Plattner <aplattner at nvidia.com>
>> ---
>> This patch is against server-1.10-branch.
>>
>> randr/rrscreen.c |   12 ++++++------
>> 1 files changed, 6 insertions(+), 6 deletions(-)
>>
>> diff --git a/randr/rrscreen.c b/randr/rrscreen.c
>> index 1bc1a9e..da6d48d 100644
>> --- a/randr/rrscreen.c
>> +++ b/randr/rrscreen.c
>> @@ -910,12 +910,6 @@ ProcRRSetScreenConfig (ClientPtr client)
>>      */
>>     width = mode->mode.width;
>>     height = mode->mode.height;
>> -    if (rotation & (RR_Rotate_90|RR_Rotate_270))
>> -    {
>> -     width = mode->mode.height;
>> -     height = mode->mode.width;
>> -    }
>> -
>>     if (width < pScrPriv->minWidth || pScrPriv->maxWidth < width) {
>>       client->errorValue = width;
>>       free(pData);
>> @@ -927,6 +921,12 @@ ProcRRSetScreenConfig (ClientPtr client)
>>       return BadValue;
>>     }
>>
>> +    if (rotation & (RR_Rotate_90|RR_Rotate_270))
>> +    {
>> +     width = mode->mode.height;
>> +     height = mode->mode.width;
>> +    }
>> +
>>     if (width != pScreen->width || height != pScreen->height)
>>     {
>>       int     c;
>> --
>> 1.7.4.1
>>
>> _______________________________________________
>> xorg-devel at lists.x.org: X.Org development
>> Archives: http://lists.x.org/archives/xorg-devel
>> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>>
>
> _______________________________________________
> xorg-devel at lists.x.org: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>

It works here. Reference:
https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/740933

Tested-by: Robert Hooker <robert.hooker at canonical.com>
Tested-by: Kent Baxley <kent.baxley at canonical.com>