[PATCH xserver v9] xf86Cursor: Deal with rotation on GPU screens using a hw-cursor

[Hans de Goede]

When a slave-output is rotated the transformation is done on the blit
from master to slave GPU, so crtc->transform_in_use is not set, but we
still need to adjust the mouse position for things to work.

This commit modifies xf86_crtc_transform_cursor_position to not rely
on crtc->f_framebuffer_to_crtc, so that it can be used with GPU screens
to and always calls it for cursors on GPU screens.

Note not using crtc->f_framebuffer_to_crtc means that crtc->transform
will not be taken into account, that is ok, because when we've a transform
active hw-cursors are not used and xf86_crtc_transform_cursor_position
will never get called.

Signed-off-by: Hans de Goede <hdegoede at redhat.com>
---
Changes in v9:
-Write out the cursor coordinate conversion instead of playing tricks
 with (ab)using xf86_crtc_rotate_coord_back.

Changes in v8:
-Fix reflection on rotated outputs (use xf86_crtc_rotate_coord_back again)
-Only call xf86_crtc_transform_cursor_position when rotating or reflecting
 (instead of always calling it for GPU screens)

Changes in v7:
-Do not use xf86_crtc_rotate_coord_back, it is not suitable for our purposes
-Modify xf86_crtc_transform_cursor_position instead of adding a new
 xf86_crtc_transform_gpu_cursor_position function

Changes in v5:
-New patch in v5 of this patch-series
---
 hw/xfree86/modes/xf86Cursors.c | 51 +++++++++++++++++++++++++++++++++---------
 1 file changed, 40 insertions(+), 11 deletions(-)

diff --git a/hw/xfree86/modes/xf86Cursors.c b/hw/xfree86/modes/xf86Cursors.c
index c836534..1bc2b27 100644
--- a/hw/xfree86/modes/xf86Cursors.c
+++ b/hw/xfree86/modes/xf86Cursors.c
@@ -367,16 +367,45 @@ xf86_crtc_transform_cursor_position(xf86CrtcPtr crtc, int *x, int *y)
     xf86CursorScreenPtr ScreenPriv =
         (xf86CursorScreenPtr) dixLookupPrivate(&screen->devPrivates,
                                                xf86CursorScreenKey);
-    struct pict_f_vector v;
-    int dx, dy;
-
-    v.v[0] = (*x + ScreenPriv->HotX) + 0.5;
-    v.v[1] = (*y + ScreenPriv->HotY) + 0.5;
-    v.v[2] = 1;
-    pixman_f_transform_point(&crtc->f_framebuffer_to_crtc, &v);
-    /* cursor will have 0.5 added to it already so floor is sufficent */
-    *x = floor(v.v[0]);
-    *y = floor(v.v[1]);
+    int dx, dy, t;
+    Bool swap_reflection = FALSE;
+
+    *x = *x - crtc->x + ScreenPriv->HotX;
+    *y = *y - crtc->y + ScreenPriv->HotY;
+
+    switch (crtc->rotation & 0xf) {
+    case RR_Rotate_0:
+        break;
+    case RR_Rotate_90:
+        t = *x;
+        *x = *y;
+        *y = crtc->mode.VDisplay - t - 1;
+        swap_reflection = TRUE;
+        break;
+    case RR_Rotate_180:
+        *x = crtc->mode.HDisplay - *x - 1;
+        *y = crtc->mode.VDisplay - *y - 1;
+        break;
+    case RR_Rotate_270:
+        t = *x;
+        *x = crtc->mode.HDisplay - *y - 1;
+        *y = t;
+        swap_reflection = TRUE;
+        break;
+    }
+
+    if (swap_reflection) {
+        if (crtc->rotation & RR_Reflect_Y)
+            *x = crtc->mode.HDisplay - *x - 1;
+        if (crtc->rotation & RR_Reflect_X)
+            *y = crtc->mode.VDisplay - *y - 1;
+    } else {
+        if (crtc->rotation & RR_Reflect_X)
+            *x = crtc->mode.HDisplay - *x - 1;
+        if (crtc->rotation & RR_Reflect_Y)
+            *y = crtc->mode.VDisplay - *y - 1;
+    }
+
     /*
      * Transform position of cursor upper left corner
      */
@@ -399,7 +428,7 @@ xf86_crtc_set_cursor_position(xf86CrtcPtr crtc, int x, int y)
     /*
      * Transform position of cursor on screen
      */
-    if (crtc->transform_in_use)
+    if (crtc->rotation != RR_Rotate_0)
         xf86_crtc_transform_cursor_position(crtc, &crtc_x, &crtc_y);
     else {
         crtc_x -= crtc->x;
-- 
2.9.3