[PATCH xserver 5/7] xfree86/modes: Adapt xf86Randr12CrtcComputeGamma() for depth 30. (v2)

[Mario Kleiner]

At screen depths > 24 bit, the color palettes passed into
xf86Randr12CrtcComputeGamma() can have a larger number of slots
than the crtc's hardware lut. E.g., at depth 30, 1024 palette
slots vs. 256 hw lut slots. This palette size > crtc gamma size
case is not handled yet and leads to silent failure, so gamma
table updates do not happen.

Add a new subsampling path for this case.

This makes lut updates work again, as tested with the xgamma
utility (uses XF86VidMode extension) and some RandR based
gamma ramp animation.

v2: Better resampling when subsampling the palette, as
    proposed by Ville. Now reaches the max index of the
    palette and deals with non-power-of-two sizes. Thanks.

Signed-off-by: Mario Kleiner <mario.kleiner.de at gmail.com>
Reviewed-by: Antoine Martin <antoine at nagafix.co.uk> (v1)
Cc: <ville.syrjala at linux.intel.com>
---
 hw/xfree86/modes/xf86RandR12.c | 96 ++++++++++++++++++++++++++++++------------
 1 file changed, 69 insertions(+), 27 deletions(-)

diff --git a/hw/xfree86/modes/xf86RandR12.c b/hw/xfree86/modes/xf86RandR12.c
index fe8770d..8180483 100644
--- a/hw/xfree86/modes/xf86RandR12.c
+++ b/hw/xfree86/modes/xf86RandR12.c
@@ -1258,40 +1258,82 @@ xf86RandR12CrtcComputeGamma(xf86CrtcPtr crtc, LOCO *palette,
 
     for (shift = 0; (gamma_size << shift) < (1 << 16); shift++);
 
-    gamma_slots = crtc->gamma_size / palette_red_size;
-    for (i = 0; i < palette_red_size; i++) {
-        value = palette[i].red;
-        if (gamma_red)
-            value = gamma_red[value];
-        else
-            value <<= shift;
+    if (crtc->gamma_size >= palette_red_size) {
+        /* Upsampling of smaller palette to larger hw lut size */
+        gamma_slots = crtc->gamma_size / palette_red_size;
+        for (i = 0; i < palette_red_size; i++) {
+            value = palette[i].red;
+            if (gamma_red)
+                value = gamma_red[value];
+            else
+                value <<= shift;
+
+            for (j = 0; j < gamma_slots; j++)
+                crtc->gamma_red[i * gamma_slots + j] = value;
+        }
+    } else {
+        /* Downsampling of larger palette to smaller hw lut size */
+        for (i = 0; i < crtc->gamma_size; i++) {
+            value = palette[i * (palette_red_size - 1) / (crtc->gamma_size - 1)].red;
+            if (gamma_red)
+                value = gamma_red[value];
+            else
+                value <<= shift;
 
-        for (j = 0; j < gamma_slots; j++)
-            crtc->gamma_red[i * gamma_slots + j] = value;
+            crtc->gamma_red[i] = value;
+        }
     }
 
-    gamma_slots = crtc->gamma_size / palette_green_size;
-    for (i = 0; i < palette_green_size; i++) {
-        value = palette[i].green;
-        if (gamma_green)
-            value = gamma_green[value];
-        else
-            value <<= shift;
+    if (crtc->gamma_size >= palette_green_size) {
+        /* Upsampling of smaller palette to larger hw lut size */
+        gamma_slots = crtc->gamma_size / palette_green_size;
+        for (i = 0; i < palette_green_size; i++) {
+            value = palette[i].green;
+            if (gamma_green)
+                value = gamma_green[value];
+            else
+                value <<= shift;
 
-        for (j = 0; j < gamma_slots; j++)
-            crtc->gamma_green[i * gamma_slots + j] = value;
+            for (j = 0; j < gamma_slots; j++)
+                crtc->gamma_green[i * gamma_slots + j] = value;
+        }
+    } else {
+        /* Downsampling of larger palette to smaller hw lut size */
+        for (i = 0; i < crtc->gamma_size; i++) {
+            value = palette[i * (palette_green_size - 1) / (crtc->gamma_size - 1)].green;
+            if (gamma_green)
+                value = gamma_green[value];
+            else
+                value <<= shift;
+
+            crtc->gamma_green[i] = value;
+        }
     }
 
-    gamma_slots = crtc->gamma_size / palette_blue_size;
-    for (i = 0; i < palette_blue_size; i++) {
-        value = palette[i].blue;
-        if (gamma_blue)
-            value = gamma_blue[value];
-        else
-            value <<= shift;
+    if (crtc->gamma_size >= palette_blue_size) {
+        /* Upsampling of smaller palette to larger hw lut size */
+        gamma_slots = crtc->gamma_size / palette_blue_size;
+        for (i = 0; i < palette_blue_size; i++) {
+            value = palette[i].blue;
+            if (gamma_blue)
+                value = gamma_blue[value];
+            else
+                value <<= shift;
 
-        for (j = 0; j < gamma_slots; j++)
-            crtc->gamma_blue[i * gamma_slots + j] = value;
+            for (j = 0; j < gamma_slots; j++)
+                crtc->gamma_blue[i * gamma_slots + j] = value;
+        }
+    } else {
+        /* Downsampling of larger palette to smaller hw lut size */
+        for (i = 0; i < crtc->gamma_size; i++) {
+            value = palette[i * (palette_blue_size - 1) / (crtc->gamma_size - 1)].blue;
+            if (gamma_blue)
+                value = gamma_blue[value];
+            else
+                value <<= shift;
+
+            crtc->gamma_blue[i] = value;
+        }
     }
 }
 
-- 
2.7.4