[PATCH 2/2] glamor: Use float matrix for render transformations

[Keith Packard]

Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:

> Okay, now I see what's going on. The old glamor code used to convert
> the 16.16 fixed point values to 32-bit floats. This is definitely not
> great because of http://en.wikipedia.org/wiki/Rounding#Double_rounding
> on the "64-bit float" -> "16.16 fixed point" -> "32-bit float"
> conversion path. Directly passing 32-bit floats to XRENDER should
> somewhat mitigate the accuracy loss, but not totally eliminate it.

You need to separate the coordinate representation issues from the
transformation representation issues. For coordinates, yes, 32-bit
floats would lose some of the low bits. For transformation matrices,
the values aren't coordinates; they're all scaling factors for the
coordinates.

        | A B C |
        | D E F |
        | G H J |

For affine transforms, A, B, D, E generally have small values, related
to the overall image scaling factor. C and F are the translation while G
and H are zero and J is one. The small values (A, B, D, E) are
multiplied by coordinates, the large values (C, F) are multiplied by
one. Having 24 bits of mantissa for all of these values offers the
maximum precision for the resulting values.

Let's try to construct a really bad case. That's where your matrix
has values which are just less than 0.5 ULP (1/2**17)

        A * x + B * y + C
  x' =  -----------------
        G * x + H * y + J

so, the maximum error here would be where the denominator was small, and
the maximum magnitude of the matrix values was large so as to keep us
From scaling the matrix up to provide precision for the smallest
members.

Here's a matrix which almost reflects in X with a tiny amount of
non-affine adjustment

        | -1    0    32767 |
        |  0    1        0 |
        |  e    0        1 |

Set e to 1/2 ULP - epsilon (a tiny bit less than 1/2**17) so that the
fixed point matrix has a zero there and the floating point matrix has a
non-zero value. For x, y of -2000,2000:

      -1 * -2000 + 0 * 2000 + 32767    34767
x' =  ----------------------------- = ------- = 35305.72
       e * -2000 + 0 * 2000 + 1       0.98474

      -1 * -2000 + 0 * 2000 + 32767    34767
x' ~= ----------------------------- = ------- = 34767
       0 * -2000 + 0 * 2000 + 1          1

This has an error of about 539 pixels.

> Is the accuracy actually good enough with this patch applied and solves
> the "off by several pixels" problem? The patch still looks messed up
> with the use of a mix of double and float.

The example above is contrived, but shows that fixed point
introduces considerable inaccuracy when used in the transformation matrix.

-- 
keith.packard at intel.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 810 bytes
Desc: not available
URL: <http://lists.x.org/archives/xorg-devel/attachments/20140818/f51506c6/attachment.sig>