[Pixman] RFC: More accurate color conversion

[Carl Worth]

Søren Sandmann <sandmann at cs.au.dk> writes:
> If we add the additional assumption that the intervals between
> corresponding to successive integers must be a constant length, then it
> follows from f(0x0000) = 0.0 and f(0xffff) = 1.0 that f(i) = i /
> 65535.0. The easiest way to see that is to consider a conversion to
> 2-bit integers instead:
>
>      0.0                                             1.0
>       |-----------------------------------------------|
>      0b00            0b01            0b10            0b11

The picture I drew on my whiteboard yesterday was similarly for a 2-bit
integer, but I drew bars for the ranges of floating-point numbers that
would map to each integer. Like so:

        +-------+---------------+---------------+-------+
       0b00     |      0b01     |      0b10     |      0b11
        +-------+---------------+---------------+-------+

Then, my drawing for cairo's current algorithm was:

        +-----------+-----------+-----------+-----------+
       0b00         |  0b01     |      0b10 |          0b11
        +-----------+-----------+-----------+-----------+

Note that positions for each integer are the same with both approaches.

Then, yes, what you are saying is true: If we wanted to reduce cairo's
error for the roundtrip conversion starting with a floating-point value,
and we wanted to do this by only changing the integer-to-floating-point
mapping, then we would end up with (as you drew):

>      0.0                                             1.0
>       |-----------------------------------------------|
>            0b00        0b01        0b10        0b11

and with my bars:

        +-----------+-----------+-----------+-----------+
        |    0b00   |    0b01   |    0b10   |    0b11   |
        +-----------+-----------+-----------+-----------+

And yes, that would be undesirable in that we would lose access to the
floating-point extremes when converting from integers.

Color me convinced. Your change minimizes the error without introducing
unwanted deviations in the floating-point values obtained from integers.

> The main motivation is that I want to extend pixman's test suite with
> more tolerance based tests, where a reference pixel is computed in
> floating point and then compared to pixman's output. It's highly
> desirable here that the _reference_ computation doesn't introduce its
> own errors.

Thanks for explaining. I'm in favor of the change, (and particularly
since the change will be backed by additional testing).

-Carl

-- 
carl.d.worth at intel.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/pixman/attachments/20131008/10df58ca/attachment.pgp>