[Pixman] [PATCH 10/13] Use floating point combiners for all operators that involve divisions

[Søren Sandmann]

Bill Spitzak <spitzak at gmail.com> writes:

>> - source:	0xff (8 bits)
>> - source alpha:	0x01 (8 bits)
>> - mask alpha:	0x7b (8 bits)
>> - dest:		0x00 (8 bits)
>> - dest alpha:	0xff (8 bits)
>>
>> When (src IN mask) is computed in 8 bits, the resulting alpha channel
>> is 0 due to rounding:
>>
>>      floor ((0x01 * 0x7b) / 255.0 + 0.5) = floor (0.9823) = 0
>>
>> which means that since Render defines any division by zero as
>> infinity, the Fa and Fb for this operator end up as follows:
>>
>>      Fa = max (1 - (1 - 1) / 0, 0) = 0
>>
>>      Fb = min (1, (1 - 0) / 1) = 1
>>
>> and so since dest is 0x00, the overall result is 0.
>>
>> However, when computed in full precision, the alpha value no longer
>> rounds to 0, and so Fa ends up being
>>
>>      Fa = max (1 - (1 - 1) / 0.0001, 0) = 1
>>
>> and so the result is now
>>
>>      s * ma * Fa + d * Fb
>>
>>    = (1.0 * (0x7b / 255.0) * 1) + d * 0
>>
>>    = 0x7b / 255.0
>>
>>    = 0.4823
>
> This example could be fixed by treating 0/0 as returning 0.

That doesn't work in general though. For example CONJOINT_ATOP which has

     Fb = 1 - sa/da

Suppose sa is zero when computed as an integer, but non-zero when
computed as floating point. If da is zero, then Fb would be 1 with
integers, but 0 with floating point.


Søren