IEE754 and maximum correct integer value

[Eike Rathke]

Hi Winfried,

On Wednesday, 2016-09-28 13:15:53 +0200, Winfried Donkers wrote:

> I'm working on some improvements of Calc's MOD function, but I struggle with some unexpected results when using large integer values.
> As I understand IEEE754, double precision means a mantissa of 53 bits, so the maximum correct unsigned integer value would be 2^53.
> 
> But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, but 0, 0 and 3.
> This test is simply in Calc with e.g. =(3^31)-((3^31)-1).

That's due to approxSub() being used for operator '-', which lessens the
available precision. approxEqual() needs to be reworked to not scale in
such cases.

I'll investigate.

  Eike

-- 
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GPG key "ID" 0x65632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A
Better use 64-bit 0x6A6CD5B765632D3A here is why: https://evil32.com/
Care about Free Software, support the FSFE https://fsfe.org/support/?erack
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/libreoffice/attachments/20160928/215a01e3/attachment.sig>