IEE754 and maximum correct integer value
Winfried Donkers
W.Donkers at dci-electronics.nl
Wed Sep 28 11:15:53 UTC 2016
Hi,
I'm working on some improvements of Calc's MOD function, but I struggle with some unexpected results when using large integer values.
As I understand IEEE754, double precision means a mantissa of 53 bits, so the maximum correct unsigned integer value would be 2^53.
But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, but 0, 0 and 3.
This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
I use a 64bit machine and master-version of LO.
What am I doing wrong here?
Winfried
More information about the LibreOffice
mailing list