IEE754 and maximum correct integer value

Luc Castermans luc.castermans at
Wed Sep 28 12:31:17 UTC 2016

Winfried,  to have an external reference I entered your input in R
(64-bit), open source
statistical SW, R confirms your math. *Calc is wrong*.  Below is from

> (3^31)-((3^31)-1)[1] 1> (3^31)-((3^31)-2)[1] 2> (3^31)-((3^31)-3)[1] 3


Below is whtat R states about precision:

All *R* platforms are required to work with values conforming to the IEC
60559 (also known as IEEE 754) standard. This basically works with a
precision of 53 bits, and represents to that precision a range of absolute
values from about *2e-308* to *2e+308*. It also has special values NaN
(many of them), plus and minus infinity and plus and minus zero (although
*R* acts as if these are the same). There are also *denormal(ized)* (or
*subnormal*) numbers with absolute values above or below the range given
above but represented to less precision.

2016-09-28 13:15 GMT+02:00 Winfried Donkers <W.Donkers at>:

> Hi,
> I'm working on some improvements of Calc's MOD function, but I struggle
> with some unexpected results when using large integer values.
> As I understand IEEE754, double precision means a mantissa of 53 bits, so
> the maximum correct unsigned integer value would be 2^53.
> But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2,
> 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result,
> but 0, 0 and 3.
> This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
> I use a 64bit machine and master-version of LO.
> What am I doing wrong here?
> Winfried
> _______________________________________________
> LibreOffice mailing list
> LibreOffice at

Luc Castermans
mailto:luc.castermans at
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the LibreOffice mailing list