IEE754 and maximum correct integer value
winfried.libreoffice at gmail.com
Wed Sep 28 14:54:17 UTC 2016
>> But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, but 0, 0 and 3.
>> This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
> That's due to approxSub() being used for operator '-', which lessens the
> available precision. approxEqual() needs to be reworked to not scale in
> such cases.
> I'll investigate.
OK, that explains the remaining problem with my MOD() fix
More information about the LibreOffice