IEE754 and maximum correct integer value

[Eike Rathke]

Hi Winfried,

On Wednesday, 2016-09-28 16:54:17 +0200, Winfried Donkers wrote:

> > > But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, but 0, 0 and 3.
> > > This test is simply in Calc with e.g. =(3^31)-((3^31)-1).
> > That's due to approxSub() being used for operator '-', which lessens the
> > available precision. approxEqual() needs to be reworked to not scale in
> > such cases.
> > 
> > I'll investigate.
> > 
> OK, that explains the remaining problem with my MOD() fix
> (https://gerrit.libreoffice.org/29365) too.

Should be cured with
https://cgit.freedesktop.org/libreoffice/core/commit/?id=2135eae2a97c17d89cb47a2074830fd2d7b2226f
Please check with a cherry-pick.

  Eike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/libreoffice/attachments/20161004/36ed57e7/attachment.sig>