[Libreoffice-bugs] [Bug 138220] calc: calculation: roundup: rounddown: fail in 7.1 alpha

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=138220

--- Comment #3 from Eike Rathke <erack at redhat.com> ---
Input of
=ROUNDUP(-999.12999999999994;11)
results in -999.13 and the formula expression becomes
=ROUNDUP(-999.13;11)
If you force a 0.000000000000 number format the resulting -999.130000000010 is
a display string conversion problem, not a wrong calculation.

Input of
=ROUNDDOWN(-999.12999999999994;11)
results in -999.13 and the formula expression becomes
=ROUNDDOWN(-999.13;11)

The input of -999.12999999999994 can't be represented in IEEE-754 double
floating-point and yields -999.13

The nearest representable value of -999.12999999999994 is
-999.1299999999999954525264911353588104248046875 with a raw hex value of
c08f390a3d70a3d7 (binary exponent.mantissa
-1111100111.0010000101000111101011100001010001111010111) which is the same
nearest value as for -999.13
Do not expect rounding down to whatever decimals would yield any
-999.12999999999990

This is no way "critical, data corruption", it is how IEEE-754 double
floating-point works.

I also get no different results in 6.1.6 apart from that the forced 12 decimals
formatted string for the first case is -999.130000000000

@zcrhonek: what and how did you test there to get to the bisected result?

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201130/c27e52c1/attachment-0001.htm>