[Libreoffice-bugs] [Bug 138360] ROUND() large numbers to decimals and to negative -5 decimals inaccurate

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=138360

Eike Rathke <erack at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Assignee|libreoffice-bugs at lists.free |erack at redhat.com
                   |desktop.org                 |
             Status|NEW                         |ASSIGNED
            Summary|calculation: basic math     |ROUND() large numbers to
                   |fail, !calc rounding wrong! |decimals and to negative -5
                   |dividing by inverse         |decimals inaccurate
                   |sometimes different from    |
                   |multiply, 7E10/1E-5 wrong,  |

--- Comment #5 from Eike Rathke <erack at redhat.com> ---
ROUND(1/0.00001;-5)

0.00001 == 1e-5 => 1.0000000000000001e-05
1/0.00001 => 99999.999999999985

1/0.00001 * 1e-5 => 0.99999999999999989
+ 0.5 + ... (correction value) => 1.5000000000000091
floor() => 1
/ 1.0000000000000001e-05 => 99999.999999999985

In round(), using the inverse factor and swapping multiplication and division
gives the proper result:

1/0.00001 / 1e5 => 0.99999999999999989
+ 0.5 + 9e-15 (correction value) => 1.5000000000000091
floor() => 1
* 1e5 => 100000


For large values like ROUND(5e15+1;9) it could be determined if the value can
be rounded at all.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201127/5f8396bf/attachment-0001.htm>