[Libreoffice-bugs] [Bug 138920] Precision issue in calc

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=138920

Eike Rathke <erack at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
                 CC|                            |erack at redhat.com
     Ever confirmed|0                           |1

--- Comment #9 from Eike Rathke <erack at redhat.com> ---
We're trying to round and due to the rounding even have some wrong display
strings resulting of the rounding to 15 decimals, but that 0.269985590000001
*is* already rounded; so yes, the entire display string conversion from binary
IEEE 754 to decimal needs to be reworked, but any "user expects" and "easy fix"
claim is just off.

Decimal input 9.25178559
nearest binary IEEE 754 representable value
9.251785590000000780719346948899328708648681640625
(1.001010000000111010100000101001110101010011101110001 * 2^3)

Decimal input 8.9818
nearest binary IEEE 754 representable value
8.98179999999999978399500832892954349517822265625
(1.00011111011010101110011111010101011001101100111101 * 2^3)

Subtraction:
1.0001010001110111000110100011110111000011110111000000 * 2^-2
0.269985590000000996724338619969785213470458984375
rounded to 15 decimals 0.269985590000001

Yes, unfortunately rounding to 15 decimals is not enough *in this* case, but 14
would be even more wrong in other cases, and sometimes 17 are required..

Fwiw, the nearest representable binary for 0.26998559 is
0.26998558999999999752361645732889883220195770263671875
(1.000101000111011100011010001111011100001111011010111 * 2^-2)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201215/954f1f43/attachment.htm>