[Libreoffice-bugs] [Bug 138920] Precision issue in calc

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=138920

--- Comment #12 from Jonny Grant <jg at jguk.org> ---
(In reply to Eike Rathke from comment #11)
> You seem to not understand. Once the value is in the IEEE 754 binary format
> there are no "8 decimal places" to use.
> 
> It's about sufficient precision and rounding when converting from binary
> floating-point to the decimal display string.
> 
> See also https://erack.de/bookmarks/D.html#010203
> Or if you want to dive deep into then
> https://erack.de/bookmarks/D.html#Computer_Arithmetic_and_IEEE_754

Dear Eike

My apologies if I was not clear. I only refer to the decimal display of the
IEEE 754 number. More specifically, when I press <Ctrl>+C, and <Ctrl>+V in an
email, i see 0.269985590000001  I'm only referring to the string conversion of
this number which is stored in floating-point format by Calc.

Calc needs to display as a string an appropriate number of decimal places to
for the result, and the copy&paste buffer should contain the same, that should
also be used so that copy&paste works as expected.

Here is an even simpler example :-

1) Type =10.1 - 9.9
2) Observe 0.2
3) <Ctrl>+C
4) Move to your text editor and press <Ctrl>+V
5) Observe 0.199999999999999
6) Switch back to Calc, resize the column
7) Observe 0.199999999999999

I'll attach another screenshot

Can the string conversion function take a hint on how many decimal places are
needed? In this example, it will never be more than 1 decimal place needed
right?

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201219/05ae49a8/attachment.htm>