[Libreoffice-bugs] [Bug 130728] New: Change Calc to display in appropriate precision

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=130728

            Bug ID: 130728
           Summary: Change Calc to display in appropriate precision
           Product: LibreOffice
           Version: 6.0 all versions
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: normal
          Priority: medium
         Component: Calc
          Assignee: libreoffice-bugs at lists.freedesktop.org
          Reporter: jg at jguk.org

There's a well known issue, Calc will display this result as 1E-04

What was expected
Expected 0.0001


0.0042
0.0043

=A2-A1
1E-04

If we expand the column it presently shows show 0.0000999999999999994




What's gone wrong
There are various ways to fix this bug, the best is to calcuate accurately, and
not display raw Double precision.

A)
Calc seems to be displaying raw double precision, and not using it's own
accurate calculation of decimal numbers.
It's pretty easy to simply have a BigNum class with an int32 and an uint32
representing the decimal precision to avoid these issues.

B)
The alternative, is to select appropriate formatting 
.
eg in this case it would be #,##0.0000



Personally I favour (A), just calculate and show exactly correct values. 


My $5 Casio calculator displays the result correctly. There is no reason to
ever show a user  1E-04 or 0.0000999999999999994


Version: 6.0.7.3
Build ID: 1:6.0.7-0ubuntu0.18.04.10
CPU threads: 8; OS: Linux 4.15; UI render: default; VCL: gtk3; 
Locale: en-GB (en_GB.UTF-8); Calc: group

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20200217/92601d26/attachment.htm>