[Libreoffice-bugs] [Bug 138360] New: calculation: basic math fail, !calc rounding wrong! dividing by inverse sometimes different from multiply, 7E10/1E-5 wrong,

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=138360

            Bug ID: 138360
           Summary: calculation: basic math fail, !calc rounding wrong!
                    dividing by inverse sometimes different from multiply,
                    7E10/1E-5 wrong,
           Product: LibreOffice
           Version: 6.1.6.3 release
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: normal
          Priority: medium
         Component: Calc
          Assignee: libreoffice-bugs at lists.freedesktop.org
          Reporter: newbie-02 at gmx.de

Description:
looking for some accuracy problems i spotted that calc's 'round' fails for
plenty values, 

as similar happened with a user defined round routine i investigated further
and came to the point that: 

'=7000000000000000*0,00001/0,00001' fails (6999~), while 
'=7000000000000000/100000*100000' holds, 

acc. school math the results should be equal, 'weitz' tells me 70000000000 /
0,00001 = 6,9999~, 

question 1: weitz and calc may share some basic problem - buggy library, wrong
compiler options or similar, 

question 2: if the error is 'unavoidable' would it be a good idea to use the
better of both formulas in calc? i 'assume' calcs rounding routine uses the
'wrong' of both formulas and thus fails sometimes ... 

(my 'user space' rounding routine works fine since i exchanged that part,
before it had similar errors as calc), 

question 3: may it be possible that this or similar errors, which can be easily
circumvented in the same or a similar way, are the cause of many other
'inaccuracies' in calc? 

(fp-math isn't 'decimal-safe' as well as decimal math isn't 'real-world-safe',
but avoidable problems should be avoided and not sorted out as 'fp-problem, it
has to be that way', 

earlier versions of calc were 'better', so i think that 'better' is possible, 

didn't do extensive testing, but on a first glance even ex$el looks better, 

and i think it is important, such elementary errors affect many other
calculations ... 

Steps to Reproduce:
1. key '=ROUND(5e15+1;9)' in a cell, 
2. hit enter, 
3. observe result 5000000000000002, 
4. try to explain this to a user with simple math knowledge
5. key '=70000000000/0,00001' in a cell, 
6. observe result 6999999999999999, 
7. if you want to say 'notabug, this is just fp-maths, learn first why it has
to be that way': 
8. try the same steps in calc ver. 4.1.6.2, it is not! unavoidable, 
9. call it a regression, 

Actual Results:
5000000000000002, 
6999999999999999

Expected Results:
5000000000000001, 
7000000000000000


Reproducible: Always


User Profile Reset: Yes



Additional Info:
Version: 7.1.0.0.alpha1+ (x64)
Build ID: b61bf7c7cfcf97a5ade6d130873af146670bc2ee
CPU threads: 8; OS: Windows 6.1 Service Pack 1 Build 7601; UI render: default;
VCL: win
Locale: de-DE (de_DE); UI: en-US
Calc:

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201120/d3fc5eb9/attachment.htm>