[Libreoffice-bugs] [Bug 138360] calculation: basic math fail, !calc rounding wrong! dividing by inverse sometimes different from multiply, 7E10/1E-5 wrong,

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=138360

Eike Rathke <erack at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |bibisectRequest
     Ever confirmed|0                           |1
             Status|UNCONFIRMED                 |NEW

--- Comment #1 from Eike Rathke <erack at redhat.com> ---
(In reply to b. from comment #0)
> '=7000000000000000*0,00001/0,00001' fails (6999~), while 
> '=7000000000000000/100000*100000' holds, 
> 
> acc. school math the results should be equal, 'weitz' tells me 70000000000 /
> 0,00001 = 6,9999~, 
You can't apply school math on IEEE-754 binary floating-point values over all
ranges. Period.

0.00001 has no exact binary representation, hence

70000000000 / 0.00001 => 6999999999999999


> question 1: weitz and calc may share some basic problem - buggy library,
> wrong compiler options or similar, 
They share IEEE-754 binary floating-point.


> question 2: if the error is 'unavoidable' would it be a good idea to use the
> better of both formulas in calc? i 'assume' calcs rounding routine uses the
> 'wrong' of both formulas and thus fails sometimes ... 
There's no extra rounding involved in multiplication and division. Also, the
formula expression is calculated as entered from left to right, there's no "use
the better".


> question 3: may it be possible that this or similar errors, which can be
> easily circumvented in the same or a similar way, are the cause of many
> other 'inaccuracies' in calc? 
There's no Calc error in this case. And this case of inaccuracy can't be
circumvented unless the final value is rounded but to what criteria?


> earlier versions of calc were 'better', so i think that 'better' is
> possible, 
Which version yielded the result 7000000000000000 for this very example? I
tried 5.3.7 and it delivers the same 6999999999999999 result.

Also doing that math in a raw C++ compiled program yields 6999999999999999.
Also Gnumeric agrees.


> didn't do extensive testing, but on a first glance even ex$el looks better, 
Maybe they apply some magic. Ask them to reveal their tricks.


> and i think it is important, such elementary errors affect many other
> calculations ... 
Yes, IEEE-754 binary floating-point is not perfect. Get over it.


> Steps to Reproduce:
> 1. key '=ROUND(5e15+1;9)' in a cell, 
> 2. hit enter, 
> 3. observe result 5000000000000002, 
I'll keep this bug for the rounding thing, we maybe could do better to not
round integer values beyond the decimal. Apart from that Gnumeric also delivers
the same 5000000000000002 result.


> 8. try the same steps in calc ver. 4.1.6.2, it is not! unavoidable, 
Oh, that version does it different? Maybe it did other things wrong. It
delivered better results for both

5000000000000002
6999999999999999

?

> 9. call it a regression, 
If so, maybe a bibisect could shed some light. Requesting. But it could also be
that compilers' IEEE-754 handling or libraries changed somewhat. And that even
both on Windows and Linux.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201125/b34169c3/attachment.htm>