[Libreoffice-bugs] [Bug 130725] ImpSvNumberInputScan::StringToDouble may produce inaccurate result

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=130725

--- Comment #29 from b. <newbie-02 at gmx.de> ---
hello @Mike Kaganski, 

while digging into calcs fp-math problems i'd say your comment (and sight?)
have some weak points: 

> see the example I put into the FAQ [2]. 

the '31000,99' sample there is less a problem of the weak precision how values
are represented as 'doubles', but of 'catastrophic cancellation' when
subtracting two operands of similar magnitude, and lack of appropriate handling
of such in LO calc, C++, or whereever. just filling the 'tail' (which is empty
reg. bit shifting) with '0' is insufficient. it is not possible to get a better
representation of the operand values with 'doubles', but it is! possible to get
a better result for the calculation, it requires some effort but it is
manageable compared to the effort spent to handele 100th of display attributes
or constructing gradients for backgrounds. 

the same applies to the time example, the result is absolutely only as exact as
the precision of its operands, neither calc nor 'Weitz' - which calculates a
somewhat different result - have a meaningful compensation for the
'cancellation', thus the result is useless for an increased precision - as it
is requested with the formatting to 0.000000 seconds. 

similar can and will occur in decimal, two pieces measured to 1,05 and 1,04
meters (which implies a precision of +/- 0,5 centimeters) may be equal or
different by up to two centimeters. 

> So here comes *your* PoV (which *we* understand clearly): that either Calc must calculate that precisely (which requires software calculations with infinite precision, and is clear WONTFIX), 

the effort to get 16 or 17 correct 'significant digits' is much less than
'infinite precision', 

> or that it at least should not show that to user automatically (unless explicitly told so), i.e. to limit aut  omatic number format precision - which is what I recommended you to file separately.

to implement more difference between displayed result accessible for the user
and 'real value' used for downstream calculation (calc already hides one to two
digits from the user) will produce more 'user irritation', not a good idea, the
opposite would fit, 

or to take the real! 0,0001 (0,00010000000000000000) as the result of '=0,0043
- 0,0042' and not the wrong 0,00010000000000000026 as calc acually does, check
with: '=RAWSUBTRACT(0,0043-0,0042;0,0001)', that's what i'd recommend, all
other ways will end in irritations at other places, 

0,00010000000000000026 is not a problem of limited precision to represent
0,0001 as double, it's a problem of how to calculate 0,0043 - 0,0042, and will
be worse for e.g.  4.000000000000003E-7 - 4.000000000000002E-7, relative error
nearly 6 percent, 

if you want to hide such errors from the user you are near to reduce each value
and calculation result to '1', 1,0000000000000007 - 1,0000000000000004
calculated to 2,22044604925031E-16 instead of 3E-16 is nearly 26% off, thus in
decimal you can't even trust the first digit,

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201214/0a1cf30f/attachment.htm>