[Libreoffice-bugs] [Bug 101837] EDITING: Calculation error for CHIINV function

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=101837

--- Comment #15 from Aron Budea <baron at caesar.elte.hu> ---
Very peculiar, in both cases the first 13 iterations of the while loop in
lcl_IterateInverse(...) ("inverse quadric interpolation with additional
brackets") are the exact same.

Then in the 14th iteration, this diverges:
fRy = rFunction.GetValue(fSx);    <= rfunction is ScChiDistFunction

fRy becomes:
- in the good case: 0
- in the bad case: -4.6945954068622342e-17

However, in both cases the result of ScChiDistFunction::GetValue(...) is
supposed to be:
0.99999999999999911 - 0.99999999999999911
(return fp - rInt.GetChiDist(x, fDF);)

Surely the displayed value doesn't reflect the actual double value.


In the good case, the while loop exits afterwards, because this part of the
condition is false: fabs(fRy) > fYEps
In the bad case the while loop does an extra 3 iterations, which results in the
final discrepancy.

For the record, in the bad case the equality check only passes with 1 digit of
rounding, ie. ROUNDSIG(A8;1)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20180418/25a4900d/attachment-0001.html>