[Libreoffice-bugs] [Bug 129606] Drag-fill series produces values like 6.00000000000001

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=129606

--- Comment #5 from Mike Kaganski <mikekaganski at hotmail.com> ---
FTR: the assumptions in comment 4 have nothing to do with reality. Both fill
series dialog and drag-fill use the same method of multiplying the increment by
number of steps. The difference between these two methods is in determining the
increment, which is trivial for fill series (the increment value is given
directly), but requires a subtraction of floating-point values of varying
magnitude in case of drag-fill. When the magnitude of initial values in
drag-fill increases, the absolute precision of the subtraction decreases. See
below the comparison of approximate values of subtraction result and 0.1 given
directly, stored as IEEE 754 64-bit binary:

1.1 - 1.0 => ~0.10000000000000009
0.1       => ~0,10000000000000001

This can be seen e.g. at http://weitz.de/ieee/.

Also worth mentioning that "15.95" from Wikipedia might be misleading. For any
number with decimal representation

a.bcdefghijklmnopq * 10^x

where a is not 0, storing it as 64-bit binary IEEE 754 produces a number which,
when converted back to base10 and rounded to original precision, has first *16*
correct digits (i.e., a to p); and 17th is off by varying degree. So "15" there
in the article is not the number of correct decimal digits, but number of
correct digits *after the most significant*.

And last note is that the citation was not from "somewhere in 'ask'", but from
stackexchange.com [1].

[1]
https://unix.stackexchange.com/questions/472618/can-i-increase-the-number-of-decimal-places-stored-by-libre-office-calc

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20191229/d2652662/attachment.htm>