[Libreoffice-bugs] [Bug 129606] Drag-fill series produces values like 6.00000000000001

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=129606

--- Comment #21 from b. <newbie-02 at gmx.de> ---
OT for the problem, but might be helpful to understand floats: 

------------ 
c#05: "Also worth mentioning that "15.95" from Wikipedia might be misleading.
For any number with decimal representation

a.bcdefghijklmnopq * 10^x

where a is not 0, storing it as 64-bit binary IEEE 754 produces a number which,
when converted back to base10 and rounded to original precision, has first *16*
correct digits (i.e., a to p)" 
------------ 

that holds up to 9,007199254740992*10^15, above that value double floats have a
'granularity' of 2 (52 binary 'after dot' digits with an exponent of 53, thus
times 2^53), 
thus for about 10% of the number range from 0 to 9,999999999999999*10^15 the
precision is '2' instead of '1', '16 correct significant decimal digits' is not
fullfilled, which is probably expressed by the value 15.95 at wikipedia, 

(keying in 9007199254740993.0 into the cited Weitz IEEE calculator is changed
on the fly to 9007199254740992.0, while 9007199254740993.1 changes to
9007199254740994.0 (which expresses an own understanding of rounding))

calc does similar but - afais - has integer precision only up to
9,007199254740991 and shortens floats to 15 digits, and does that more by
truncating than roundig (only for the display string?) which leads to funny
number sequences if you repeat adding 1 to the predecessor starting with e.g.
9007199254740989.0 (ok, '1' is cutted away instead of rounded up, try
1,000000000000001)

similar happens for e.g. 4.503.599.627.370.494 with repeated '+ 0,5'

4.503.599.627.370.494,000
4.503.599.627.370.490,000  too harsh rouding (15 digits), 
                    ^
4.503.599.627.370.495,000  correct
4.503.599.627.370.500,000  too harsh rounding, 
                  ^^^
4.503.599.627.370.496,000  correct
4.503.599.627.370.496,000  not rounded up, 
                    ^ ^
4.503.599.627.370.496,000  wrong value carried on, 
                    ^
4.503.599.627.370.496,000  ...
                    ^
4.503.599.627.370.496,000
                    ^
4.503.599.627.370.496,000
                    ^
4.503.599.627.370.496,000
                    ^
questionable rounding, 
'holding precision internally' not fulfilled, 


4.503.599.627.370.494 repeatedly '+ 0,5000000000000001'

4.503.599.627.370.494,000
4.503.599.627.370.490,000  in 16 digits ~495 (rounded) is possible,  
                    ^
4.503.599.627.370.495,000  correct
4.503.599.627.370.500,000  too harsh rounding, 
                  ^^^
4.503.599.627.370.496,000  correct, 
4.503.599.627.370.497,000
4.503.599.627.370.498,000  rounded value taken as argument? 
                    ^
4.503.599.627.370.499,000  rounded value taken as argument? 
                    ^
4.503.599.627.370.500,000  again ...
                  ^^^
4.503.599.627.370.501,000  again ...
                  ^^^
4.503.599.627.370.502,000  again ...
                  ^^^
4.503.599.627.370.503,000  again ...
                  ^^^
4.503.599.627.370.504,000  again ...
                   ^^
questionable rounding, 
'holding precision internally' not fulfilled, 

irritating for simple minded users like me ... 

similar wants to happen for other values, especially when crossing precision
range borders, but overall the 'rounding concept' of calc seems problematic to
me ... 

(ex$el (2010) is worse in rounding / truncating, but - afais - produces series
with better monotonity while matching or exceeding the other errors of calc,
including the 0,0999... instead of 0,1 on sheet - fill)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20200927/fb6c12a3/attachment-0001.htm>