[Libreoffice-bugs] [Bug 86005] EDITING: Wrongly calculated value 1 in calc

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=86005

b. <newbie-02 at gmx.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |newbie-02 at gmx.de

--- Comment #7 from b. <newbie-02 at gmx.de> ---
just to set some things clear: 

the intial fail for this bug isn't any function like INT, MOD or FLOOR ... but
the representation of originally 'decimal' defined values by dyadic / binary
'floating-point' numbers with limited length, and the subtraction of such
values 'with similar magnitude', 

e.g. '8,45 - 8' 

on such operations the 'upper bits' are zeroed by 'cancellation', and only few
significant bits are left to define the result, 

(consider metering a distance to 1,052 +/-0,0005 meters, that's about +/- 0,5
promille accuracy, now subtract 1 meter with similar precision, you have 0,052
meter with a potential error of +/- 0,001 m, that's about 2 percent error
range),  

thus the result has as much 'missing accuracy' as the magnitude of the result
is below the magnitude of the arguments, because the absolute error of up to
+/- 0,5 ULP, which is inherent in any fp-value, is kept, but is much bigger
relative to a smaller result than the bigger operand before, and thus becomes
visible, 

(it becomes more than one ULP of the result by 'bit shifting' to gain
'normalized' representations with a '1' in the first place), 

8,45 is in the range where fp's are puzzeled together from '1,x * 2^3', while
0,45 as result is in a range where they are constructed from 1,y * 2^-2, thus
an error of up to 0,5*2^-49 injected from 8,45 (where it vanishes in the lack
of precision / granularity of values) is now affecting a number ~2^5 times
smaller, and that is! visible ... and if not instantly spotted or rounded away
for display (column to small or numberformat insufficient) it becomes visible
by 'flip-flop' ('borderline') functions like INT, MOD, FLOOR etc. which don't
calculate smooth 'near target', but make digital decisions 'in' or 'out', 

german proverb: 'just beside it is also over'

as long as calc doesn't implement 'decimal-safe' datatypes or starts to
consider the (decimal) string representation for the calculations ... the only
chance for users is to try to avoid such calculations ... the developers try
their best and achieve small improvements every now and then, but the basic
problem remains and cannot be solved 'within' the simple IEEE 754 standard from
1985, 

'1/3 is periodically ended number but the result of the equation above is not'
- the result shouldn't be, but it suffers from intermediate values like 8,45
being endless fractions in binary, and from the 'error blow up' by
'high-bit-cancellation', to get 0,45 you need 01101 in the end of the mantissa,
but the mantissa doesn't know and pulls 5 zeroes from the 'off', 

'So it appears that while LibreOffice may format a number to 20 places' - the
20 decimals are 'fake' in most cases, for values like 1,222... you get 14, for
0,345... you get 15 valid decimal places and 6/5 padding '0', 0,00000123... can
use the whole range of digits, 

'the internal calculations are only valid up to 15 places.' - 64 bit fp allows
15,95 decimal digit accuracy, cutted down to 15 by calc, of which some
sometimes get lost in rounding, scientific format shows max 14 digits but has
at least one more digit internal (you can key it in, but cannot see ... )

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201108/72e27a1c/attachment.htm>