how does arithmetic exactly work in Calc?
erack at redhat.com
Thu Dec 3 16:41:37 UTC 2020
On Thursday, 2020-12-03 13:48:25 +0100, Michel Onoff wrote:
> I would like to understand how simple arithmetic (+ - * /) works in Calc.
> I kind of guess that the underlying internal number representation is
> IEEE 754 double (64 bit).
> I also guess that when a double is shown on the sheet it is approximated
> by a decimal with at most 15 significand digits.
> To reproduce the behavior below, use scientific notation with 20 digits
> or more.
Note that with increasing digits beyond 15 significands you currently
will not gain anything except additional zeros.
> I have two slightly different number x and x'.
> x is 2^-49 (a formula) while x' is 1.77635683940025E-15 (a literal).
> Their decimal representation appear equal on the sheet, but they are,
> indeed, slightly different internally. You can set them apart as follows:
> y = x - 1.7763568394002E-15
> y' = x' - 1.7763568394002E-15
> that is, by subtracting the same number from x and x'.
> y and y' appear differently on the sheet, meaning that x and x' are
> different internally to start with.
> However, x - x' is exactly 0. In IEEE 754 arithmetic, two numbers are
> equal if and only if their difference is 0. That would mean that x and
> x' are equal, which they are not from the above.
The Calc + and - operators try to cater for numeric inaccuracies and tie
to zero in some ranges, otherwise the famous example of 0.3 - 0.2 - 0.1
would not be 0.0 but -2.77555756156289E-17 instead.
If you want the "raw" subtraction of your two values you can check with
that yields 3.94430452610506E-31
The double to string conversion needs further improvement to be able to
more precisely represent values in decimals so could actually convert
2^-49 to 1.7763568394002505E-15
GPG key 0x6A6CD5B765632D3A - 2265 D7F3 A7B0 95CC 3918 630B 6A6C D5B7 6563 2D3A
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 833 bytes
Desc: not available
More information about the LibreOffice