how does arithmetic exactly work in Calc?

[Eike Rathke]

Hi Michel,

On Thursday, 2020-12-03 13:48:25 +0100, Michel Onoff wrote:

> I would like to understand how simple arithmetic (+ - * /) works in Calc.
> I kind of guess that the underlying internal number representation is
> IEEE 754 double (64 bit).

Yes.

> I also guess that when a double is shown on the sheet it is approximated
> by a decimal with at most 15 significand digits.

Yes.

> To reproduce the behavior below, use scientific notation with 20 digits
> or more.

Note that with increasing digits beyond 15 significands you currently
will not gain anything except additional zeros.

> I have two slightly different number x and x'.
> x is 2^-49 (a formula) while x' is 1.77635683940025E-15 (a literal).
> Their decimal representation appear equal on the sheet, but they are,
> indeed, slightly different internally. You can set them apart as follows:
> 
> y = x - 1.7763568394002E-15
> y' = x' - 1.7763568394002E-15
> 
> that is, by subtracting the same number from x and x'.
> y and y' appear differently on the sheet, meaning that x and x' are
> different internally to start with.

Yes.

> However, x - x' is exactly 0. In IEEE 754 arithmetic, two numbers are
> equal if and only if their difference is 0. That would mean that x and
> x' are equal, which they are not from the above.

The Calc + and - operators try to cater for numeric inaccuracies and tie
to zero in some ranges, otherwise the famous example of 0.3 - 0.2 - 0.1
would not be 0.0 but -2.77555756156289E-17 instead.

If you want the "raw" subtraction of your two values you can check with
=RAWSUBTRACT(2^-49;1.77635683940025E-015)
that yields 3.94430452610506E-31

The double to string conversion needs further improvement to be able to
more precisely represent values in decimals so could actually convert
2^-49 to 1.7763568394002505E-15

  Eike

-- 
GPG key 0x6A6CD5B765632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/libreoffice/attachments/20201203/d9c5d845/attachment.sig>