how does arithmetic exactly work in Calc?

[Michel Onoff]

Hi Regina,

by quickly browsing through the code, it seems like there's an
"optimization" that returns 0 when subtracting nearby values, so that's
at odd with the IEEE 754 spec.

Do you know if there's something similar to a spec for Calc, in addition
to and complementing the implementation? Or should one consider the
implementation as *the* definitive spec? I ask because while most
interested people can probably read and understand a spec, not everybody
is able to read C++ code.

Thanks!
Michel




On 2020-12-03 17:27, Regina Henschel wrote:
> Hi Michel,
>
> for example look for ScAdd in
> https://opengrok.libreoffice.org/xref/core/sc/source/core/tool/interpr5.cxx
> and for the rtl::math things in
> https://opengrok.libreoffice.org/xref/core/include/rtl/math.hxx
>
> Kind regards
> Regina
>
> Michel Onoff schrieb am 03-Dec-20 um 13:48:
>> Hi,
>>
>> I would like to understand how simple arithmetic (+ - * /) works in Calc.
>> I kind of guess that the underlying internal number representation is
>> IEEE 754 double (64 bit).
>> I also guess that when a double is shown on the sheet it is approximated
>> by a decimal with at most 15 significand digits.
>>
>> To reproduce the behavior below, use scientific notation with 20 digits
>> or more.
>>
>> I have two slightly different number x and x'.
>> x is 2^-49 (a formula) while x' is 1.77635683940025E-15 (a literal).
>> Their decimal representation appear equal on the sheet, but they are,
>> indeed, slightly different internally. You can set them apart as follows:
>>
>> y = x - 1.7763568394002E-15
>> y' = x' - 1.7763568394002E-15
>>
>> that is, by subtracting the same number from x and x'.
>> y and y' appear differently on the sheet, meaning that x and x' are
>> different internally to start with.
>>
>> However, x - x' is exactly 0. In IEEE 754 arithmetic, two numbers are
>> equal if and only if their difference is 0. That would mean that x and
>> x' are equal, which they are not from the above.
>>
>> So, what is the exact model for Calc's arithmetic?
>> Which calculations are performed exactly on fundamental arithmetic
>> operations?
>>
>> Thanks
>> Michel
>> _______________________________________________
>> LibreOffice mailing list
>> LibreOffice at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/libreoffice
>>
>