how does arithmetic exactly work in Calc?

[Regina Henschel]

Hi Michel,

for example look for ScAdd in
https://opengrok.libreoffice.org/xref/core/sc/source/core/tool/interpr5.cxx
and for the rtl::math things in
https://opengrok.libreoffice.org/xref/core/include/rtl/math.hxx

Kind regards
Regina

Michel Onoff schrieb am 03-Dec-20 um 13:48:
> Hi,
> 
> I would like to understand how simple arithmetic (+ - * /) works in Calc.
> I kind of guess that the underlying internal number representation is
> IEEE 754 double (64 bit).
> I also guess that when a double is shown on the sheet it is approximated
> by a decimal with at most 15 significand digits.
> 
> To reproduce the behavior below, use scientific notation with 20 digits
> or more.
> 
> I have two slightly different number x and x'.
> x is 2^-49 (a formula) while x' is 1.77635683940025E-15 (a literal).
> Their decimal representation appear equal on the sheet, but they are,
> indeed, slightly different internally. You can set them apart as follows:
> 
> y = x - 1.7763568394002E-15
> y' = x' - 1.7763568394002E-15
> 
> that is, by subtracting the same number from x and x'.
> y and y' appear differently on the sheet, meaning that x and x' are
> different internally to start with.
> 
> However, x - x' is exactly 0. In IEEE 754 arithmetic, two numbers are
> equal if and only if their difference is 0. That would mean that x and
> x' are equal, which they are not from the above.
> 
> So, what is the exact model for Calc's arithmetic?
> Which calculations are performed exactly on fundamental arithmetic
> operations?
> 
> Thanks
> Michel
> _______________________________________________
> LibreOffice mailing list
> LibreOffice at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/libreoffice
>