[Libreoffice-bugs] [Bug 92016] Return value from Function

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=92016

--- Comment #4 from Gerhard Weydt <gerhard.weydt at t-online.de> ---
This is not a bug, but simply due to the fact that a variable of type single
has less precision than a number in Calc, in standard format, if this number
has only a small integer part. And it’s not a behaviour restricted to
LibreOffice Basic, but a general one.
Numbers of type single or double are stored in a binary format; you will find
the details in
https://en.wikipedia.org/wiki/Single-precision_floating-point_format.
By this transformation to binary format and back again to decimal format value
is changed to a small extent, depending on the precision used. You can get an
idea of the difference by executing the small program:

sub test
dim a as single, ai as long, af as double
a = 25.4
ai = Int(a)     'integer part
af = a – ai     'fraction part
msgbox af
end sub

It yields 0,399999618530273 if a is of type single, and 0,399999999999999, if a
is of type double I used a double variable for the result, to show as much
digits.).
Now Calc uses 13 digits after the decimal point in our case, where there only
two digits before it. I can not swear that Calc does a rounding when
transforming the return value to fit its standard format, but it does so, if
you change the number of decimal digits, and this fits the results we obtain.
If you round the result of our test programm to 13 decimal digits in the case
of the single type, you get 0,399996185303, exactly the result I get using your
function. The number you quote is slightly different, I don’t know why, but
this is not important for the argument. If, on  the other hand, you round the
second result, this yields 0.4.

So the difference between using a single or double variable is explainable, and
there will be no change in future releases. If you wish to have an exact value
even when using a high precision, as the standard format does in your case, you
have to use the double type. If you want to use only, say, two decimal digits,
then the single type is sufficient.

If you accept this explanation, you should change the status to resolved.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20170705/68fc987b/attachment.html>