[Bug 90419] Accuracy of calculation in Fill

bugzilla-daemon at bugs.documentfoundation.org bugzilla-daemon at bugs.documentfoundation.org
Fri Feb 12 10:13:56 UTC 2016


--- Comment #5 from Piet van Oostrum <piet at vanoostrum.org> ---
(In reply to Eike Rathke from comment #1)
> While generally all floating point numbers that are not exactly
> representable in an IEEE 754 double lose precision, this specific case of
> filling a series of numbers could be improved by using multiplication
> instead of incrementing (adding delta) values.

Actually, this is already how it is done.
In ScTable::FillSeries:

        case FILL_LINEAR:
                //  use multiplication instead of repeated addition
                //  to avoid accumulating rounding errors
                nVal = nStartVal;
                double nAdd = nStepValue;
                if ( !SubTotal::SafeMult( nAdd, (double) ++nIndex ) ||
                        !SubTotal::SafePlus( nVal, nAdd ) )
                    bError = true;

However, depending on the initial values, this may still give rise to
"unexpected" values (that is unexpected with regard to the "intuitive"
expectations). But that's life with floating point numbers.

You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice/attachments/20160212/a622187e/attachment.html>

More information about the LibreOffice mailing list