[Bug 90419] Accuracy of calculation in Fill

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=90419

--- Comment #5 from Piet van Oostrum <piet at vanoostrum.org> ---
(In reply to Eike Rathke from comment #1)
> While generally all floating point numbers that are not exactly
> representable in an IEEE 754 double lose precision, this specific case of
> filling a series of numbers could be improved by using multiplication
> instead of incrementing (adding delta) values.

Actually, this is already how it is done.
In ScTable::FillSeries:

        case FILL_LINEAR:
            {
                //  use multiplication instead of repeated addition
                //  to avoid accumulating rounding errors
                nVal = nStartVal;
                double nAdd = nStepValue;
                if ( !SubTotal::SafeMult( nAdd, (double) ++nIndex ) ||
                        !SubTotal::SafePlus( nVal, nAdd ) )
                    bError = true;
            }

However, depending on the initial values, this may still give rise to
"unexpected" values (that is unexpected with regard to the "intuitive"
expectations). But that's life with floating point numbers.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice/attachments/20160212/a622187e/attachment.html>