[Libreoffice-bugs] [Bug 55960] Filling cells with incremental numbers – wrong numbers

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=55960

--- Comment #16 from Mike Kaganski <mikekaganski at hotmail.com> ---
In double precision,

> 0.05*39=1.9500000000000002
> -2+0.05*39=-0.049999999999999822

First idea is that if we returned not only the increment from the approxDiff,
but also the number of decimals for rounding (which is 15 in this case), and
then used the rounding in the following calculation of the resulting increment
for each cell, then this specific sequence would be OK.

Yet, this is not a full solution, since this would fail with longer sequences,
like "-22;-21.95;...". That is because when multiplying the atomic increment,
we also need to decrease resulting absolute precision proportionally (so if
initially it's 5E-16 for 0.05, then for 0.05*39, it's 5*E-16*39=1.95E-14, so we
need to round the result of 0.05*39 to 14 decimals).

Eike: if agreed, I suggest to make this an easy hack.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20200225/ab0a5993/attachment.htm>