[Libreoffice-bugs] [Bug 138360] calculation: basic math fail, !calc rounding wrong! dividing by inverse sometimes different from multiply, 7E10/1E-5 wrong,

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=138360

--- Comment #4 from Eike Rathke <erack at redhat.com> ---
(In reply to b. from comment #2)
> 0,5 ULP precision at
> granularity borders ... produces overlapping 'definition ranges'? as e.g.
> 0,5 ULP of 2^53 reaches 'down' to including '2^53 - 1'
Say what? I don't understand what you are trying to tell.

> while 2^53-1 itself
> covers the range from 2^53-1-0,5 to 2^53-1+0,4999~, 
It does not cover a range. In an IEEE-754 binary64 magnitude with a distance
precision of 1, i.e. values in the interval [2^52,2^53) are integers, any value
or result that would lie in between integer values is tied to the nearest
representable value. There is no range between two integer values in that
interval.

Btw, it would help people and specifically developers reading (and maybe
trying) your examples if you used '.' decimal point instead of ',' decimal
comma.

> that leads to some
> ambiguity as e.g. 2^53-1+0,4 is falling into both 'ULP-precision-ranges', is
> that handeled in a meaningful way?
What both "ULP-precision-ranges"? 2^53-1+0.4 results in 2^53-1 as that is the
nearest representable value. There is no ambiguity.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201127/e47cadde/attachment.htm>