[Libreoffice-bugs] [Bug 138595] not a [calc ROUND() precision bug] but: calculation with day-time-fractions imprecise as 8 hours are represented by 1/3, thus 0.3333333333333333 with a small roundoff error, (and other fp-imprecisions?)

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=138595

b. <newbie-02 at gmx.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |NOTABUG
             Status|UNCONFIRMED                 |RESOLVED
            Summary|calc ROUND() precision bug  |not a [calc ROUND()
                   |                            |precision bug] but:
                   |                            |calculation with
                   |                            |day-time-fractions
                   |                            |imprecise as 8 hours are
                   |                            |represented by 1/3, thus
                   |                            |0.3333333333333333 with a
                   |                            |small roundoff error, (and
                   |                            |other fp-imprecisions?)

--- Comment #9 from b. <newbie-02 at gmx.de> ---
ok, the problem (your error) has at least three stages: 

1.) the values you propagated as 'exact numbers' to be calculated with are not
'exact' numbers but in themself computed from other numbers (as it already is
explained in 'ask'), 

2.) numbers you see on screen are - if computed - not! (mostly not) an exact
decimal representation of what the computer is calculating with, but something
'prettyfied' for better acceptance by humans, 

3.) somebody sometimes decided to calculate time values in calc (as in ex$el
and in 1-2-3? before) in a manner that '1' represents one day, and thus 8 hours
are represented by a third (8/24) of 1, '0,333333333333~', a number already
difficult for decimal calculations, becoming even more difficult with limited
count of digits and not! becoming 'better' in conversion to binary, thus each
of your seven 'eights' accumulated contains a litte roundoff error of about a 3
in the 17th position, that's hidden from you as a user to avoid some
irritations, but produces others as you see, 

in detail your calculation in cell C38 is '=1/3*7*408/8*24', shown to you as
'2856', but for the computer internally about '-0,00000000000045474735' 'off',
and thus something with 2855,9999999999995452...', what causes irritations in
rounding and downstream calculations, 

you can check it with '=RAWSUBTRACT(1/3*7*408/8*24;2856)'

ways to avoid: 

- calculate with 'real' numbers for hours instead of 'time formatted day
fractions', '=8*7*408/8' shouldn't have such problems, 

- let the people work 6 hours a day, 6/24 has less problems than 8/24, 

- round your results at well selected steps to well selected precision,
'=ROUND((1/3*7)*408/8*24;2)' - you expect no 100'th of TB? - would hold, 

setting 'notabug', hope you agree,

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201208/c9470019/attachment.htm>