[Libreoffice-bugs] [Bug 139173] New: Incorrect bounds on formula DEC2HEX

[bugzilla-daemon at bugs.documentfoundation.org]

https://bugs.documentfoundation.org/show_bug.cgi?id=139173

            Bug ID: 139173
           Summary: Incorrect bounds on formula DEC2HEX
           Product: LibreOffice
           Version: 6.4.7.2 release
          Hardware: All
                OS: Linux (All)
            Status: UNCONFIRMED
          Severity: minor
          Priority: medium
         Component: Calc
          Assignee: libreoffice-bugs at lists.freedesktop.org
          Reporter: cameron.pinnegar at gmail.com

Description:
In MS Excel the formulae DEC2BIN, DEC2OCT, and DEC2HEX have valid inputs as
integers m in the range -2^n <= m < 2^n, where n is given by

DEC2BIN: n=9
DEC2OCT: n=29
DEC2HEX: n=39

LibreOffice Calc gets this right for DEC2BIN and DEC2OCT, but not DEC2HEX.
There, the valid range is erroneously: -2^39 - 1 <= m < 2^39 + 1.

In other words, in Excel we have:

=DEC2HEX(2^39) == #NUM!
=DEC2HEX(-2^39 - 1) == #NUM!

But in LOCalc:

=DEC2HEX(2^39) == "8000000000"
=DEC2HEX(-2^39 - 1) == "7FFFFFFFFF"

This is surprising and does not seem like intended behaviour. I have not yet
investigated whether the other base conversion functions have inconsistent
bounds as well.

Steps to Reproduce:
In a cell do =DEC2HEX(2^39) or DEC2HEX(-2^39 - 1)

Actual Results:
Answers are "8000000000" and "7FFFFFFFFF" respectively.

Expected Results:
Answers should be #NUM! in both cases, in accordance with Excel.


Reproducible: Always


User Profile Reset: No



Additional Info:
OS: Manjaro Linux
LibreOffice Calc version: 6.4.7.2

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice-bugs/attachments/20201223/03a9c801/attachment-0001.htm>