<div dir="ltr"><div>Winfried, to have an external reference I entered your input in R (64-bit), open source<br></div>statistical SW, R confirms your math. <b>Calc is wrong</b>. Below is from R-commandline:<br><div><br><br><div><table class="gmail-GCWXI2KCCKB gmail-ace_text-layer gmail-ace_line gmail-GCWXI2KCAU" style="font-family:"lucida console";font-size:10pt;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;padding-left:6px;padding-bottom:8px;border-width:medium;border-style:none;border-color:-moz-use-text-color;outline:medium none;word-wrap:break-word;white-space:pre-wrap;color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;background-color:rgb(255,255,255);width:866px" cellpadding="0" cellspacing="0"><tbody><tr><td style="font-family:"lucida console";line-height:1.2;font-size:10pt;vertical-align:top" align="left"><pre tabindex="0" class="gmail-GCWXI2KCJKB" id="gmail-rstudio_console_output" style="font-family:"lucida console";font-size:10pt;outline:medium none;border-width:medium;border-style:none;border-color:-moz-use-text-color;word-break:break-all;margin:0px;white-space:pre-wrap;line-height:1.2"><span class="gmail-GCWXI2KCPKB gmail-ace_keyword" style="color:blue;white-space:pre">> </span><span class="gmail-GCWXI2KCPJB gmail-ace_keyword" style="color:blue">(3^31)-((3^31)-1)
</span>[1] 1
<span class="gmail-GCWXI2KCPKB gmail-ace_keyword" style="color:blue;white-space:pre">> </span><span class="gmail-GCWXI2KCPJB gmail-ace_keyword" style="color:blue">(3^31)-((3^31)-2)
</span>[1] 2
<span class="gmail-GCWXI2KCPKB gmail-ace_keyword" style="color:blue;white-space:pre">> </span><span class="gmail-GCWXI2KCPJB gmail-ace_keyword" style="color:blue">(3^31)-((3^31)-3)
</span>[1] 3
</pre></td></tr><tr><td style="font-family:"lucida console";line-height:1.2;font-size:10pt;vertical-align:top" align="left"><br></td></tr><tr><td style="font-family:"lucida console";line-height:1.2;font-size:10pt;vertical-align:top" align="left"><table style="width:866px" cellpadding="0" cellspacing="0"><tbody><tr><td rowspan="1" style="font-family:"lucida console";line-height:1.2;font-size:10pt;vertical-align:top" align="left" height="" width="1"><div class="gmail-GCWXI2KCPKB gmail-ace_keyword" style="color:blue;white-space:pre">> </div></td></tr></tbody></table></td></tr></tbody></table><br>Below is whtat R states about precision:<br><br>All <span style="font-family:courier new,courier;color:rgb(102,102,102)"><b>R</b></span> platforms are required to work with values conforming to the
IEC 60559 (also known as IEEE 754) standard. This basically works
with a precision of 53 bits, and represents to that precision a range
of absolute values from about <i>2e-308</i> to
<i>2e+308</i>. It also has special values
<code><a href="https://stat.ethz.ch/R-manual/R-devel/library/base/html/is.finite.html">NaN</a></code> (many of them), plus and minus infinity and plus and
minus zero (although <span style="font-family:courier new,courier;color:rgb(102,102,102)"><b>R</b></span> acts as if these are the same). There are
also <em>denormal(ized)</em> (or <em>subnormal</em>) numbers with absolute
values above or below the range given above but represented to less
precision.
<br><br><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-09-28 13:15 GMT+02:00 Winfried Donkers <span dir="ltr"><<a href="mailto:W.Donkers@dci-electronics.nl" target="_blank">W.Donkers@dci-electronics.nl</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I'm working on some improvements of Calc's MOD function, but I struggle with some unexpected results when using large integer values.<br>
As I understand IEEE754, double precision means a mantissa of 53 bits, so the maximum correct unsigned integer value would be 2^53.<br>
<br>
But when I try 3^31 (which is less than 2^52) and subtract 3^31-1, 3^31-2, 3^31-3 (resp. 1, 2 and 3 less than 3^31) I don't get 1, 2 and 3 as result, but 0, 0 and 3.<br>
This test is simply in Calc with e.g. =(3^31)-((3^31)-1).<br>
I use a 64bit machine and master-version of LO.<br>
<br>
What am I doing wrong here?<br>
<br>
Winfried<br>
<br>
<br>
______________________________<wbr>_________________<br>
LibreOffice mailing list<br>
<a href="mailto:LibreOffice@lists.freedesktop.org">LibreOffice@lists.freedesktop.<wbr>org</a><br>
<a href="https://lists.freedesktop.org/mailman/listinfo/libreoffice" rel="noreferrer" target="_blank">https://lists.freedesktop.org/<wbr>mailman/listinfo/libreoffice</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Luc Castermans<br>mailto:<a href="mailto:luc.castermans@gmail.com" target="_blank">luc.castermans@gmail.com</a></div>
</div>