tests sc_array_functions_test fail due to floating point precision

[Milton Vandersloot]

Hi Eike

On December 7, 2017 at 12:40 PM Eike Rathke wrote:
> On Saturday, 2017-12-02 02:47:03 -0500, Milton Vandersloot wrote:
> > I've also opened the other test documents in
> > sc/qa/unit/data/functions/array/fods/ using LibreOffice. The documents which
> > fail are linest.fods and also logest.fods. The others succeed.
>
>If your build completed ignoring these tests (i.e. produced a
>usable office under ./instdir/), could you please open the
>failing documents and tell us the cell addresses on the second
>sheet where tests fail and the actual values they produced?

Below are the numbers. Some remarks: Many errors are in the
range of 1E-11 to 1E-12 but some are way larger >1E30. I assume
this is due some division of a very small error.

Has someone done a/some theoretical error analysis of the
algorithms involved? (e.g. numeric (forward/backward) stability
and error estimates similar to what is well-known for LU
factorization and other algorithms.)

Who/What provided the reference values -- or reformulated --
what error do these numbers have (wrt to the mathematical
solution).

Best
Milton


Below are the first three columns (A-C). They are prefixed by the
line number (e.g. 'l21'). At the end I added the formula of the
TRUE/FALSE column. I included this as some tests round to
something different than 1E-12, e.g. line 5 in linest.fods.


logest.fods Sheet 2:
--------------------

l21: 538.89039831558	538.890398315583	FALSE	(=ROUND(A21,12)=ROUND(B21,12))
off by 3E-12



linest.fods Sheet 2:
--------------------

l39: 12237.3616028624	12237.3616028623	FALSE	(=ROUND(A39,12)=ROUND(B39,12))
off by 1E-10

l46: 1.15420554120705E+030	5.5523702096735E+030	FALSE	(=ROUND(A46,12)=ROUND(B46,12))
off by 4.4E30

l66: 1.36570269395312E+030	3.73790456956461E+029	FALSE	(=ROUND(A66,12)=ROUND(B66,12))
off by 1.E30

l99: 2029957.60481663	2029957.60481605	FALSE	(=ROUND(A99,12)=ROUND(B99,12))
off by 6.E-7

l113: 1.11356775201247E+027	7.3824164159995E+026	FALSE	(=ROUND(A113,12)=ROUND(B113,12))
off by 4E26

l156: 9942283.62964539	9942283.62965002	FALSE	(=ROUND(A156,12)=ROUND(B156,12))
off by 5E-6
l157: 11577901.4177444	11577901.4177444	FALSE	(=ROUND(A157,12)=ROUND(B157,12))
off by 0. This keeps me puzzling ... (increasing the number of significant digits yields zeros on both entries)

l163: -108.083609022555	-108.083609022579	FALSE	(=ROUND(A163,12)=ROUND(B163,12))
off by 2.4E-11
l164: 34.8435504725302	34.8435504725221	FALSE	(=ROUND(A164,12)=ROUND(B164,12))
off by 1.2E-11

l171: 2484916.49382756	2484916.49382742	FALSE	(=ROUND(A171,12)=ROUND(B171,12))
off by 1.4E-6

l184: 2.7060018272122E-14	9.25081260404044E-13	FALSE	(=ROUND(A184,12)=ROUND(B184,12))
off by 9E-13

l186: 4.04703891013103E+033	3.46285083263994E+030	FALSE	(=ROUND(A186,12)=ROUND(B186,12))
off by 4E33

l189: 5.41123064912622E-14	1.84989825908131E-12	FALSE	(=ROUND(A189,12)=ROUND(B189,12))
off by 1.8E-12

l201: 786024.499999976	786024.50000001	FALSE	(=ROUND(A201,12)=ROUND(B201,12))
off by 4E-8

l216: 786024.499999976	786024.50000001	FALSE	(=ROUND(A216,12)=ROUND(B216,12))
off by 4E-8