Decimal-Fidelity in Calc Interchange (was RE: Difficulties with Flat XML under source control)

[Dennis E. Hamilton]

Thanks to the laws of serendipity, I just ran into the paper by Michel Hack on "On Intermediate Precision Requirements for Correctly-Rounding Decimal-to-Binary Floating-Point Conversion."  The PDF is one of the papers available at <http://www.informatik.uni-trier.de/Reports/TR-08-2004/>.  The references are all valuable, with citation of the work of David Matula and others.  I recall that Guy Steele looked at this issue in the work on Common Lisp, but I no longer have source information.  I suspect that Java and Fortress work would address this, especially because Java specifies IEEE binary as the internal form.

There may be other papers in this family that are of interest in taking the bumps out of the use of decimal floating-point values in the persistent form of cell values in spreadsheet documents and in formulas having decimal-notation literal values, especially where the internal use is via conversion to and then from a decimal-incommensurate arithmetic.

I am changing the subject so this thread is recognized for where it has wandered. I've also stitched the part related to number drift and conversion fidelity back in.

 - Dennis

-----Original Message-----
From: Dennis E. Hamilton [mailto:dennis.hamilton at acm.org] 
Sent: Wednesday, June 20, 2012 10:57
To: 'Thorsten Behrens'
Cc: 'libreoffice-dev'
Subject: RE: Difficulties with Flat XML under source control

It occurs to me that Postscript and PDF have dealt with this for imaging models that work consistently.  Here, the "in" is to a renderer, but the model for representation of decimal expressions of find-sensitivity values seems to have been handled (for years).  Those specifications may be some help too.

 - Dennis

-----Original Message-----
From: Thorsten [mailto:netsroth at googlemail.com] On Behalf Of Thorsten Behrens
Sent: Wednesday, June 20, 2012 06:32
To: Dennis E. Hamilton
Cc: 'libreoffice-dev'
Subject: Re: Difficulties with Flat XML under source control

Dennis E. Hamilton wrote:
> For out-in (which this is, presumably), you want to record a
> decimal expression of the internal value that will convert back to
> the exact internal value on re-input.  (The in-out case is that
> the input conversion provide whatever internal representation that
> will convert to the read value on re-output.  Without additional
> information, it is generally very difficult to have these be the
> same.)
> 
> It is also desirable, of course, that any other ODF consumer use
> the same technique so that its in-out conversion satisfies the
> out-in condition of the original source of the decimal expression
> of the value.  
>
Hi Dennis,

yes - but in a first approximation, one can probably relax this a
bit (for the use case at hand): only _after_ the first save
operation this needs to hold. Also, most people would probably be
contempt with this to work for *one* ODF editing application.

> It is also desirable, of course, that any other ODF consumer use
> the same technique so that its in-out conversion satisfies the
> out-in condition of the original source of the decimal expression
> of the value.  
> 
Note that there's a difference between spreadsheet values (for which
I think de facto the above holds true - likely everyone stores those
in IEEE doubles), and other content: consumers might employ rather
complex transformations to arrive at internal values, given e.g. a
gradient center coordinate - asking for common behaviour is very
close to asking for a common ODF application model.

Cheers,

-- Thorsten

-----Original Message-----
From: libreoffice-bounces+dennis.hamilton=acm.org at lists.freedesktop.org [mailto:libreoffice-bounces+dennis.hamilton=acm.org at lists.freedesktop.org] On Behalf Of Thorsten Behrens
Sent: Wednesday, June 20, 2012 05:49
To: Johannes Sixt
Cc: libreoffice-dev
Subject: Re: Difficulties with Flat XML under source control

Johannes Sixt wrote:
> >> - Measurements change. E.g. (just to pick one case), in
> >> <style:graphic-properties> the draw:visible-area-width changes from
> >> 6.088cm to 6.089cm. Is there a remedy to avoid changes of this kind?
> > 
> > 	Ah; nasty, some rounding problem / internal representation issue -
> > possibly again looking at the code we could do better here to make it
> > more predictable; possibly using more precision we could do better
> > (doubles instead of floats) ?
> 
> Probably. Looking at this again, these changes seem to happen only for
> draw:visible-area-*. Hence, it may also be a matter of conversion
> between screen dimensions (pixels?) and cm/mm/in/etc.
> 
Hrm, yeah - and we *really* don't want this slow drift - any chance
you can file a bug with a preferrably small sample doc?

Thanks,

-- Thorsten