[poppler] pdftotext needs support for surrogates outside the BMP plane
jonathan_kew at sil.org
Mon Jun 2 14:07:01 PDT 2008
On 2 Jun 2008, at 9:51 pm, Ross Moore wrote:
> Q: How do I convert an unpaired UTF-16 surrogate to UTF-8?
> A different issue arises if an unpaired surrogate is encountered when
> converting ill-formed UTF-16 data. By represented such an unpaired
> surrogate on its own as a 3-byte sequence, the resulting UTF-8 data
> stream would become ill-formed. While it faithfully reflects the
> nature of the input, Unicode conformance requires that encoding form
> conversion always results in valid data stream. Therefore a converter
> must treat this as an error. [AF]
> So yes, throwing an error is recommended; but, IMHO dropping the
> character and continuing as far as possible would be a friendly thing
> to do, as part of how the error is presented.
Simply dropping it is a bad thing; replacing it with U+FFFD
REPLACEMENT CHARACTER is better.
> Now here's my concern about that conversion formula:
> Unicode uu = (u[i] & 0x3ff) << 10 | (u[i+1] & 0x3ff) | 0x10000;
> With | being bitwise 'or', doesn't this convert <d840 dc00> to
> 0x10000 when the correct result is 0x20000 ?
You're right. Good catch; I should have noticed that too.
> Thus this formula works correctly for Plane 1 characters only, and
> not for higher planes.
> A surrogate pair denotes the code point
> 10000 + (H - D800 ) × 400 + (L - DC00)
> where H and L are the hex values of the high and low surrogates
> Doesn't this translate into the following ?
> Unicode uu = ((u[i] & 0x7ff) << 10 ) + (u[i+1] & 0x3ff) +
The expression (u[i] & 0x3ff) in the original is fine; the only
problem is that it should ADD the 0x10000, not OR it with the rest of
the value. (The 0x400 bit can never be set on a high surrogate; if it
were, it would have been outside the range D800..DBFF.)
More information about the poppler