[Libreoffice] [REVIEW] fdo#37668 ServerFont::GetGlyphData crash proposed fix for 3-4

[Caolán McNamara]

So, for https://bugs.freedesktop.org/show_bug.cgi?id=37668 
and https://bugzilla.redhat.com/show_bug.cgi?id=709503 there is a crash
in ServerFont::GetGlyphData, apparently only on 32bit platforms.

I can't reproduce this crash, but I can theorize how its happening. From
https://bugzilla.redhat.com/attachment.cgi?id=502106 I can see that its
happening on nFallbackLevel=9 in OutputDevice::forceFallbackFontToFit

Now, the code is ...

long nLGlyph = ...
nLGlyph |= (nFallbackLevel << GF_FONTSHIFT)


GetGlyphBoundRect(long nGlyphIndex) {
...
    int nLevel = nGlyphIndex >> GF_FONTSHIFT;
...
    ServerFont* pSF = m_pServerFont[ nLevel ];
...

    nGlyphIndex &= GF_IDXMASK;
}

where GF_FONTSHIFT is 24. i.e. shift the fallback level up to some high
bits, then shift it down again. So on 32bit platforms, 9 left shifted 24
bits into a long gives a value where the sign bit is on!, so right
shifting it again retains a negative value, its signed extended. On
64bit we don't get as far as the sign bit, so we get away with it.

So, lets change long int to the unsigned int typedef sal_Glyphid, the
code in forceFallbackFontToFit was pasted in from GetGlyphOutline so
change that as well, and then propagate that through the different
platform backends, including some apparently missing masks on the
glyphid.

http://cgit.freedesktop.org/libreoffice/libs-gui/commit/?id=0adcbfa45f1c387d35acdc34c4a945dc88fecead

C.