[global-libreoffice-ci] UBSAN Linux Build - Build # 1738 - Failure!

[Stephan Bergmann]

On 25/08/2020 22:16, Eike Rathke wrote:
> On Tuesday, 2020-08-25 13:57:02 +0200, Stephan Bergmann wrote:
>> On 25/08/2020 04:26, ci at libreoffice.org wrote:
>>> /sc/source/filter/excel/xlescher.cxx:116:43: runtime error: -764.94 is outside the range of representable values of type 'unsigned int'
>>>      #0 0x2b30c609bf3b in (anonymous namespace)::lclGetRowFromY(ScDocument const&, short, unsigned int&, unsigned int&, unsigned int, unsigned int, long&, long, double) /sc/source/filter/excel/xlescher.cxx:116:43
>>>      #1 0x2b30c609b1ff in XclObjAnchor::SetRect(XclRoot const&, short, tools::Rectangle const&, MapUnit) /sc/source/filter/excel/xlescher.cxx:174:5
>>>      #2 0x2b30c4e4c027 in XclExpDffSheetAnchor::ImplCalcAnchorRect(tools::Rectangle const&, MapUnit) /sc/source/filter/excel/xeescher.cxx:308:14
>>>      #3 0x2b30c4e4b4f4 in XclExpDffAnchorBase::WriteData(EscherEx&, tools::Rectangle const&) /sc/source/filter/excel/xeescher.cxx:277:5
>>>      #4 0x2b306e07562c in EscherExHostAppData::WriteClientAnchor(EscherEx&, tools::Rectangle const&) /include/filter/msfilter/escherex.hxx:958:51
>>>      #5 0x2b306e14480f in ImplEESdrWriter::ImplWriteShape(ImplEESdrObject&, EscherSolverContainer&, bool) /filter/source/msfilter/eschesdo.cxx:677:28
>>> [...]
>>
>> This is apparently caused by <https://git.libreoffice.org/core/+/5e8875780d665b7ae0fee1a053b5ce78ec513f69%5E!/>
>> "tdf#135828 XLSX shape export: fix distortion of rotated shapes" adding
>> ScExportTest::testTdf135828_Shape_Rect, but it may well be a pre-existing
>> issue that only happens to get triggered by that test.
>>
>> Debugging this a bit, I found that negative values start to appear in
>> ImplEESdrWriter::ImplWriteShape (filter/source/msfilter/eschesdo.cxx, frame
>> #5), where
>>
>>>          if( rObj.GetAngle() )
>>>              ImplFlipBoundingBox( rObj, aPropOpt );
>>
>> (lines 665--666) turns rObj.GetRect() from 463x3130@(3051,87) to
>> 463x3130@(4396,-898), as rObj.GetAngle() returns 28306.  That -898 then gets
>> passed down to lclGetRowFromY (sc/source/filter/excel/xlescher.cxx, frame
>> #0) as nY.  No idea where and how to fix things, though.
> 
> As I mentioned on IRC today, my understanding is that drawing layer
> positions should not be negative, at least not for Calc (eported to BIFF
> or OOXML), so that flipped Rectangle would already be wrong; but I'm not
> certain about the origin or what actually should be done here instead.

<https://gerrit.libreoffice.org/c/core/+/101979> "Avoid UBSan negative 
double -> unsigned int conversion", for better or worse