[Libreoffice-commits] core.git: Branch 'libreoffice-6-3' - sw/source

[Stephan Bergmann (via logerrit)]

 sw/source/core/text/frmform.cxx |    9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

New commits:
commit 9df327e2dda452dd557eb6bcc0b6f54cfe1fef68
Author:     Stephan Bergmann <sbergman at redhat.com>
AuthorDate: Thu Jun 27 13:33:27 2019 +0200
Commit:     Michael Stahl <michael.stahl at cib.de>
CommitDate: Mon Dec 23 12:08:26 2019 +0100

    tdf#126127: Make nTmp smaller still, avoid -fsanitize=signed-integer-overflow
    
    ...after f2e3655255db4032738849cd4b77ce67a6e2c984 "Avoid
     -fsanitize=signed-integer-overflow" had already reduced it from using LONG_MAX
    to TWIPS_MAX/2 in the past.  This time, avoid the computation of
    
    > const sal_uInt64 nCurrentDist = sal_Int64(aDiff.getX()) * sal_Int64(aDiff.getX()) + sal_Int64(aDiff.getY()) * sal_Int64(aDiff.getY()); // opt: no sqrt
    
    in GetFrameOfModify (sw/source/core/layout/frmtool.cxx) from overflowing (where
    aDiff.getY() derives from nTmp and can be close to it in magnitude, so computing
    its square would overflow on platforms where TWIPS_MAX is a large sal_Int64
    value).
    
    (The "empirically shown to be large enough in practice" in the comment is a
    successful `make check` on Linux 64-bit with UBSan.)
    
    Change-Id: Ic7f058bd6853ff04ccb50a150509e98f850d12d2
    Reviewed-on: https://gerrit.libreoffice.org/74801
    Reviewed-by: Michael Stahl <michael.stahl at cib.de>
    Tested-by: Jenkins
    (cherry picked from commit 8723ac4e20eda87a82393f2f6c7d28ece8514238)
    Reviewed-on: https://gerrit.libreoffice.org/85733

diff --git a/sw/source/core/text/frmform.cxx b/sw/source/core/text/frmform.cxx
index 492a738941c3..7e7460dd3812 100644
--- a/sw/source/core/text/frmform.cxx
+++ b/sw/source/core/text/frmform.cxx
@@ -835,7 +835,14 @@ bool SwTextFrame::CalcPreps()
                 }
                 else
                 {
-                    SwTwips nTmp  = TWIPS_MAX/2 - (getFrameArea().Top()+10000);
+                    // nTmp should be very large, but not so large as to cause overflow later (e.g.,
+                    // GetFrameOfModify in sw/source/core/layout/frmtool.cxx calculates nCurrentDist
+                    // from, among others, the square of aDiff.getY(), which can be close to nTmp);
+                    // the previously used value TWIPS_MAX/2 (i.e., (LONG_MAX - 1)/2) depended on
+                    // the range of 'long', while the value (SAL_MAX_INT32 - 1)/2 (which matches the
+                    // old value on platforms where 'long' is 'sal_Int32') is empirically shown to
+                    // be large enough in practice even on platforms where 'long' is 'sal_Int64':
+                    SwTwips nTmp  = (SAL_MAX_INT32 - 1)/2 - (getFrameArea().Top()+10000);
                     SwTwips nDiff = nTmp - getFrameArea().Height();
 
                     {