[Poppler-bugs] [Bug 39361] New: Numerical overflow in libopenjpeg JPXStream::doLookChar()

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=39361

           Summary: Numerical overflow in libopenjpeg
                    JPXStream::doLookChar()
           Product: poppler
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: medium
         Component: general
        AssignedTo: poppler-bugs at lists.freedesktop.org
        ReportedBy: daniel-gl at gmx.net


There is a line

unsigned char rc = (unsigned char) ((r >> adjust)+((r >> (adjust-1))%2));

in that function which overflows a component's value if (r >> (adjust - 1)) ==
511.
It is also wrong to execute this at all if adjust == 0.

According to oprofile a lot of time is spent in this function.
See the attached patch for my try to fix and optimize this a little.

-- 
Configure bugmail: https://bugs.freedesktop.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.