[Bug 746247] New: div255 implementation is incorrect

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=746247

            Bug ID: 746247
           Summary: div255 implementation is incorrect
    Classification: Platform
           Product: GStreamer
           Version: git master
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: Normal
         Component: orc
          Assignee: gstreamer-bugs at lists.freedesktop.org
          Reporter: thaytan at noraisin.net
        QA Contact: gstreamer-bugs at lists.freedesktop.org
                CC: ds at schleef.org
     GNOME version: ---

Created attachment 299452
  --> https://bugzilla.gnome.org/attachment.cgi?id=299452&action=edit
test div255 app

div255 implements this algorithm everywhere:
  d = (s + 128 + (s+128)>>8) >> 8
which produces a result that is off-by-one for roughly half the 0..65535 input
range.

A correct implementation is:
  d = (s + 1 + (s >> 8)) >> 8

Test python app, and a fix which implements the new algorithm attached.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.