[gstreamer-bugs] [Bug 576997] New: floating point stack corruption : emms_c macro not working in ffdeinterlace

[GStreamer (bugzilla.gnome.org)]

If you have any questions why you received this email, please see the text at
the end of this email. Replies to this email are NOT read, please see the text
at the end of this email. You can add comments to this bug at:
  http://bugzilla.gnome.org/show_bug.cgi?id=576997

  GStreamer | gst-ffmpeg | Ver: 0.10.5
           Summary: floating point stack corruption : emms_c macro not
                    working in ffdeinterlace
           Product: GStreamer
           Version: 0.10.5
          Platform: Other
        OS/Version: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: gst-ffmpeg
        AssignedTo: gstreamer-bugs at lists.sourceforge.net
        ReportedBy: ivanohe at abacom.com
         QAContact: gstreamer-bugs at lists.sourceforge.net
     GNOME version: Unspecified
   GNOME milestone: Unspecified


Hi,

I spent lots of time tracking a voodoo issue on one of our encoder on a 64 bit
machine : There was a big floating point error once when the encoder was
compiled with icc. Doing the calculation once (to get the error) and again will
give correct result afterward...

By compiling with icc using the -fp-stack-check, I got a crash in my encoder at
the place of the floating point calculation error. This means that the floating
point context was not clean at this point. 

After lots of trial, I found the cause of the problem : ffdeinterlace.

There is some mmx instruction in the deinterlace code. Normally, the emms_c()
macro will take care of calling emms() function to clear the state of the
floating point unit. However, the emms_c() macro expect the global mm_flags to
be set properly (mmx support) or else, the macro does nothing.

I also see that this problem occur in the current ffmpeg (2009-03-24) when
using a deinterlace in a raw to raw scenario like this : 

./ffmpeg_g -i test.avi -vcodec rawvideo -an -deinterlace out2.avi

using a compress input file or choosing an encoder will call the dsputil_init()
function and properly set the mm_flags.

I made a patch to ffdeinterlace that fix the issue for me. If there is a fix in
ffmpeg, my patch might not be need after that point.

I am using gst-ffmpeg 0.10.5 but since the problem is cause by ffmpeg code,
this patch still apply for latest gst-ffmpeg


-- 
See http://bugzilla.gnome.org/page.cgi?id=email.html for more info about why you received
this email, why you can't respond via email, how to stop receiving
emails (or reduce the number you receive), and how to contact someone
if you are having problems with the system.

You can add comments to this bug at http://bugzilla.gnome.org/show_bug.cgi?id=576997.