[Bug 749088] New: vtenc: VTCompressionSessionEncodeFrame failing with vt_status == noErr -> memory leak

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=749088

            Bug ID: 749088
           Summary: vtenc: VTCompressionSessionEncodeFrame failing with
                    vt_status == noErr -> memory leak
    Classification: Platform
           Product: GStreamer
           Version: git master
                OS: Mac OS
            Status: NEW
          Severity: normal
          Priority: Normal
         Component: gst-plugins-bad
          Assignee: gstreamer-bugs at lists.freedesktop.org
          Reporter: ilya.konstantinov at gmail.com
        QA Contact: gstreamer-bugs at lists.freedesktop.org
     GNOME version: ---

vtenc currently assumes that either:
1. gst_vtenc_encode_frame is called, or
2. VTCompressionSessionEncodeFrame returns failure code

In fact, in some cases it's neither -- for example, when this cryptic error is
print to the console:

vtCompressionSessionTrackFrameExitedEncoder signalled err=-12902
(kVTParameterErr) (Unrecognised VTVideoEncoderFrame token passed in by encoder)
at
/SourceCache/CoreMedia_frameworks/CoreMedia-1562.235/Sources/VideoToolbox/VTCompressionSession.c
line 2222
<<<< GVAVideoEncoder >>>> GVAVideoEncoder_EmitEncodedFrame: AVF encoder error:
VT, GVAVideoEncoder_EmitEncodedFrame: outputFrameInfoPtr ==NULL

While we certainly shouldn't deal with cryptic errors, we also shouldn't make
things worse. Right now, in such cases we don't release frames at all, and
memory keeps growing indefinitely.

The solution is to release the data regardless. We must invoke
gst_video_encoder_finish_frame, since it's the only way to make GstVideoEncoder
release its reference to the frame.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.