[Bug 778496] vtenc: should support GLMemory

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=778496

--- Comment #5 from Nick Kallen <socialmediamaster9000 at gmail.com> ---
This latest patch is non-trivial and although it works correctly I wanted to
get feedback on implementation details.

Firstly, the old code was extremely deadlock-prone, look at code like this:

  /* We need to unlock the stream lock here because
   * it can wait for gst_vtenc_enqueue_buffer() to
   * handle a buffer... which will take the stream
   * lock from another thread and then deadlock */
  GST_VIDEO_ENCODER_STREAM_UNLOCK (self);
  vt_status =
      VTCompressionSessionCompleteFrames (self->session,
      kCMTimePositiveInfinity);

The main problem is the various VTC* methods may or may not block, but they
usually execute code asynchronously. And the callback in this case always
called gst_video_encoder_get_frame, which required the video encoder stream
lock.

I haven't removed these UNLOCKs in this commit yet, just to be careful, but I'm
pretty sure this removes all possibility of deadlock. But it relies on passing
frame pointers between threads -- I wanted to double check that's safe. Seems
like it should be, but it's surprising the old code didn't do that in the first
place.

--

The resizing-related __unfix_resolution code is elaborate but turned out to be
necessary to get caps to agree. The pipelines I have in mind are:

 uridecodebin ! vtenc ! video/x-h264,width=x,height=y

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.