[Bug 103417] New: gstreamer nvh264enc deadlock and crashed

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=103417

            Bug ID: 103417
           Summary: gstreamer nvh264enc deadlock and crashed
           Product: Spice
           Version: unspecified
          Hardware: Other
                OS: All
            Status: NEW
          Severity: normal
          Priority: medium
         Component: server
          Assignee: spice-bugs at lists.freedesktop.org
          Reporter: bochuan.ding at bridgetek.cn

Spice server version 0.13.90
Host OS: Ubuntu 16.04
Guest OS: Windows 7

For hardware accelerate, I use GStreamer nvh264enc in place of xh264enc.
However, it often get deadlock when GStreamer get restarted. I debugged the
gstreamer plugin nvenc and I find the problem in the free_pipeline() and
create_pipeline()。 Because there is no mutex lock to protect the process
create/free pipeline，when the pipeline status switched in a short time,e.g
start -> stop -> start, the nvenc would be in unknown status. In my case, it
causes the deadlock inside the nvenc plugin.

I try to fix this bug in a simple way. In the configure_pipeline(), call
free_pipeline() before create_pipeline(). This will make sure that we do free
the resource of pipeline and it's safe even if the encoder->pipeline is NULL.

The source code is below. Here is the only way to call create_pipeline() except
the gstreamer_encoder_new(), but it's like a initialized constructor, so we
just take care about the configure_pipeline() which takes responsibility for
switching the status of the pipeline.

static gboolean configure_pipeline(SpiceGstEncoder *encoder)
{
    if (!encoder->set_pipeline) {
        return TRUE;
    }

    free_pipeline(encoder->pipeline);

    if (!encoder->pipeline && !create_pipeline(encoder)) {
        return FALSE;
    }
    ...
}


I can't guarantee that it doesn't bring in other bugs. I think there should be
a more grace method to fix this problem.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/spice-bugs/attachments/20171023/eb335cc2/attachment.html>