[Bug 684981] Pipeline hangs on PREROLLING negotiating caps

[GStreamer (bugzilla.gnome.org)]

https://bugzilla.gnome.org/show_bug.cgi?id=684981
  GStreamer | gstreamer (core) | 1.0.0

--- Comment #1 from Andre Moreira Magalhaes <andrunko at gmail.com> 2012-09-27 17:23:40 UTC ---
Created an attachment (id=225274)
 View: https://bugzilla.gnome.org/attachment.cgi?id=225274
 Review: https://bugzilla.gnome.org/review?bug=684981&attachment=225274

Do not hang on caps intersect when the caps have duplicated entries

After some investigation, the issue happens when caps with duplicated entries
are intersected. gst_caps_intersect_full will be called from gst_base_transform
and will hang on gst_structure_intersect, resulting in a caps with many
duplicated lists inside lists entries.

This happened cause the gamma element set it caps tempĺate to    
GST_STATIC_CAPS (GST_VIDEO_CAPS_MAKE ("{ AYUV, "
            "ARGB, BGRA, ABGR, RGBA, ABGR, RGBA, Y444, "
            "xRGB, RGBx, xBGR, BGRx, RGB, BGR, Y42B, NV12, "
            "NV21, YUY2, UYVY, YVYU, I420, YV12, IYUV, Y41B }"))
where RGBA and ABGR are duplicated entries. 

After caps negotiation finally finishes, the result is:
video/x-raw, format=(string){ I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB,
xBGR, { RGBA, RGBA, { RGBA, RGBA }, { RGBA, RGBA, { RGBA, RGBA } }, { RGBA,
RGBA, { RGBA, RGBA }, { RGBA, RGBA, { RGBA, RGBA } } }, { RGBA, RGBA, { RGBA,
RGBA }, { RGBA, RGBA, { RGBA, RGBA } }, { RGBA, RGBA, { RGBA, RGBA }, { RGBA,
RGBA, { RGBA, RGBA } } } }, { RGBA, RGBA, { RGBA, RGBA }, { RGBA, RGBA,...

Fixing gstgamma also fixes the issue, but a more generic fix is attached.

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.