[Bug 794306] vaapih264enc caps negotiation is broken by default

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=794306

--- Comment #19 from Víctor Manuel Jáquez Leal <vjaquez at igalia.com> ---
Created attachment 372563
  --> https://bugzilla.gnome.org/attachment.cgi?id=372563&action=edit
vaapiencode: h264: find profile in available and allowed caps

The commit 67e33d3de225d0e006d7bf606e7abb20d4544eab ("vaapiencode: h264:
find best profile in those available") changed the code to pick a profile
that is actually supported by the hardware. Unfortunately it dropped the
downstream constraints. This can cause negotiation failures under certain
circumstances.

The fix is split in two cases:

1\ the available VA-API caps doesn't intersect with pipeline's allowed
caps:
 * The best allowed profile (pipeline's caps) is set as the encoding
   target profile (it will be adjusted later by the available profiles
   and properties)

2\ the available VA-API caps does intersect with pipeline's allowed
caps:
 * The intersected caps are fixed, and its profile is set as the
   encoding target profile. In this case the is not the best profile,
   but the minimal one (if VA-API reports the profiles in order).
   Setting the minimal profile of the intersected caps is better for
   compatibility.

This patch fixes other tests related with caps negotiation, for
example, it handles baseline profile, even when VA only supports
constrained-baseline.

Original-patch-by: Michael Olbrich <m.olbrich at pengutronix.de>

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.