[Bug 711155] wayland: add wl_drm support to wayland sink

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=711155

--- Comment #60 from George Kiagiadakis <george.kiagiadakis at collabora.com> ---
I've worked a bit on this recently and I've updated my branch with the split
sink approach:

https://git.collabora.com/cgit/user/gkiagia/gst-plugins-bad.git/log/?h=waylandsink-dmabuf

Now wldmabufsink is fully implemented and works, but it is still unclear how to
do proper negotiation of format and allocator.

Currently, the linux_dmabuf v1 protocol that I am using has no way of informing
the client of what requirements it has on the dmabuf that it can import. It can
inform it only about the colour format (although this is not implemented at the
moment), but it cannot inform it about the alignment constraints as well as the
potential hardware constraints (i.e. a certain device may not be able to share
dmabuf with another certain device in some platform maybe?). However, the
protocol does have a way to find out early-ish if a certain configuration is
not going to work. This works like this: you allocate a dmabuf from your
potential allocator, you try to import it to the compositor with the format and
alignment that you want and if the compositor cannot use this, then it sends
back an event saying so. This can be done without creating a surface, so it can
be used as an early negotiation mechanism.

When I started writing wldmabufsink, I was thinking that I could do this in
set_caps() in order to check if the negotiated format is OK (since listing
formats from the compositor is not implemented). However, the problem is that
at that point the allocator is unknown. One solution would be to write a drm
dumb-buffers-based allocator in waylandsink, like the one that Benjamin has
written in his patch, and then allocate a dummy buffer from there with the
negotiated format and see if it works. But this is wrong in many ways. Apart
from adding a dependency to drm and limiting waylandsink for use on drm-based
systems, it would also not really prove anything about the negotiation, because
the compositor may accept dumb buffers and reject v4l2 buffers for example, or
vice versa (in a v4l2{src,dec} ! wldmabufsink scenario).

A better solution would be instead to *negotiate* the allocator. In GStreamer
this is currently not supported. Downstream gets to make a proposal of its
preferred allocator and upstream gets to make a decision, which may be totally
incompatible with what downstream can use (in this specific case of
wldmabufsink, downstream has nothing to propose and upstream gets to select
whatever allocator it wants). Downstream instead needs a way to validate that a
chosen allocator works and upstream needs a way to propose an allocator and get
feedback from downstream on its decision. This could be achieved by extending
the semantics of the existing ALLOCATION query and adding a couple of vfuncs
for them in GstBase* classes, plus some more GstQuery manipulation functions.
Upstream could set one or more allocators on the query before sending it
downstream and downstream could then check that these allocators work. If a
certain allocator doesn't work, downstream should remove it from the list and
maybe also sort the working ones in order of preference. Eventually upstream
would then get a list of allocators that are guaranteed to work, or an empty
list in case nothing works and negotiation has failed.

Now this doesn't fully solve the problem again, because there is still no way
to negotiate the format. But this is something that eventually will be
implemented, I hope, in the wayland linux_dmabuf protocol, so we will be able
to use normal caps to negotiate the format first and then negotiate the
allocator using the mechanism described above. Fingers crossed, though, as I'm
a bit sceptical wrt the reason that this feature is not implemented yet.

Regarding the drm dumb buffers allocator, it would be really nice to have it
around so that something like videotestsrc ! wldmabufsink can work. However, as
I said above, I do not agree with adding a dependency on drm in waylandsink.
Thinking about it, this allocator is a lot like the allocator that I have done
in the "inteldmabufupload" element [1]. The difference is that the inteldmabuf
upload element allocates buffers from GEM, which are GPU buffers meant to be
used by GL internally, and uses the intel-specific API. Dumb buffers are in a
different memory and are cross-GPU compatible. What I would propose there is to
keep wldmabufsink without any allocator and instead require the use of "upload"
elements, like inteldmabufupload. We can have dumbdmabufupload, intel*,
radeon*, nouveau*, ion*, whatever, if there are use cases for them. Wayland
does not really require any specific allocator, it is up to the negotiation
mechanism to decide what to use.

Eventually, with this design, we can have 2 kinds of pipelines:

1) Software source, using mmap()'able dmabuf:

   videotestsrc ! dumbdmabufupload ! wldmabufsink
   filesrc ! qtdemux ! avdec_h264 ! dumbdmabufupload ! wldmabufsink

2) Hardware source, eliminating the need for an upload element:

   v4l2src ! wldmabufsink
   filesrc ! qtdemux ! v4l2dec ! wldmabufsink

And of course things like "videotestsrc ! wldmabufsink" would fail in the
negotiation of the allocator, as the list of acceptable allocators should be
empty in this case.

[1]. https://bugzilla.gnome.org/show_bug.cgi?id=732281

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.