Full-motion zero-copy screen capture in Weston

[Hoosier, Matt]

> -----Original Message-----
> From: Pekka Paalanen <pekka.paalanen at collabora.com>
> Sent: Wednesday, June 12, 2024 4:03 AM
> To: Hoosier, Matt <Matt.Hoosier at garmin.com>
> Cc: sir at cmpwn.com; contact at emersion.fr; Marius Vlad
> <marius.vlad at collabora.com>; wayland-devel at lists.freedesktop.org; Daniel
> Stone <daniel at fooishbar.org>
> Subject: Re: Full-motion zero-copy screen capture in Weston
> 
> On Mon, 10 Jun 2024 14:18:39 +0000
> "Hoosier, Matt" <Matt.Hoosier at garmin.com> wrote:
> 
> > Yes, the native Linux driver for the hardware still does end up being
> > responsible for one or more planes.
> >
> > Other than trying to arrange the pieces so that there's some API that
> > offers the client an option to guarantee that the source of the screen
> > capture involves the writeback connector (similar to what you've done
> > with weston_output_capture), I don't really think it would be a good
> > idea to make Weston explicitly aware of any of this funny hypervisor
> > business going on.
> >
> 
> I very much agree.

Ack


> 
> > The title on this email conversation was probably poorly chosen, in
> > retrospect. I'm not so much concerned with keeping absolutely to a
> > zero-copy mechanism as I am to using hardware mechanisms (GL
> > rendering, DRI writeback, hardware-accelerated blits as necessary) all
> > the way through.
> >
> > After seeing the way that the Pipewire backend handles streaming
> > (especially with
> > https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/1366),
> > I'm almost thinking that it would be preferable to maybe structure the
> > design of this feature something like this:
> >
> > * Add some sort of API on weston_output by which it can advertise the
> > ability to lay hands on the explicit renderbuffer (e.g.
> > gbm_surface_get_front_buffer()). This roughly equates to whether it's
> > a primary, non-nested backend. That is, DRM.
> >
> > * When processing the weston.ini mirror-of relationships at startup,
> > check whether the source output of the mirror-of declaration supports
> > that ability above. If so:
> >   - Wire up a signal so that the source output announces each newly
> > rendered frame to any/all mirror-of outputs.
> >   - The virtual output's backend allocates its own buffer pool in
> > exactly the same way that MR 1366 already does.
> >   - Upon receipt of each signal announcing a new frame's renderbuffer
> > from the source output, the virtual output does a very cheap
> > glBlitFramebuffer() to get the contents into its own buffer pool.
> > This avoids the possibility of an unresponsive client tying up the
> > source output's buffer.
> >
> > * If the source output isn't one that supports exposing the underlying
> > renderbuffer, then the mirror-of relationship continues as with MR
> > 1476 just to invoke an explicit weston_renderer pass to draw the
> > correct logical contents.
> >
> > How does this strike you?
> 
> Sorry, I don't understand how that idea is relevant to the KMS writeback case.
> Did you imply that DRM-backend could deliver a KMS-writeback FB instead of
> the rendered FB?

Just for the same of argument, yes. But I take your point below that this entire idea to drive the mirroring output directly from the source output's rendering doesn't fit the plan for the mirror-of semantics.

> 
> This is not the current plan for mirror-of. It does not allow the mirror output to
> run on its own pace independently of the source output. Re-using the source
> output's rendered FB would also be a problem for color management. The FB
> is rendered for that output, and the color properties of the mirror may be
> different, needing an independent rendering anyway.
> 
> The fundamental difference is who defines the properties of the stream.
> KMS writeback steam properties are necessarily defined by the KMS output.
> Mirror-of is for the opposite, for full flexibility. E.g. a remote mirror may not
> want to run at the full framerate, the physical monitor resolution, or with the
> color capabilities of the source output in order to save bandwidth and improve
> latency.

Okay, understood. Although I'm a bit curious how you can say that one output "mirrors" another whose resolution doesn't even match. Maybe you have scaling in mind?

> 
> It seems to me that we will need two different mechanisms for the two cases. I
> believe KMS writeback streaming is worthwhile to support, but not via mirror-
> of key. Maybe the writeback streaming should be built into the DRM-backend
> as a special pipewire source? Then it would also be guaranteed to be KMS
> writeback. It is some amount of code and testing duplication, but I think it
> would be the simplest.

Fair enough. I had the beginning of a complicated scheme of signals fallback rendering in mind. If you think it's justified to just add a special case directly for this, that's better to me.

> 
> I don't see KMS writeback streaming as specific to your curious virtualization
> case. I can imagine it being useful in general, to ensure that display controller
> output is correct for example, or when overall rendering efficiency is more
> important than optimal stream format.
> 
> 
> Thanks,
> pq

The backends mechanics to do that seem fairly obvious. Once https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/1366/ merges, there will be a ready-made way to allocate PipeWire buffers that are backed by dmabuf memory with whatever GBM flags seem appropriate to make the dmabuf eligible to register as a KMS framebuffer.

What do you think about the front-end logic to configure and activate this special KMS writeback PipeWire source? Using dedicated keys in the weston.ini file's [output] section for the KMS connector seems like the straightforward way to me. I suppose we could talk about a protocol extension, but that's (a) a lot more work and (b) obscures the KMS connector IDs to the client, who sees only the wl_output.

Thanks,
Matt