Full-motion zero-copy screen capture in Weston

[Marius Vlad]

On Wed, Jun 12, 2024 at 09:35:48PM +0000, Hoosier, Matt wrote:
> > -----Original Message-----
> > From: Hoosier, Matt
> > Sent: Wednesday, June 12, 2024 8:37 AM
> > To: Pekka Paalanen <pekka.paalanen at collabora.com>
> > Cc: sir at cmpwn.com; contact at emersion.fr; Marius Vlad
> > <marius.vlad at collabora.com>; wayland-devel at lists.freedesktop.org; Daniel
> > Stone <daniel at fooishbar.org>
> > Subject: RE: Full-motion zero-copy screen capture in Weston
> > 
> > > -----Original Message-----
> > > From: Pekka Paalanen <pekka.paalanen at collabora.com>
> > > Sent: Wednesday, June 12, 2024 4:03 AM
> > > To: Hoosier, Matt <Matt.Hoosier at garmin.com>
> > > Cc: sir at cmpwn.com; contact at emersion.fr; Marius Vlad
> > > <marius.vlad at collabora.com>; wayland-devel at lists.freedesktop.org; Daniel
> > > Stone <daniel at fooishbar.org>
> > > Subject: Re: Full-motion zero-copy screen capture in Weston
> > >
> > > On Mon, 10 Jun 2024 14:18:39 +0000
> > > "Hoosier, Matt" <Matt.Hoosier at garmin.com> wrote:
> > >
> > > > Yes, the native Linux driver for the hardware still does end up being
> > > > responsible for one or more planes.
> > > >
> > > > Other than trying to arrange the pieces so that there's some API that
> > > > offers the client an option to guarantee that the source of the screen
> > > > capture involves the writeback connector (similar to what you've done
> > > > with weston_output_capture), I don't really think it would be a good
> > > > idea to make Weston explicitly aware of any of this funny hypervisor
> > > > business going on.
> > > >
> > >
> > > I very much agree.
> > 
> > Ack
> > 
> > 
> > >
> > > > The title on this email conversation was probably poorly chosen, in
> > > > retrospect. I'm not so much concerned with keeping absolutely to a
> > > > zero-copy mechanism as I am to using hardware mechanisms (GL
> > > > rendering, DRI writeback, hardware-accelerated blits as necessary) all
> > > > the way through.
> > > >
> > > > After seeing the way that the Pipewire backend handles streaming
> > > > (especially with
> > > > https://gitlab.freedesktop.org/wayland/weston/-
> > /merge_requests/1366),
> > > > I'm almost thinking that it would be preferable to maybe structure the
> > > > design of this feature something like this:
> > > >
> > > > * Add some sort of API on weston_output by which it can advertise the
> > > > ability to lay hands on the explicit renderbuffer (e.g.
> > > > gbm_surface_get_front_buffer()). This roughly equates to whether it's
> > > > a primary, non-nested backend. That is, DRM.
> > > >
> > > > * When processing the weston.ini mirror-of relationships at startup,
> > > > check whether the source output of the mirror-of declaration supports
> > > > that ability above. If so:
> > > >   - Wire up a signal so that the source output announces each newly
> > > > rendered frame to any/all mirror-of outputs.
> > > >   - The virtual output's backend allocates its own buffer pool in
> > > > exactly the same way that MR 1366 already does.
> > > >   - Upon receipt of each signal announcing a new frame's renderbuffer
> > > > from the source output, the virtual output does a very cheap
> > > > glBlitFramebuffer() to get the contents into its own buffer pool.
> > > > This avoids the possibility of an unresponsive client tying up the
> > > > source output's buffer.
> > > >
> > > > * If the source output isn't one that supports exposing the underlying
> > > > renderbuffer, then the mirror-of relationship continues as with MR
> > > > 1476 just to invoke an explicit weston_renderer pass to draw the
> > > > correct logical contents.
> > > >
> > > > How does this strike you?
> > >
> > > Sorry, I don't understand how that idea is relevant to the KMS writeback
> > case.
> > > Did you imply that DRM-backend could deliver a KMS-writeback FB instead
> > of
> > > the rendered FB?
> > 
> > Just for the same of argument, yes. But I take your point below that this entire
> > idea to drive the mirroring output directly from the source output's rendering
> > doesn't fit the plan for the mirror-of semantics.
> > 
> > >
> > > This is not the current plan for mirror-of. It does not allow the mirror output
> > to
> > > run on its own pace independently of the source output. Re-using the source
> > > output's rendered FB would also be a problem for color management. The FB
> > > is rendered for that output, and the color properties of the mirror may be
> > > different, needing an independent rendering anyway.
> > >
> > > The fundamental difference is who defines the properties of the stream.
> > > KMS writeback steam properties are necessarily defined by the KMS output.
> > > Mirror-of is for the opposite, for full flexibility. E.g. a remote mirror may not
> > > want to run at the full framerate, the physical monitor resolution, or with
> > the
> > > color capabilities of the source output in order to save bandwidth and
> > improve
> > > latency.
> > 
> > Okay, understood. Although I'm a bit curious how you can say that one
> > output "mirrors" another whose resolution doesn't even match. Maybe you
> > have scaling in mind?
> > 
> > >
> > > It seems to me that we will need two different mechanisms for the two
> > cases. I
> > > believe KMS writeback streaming is worthwhile to support, but not via
> > mirror-
> > > of key. Maybe the writeback streaming should be built into the DRM-
> > backend
> > > as a special pipewire source? Then it would also be guaranteed to be KMS
> > > writeback. It is some amount of code and testing duplication, but I think it
> > > would be the simplest.
> > 
> > Fair enough. I had the beginning of a complicated scheme of signals fallback
> > rendering in mind. If you think it's justified to just add a special case directly
> > for this, that's better to me.
> > 
> > >
> > > I don't see KMS writeback streaming as specific to your curious virtualization
> > > case. I can imagine it being useful in general, to ensure that display controller
> > > output is correct for example, or when overall rendering efficiency is more
> > > important than optimal stream format.
> > >
> > >
> > > Thanks,
> > > pq
> > 
> > The backends mechanics to do that seem fairly obvious. Once
> > https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/1366/
> > merges, there will be a ready-made way to allocate PipeWire buffers that are
> > backed by dmabuf memory with whatever GBM flags seem appropriate to
> > make the dmabuf eligible to register as a KMS framebuffer.
> 
> One thing that's not clear to me: what are the rules about which bits
> of backend code are allowed to link to various libraries? Weston has
> two different endpoints that separately integrate Pipewire: the
> pipewire-plugin and the pipewire-backend. The plugin seems to be
> mainly concerned to implement the API for Weston's virtual outputs,
> and I don't see any obvious way that the pipewire-backend
> module interacts with pipewire-plugin at all.
DRM virtual API and with it, its plug-ins are being proposed to be
removed: https://gitlab.freedesktop.org/wayland/weston/-/issues/914
> 
> If we added some support for streaming out the contents of a DRM output over PipeWire, would it be expected to somehow route all access to libpipewire through one or the other of those module through one of the API structs? Or should it be done by just directly using libpipewire in backend-drm?
> 
> > 
> > What do you think about the front-end logic to configure and activate this
> > special KMS writeback PipeWire source? Using dedicated keys in the
> > weston.ini file's [output] section for the KMS connector seems like the
> > straightforward way to me. I suppose we could talk about a protocol
> > extension, but that's (a) a lot more work and (b) obscures the KMS connector
> > IDs to the client, who sees only the wl_output.
> > 
> > Thanks,
> > Matt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20240613/8c25b14e/attachment.sig>