Full-motion zero-copy screen capture in Weston

Thu Jun 13 08:32:13 UTC 2024

On Wed, 12 Jun 2024 21:35:48 +0000
"Hoosier, Matt" <Matt.Hoosier at garmin.com> wrote:

> > -----Original Message-----
> > From: Hoosier, Matt
> > Sent: Wednesday, June 12, 2024 8:37 AM
> > To: Pekka Paalanen <pekka.paalanen at collabora.com>
> > Cc: sir at cmpwn.com; contact at emersion.fr; Marius Vlad
> > <marius.vlad at collabora.com>; wayland-devel at lists.freedesktop.org; Daniel
> > Stone <daniel at fooishbar.org>
> > Subject: RE: Full-motion zero-copy screen capture in Weston
> >   
> > > -----Original Message-----
> > > From: Pekka Paalanen <pekka.paalanen at collabora.com>
> > > Sent: Wednesday, June 12, 2024 4:03 AM
> > > To: Hoosier, Matt <Matt.Hoosier at garmin.com>
> > > Cc: sir at cmpwn.com; contact at emersion.fr; Marius Vlad
> > > <marius.vlad at collabora.com>; wayland-devel at lists.freedesktop.org; Daniel
> > > Stone <daniel at fooishbar.org>
> > > Subject: Re: Full-motion zero-copy screen capture in Weston
> > >
> > > On Mon, 10 Jun 2024 14:18:39 +0000
> > > "Hoosier, Matt" <Matt.Hoosier at garmin.com> wrote:
> > >  
> > > > Yes, the native Linux driver for the hardware still does end up being
> > > > responsible for one or more planes.
> > > >
> > > > Other than trying to arrange the pieces so that there's some API that
> > > > offers the client an option to guarantee that the source of the screen
> > > > capture involves the writeback connector (similar to what you've done
> > > > with weston_output_capture), I don't really think it would be a good
> > > > idea to make Weston explicitly aware of any of this funny hypervisor
> > > > business going on.
> > > >  
> > >
> > > I very much agree.  
> > 
> > Ack
> > 
> >   
> > >  
> > > > The title on this email conversation was probably poorly chosen, in
> > > > retrospect. I'm not so much concerned with keeping absolutely to a
> > > > zero-copy mechanism as I am to using hardware mechanisms (GL
> > > > rendering, DRI writeback, hardware-accelerated blits as necessary) all
> > > > the way through.
> > > >
> > > > After seeing the way that the Pipewire backend handles streaming
> > > > (especially with
> > > > https://gitlab.freedesktop.org/wayland/weston/-  
> > /merge_requests/1366),  
> > > > I'm almost thinking that it would be preferable to maybe structure the
> > > > design of this feature something like this:
> > > >
> > > > * Add some sort of API on weston_output by which it can advertise the
> > > > ability to lay hands on the explicit renderbuffer (e.g.
> > > > gbm_surface_get_front_buffer()). This roughly equates to whether it's
> > > > a primary, non-nested backend. That is, DRM.
> > > >
> > > > * When processing the weston.ini mirror-of relationships at startup,
> > > > check whether the source output of the mirror-of declaration supports
> > > > that ability above. If so:
> > > >   - Wire up a signal so that the source output announces each newly
> > > > rendered frame to any/all mirror-of outputs.
> > > >   - The virtual output's backend allocates its own buffer pool in
> > > > exactly the same way that MR 1366 already does.
> > > >   - Upon receipt of each signal announcing a new frame's renderbuffer
> > > > from the source output, the virtual output does a very cheap
> > > > glBlitFramebuffer() to get the contents into its own buffer pool.
> > > > This avoids the possibility of an unresponsive client tying up the
> > > > source output's buffer.
> > > >
> > > > * If the source output isn't one that supports exposing the underlying
> > > > renderbuffer, then the mirror-of relationship continues as with MR
> > > > 1476 just to invoke an explicit weston_renderer pass to draw the
> > > > correct logical contents.
> > > >
> > > > How does this strike you?  
> > >
> > > Sorry, I don't understand how that idea is relevant to the KMS writeback  
> > case.  
> > > Did you imply that DRM-backend could deliver a KMS-writeback FB instead  
> > of  
> > > the rendered FB?  
> > 
> > Just for the same of argument, yes. But I take your point below that this entire
> > idea to drive the mirroring output directly from the source output's rendering
> > doesn't fit the plan for the mirror-of semantics.
> >   
> > >
> > > This is not the current plan for mirror-of. It does not allow the mirror output  
> > to  
> > > run on its own pace independently of the source output. Re-using the source
> > > output's rendered FB would also be a problem for color management. The FB
> > > is rendered for that output, and the color properties of the mirror may be
> > > different, needing an independent rendering anyway.
> > >
> > > The fundamental difference is who defines the properties of the stream.
> > > KMS writeback steam properties are necessarily defined by the KMS output.
> > > Mirror-of is for the opposite, for full flexibility. E.g. a remote mirror may not
> > > want to run at the full framerate, the physical monitor resolution, or with  
> > the  
> > > color capabilities of the source output in order to save bandwidth and  
> > improve  
> > > latency.  
> > 
> > Okay, understood. Although I'm a bit curious how you can say that one
> > output "mirrors" another whose resolution doesn't even match. Maybe you
> > have scaling in mind?

It's hard to pick the right words. See e.g. the thread at
https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/1476#note_2435244

Current upstream has "same-as" for shared-CRTC KMS cloning, which would
probably be renamed to "clone-of". Do we then re-purpose "same-as" for
something else? We've also talked about "mirror-of" and
"share-position-with".

What the exact words and keys will be, we're not committed to yet. There
are several different kinds of mirroring. All the kinds that expose a
new wl_output and have a separate [output] section in weston.ini,
except for old "same-as" / new "clone-of", will create an independent
weston_output and be mirroring based on exposing the same desktop area
as some source output.

(If one thinks of mirrors in real world, they almost never reproduce
the original exactly, and they can be made to distort the image. They
tend to at least "mirror" things, that is, swap left and right for
instance.)

> > >
> > > It seems to me that we will need two different mechanisms for the two  
> > cases. I  
> > > believe KMS writeback streaming is worthwhile to support, but not via  
> > mirror-  
> > > of key. Maybe the writeback streaming should be built into the DRM-  
> > backend  
> > > as a special pipewire source? Then it would also be guaranteed to be KMS
> > > writeback. It is some amount of code and testing duplication, but I think it
> > > would be the simplest.  
> > 
> > Fair enough. I had the beginning of a complicated scheme of signals fallback
> > rendering in mind. If you think it's justified to just add a special case directly
> > for this, that's better to me.
> >   
> > >
> > > I don't see KMS writeback streaming as specific to your curious virtualization
> > > case. I can imagine it being useful in general, to ensure that display controller
> > > output is correct for example, or when overall rendering efficiency is more
> > > important than optimal stream format.
> > >
> > >
> > > Thanks,
> > > pq  
> > 
> > The backends mechanics to do that seem fairly obvious. Once
> > https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/1366/
> > merges, there will be a ready-made way to allocate PipeWire buffers
> > that are backed by dmabuf memory with whatever GBM flags seem
> > appropriate to make the dmabuf eligible to register as a KMS
> > framebuffer.  
> 
> One thing that's not clear to me: what are the rules about which bits
> of backend code are allowed to link to various libraries? Weston has
> two different endpoints that separately integrate Pipewire: the
> pipewire-plugin and the pipewire-backend. The plugin seems to be
> mainly concerned to implement the API for Weston's virtual outputs,
> and I don't see any obvious way that the pipewire-backend module
> interacts with pipewire-plugin at all.

Pipewire-backend and pipewire-plugin are completely independent
entities, they are not intended to interact. Pipewire-plugin is also on
its way out:
https://gitlab.freedesktop.org/wayland/weston/-/issues/914

Backends are somewhat more free to pull in new dependencies than
libweston core, because backends can be not built. Of course,
DRM-backend being such a central component, we'd probably want a
separate build option to enable "DRM KMS writeback Pipewire support" or
such, defaulting to enabled.

> If we added some support for streaming out the contents of a DRM
> output over PipeWire, would it be expected to somehow route all
> access to libpipewire through one or the other of those module
> through one of the API structs? Or should it be done by just directly
> using libpipewire in backend-drm?

Just use libpipewire directly. We would be having two independent
pipewire users then: pipewire-backend and DRM-backend, with the
pipewire-plugin going away. I'm not familiar with libpipewire though,
so if it makes it difficult or somehow sub-optimal to keep the two
users completely separate, then we'll have to figure out a way to share
the pipewire context. I wouldn't go wrapping pipewire API per se.

Maybe an internal static helper library, and whoever initializes it
first, the helper will register itself with the "plugin registry" and
further initializers will automatically get the existing instance.

> > 
> > What do you think about the front-end logic to configure and
> > activate this special KMS writeback PipeWire source? Using
> > dedicated keys in the weston.ini file's [output] section for the
> > KMS connector seems like the straightforward way to me. I suppose
> > we could talk about a protocol extension, but that's (a) a lot more
> > work and (b) obscures the KMS connector IDs to the client, who sees
> > only the wl_output.

Yes, I think the way to go is to do exactly that with weston.ini. If
that's enough, then no need to consider any IPC or protocol.

Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20240613/c8753e1f/attachment.sig>