Full-motion zero-copy screen capture in Weston

Fri Jun 14 18:36:57 UTC 2024

> -----Original Message-----
> From: Pekka Paalanen <pekka.paalanen at collabora.com>
> Sent: Thursday, June 13, 2024 3:32 AM
> To: Hoosier, Matt <Matt.Hoosier at garmin.com>
> Cc: 'sir at cmpwn.com' <sir at cmpwn.com>; 'contact at emersion.fr'
> <contact at emersion.fr>; 'Marius Vlad' <marius.vlad at collabora.com>;
> 'wayland-devel at lists.freedesktop.org' <wayland-
> devel at lists.freedesktop.org>; 'Daniel Stone' <daniel at fooishbar.org>
> Subject: Re: Full-motion zero-copy screen capture in Weston
> 
> On Wed, 12 Jun 2024 21:35:48 +0000
> "Hoosier, Matt" <Matt.Hoosier at garmin.com> wrote:
> 
> > > -----Original Message-----
> > > From: Hoosier, Matt
> > > Sent: Wednesday, June 12, 2024 8:37 AM
> > > To: Pekka Paalanen <pekka.paalanen at collabora.com>
> > > Cc: sir at cmpwn.com; contact at emersion.fr; Marius Vlad
> > > <marius.vlad at collabora.com>; wayland-devel at lists.freedesktop.org;
> > > Daniel Stone <daniel at fooishbar.org>
> > > Subject: RE: Full-motion zero-copy screen capture in Weston
> > >
> > > > -----Original Message-----
> > > > From: Pekka Paalanen <pekka.paalanen at collabora.com>
> > > > Sent: Wednesday, June 12, 2024 4:03 AM
> > > > To: Hoosier, Matt <Matt.Hoosier at garmin.com>
> > > > Cc: sir at cmpwn.com; contact at emersion.fr; Marius Vlad
> > > > <marius.vlad at collabora.com>; wayland-devel at lists.freedesktop.org;
> > > > Daniel Stone <daniel at fooishbar.org>
> > > > Subject: Re: Full-motion zero-copy screen capture in Weston
> > > >
> > > > On Mon, 10 Jun 2024 14:18:39 +0000 "Hoosier, Matt"
> > > > <Matt.Hoosier at garmin.com> wrote:
> > > >
> > > > > Yes, the native Linux driver for the hardware still does end up
> > > > > being responsible for one or more planes.
> > > > >
> > > > > Other than trying to arrange the pieces so that there's some API
> > > > > that offers the client an option to guarantee that the source of
> > > > > the screen capture involves the writeback connector (similar to
> > > > > what you've done with weston_output_capture), I don't really
> > > > > think it would be a good idea to make Weston explicitly aware of
> > > > > any of this funny hypervisor business going on.
> > > > >
> > > >
> > > > I very much agree.
> > >
> > > Ack
> > >
> > >
> > > >
> > > > > The title on this email conversation was probably poorly chosen,
> > > > > in retrospect. I'm not so much concerned with keeping absolutely
> > > > > to a zero-copy mechanism as I am to using hardware mechanisms
> > > > > (GL rendering, DRI writeback, hardware-accelerated blits as
> > > > > necessary) all the way through.
> > > > >
> > > > > After seeing the way that the Pipewire backend handles streaming
> > > > > (especially with
> > > > > https://gitlab.freedesktop.org/wayland/weston/-
> > > /merge_requests/1366),
> > > > > I'm almost thinking that it would be preferable to maybe
> > > > > structure the design of this feature something like this:
> > > > >
> > > > > * Add some sort of API on weston_output by which it can
> > > > > advertise the ability to lay hands on the explicit renderbuffer (e.g.
> > > > > gbm_surface_get_front_buffer()). This roughly equates to whether
> > > > > it's a primary, non-nested backend. That is, DRM.
> > > > >
> > > > > * When processing the weston.ini mirror-of relationships at
> > > > > startup, check whether the source output of the mirror-of
> > > > > declaration supports that ability above. If so:
> > > > >   - Wire up a signal so that the source output announces each
> > > > > newly rendered frame to any/all mirror-of outputs.
> > > > >   - The virtual output's backend allocates its own buffer pool
> > > > > in exactly the same way that MR 1366 already does.
> > > > >   - Upon receipt of each signal announcing a new frame's
> > > > > renderbuffer from the source output, the virtual output does a
> > > > > very cheap
> > > > > glBlitFramebuffer() to get the contents into its own buffer pool.
> > > > > This avoids the possibility of an unresponsive client tying up
> > > > > the source output's buffer.
> > > > >
> > > > > * If the source output isn't one that supports exposing the
> > > > > underlying renderbuffer, then the mirror-of relationship
> > > > > continues as with MR
> > > > > 1476 just to invoke an explicit weston_renderer pass to draw the
> > > > > correct logical contents.
> > > > >
> > > > > How does this strike you?
> > > >
> > > > Sorry, I don't understand how that idea is relevant to the KMS
> > > > writeback
> > > case.
> > > > Did you imply that DRM-backend could deliver a KMS-writeback FB
> > > > instead
> > > of
> > > > the rendered FB?
> > >
> > > Just for the same of argument, yes. But I take your point below that
> > > this entire idea to drive the mirroring output directly from the
> > > source output's rendering doesn't fit the plan for the mirror-of semantics.
> > >
> > > >
> > > > This is not the current plan for mirror-of. It does not allow the
> > > > mirror output
> > > to
> > > > run on its own pace independently of the source output. Re-using
> > > > the source output's rendered FB would also be a problem for color
> > > > management. The FB is rendered for that output, and the color
> > > > properties of the mirror may be different, needing an independent
> rendering anyway.
> > > >
> > > > The fundamental difference is who defines the properties of the stream.
> > > > KMS writeback steam properties are necessarily defined by the KMS
> output.
> > > > Mirror-of is for the opposite, for full flexibility. E.g. a remote
> > > > mirror may not want to run at the full framerate, the physical
> > > > monitor resolution, or with
> > > the
> > > > color capabilities of the source output in order to save bandwidth
> > > > and
> > > improve
> > > > latency.
> > >
> > > Okay, understood. Although I'm a bit curious how you can say that
> > > one output "mirrors" another whose resolution doesn't even match.
> > > Maybe you have scaling in mind?
> 
> It's hard to pick the right words. See e.g. the thread at
> https://gitlab.freedesktop.org/wayland/weston/-
> /merge_requests/1476#note_2435244
> 
> Current upstream has "same-as" for shared-CRTC KMS cloning, which would
> probably be renamed to "clone-of". Do we then re-purpose "same-as" for
> something else? We've also talked about "mirror-of" and "share-position-
> with".
> 
> What the exact words and keys will be, we're not committed to yet. There are
> several different kinds of mirroring. All the kinds that expose a new wl_output
> and have a separate [output] section in weston.ini, except for old "same-as" /
> new "clone-of", will create an independent weston_output and be mirroring
> based on exposing the same desktop area as some source output.
> 
> (If one thinks of mirrors in real world, they almost never reproduce the original
> exactly, and they can be made to distort the image. They tend to at least
> "mirror" things, that is, swap left and right for
> instance.)
> 
> > > >
> > > > It seems to me that we will need two different mechanisms for the
> > > > two
> > > cases. I
> > > > believe KMS writeback streaming is worthwhile to support, but not
> > > > via
> > > mirror-
> > > > of key. Maybe the writeback streaming should be built into the
> > > > DRM-
> > > backend
> > > > as a special pipewire source? Then it would also be guaranteed to
> > > > be KMS writeback. It is some amount of code and testing
> > > > duplication, but I think it would be the simplest.
> > >
> > > Fair enough. I had the beginning of a complicated scheme of signals
> > > fallback rendering in mind. If you think it's justified to just add
> > > a special case directly for this, that's better to me.
> > >
> > > >
> > > > I don't see KMS writeback streaming as specific to your curious
> > > > virtualization case. I can imagine it being useful in general, to
> > > > ensure that display controller output is correct for example, or
> > > > when overall rendering efficiency is more important than optimal stream
> format.
> > > >
> > > >
> > > > Thanks,
> > > > pq
> > >
> > > The backends mechanics to do that seem fairly obvious. Once
> > > https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/1366/
> > > merges, there will be a ready-made way to allocate PipeWire buffers
> > > that are backed by dmabuf memory with whatever GBM flags seem
> > > appropriate to make the dmabuf eligible to register as a KMS
> > > framebuffer.
> >
> > One thing that's not clear to me: what are the rules about which bits
> > of backend code are allowed to link to various libraries? Weston has
> > two different endpoints that separately integrate Pipewire: the
> > pipewire-plugin and the pipewire-backend. The plugin seems to be
> > mainly concerned to implement the API for Weston's virtual outputs,
> > and I don't see any obvious way that the pipewire-backend module
> > interacts with pipewire-plugin at all.
> 
> Pipewire-backend and pipewire-plugin are completely independent entities,
> they are not intended to interact. Pipewire-plugin is also on its way out:
> https://gitlab.freedesktop.org/wayland/weston/-/issues/914
> 
> Backends are somewhat more free to pull in new dependencies than libweston
> core, because backends can be not built. Of course, DRM-backend being such
> a central component, we'd probably want a separate build option to enable
> "DRM KMS writeback Pipewire support" or such, defaulting to enabled.
> 
> > If we added some support for streaming out the contents of a DRM
> > output over PipeWire, would it be expected to somehow route all access
> > to libpipewire through one or the other of those module through one of
> > the API structs? Or should it be done by just directly using
> > libpipewire in backend-drm?
> 
> Just use libpipewire directly. We would be having two independent pipewire
> users then: pipewire-backend and DRM-backend, with the pipewire-plugin
> going away. I'm not familiar with libpipewire though, so if it makes it difficult or
> somehow sub-optimal to keep the two users completely separate, then we'll
> have to figure out a way to share the pipewire context. I wouldn't go wrapping
> pipewire API per se.
> 
> Maybe an internal static helper library, and whoever initializes it first, the
> helper will register itself with the "plugin registry" and further initializers will
> automatically get the existing instance.
> 
> > >
> > > What do you think about the front-end logic to configure and
> > > activate this special KMS writeback PipeWire source? Using dedicated
> > > keys in the weston.ini file's [output] section for the KMS connector
> > > seems like the straightforward way to me. I suppose we could talk
> > > about a protocol extension, but that's (a) a lot more work and (b)
> > > obscures the KMS connector IDs to the client, who sees only the
> > > wl_output.
> 
> Yes, I think the way to go is to do exactly that with weston.ini. If that's enough,
> then no need to consider any IPC or protocol.
> 
> 
> Thanks,
> pq

Hmm. As I read through the history of the original support for writeback screenshots (https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/458) and get some initial results just trying to drive a steady stream of writeback composition, I'm not sure this can work effectively.

MR 458 has a fairly long discussion about the restriction that no further commits must be made to the source CRTC or any of the other KMS objects reachable on its graph, while a writeback composition referencing that CRTC is still in flight.

Daniel Vetter confirmed that interpretation.

I think this means that unless you are extremely lucky that:

(a) your hardware's writeback mechanism is synchronous with scanout, and
(b) the fence fd manages to fire and get dispatched immediately upon scanout/writeback passing through the final scanline, and
(c) there's a really large vertical back porch

, there will be no time left to enqueue a page flip for the immediately succeeding vblank.

Seems like this will almost always cut the frame achievable update rate on the main connector in half.

Did I misinterpret something in there? Early results look like the kernel state machine gets confused (with VKMS anyway) if I queue up two writeback operations in flight at the same time.

-Matt