Full-motion zero-copy screen capture in Weston

[Hoosier, Matt]

Yes, the native Linux driver for the hardware still does end up being responsible for one or more planes.

Other than trying to arrange the pieces so that there's some API that offers the client an option to guarantee that the source of the screen capture involves the writeback connector (similar to what you've done with weston_output_capture), I don't really think it would be a good idea to make Weston explicitly aware of any of this funny hypervisor business going on.

The title on this email conversation was probably poorly chosen, in retrospect. I'm not so much concerned with keeping absolutely to a zero-copy mechanism as I am to using hardware mechanisms (GL rendering, DRI writeback, hardware-accelerated blits as necessary) all the way through.

After seeing the way that the Pipewire backend handles streaming (especially with https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/1366), I'm almost thinking that it would be preferable to maybe structure the design of this feature something like this:

* Add some sort of API on weston_output by which it can advertise the ability to lay hands on the explicit renderbuffer (e.g. gbm_surface_get_front_buffer()). This roughly equates to whether it's a primary, non-nested backend. That is, DRM.

* When processing the weston.ini mirror-of relationships at startup, check whether the source output of the mirror-of declaration supports that ability above. If so:
  - Wire up a signal so that the source output announces each newly rendered frame to any/all mirror-of outputs.
  - The virtual output's backend allocates its own buffer pool in exactly the same way that MR 1366 already does.
  - Upon receipt of each signal announcing a new frame's renderbuffer from the source output, the virtual output does a very cheap glBlitFramebuffer() to get the contents into its own buffer pool. This avoids the possibility of an unresponsive client tying up the source output's buffer.

* If the source output isn't one that supports exposing the underlying renderbuffer, then the mirror-of relationship continues as with MR 1476 just to invoke an explicit weston_renderer pass to draw the correct logical contents.

How does this strike you?

-Matt

> -----Original Message-----
> From: Pekka Paalanen <pekka.paalanen at collabora.com>
> Sent: Monday, June 10, 2024 3:03 AM
> To: Hoosier, Matt <Matt.Hoosier at garmin.com>
> Cc: sir at cmpwn.com; contact at emersion.fr; Marius Vlad
> <marius.vlad at collabora.com>; wayland-devel at lists.freedesktop.org
> Subject: Re: Full-motion zero-copy screen capture in Weston
> 
> On Fri, 7 Jun 2024 14:16:32 +0000
> "Hoosier, Matt" <Matt.Hoosier at garmin.com> wrote:
> 
> > > What do you mean you can capture all virtual machines with KMS
> > > writeback?
> > >
> > > What's the architecture there? How do virtual machines access KMS
> > > hardware? Why would Weston be able to capture their outputs at all?
> >
> > The world of virtualization on commercially supported embedded SOCs
> > for big-scale production use is wild. Every vendor typically has only
> > a narrow range of supported hypervisors. Usually, one -- and an
> > in-house model at that. There will typically be at least one bizarre
> > twist on the virtualization method for display controllers.
> >
> > One fad is for one of the virtual machines -- typically, the one
> > running a normal-style GNU/Linux Yocto system -- to own privileged I/O
> > maps of most of the hardware, and it more or less runs the same
> > drivers inside this VM that the SOC maker has already written for
> > their bare-metal Linux deployment. Most hardware peripherals are just
> > exposed to the other guest VMs by having the privileged Linux VM host
> > export some sort of hypervisor-integrated VirtIO-style backend for the
> > hardware. The guests see VirtIO devices. This approach goes by the
> > name of "paravirtualization".
> >
> > But for graphics and display, there is almost always some additional
> > mechanism to sidestep this pure paravirtualization because it's
> > perceived as too expensive. So the vendor may do something like
> > designate some subset of planes on each connector to be "directly"
> > manipulated by the non-GNU/Linux guest VMs. The hypervisor executive
> > runs a tiny little server that receives the stream of plane updates,
> > and during the vsync it programs the appropriate display controller
> > hardware registers to refer to the new frame's contents. It's very
> > limited -- the guests VMs whose scene updates are couched using this
> > mechanism are not able to do modesets or reconfigure the overall DRI
> > scene topology.
> >
> > The key point here is that because Linux is running a full physical
> > driver, the writeback connector gets the results of blending all the
> > layers -- even those whose contents are programmed using the awkward
> > side channel.
> >
> > I'm not a big fan on this approach. But it is there, and I'd like to
> > cope with it. I have a use-case that requires Linux to get a complete
> > picture of the physical contents getting scanned out by the connector.
> >
> 
> I see. On such connectors, does Weston still paint at least one KMS plane?
> 
> If we can pretend that the connector is still a normal output for Weston, just
> with less planes, I feel that this should be implementable in upstream Weston.
> 
> 
> Thanks,
> pq