Full-motion zero-copy screen capture in Weston

Thu May 30 13:40:15 UTC 2024

Okay, interesting thoughts on all of that.

I'm not sure how far I'm going to get toward a complete overhaul of the capture mechanism. Maybe it would be useful to know a couple things about the current one (weston_output_capture_v1), so that I don't commit mistakes that somebody already considered and guarded against:

* Why was it explicitly picked only for still shots? I can image that it wouldn't have been a whole lot more work to design this API with a bounce-buffering scheme rather than a setup/do-it-once/destroy model.

* Why was wl_shm explicitly picked for the buffer type? I was thinking here that it would have worked equally well to specify that the client must supply a linear-layout buffer manufactured through zwp_linux_dmabuf. This would be eligible for direct targeting by the GL renderer/KMS writeback, but also can be mapped with gbm_bo_map() in the compositor's Pixman backend and/or a client who wants to map it to the CPU upon delivery.

-Matt

On 5/30/24, 3:22 AM, "Pekka Paalanen" <pekka.paalanen at collabora.com <mailto:pekka.paalanen at collabora.com>> wrote:

On Wed, 29 May 2024 15:18:16 +0000
"Hoosier, Matt" <Matt.Hoosier at garmin.com <mailto:Matt.Hoosier at garmin.com>> wrote:

> Thanks, Pekka.
> 
> So what would be an upstream-friendly way to put hooks deep down into
> the DRM backend to exploit the KMS writeback connectors, but still
> allow an XDG Desktop portal implementation to be usable on all the
> backends?

Hi Matt,

I don't know how that would look like. "Hooks" feels like a
side-channel to me, while I'd prefer a first class API. This probably
requires adjusting some of the existing internal interfaces so that
they can handle both one-shot and streaming captures (timestamps!), and
even multiple capture consumers simultaneously.

While the private screenshooting protocol currently explicitly defines
how the screenshot must be taken, this improved API should be able to
fall back as necessary and if desired: use KMS writeback when it works,
fall back to renderer otherwise (which disables KMS planes composition).

There is also the dmabuf vs. wl_shm buffer issue.

All the external APIs should be implemented on top of such improved
internal API: screenshots, screen streaming, anything that wants to get
the contents of another backend's output and not create an independent
output of its own with an independent update/rendering cadence.

> Or maybe that's an unnecessary goal to begin with; a desktop session
> somewhat implies the compositor is driving the physical hardware?

I think the improved API needs to be usable on all backends, yes. Only
KMS writeback needs backend support, the rest should be doable in the
renderers which are shared by all backends. Non-DRM backends just don't
have KMS writeback.

Something like this would be ideal, I believe. Unfortunately I don't
think I can personally spare time to help with this. This is more of a
wish than a requirement.

Thanks,
pq

> 
> > -----Original Message-----
> > From: Pekka Paalanen <pekka.paalanen at collabora.com <mailto:pekka.paalanen at collabora.com>>
> > Sent: Wednesday, May 29, 2024 2:43 AM
> > To: Marius Vlad <marius.vlad at collabora.com <mailto:marius.vlad at collabora.com>>; Hoosier, Matt
> > <Matt.Hoosier at garmin.com <mailto:Matt.Hoosier at garmin.com>>
> > Cc: wayland-devel at lists.freedesktop.org <mailto:wayland-devel at lists.freedesktop.org>
> > Subject: Re: Full-motion zero-copy screen capture in Weston
> > 
> > On Tue, 28 May 2024 20:04:45 +0300
> > Marius Vlad <marius.vlad at collabora.com <mailto:marius.vlad at collabora.com>> wrote:
> > 
> > > On Tue, May 28, 2024 at 03:53:23PM +0000, Hoosier, Matt wrote: 
> > > > Hi Marius, 
> > > Hi, 
> > > >
> > > > Okay, I guess that answers the bit about needing to screen-scrape to get 
> > content into Pipewire now. 
> > > >
> > > > But I'm still a little unclear about a couple things, if I were to try to build on 
> > this PW backend as a starting point: 
> > > >
> > > > First, it looks to me like when you use the PW backend to Weston,
> > > > that becomes your display. That is, rendering directly targets it. I
> > > > was hoping for a way to get it to broadcast the very same
> > > > framebuffer(s) that are getting scanned out for the current frame by
> > > > the DRM backend. 
> > > With
> > > https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/341 <https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/341>
> > > the PipeWire output can mirror out the native DRM one. It aims at
> > > having a way to configure VNC, RDP and PipeWire to screen-share the 
> > output. 
> > > >
> > > > Second, I'm don't see the path to getting this to leverage the
> > > > DRM_MODE_CONNECTOR_WRITEBACK hardware (like the
> > > > weston_screen_recorder does). I think that any layering would be
> > > > forced to be offloaded to the GPU ahead of time. Maybe I missed some
> > > > implication of what you were pointing out here? 
> > > No sorry, I haven't really implied that, just pointed that out there's
> > > some work for PipeWire gaining dmabuf.
> > >
> > > Screen-sharing an output is done
> > > with multiple backends, and configuring Weston front-end is it with
> > > https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/341. <https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/341.  >
> > 
> > Hi Matt, Marius,
> > 
> > multiple backends necessarily means that each backend renders its own
> > weston_output independently. While it's fine for most screen-sharing use
> > cases, I don't think it can ever be as efficient as KMS writeback could be.
> > 
> > Getting a dmabuf-based video stream from DRM-backend using KMS
> > writeback needs integration in the DRM-backend. I don't know how that
> > would look like, but I suspect these ideas would be good:
> > 
> > - builds on and extends the existing writeback support instead of
> > something new on the side
> > 
> > - does not use the DRM-backend "virtual output" API; this was a
> > workaround for the lack of multi-backend support in the past and has
> > the same disadvantages as multi-backend. (I don't think the DRM
> > virtual output API has any reason to exist anymore, multi-backend
> > should supersede it after converting the users.)
> > 
> > I also think that implementing the usual desktop portals in Weston would be a
> > good thing, and not something to avoid, if those can provide a nice external
> > interface for the use case.
> > 
> > 
> > Thanks,
> > pq 