Screen shooting and recording protocols (Re: Authorized clients)

[Pekka Paalanen]

On Thu, 09 Jan 2014 13:05:28 +0100
Martin Peres <martin.peres at free.fr> wrote:

> Le 09/01/2014 10:00, Pekka Paalanen a écrit :
> > Hi,
> >
> > what I'm replying to has nothing to do with security anymore, so I
> > changed the topic.
> >
> > The security issue is what and when can use a specific protocol
> > interface, the below is about how to use the interfaces once you
> > already have access
> This is indeed something that needs to be discussed independently
> of any security concern.
> 
> > On Wed, 8 Jan 2014 23:30:29 +0100
> > Maarten Baert <maarten-baert at hotmail.com> wrote:
> >
> >> On 08/01/14 13:52, Martin Peres wrote:
> >>> I think the screenshot API and the video recording one should be
> >>> separate.
> >> Ideally that's true, but it creates extra work for the compositor
> >> developers. And I don't see a huge benefit, actually. On X11 I can
> >> simply take 30 screenshots per second and this works great, the
> >> performance is about as good as it can get (2ms for a 1920x1080
> >> frame, that's almost as fast as a simple copy) and the only issue
> >> is tearing (which is something that a good screenshot protocol
> >> should already solve, obviously we don't want tearing in
> >> screenshots either). So I would be perfectly happy with a
> >> screenshot interface that works just like X11 minus the tearing.
> >> Do we really need a separate API for that?
> >>
> >> There's another gray area here: what about timelapse videos? These
> >> are created by taking a screenshot every few seconds and then
> >> playing them back at much higher speed. Should such an application
> >> use the screenshot or the video API?
> > There are differences in the implementation details of shooting
> > (stills) vs. recording (videos).
> >
> > Weston supports (though disabled atm, AFAIK) hw overlays in
> > addition to the GL renderer. To make a screenshot, the overlays are
> > temporarily disabled, so that GL can composite the whole image
> > which can then be sent to the shooting client.
> >
> > If you do repeated shooting, you probably end up doing a lot of
> > useless back and forth work, enabling and disabling hw overlays.
> > Also, shooting has one frame latency, because you have to wait for
> > a repaint after you disabled the overlays. It will probably be
> > quite hard to get a stable framerate.
> >
> > If the compositor knows we are doing recording instead of shooting,
> > it can disable overlays for the whole duration. You only have the
> > latency for the first frame, not all frames. There are probably
> > other details that can be optimized, too.
> >
> > Shooting can easily be implemented like the current screenshooter
> > protocol: a client creates a wl_buffer, and tells the compositor to
> > fill it in. There is an inevitable copy, but since shooting is a
> > rare action and usually triggered by the user directly, that cost
> > is acceptable. The timing requirements are also very loose.
> >
> > That scheme might not work too well with recording, and achieving a
> > zero-copy path to the video encoder will be hard. In that sense, the
> > compositor autonomously pushing buffers to the client would be more
> > performant, but then we have the problem of synchronizing buffer
> > reads, writes, and re-uses. In the opposite direction, the
> > synchronization was solved with the wl_surface.attach/commit and
> > wl_buffer.release messages, but here an accidentally malfunctioning
> > client has a greater risk to crash the compositor, if it does not
> > return the buffers to the compositor in time, so the compositor
> > needs a failsafe mechanism.
> >
> > Those are some reasons why screen recording (video) is easier to do
> > as a compositor plugin, like it is currently in Weston. A separate
> > client would need a non-trivial amount of new Wayland protocol to
> > work well.
> 
> There is a way to limit the memory consumption of apps that don't
> consume buffers. We could have a small ring buffer of wl_buffer or
> dma-buf (if we want 0 copy) on the compositor side. When the
> ring buffer is full, the compositor should not copy or store any new
> buffer. This way, it won't use up all the video memory.

Yeah, that would be part of the failsafe, except the implementation
details and protocol get complicated as you probably do not want the
compositor to depend on the client in allocating the buffers the
compositor is rendering into, which means you cannot use wl_drm.

Or maybe you could... compositor has its own 2-3 private buffers it
normally uses, and the recorder client could give it some "empty"
buffers. The compositor renders into client's buffers one by one if they
are suitable and not busy, *and* flips them on screen. As the fallback
it can use its own buffers. That way the client is the one that gets
"killed on OOM". Maybe that'd work.

> Then, we need a way for the compositor to know when the user
> consumes a buffer. Having a simple "get_next_frame()" function
> returning the next wl_buffer or the next dma-buf fd would be
> enough. We could also tell that the semantics of the call means
> that the application should be done using a frame before
> requiring a new one. If we really want to be conservative, we could
> only use 2 entries in the ring buffer.
> 
> This way, I don't think we would need any other form of
> synchronization, would we?

Such a function would be higher level API than the protocol, since the
wire protocol has no concept of a "return value".

Yes, that should be all that's needed. "Here it is, just say when
you're done."

> In order to reduce the frame rate, another function in the protocol
> is needed to set the desired framerate otherwise a lot of copies will
> be made for no good reason, as Maarten rightly suggested!

Not really. We want zero-copy, and then it's ok to push frames
at the compositor's repaint rate (which is variable!). If the client
doesn't need some of them, it just acks them without processing. I'm
assuming a frame is sent only when the compositor repaints, i.e.
something on screen changes. That's how the current recorder works, too.

We do not want to cap the compositor's repaint rate with this in any
case.


Thanks,
pq