Screen shooting and recording protocols (Re: Authorized clients)

[Maarten Baert]

On 09/01/14 20:14, Bill Spitzak wrote:
> My quick impression is that a framerate hint is not needed. Instead
> they are throttled by the client not releasing the buffers.
I tried that the first time I implemented OpenGL recording (using a
5-frame ring buffer). It worked, kind of, but the frame rate was very
unsteady (read: almost unusable). There's also a lot of latency when you
intentionally leave the ring buffer full, and you have no safety margin
to deal with short interruptions. So please, don't use the ring buffer
for throttling. There was a massive quality improvement when I
programmed the video source to do some basic rate control:

int64_t timestamp = hrt_time_micro();
int64_t interval = 1000000 / target_fps;
if(timestamp < m_next_frame_time - interval)
    return;
m_next_frame_time = std::max(m_next_frame_time + interval, timestamp);

It's a simple compromise between long-term accuracy (average frame rate)
and flexibility/robustness to deal with irregular frame rates. If the
compositor is guaranteed to be synchronized to the monitor refresh rate
at all times (i.e. no missed deadlines, no suspends, no monitor changes,
...), this robustness isn't even needed, and it becomes even simpler:

int64_t timestamp = hrt_time_micro();
int64_t interval = 1000000 / target_fps;
if(timestamp < m_next_frame_time)
    return;
m_next_frame_time += interval;

Still, I would keep the more robust version because it has no
significant downsides.

But I agree that if the compositor is able to capture frames with zero
overhead, then the compositor should just capture every frame and let
the application decide. Is this realistic?

On 09/01/14 22:14, Martin Peres wrote:
> I'm not saying supporting the acquisition of just a rectangle isn't a
> good idea but if what the user wants is the output of a window, it is
> stupid
> to grab the whole screen. Shouldn't we try to make stuff just work,
> whatever the user does? 
Yes of course, this is something that is needed regardless of whether
window picking is supported or not. So what I'm proposing is basically
the ability to:
- capture a single output (applications can deal with synchronization of
multiple outputs if needed, synchronization code is needed for audio
anyway so it's actually quite easy to add)
- capture a part of a single output (video resolutions are often lower
than screen resolutions, so some users prefer to record only a 1280x720
or 854x480 rectangle of their 1440x900 display and they just move
whatever they want to record into that rectangle)
The window picking would then just be an easy way to quickly get the
correct x/y/w/h values rather than entering them manually.

My main point is that when the user wants to capture a window, the
compositor should capture it as it appears on the screen rather than
capturing the SHM buffer of the application (or have support for both):
- The SHM buffer doesn't show subwindows, dialog windows, tooltips,
drop-down menus, things like that.
- Applications can be partially transparent. None of the popular video
formats support transparency, and I really can't think of a sane way to
display a transparent video in the first place. Maybe transparency is a
cool feature for screenshots, but for video it is really a problem, so
there should be a way to avoid that.
- SHM buffers can use any sample format the application wants (if
supported by the compositor of course). A screenshot/recording
application can easily add code for the most common screen formats
(these days almost every desktop uses BGRX or BGRA anyway), but dealing
with whatever weird formats some applications might want to use will be
a nightmare. The compositor already has to convert the buffer to the
screen format when it is drawn, so it's much easier and probably also
more efficient to capture the screen rather than doing a second conversion.

Maarten Baert