Time in wayland presentation extension

[Pekka Paalanen]

On Fri, 12 Dec 2014 13:48:44 +0100
Dan Oscarsson <Dan.Oscarsson at tieto.com> wrote:

> tis 2014-12-09 klockan 17:32 +0200 skrev Pekka Paalanen:
> > > I have expected FreeSync, Adaptive-Sync, Nvidia G-sync to actually allow
> > > the video player (not the compositor) to play video and get each frame
> > > to be displayed at exactly the correct time for all movie speeds. It
> > > would also handle videos with variable frame rate to be display
> > > correctly.
> > 
> > Yes, but always through the compositor. It can bypass compositing, but
> > not the compositor.
> > 
> > X11 has special "let's use a hardware overlay" protocol extension, but
> > we deliberately do not want such thing in Wayland. The compositor is
> > always in full control of all presentation. Using a hardware overlay is
> > an internal decision that the compositor does every output frame. This
> > way the compositor will composite when it has to, and use hardware
> > resources when it can. This leads to optimal usage of display hardware
> > resources without complicated negotiations with clients.
> 
> From what I understand vdpau have a hardware presentation queue. I
> wonder how that will integrate with a compositor.

Would be interesting to read about that, where did you see it?

I can't immediately imagine what kernel interface would expose such a
thing. DRM KMS is not it, at least.

> When I play a video in fullscreen I want it to be played perfectly.
> The hardware frame queue in vdpau helps making that possible even if the
> computer CPU have other things to do. So forcing only compositor to
> handle this might make movie viewing less good than with vdpau today
> under X11.

Well, DRM KMS does not support queueing, and KMS is how we drive e.g.
the hardware overlays, so until that happens, I don't see a way to
support it on Linux.

The Presentation queueing is intended to fully support hardware queues
if we get those some day. There just isn't a way for a compositor to
build that queue in the kernel/hardware yet AFAIK.

> > > Before I got a modern LCD tv which included frame interpolation the
> > > world was simpler for display of video. But now you have to display
> > > frames at correct vsync or the video will jump which is bad for viewing.
> > 
> > I fully intend you to be able to target the correct vsync when you so
> > wish and the vsync is also predictable by the client.
> > 
> > That's a good point about TVs.
> > 
> > Does frame interpolation have any overlap with dynamic vsync? I mean,
> > can they exist at the same time in the same display device? Would that
> > ever make sense?
> 
> Frame interpolation can be used by TVs (and probably monitors) today
> that have fixed refresh rate. I do not think it can be done with dynamic
> vsync that well. To be able to interpolate you have to have at least two
> frames (to my knowledge). With dynamic vsync there can be a long time
> between two frames and you cannot delay the frames displayed to much so
> that will complicate things. Also I think it is easier to make hardware
> for frame interpolation for fixed frame rates.
> 
> So my guess is that for fixed frame rates, frame interpolation will be
> used. But with dynamic vsyncs it will be turned off.

That would be a relief.


> > > Having a compositor complicates things.
> > > When running i full screen mode, the video player should be totally in
> > > control of refresh rate and when to show frames. This works fine for
> > > both fixed and dynamic refresh rate.
> > 
> > No, that's not how Wayland is designed. We cannot give direct hardware
> > access to clients. The purpose of a window system is to make apps
> > cooperate. No client can overrule the compositor decisions. It protects
> > users from malfunctioning apps, and it also improves security.
> 
> Yes, that is nice. Though I think allowing client side window
> decorations will make apps cooperate less regarding giving the user a
> consistent way to manipulate windows.
> 
> 
> > But, e.g. for fullscreen windows, we have had the idea for a client to
> > say that they would prefer a certain video mode and refresh rate. It is
> > still up to the compositor to decide when it honours that, if at all.
> > This would be part of the public shell protocol.
> 
> For movie playing, especially if it is done to a screen that does frame
> interpolation (like my TV), it is extremely important that correct
> refresh rate is used and that each frame is displayed at correct vsync.
> Otherwise frame interpolation will miss frames needed to calculate the
> interpolated frames resulting in jumpy video.

That would then be a bad compositor. Bugs happen, specs cannot prevent
that. The proper remedy there is to fix the compositor, not let every
random app go and dictate how the display hardware runs like they do in
X11 and then fight each other and the DE.

> > > As a side note - do not know if it is fixed in Wayland - when I am using
> > > Gnome 3.10 with several screens their compositor syncs updating to the
> > > screen with lowest refresh rate. I hope this is not so with Wayland. I
> > > have a LCD tv running at 24 Hz, and a normal monitor running at 60 Hz.
> > > It gets really bad to have the compositor just updating at 24 hz on the
> > > 60 hz screen. The correct solution is to update each screen with the
> > > refresh rate of the screen - more complex but much nicer to look at.
> > 
> > That is mostly a compositor issue. Wayland is largely irrelevant there.
> > (However, for the problem under X, X.org is relevant, and X11 maybe.
> > Maybe you wouldn't have that problem if the monitors were separate
> > SCREENs, i.e. zaphod mode, but with the other caveats it brings.)
> 
> Currently I run my screens as separate screens just because of the above
> problem. Unfortunately som window managers (for example gnome-shell) and
> applications do not understand "separate screens" anymore and do stupid
> things.
> 
> 
> > Compositors are perfectly able to refresh each display individually,
> > and that is what Weston does. There is a repaint loop per output.
> 
> Good.
> 
> > > If I have queued a frame for update, can I cancel that if it has not
> > > been display yet? vdpau is missing that so if you queue several frames
> > > in advance you cannot stop displaying immediately.
> > 
> > Yes, you can cancel, but in the proposal the only option is to cancel
> > all queued frames. Would be a lot more complicated to allow cancelling
> > individual frames and there hasn't been a use case for it yet.
> 
> I think that is good enough. The case where I know cancelling would be
> good is when the user pauses the video - then you want to cancel all
> queued frames as you want to pause to occur immediately.

Indeed.

> Hopefully NVIDIA will soon make it possible to run Wayland with their
> binary driver so I can try it on my main machine.

Thanks,
pq