wl_output ambiguity, xdg_shell fullscreen refresh rate

[Philipp Kerling]

Am Dienstag, den 16.05.2017, 12:28 +0300 schrieb Pekka Paalanen:
> On Mon, 15 May 2017 17:31:57 +0200
> Philipp Kerling <pkerling at casix.org> wrote:
> 2017-05-15 (月) の 11:18 +0300 に Pekka Paalanen さんは書きました:
> > > On Sun, 14 May 2017 14:43:44 +0200
> > > Philipp Kerling <pkerling at casix.org> wrote:  
> > > > To start off:
> > > >  * Kodi should offer the user the opportunity to select inside
> > > > the
> > > >    application on which monitor he/she wants to have Kodi
> > > > displayed
> > > > on
> > > >    when in full-screen mode. Now I see that I can do this with
> > > > either
> > > >    wl_shell_surface::set_fullscreen and
> > > > zxdg_toplevel::set_fullscreen,
> > > >    which expect a wl_output to display the window on. But I am
> > > > not
> > > > sure
> > > >    how I should match the wl_output instance I get from Wayland
> > > > with
> > > >    the monitor that the user selected. As far as I can see, I
> > > > can
> > > > only
> > > >    identify the monitor by way of "make" and "model". This
> > > > seems
> > > >    reasonable enough at first glance, but I think that in
> > > > multi-
> > > > monitor 
> > > >    configurations it is not so uncommon to buy the same monitor
> > > > model
> > > >    multiple times - I personally have done this. Naturally, the
> > > > "make"
> > > >    and "model" of both monitors are identical, so I cannot
> > > > discern
> > > >    which is which by looking at the wl_output metadata. Am I
> > > > missing
> > > >    something here?  
> > > 
> > > That is all true, you are not missing anything.
> > > 
> > > It is a missing piece of protocol - I believe no-one has had both
> > > the
> > > need and the time to work on getting anything better yet.
> > >   
> > > > Why isn't e.g. the connector the output is connected
> > > >    to exported?  
> > > 
> > > Because connector names are specific to DRM while Wayland (core
> > > protocol at least) is not.  
> > 
> > I'm not sure I agree. See further below.
> > 
> > > Also, would you not want to identify the
> > > monitor rather than which connector it happens to be plugged
> > > in?   
> > 
> > I think that it's not quite decided yet what we want the user
> > experience to be on the Kodi side, i.e. do we want to bind output
> > to
> > the physical monitor (however it is identified) or the GPU
> > connector?
> > You can probably find use cases for both, so it's not an easy
> > decision
> > to make. With X11 it was limited to the latter, so my first
> > approach
> > was to mimic that. Ideally, both would be possible with Wayland and
> > the
> > application can decide what is best for its specific use case. I
> > realize that this might be too much to ask for all at once, so I'll
> > talk to my mentor to decide which case we want to focus on for
> > Kodi.
> > There are also two different points involved in this discussion:
> >    I. How the *user* identifies which output/monitor the fullscreen
> >       display should be on (e.g. with make+model+connector name, or
> > by
> >       compositor-specific numbering).
> >   II. How *Kodi* or any other application then stores this, re-
> > identifies
> >       this output/monitor and maps it to a wl_output on application
> > start
> >       and hotplug. Possiblities include saving information
> > pertaining to
> >       the output, the connector, the physical device, or any
> > combination.
> 
> Hi,
> 
> very much agreed.
> 
> > > Would
> > > a user even know which connector name matches to which monitor?  
> > 
> > Yes, I also thought about this. Tech-savvy users might actually be
> > able
> > to identify the connector name, but you are absolutely right that
> > usually users should not be expected to. In fact, having make and
> > model
> > available actually is an improvement in that regard (on X11, Kodi
> > *only* displays connector since that is what's available via
> > RANDR).
> > So: How do you suggest to identify a particular monitor then? As I
> > said, make and model are not sufficient - but serial number will
> > not
> > help the user in identification. If there are multiple monitors of
> > the
> > same model, I do not think that they (i.e. the physical devices)
> > can be
> > reasonably discerned in a way that is transparent to the user.
> > That's
> > why I suggested the connector name.
> 
> One idea coming to mind is that in the DE's (compositor's) display
> configuration dialog, the user would be allowed to identify monitors
> by
> visual clues, and would be given the opportunity to give the monitors
> (or connectors!) custom names. An application could then show all the
> information plus the custom names so that the user can pick the one
> he
> wants.
What do you think makes more sense here - having the compositor provide
this information such as the custom names to the application which can
then display a selection box to the user (with the option to ask the
compositor to briefly display said information on all screens as
overlay), or having the compositor provide the whole UI for output
selection on application request and then return the wl_output (or
whatever) that was selected?

> > > To expose such a name in the protocol, we should define the space
> > > or
> > > meaning of the names. Human-readable? That's not the connector
> > > name,
> > > IMHO. Machine-parseable? Needs a very strict definition to work.
> > > Both?  
> > 
> > RANDR output names (which are somewhat close to what I refer to as
> > connector names - it's a matter of debate whether output or
> > connector
> > name would be more fitting for the Kodi use case) are specifically
> > meant to be "presented to the user", but I can see your point. I
> > think
> > that machine-parseable is doable, also considering that RANDR did
> > have
> > a "ConnectorType" property which is probably part of what we'd want
> > here. 
> 
> I think you are right and we have to cater for two very different
> cases: how a user would identify a connector or a monitor, and how a
> program would recognize a previously chosen connector or monitor.
> It's
> also possible one would need to let the user decide if one is
> matching
> the connector or the monitor.
> 
> When thinking about naming schemes, it easy to forget about multiple
> DRM display devices. I don't think DRM connector names are unique
> globally, they are unique for a single DRM device. Any global
> naming needs to somehow identify the DRM device too. That's
> something the compositor needs to solve, but it has some consequences
> on how the protocol specification should be written.
You're absolutely right, I was totally missing that.

> 
> DRM drivers can be initialized in random order, so one cannot rely on
> the ordering.
> 
> > > A simple numbering with a way in the compositor to identify the
> > > monitors? Maybe.  
> > 
> > What exactly do you mean by "a way in the compositor to identify
> > the
> > monitors"? Like you can tell the compositor to display the output
> > names
> > (briefly) on the outputs so the user knows which is which?
> 
> Yes.
> 
> > If so, couldn't this also be done by the application at least with
> > more
> > or less arbitrarily assigned numbers by mapping a surface to each
> > known
> > output that displays said number?
> 
> Not conveniently, and not in a way guaranteed to work. As I wrote in
> my
> other email, Wayland clients ask to do things, but the compositor may
> decide otherwise. E.g. it is not guaranteed a new window will be
> placed
> top-most to be visible. Clients also cannot control the positioning
> much.
Yes, considering this it would probably be too error-prone. I also
realized just now that while the definition of
wl_shell_surface::set_fullscreen forced the compositor to honor the
requested output provided by the client, this was dropped in
xdg_shell::set_fullscreen, so even that cannot be relied on.

> 
> > I'm beginning to think that something along those lines might be
> > one of
> > the more user-friendly options of solving I. above.
> 
> Yes, but the visual cues for identifying the monitors really must be
> in the compositor. Also, once it has been saved in the compositor,
> e.g.
> by assigning custom names, there is no need to ever do it again until
> monitors change. IOW, I believe identification is compositor
> configuration, not something every app should do on its own.
Agreed.

> 
> > > > This would allow way better matching and also give the
> > > >    user a chance to select the right monitor without having to
> > > > try
> > > > all
> > > >    options. Serial number (or anything similar - not sure if
> > > > the
> > > > EDID
> > > >    has this though) could also be OK.  
> > > 
> > > EDID may have a serial number, yes.
> > > 
> > > Physical monitor devices are not the only things though. An
> > > output
> > > could be just a window in another window system. An output could
> > > be
> > > at
> > > the end of a streaming network connection. Etc. So aiming for a
> > > generic
> > > solution here would have a really wide scope.  
> > 
> > Also answering the first question above: I don't see how "make" and
> > "model" (and the mode list for that matter) in wl_output fit with
> > this
> > statement. They are also limited to traditional physical devices
> > (monitors), yet are in the core protocol. I don't suppose that it
> > makes
> > sense to classify an output in another window system with a
> > manufacturer and model. That said, I also don't see why "connector"
> > (or
> > something similar) would be fundamentally different from what is
> > already there.
> 
> You're absolutely right. wl_output is one of the oldest designs in
> Wayland, and carries quite many things that could have been designed
> better in hindsight. Let's not make it any worse than it already is.
> 
> Of course, one could put things like "rtp://..." or "Window 0x65" in
> connector attribute, just like one can put arbitrary things in make
> and
> model, but could we not do better? Or do we not need to do any
> better?
> 
> I'm honestly unsure.
> 
> However, one thing is certain: if we extend wl_output interface, it
> will be immediately set in stone forever, because it's in
> wayland.xml,
> and wl_surface references wl_output objects. There will be no turning
> back. The only way to deprecate things is to say they have become
> meaningless, but they can never be actually removed.
> 
> If instead we write another protocol extension, that extension is
> much
> easier to evolve and can be deprecated and even removed and replaced
> if
> necessary. It will also be optional to implement, for better and
> worse.
I can understand your concerns and you have convinced me that the
connector or output name should not be put there.

> > > However, there already is a unique numbering for wl_outputs, even
> > > though it's not exactly user-friendly. It is the wl_output "name"
> > > advertised by wl_registry, which you use in wl_registry.bind
> > > request
> > > to
> > > create a wl_output instance. This "name" is the same for all
> > > clients.  
> > 
> > Is this name (which is usually the integer number of the protocol
> > in
> > enumeration order if I got that right) *guaranteed* to stay the
> > same
> > (for a given wl_output)
> >  * when I restart the application?
> 
> Yes.
> 
> >  * when I restart the compositor?
> 
> No.
> 
> >  * when I restart the computer?
> 
> No.
> 
> >  * when I connect additional monitors?
> 
> Yes, already existing wl_outputs do not change their "name".
> 
> >  * when I rearrange monitors in the compositor?
> 
> Depends on what the compositor does.
> 
> >  * when I start the application with the monitor unplugged and plug
> > it
> >    in later (while the app is running)?
> 
> No.
> 
> If you unplug an re-plug a monitor into the same connector, the
> "name"
> will be different.
Thanks for the detailed evaluation. I gather from this that there is no
use relying on it for output/monitor identification across restarts.

> 
> "For a given wl_output" is a bit awkward here, because the "name"
> defines the wl_output identity. I assume you wanted to refer to a
> connector or a monitor instead.
Yes, sorry - I wanted to refer to the actual physical output (however
you want to define that) behind the wl_output instance, not to the
transiently bound Wayland protocol.

> > If not, this would not be useful to Kodi. Whether other clients see
> > the
> > same "name" or not is not especially relevant here. The specific
> > use
> > case is to save the monitor/output identifier to the configuration
> > file
> > and have it persist across both app and device restarts and output
> > configuration changes. We would especially like to support the last
> > point of the enumeration above, i.e. starting on a fallback
> > (unspecified) monitor and then switching fullscreen output to the
> > preferred monitor once it is connected by the user.
> 
> Right.
> 
> What about across different compositors?
> 
> If a user runs Kodi on one compositor and configures it, switches to
> another compositor, do you still expect the saved configuration to be
> valid?
> 
> I assume you do
Actually, I don't :-) But that is just my personal opinion and has not
been discussed within Kodi yet. Also, if this becomes an extension, it
should obviously consider a variety of use cases not limited to Kodi.

> , which makes it crucially important to define in the
> protocol specification on what names or other information to put in
> wl_output attributes. When one needs to be so excruciatingly accurate
> in the specification, you pretty much want to restrict the scope to
> e.g. DRM display devices only or refer to specific EDID fields.
Yes. If this were to be supported, the specification will have to be
very accurate. But at the same time, it will get increasingly platform-
specific (to DRM in this case). Which is in principle OK for a protocol
extension I guess, but it also comes with problems. Assuming that there
will be additional specific extensions for other platforms,
applications would have to implement each one and have the
identification code depend on the platform. Or maybe there is a way to
write the specification such that identification by the app is always
possible by string matching (without having the application interpret
the contents), I don't know.

I'm really not sure what to do here. On the one hand, I understand that
users that e.g. frequently switch between, say, GNOME and KDE would be
annoyed if they have to reselect the output each time they switch
compositors - in every application that uses this theoretical
interface. More so because "it worked with X". On the other hand, it
will make the protocol and implementation a lot more difficult.
Opinions?

Also, if we're identifying the physical device, this information will
not be necessary at all. Instead, we'd have to use the tuple
(make,model,serial number), which also has the problem of being
composed of arbitrary text strings. I'd expect most compositors to read
it out of the EDID, but the protocol does not mandate this. Making sure
this reliably works would also require an extension that provides this
information but also mandates rules how they should be obtained by the
compositor.

> > > What we would really need is a new protocol extension designed to
> > > cater
> > > for specific use cases and offering the additional information to
> > > help
> > > make user-friendly UIs. Maybe compositors should a) number of the
> > > outputs, and b) let the user give them arbitrary names to be
> > > shown in
> > > application UIs?  
> > 
> > I don't see the real benefit in b), it is IMHO too much effort for
> > the
> > average user. I'm also not sure what I would personally label my
> > screens as besides "Left screen" and "Right screen", which the
> > compositor could as well figure out on its own :-)
> 
> It would be an option to customize, in case the default names
> manufactured from DRM connector, make, model etc. are not good
> enough.
> Yes, "left screen" and "right screen" are also good default names,
> but
> it still means that the protocol specification has to have a way to
> communicate them to clients.
Ah, this clears up a lot. This is only necessary to communicate if the
applications themselves render the output selection UI (see above),
yes?

> 
> > > But then, do you tie the output identity to the
> > > connector or the physical device? Or an intelligent mix?  
> > 
> > As I said, best is probably if the application has enough
> > information
> > to be able to do either.
> > 
> > > Would it be in scope for your work to design such an extension?
> > > It's
> > > relatively easy to write a protocol proposal, but getting it
> > > reviewed
> > > and hopefully ack'd by more than one DE project is often hard and
> > > time
> > > consuming. And then you'd need to provide an implementation on at
> > > least
> > > one compositor, unless you find someone else to collaborate with
> > > you.  
> > 
> > If the consensus is that this will, in some form or another, need a
> > protocol extension, I'll have to discuss with my mentor whether
> > this
> > can be pursued as part of my GSoC work, especially the reference
> > implementation part. I think that it is a possibility, but does
> > require
> > that there is a realistic expectation that at least mutter or KWin
> > will
> > support the extension in the future. Otherwise it could be very
> > hard to
> > justify.
> 
> I do believe an extension is necessary sooner or later. We can also
> use
> Weston for proving a sample implementation, but that won't help your
> users. If you decide to tackle this, I warmly recommend contacting
> the
> DE projects ASAP. Who knows, maybe someone has started on something
> already.
Thanks for the hint. Will do.

> 
> > > >    quite a problem. I can see that zxdg_toplevel functionality
> > > > is
> > > >    mostly superior to that of wl_shell_surface, but it has one
> > > > omission
> > > >    that is crucial for Kodi: the ability to request a specific
> > > > refresh
> > > >    rate for fullscreen display. This is needed for closely
> > > > matching
> > > > the
> > > >    display and video FPS so duplicated and skipped frames are
> > > > kept
> > > > to a
> > > >    minimum. Is this an intentional omission and/or is there
> > > > anything
> > > >    that provides this functionality?  
> > > 
> > > Hopefully the xdg_shell designers would answer that, but I
> > > believe it
> > > has been omitted for now to make it easier to declare the first
> > > fundamental parts of xdg_shell stable. It is missing a lot, but
> > > the
> > > foundation must agreed upon first before building more on top.  
> > 
> > So is there any realistic chance to have such functionality
> > integrated?
> > What steps would have to be done?
> 
> My personal piece of advice: do not make it a blocker for your own
> work. xdg_shell has been in development for years by now and still
> has
> not been declared stable.
Yep, I don't intend to sit around and wait for this. Still, I think it
is an important missing piece of functionality missing to make video
players and media centers work great on Wayland.

Actually, what is the roadmap for xdg_shell?
If Kodi depends on zxdg_shell_v6 being there (like most toolkits seem
to do), what will happen once it gets moved to stable?
Should I implement using wl_shell_surface as fallback to be sure?

> I don't think people would disagree much that some way of
> communicating
> video timing related information would be good, but beyond that I
> can't
> tell. It might be best to restrict this out of your project scope.
> 
> For example: variable refresh rate monitors exist and the software
> infrastructure is slowly building up, I believe. There it is
> questionable if one would want windows to declare their optimal
> refresh
> rates, or just submit content with target timestamps for the
> compositor
> to arrange everything.
> Some years ago when I was working on the
> Prensentation timing extension, it also included a feature to
> queue window content updates with target timestamps. That feature was
> hard to define well, and eventually I abandoned it to get at least
> the
> feedback part stabilized, which is the current Presentation timing
> extension.
It'd be great to see support for variable refresh rate come to the
Linux graphics stack. But realistically, fixed refresh rate displays
and infrastructure will still be with us for the foreseeable future and
must be supported as such by compositors and media centers and the
combination of both to the best of their abilities.
So, I don't think it would be harmful to declare the FPS on the surface
anyway if that information is available.
Actually, that might allow the compositor to correctly drive the
variable refresh rate display correctly without any additional timing
information or queued updates. At least in special cases such as full-
screen output it could just reduce the update rate to the requested FPS
(similar to what would happen if an actual mode-switch had occured on a
fixed refresh rate display).
That is not to say that queued updates etc. are unnecessary, but it
could allow applications that exist now to work on variable refresh
monitors without modification.

Again thanks a lot for your precious time,
Philipp