[PATCH] protocol: Add buffer_scale to wl_surface and wl_output

Wed May 15 01:13:11 PDT 2013

On Tue, 14 May 2013 12:26:48 +0200
alexl at redhat.com wrote:

> From: Alexander Larsson <alexl at redhat.com>
> 
> This adds wl_surface_set_buffer_scale() to set the buffer scale of a
> surface.
> 
> It is similar to set_buffer_transform that the buffer is stored in a
> way that has been transformed (in this case scaled). This means that
> if an output is scaled we can directly use the pre-scaled buffer with
> additional data, rather than having to scale it.
> 
> It also adds a geometry2 event with a scale member to wl_output that
> specifies the scaling of an output.
> 
> This is meant to be used for outputs with a very high DPI to tell the
> client that this particular output has subpixel precision. Coordinates
> in other parts of the protocol, like input events, relative window
> positioning and output positioning are still in the compositor space
> rather than the scaled space. However, input has subpixel precision
> so you can still get input at full resolution.

I think I can understand this paragraph, but could we express it more
clearly?

"Output with sub-pixel precision" could probably use some explanation
about what it is here, like is it about sub-pixels in the RGB pixel
parts sense. Can this be used for RGB-sub-pixel things somehow?

"Compositor space" and "scaled space" need to be clearly defined, I am
not sure what they refer to here. The well-known coordinate spaces we
already have are surface (local) coordinates, and global coordinates.
Output coordinates likely too, and buffer coordinates will be
introduced with the surface crop & scale extension[1] to further
differentiate from surface coordinates.

> This setup means global properties like mouse acceleration/speed,
> pointer size, monitor geometry, etc can be specified in a "mostly
> similar" resolution even on a multimonitor setup where some monitors
> are low dpi and some are e.g. retina-class outputs.
> ---
>  protocol/wayland.xml | 41 +++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 39 insertions(+), 2 deletions(-)
> 
> diff --git a/protocol/wayland.xml b/protocol/wayland.xml
> index 3bce022..e5744c7 100644
> --- a/protocol/wayland.xml
> +++ b/protocol/wayland.xml
> @@ -876,7 +876,7 @@
>      </event>
>    </interface>
>  
> -  <interface name="wl_surface" version="2">
> +  <interface name="wl_surface" version="3">

You have to bump also the wl_compositor version. wl_surface is not a
global, and only globals can have their interface version negotiated.
The global that can create wl_surface objects is wl_compositor, and
wl_compositor only.

The version of wl_surface in use will be implied by the negotiated
version of wl_compositor.

(Yes, it is a bit strange perhaps, but that is how it is.)

>      <description summary="an onscreen surface">
>        A surface is a rectangular area that is displayed on the screen.
>        It has a location, size and pixel contents.
> @@ -1110,6 +1110,30 @@
>        </description>
>        <arg name="transform" type="int"/>
>      </request>
> +
> +    <!-- Version 3 additions -->
> +
> +    <request name="set_buffer_scale" since="3">
> +      <description summary="sets the buffer scale">
> +	This request sets an optional scaling factor on how the compositor
> +	interprets the contents of the buffer attached to the surface. A
> +	value larger than 1, like e.g. 2 means that the buffer is 2 times the
> +	size of the surface.

..in each dimension. Ok.

> +
> +	Buffer scale is double-buffered state, see wl_surface.commit.
> +
> +	A newly created surface has its buffer scale set to 1.
> +
> +	The purpose of this request is to allow clients to supply higher resolution
> +	buffer data for use on high-resolution outputs where the output itself
> +	has a scaling factor set. For instance, a laptop with a high DPI
> +	internal screen and an low DPI external screen would have two outputs
> +	with different scaling, and a wl_surface rendered on the scaled output
> +	would normally be scaled up. To avoid this upscaling the app can supply
> +	a pre-scaled version with more detail by using set_buffer_scale.

You could also mention, that it is expected that clients will use
an output's scale property value as the set_buffer_scale argument. Or
at least that is the intended use here.

> +      </description>
> +      <arg name="scale" type="fixed"/>

Are you sure you really want fixed as the type?
Integer scaling factors sounded a lot more straightforward. When we are
dealing with pixel buffers, integers make sense.

Also, I do not buy the argument, that integer scaling factors are not
finely grained enough. If an output device (monitor) has such a hidpi,
and a user wants the default scaling, then we will simply have an
integer scaling factor >1, for example 2. Clients will correspondingly
somehow see, that the output resolution is "small", so they will adapt,
and the final window size will not be doubled all the way unless it
actually fits the output. This happens by the client choosing to draw a
smaller window to begin with, not by scaling, when compared to what it
would do if the default scaling factor was 1. Fractional scaling factors
are simply not needed here, in my opinion.

Can we have any use for scales less than one?

In other words, can you even imagine a case, where the compositor would
by default down-scale all surfaces? I cannot, since I think that default
behaviour could only make the image worse, even if it could fit more on
screen. Rather than that, clients know the current output resolution,
and need to adapt to it directly by initially not making a window too
large.

Also, one issue raised was that if an output has a scaling factor A,
and a buffer has a scaling factor B, then final scaling factor is
rational. To me that is a non-issue. It can only occur for a
misbehaving client, in which case it gets what it deserves, or in a
multi-output case of one surface spanning several non-identical
monitors. I think the latter case it not worth caring about.
Non-identical monitors are not identical, and you get what you happen
to get when you use a single buffer to composite to both.

> +    </request>
>     </interface>
>  
>    <interface name="wl_seat" version="1">
> @@ -1467,7 +1491,7 @@
>      </event>
>    </interface>
>  
> -  <interface name="wl_output" version="1">
> +  <interface name="wl_output" version="2">
>      <description summary="compositor output region">
>        An output describes part of the compositor geometry.  The
>        compositor works in the 'compositor coordinate system' and an
> @@ -1520,6 +1544,8 @@
>  	The geometry event describes geometric properties of the output.
>  	The event is sent when binding to the output object and whenever
>  	any of the properties change.
> +
> +        Some additional properties were later added, see wl_output.geometry2.
>        </description>
>        <arg name="x" type="int"
>  	   summary="x position within the global compositor space"/>
> @@ -1565,6 +1591,17 @@
>        <arg name="height" type="int" summary="height of the mode in pixels"/>
>        <arg name="refresh" type="int" summary="vertical refresh rate in mHz"/>
>      </event>
> +
> +    <event name="geometry2" since="2">

The event needs a better name. "scale"? "default_scale"?

> +      <description summary="additional properties of the output">

The summary should refer to the scale specifically.

> +	This event contains additional geometric information
> +        that are not in the geometry event. Whenever it is sent
> +        it will be followed by a geometry event. This way you can
> +        tell by the time the geometry event is recieved whether a
> +        geometry2 event will be seen or not.

The "geometry event always follows a geometry2 event" could also be
documented generally in the geometry event: all (any?) supported
additional output properties are sent before the geometry event, and
then the geometry event indicates the end of events for this batch.

Here you should document, what this scaling factor actually means, and
then refer to wl_surface.set_buffer_scale.

> +      </description>
> +      <arg name="scale" type="fixed" summary="scaling factor of output"/>
> +    </event>
>    </interface>
>  
>    <interface name="wl_region" version="1">

We still need to define precisely how this interacts with the various
coordinate systems. Let's look at it first without the crop & scale
extension in between. If I understood right, it would go like this:

Coordinate systems the client is aware of:

A. Buffer pixel coordinates, which are essentially equivalent to byte
addresses of pixels in the buffer.

- Apply (inverse?) buffer_transform and inverse buffer_scale to get:

B. Surface coordinates, i.e. the local coordinates where all window
management and input happens.

Coordinate systems internal to a compositor:

- Apply surface transform (usually only translation, but visual effects
  may do other stuff, too) to get:

C. Global coordinates, where all surfaces and outputs are laid out.

- Apply output transform and output scale, to get:

D. Output framebuffer pixel coordinates, which correspond to the byte
addresses of pixels in the scanout buffer.

The important thing is to make all client-visible coordinate systems
consistent and logical.

Buffer size in coordinate system A is always integers, naturally. We
also want to keep the surface size in coordinate system B integers,
because the protocol already assumes that (wl_surface.attach, shell
protocol, maybe others). Therefore, buffer_scale should be an integer
to guarantee that.

When a surface comes to coordinate systems C and D, there is no
requirement to have its size or position as integers. The surface
transform, that is private to the compositor, could warp the surface
along a curve for all we care, but usually it doesn't. Usually it only
does a translation to position the surface, and does not change the
scale. Then the renderer in a compositor just does its best to realize
the total transformation from buffer pixels to framebuffer pixels.

In the best case, when buffer_scale equals output_scale and surface
transform is only a translation, the pixels from a surface's buffer are
mapped 1:1 to the framebuffer pixels.

If buffer_scale is 1, set explicitly by a client or just by default,
the buffer pixels will be scaled by the integer factor output_scale.

These two are the most important cases to my understanding.

If additionally to the best case, the translation in the surface
transform is zero, buffer transform matches the output transform, and
the buffer is the size of the output resolution, the buffer could be
scanned out directly, bypassing the compositing. Overlay hardware can
allow scanout even with some scaling.

Thinking about input events, there are no problems. Input always comes
in coordinate system B, the surface coordinates. As position/motion in
input events is always of type wl_fixed, you can address individual
pixels in a buffer up to scaling factor 256.

The usual shell operations should work as is, too.
wl_shell_surface.configure tells you the desired surface size in
surface coordinates, and then you realize that any way you want in a
client. Transient and pop-up surfaces that are relative to a parent
surface work, too, since the child surface's position and size is
negotiated and given in parent surface's coordinates. The same works
with sub-surfaces.

And now the questions.

If an output has a scaling factor f, what does the wl_output
report as the output's current and supported video modes?

What about x,y in the wl_output.geometry event (which I think are just a
global coordinate space leak that should not be there)?

The video modes are important because of the
wl_shell_surface.set_fullscreen with method DRIVER. A fullscreen
surface with method DRIVER implies, that the client wants the
compositor to change the video mode to match this surface. Of course
this usually only happens when the fullscreen surface is topmost and
active.

A client can use the list of supported video modes from the wl_output
to choose the size of its fullscreen surface. E.g. if video mode
800x600 is supported, the client may choose to make the fullscreened
surface of size 800x600, and assume that the server ideally switches
the video mode 800x600 to present the surface.

How do buffer_scale and output scale interact with this mechanism?

I'll leave the interactions with the crop & scale extension for later,
I have a feeling there is a relatively clean and logical solution.

Thanks,
pq

[1] http://lists.freedesktop.org/archives/wayland-devel/2013-April/008927.html