[PATCH 0/2] Support for high DPI outputs via scaling

[John Kåre Alsaker]

On Mon, May 13, 2013 at 8:52 PM, Jason Ekstrand <jason at jlekstrand.net>wrote:

> On Mon, May 13, 2013 at 9:54 AM, Alexander Larsson <alexl at redhat.com>wrote:
>
>>
>> On mån, 2013-05-13 at 14:40 +0200, John Kåre Alsaker wrote:
>>
>> >
>> >         I don't think this will work in practice. I know for sure that
>> >         e.g. Gtk
>> >         is not set up to do any reasonable non-integer scaling. It
>> >         will just
>> >         scale up all drawing by a fractional factor without any
>> >         rounding
>> >         anywhere, causing everything to become fuzzy. We will
>> >         eventually have
>> >         alternative icons for higher resolutions, but these will be at
>> >         2x scale,
>> >         not at generic fractional factor (that is not really doable
>> >         without
>> >         using pure-vector icons with some complex hinting method).
>> >         Although, I
>> >         guess that in the case of text the scaling will be ok, so for
>> >         some
>> >         usecases it might be OK.
>> > It will work very well for things designed to be scalable, browsers
>> > are an example. GTK could just fall back to integer scaling.
>>
>> Browsers are not really scalable like that, css page layout is generally
>> based on
>> the CSS "Px" definition, and per e.g.
>> http://static.zealous-studios.co.uk/projects/web_tests/PPI%20tests.html:
>>
>>         For lower-resolution devices [i.e. non-print], and devices with
>> unusual
>>         viewing distances, it is recommended instead that the anchor unit
>> be the
>>         pixel unit. For such devices it is recommended that the pixel
>> unit refer
>>         to the whole number of device pixels that best approximates the
>>         reference pixel.
>>
>> I.e. "whole number of pixels" => scale by integer matches best what CSS
>> wants.
>>
> My point was that browsers can scale up websites by fractional amounts
even though it will look somewhat better with integer factors.


>
>> >         So, it's my opinion that supporting fractional scaling is an
>> >         unnecessary
>> >         burden on compositors for something that is not very useful in
>> >         practice.
>> >         Thats just my opinion though, and the proposal as such can
>> >         handle
>> >         fractional scaling fine by just changing the scale factors to
>> >         fixed
>> >         type.
>> > It is of no burden on compositors at all, only for clients which
>> > choose to implement it.
>>
>> It is, as long as clients are allowed to specify a fractional scaling
>> factors compositors need to be able to handle scaling by that (for
>> instance if the window appears partially on a non-highres monitor.
>>
>
> While this isn't a huge problem on GL-based compositors it will cause a
> problem for software compositors.  Any scaling for that matter is a
> potential problem there.  However, a factor of two or something shouldn't
> be too bad in software.
>
Compositors do not have to scale anything at all. Even with fractional
scaling factors a compositor could limit itself to just scaling with
integer factors. I don't think software compositor is a case worth
considering either. Nobody wants to have those, especially with high-DPI
displays.


>
>
>>  >         > In mirrored/clone mode only a single wl_output would be
>> >         presented to
>> >         > clients with a single scale factor, so priorities won't
>> >         matter in that
>> >         > case.
>> >
>> >
>> >         That is not necessarily so. We might very well want to know
>> >         that there
>> >         are two displays, with say different RGB subpixel order, etc.
>> > The compositor could unify this information into a single wl_output.
>>
>> And what would it then report for physical width/height, subpixel, make,
>> model, etc? Seems pretty weird.
>>
> Even if two wl_outputs would be presented, they would have the same
resolution and scaling factor so priorities still won't matter.


>
>> > I suggest we send 3 scaling factors to clients. A lower bound where
>> > factors lower will be scaled up. The desired scaling factor. A upper
>> > bound where factors above will be scaled down. Clients should find the
>> > scaling factor in lower bound to upper bound they are able to render
>> > at which is closes to the desired factor. Failing that they should
>> > find the first factor they are able to render at above the upper
>> > bound. When displayed on multiple monitors it could try to remain
>> > sharp on as many monitors as possible.
>>
>> I don't quite understand this. Any factor lower than the "desired"
>> factor has to be scaled up, no?
>>
> I expect a compositor to render deviations of the desired scaling factor
without scaling windows. The range when this is allowed is reported to
clients so they can try to render at a size which will avoid scaling.

For example a compositor may want to use a 1-1.2 range with 1.1 as the
desired scaling factor. A clients which are only able to draw at integer
scaling factor would round that up to 2 and let the compositor downscale
it. When the range for which compositor won't scale is send to clients we
can avoid this.


>> Lets take a concrete example. The macbook pro 15" has a 2880 x 1800
>> native panel. This is "normally" driven as 1440 x 900 with a scale
>> factor of 2. On OSX you can also select a 1920x1200 resolution, which is
>> nominally a scale factor of 1.5. However, on OSX what they do is that
>> apps render to a framebuffer at twice the resolution (3840x2400) and its
>> then scaled it down to 2880x1800 to make it look reasonable.
>>
> Rendering at 3840x2400 then scaling down to 2880x1800 is bad for
performance and makes the result unsharp. It is much preferable that
clients render into 2880x1800 directly with a 1.5 scaling factor (for
clients that are capable of doing that).


> Actually, this seems like a fairly reasonable way for the client to
> render.  It can simply render at a sufficiently high integer multiple of
> the resolution it wants to look really good on such that it's always
> downscaling.  Or, if it's worried about memory, let the compositor
> upscale.  However, the key here is that the scale factor is an integer
> multiple of *one* of the monitor scale factors.  Whether or not the monitor
> scale factor is an integer doesn't matter for this.
>
We can't really downscale with integer multipliers, just upscale. If we
downscale, rendering at a (much) higher resolution doesn't really help
much. We want to avoid upscaling though.


>
>> On such a setup in wayland the wl_output would be 1920x1200, with a
>> scaling factor of 1.5. For a 800x601 window a hidpi applications could
>> then either pick a buffer of 1600x1202 buffer and set a scaling factor
>> of 2 causing the compositor to downscale it (similar to OSX above), or
>> it could create a buffer at 1200x902 (nearest to the true 901.5) and and
>> set scaling factor to 902/600 (i.e. 1.50333..) and have the compositor
>> not scale it (which is a minor issue due to the 1.5 != 1.50333..).
>>
>> In this case the desired scaling factor would be 1.5. What do you expect
>> the upper and lower bounds to be?
>>
> If the compositor allows deviations in area of 10%, I'd expect the lower
and upper bounds to be 1.42 ((1.5² * 0.9)^½) and 1.57 ((1.5² * 1.1)^½)
respectively.


> I don't know if the upper/lower limits is the solution, but this is
> certainly an issue.  In a case like that, I would say that the application
> simply works in the coordinates of the "prefered" output.  This includes
> input events etc.  This may mean that the size of the window in points
> isn't an integer.  If the toolkit doesn't want to deal with that, they can
> come up with a solution (maybe just require a multiple of 2 for the size in
> this case).
>
The size of a window should always be in pixels at the protocol level.


> Allow me to attempt to restate something I tried to point out in an
> earlier e-mail (now that I understand the problem better).  As a
> disclaimer, while I have worked with a number of toolkits from the
> client-side, I am not a toolkit developer.  Feel free to tell me when there
> are hidden issues that have to do with toolkit innards (but please be
> specific so I can understand the exact problem).
>
> One line that I think needs to be drawn more distinctly here is what
> happens at the application level, the toolkit level, and the compositor
> level.  At the application level, we would rather they not even notice the
> change unless the want to.  While it might be convenient at times to know
> this information in a GTK or Qt app, apps should work nicely on hidpi
> displays without any modification.  They will probably need
> higher-resolution or multi-resolution icons, but other than that they
> should "just work".
>
> Where to separate the toolkit from the compositor becomes more of an
> interesting issue.  The only place I have seen multiple densities handled
> well is Android (Windows 8 does something, but they're using HTML5/CSS so
> they can cheat a bit).  On Android, they have a concept of "display pixels"
> or dp which basically corresponds to "points" in your above proposal.  When
> you lay out an android UI, you do so in display pixels and then the toolkit
> automatically handles scaling everything.  Android basically has 4
> different "scale factors": 0.75, 1.0, 1.5, 2.0 (I think there's a 3.0 for
> some of these crazy 450 DPI phones).  The reason why Android's system works
> so well (and looks so good) is that all the scaling is done at the toolkit
> level and they don't really have to bother with multiple outputs.
>
> Unfortunately, we do have to bother with multiple outputs.  We all know
> how apple solved this and it assumes that there will only ever be two
> screen densities in existence.  Since apple controls their hardware, they
> could assume this.  We cant.  I think what I would suggest is a hybrid
> toolkit/compositor approach as follows.
>
> 1. Apps simply define everything in points with the understanding that it
> may not directly correspond to pixels on the display.  For icons and other
> images, they can provide multiple versions or use vector graphics or
> something.
>
> 2. The toolkit picks a "nice" scaling factor to go between pixels on its
> preferred output to points for the client.  By "nice" I mean that you'd
> rather scale by 1.5 than, say, 1.587.  It then renders at the resolution of
> the preferred output (or possibly an integer multiple so other outputs
> downscale instead of upscale).  When it hands the buffer to the compositor,
> it gives the compositor a (probably integer) scale factor relative to a
> particular output.  Yes, this means the toolkit may have to ceiling or
> floor some values to make everything fit nicely.  However, the point is
> that the scale factor presented to the user doesn't have to match what
> happens in the toolkit.
>
> 3. The compositor then takes the image provided by the toolkit and scales
> it for all of the outputs.  On the toolkit's preferred output, it shouldn't
> scale at all (or possibly an integer multiple).  On the other outputs, it
> would scale based on how the outputs are scaled relative to each other and
> the provided scale factor.  Whether the compositor sends events in pixels
> or points probably doesn't matter that much (up to rounding error).
> However, I think I would vote for pixels on the client's preferred monitor
> so that pixel-perfect clicking is preserved.
>
> There are a couple of other things I think are worth mentioning here as
> well.  First, is that the client can't know how much of it is on which
> surface.  There is nothing in the protocol (a this point) to indicate that
> beyond which outputs it is on.  Surfaces don't know their absolute
> positions.
>
> Second, we are going to have to deal with fractional scaling even if we
> restrict to integers.  Consider the following example.  Say I have two
> monitors: A and B which are at 200 DPI, and 300 DPI respectively.  An
> application has a number of choices for resolution:
>
>  - It could render at 200 DPI and then scale by 3/2 for B.
>  - It could render at 300 DPI and scale by 3/2 for A.
>  - It could render at 100 DPI and scale by 2 for A and 3 for B and look
> bad on both
>  - It could render at 400 DPI and scale by 3/4 for B and 1/2 for A
>
> and I'm sure you could come up with more.  However, the point is that
> unless we are going to make the extremely restrictive assumption that
> monitors come in powers of two, we cannot get around fractional scaling.
>
Yes, we do need fractional scaling even without fractional scaling factor
to give a good experience.


> Thanks,
> --Jason Ekstrand
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/wayland-devel/attachments/20130514/d494ec48/attachment-0001.html>