surface buffer cardinality and outputs

[Pekka Paalanen]

On Sat, 23 Mar 2013 14:42:51 -0400
Jerome Glisse <j.glisse at gmail.com> wrote:

> On Fri, Mar 22, 2013 at 5:51 PM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> > On Fri, 22 Mar 2013 10:41:00 -0400
> > Jerome Glisse <j.glisse at gmail.com> wrote:
> >
> >> On Fri, Mar 22, 2013 at 6:35 AM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> >> > On Wed, 20 Mar 2013 13:09:48 -0400
> >> > Jerome Glisse <j.glisse at gmail.com> wrote:
> >> >
> >> >> On Wed, Mar 20, 2013 at 4:32 AM, RenoX <renozyx at gmail.com> wrote:
> >> >> >
> >> >> >
> >> >> > On Tue, Mar 19, 2013 at 11:34 AM, Jason Ekstrand <jason at jlekstrand.net>
> >> >> > wrote:
> >> >> >> I'm not sure exactly what I think of all this surface transform
> >> >> >> passing.  I'll get back to that once I get a chance to think about it.
> >> >> >>  Part of the problem is that you don't want to try and make your
> >> >> >> animations subpixel-perfect because that would require a lot of
> >> >> >> round-trips to make it right.
> >> >> >
> >> >> > And you(Jerome Glisse) replied:
> >> >> >> It doesn't add more roundtrip, client will get next matrix like a
> >> >> >> frame event, client then draw in screen space and then draw with new
> >> >> >> transformation matrix. If client can't keep up it wouldn't either keep
> >> >> >> up in the non screen space rendering case.
> >> >> >
> >> >> > I disagree: currently Weston move windows server side only, that is to say
> >> >> > without roundtrip,
> >> >> > if you want to be 'subpixel perfect' then moving a window imply to re-render
> >> >> > it from the client, which will add a lot of roundtrip in this case.
> >> >> >
> >> >> > So the current design is 'pixel perfect', but not 'subpixel perfect', and
> >> >> > cannot be without big changes..
> >> >> >
> >> >> > regards,
> >> >> > renoX
> >> >>
> >> >> What i propose is an extension ie application can still render non
> >> >> subpixel and can still render non subpixel when moving/resizing. Note
> >> >> also that moving a window around is not so much different then
> >> >> resizing a window in which case wayland will round trip with the
> >> >> client. So really if a client want subpixel rendering at all time it
> >> >> just add roundtrip in the window is moving case.
> >> >
> >> > The server can also move surfaces on its own, without the client
> >> > initiating the move. This is move by hotkey. Therefore you would
> >> > incur at least one extra roundtrip during the beginning of such move.
> >> >
> >> >> Again i stress it's not a lot of roundtrip it's the same amount of
> >> >> roundtrip as with resize window.
> >> >
> >> > Resize is actually a huge deal. It is very very heavy. At least I
> >> > do not want to incur the same penalty to moving, now that using a
> >> > compositor has finally made moving a very cheap operation.
> >> >
> >> >> Change i propose are not big, it's a fairly small contained change, i
> >> >> will try to hack something over the weekend.
> >> >
> >> > This discussion is has lead me to conclude, that it is
> >> > near-impossible to reliably support and present sub-pixel rendering
> >> > in general without telling clients where their surfaces are on an
> >> > output.
> >> >
> >> > Not telling clients where their surfaces are has been a major
> >> > design decision in Wayland this far, so it seems Wayland is somewhat
> >> > incompatible with the concept of sub-pixel rendering. Feasible
> >> > cases are limited to just some special cases, where the output
> >> > device has a translation-invariant pixel pattern, and the
> >> > compositor does not apply any other transformations than
> >> > translation.
> >> >
> >> > With that in mind, I think it's ok to enable sub-pixel rendering
> >> > where it is feasible, but I would not expose surface positions or
> >> > transformations to clients for it.
> >> >
> >> > Therefore I'd simply propose for compositors to advertise outputs
> >> > with sub-pixel order as unknown or none, unless the output is known
> >> > to be translation invariant and surfaces are only translated.
> >> >
> >> >
> >> > Thanks,
> >> > pq
> >>
> >> I still believe that my proposal is totally doable and does not break
> >> wayland paradigm. I should stress that the compositor (wayland server)
> >> is always in charge that means that the compositor can decide to not
> >> ping pong with the application while moving the window but only update
> >> the transformation matrix and ask for a redraw from client once the
> >> window no longer move.
> >
> > This introduces temporary glitches, which we work hard to eliminate.
> > Unless you mean window outline moves instead of window content?
> >
> > (Window outline resizes are actually something I'd personally like
> > to see, would make Weston on Raspberry Pi a lot more smooth
> > experience.)
> 
> Yes temporary glitch but better rendering for the common case where
> window does not move.
> 
> >> Window being move or resized is not typical usage of desktop, by that
> >> i means user is not going to constantly move its window or resize
> >> them, it happens seldomly during a session. That being said i stress
> >> again that the compositor is in charge and if it want each frame
> >> sub-pixel perfect it could ask for a redraw by the application, or it
> >> could just wait for the window to reach its final position as attended
> >> by the user and then send the matrix and ask for new sub-pixel perfect
> >> content.
> >>
> >> You can get both, autonomous compositor window move (with no client
> >> communication) and sub-pixel perfect rendering on non moving window.
> >
> > Yeah, with glitches, if the monitor sub-pixel pattern is not
> > translation invariant.
> 
> No you can do proper rendering if you know the sub-pixel pattern and
> the transformation matrix.

I always tend to assume we do not communicate the matrix, since that
exposes the global or output coordinate system, and surface
positions.

Keeping surface positions as a secret is a special feature of
Wayland that differs from most other windowing systems. I would not
want to expose them even via an extension, since that opens an
opportunity to abuse it to do global positioning from clients.

> >> My point here is that i would hate for an api that just encompass
> >> special cases, while transmitting the transformation matrix as long as
> >> providing per output dpi/geometry informations would allow to handle
> >> any case we could ever face.
> >
> > How would you transmit transformations that are not representable
> > by a matrix? Nothing says we are limited to matrices, that is also
> > just a special case. Or would you introduce that limitation in the
> > protocol?
> 
> What kind of transformation are not representable by matrix, i am
> really curious on that. Unless you want to break the window into tile
> and show tile at different position which would be very weird. But
> still doable just have the server send an identify matrix.

Like Andreas said, mapping on a curved geometry, e.g. as a circle segment.
Wobbly windows are another example.

> > In my opinion, the whole idea of clients rendering in screen space
> > goes against a compositing architecture. Once you let clients to draw
> > in screen space, the compositor is no longer free to move and
> > transform the image. Even blending might break the image. The only
> > thing I can think of what the compositor can do without asking and
> > waiting for a new image from the client is re-stacking. Either
> > that, or temporarily produce wrong output.
> 
> As i sais compositor is still in charge and nothing i propose to add
> forbid or change what compositor can do. Will there be a slight
> difference in rendering btw non moving/changing/resize/... window and
> stable one, depend on the compositor it can either request redraw
> supixel perfect for each step of the animation or leave with the fact
> of unperfect result during animation and it won't be terribly wrong
> output you make it sounds like it will be horrible.

Yes, I think it is horrible, because "every frame is perfect" is an
important design principle in Wayland, and we go to great lengths
to ensure that. It also means possible temporary glitches are
unacceptable.

Implementations may be lacking, but the protocol must be designed
for guaranteed perfectness. It is the reason we added e.g.
wl_surface.commit, and I spent weeks in designing the sub-surface
protocol to guarantee a way for glitch-free window updates even in
threaded clients.

But, if you design it as an optional extension, it is more
acceptable, and also means people can try it without modifying
libwayland. That way systems, that can implement it properly and
still remain usable performance-wise, can make use of it.

> > Formulating your idea as an optional extension would be ok, and
> > maybe that is your plan.
> >
> > Or maybe we should just propose to remove all sub-pixel features from
> > the core protocol? Well, not remove, but add a note that the
> > sub-pixel information is not useful without further extensions.
> 
> I think the sub-pixel information is just too limited but still usefull.

Yeah, it doesn't support translation variant patterns. Seems like
you would have to replace that feature in your extension anyway.

> > About dpi. There is going to be a clipping and scaling extension to
> > allow taking advantage of hardware scalers, and that will be an
> > orthogonal companion extension to sub-surfaces.
> >
> > We don't have any proposal of the clipping & scaling extension, but
> > writing it in terms of DPI does not sound right to me. It probably will
> > somehow relay the intended surface size separately from the buffer
> > size, and also allow to use just a sub-rectangle of the buffer contents.
> > These would be sizes in integer pixels, so we avoid any rounding
> > problems from multiplying fixed or floating point numbers.
> >
> > Helping with that would be very useful. The idea is to have an
> > extension interface to wl_surface, just like wl_shell_surface and
> > wl_subsurface are.
> 
> On DPI side i only proposed that client tell the compositor the DPI at
> which it rendered its buffer so that compositor can upscale on high
> dpi display.

Exactly that is going to be a problem, since I expect rounding
errors to produce nice 1-pixel glitches in some cases and not in
others. I believe it is better to use explicit sizes in pixels, not
dpi multipliers, to avoid any possible rounding problems. After
all, it is the scaled size in pixels the clients wants to control,
not physical size directly.

Currently Wayland is a raster protocol, based on units of discrete
pixels. Using real numbers like DPI in the protocol will be
complicated because they work in a different domain and so need to
be converted to the discrete domain. This conversion is only
approximate. You could of course transmit DPI as a ratio N:M, but
then it will be equivalent to just transmitting the scaled size in
pixels, just more complex. We are not using physical units for
images or geometry, so DPI is meaningless in that respect.


Thanks,
pq