Question about the future of Xorg

[Carsten Haitzler]

On Wed, 11 Jun 2025 14:50:33 -0400 (EDT) Vladimir Dergachev
<volodya at mindspring.com> said:

> 
> 
> On Wed, 11 Jun 2025, Carsten Haitzler wrote:
> 
> >>> the windows on screen with a different offset. keeping other controls like
> >>> your pager/shelf/whatever overlayed where they are is far more useful than
> >>> having them panned away and off screen.
> >>
> >> But being primitive is the whole point. The apps render to the large
> >> screen as is, and one or more (or none) CRTCs display it on monitors.
> >
> > the apps don't know or care.
> 
> They do. Consider two apps - one renders into a window that is partly 
> over the monitor edge, and the other app capture data from that window.
> 
> The first app only renders into the visible area, as it thinks the over 
> the edge pixels are not displayed.

not in a wayland world. the app must render all of its content. doing otherwise
would be a bug on the part of the application. the compositor may transform
the window - rotate it, resize it... wrap it around the surface of a 3d bunny
rabbit. all of the window content should be there because it may be modified
and transformed in any way the compositor likes.

> The second app captures the window and part of it is black or something 
> else. Not good. I have seen this behavious with zoom and other apps.

in a universe where this is implemented in wayland the compositor captures. it
can do what it likes when capturing and fake content if it likes - mask out
content as it likes. this is by design.

x11 is a much leakier abstraction. it's very explicit as to how it works. i was
talking how i'd implement it in a wayland world if you saw the context.

> This needs to be handled at the protocol level (i.e. Wayland), with 
> specific definitions that discuss what happens with partially obscured 
> window.

wayland clients should render all of their content because the window may be
transformed in all sorts of ways the client may not expect.

an example. i might write a compositor that actually zooms out windows stacked
below the focused one. it also blurs them to look out of focus. so it gives a
deep 3d depth to the screen. in fact i may have a compositor render this
stereoscopically across 2 eyes in a vr headset to give even more depth with
each eye having the windows offset to give true depth perception.

the compositor could also rotate the windows and scatter them like a set of
cards thrown about a table. the apps should provide all the content. that's the
expectation.

also for captures, the compositor can do what it likes here. given pipewire
capture or the new wl protocol for captured buffers, the compositor can render
something different from what you see on the screen. it can hide or blur or
mask out windows. it can hide specific content. it can render at higher
resolution or lower - it can do whatever it likes really. int he above
scenario, a capture may not blur windows further away... or maybe it blurs all
windows that are not owned by the client requesting the capture, for privacy
reasons.

> The virtual framebuffer that people ask for is nice because the apps do 
> not know or care that not everything is displayed - they work just as 
> usual.
> 
> If you do it at the compositor then an app can attempt to detect whether 
> its output is being recorded by creating an obscured area and monitoring 
> whether it is requested to render to it.

it will be requested to render in frame requests for any reason at all - every
screen refresh or something else. these do not tell you if you are being
recorded. the compositor already has all buffers for all windows anyway so as
long as nothing has updated everything can be recorded by the compositor as it
already rendered/composited that frame and can use that. clients won't know.

if privacy/security is a factor, then it's the compositor's job to enforce that
here. apps in general have no access to any other client's windows in a
wayland world (unlike x). the only leaks are via a compositor and the leak
there is pretty much screenshots (pipewire if implemented - that's dbus) or the
new wl screenshot protocol. the compositor can put in this anything it likes as
above. it's the compositor's job then to enforce privacy/security by
blanking/blurring/hiding content. it could have a default policy of deny such
requests until a user says "ok" and then indicate if they want everything or
only a subset of content to be recorded (compositor can remember this then or
even remember specifically for that client - right now this is actually an area
that at least last i knew was un-solved in wayland how to persistently and
reliably identify a specific app/client - some kind of persistent key that a
compositor can use to class/separate clients apart?). at the moment the best
they can do is follow the unix socket fd back to the original process and
inspect relevant info there like maybe UID/GID or smack label for example and
then use some policies like if uid is the same then allow content to be seen -
if they differ mask out non-matching content (and perhaps gid too
depending...). if smack exists then this could be used too (you'd need a
scheme of smack labels deciding how to separate content).

> >> You don't need the apps to handle the rerendering calls, and the data is
> >> always there. For example, you can render a large 8K virtual framebuffer
> >> and record it in MP3 file, or view it with x11vnc.
> >
> > see above. they don't know or care. in a composited world all of a window;'s
> > pixels exist in another buffer already - the app already rendered them,. the
> > compositor then renders this again when compositing where the window is
> > meant to be with whatever transforms/sizes it wants. app doesn't need to
> > know or care.
> 
> So, I agree that the app should not care - it should just render as usual. 
> It is the user that cares and might want to configure his hardware to do 
> what they want.

no - they don't configure the hardware.. they configure their compositor... :)
the compositor does with the hw as it pleases to get the effect the user
requested. it may be the maximum buffer size the hw can handle is e.g. 4096
pixels wide. the screen is 3840. it would be impossible to do a 7860 wide
buffer as the hw literally cannot handle buffer spans wider than 4096. this
kind of thing exists in real hardware. the compositor could fake a 7680 wide
virtual screen but the display hw could never handle a buffer with spans that
wide - so the compositor will do this whoever it can to essentially get the
effect... if this is a compositor feature.

but in the end it is a compositor feature... not a wayland feature. it is
between a compositor and users to offer something like this. ask compositor
authors.. or write your own or send them patches.

> >> You should be able to render to a framebuffer of *your* choice, and you
> >> should be able to configure CRTCs to use framebuffer of *your* choice.
> >
> > talk to the compositor devs of your chosen wayland compositor, as i keep
> > mentioning. wayland has no clue. asking your local bakery for a new hard
> > drive is not going to help and the more you complain to them that they
> > don't have a hard drive, the grumpier they will get. wayland is a PROTOCOL.
> > it is a LANGUAGE. if you went and complained about government policies to
> > the authors of the oxford english dictionary is not going to get you
> > anywhere... just because your government may happen to speak english to its
> > citizens (for example).
> 
> You can change a lot by modifying the language..

then you fail to grasp the problem here nor my analogy. there is only so much i
can do to repeat the same thing.

> But more on point, I can see how you could modify wayland compositor to 
> support virtual screens - but then the control app would have to talk to 
> the compositor to request changes in virtual screen configuration in a 
> different way than through wayland library.

it'd be internal to that compositor. it could just have a config dialog. it
doesnt need to be any "control app".

example: enlightenment is a wayland compositor. it reads AND writes its own
config files. all of its settings dialogs are internal to the wm/compositor. if
you want to reconfigure screen setup you use its screen setup dialogs. you
click some buttons. it will modify the screen setup. in wayland or in x11 mode.
it will save this and restore it. it will remember which screens had what
config (based on output+edid data) and restore that when they come back. there
are ZERO tools to do this. it's entirely internal all inside the same process
that takes care of itself. it;'s the same way gimp has a settings dialog or
inkscape does or blender or any other app - they have their own settings
dialogs built in and you clicky-pointy to change things and the app responds.
enlightenment works the same way. there are no config apps because it doesn't
need any.

now that is one way to do things. other compositors may work differently - they
may implement a special protocol for this and have tools to modify config live
on the fly. that protocol doesn't even have to be wayland. it could be they
have no protocol, and any config changes require you to edit some file and
restart the compositor. in the end it's not a standard wayland protocol that
compositors will support. you'll likely find a wide agreement with compositor
authors that giving carte blanche protocol access to client apps to modify
screen config is a very bad thing and it'll just be NAK'd - if apps need things
that may result in different screen setups.. then it should be done at a very
high level with high level protocol info for windows (surfaces) that then may
result in compositors doing something special.

> best
> 
> Vladimir Dergachev
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
Carsten Haitzler - raster at rasterman.com