Possible new standards for XR desktops

[Nova King]

Hello everyone, I'm Nova. I'm making an XR desktop that's extremely modular
called Stardust (https://github.com/technobaboo/stardust-xr) and I was
wondering how I should go about desktop standards for XR when none
currently exist, and I figured Freedesktop would be the right place to ask
and propose some areas where I feel standards are needed and I need your
help to flesh out into proper standards.

So first of all, an XR desktop requires massive changes in some areas
compared to a 2D desktop, most notably in graphics.

For GUI there should be a standard to create 3D apps.The problem comes with
compositing all those 3D apps, and this is where I'm highly unsure what to
do. Stardust currently has its own renderer in the core that takes all the
module UIs and renders them all in 1 place. This means it'd be tied to
Stardust and while Stardust is the first of its kind that I can tell, I
feel I should make it interoperable with any others that may come along. So
the issue is this: In 2D there can be a simple layering system using
transparency and drawing some apps over others just using buffers in the
GPU, but to do that in 3D using existing standards and rendering
technologies requires depthbuffers as well as color buffers. There is an
extension to the OpenXR spec that allows for depth compositing however it
is not implemented in all runtimes and OpenXR is not made for
self-contained apps, more immersive experiences. While XR should be not
renderer-dependent depth compositing does not have the capability to render
transparency correctly. Imagine a sphere inside of a transparent cube that
has its front and back faces rendered. If you were to depth composite the
two, the cube would still have its back faces rendered on top of the sphere
even if the sphere is inside the cube and smaller. A custom renderer would
need to be used that effectively had voxels in camera frustum space for
each layer. That's no easy task and it'd be really hard to get every
renderer to comply to fix the transparency issue. The other way is to have
a standard that sends over meshes, shaders, textures, etc. to a single
engine which then renders them all together and displays the result through
OpenXR. This is difficult as well, but I figure the FreeDesktop team could
help me find a suitable compromise so we can make a specification for how
this is handled for 3D apps? Monado might be a good project to add to this
conversation as they're making XRDesktop (which is currently not focused on
3D apps but more bringing 2D apps into XR, they'd still be nice to have
around though).

For the input situation I have some ideas already as to a possible standard
though they would need to be fleshed out. Unlike the 4 standard input
methods in 2D (keyboard, mouse, touch, and drawing tablet) XR has 4 main
inputs as well (non-spatial inputs like gamepads, pointers that shoot a ray
out from a controller, a controller that is in 3D space
positionally/rotationally tracked, and hand tracking input with a full 21
or 22-bone skeleton per hand). I have some thoughts as to how simply
providing an app with free access of these controls whenever they want
could cause privacy issues for apps who are more proprietary unlike simple
keyboard and mouse inputs, but I'm not sure privacy is what standards
groups deal with in these respects.

Now onto the launcher scenario. While the desktop entires standard is a
good one I'm not entirely convinced XR apps can be as simple as a single
executable file to launch. Many XR apps could use being able to have parts
of them launched rather than a whole and it's not always easy or practical
to put those parts in seperate executables or run multiple instances with
IPC and shut off the other functions.

Drag and drop in X11 is a very nice spec and I feel something similar is
needed for XR, especially because holding object representations of files,
photos, music, etc. is a very nice UI feature (I'm an XR interaction
designer/developer). So a sort of drag and drop inside a 3D boundary
standard would be very good to have.

That's all the standards ideas I have for now, I apologize if this isn't
the proper channel to ask these questions in. If not, could you please
direct me to where a better place would be?

Thank you all so much for making desktops open and free!

--Nova
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/xdg/attachments/20200331/f2a0ad02/attachment.htm>