[RFC] Wayland Security Modules

[Sloane, Brandon]

Hello,

I've spent the past few months prototyping a security modules system for Wayland. Our specific motivation for this is to support SELinux integration to meet some rather unique security requirements. However, what we are proposing here is a rather general purpose security module system that provides high level hooks modules can then implement. Potential usecases for this system are:

* Creating SELinux permissions for Wayland actions.
* Integrating with non-SELinux Linux Security Modules (AppArmor/SMACK/etc).
* Integrating with PolicyKit.
* Disabling privileged protocols that a specific compositor implements.
* Restricting privileged protocols to trusted clients.
* Creating backends for wp_security_context_manager.

Our current proof of concept is here: https://gitlab.freedesktop.org/bsloane1650/wayland.
Some more in-depth technical discussion is available in the doc/WSM.md file in that repository.

We also have some modules in development here: https://gitlab.freedesktop.org/bsloane1650/wayland-security-modules
* Logger - a basic proof of concept that demonstrates instantiating a module and logging every access.
* Allow-list - A basic proof of concept that demonstrates globally restricting what interfaces can be used.
* SELinux - A more complex module that defers all access decisions to the system's SELinux policy (under active development)

The overall design is to add hooks at key points in libwayland:
* Creation and destruction of core libwayland objects: wl_client, wl_display, wl_global, wl_resource
* Prior to sending an event to the client
* Prior to invoking the request handler after receiving a request from a client.
* Prior to publishing a global object
* Prior to binding a global object.

A security module is represented by a "struct wsm" object, which has void* for modules to use, and a function pointer for each hook. Compositors can instantiate these structs however they like and pass them into a new wl_display_create_with_wsms method. The existing wl_display_create method is modified to dynamically load shared object files based on the new WAYLAND_SECURITY_MODULES environmental variable. These shared objects are expected to export a wl_wsm_init symbol that instantiates a wsm structure.

We have had success running this by linking unmodified compositors (mostly Weston) against an updated libwayland. Depending on what accesses the module blocks, existing compositors work without even needing a recompile. However, to be useful, we have found a couple of areas that additional compositor integration is needed. Mostly this has been shifting from wl_resourc_create to wl_resource_create_with_related in a few key places (such as creating a wl_data_offer) to allow the security modules to associated resources that are shared between clients. 

We have also found a need to modify compositors to deal with denials associated with new_id type requests. We think we have a workable solution implemented in libwsm_compositor that compositors can incorporate with a few library calls; but I still consider this the most questionable part of the project. This issue is discussed rather in much more detail in doc/WSM.md. Any input on this would be greatly appreciated.

We have also experimented with per-surface screenshot restrictions in Weston, which needs to be implemented almost entirely in the compositor itself.

We probably should have gone public with this far earlier in the design process. However, despite the late stage we find ourselves in, we are open to significant revisions based on community feedback.

Thanks,
Brandon