[systemd-devel] deny access to GPU devices

[Lennart Poettering]

On Mon, 07.11.16 16:15, Markus Koeberl (markus.koeberl at tugraz.at) wrote:

> hi!
> 
> I am using slurm to manage GPU resources. On a host with several
> GPUs installed a user gets only access to the GPUs he asks slurm
> for. This is implemented by using the devices cgroup controller. For
> each job slurm starts, all devices which are not allowed get denied
> using cgroup devices.deny.  But by default users get access to all
> GPUs at login. As my users have ssh access to the host they can
> bypass slurm and access all GPUs directly. Therefore I would like to
> deny access to GPU devices for all user logins.

I have no idea what "slurm" is, but do note that the "devices" cgroup
controller has no future, it is unlikely to ever become available in
cgroupsv2.

Device access to local users is normally managed through ACLs on the
device node, via udev/logind's "uaccess" logic. Using the "devices"
cgroup controller for this appears pretty misguided...

> Basically what I want is for all users logins: 
> echo "c 195:* rwm" > /sys/fs/cgroup/devices/... /devices.deny
> Which should deny access to all Nvidia GPUs (this is what slurm does
> in his own hierarchy which looks like
> /sys/fs/cgroup/devices/slurm/uid_1044/job_555359/step_0).

Well, this is just broken. If you use systemd, then the cgroup tree in
the hierarchies it manages are property of systemd, and if you want to
make cgroups, you can do so only in subhierarchies of systemd's own
tree, by setting on the Delegate=yes setting. The path above however
indicates that this is not done here. hence you are really on your
own, sorry.

Also, were does 195 come from? is that a hardcoded major of the
closed-source nvidia driver? Yuck, code really shouldn't hardcode
major/minor numbers these days... And sec

> I did not find anything in the documentation how to implement
> this. It seams to me that there is no way at the moment to configure
> sytemd to alter the cgroup device config when creating the session
> for the user.  It would be nice if somebody could give me some hints
> how to implement this or a link to an implementation or the right
> documentation.

You can alter the DevicesAllow= property of the "user-1000.slice"
(where 1000 is the uid of your user) unit. But do note that the whole
"devices" cgroup controller is going away (as mentioned above), so
this is not future proof. And in general ACL-based device access
management is usually the better idea.

Lennart

-- 
Lennart Poettering, Red Hat