[systemd-devel] deny access to GPU devices

[Markus Koeberl]

hi!

I am using slurm to manage GPU resources. On a host with several GPUs installed a user gets only access to the GPUs he asks slurm for. This is implemented by using the devices cgroup controller. For each job slurm starts, all devices which are not allowed get denied using cgroup devices.deny.
But by default users get access to all GPUs at login. As my users have ssh access to the host they can bypass slurm and access all GPUs directly. Therefore I would like to deny access to GPU devices for all user logins.
Basically what I want is for all users logins: 
echo "c 195:* rwm" > /sys/fs/cgroup/devices/... /devices.deny
Which should deny access to all Nvidia GPUs (this is what slurm does in his own hierarchy which looks like /sys/fs/cgroup/devices/slurm/uid_1044/job_555359/step_0).

On my system for my user with UID=1044 this would be:
echo "c 195:* rwm" > /sys/fs/cgroup/devices/user.slice/user-1044.slice/devices.deny
based on:
$ awk -F':' '$2 ~ /devices/ {print $3}' /proc/self/cgroup 
/user.slice/user-1044.slice

I did not find anything in the documentation how to implement this. It seams to me that there is no way at the moment to configure sytemd to alter the cgroup device config when creating the session for the user.
It would be nice if somebody could give me some hints how to implement this or a link to an implementation or the right documentation.

My idea how to implement this but I am not sure if it is the right way or if it will work:
write a PAM session module which runs the echo "c 195:* rwm" to the right cgroup devices.deny file based on the information from /proc/self/cgroup

I am using debian stable/unstable, at the moment I have installed systemd 230 from jessie-backports. I saw systemd 232 in unstable which should be no problem to install.

Thanks for any help or advise!


regards
Markus Köberl
-- 
Markus Koeberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: markus.koeberl at tugraz.at