[systemd-devel] deny access to GPU devices

[Lennart Poettering]

On Mon, 14.11.16 13:13, Markus Koeberl (markus.koeberl at tugraz.at) wrote:

> > > I am using slurm to manage GPU resources. On a host with several
> > > GPUs installed a user gets only access to the GPUs he asks slurm
> > > for. This is implemented by using the devices cgroup controller. For
> > > each job slurm starts, all devices which are not allowed get denied
> > > using cgroup devices.deny.  But by default users get access to all
> > > GPUs at login. As my users have ssh access to the host they can
> > > bypass slurm and access all GPUs directly. Therefore I would like to
> > > deny access to GPU devices for all user logins.
> > 
> > I have no idea what "slurm" is, but do note that the "devices" cgroup
> > controller has no future, it is unlikely to ever become available in
> > cgroupsv2.
> 
> That are bad news. Is there a place where I can read about the
> future of cgroups?

There have been articles on LWN, and there were various discussions at the
Linux Plumber Conference. Also, ping Tejun, he's the cgroups guy to go to.

> > Device access to local users is normally managed through ACLs on the
> > device node, via udev/logind's "uaccess" logic. Using the "devices"
> > cgroup controller for this appears pretty misguided...
> 
> Using devices cgroup there is the possibility to extend this:
> It is possible to grand one process of a user access to a device 
> and at the same time deny an other process of the same user access
> to the same device.

Hmm, on UNIX the primary credentials used for access controls are
users and groups of course (with secondary concepts such as labels,
and caps and suchlike), but they usually are attached to the process,
instead of some external concept such as a cgroup. As such the devices
cgroup subsystem is kind of an outlier on this one already...

> > > I did not find anything in the documentation how to implement
> > > this. It seams to me that there is no way at the moment to configure
> > > sytemd to alter the cgroup device config when creating the session
> > > for the user.  It would be nice if somebody could give me some hints
> > > how to implement this or a link to an implementation or the right
> > > documentation.
> > 
> > You can alter the DevicesAllow= property of the "user-1000.slice"
> > (where 1000 is the uid of your user) unit. But do note that the whole
> > "devices" cgroup controller is going away (as mentioned above), so
> > this is not future proof. And in general ACL-based device access
> > management is usually the better idea.
> 
> I had the impression that I need the opposite because using this to deny access wont work but I have to admit I did not test it.
> Would "DeviceAllow=/dev/nvidia? " (omit rwm) remove r, w and m
> attributes form /dev/nvidia[0-9]

Omiting the rwm thing does the right thing. But if you want to block
entire subsystems you need a syntax like "DeviceAllow=char-foobar",
where "foobar" is a subsystem as listed in /proc/devices.

> I also did not see a way to specify this for all users therefore
> this would mean to maintain the configuration on all hosts for each
> individual user which I do not like. Although I have a small number
> of users and hosts this sounds complicated to maintain especially in
> my case the environment is highly inhomogeneous.

Yes, this is kinda nasty I have to admit, the way it is right
now. Ideally we could stuff this kind of information into the user
database, but UNIX is pretty limited there right now I fear and
I don't see this changing anytime soon.

Lennart

-- 
Lennart Poettering, Red Hat