[RFC] Per file OOM badness

[Christian König]

Hello everyone,

I'm currently working on the issue that when device drivers allocate memory on
behalf of an application the OOM killer usually doesn't knew about that unless
the application also get this memory mapped into their address space.

This is especially annoying for graphics drivers where a lot of the VRAM
usually isn't CPU accessible and so doesn't make sense to map into the
address space of the process using it.

The problem now is that when an application starts to use a lot of VRAM those
buffers objects sooner or later get swapped out to system memory, but when we
now run into an out of memory situation the OOM killer obviously doesn't knew
anything about that memory and so usually kills the wrong process.

The following set of patches tries to address this problem by introducing a per
file OOM badness score, which device drivers can use to give the OOM killer a
hint how many resources are bound to a file descriptor so that it can make
better decisions which process to kill.

So question at every one: What do you think about this approach?

My biggest concern right now is the patches are messing with a core kernel
structure (adding a field to struct file). Any better idea? I'm considering
to put a callback into file_ops instead.

Best regards and feel free to tear this idea apart,
Christian.