[BUG] gpu: drm: possible ABBA deadlock in drm_gem_prime_fd_to_handle() and drm_gem_prime_handle_to_fd()

[Jia-Ju Bai]

Hello,

My static analysis tool reports a possible ABBA deadlock in the drm 
driver in Linux 5.10:

drm_gem_prime_fd_to_handle()
   mutex_lock(&dev->object_name_lock); --> Line 313 (Lock A)
   drm_gem_handle_delete()
     drm_gem_object_release_handle()
       drm_gem_remove_prime_handles()
         mutex_lock(&filp->prime.lock); --> Line 16 (Lock B)

drm_gem_prime_handle_to_fd()
   mutex_lock(&file_priv->prime.lock); --> Line 433 (Lock B)
   mutex_lock(&dev->object_name_lock); --> Line 466 (Lock A)

When drm_gem_prime_fd_to_handle() and drm_gem_prime_handle_to_fd() are 
concurrently executed, the deadlock can occur.

I am not quite sure whether this possible deadlock is real and how to 
fix it if it is real.
Any feedback would be appreciated, thanks :)

Reported-by: TOTE Robot <oslab at tsinghua.edu.cn>


Best wishes,
Jia-Ju Bai