BUG / design challenge: vmwgfx + PRIME handle free == clobbering errno

[Stefan Hoffmeister]

Hi,

vmwgfx implements drmPrimeFDToHandle in terms of the TTM resource manager.

At the same time, the driver advertises

	.driver_features =
	DRIVER_MODESET | DRIVER_RENDER | DRIVER_ATOMIC | DRIVER_GEM,

Given that, a userland application will call drmCloseBufferHandle to  
correctly release any previously acquired handle.

In kernel language, this translates to ioctl  
DRM_IOCTL_PRIME_FD_TO_HANDLE and DRM_IOCTL_GEM_CLOSE.

Problems:

a) because the application uses drmCloseBufferHandle, and that goes  
through GEM (the driver claims "I am GEM"!), vmwgfx will always return  
EINVAL, because the driver is _not_ GEM, but TTM on that aspect.

This will always clobber errno for userspace, which is not at all  
helpful for meaningful error handling.

Additionally, the driver offers no formal API contract which would  
allow applications to detect that the driver does not like be talked  
to like a GEM driver, although it is _claims_ to be GEM driver.

b) there is no (documented) means to release the handle acquired from  
a call to drmPrimeFDToHandle; this implies that in principle any call  
to drmPrimeFDToHandle will leak the handle.

Questions:

a) how can errno clobbering by vmwgfx with EINVAL be avoided?

b) what is the correct way to release the TTM resources allocated  
through drmPrimeFDToHandle?


For context: the KDE Plasma Desktop kwin DRM integration layer makes  
extensive use of drmPrimeFDToHandle. With the way the VMware vmwgfx  
driver behaves, I see these options:

a) correctly check the return code of drmCloseBufferHandle and blindly  
log all the EINVAL errors triggered by the VMware driver, putting  
blame on the application for mis-managing handles
b) drop any return code from drmCloseBufferHandle onto the floor,  
completely disregarding application integrity checking
c) hard-code a check for "vmwgfx" as the driver in use and spam "this  
driver owned by VMware / Broadcom is broken, ignore the following  
EINVAL and secondary effects" for integrity checks

I do not like any of these options, to be honest.

Many thanks for any input
Stefan