[PATCH RFC wayland-protocols] unstable/linux-dmabuf: add wp_linux_dmabuf_device_hint

[Philipp Zabel]

Hi Pekka,

thank you for the explanation.

On Wed, 2018-11-14 at 11:03 +0200, Pekka Paalanen wrote:
[...]
> The hints protocol we are discussing here is a subset of what
> https://github.com/cubanismo/allocator aims to achieve. Originally we
> only concentrated on getting the format and modifier more optimal, but
> the question of where and how to allocate the buffers is valid too. Is
> it in scope for this extension is the big question below.

My guess is: probably not. Either way, I'd prefer the protocol docs to
be explicit about this.

> Ideally, the protocol would do something like this:
> 
> - Tell the client which device and for which use case the device must
>   be able to access the buffer at minimum and always.
> 
> - Tell the client that if it could make the buffer suitable also for a
>   secondary device and a secondary use case, the compositor could do a
>   more optimal job (e.g. putting the buffer in direct scanout,
>   bypassing composition, or a hardware video encoder in case the output
>   is going to be streamed).
> 
> We don't have the vocabulary for use cases and there are tons of
> different details to be taken into account, which is the whole point of
> the allocator project. So we cannot do the complete solution here and
> now, but we can do an approximate solution by negotiating pixel
> formats and modifiers.
> 
> The primary device is what the compositor uses for the fallback path,
> which is compositing with a GPU.
>
>  Therefore at very minimum, clients
> need to allocate buffers that can be used with the primary device. We
> guarantee this in the zwp_linux_dmabuf protocol by having the
> compositor test the buffer import into EGL (or equivalent) before it
> accepts that the buffer even exists. The client does not absolutely
> necessarily need the primary device for this, but it will have much
> better chances of making usable buffers if it uses it for allocation at
> least.

So the client must provide buffers that the primary device can import
and sample a texture from, ideally directly.
Can something like this be added to the interface description, to make
it clear what the primary device actually is supposed to be in this
context?

> The primary device also has another very different meaning: the
> compositor will likely be using the primary device anyway so it is kept
> active and if clients use the same device instead of some other device,
> it probably results in considerable power savings. IOW, the primary
> device is the preferred rendering device as well. Or so I assume, these
> two concepts could be decoupled as well.

And the client should default to using the same primary device for
rendering for power savings.

> A secondary device is optional. In system where the GPU and display
> devices are separate DRM devices, the GPU will be the primary device,
> and the display device would be the secondary device. So there seems to
> be a use case for sending the secondary device (or devices?) in
> addition to the primary device.
> 
> AFAIK, the unix device memory allocator project does not yet have
> anything we should be encoding as a Wayland extension, so all we seem
> to be able to do is to deliver the device file descriptors and the
> format+modifier sets.

Ok.

> Now the design question: do we want to communicate the secondary
> devices in this extension? Quite likely we need a different extension
> to be used with the allocator project.

As long as the use case is not clear, I'd say leave it out.
A "secondary_device" event may be added later with a version update if
needed.

> Is communicating the display device fd useful already when it differs
> from the rendering device? Is there a way for generic client userspace
> to use it effectively, or would it rely on hardware-specific code in
> clients rather than in e.g. Mesa drivers? Are there EGL or Vulkan APIs
> to tell the driver it should make the buffer work on one device while
> rendering on another?

I have not found anything specific about this in the Vulkan spec.

The VK_KHR_external_memory extension even states:

    "However, only the same concrete physical device can be used when
     sharing memory, [...]"

and:

    "Note this does not attempt to address cross-device transitions, nor
     transitions to engines on the same device which are not visible
     within the Vulkan API.
     Both of these are beyond the scope of this extension."

in the issues. So even though with VK_EXT_external_memory_dma_buf
and VK_EXT_image_drm_format_modifier bolted on top, sharing between
different devices should be possible, it is not the main focus of these
extensions.

> My current opinion is that if there is no generic way for an
> application to benefit from the secondary device fd, then we should not
> add secondary devices in this extension yet.

I agree.

regards
Philipp