On 18/08/2021 11:00, Christian König wrote:
Am 18.08.21 um 09:43 schrieb Gal Pressman:
Hey all,
Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).
While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.
The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.
This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.
Well NAK to the general approach, this can be solved much easier.
If you can't support dynamic moves while using the buffer then just pin all buffers during import/export.
This avoids the move notification and the framework/exporter can still correctly account for pinned buffers.
But please note that at least amdgpu never uses P2P support for pinned buffers since we want to avoid that unmoveable buffers clutter video memory and create conflicts with V4L and scanout.
If you don't have such concerns in habanalabs then you can implement the pinning there while keeping P2P still enabled.
Thanks Christian!
Are you suggesting to pass an empty move_notify callback instead of passing NULL? Also, doesn't the pin operation move the memory from the device to host memory?