[RFC] Experimental DMA-BUF Device Heaps

[James Jones]

On 8/23/20 1:46 PM, Laurent Pinchart wrote:
> Hi James,
> 
> On Sun, Aug 23, 2020 at 01:04:43PM -0700, James Jones wrote:
>> On 8/20/20 1:15 AM, Ezequiel Garcia wrote:
>>> On Mon, 2020-08-17 at 20:49 -0700, James Jones wrote:
>>>> On 8/17/20 8:18 AM, Brian Starkey wrote:
>>>>> On Sun, Aug 16, 2020 at 02:22:46PM -0300, Ezequiel Garcia wrote:
>>>>>> This heap is basically a wrapper around DMA-API dma_alloc_attrs,
>>>>>> which will allocate memory suitable for the given device.
>>>>>>
>>>>>> The implementation is mostly a port of the Contiguous Videobuf2
>>>>>> memory allocator (see videobuf2/videobuf2-dma-contig.c)
>>>>>> over to the DMA-BUF Heap interface.
>>>>>>
>>>>>> The intention of this allocator is to provide applications
>>>>>> with a more system-agnostic API: the only thing the application
>>>>>> needs to know is which device to get the buffer for.
>>>>>>
>>>>>> Whether the buffer is backed by CMA, IOMMU or a DMA Pool
>>>>>> is unknown to the application.
>>>>>>
>>>>>> I'm not really expecting this patch to be correct or even
>>>>>> a good idea, but just submitting it to start a discussion on DMA-BUF
>>>>>> heap discovery and negotiation.
>>>>>>
>>>>>
>>>>> My initial reaction is that I thought dmabuf heaps are meant for use
>>>>> to allocate buffers for sharing across devices, which doesn't fit very
>>>>> well with having per-device heaps.
>>>>>
>>>>> For single-device allocations, would using the buffer allocation
>>>>> functionality of that device's native API be better in most
>>>>> cases? (Some other possibly relevant discussion at [1])
>>>>>
>>>>> I can see that this can save some boilerplate for devices that want
>>>>> to expose private chunks of memory, but might it also lead to 100
>>>>> aliases for the system's generic coherent memory pool?
>>>>>
>>>>> I wonder if a set of helpers to allow devices to expose whatever they
>>>>> want with minimal effort would be better.
>>>>
>>>> I'm rather interested on where this goes, as I was toying with using
>>>> some sort of heap ID as a basis for a "device-local" constraint in the
>>>> memory constraints proposals Simon and I will be discussing at XDC this
>>>> year.  It would be rather elegant if there was one type of heap ID used
>>>> universally throughout the kernel that could provide a unique handle for
>>>> the shared system memory heap(s), as well as accelerator-local heaps on
>>>> fancy NICs, GPUs, NN accelerators, capture devices, etc. so apps could
>>>> negotiate a location among themselves.  This patch seems to be a step
>>>> towards that in a way, but I agree it would be counterproductive if a
>>>> bunch of devices that were using the same underlying system memory ended
>>>> up each getting their own heap ID just because they used some SW
>>>> framework that worked that way.
>>>>
>>>> Would appreciate it if you could send along a pointer to your BoF if it
>>>> happens!
>>>
>>> Here is it:
>>>
>>> https://linuxplumbersconf.org/event/7/contributions/818/
>>>
>>> It would be great to see you there and discuss this,
>>> given I was hoping we could talk about how to meet a
>>> userspace allocator library expectations as well.
>>
>> Thanks!  I hadn't registered for LPC and it looks like it's sold out,
>> but I'll try to watch the live stream.
>>
>> This is very interesting, in that it looks like we're both trying to
>> solve roughly the same set of problems but approaching it from different
>> angles.  From what I gather, your approach is that a "heap" encompasses
>> all the allocation constraints a device may have.
>>
>> The approach Simon Ser and I are tossing around so far is somewhat
>> different, but may potentially leverage dma-buf heaps a bit as well.
>>
>> Our approach looks more like what I described at XDC a few years ago,
>> where memory constraints for a given device's usage of an image are
>> exposed up to applications, which can then somehow perform boolean
>> intersection/union operations on them to arrive at a common set of
>> constraints that describe something compatible with all the devices &
>> usages desired (or fail to do so, and fall back to copying things around
>> presumably).  I believe this is more flexible than your initial proposal
>> in that devices often support multiple usages (E.g., different formats,
>> different proprietary layouts represented by format modifiers, etc.),
>> and it avoids adding a combinatorial number of heaps to manage that.
>>
>> In my view, heaps are more like blobs of memory that can be allocated
>> from in various different ways to satisfy constraints.  I realize heaps
>> mean something specific in the dma-buf heap design (specifically,
>> something closer to an association between an "allocation mechanism" and
>> "physical memory"), but I hope we don't have massive heap/allocator
>> mechanism proliferation due to constraints alone.  Perhaps some
>> constraints, such as contiguous memory or device-local memory, are
>> properly expressed as a specific heap, but consider the proliferation
>> implied by even that simple pair of examples: How do you express
>> contiguous device-local memory?  Do you need to spawn two heaps on the
>> underlying device-local memory, one for contiguous allocations and one
>> for non-contiguous allocations?  Seems excessive.
>>
>> Of course, our approach also has downsides and is still being worked on.
>>    For example, it works best in an ideal world where all the allocators
>> available understand all the constraints that exist.
> 
> Shouldn't allocators be decoupled of constraints ? In my imagination I
> see devices exposing constraints, and allocators exposing parameters,
> with a userspace library to reconcile the constraints and produce
> allocator parameters from them.

Perhaps another level of abstraction would help.  I'll have to think 
about that.

However, as far as I can tell, it wouldn't remove the need to 
communicate a lot of constraints from multiple engines/devices/etc. to 
the allocator (likely a single allocator.  I'd be interested to know if 
anyone has a design that effectively uses multiple allocators to satisfy 
a single allocation request, but I haven't come up with a good one) 
somehow.  Either the constraints are directly used as the parameters, or 
there's a translation/second level of abstraction, but either way much 
of the information needs to make it to the allocator, or represent the 
need to use a particular allocator.  Simple things like pitch and offset 
alignment can be done without help from a kernel-level allocator, but 
others such as cache coherency, physical memory bank placement, or 
device-local memory will need to make it all the way down to the kernel 
some how I believe.

Thanks,
-James

>> Dealing with a
>> reality where there are probably a handful of allocators, another
>> handful of userspace libraries and APIs, and still more applications
>> trying to make use of all this is one of the larger remaining challenges
>> of the design.
>>
>> We'll present our work at XDC 2020.  Hope you can check that out as well!
>>
>>>>> 1. https://lore.kernel.org/dri-devel/57062477-30e7-a3de-6723-a50d03a402c4@kapsi.fi/
>>>>>
>>>>>> Given Plumbers is just a couple weeks from now, I've submitted
>>>>>> a BoF proposal to discuss this, as perhaps it would make
>>>>>> sense to discuss this live?
>>>>>>
>>>>>> Not-signed-off-by: Ezequiel Garcia <ezequiel at collabora.com>
>