<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"> <div dir="ltr">Sent from my iPad</div><div dir="ltr"> <blockquote type="cite">On Aug 2, 2022, at 3:33 PM, Tomasz Figa <tfiga@chromium.org> wrote: </blockquote></div><blockquote type="cite"><div dir="ltr">On Mon, Aug 1, 2022 at 8:43 PM ayaka <ayaka@soulik.info> wrote: <blockquote type="cite"> </blockquote><blockquote type="cite"> </blockquote><blockquote type="cite"> </blockquote><blockquote type="cite">Sent from my iPad </blockquote><blockquote type="cite"> </blockquote><blockquote type="cite"><blockquote type="cite">On Aug 1, 2022, at 5:46 PM, Tomasz Figa <tfiga@chromium.org> wrote: </blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </blockquote></blockquote><blockquote type="cite"><blockquote type="cite">CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. </blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">On Mon, Aug 1, 2022 at 3:44 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">On 8/1/22 14:19, Tomasz Figa wrote: </blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Hello Tomasz </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">?Hi Randy, </blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">On Mon, Aug 1, 2022 at 5:21 AM <ayaka@soulik.info> wrote: </blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">From: Randy Li <ayaka@soulik.info> </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">This module is still at a early stage, I wrote this for showing what </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">APIs we need here. </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Let me explain why we need such a module here. </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">If you won't allocate buffers from a V4L2 M2M device, this module </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">may not be very useful. I am sure the most of users won't know a </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">device would require them allocate buffers from a DMA-Heap then </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">import those buffers into a V4L2's queue. </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Then the question goes back to why DMA-Heap. From the Android's </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">description, we know it is about the copyright's DRM. </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">When we allocate a buffer in a DMA-Heap, it may register that buffer </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">in the trusted execution environment so the firmware which is running </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">or could only be acccesed from there could use that buffer later. </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">The answer above leads to another thing which is not done in this </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">version, the DMA mapping. Although in some platforms, a DMA-Heap </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">responses a IOMMU device as well. For the genernal purpose, we would </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">be better assuming the device mapping should be done for each device </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">itself. The problem here we only know alloc_devs in those DMAbuf </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">methods, which are DMA-heaps in my design, the device from the queue </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">is not enough, a plane may requests another IOMMU device or table </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">for mapping. </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Signed-off-by: Randy Li <ayaka@soulik.info> </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">--- </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">drivers/media/common/videobuf2/Kconfig | 6 + </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">drivers/media/common/videobuf2/Makefile | 1 + </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">.../common/videobuf2/videobuf2-dma-heap.c | 350 ++++++++++++++++++ </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">include/media/videobuf2-dma-heap.h | 30 ++ </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">4 files changed, 387 insertions(+) </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">create mode 100644 drivers/media/common/videobuf2/videobuf2-dma-heap.c </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">create mode 100644 include/media/videobuf2-dma-heap.h </blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">First of all, thanks for the series. </blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Possibly a stupid question, but why not just allocate the DMA-bufs </blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">directly from the DMA-buf heap device in the userspace and just import </blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">the buffers to the V4L2 device using V4L2_MEMORY_DMABUF? </blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Sometimes the allocation policy could be very complex, let's suppose a </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">multiple planes pixel format enabling with frame buffer compression. </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Its luma, chroma data could be allocated from a pool which is delegated </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">for large buffers while its metadata would come from a pool which many </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">users could take some few slices from it(likes system pool). </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Then when we have a new users knowing nothing about this platform, if we </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">just configure the alloc_devs in each queues well. The user won't need </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">to know those complex rules. </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">The real situation could be more complex, Samsung MFC's left and right </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">banks could be regarded as two pools, many devices would benefit from </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">this either from the allocation times or the security buffers policy. </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">In our design, when we need to do some security decoding(DRM video), </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">codecs2 would allocate buffers from the pool delegated for that. While </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">the non-DRM video, users could not care about this. </blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> </blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I'm a little bit surprised about this, because on Android all the </blockquote></blockquote><blockquote type="cite"><blockquote type="cite">graphics buffers are allocated from the system IAllocator and imported </blockquote></blockquote><blockquote type="cite"><blockquote type="cite">to the specific devices. </blockquote></blockquote><blockquote type="cite">In the non-tunnel mode, yes it is. While the tunnel mode is completely vendor defined. Neither HWC nor codec2 cares about where the buffers coming from, you could do what ever you want. </blockquote><blockquote type="cite"> </blockquote><blockquote type="cite">Besides there are DRM video in GNU Linux platform, I heard the webkit has made huge effort here and Playready is one could work in non-Android Linux. </blockquote><blockquote type="cite"><blockquote type="cite">Would it make sense to instead extend the UAPI to expose enough </blockquote></blockquote><blockquote type="cite"><blockquote type="cite">information about the allocation requirements to the userspace, so it </blockquote></blockquote><blockquote type="cite"><blockquote type="cite">can allocate correctly? </blockquote></blockquote><blockquote type="cite">Yes, it could. But as I said it would need the users to do more works. </blockquote><blockquote type="cite"><blockquote type="cite">My reasoning here is that it's not a driver's decision to allocate </blockquote></blockquote><blockquote type="cite"><blockquote type="cite">from a DMA-buf heap (and which one) or not. It's the userspace which </blockquote></blockquote><blockquote type="cite"><blockquote type="cite">knows that, based on the specific use case that it wants to fulfill. </blockquote></blockquote><blockquote type="cite">Although I would like to let the users decide that, users just can’t do that which would violate the security rules in some platforms. </blockquote><blockquote type="cite">For example, video codec and display device could only access a region of memory, any other device or trusted apps can’t access it. Users have to allocate the buffer from the pool the vendor decided. </blockquote><blockquote type="cite"> </blockquote><blockquote type="cite">So why not we offer a quick way that users don’t need to try and error. </blockquote> In principle, I'm not against integrating DMA-buf heap with vb2, however I see some problems I mentioned before: 1) How would the driver know if it should allocate from a DMA-buf heap or not?</div></blockquote>struct vb2_queue.mem_ops<div> </div><div>int (*queue_setup)(struct vb2_queue *q,unsigned int *num_buffers, unsigned int *num_planes, unsigned int sizes[], struct device *alloc_devs[]);</div><div> <div><blockquote type="cite"><div dir="ltr">2) How would the driver know which heap to allocate from? </div></blockquote><div>From vb2_queue.alloc_devs</div><div> </div><div>What I did now is likes what MFC does, create some mem_alloc_devs.</div><div>It would be better that we could retrieve the DMA-heaps’ devices from kernel, but that is not enough, we need a place to store the heap flags although none of them are defined yet.</div><div> </div><div>From Android documents, I think it is unlikely we would have heap flags.</div><div>“Standardization: The DMA-BUF heaps framework offers a well-defined UAPI. ION allowed custom flags and heap IDs that prevented developing a common testing framework because each device’s ION implementation could behave differently.”</div><div> <blockquote type="cite"><div dir="ltr">3) How would the heap know how to allocate properly for the device? </div></blockquote>Because “each DMA-BUF heap is a separate character device”.</div><div>But as I said in the first draft I am not sure about the DMA mapping part. alloc_devs responds for the heap, we have a device variable in the queue that mapping function could access, but that may not be enough. A plane may apply a different mapping policy or IOMMU here.</div><div> </div><div>Would it be better that I create a interface here that creating a memdev with DMA-heap description ? <blockquote type="cite"><div dir="ltr">Best regards, Tomasz </div></blockquote></div></div></div></body></html>