[PATCH v2 12/12] Documentation: iio: Document high-speed DMABUF based API
Daniel Vetter
daniel at ffwll.ch
Tue Mar 29 14:07:21 UTC 2022
On Tue, Mar 29, 2022 at 10:47:23AM +0100, Paul Cercueil wrote:
> Hi Daniel,
>
> Le mar., mars 29 2022 at 10:54:43 +0200, Daniel Vetter <daniel at ffwll.ch> a
> écrit :
> > On Mon, Feb 07, 2022 at 01:01:40PM +0000, Paul Cercueil wrote:
> > > Document the new DMABUF based API.
> > >
> > > v2: - Explicitly state that the new interface is optional and is
> > > not implemented by all drivers.
> > > - The IOCTLs can now only be called on the buffer FD returned by
> > > IIO_BUFFER_GET_FD_IOCTL.
> > > - Move the page up a bit in the index since it is core stuff
> > > and not
> > > driver-specific.
> > >
> > > Signed-off-by: Paul Cercueil <paul at crapouillou.net>
> > > ---
> > > Documentation/driver-api/dma-buf.rst | 2 +
> > > Documentation/iio/dmabuf_api.rst | 94
> > > ++++++++++++++++++++++++++++
> > > Documentation/iio/index.rst | 2 +
> > > 3 files changed, 98 insertions(+)
> > > create mode 100644 Documentation/iio/dmabuf_api.rst
> > >
> > > diff --git a/Documentation/driver-api/dma-buf.rst
> > > b/Documentation/driver-api/dma-buf.rst
> > > index 2cd7db82d9fe..d3c9b58d2706 100644
> > > --- a/Documentation/driver-api/dma-buf.rst
> > > +++ b/Documentation/driver-api/dma-buf.rst
> > > @@ -1,3 +1,5 @@
> > > +.. _dma-buf:
> > > +
> > > Buffer Sharing and Synchronization
> > > ==================================
> > >
> > > diff --git a/Documentation/iio/dmabuf_api.rst
> > > b/Documentation/iio/dmabuf_api.rst
> > > new file mode 100644
> > > index 000000000000..43bb2c1b9fdc
> > > --- /dev/null
> > > +++ b/Documentation/iio/dmabuf_api.rst
> > > @@ -0,0 +1,94 @@
> > > +===================================
> > > +High-speed DMABUF interface for IIO
> > > +===================================
> > > +
> > > +1. Overview
> > > +===========
> > > +
> > > +The Industrial I/O subsystem supports access to buffers through a
> > > file-based
> > > +interface, with read() and write() access calls through the IIO
> > > device's dev
> > > +node.
> > > +
> > > +It additionally supports a DMABUF based interface, where the
> > > userspace
> > > +application can allocate and append DMABUF objects to the buffer's
> > > queue.
> > > +This interface is however optional and is not available in all
> > > drivers.
> > > +
> > > +The advantage of this DMABUF based interface vs. the read()
> > > +interface, is that it avoids an extra copy of the data between the
> > > +kernel and userspace. This is particularly useful for high-speed
> > > +devices which produce several megabytes or even gigabytes of data
> > > per
> > > +second.
> > > +
> > > +The data in this DMABUF interface is managed at the granularity of
> > > +DMABUF objects. Reducing the granularity from byte level to block
> > > level
> > > +is done to reduce the userspace-kernelspace synchronization
> > > overhead
> > > +since performing syscalls for each byte at a few Mbps is just not
> > > +feasible.
> > > +
> > > +This of course leads to a slightly increased latency. For this
> > > reason an
> > > +application can choose the size of the DMABUFs as well as how many
> > > it
> > > +allocates. E.g. two DMABUFs would be a traditional double buffering
> > > +scheme. But using a higher number might be necessary to avoid
> > > +underflow/overflow situations in the presence of scheduling
> > > latencies.
> >
> > So this reads a lot like reinventing io-uring with pre-registered
> > O_DIRECT
> > memory ranges. Except it's using dma-buf and hand-rolling a lot of
> > pieces
> > instead of io-uring and O_DIRECT.
>
> I don't see how io_uring would help us. It's an async I/O framework, does it
> allow us to access a kernel buffer without copying the data? Does it allow
> us to zero-copy the data to a network interface?
With networking, do you mean rdma, or some other kind of networking?
Anything else than rdma doesn't support dma-buf, and I don't think it will
likely ever do so. Similar it's really tricky to glue dma-buf support into
the block layer.
Wrt io_uring, yes it's async, but that's not the point. The point is that
with io_uring you pre-register ranges for reads and writes to target,
which in combination with O_DIRECT, makes it effectively (and efficient!)
zero-copy. Plus it has full integration with both networking and normal
file io, which dma-buf just doesn't have.
Like you _cannot_ do zero copy from a dma-buf into a normal file. You
absolutely can do the same with io_uring.
> > At least if the entire justification for dma-buf support is zero-copy
> > support between the driver and userspace it's _really_ not the right
> > tool
> > for the job. dma-buf is for zero-copy between devices, with cpu access
> > from userpace (or kernel fwiw) being very much the exception (and often
> > flat-out not supported at all).
>
> We want both. Using dma-bufs for the driver/userspace interface is a
> convenience as we then have a unique API instead of two distinct ones.
>
> Why should CPU access from userspace be the exception? It works fine for IIO
> dma-bufs. You keep warning about this being a terrible design, but I simply
> don't see it.
It depends really on what you're trying to do, and there's extremely high
chances it will simply not work.
Unless you want to do zero copy with a gpu, or something which is in that
ecosystem of accelerators and devices, then dma-buf is probably not what
you're looking for.
-Daniel
>
> Cheers,
> -Paul
>
> > > +
> > > +2. User API
> > > +===========
> > > +
> > > +``IIO_BUFFER_DMABUF_ALLOC_IOCTL(struct iio_dmabuf_alloc_req *)``
> > > +----------------------------------------------------------------
> > > +
> > > +Each call will allocate a new DMABUF object. The return value (if
> > > not
> > > +a negative errno value as error) will be the file descriptor of
> > > the new
> > > +DMABUF.
> > > +
> > > +``IIO_BUFFER_DMABUF_ENQUEUE_IOCTL(struct iio_dmabuf *)``
> > > +--------------------------------------------------------
> > > +
> > > +Place the DMABUF object into the queue pending for hardware
> > > process.
> > > +
> > > +These two IOCTLs have to be performed on the IIO buffer's file
> > > +descriptor, obtained using the `IIO_BUFFER_GET_FD_IOCTL` ioctl.
> > > +
> > > +3. Usage
> > > +========
> > > +
> > > +To access the data stored in a block by userspace the block must be
> > > +mapped to the process's memory. This is done by calling mmap() on
> > > the
> > > +DMABUF's file descriptor.
> > > +
> > > +Before accessing the data through the map, you must use the
> > > +DMA_BUF_IOCTL_SYNC(struct dma_buf_sync *) ioctl, with the
> > > +DMA_BUF_SYNC_START flag, to make sure that the data is available.
> > > +This call may block until the hardware is done with this block.
> > > Once
> > > +you are done reading or writing the data, you must use this ioctl
> > > again
> > > +with the DMA_BUF_SYNC_END flag, before enqueueing the DMABUF to the
> > > +kernel's queue.
> > > +
> > > +If you need to know when the hardware is done with a DMABUF, you
> > > can
> > > +poll its file descriptor for the EPOLLOUT event.
> > > +
> > > +Finally, to destroy a DMABUF object, simply call close() on its
> > > file
> > > +descriptor.
> > > +
> > > +For more information about manipulating DMABUF objects, see:
> > > :ref:`dma-buf`.
> > > +
> > > +A typical workflow for the new interface is:
> > > +
> > > + for block in blocks:
> > > + DMABUF_ALLOC block
> > > + mmap block
> > > +
> > > + enable buffer
> > > +
> > > + while !done
> > > + for block in blocks:
> > > + DMABUF_ENQUEUE block
> > > +
> > > + DMABUF_SYNC_START block
> > > + process data
> > > + DMABUF_SYNC_END block
> > > +
> > > + disable buffer
> > > +
> > > + for block in blocks:
> > > + close block
> > > diff --git a/Documentation/iio/index.rst
> > > b/Documentation/iio/index.rst
> > > index 58b7a4ebac51..669deb67ddee 100644
> > > --- a/Documentation/iio/index.rst
> > > +++ b/Documentation/iio/index.rst
> > > @@ -9,4 +9,6 @@ Industrial I/O
> > >
> > > iio_configfs
> > >
> > > + dmabuf_api
> > > +
> > > ep93xx_adc
> > > --
> > > 2.34.1
> > >
> >
> > --
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > http://blog.ffwll.ch
>
>
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
More information about the dri-devel
mailing list