[Intel-gfx] [Mesa-dev] [PATCH] dma-buf: Document dma-buf implicit fencing/resv fencing rules

[Daniel Stone]

Hi,

On Wed, 23 Jun 2021 at 17:20, Daniel Vetter <daniel.vetter at ffwll.ch> wrote:
> +        *
> +        * IMPLICIT SYNCHRONIZATION RULES:
> +        *
> +        * Drivers which support implicit synchronization of buffer access as
> +        * e.g. exposed in `Implicit Fence Poll Support`_ should follow the
> +        * below rules.

'Should' ... ? Must.

> +        * - Drivers should add a shared fence through
> +        *   dma_resv_add_shared_fence() for anything the userspace API
> +        *   considers a read access. This highly depends upon the API and
> +        *   window system: E.g. OpenGL is generally implicitly synchronized on
> +        *   Linux, but explicitly synchronized on Android. Whereas Vulkan is
> +        *   generally explicitly synchronized for everything, and window system
> +        *   buffers have explicit API calls (which then need to make sure the
> +        *   implicit fences store here in @resv are updated correctly).
> +        *
> +        * - [...]

Mmm, I think this is all right, but it could be worded much more
clearly. Right now it's a bunch of points all smashed into one, and
there's a lot of room for misinterpretation.

Here's a strawman, starting with most basic and restrictive, working
through to when you're allowed to wriggle your way out:

Rule 1: Drivers must add a shared fence through
dma_resv_add_shared_fence() for any read accesses against that buffer.
This appends a fence to the shared array, ensuring that any future
non-read access will be synchronised against this operation to only
begin after it has completed.

Rule 2: Drivers must add an exclusive fence through
dma_resv_add_excl_fence() for any write accesses against that buffer.
This replaces the exclusive fence with the new operation, ensuring
that all future access will be synchronised against this operation to
only begin after it has completed.

Rule 3: Drivers must synchronise all accesses to buffers against
existing implicit fences. Read accesses must synchronise against the
exclusive fence (read-after-write), and write accesses must
synchronise against both the exclusive (write-after-write) and shared
(write-after-read) fences.

Note 1: Users like OpenGL and window systems on non-Android userspace
are generally implicitly synchronised. An implicitly-synchronised
userspace is unaware of fences from prior operations, so the kernel
mediates scheduling to create the illusion that GPU work is FIFO. For
example, an application will flush and schedule GPU write work to
render its image, then immediately tell the window system to display
that image; the window system may immediately flush and schedule GPU
read work to display that image, with neither waiting for the write to
have completed. The kernel provides coherence by synchronising the
read access against the write fence in the exclusive slot, so that the
image displayed is correct.

Note 2: Users like Vulkan and Android window system are generally
explicitly synchronised. An explicitly-synchronised userspace is
responsible for tracking its own read and write access and providing
the kernel with synchronisation barriers. For instance, a Vulkan
application rendering to a buffer and subsequently using it as a read
texture, must annotate the read operation with a read-after-write
synchronisation barrier.

Note 3: Implicit and explicit userspace can coexist. For instance, an
explicitly-synchronised Vulkan application may be running as a client
of an implicitly-synchronised window system which uses OpenGL for
composition; an implicitly-synchronised OpenGL application may be
running as a client of a window system which uses Vulkan for
composition.

Note 4: Some subsystems, for example V4L2, do not pipeline operations,
and instead only return to userspace when the scheduled work against a
buffer has fully retired.

Exemption 1: Fully self-coherent userspace may skip implicit
synchronisation barriers. For instance, accesses between two
Vulkan-internal buffers allocated by a single application do not need
to synchronise against each other's implicit fences, as the client is
responsible for explicitly providing barriers for access. A
self-contained OpenGL userspace also has no need to implicitly
synchronise its access if the driver instead tracks all access and
inserts the appropriate synchronisation barriers.

Exemption 2: When implicit and explicit userspace coexist, the
explicit side may skip intermediate synchronisation, and only place
synchronisation barriers at transition points. For example, a Vulkan
compositor displaying a buffer from an OpenGL application would need
to synchronise its first access against the fence placed in the
exclusive implicit-synchronisation slot. Once this read has fully
retired, the compositor has no need to participate in implicit
synchronisation until it is ready to return the buffer to the
application, at which point it must insert all its non-retired
accesses into the shared slot, which the application will then
synchronise future write accesses against.

Cheers,
Daniel