[RFC PATCH 3/5] drm/amdgpu: Allow explicit sync for VM ops.

[Daniel Vetter]

Digging out of a hole, apologies to everyone.

On Fri, Jun 17, 2022 at 03:08:00PM +0200, Christian König wrote:
> Am 17.06.22 um 15:03 schrieb Bas Nieuwenhuizen:
> > [SNIP]
> > > > > BOOKKEEP can only be used by VM updates themselves. So that they don't
> > > > > interfere with CS.
> > > > That is the point why we would go BOOKKEEP for explicit sync CS
> > > > submissions, no? Explicit submission shouldn't interfere with any
> > > > other CS submissions. That includes being totally ignored by GL
> > > > importers (if we want to have synchronization there between an
> > > > explicit submission and GL, userspace is expected to use Jason's
> > > > dmabuf fence import/export IOCTLs)
> > > No, that would break existing DMA-buf rules.
> > > 
> > > Explicit CS submissions are still a dependency for implicit submissions.
> > This is explicitly what we don't want for explicit submissions and why
> > I waited with this series until the DMA_RESV_USAGE series landed. We
> > wish to opt out from implicit sync completely, and just use the IOCTLs
> > Jason wrote for back-compat with windowing systems that need it.
> > 
> > If BOOKKEEP isn't for that, should we add a new USAGE?
> 
> BOOKKEEP is exactly for that, but as discussed with Daniel that's not what
> we want in the kernel.

Not sure which Daniel you talked to, but this wasn't me.

> When you mix implicit with explicit synchronization (OpenGL with RADV for
> example) it should be mandatory for the OpenGL to wait for any RADV
> submission before issuing an operation.
> 
> What you want to do is intentionally not supported.

vk is very intentional in it's rejecting of any implicit sync. Which means
when you share a buffer with gl, even in _that_ case there must be no sync
automatically, or your implementation is kinda shit. Instead anyone
sharing a buffer with vk and using it in gl must take care of sync by
importing the timeline syncobj to gl, that's why all these extensions got
added.

This leaves libva in the cold, but hey libva didn't even get around to
adding the full set of modifier extensions so I can't really get myself to
care.

So summary this means:

- a CS/execbuf for vk should _only_ set BOOKKEEPING fences (except ofc if
  there's memory management moves in the preparation, which use KERNEL
  fences and then become additional dependencies for the job)

- because vk memory model is that always everything currently bound can be
  used this means you set BOOKKEEPING on absolutely everything. The
  current clever trick amdgpu has with shared buffers is also not really
  the right thing.

- implicit sync is only controlled through the new import/export ioctl on
  the dma-buf

- if you set any READ/WRITE fences anywhere else, you have potential
  oversync compared to what the vk spec would want

- userspace gets to keep absolutely all the pieces here. Which is not an
  issue, because userspace is totally allowed to fill a buffer with
  garbage and hand that to the compositor already, so there's nothing new
  going wrong here.

- ideally (definitely required for vk sparse) when you unbind or rebind
  then the BOOKKEEPING fences for the vm/ctx get for the old buffers get
  simply replaced by the pte clearing and tlb flushing fences (like amdkfd
  does for compute, vk really just wants to look like compute in
  everything). In practice, especially with partial and multiple mappings
  of the same underlying bo involved, this might be too expensive to
  accurately track since you can only do the replacement trick when the
  last mapping is gone. It might be worth it for private bo though, dunno.

For amdgpu the current special owner checks mostly allow you to get the
semantics vulkan wants. But it breaks down when you have cross-device or
cross-process sharing.

We should probably also document this in the kerneldoc for the BOOKKEEPING
usage that this is the fence type that vulkan cs should use in all
drivers, otherwise this will become an endless mess of driver specific
hacks (i.e. the world we currently live in).
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch