About migrating framebuffers in multi-GPU compositors

[Alex Deucher]

On Thu, Mar 24, 2022 at 3:40 PM Hoosier, Matt <Matt.Hoosier at garmin.com> wrote:
>
> On Thu, 2022-03-24 at 15:28 -0400, Alex Deucher wrote:
>
> CAUTION - EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe.
>
>
>
> On Thu, Mar 24, 2022 at 9:43 AM Hoosier, Matt <
>
> Matt.Hoosier at garmin.com
>
> > wrote:
>
>
> On Thu, 2022-03-24 at 11:56 +0200, Pekka Paalanen wrote:
>
>
> On Wed, 23 Mar 2022 14:19:08 +0000
>
>
> "Hoosier, Matt" <
>
>
> Matt.Hoosier at garmin.com
>
>
>
> wrote:
>
>
>
> Hi,
>
>
>
> I recently had a reason to wade through Mutter's code to support
>
>
> systems with more than one GPU. I was a bit surprised to see that
>
>
> it supports several different strategies for dealing with
>
>
> scanning out a buffer on a KMS output not associated with the GPU
>
>
> where the buffer was originally rendered.
>
>
>
> Hi,
>
>
>
> indeed. The reason for multiple paths is that different systems
>
>
> don't support all ways, or do support some of the ways but the
>
>
> performance might be abysmal. Knowing which path to take is a
>
>
> difficult problem, and other than benchmarking (which I didn't
>
>
> implement in Mutter) you can't really know if what you picked is
>
>
> going to be fine.
>
>
>
> In particular, the approach of using the secondary GPU's OpenGL
>
>
> implementation to blit into a dumb buffer was really unexpected.
>
>
> Typically, dumb buffers get described as a really slow, uncached
>
>
> mapping of GPU memory into the CPU.
>
>
>
> The support got added here (by Pekka):
>
>
>
> https://urldefense.com/v3/__https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/615__;!!EJc4YC3iFmQ!FFkanSe1AOe2ya_-thuTSNX7kr4pCENbg2UXiNPEhVEP_uxwjWvWCTHD-4dIcSsrfA$
>
>
>
>
>
> That MR is using the primary GPU to blit, not the secondary GPU.
>
>
>
> If a secondary GPU can have a hardware accelerated OpenGL context,
>
>
> I don't know why anyone would deliberately use dumb buffers on that
>
>
> device with OpenGL. GBM offers better ways.
>
>
>
>
> This MR cover letter has a better overview of all the methods:
>
>
> https://urldefense.com/v3/__https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/810__;!!EJc4YC3iFmQ!FFkanSe1AOe2ya_-thuTSNX7kr4pCENbg2UXiNPEhVEP_uxwjWvWCTHD-4d7YGnc1w$
>
>
>
>
> Ah, even nicer. Thanks!
>
>
> In the ranked-order list of strategies there, the zero-copy technique is less preferred than the secondary GPU copy technique. Seems like you'd rarely ever fall through to the zero-copy strategy even if the GPU drivers do both support it. Anything subtle going on there that's good to be aware of? Like maybe a given driver typically supports secondary-GPU-copy XOR zero-copy, so you're fairly likely to reach the second strategy on systems that can handle it.
>
>
>
>
> If I follow this right, the blit occurs directly between video
>
>
> memory owned by the primary GPU into dumb-buffer memory owned by
>
>
> the secondary GPU, without laboriously using the CPU to do PIO.
>
>
>
> Correct.
>
>
>
> Does this imply that the two GPUs' drivers have to be at least
>
>
> minimally aware of each other to negotiate some kind of DMA path
>
>
> directly between the two?
>
>
>
> I don't know the details. It depends on whether you can map
>
>
> secondary GPU memory to be written by the primary GPU. The specific
>
>
> use case here is iGPU as primary and virtual as secondary, which
>
>
> means that video memory for both is more or less "system RAM". No
>
>
> discrete VRAM involved.
>
>
>
> Oh interesting. I hadn't realized that on the hybrid GPU systems even the dGPU uses system RAM. But on thinking about it, that's probably the only efficient way for the hardware to be designed.
>
>
>
> It is accomplished through the kernel dmabuf framework where
>
>
> drivers export and import dmabuf.
>
>
>
> Right, makes sense.
>
>
> So I wonder how I should reason about a system that's configured with 2x of the same discrete graphics card (AMD, if it matters). The compositor would arbitrarly pick whichever of those happened to enumerate first as the primary, and then it's down to the driver details as to which of the four migration paths gets chosen? For the moment, let's assume that none of the stock applications is bothering to use any sort of advanced dmabuf hinting to pick the right GPU node to correspond to the output on which it will eventually display.
>
>
>
> Since you asked about AMD GPUs, the iGPU render engines can support
>
> reading or writing to carve out (system memory carved out for iGPU
>
> use), system memory, or remote device memory (e.g., remote device PCI
>
> BARs).  The iGPU display hardware supports reading from carve out or
>
> system memory.  For dGPUs, the render engines can support reading and
>
> writing to system memory (including iGPU cave out), local VRAM, or
>
> remote device memory (e.g., remote device PCI BARs).  That said, dGPUs
>
> perform best when the buffers are in local VRAM.  So ideally if you
>
> are using the dGPU for rendering, that would be done in VRAM.  Then if
>
> the app is being rendered by the dGPU and the app is fullscreen and
>
> the display is attached to the iGPU, you'd need to copy that buffer to
>
> system memory or carve out so it could be displayed by the iGPU
>
> display hardware. If the compositor is compositing the rendering with
>
> other buffers on the iGPU, then it may make more sense to keep it in
>
> VRAM and just read it from the render engine in the iGPU and write the
>
> resulting frame to the display buffer on the iGPU.  This also makes
>
> sense if you have two dGPUs.  Ideally you'd read directly from the
>
> remote device memory rather than taking a trip through system ram.  As
>
> Pekka said, it gets complicated quickly.
>
>
> What you really want to avoid is reading device memory or carve out
>
> with the CPU.  Not only does it go over the PCI bus, but MMIO space is
>
> usually mapped uncached on the CPU, so you'll be doing uncached reads
>
> over a relatively slow bus.  If you need to get data out of device
>
> memory, it is much better to have a device DMA it to somewhere else.
>
> Either the device where the memory is attached (e.g., DMA from local
>
> VRAM to system memory or local VRAM to remote VRAM on another device),
>
> or the device who wants access to the memory (e.g., DMA from remote
>
> VRAM to system memory or remote VRAM to local VRAM on the device).
>
> Displaylink devices are a bad example of this.  Their display hardware
>
> is fed from system memory, so you need to get the data from the render
>
> device to system memory.  If you try to do the copy with the CPU, the
>
> performance will be unusable.  This should largely work with dma-buf
>
> since the dma-buf will be moved to system memory if the importer
>
> doesn't support peer to peer DMA, but in a lot of cases, user mode
>
> just mmaps the buffer in VRAM rather than importing it as dma-buf and
>
> then copies it using the CPU.  That really only works if the source
>
> buffer is in cached system memory.
>
>
> Thanks, Pekka and Alex, for these really interesting descriptions.
>
> Just out of curiosity, I wonder how something like Windows decides on a general policy for coordinating usage of VRAM from multiple GPUs. I'm not so much interested in doing anything with Windows, but it would be an interesting reference to see how a widely adopted commercial systems attacks the problem of handling multiple discrete video cards.
>

I'm not an expert on windows, but my understanding is that all shared
buffers take a trip through system memory on windows and all display
drivers are required to support buffer sharing.  I suspect in most
cases, you end up with two copies, device to system in the exporter,
system to device on the importer, but the secondary copy could be
short circuited in some cases (e.g., if the importing device supports
access to buffers in system memory for whatever operation it needs to
do).  I think the important part is the buffer sharing API
requirement.  At least every driver supports shared buffers via system
memory.  Then at least you have a common API with a decent baseline to
optimize in the drivers involved.  Unfortunately not all drm drivers
support dma-buf.

Alex