<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size:11.0pt">Hi Gurchetan,<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt">Thank you for taking a look at this patch series!<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <p class="MsoNormal">On Thu, Mar 28, 2024 at 2:01<span style="font-family:"Arial",sans-serif"> </span>AM Vivek Kasireddy <<a href="mailto:vivek.kasireddy@intel.com" target="_blank">vivek.kasireddy@intel.com</a>> wrote:<o:p></o:p></p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> <p class="MsoNormal">Having virtio-gpu import scanout buffers (via prime) from other<br> devices means that we'd be adding a head to headless GPUs assigned<br> to a Guest VM or additional heads to regular GPU devices that are<br> passthrough'd to the Guest. In these cases, the Guest compositor<br> can render into the scanout buffer using a primary GPU and has the<br> secondary GPU (virtio-gpu) import it for display purposes.<br> <br> The main advantage with this is that the imported scanout buffer can<br> either be displayed locally on the Host (e.g, using Qemu + GTK UI)<br> or encoded and streamed to a remote client (e.g, Qemu + Spice UI).<br> Note that since Qemu uses udmabuf driver, there would be no copies<br> made of the scanout buffer as it is displayed. This should be<br> possible even when it might reside in device memory such has VRAM.<br> <br> The specific use-case that can be supported with this series is when<br> running Weston or other guest compositors with "additional-devices"<br> feature (./weston --drm-device=card1 --additional-devices=card0).<br> More info about this feature can be found at:<br> <a href="https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/736" target="_blank">https://gitlab.freedesktop.org/wayland/weston/-/merge_requests/736</a><br> <br> In the above scenario, card1 could be a dGPU or an iGPU and card0<br> would be virtio-gpu in KMS only mode. However, the case where this<br> patch series could be particularly useful is when card1 is a GPU VF<br> that needs to share its scanout buffer (in a zero-copy way) with the<br> GPU PF on the Host. Or, it can also be useful when the scanout buffer<br> needs to be shared between any two GPU devices (assuming one of them<br> is assigned to a Guest VM) as long as they are P2P DMA compatible.<o:p></o:p></p> </blockquote> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Is passthrough iGPU-only or passthrough dGPU-only something you intend to use?<o:p></o:p></p> <p class="MsoNormal"><span style="font-size:11.0pt">Our main use-case involves passthrough’g a headless dGPU VF device and sharing <o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt">the Guest compositor’s scanout buffer with dGPU PF device on the Host. Same goal for<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt">headless iGPU VF to iGPU PF device as well.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt">However, using a combination of iGPU and dGPU where either of them can be passthrough’d<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:11.0pt">to the Guest is something I think can be supported with this patch series as well. <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">If it's a dGPU + iGPU setup, then the way other people seem to do it is a "virtualized" iGPU (via virgl/gfxstream/take your pick) and pass-through the dGPU.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">For example, AMD seems to use virgl to allocate and import into the dGPU.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal"><a href="https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23896" target="_blank">https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23896</a><o:p></o:p></p> </div> <div> <p class="MsoNormal"><a href="https://lore.kernel.org/all/20231221100016.4022353-1-julia.zhang@amd.com/" target="_blank">https://lore.kernel.org/all/20231221100016.4022353-1-julia.zhang@amd.com/</a><o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">ChromeOS also uses that method (see <a href="http://crrev.com/c/3764931"> crrev.com/c/3764931</a>) [cc: dGPU architect <a href="mailto:dbehr@google.com">+Dominik Behr</a>]<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">So if iGPU + dGPU is the primary use case, you should be able to use these methods as well. The model would "virtualized iGPU" + passthrough dGPU, not split SoCs. <o:p></o:p></p> <p class="MsoNormal">In our use-case, the goal is to have only one primary GPU (passthrough’d iGPU/dGPU)<o:p></o:p></p> <p class="MsoNormal">do all the rendering (using native DRI drivers) for clients/compositor and all the outputs<o:p></o:p></p> <p class="MsoNormal">and share the scanout buffers with the secondary GPU (virtio-gpu). Since this is mostly<o:p></o:p></p> <p class="MsoNormal">how Mutter (and also Weston) work in a multi-GPU setup, I am not sure if virgl is needed.<o:p></o:p></p> </div> <div> <p class="MsoNormal"> <o:p></o:p></p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> <p class="MsoNormal">As part of the import, the virtio-gpu driver shares the dma<br> addresses and lengths with Qemu which then determines whether the<br> memory region they belong to is owned by a PCI device or whether it<br> is part of the Guest's system ram. If it is the former, it identifies<br> the devid (or bdf) and bar and provides this info (along with offsets<br> and sizes) to the udmabuf driver. In the latter case, instead of the<br> the devid and bar it provides the memfd. The udmabuf driver then<br> creates a dmabuf using this info that Qemu shares with Spice for<br> encode via Gstreamer.<br> <br> Note that the virtio-gpu driver registers a move_notify() callback<br> to track location changes associated with the scanout buffer and<br> sends attach/detach backing cmds to Qemu when appropriate. And,<br> synchronization (that is, ensuring that Guest and Host are not<br> using the scanout buffer at the same time) is ensured by pinning/<br> unpinning the dmabuf as part of plane update and using a fence<br> in resource_flush cmd.<o:p></o:p></p> </blockquote> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">I'm not sure how QEMU's display paths work, but with crosvm if you share the guest-created dmabuf with the display, and the guest moves the backing pages, the only recourse is the destroy the surface and show a black screen to the user: not the best thing experience wise.<o:p></o:p></p> <p class="MsoNormal">Since Qemu GTK UI uses EGL, there is a blit done from the guest’s scanout buffer onto an EGL<o:p></o:p></p> <p class="MsoNormal">backed buffer on the Host. So, this problem would not happen as of now. <o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">Only amdgpu calls dma_buf_move_notfiy(..), and you're probably testing on Intel only, so you may not be hitting that code path anyways. <o:p></o:p></p> <p class="MsoNormal">I have tested with the Xe driver in the Guest which also calls dma_buf_move_notfiy(). However,<o:p></o:p></p> <p class="MsoNormal">note that for dGPUs, both Xe and amdgpu migrate the scanout buffer from vram to system<o:p></o:p></p> <p class="MsoNormal">memory as part of export, because virtio-gpu is not P2P compatible.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">I forgot the exact reason, but apparently udmabuf may not work with amdgpu displays and it seems the virtualized iGPU + dGPU is the way to go for amdgpu anyways.<o:p></o:p></p> <p class="MsoNormal">I am curious why udmabuf would not work with amdgpu?<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">So I recommend just pinning the buffer for the lifetime of the import for simplicity and correctness. <o:p></o:p></p> <p class="MsoNormal">Yeah, in this patch series, the dmabuf is indeed pinned but only for a short duration in the Guest –<o:p></o:p></p> <p class="MsoNormal">just until the Host is done using it (blit or encode).<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Thanks,<o:p></o:p></p> <p class="MsoNormal">Vivek<o:p></o:p></p> </div> <div> <p class="MsoNormal"> <o:p></o:p></p> </div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> <p class="MsoNormal" style="margin-bottom:12.0pt">This series is available at:<br> <a href="https://gitlab.freedesktop.org/Vivek/drm-tip/-/commits/virtgpu_import_rfc" target="_blank">https://gitlab.freedesktop.org/Vivek/drm-tip/-/commits/virtgpu_import_rfc</a><br> <br> along with additional patches for Qemu and Spice here:<br> <a href="https://gitlab.freedesktop.org/Vivek/qemu/-/commits/virtgpu_dmabuf_pcidev" target="_blank">https://gitlab.freedesktop.org/Vivek/qemu/-/commits/virtgpu_dmabuf_pcidev</a><br> <a href="https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf_v4" target="_blank">https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf_v4</a> <br> <br> Patchset overview:<br> <br> Patch 1: Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd<br> Patch 2-3: Helpers to initalize, import, free imported object<br> Patch 4-5: Import and use buffers from other devices for scanout<br> Patch 6-7: Have udmabuf driver create dmabuf from PCI bars for P2P DMA<br> <br> This series is tested using the following method:<br> - Run Qemu with the following relevant options:<br> qemu-system-x86_64 -m 4096m ....<br> -device vfio-pci,host=0000:03:00.0<br> -device virtio-vga,max_outputs=1,blob=true,xres=1920,yres=1080<br> -spice port=3001,gl=on,disable-ticketing=on,preferred-codec=gstreamer:h264<br> -object memory-backend-memfd,id=mem1,size=4096M<br> -machine memory-backend=mem1 ...<br> - Run upstream Weston with the following options in the Guest VM:<br> ./weston --drm-device=card1 --additional-devices=card0<br> <br> where card1 is a DG2 dGPU (passthrough'd and using xe driver in Guest VM),<br> card0 is virtio-gpu and the Host is using a RPL iGPU.<br> <br> Cc: Gerd Hoffmann <<a href="mailto:kraxel@redhat.com" target="_blank">kraxel@redhat.com</a>><br> Cc: Dongwon Kim <<a href="mailto:dongwon.kim@intel.com" target="_blank">dongwon.kim@intel.com</a>><br> Cc: Daniel Vetter <<a href="mailto:daniel.vetter@ffwll.ch" target="_blank">daniel.vetter@ffwll.ch</a>><br> Cc: Christian Koenig <<a href="mailto:christian.koenig@amd.com" target="_blank">christian.koenig@amd.com</a>><br> Cc: Dmitry Osipenko <<a href="mailto:dmitry.osipenko@collabora.com" target="_blank">dmitry.osipenko@collabora.com</a>><br> Cc: Rob Clark <<a href="mailto:robdclark@chromium.org" target="_blank">robdclark@chromium.org</a>><br> Cc: Thomas Hellström <<a href="mailto:thomas.hellstrom@linux.intel.com" target="_blank">thomas.hellstrom@linux.intel.com</a>><br> Cc: Oded Gabbay <<a href="mailto:ogabbay@kernel.org" target="_blank">ogabbay@kernel.org</a>><br> Cc: Michal Wajdeczko <<a href="mailto:michal.wajdeczko@intel.com" target="_blank">michal.wajdeczko@intel.com</a>><br> Cc: Michael Tretter <<a href="mailto:m.tretter@pengutronix.de" target="_blank">m.tretter@pengutronix.de</a>><br> <br> Vivek Kasireddy (7):<br> drm/virtio: Implement VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING cmd<br> drm/virtio: Add a helper to map and note the dma addrs and lengths<br> drm/virtio: Add helpers to initialize and free the imported object<br> drm/virtio: Import prime buffers from other devices as guest blobs<br> drm/virtio: Ensure that bo's backing store is valid while updating<br> plane<br> udmabuf/uapi: Add new ioctl to create a dmabuf from PCI bar regions<br> udmabuf: Implement UDMABUF_CREATE_LIST_FOR_PCIDEV ioctl<br> <br> drivers/dma-buf/udmabuf.c | 122 ++++++++++++++++--<br> drivers/gpu/drm/virtio/virtgpu_drv.h | 8 ++<br> drivers/gpu/drm/virtio/virtgpu_plane.c | 56 ++++++++-<br> drivers/gpu/drm/virtio/virtgpu_prime.c | 167 ++++++++++++++++++++++++-<br> drivers/gpu/drm/virtio/virtgpu_vq.c | 15 +++<br> include/uapi/linux/udmabuf.h | 11 +-<br> 6 files changed, 368 insertions(+), 11 deletions(-)<br> <br> -- <br> 2.43.0<o:p></o:p></p> </blockquote> </div> </div> </div> </div> </body> </html>