Help understanding allocation and mapping flow of virtio-gpu 3D resource blobs
Alex Bennée
alex.bennee at linaro.org
Sun Feb 4 14:52:17 UTC 2024
Alex Bennée <alex.bennee at linaro.org> writes:
(doh, replying to virglrender-devel thanks to autocomplete failure)
> Hi,
>
> I'm trying to get an understanding of the blob allocation and mapping
> flow for virtio-gpu for Vulkan and Rutabaga. Having gotten all the
> various libraries setup I'm still seeing failures when running a TCG
> guest (buildroot + latest glm, mesa, vkmark) with:
>
> ./qemu-system-aarch64 \
> -M virt -cpu cortex-a76 \
> -m 8192 \
> -object memory-backend-memfd,id=mem,size=8G,share=on \
> -serial mon:stdio \
> -kernel ~/lsrc/linux.git/builds/arm64.initramfs/arch/arm64/boot/Image \
> -append "console=ttyAMA0" \
> -device virtio-gpu-gl,context_init=true,blob=true,hostmem=4G \
> -display sdl,gl=on -d guest_errors,trace:virtio_gpu_cmd_res\*,trace:virtio_gpu_virgl_process_command -D debug.log
>
> which shows up as detected in dmesg but not to vulkaninfo:
>
> [ 0.644879] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)
> [ 0.648643] virtio-pci 0000:00:02.0: enabling device (0000 -> 0002)
> [ 0.672391] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> [ 0.678071] Serial: AMBA driver
> [ 0.682122] [drm] pci: virtio-gpu-pci detected at 0000:00:02.0
> [ 0.683249] [drm] Host memory window: 0x8000000000 +0x100000000
> [ 0.683420] [drm] features: +virgl +edid +resource_blob +host_visible
> [ 0.683470] [drm] features: +context_init
> [ 0.695695] [drm] number of scanouts: 1
> [ 0.695837] [drm] number of cap sets: 3
> [ 0.716173] [drm] cap set 0: id 1, max-version 1, max-size 308
> [ 0.716499] [drm] cap set 1: id 2, max-version 2, max-size 1384
> [ 0.716686] [drm] cap set 2: id 4, max-version 0, max-size 160
> [ 0.726001] [drm] Initialized virtio_gpu 0.1.0 0 for 0000:00:02.0 on minor 0
> virgl_resource_create: err=0, res=2
> virgl_renderer_resource_attach_iov: 0x55b843c17a80/2
> virgl_resource_attach_iov: pipe_resource: 0x55b8434da8f0
> vrend_pipe_resource_attach_iov: 0x43
>
> ...
>
> # vulkaninfo --summary
> WARNING: [Loader Message] Code 0 : terminator_CreateInstance: Failed to CreateInstance in ICD 1. Skipping ICD.
> error: XDG_RUNTIME_DIR is invalid or not set in the environment.
> ==========
> VULKANINFO
> ==========
>
> Vulkan Instance Version: 1.3.262
>
>
> Instance Extensions: count = 12
> -------------------------------
> VK_EXT_debug_report : extension revision 10
> VK_EXT_debug_utils : extension revision 2
> VK_KHR_device_group_creation : extension revision 1
> VK_KHR_external_fence_capabilities : extension revision 1
> VK_KHR_external_memory_capabilities : extension revision 1
> VK_KHR_external_semaphore_capabilities : extension revision 1
> VK_KHR_get_physical_device_properties2 : extension revision 2
> VK_KHR_get_surface_capabilities2 : extension revision 1
> VK_KHR_portability_enumeration : extension revision 1
> VK_KHR_surface : extension revision 25
> VK_KHR_surface_protected_capabilities : extension revision 1
> VK_LUNARG_direct_driver_loading : extension revision 1
>
> Instance Layers:
> ----------------
>
> Devices:
> ========
> GPU0:
> apiVersion = 1.3.267
> driverVersion = 0.0.1
> vendorID = 0x10005
> deviceID = 0x0000
> deviceType = PHYSICAL_DEVICE_TYPE_CPU
> deviceName = llvmpipe (LLVM 15.0.3, 128 bits)
> driverID = DRIVER_ID_MESA_LLVMPIPE
> driverName = llvmpipe
> driverInfo = Mesa 23.3.2 (LLVM 15.0.3)
> conformanceVersion = 1.3.1.1
> deviceUUID = 6d657361-3233-2e33-2e32-000000000000
> driverUUID = 6c6c766d-7069-7065-5555-494400000000
>
> With an older and more hacked up set of the blob patches I can get
> vulkaninfo to work but I see multiple GPUs and vkmark falls over when
> mapping stuff:
>
> # vulkaninfo --summary
> render_state_create_resource: res_id = 5
> vkr_context_add_resource: res_id = 5
> vkr_context_import_resource_internal: res_id = 5
> virgl_resource_create: err=0, res=5
> render_state_create_resource: res_id = 6
> vkr_context_add_resource: res_id = 6
> vkr_context_import_resource_internal: res_id = 6
> virgl_resource_create: err=0, res=6
> error: XDG_RUNTIME_DIR is invalid or not set in the environment.
> ==========
> VULKANINFO
> ==========
>
> Vulkan Instance Version: 1.3.262
>
>
> Instance Extensions: count = 12
> -------------------------------
> VK_EXT_debug_report : extension revision 10
> VK_EXT_debug_utils : extension revision 2
> VK_KHR_device_group_creation : extension revision 1
> VK_KHR_external_fence_capabilities : extension revision 1
> VK_KHR_external_memory_capabilities : extension revision 1
> VK_KHR_external_semaphore_capabilities : extension revision 1
> VK_KHR_get_physical_device_properties2 : extension revision 2
> VK_KHR_get_surface_capabilities2 : extension revision 1
> VK_KHR_portability_enumeration : extension revision 1
> VK_KHR_surface : extension revision 25
> VK_KHR_surface_protected_capabilities : extension revision 1
> VK_LUNARG_direct_driver_loading : extension revision 1
> VK_LUNARG_direct_driver_loading : extension revision 1 [0/7869]
>
> Instance Layers:
> ----------------
>
> Devices:
> ========
> GPU0:
> apiVersion = 1.3.230
> driverVersion = 23.3.4
> vendorID = 0x8086
> deviceID = 0xa780
> deviceType = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
> deviceName = Virtio-GPU Venus (Intel(R) Graphics (RPL-S))
> driverID = DRIVER_ID_MESA_VENUS
> driverName = venus
> driverInfo = Mesa 23.3.4
> conformanceVersion = 1.3.0.0
> deviceUUID = 29d2e940-a1a0-3054-0f9a-9f7dec52a084
> driverUUID = b11fafe9-8706-9ab8-0f16-8b272cf893ca
> GPU1:
> apiVersion = 1.2.0
> driverVersion = 23.3.4
> vendorID = 0x10005
> deviceID = 0x0000
> deviceType = PHYSICAL_DEVICE_TYPE_CPU
> deviceName = Virtio-GPU Venus (llvmpipe (LLVM 15.0.6, 256 bits))
> driverID = DRIVER_ID_MESA_VENUS
> driverName = venus
> driverInfo = Mesa 23.3.4
> conformanceVersion = 1.3.0.0
> deviceUUID = 5fb5c03f-c537-f0fe-a7e6-9cd5866acb8d
> driverUUID = b11fafe9-8706-9ab8-0f16-8b272cf893ca
> GPU2:
> apiVersion = 1.3.267
> driverVersion = 0.0.1
> vendorID = 0x10005
> deviceID = 0x0000
> deviceType = PHYSICAL_DEVICE_TYPE_CPU
> deviceName = llvmpipe (LLVM 15.0.3, 128 bits)
> driverID = DRIVER_ID_MESA_LLVMPIPE
> driverName = llvmpipe
> driverInfo = Mesa 23.3.2 (LLVM 15.0.3)
> conformanceVersion = 1.3.1.1
> deviceUUID = 6d657361-3233-2e33-2e32-000000000000
> driverUUID = 6c6c766d-7069-7065-5555-494400000000
> render_state_destroy_resource: res_id = 5
> vkr_context_remove_resource: res_id = 5
> virgl_resource_destroy_func: res=5
> render_state_destroy_resource: res_id = 6
> vkr_context_remove_resource: res_id = 6
> virgl_resource_destroy_func: res=6
>
> running vkmark gives:
>
> # vkmark --winsys kms
> render_state_create_resource: res_id = 7
> vkr_context_add_resource: res_id = 7
> vkr_context_import_resource_internal: res_id = 7
> virgl_resource_create: err=0, res=7
> render_state_create_resource: res_id = 8
> vkr_context_add_resource: res_id = 8
> vkr_context_import_resource_internal: res_id = 8
> virgl_resource_create: err=0, res=8
> virgl_resource_create: err=0, res=9
> virgl_renderer_resource_attach_iov: 0x55615acf7f40/9
> virgl_resource_attach_iov: pipe_resource: 0x55615acf7db0
> vrend_pipe_resource_attach_iov: 0x43
>
> this bit does nothing as VREND_STORAGE_HOST_SYSTEM_MEMORY isn't set.
>
> virgl_resource_create: err=0, res=10
> virgl_renderer_resource_attach_iov: 0x55615ae569a0/10
> virgl_resource_attach_iov: pipe_resource: 0x55615a99ce20
> vrend_pipe_resource_attach_iov: 0x43
> Warning: KMSWindowSystem: Using VK_IMAGE_TILING_OPTIMAL for dmabuf with invalid modifier, but this is not guaranteed to work.
> vkr_dispatch_vkAllocateMemory: mem_index_type:0
> virgl_render_server[2889817]: vkr: failed to import resource: invalid res_id 9
> virgl_render_server[2889817]: vkr: vkAllocateMemory resulted in CS error
> virgl_render_server[2889817]: vkr: ring_submit_cmd: vn_dispatch_command failed
> MESA-VIRTIO: debug: vn_ring_submit abort on fatal
> render_state_destroy_resource: res_id = 7
> vkr_context_remove_resource: res_id = 7
> render_state_destroy_resource: res_id = 8
> vkr_context_remove_resource: res_id = 8
> virgl_resource_destroy_func: res=7
> virgl_resource_destroy_func: res=8
> virgl_resource_destroy_func: res=10
> virgl_resource_destroy_func: res=9
> Aborted
>
> The debug printfs are throughout QEMU and virlgrenderer while trying to
> work out what was going on. While the eventual aim is to enable this
> stack on a ARM64 platform with Xen I wanted to make sure I understand
> the blob resource flow first.
>
> As I understand it we need resource blobs to hold 3D data for rendering
> that is visible to the underlying GPU that will be doing the work. While
> these blobs can be copied back and forth the most efficient way to do
> this is to allocate HOST VISIBLE blobs that are appropriately placed
> and aligned for the host GPU. For Vulkan there will also be the command
> stream which will need to be translated from the Vulkan's portable
> shader IR (as emitted by mesa on the guest) to the underlying shader
> commands on the host system (via mesa on the host).
>
> I can see the host visible region is created as a large chunk of the PCI
> BAR space and is outside of the guests system RAM.
>
> The guest creates a unique resource ID by submitting a
> VIRTIO_GPU_CMD_RESOURCE_CREATE_BLOB command.
>
> Q1: is the actual allocation done here? I assume this happens somewhere
> in the depths of virglrenderer or is it the kernel DRM subsystem?
>
> Q2: should the memory for this resource be visible to the host
> userspace at this point? Where is the mapping into userspace done?
>
> The guest submits a VIRTIO_GPU_CMD_RESOURCE_MAP_BLOB command with a
> unique resource ID and requests it is mapped at an offset into the host
> visible memory region.
>
> Q3: Is all the mapping for host and guest handled by QEMU's memory_region
> code?
>
> Q4: Are there any differences between cards with VRAM and those with a
> unified memory architecture (e.g. using system RAM)?
>
> Finally should we expect to see any other resources
> (RESOURCE_CREATE_2D/3D, TRANSFER_TO_HOST, ATTACH_BACKING) if we have
> host visible blobs properly allocated and working?
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
More information about the virglrenderer-devel
mailing list