<div dir="ltr"> <div class="gmail_quote"><div dir="ltr">On Wed, Sep 26, 2018 at 3:29 AM Gurchetan Singh <<a href="mailto:gurchetansingh@chromium.org">gurchetansingh@chromium.org</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Tue, Sep 25, 2018 at 2:10 AM Gerd Hoffmann <<a href="mailto:kraxel@redhat.com" target="_blank">kraxel@redhat.com</a>> wrote: > > Hi, > > > > Who will do the actual allocations? I expect we need new virglrenderer > > > functions for that? > > > > The decision to back memory via iovecs or host memory is up to the > > VMM. > > What exactly do you mean with "via iovecs"? The current way to allocate > resources? They are guest-allocated and the iovecs passed to > virglrenderer point into guest memory. So that clearly is *not* in the > hands of the VMM. Or do you mean something else? I guess I'm just brainstorming about one-copy with virgl and how we want to implement it. The simplest case that can be improved with host memory is: (1) Guest app maps the buffer (glMapBufferRange --> virgl_buffer_transfer_map). (i) If the buffer is not marked clean (texture buffers, SSBOs) this will trigger a TRANSFER_FROM_HOST_3D (copies++) (2) The guest app copies the buffer data (unavoidable - copies++) (3) Guest unmaps the buffer, triggering a TRANSFER_TO_HOST (copies++). For host GL buffers, the copies are done in {vrend_renderer_transfer_write_iov, vrend_renderer_transfer_send_iov}. If there are N iovecs backing the guest resource, we will have N copies (see vrend_write_to_iovec, vrend_read_from_iovec). udmabuf could be helpful, since it bundles up the iovecs and it will make the N small copies into one big copy. udmabuf could also eliminate some copies for textures completely. Right now, for most textures, virglrenderer copies iovecs into a temporary buffer (see read_transfer_data), and then calls glTexSubImage2D*. Just mmaping the udmabuf and calling glTexSubImage2D* is definite win. But making host memory guest visible will bring the worst-case buffer copies from 3 to 1. For textures, if we start counting when the GPU buffer gets detiled, there will be 5 copies currently, 3 with udmabuf, and 1 with host exposed memory. > > > To make sure we all are on the same page wrt. resource allocation, the > workflow we have now looks like this: > > (1) guest virtio-gpu driver allocates resource. Uses normal (guest) ram. > Resources can be scattered. > (2) guest driver creates resources (RESOURCE_CREATE_*). > (3) qemu (virgl=off) or virglrenderer (virgl=on) creates host resource. > virglrenderer might use a different format (tiling, ...). > (4) guest sets up backing storage (RESOURCE_ATTACH_BACKING). > (5) qemu creates a iovec for the guest resource. > (6) guest writes data to resource. > (7) guest requests a transfer (TRANSFER_TO_HOST_*). > (8) qemu or virglrenderer copy data from guest resource to > host resource, possibly converting (again tiling, ...). > (9) guest can use the resource now ... > > > One thing I'm prototyping right now is zerocopy resources, the workflow > changes to look like this: > > (2) guest additionally sets a flag to request a zerocopy buffer. > (3) not needed (well, the bookkeeping part of it is still needed, but > it would *not* allocate a host resource). > (5) qemu additionally creates a host dma-buf for the guest resource > using the udmabuf driver. > (7+8) not needed. > > Right now I have (not tested yet) code to handle dumb buffers. > Interfacing to guest userspace (virtio-gpu driver ioctls) is not > there yet. Interfacing with virglrenderer isn't there yet either. > > I expect that doesn't solve the coherent mapping issue. The host gpu > could import the dma-buf of the resource, but as it has no control over > the allocation it might not be able to use it without copying. > > > I'm not sure how the API for coherent resources should look like. > One option I see is yet another resource flag, so the workflow would > change like this (with virgl=on only ...): > > (2) guest additionally sets a flag to request a coherent resource. > (3) virglrenderer would create a coherent host resource. > (4) guest finds some address space in the (new) pci bar and asks > for the resource being mapped there (new command needed for > this). > (5) qemu maps the coherent resource into the pci bar. > (7+8) not needed. > > Probably works for GL_MAP_COHERENT_BIT use case. Dunno about vulkan. > > Interfaces to guest userspace and virglrenderer likewise need updates > to support this. > > > > A related question: are we going to also expose host memory to the > > guest for the non-{GL_MAP_COHERENT_BIT, > > VK_MEMORY_PROPERTY_HOST_COHERENT_BIT} cases? > > The guest should be able to do that, yes. In case both coherent and > zerocopy resources are supported by the host it can even pick. > > coherent resources will be limited though (pci bar size, also because > we don't want allow guests allocate unlimited host memory for security > reasons), so using them for everything is probably not a good idea. The 64-bit BAR should be enough, especially if it's managed intelligently. Vulkan may take some time and I don't think stacks with host GLES drivers support GL_MAP_COHERENT_BIT, so there will be cases when that space goes unused. Here's one possible flow: i) virtio_gpu_resource_create_coherent -- for strictly coherent needs (i.e, no unmap needed) ii) virtio_gpu_resource_create_3d -- may or may not be host backed (depends on the PCI bar size, platform-specific information -- guest doesn't need to know) The guest would still issue the transfer ioctls for the virtio_gpu_resource_create_3d resource, but the work performed would be pared down when backed by host memory. This will require increased VMM <--> virglrenderer inter-op. Maybe behind a flag that QEMU doesn't set, but cros_vm will. WDTY? </blockquote><div>I think that's a pretty good middle ground. The amount of inter-op between vmm and virglrenderer is already fairly high, especially with the stuff crosvm does to ensure virtio_gpu resource are allocated in a way so that they can't be sent over wayland connections.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> > > cheers, > Gerd > _______________________________________________ virglrenderer-devel mailing list <a href="mailto:virglrenderer-devel@lists.freedesktop.org" target="_blank">virglrenderer-devel@lists.freedesktop.org</a> <a href="https://lists.freedesktop.org/mailman/listinfo/virglrenderer-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/virglrenderer-devel</a> </blockquote></div></div>