<html><head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body> <div class="moz-cite-prefix">On 2021-12-07 5:16 p.m., Alex Deucher wrote: </div> <blockquote type="cite" cite="mid:CADnq5_NkeJsOC_bW2xW7V-aNKd3hhOZ-JmGz0-4gf6QZM2c-ew@mail.gmail.com"> <pre class="moz-quote-pre" wrap="">On Tue, Dec 7, 2021 at 4:59 PM Felix Kuehling <a class="moz-txt-link-rfc2396E" href="mailto:felix.kuehling@amd.com"><felix.kuehling@amd.com></a> wrote: </pre> <blockquote type="cite"> <pre class="moz-quote-pre" wrap=""> On 2021-12-07 9:59 a.m., Philip Yang wrote: </pre> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">If host and amdgpu IOMMU is not enabled or IOMMU is pass through mode, dma_map_page return address is equal to page physical address, use this to set adev->iommu_isolation flag which will be used to optimize memory usage for multi GPU mappings. Signed-off-by: Philip Yang <a class="moz-txt-link-rfc2396E" href="mailto:Philip.Yang@amd.com"><Philip.Yang@amd.com></a> --- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 2 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 27 ++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h index c5cfe2926ca1..fbbe8c7b5d0c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h @@ -1097,6 +1097,8 @@ struct amdgpu_device { struct amdgpu_reset_control *reset_cntl; uint32_t ip_versions[MAX_HWIP][HWIP_MAX_INSTANCE]; + + bool iommu_isolation; }; static inline struct amdgpu_device *drm_to_adev(struct drm_device *ddev) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 3c5afa45173c..6d0f3c477670 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -3364,6 +3364,31 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) return ret; } +/** + * amdgpu_device_check_iommu_isolation - check if IOMMU isolation is enabled + * + * @adev: amdgpu_device pointer + * + * device is in IOMMU isolation mode if dma_map_page return address is not equal + * to page physical address. + */ +static void amdgpu_device_check_iommu_isolation(struct amdgpu_device *adev) +{ + struct page *page; + dma_addr_t addr; + + page = alloc_page(GFP_KERNEL); + if (!page) + return; + addr = dma_map_page(adev->dev, page, 0, PAGE_SIZE, DMA_BIDIRECTIONAL); + if (dma_mapping_error(adev->dev, addr)) + goto out_free_page; + adev->iommu_isolation = (addr != page_to_phys(page)); + dma_unmap_page(adev->dev, addr, PAGE_SIZE, DMA_BIDIRECTIONAL); </pre> </blockquote> <pre class="moz-quote-pre" wrap=""> This is a bit of a hack. Unfortunately it seems there isn't a much better way to do this. I guess you could copy the implementation of dma_map_direct in kernel/dma/mapping.c, but that's more brittle. I think this hack only tells you whether system memory is direct-mapped. The answer may be different for peer VRAM (which isn't supported upstream yet, but it's coming). I think this can happen when the IOMMU is in pass-through mode by default but still used to DMA map physical addresses that are outside the dma mask of the GPU. So a more future proof way would be to store a direct-mapped flag for each GPU-GPU and GPU-System pair somehow. For the GPU->GPU direct mapping flag you'd need to try to DMA-map a page from one GPU's VRAM to the other device. Anyway, that can be done in a later change. For now I'd just change the name of the flag from iommu_isolation to direct_map_ram or ram_is_direct_mapped or similar to be more specific about what it means. </pre> </blockquote> <pre class="moz-quote-pre" wrap=""> This thread might be useful: <a class="moz-txt-link-freetext" href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2021-November%2F330557.html&data=04%7C01%7CPhilip.Yang%40amd.com%7Cfe724d9cbcd743c6156e08d9b9cf464d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637745122159896596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=YW0Hxm8n5xONavU4skohVV3nbrDUEPn3ehX6f9hJe6M%3D&reserved=0">https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2021-November%2F330557.html&data=04%7C01%7CPhilip.Yang%40amd.com%7Cfe724d9cbcd743c6156e08d9b9cf464d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637745122159896596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=YW0Hxm8n5xONavU4skohVV3nbrDUEPn3ehX6f9hJe6M%3D&reserved=0</a> <a class="moz-txt-link-freetext" href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2021-November%2F330583.html&data=04%7C01%7CPhilip.Yang%40amd.com%7Cfe724d9cbcd743c6156e08d9b9cf464d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637745122159896596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZaJgJPDa3akycHAob1y5OWr5iHMhHgqGZSijMDCpLbQ%3D&reserved=0">https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Farchives%2Fdri-devel%2F2021-November%2F330583.html&data=04%7C01%7CPhilip.Yang%40amd.com%7Cfe724d9cbcd743c6156e08d9b9cf464d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637745122159896596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZaJgJPDa3akycHAob1y5OWr5iHMhHgqGZSijMDCpLbQ%3D&reserved=0</a> </pre> </blockquote> Thanks for the helpful info, after testing on different platforms, I will send out v2 patch. Regards, Philip <blockquote type="cite" cite="mid:CADnq5_NkeJsOC_bW2xW7V-aNKd3hhOZ-JmGz0-4gf6QZM2c-ew@mail.gmail.com"> <pre class="moz-quote-pre" wrap=""> Alex </pre> <blockquote type="cite"> <pre class="moz-quote-pre" wrap=""> Regards, Felix </pre> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">+out_free_page: + __free_page(page); +} + static const struct attribute *amdgpu_dev_attributes[] = { &dev_attr_product_name.attr, &dev_attr_product_number.attr, @@ -3767,6 +3792,8 @@ int amdgpu_device_init(struct amdgpu_device *adev, queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work, msecs_to_jiffies(AMDGPU_RESUME_MS)); + amdgpu_device_check_iommu_isolation(adev); + return 0; release_ras_con: </pre> </blockquote> </blockquote> </blockquote> </body> </html>