<div class="__aliyun_email_body_block"><div style="line-height:1.7;font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;"><div style="clear:both;">[snip]<p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Fd0 = open(card0)</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Fd1 = open(card1)</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Vm0 =xe_vm_create(fd0) //driver create process xe_svm on the process's first vm_create</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Vm1 = xe_vm_create(fd1) //driver re-use xe_svm created above if called from same process</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Queue0 = xe_exec_queue_create(fd0, vm0)</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Queue1 = xe_exec_queue_create(fd1, vm1)</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">//check p2p capability calling L0 API….</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span class=" __aliyun_node_has_bgcolor" style="margin:.0px;padding:.0px;border:.0px;outline:.0px;background:yellow;">ptr = malloc()//this replace bo_create, vm_bind, dma-import/export</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Xe_exec(queue0,<span > </span><span class=" __aliyun_node_has_color" style="margin:.0px;padding:.0px;border:.0px;outline:.0px;color:#fa0000;">pt</span>r)//submit gpu job which use ptr, on card0</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span style="margin:.0px;padding:.0px;border:.0px;outline:.0px;">Xe_exec(queue1,<span > </span><span class=" __aliyun_node_has_color" style="margin:.0px;padding:.0px;border:.0px;outline:.0px;color:#fa0000;">ptr</span>)//submit gpu job which use ptr, on card1</span></p><p style="color:#000000;font-family:Helvetica,Tahoma,Arial;font-size:14.0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:.0px;text-transform:none;word-spacing:.0px;white-space:normal;text-decoration-thickness:initial;text-decoration-style:initial;text-decoration-color:initial;margin-left:72.0px;"><span class=" __aliyun_node_has_color" style="margin:.0px;padding:.0px;border:.0px;outline:.0px;color:#fa0000;">//Gpu page fault handles memory allocation/migration/mapping to gpu</span></p></div><div style="clear:both;">[snip]</div><div style="clear:both;">Hi Oak,</div><div style="clear:both;">From your sample code, you not only need va-manager cross gpu devices, but also cpu, right?</div><div style="clear:both;">I think you need a UVA (unified va) manager in user space and make range of drm_gpuvm reserved from cpu va space. In that way, malloc's va and gpu va are in same space and will not conflict. And then via HMM mechanism, gpu devices can safely use VA passed from HMM.</div><div style="clear:both;"><br ></div><div style="clear:both;">By the way, I'm not familiar with drm_gpuvm, traditionally, gpu driver often put va-manager in user space, not sure what's benefit we can get from drm_gpuvm invented in kernel space. Can anyone help explain more?</div><div style="clear:both;"><br ></div><div style="clear:both;">- Chunming</div><blockquote style="margin-right:.0px;margin-top:.0px;margin-bottom:.0px;font-family:Tahoma,Arial,STHeiti,SimSun;font-size:14.0px;color:#000000;"><div style="clear:both;">------------------------------------------------------------------</div><div style="clear:both;">发件人：Zeng, Oak <oak.zeng@intel.com></div><div style="clear:both;">发送时间：2024年1月25日(星期四) 09:17</div><div style="clear:both;">收件人："Christian König" <christian.koenig@amd.com>; Danilo Krummrich <dakr@redhat.com>; Dave Airlie <airlied@redhat.com>; Daniel Vetter <daniel@ffwll.ch>; Felix Kuehling <felix.kuehling@amd.com>; "Shah, Ankur N" <ankur.n.shah@intel.com>; "Winiarski, Michal" <michal.winiarski@intel.com></div><div style="clear:both;">抄 送："Brost, Matthew" <matthew.brost@intel.com>; Thomas.Hellstrom@linux.intel.com <Thomas.Hellstrom@linux.intel.com>; "Welty, Brian" <brian.welty@intel.com>; dri-devel@lists.freedesktop.org <dri-devel@lists.freedesktop.org>; "Ghimiray, Himal Prasad" <himal.prasad.ghimiray@intel.com>; "Gupta, saurabhg" <saurabhg.gupta@intel.com>; "Bommu, Krishnaiah" <krishnaiah.bommu@intel.com>; "Vishwanathapura, Niranjana" <niranjana.vishwanathapura@intel.com>; intel-xe@lists.freedesktop.org <intel-xe@lists.freedesktop.org></div><div style="clear:both;">主 题：RE: Making drm_gpuvm work across gpu devices</div><div style="clear:both;"><br ></div><div class="WordSection1" style="page:WordSection1;"><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">Hi Christian,</p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">Even though I mentioned KFD design, I didn’t mean to copy the KFD design. I also had hard time to understand the difficulty of KFD under virtualization environment.</p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">For us, Xekmd doesn't need to know it is running under bare metal or virtualized environment. Xekmd is always a guest driver. All the virtual address used in xekmd is guest virtual address. For SVM, we require all the VF devices share one
single shared address space with guest CPU program. So all the design works in bare metal environment can automatically work under virtualized environment. +<a href="mailto:ankur.n.shah@intel.com" target="_blank"><span style="font-family:Calibri,sans-serif;text-decoration:none;">@Shah,
Ankur N</span></a> +<a href="mailto:michal.winiarski@intel.com" target="_blank"><span style="font-family:Calibri,sans-serif;text-decoration:none;">@Winiarski, Michal</span></a> to backup me if I am wrong.</p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">Again, shared virtual address space b/t cpu and all gpu devices is a hard requirement for our system allocator design (which means malloc’ed memory, cpu stack variables, globals can be directly used in gpu program. Same requirement as kfd
SVM design). This was aligned with our user space software stack. </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">For anyone who want to implement system allocator, or SVM, this is a hard requirement. I started this thread hoping I can leverage the drm_gpuvm design to manage the shared virtual address space (as the address range split/merge function
was scary to me and I didn’t want re-invent). I guess my takeaway from this you and Danilo is this approach is a NAK. Thomas also mentioned to me drm_gpuvm is a overkill for our svm address range split/merge. So I will make things work first by manage address
range xekmd internally. I can re-look drm-gpuvm approach in the future.</p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">Maybe a pseudo user program can illustrate our programming model:</p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >Fd0 = open(card0)</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >Fd1 = open(card1)</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >Vm0 =xe_vm_create(fd0) //driver create process xe_svm on the process's first vm_create</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >Vm1 = xe_vm_create(fd1) //driver re-use xe_svm created above if called from same process</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >Queue0 = xe_exec_queue_create(fd0, vm0)</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >Queue1 = xe_exec_queue_create(fd1, vm1)</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >//check p2p capability calling L0 API….</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span class=" __aliyun_node_has_bgcolor" style="background:yellow;">ptr = malloc()//this replace bo_create, vm_bind, dma-import/export</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >Xe_exec(queue0, <span class=" __aliyun_node_has_color" style="color:#fa0000;">
pt</span>r)//submit gpu job which use ptr, on card0</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span >Xe_exec(queue1, <span class=" __aliyun_node_has_color" style="color:#fa0000;">
ptr</span>)//submit gpu job which use ptr, on card1</span></p><p style="margin-left:72.0px;margin-top:14.0px;margin-bottom:14.0px;"><span class=" __aliyun_node_has_color" style="color:#fa0000;">//Gpu page fault handles memory allocation/migration/mapping to gpu</span></p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><span > </span></p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><span >As you can see, from above model, our design is a little bit different than the KFD design. user need to explicitly create gpuvm (vm0 and vm1 above) for each gpu device. Driver internally have a xe_svm represent the shared
address space b/t cpu and multiple gpu devices. But end user doesn’t see and no need to create xe_svm. The shared virtual address space is really managed by linux core mm (through the vma struct, mm_struct etc). From each gpu device’s perspective, it just
operate under its own gpuvm, not aware of the existence of other gpuvm, even though in reality all those gpuvm shares a same virtual address space.</span></p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><span > </span></p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><span >See one more comment inline</span></p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><div style="border-top:none;border-right:none;border-bottom:none;border-image:initial;border-left:2.0px solid blue;padding:.0px .0px .0px 5.0px;"><div style="border-right:none;border-bottom:none;border-left:none;border-image:initial;border-top:1.0px solid #e1e1e1;padding:4.0px .0px .0px;"><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><span style="font-weight:700;">From:</span><span > Christian König <christian.koenig@amd.com>
<br ><b >Sent:</b> Wednesday, January 24, 2024 3:33 AM<br ><b >To:</b> Zeng, Oak <oak.zeng@intel.com>; Danilo Krummrich <dakr@redhat.com>; Dave Airlie <airlied@redhat.com>; Daniel Vetter <daniel@ffwll.ch>; Felix Kuehling <felix.kuehling@amd.com><br ><b >Cc:</b> Welty, Brian <brian.welty@intel.com>; dri-devel@lists.freedesktop.org; intel-xe@lists.freedesktop.org; Bommu, Krishnaiah <krishnaiah.bommu@intel.com>; Ghimiray, Himal Prasad <himal.prasad.ghimiray@intel.com>; Thomas.Hellstrom@linux.intel.com; Vishwanathapura,
Niranjana <niranjana.vishwanathapura@intel.com>; Brost, Matthew <matthew.brost@intel.com>; Gupta, saurabhg <saurabhg.gupta@intel.com><br ><b >Subject:</b> Re: Making drm_gpuvm work across gpu devices</span></p></div><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">Am 23.01.24 um 20:37 schrieb Zeng, Oak:<br ><br ></p><div ><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">[SNIP] </p><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';"> </pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">Yes most API are per device based.</pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';"> </pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">One exception I know is actually the kfd SVM API. If you look at the svm_ioctl function, it is per-process based. Each kfd_process represent a process across N gpu devices.</pre></div><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><br >
Yeah and that was a big mistake in my opinion. We should really not do that ever again.<br ><br ><br ></p><div ><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">Need to say, kfd SVM represent a shared virtual address space across CPU and all GPU devices on the system. This is by the definition of SVM (shared virtual memory). This is very different from our legacy gpu *device* driver which works for only one device (i.e., if you want one device to access another device's memory, you will have to use dma-buf export/import etc).</pre></div><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><br >
Exactly that thinking is what we have currently found as blocker for a virtualization projects. Having SVM as device independent feature which somehow ties to the process address space turned out to be an extremely bad idea.<br ><br >
The background is that this only works for some use cases but not all of them.<br ><br >
What's working much better is to just have a mirror functionality which says that a range A..B of the process address space is mapped into a range C..D of the GPU address space.<br ><br >
Those ranges can then be used to implement the SVM feature required for higher level APIs and not something you need at the UAPI or even inside the low level kernel memory management.<br ><br >
When you talk about migrating memory to a device you also do this on a per device basis and *not* tied to the process address space. If you then get crappy performance because userspace gave contradicting information where to migrate memory then that's a bug
in userspace and not something the kernel should try to prevent somehow.<br ><br >
[SNIP]<br ><br ></p><div ><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">I think if you start using the same drm_gpuvm for multiple devices you</pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">will sooner or later start to run into the same mess we have seen with</pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">KFD, where we moved more and more functionality from the KFD to the DRM</pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">render node because we found that a lot of the stuff simply doesn't work</pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">correctly with a single object to maintain the state.</pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';"> </pre><pre style="margin:.0px;font-size:13.3px;font-family:'Courier New';">As I understand it, KFD is designed to work across devices. A single pseudo /dev/kfd device represent all hardware gpu devices. That is why during kfd open, many pdd (process device data) is created, each for one hardware device for this process.</pre></div><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><br >
Yes, I'm perfectly aware of that. And I can only repeat myself that I see this design as a rather extreme failure. And I think it's one of the reasons why NVidia is so dominant with Cuda.<br ><br >
This whole approach KFD takes was designed with the idea of extending the CPU process into the GPUs, but this idea only works for a few use cases and is not something we should apply to drivers in general.<br ><br >
A very good example are virtualization use cases where you end up with CPU address != GPU address because the VAs are actually coming from the guest VM and not the host process.</p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">I don’t get the problem here. For us, under virtualization, both the cpu address and gpu virtual address operated in xekmd is guest virtual address. They can still share the same virtual address space (as SVM required)</p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"> </p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;">Oak</p><p class="MsoNormal" style="margin:.0px;font-size:14.7px;font-family:Calibri,sans-serif;"><br ><br >
SVM is a high level concept of OpenCL, Cuda, ROCm etc.. This should not have any influence on the design of the kernel UAPI.<br ><br >
If you want to do something similar as KFD for Xe I think you need to get explicit permission to do this from Dave and Daniel and maybe even Linus.<br ><br >
Regards,<br >
Christian.</p></div></div></blockquote></div></div>