<div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>I found that if I have HDMI cable connected and desktop environment running, I can still reproduce the low memory speed 100% by logging in over ssh and starting my "cl_slow_test" program. It starts up, does the clCreateContext and starts doing slow memcpy in loop. Now as soon as I move the mouse in desktop env, immediatelly the memcpy speed goes normal.</div><div><br></div><div>This time I enabled amdgpu logging also in kernel driver this way:<br class="m_-1897352530028054162gmail-Apple-interchange-newline"></div><div><br></div><div><div>sudo trace-cmd start -e dma_fence -e gpu_scheduler -e amdgpu -v -e "amdgpu_dm:amdgpu_dc_rreg" -e "amdgpu_dm:amdgpu_dc_wreg" -e "amdgpu_dm:amdgpu_dc_performance" -e "amdgpu:amdgpu_mm_rreg" -e "amdgpu:amdgpu_mm_wreg" -e "amdgpu:amdgpu_iv" -e "amdgpu:amdgpu_bo_create" -e "amdgpu:amdgpu_cs" -e "amdgpu:amdgpu_cs_ioctl" -e "amdgpu:amdgpu_sched_run_job" -e "amdgpu:amdgpu_vm_grab_id" -e "amdgpu:amdgpu_vm_bo_map" -e "amdgpu:amdgpu_vm_bo_unmap" -e "amdgpu:amdgpu_vm_bo_update" -e "amdgpu:amdgpu_vm_bo_cs" -e "amdgpu:amdgpu_vm_set_ptes" -e "amdgpu:amdgpu_vm_copy_ptes" -e "amdgpu:amdgpu_vm_flush" -e "amdgpu:amdgpu_pasid_allocated" -e "amdgpu:amdgpu_pasid_freed" -e "amdgpu:amdgpu_bo_list_set" -e "amdgpu:amdgpu_cs_bo_status" -e "amdgpu:amdgpu_cs_bo_status" -e "amdgpu:amdgpu_bo_move" -e "amdgpu:amdgpu_ib_pipe_sync"<br></div><div>cd /sys/kernel/debug/tracing && cat trace_pipe | tee slow_test_amdgpu_log.txt<br></div><div><br></div><div>Attached the log file. In the log file there are 3 annotations: "CL SLOW TEST STARTED, MEMORY SPEED SLOW", "MOUSE_MOVED, MEMORY SPEED RESTORES" and "CL SLOW TEST CLOSED".</div><div><br></div><div>Maybe this helps to understand what magic the mouse moving does to the system? I am really stuck with this - 100x slowdown is obviously not acceptable. Maybe there is a way how to work around the issue at least?</div><div><br></div><div>--</div></div><div>Lauri</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Mar 10, 2019 at 9:13 PM Lauri Ehrenpreis <<a href="mailto:laurioma@gmail.com" target="_blank">laurioma@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Seems sysbench cpu test does not slow down:<div>1) run sysbench cpu on idle machine:</div><div> sysbench cpu run<br></div><div>...</div><div><div>General statistics:</div><div> total time: 10.0033s</div><div> total number of events: 19052</div></div><div>2) start ./cl_slow_test 1 10000 in background</div><div>3) run sysbench again</div><div>sysbench cpu run<br></div><div>..</div><div><div>General statistics:</div><div> total time: 10.0036s</div><div> total number of events: 18979</div></div><div><br></div><div>So if did not slow down considerably.</div><div><br></div><div>If I do similar test with sysbench memory test I get following results:</div><div>1) run sysbench memory on idle machine:</div><div>sysbench memory --memory-block-size=32M --memory-total-size=100G run<br></div><div>...</div><div><div>66432.00 MiB transferred (6638.95 MiB/sec)</div></div><div>2) start ./cl_slow_test 1 10000 in background<br></div><div>3) sysbench memory --memory-block-size=32M --memory-total-size=100G run</div><div>...</div><div><div>672.00 MiB transferred (66.40 MiB/sec)</div></div><div><br></div><div>It confirms that memory speed is reduced 100x :(</div><div><br></div><div>--</div><div>Lauri</div><div><br></div></div></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Mar 9, 2019 at 9:22 PM Jan Vesely <<a href="mailto:jan.vesely@rutgers.edu" target="_blank">jan.vesely@rutgers.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sat, Mar 9, 2019 at 1:54 AM Lauri Ehrenpreis <<a href="mailto:laurioma@gmail.com" target="_blank">laurioma@gmail.com</a>> wrote:<br>
><br>
> Even if it's using CPU for OCL (I know it's not doing this), why does memcpy on CPU slow down permanently, if I'm not doing anything with OpenCL after clCreateContext?<br>
><br>
> As you see from test program it just does clCreateContext and then a loop of memcpy-s on CPU.<br>
><br>
> Also I found out that writing different values to /sys/class/drm/card0/device/power_dpm_force_performance_level changes my max memcpy speed on CPU:<br>
><br>
> echo "low" > /sys/class/drm/card0/device/power_dpm_force_performance_level<br>
> ./cl_slow_test 1 5<br>
> got 1 platforms 1 devices<br>
> speed 731.810425 avg 731.810425 mbytes/s<br>
> speed 163.425583 avg 447.618011 mbytes/s<br>
> speed 123.441612 avg 339.559235 mbytes/s<br>
> speed 121.655266 avg 285.083252 mbytes/s<br>
> speed 123.806801 avg 252.827972 mbytes/s<br>
><br>
> echo "high" > /sys/class/drm/card0/device/power_dpm_force_performance_level<br>
> ./cl_slow_test 1 5<br>
> got 1 platforms 1 devices<br>
> speed 3742.063721 avg 3742.063721 mbytes/s<br>
> speed 836.148987 avg 2289.106445 mbytes/s<br>
> speed 189.379166 avg 1589.197266 mbytes/s<br>
> speed 189.271393 avg 1239.215820 mbytes/s<br>
> speed 188.290451 avg 1029.030762 mbytes/s<br>
><br>
> echo "profile_standard" > /sys/class/drm/card0/device/power_dpm_force_performance_level<br>
> ./cl_slow_test 1 5<br>
> got 1 platforms 1 devices<br>
> speed 2303.955566 avg 2303.955566 mbytes/s<br>
> speed 2298.224121 avg 2301.089844 mbytes/s<br>
> speed 2295.585205 avg 2299.254883 mbytes/s<br>
> speed 2295.762939 avg 2298.381836 mbytes/s<br>
> speed 2288.766602 avg 2296.458740 mbytes/s<br>
><br>
> echo "profile_peak" > /sys/class/drm/card0/device/power_dpm_force_performance_level<br>
> ./cl_slow_test 1 5<br>
> got 1 platforms 1 devices<br>
> speed 3710.360352 avg 3710.360352 mbytes/s<br>
> speed 3713.660400 avg 3712.010254 mbytes/s<br>
> speed 3797.630859 avg 3740.550537 mbytes/s<br>
> speed 3708.004883 avg 3732.414062 mbytes/s<br>
> speed 3796.403076 avg 3745.211914 mbytes/s<br>
><br>
> However none of those is close to the memcpy speed I get when I don't do clCreateContext (my test prog has first arg 0):<br>
> ./cl_slow_test 0 5<br>
> speed 7299.201660 avg 7299.201660 mbytes/s<br>
> speed 9298.841797 avg 8299.021484 mbytes/s<br>
> speed 9360.181641 avg 8652.742188 mbytes/s<br>
> speed 9004.759766 avg 8740.746094 mbytes/s<br>
> speed 9414.607422 avg 8875.518555 mbytes/s<br>
><br>
> Also attached clinfo.txt. It shows that opencl is using GPU so device node permissions are probably not the issue.<br>
<br>
Is it only memory accesses or does overall CPU performance degrade<br>
(including compute - say sysbench) as well?<br>
<br>
Jan<br>
<br>
> --<br>
> Lauri<br>
><br>
> On Fri, Mar 8, 2019 at 10:35 PM Alex Deucher <<a href="mailto:alexdeucher@gmail.com" target="_blank">alexdeucher@gmail.com</a>> wrote:<br>
>><br>
>> I think you are probably using the CPU for OCL in the remote login<br>
>> case. When you log into the desktop, the permissions on the device<br>
>> nodes get changed dynamically to support accelerated rendering. You<br>
>> probably need to change the permissions on the device nodes manually<br>
>> if you are not logging into the desktop.<br>
>><br>
>> Alex<br>
>><br>
>> On Fri, Mar 8, 2019 at 2:43 PM Lauri Ehrenpreis <<a href="mailto:laurioma@gmail.com" target="_blank">laurioma@gmail.com</a>> wrote:<br>
>> ><br>
>> > Hi!<br>
>> ><br>
>> > I am using Ryzen 2400G with Gigabyte AMD B450 AORUS board. I have latest bios, ubuntu 18.04 and latest mainline kernel (5.0.0-050000-generic) installed. Also I have rocm-dev 2.1.96 but no rock-dkms installed.<br>
>> ><br>
>> > I found that when I log in over ssh and try to use OpenCL (doing clCreateContext is enough) then cpu memory accesses after that will slow down by 100x.<br>
>> > If I connect HDMI cable and log in to desktop mode then this does not happen. Also if I don't call clCreateContext then everything works properly.<br>
>> ><br>
>> > Attached the test program and kernel log also. Test works like that :<br>
>> > g++ cl_slow_test.cpp -o cl_slow_test -I /opt/rocm/opencl/include/ -L /opt/rocm/opencl/lib/x86_64/ -lOpenCL<br>
>> > lauri@rv:~$ ./cl_slow_test 0 5<br>
>> > speed 7003.145508 avg 7003.145508 mbytes/s<br>
>> > speed 8427.357422 avg 7715.251465 mbytes/s<br>
>> > speed 9203.049805 avg 8211.184570 mbytes/s<br>
>> > speed 9845.956055 avg 8619.877930 mbytes/s<br>
>> > speed 9882.748047 avg 8872.452148 mbytes/s<br>
>> > lauri@rv:~$ ./cl_slow_test 1 5<br>
>> > got 1 platforms 1 devices<br>
>> > speed 1599.803589 avg 1599.803589 mbytes/s<br>
>> > speed 1665.426392 avg 1632.614990 mbytes/s<br>
>> > speed 146.137253 avg 1137.122437 mbytes/s<br>
>> > speed 121.056877 avg 883.106018 mbytes/s<br>
>> > speed 122.428970 avg 730.970581 mbytes/s<br>
>> ><br>
>> > I also tried latest amd-staging kernel <a href="https://github.com/M-Bab/linux-kernel-amdgpu-binaries" rel="noreferrer" target="_blank">https://github.com/M-Bab/linux-kernel-amdgpu-binaries</a> and it had the same issue.<br>
>> ><br>
>> > Can anyone point me into right direction?<br>
>> ><br>
>> > Br,<br>
>> > Lauri<br>
>> > _______________________________________________<br>
>> > amd-gfx mailing list<br>
>> > <a href="mailto:amd-gfx@lists.freedesktop.org" target="_blank">amd-gfx@lists.freedesktop.org</a><br>
>> > <a href="https://lists.freedesktop.org/mailman/listinfo/amd-gfx" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/amd-gfx</a><br>
><br>
> _______________________________________________<br>
> amd-gfx mailing list<br>
> <a href="mailto:amd-gfx@lists.freedesktop.org" target="_blank">amd-gfx@lists.freedesktop.org</a><br>
> <a href="https://lists.freedesktop.org/mailman/listinfo/amd-gfx" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/amd-gfx</a><br>
</blockquote></div>
</blockquote></div>