<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi Lauri, Thanks for your persistence. Seeing that this is reproducible on several boards with up-to-date BIOS is really helpful and gives me some confidence that it's more than a weird vendor or board-specific corner case and that we should be able to reproduce it. Yong is going to start looking into this problem. Regards, Felix <div class="moz-cite-prefix">On 3/14/2019 12:41 PM, Lauri Ehrenpreis wrote: </div> <blockquote type="cite" cite="mid:CAGyaPbCvOijB=pWe3zkMze3aT+Yr86aQYK2OP=P8qpOdCzaORg@mail.gmail.com"> <div dir="ltr">Yes it affects this a bit but it doesn't get the speed up to "normal" level. I got best results with "profile_peak" - then the memcpy speed on CPU is 1/3 of what it is without opencl initialization: <div> </div> <div> <div> echo "profile_peak" > /sys/class/drm/card0/device/power_dpm_force_performance_level </div> <div> <div>./cl_slow_test 1 5</div> <div>got 1 platforms 1 devices</div> <div>speed 3710.360352 avg 3710.360352 mbytes/s</div> <div>speed 3713.660400 avg 3712.010254 mbytes/s</div> <div>speed 3797.630859 avg 3740.550537 mbytes/s</div> <div>speed 3708.004883 avg 3732.414062 mbytes/s</div> <div>speed 3796.403076 avg 3745.211914 mbytes/s</div> </div> <div> </div> <div>Without calling clCreateContext:</div> <div> <div>./cl_slow_test 0 5</div> <div>speed 7299.201660 avg 7299.201660 mbytes/s</div> <div>speed 9298.841797 avg 8299.021484 mbytes/s</div> <div>speed 9360.181641 avg 8652.742188 mbytes/s</div> <div>speed 9004.759766 avg 8740.746094 mbytes/s</div> <div>speed 9414.607422 avg 8875.518555 mbytes/s</div> </div> </div> <div> </div> <div>--</div> <div>Lauri</div> </div> <div class="gmail_quote"> <div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 5:46 PM Ernst Sjöstrand <<a href="mailto:ernstp@gmail.com" moz-do-not-send="true">ernstp@gmail.com</a>> wrote: </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Does echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level or setting cpu scaling governor to performance affect it at all? Regards //Ernst Den tors 14 mars 2019 kl 14:31 skrev Lauri Ehrenpreis <<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a>>: > > I tried also with those 2 boards now: > <a href="https://www.asrock.com/MB/AMD/Fatal1ty%20B450%20Gaming-ITXac/index.asp" rel="noreferrer" target="_blank" moz-do-not-send="true"> https://www.asrock.com/MB/AMD/Fatal1ty%20B450%20Gaming-ITXac/index.asp</a> > <a href="https://www.msi.com/Motherboard/B450I-GAMING-PLUS-AC" rel="noreferrer" target="_blank" moz-do-not-send="true"> https://www.msi.com/Motherboard/B450I-GAMING-PLUS-AC</a> > > Both are using latest BIOS, ubuntu 18.10, kernel <a href="https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0.2/" rel="noreferrer" target="_blank" moz-do-not-send="true"> https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0.2/</a> > > There are some differences in dmesg (asrock has some amdgpu assert in dmesg) but otherwise results are exactly the same. > In desktop env cl_slow_test works fast, over ssh terminal it doesn't. If i move mouse then it starts working fast in terminal as well. > > So one can't use OpenCL without monitor and desktop env running and this happens with 2 different chipsets (b350 & b450), latest bios from 3 different vendors, latest kernel and latest rocm. This doesn't look like edge case with unusual setup to me.. > > Attached dmesg, dmidecode, and clinfo from both boards. > > -- > Lauri > > On Wed, Mar 13, 2019 at 10:15 PM Lauri Ehrenpreis <<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a>> wrote: >> >> For reproduction only the tiny cl_slow_test.cpp is needed which is attached to first e-mail. >> >> System information is following: >> CPU: Ryzen5 2400G >> Main board: Gigabyte AMD B450 AORUS mini itx: <a href="https://www.gigabyte.com/Motherboard/B450-I-AORUS-PRO-WIFI-rev-10#kf" rel="noreferrer" target="_blank" moz-do-not-send="true"> https://www.gigabyte.com/Motherboard/B450-I-AORUS-PRO-WIFI-rev-10#kf</a> >> BIOS: F5 8.47 MB 2019/01/25 (latest) >> Kernel: <a href="https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0/" rel="noreferrer" target="_blank" moz-do-not-send="true"> https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.0/</a> (amd64) >> OS: Ubuntu 18.04 LTS >> rocm-opencl-dev installation: >> wget -qO - <a href="http://repo.radeon.com/rocm/apt/debian/rocm.gpg.key" rel="noreferrer" target="_blank" moz-do-not-send="true"> http://repo.radeon.com/rocm/apt/debian/rocm.gpg.key</a> | sudo apt-key add - >> echo 'deb [arch=amd64] <a href="http://repo.radeon.com/rocm/apt/debian/" rel="noreferrer" target="_blank" moz-do-not-send="true"> http://repo.radeon.com/rocm/apt/debian/</a> xenial main' | sudo tee /etc/apt/sources.list.d/rocm.list >> sudo apt install rocm-opencl-dev >> >> Also exactly the same issue happens with this board: <a href="https://www.gigabyte.com/Motherboard/GA-AB350-Gaming-3-rev-1x#kf" rel="noreferrer" target="_blank" moz-do-not-send="true"> https://www.gigabyte.com/Motherboard/GA-AB350-Gaming-3-rev-1x#kf</a> >> >> I have MSI and Asrock mini itx boards ready as well, So far didn't get amdgpu & opencl working there but I'll try again tomorrow.. >> >> -- >> Lauri >> >> >> On Wed, Mar 13, 2019 at 8:51 PM Kuehling, Felix <<a href="mailto:Felix.Kuehling@amd.com" target="_blank" moz-do-not-send="true">Felix.Kuehling@amd.com</a>> wrote: >>> >>> Hi Lauri, >>> >>> I still think the SMU is doing something funny, but rocm-smi isn't >>> showing enough information to really see what's going on. >>> >>> On APUs the SMU firmware is embedded in the system BIOS. Unlike discrete >>> GPUs, the SMU firmware is not loaded by the driver. You could try >>> updating your system BIOS to the latest version available from your main >>> board vendor and see if that makes a difference. It may include a newer >>> version of the SMU firmware, potentially with a fix. >>> >>> If that doesn't help, we'd have to reproduce the problem in house to see >>> what's happening, which may require the same main board and BIOS version >>> you're using. We can ask our SMU firmware team if they've ever >>> encountered your type of problem. But I don't want to give you too much >>> hope. It's a tricky problem involving HW, firmware and multiple driver >>> components in a fairly unusual configuration. >>> >>> Regards, >>> Felix >>> >>> On 2019-03-13 7:28 a.m., Lauri Ehrenpreis wrote: >>> > What I observe is that moving the mouse made the memory speed go up >>> > and also it made mclk=1200Mhz in rocm-smi output. >>> > However if I force mclk to 1200Mhz myself then memory speed is still >>> > slow. >>> > >>> > So rocm-smi output when memory speed went fast due to mouse movement: >>> > rocm-smi >>> > ======================== ROCm System Management Interface >>> > ======================== >>> > ================================================================================================ >>> > GPU Temp AvgPwr SCLK MCLK PCLK Fan Perf >>> > PwrCap SCLK OD MCLK OD GPU% >>> > GPU[0] : WARNING: Empty SysFS value: pclk >>> > GPU[0] : WARNING: Unable to read >>> > /sys/class/drm/card0/device/gpu_busy_percent >>> > 0 44.0c N/A 400Mhz 1200Mhz N/A 0% manual N/A >>> > 0% 0% N/A >>> > ================================================================================================ >>> > ======================== End of ROCm SMI Log >>> > ======================== >>> > >>> > And rocm-smi output when I forced memclk=1200MHz myself: >>> > rocm-smi --setmclk 2 >>> > rocm-smi >>> > ======================== ROCm System Management Interface >>> > ======================== >>> > ================================================================================================ >>> > GPU Temp AvgPwr SCLK MCLK PCLK Fan Perf >>> > PwrCap SCLK OD MCLK OD GPU% >>> > GPU[0] : WARNING: Empty SysFS value: pclk >>> > GPU[0] : WARNING: Unable to read >>> > /sys/class/drm/card0/device/gpu_busy_percent >>> > 0 39.0c N/A 400Mhz 1200Mhz N/A 0% manual N/A >>> > 0% 0% N/A >>> > ================================================================================================ >>> > ======================== End of ROCm SMI Log >>> > ======================== >>> > >>> > So only difference is that temperature shows 44c when memory speed was >>> > fast and 39c when it was slow. But mclk was 1200MHz and sclk was >>> > 400MHz in both cases. >>> > Can it be that rocm-smi just has a bug in reporting and mclk was not >>> > actually 1200MHz when I forced it with rocm-smi --setmclk 2 ? >>> > That would explain the different behaviour.. >>> > >>> > If so then is there a programmatic way how to really guarantee the >>> > high speed mclk? Basically I want do something similar in my program >>> > what happens if I move >>> > the mouse in desktop env and this way guarantee the normal memory >>> > speed each time the program starts. >>> > >>> > -- >>> > Lauri >>> > >>> > >>> > On Tue, Mar 12, 2019 at 11:36 PM Deucher, Alexander >>> > <<a href="mailto:Alexander.Deucher@amd.com" target="_blank" moz-do-not-send="true">Alexander.Deucher@amd.com</a> <mailto:<a href="mailto:Alexander.Deucher@amd.com" target="_blank" moz-do-not-send="true">Alexander.Deucher@amd.com</a>>> wrote: >>> > >>> > Forcing the sclk and mclk high may impact the CPU frequency since >>> > they share TDP. >>> > >>> > Alex >>> > ------------------------------------------------------------------------ >>> > *From:* amd-gfx <<a href="mailto:amd-gfx-bounces@lists.freedesktop.org" target="_blank" moz-do-not-send="true">amd-gfx-bounces@lists.freedesktop.org</a> >>> > <mailto:<a href="mailto:amd-gfx-bounces@lists.freedesktop.org" target="_blank" moz-do-not-send="true">amd-gfx-bounces@lists.freedesktop.org</a>>> on behalf of Lauri >>> > Ehrenpreis <<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a> <mailto:<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a>>> >>> > *Sent:* Tuesday, March 12, 2019 5:31 PM >>> > *To:* Kuehling, Felix >>> > *Cc:* Tom St Denis; <a href="mailto:amd-gfx@lists.freedesktop.org" target="_blank" moz-do-not-send="true"> amd-gfx@lists.freedesktop.org</a> >>> > <mailto:<a href="mailto:amd-gfx@lists.freedesktop.org" target="_blank" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a>> >>> > *Subject:* Re: Slow memory access when using OpenCL without X11 >>> > However it's not only related to mclk and sclk. I tried this: >>> > rocm-smi --setsclk 2 >>> > rocm-smi --setmclk 3 >>> > rocm-smi >>> > ======================== ROCm System Management Interface >>> > ======================== >>> > ================================================================================================ >>> > GPU Temp AvgPwr SCLK MCLK PCLK Fan Perf >>> > PwrCap SCLK OD MCLK OD GPU% >>> > GPU[0] : WARNING: Empty SysFS value: pclk >>> > GPU[0] : WARNING: Unable to read >>> > /sys/class/drm/card0/device/gpu_busy_percent >>> > 0 34.0c N/A 1240Mhz 1333Mhz N/A 0% >>> > manual N/A 0% 0% N/A >>> > ================================================================================================ >>> > ======================== End of ROCm SMI Log >>> > ======================== >>> > >>> > ./cl_slow_test 1 >>> > got 1 platforms 1 devices >>> > speed 3919.777100 avg 3919.777100 mbytes/s >>> > speed 3809.373291 avg 3864.575195 mbytes/s >>> > speed 585.796814 avg 2771.649170 mbytes/s >>> > speed 188.721848 avg 2125.917236 mbytes/s >>> > speed 188.916367 avg 1738.517090 mbytes/s >>> > >>> > So despite forcing max sclk and mclk the memory speed is still slow.. >>> > >>> > -- >>> > Lauri >>> > >>> > >>> > On Tue, Mar 12, 2019 at 11:21 PM Lauri Ehrenpreis >>> > <<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a> <mailto:<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a>>> wrote: >>> > >>> > IN the case when memory is slow, the rocm-smi outputs this: >>> > ======================== ROCm System Management >>> > Interface ======================== >>> > ================================================================================================ >>> > GPU Temp AvgPwr SCLK MCLK PCLK Fan >>> > Perf PwrCap SCLK OD MCLK OD GPU% >>> > GPU[0] : WARNING: Empty SysFS value: pclk >>> > GPU[0] : WARNING: Unable to read >>> > /sys/class/drm/card0/device/gpu_busy_percent >>> > 0 30.0c N/A 400Mhz 933Mhz N/A 0% >>> > auto N/A 0% 0% N/A >>> > ================================================================================================ >>> > ======================== End of ROCm SMI Log >>> > ======================== >>> > >>> > normal memory speed case gives following: >>> > ======================== ROCm System Management >>> > Interface ======================== >>> > ================================================================================================ >>> > GPU Temp AvgPwr SCLK MCLK PCLK Fan >>> > Perf PwrCap SCLK OD MCLK OD GPU% >>> > GPU[0] : WARNING: Empty SysFS value: pclk >>> > GPU[0] : WARNING: Unable to read >>> > /sys/class/drm/card0/device/gpu_busy_percent >>> > 0 35.0c N/A 400Mhz 1200Mhz N/A 0% >>> > auto N/A 0% 0% N/A >>> > ================================================================================================ >>> > ======================== End of ROCm SMI Log >>> > ======================== >>> > >>> > So there is a difference in MCLK - can this cause such a huge >>> > slowdown? >>> > >>> > -- >>> > Lauri >>> > >>> > On Tue, Mar 12, 2019 at 6:39 PM Kuehling, Felix >>> > <<a href="mailto:Felix.Kuehling@amd.com" target="_blank" moz-do-not-send="true">Felix.Kuehling@amd.com</a> <mailto:<a href="mailto:Felix.Kuehling@amd.com" target="_blank" moz-do-not-send="true">Felix.Kuehling@amd.com</a>>> wrote: >>> > >>> > [adding the list back] >>> > >>> > I'd suspect a problem related to memory clock. This is an >>> > APU where >>> > system memory is shared with the CPU, so if the SMU >>> > changes memory >>> > clocks that would affect CPU memory access performance. If >>> > the problem >>> > only occurs when OpenCL is running, then the compute power >>> > profile could >>> > have an effect here. >>> > >>> > Laurie, can you monitor the clocks during your tests using >>> > rocm-smi? >>> > >>> > Regards, >>> > Felix >>> > >>> > On 2019-03-11 1:15 p.m., Tom St Denis wrote: >>> > > Hi Lauri, >>> > > >>> > > I don't have ROCm installed locally (not on that team at >>> > AMD) but I >>> > > can rope in some of the KFD folk and see what they say :-). >>> > > >>> > > (in the mean time I should look into installing the ROCm >>> > stack on my >>> > > Ubuntu disk for experimentation...). >>> > > >>> > > Only other thing that comes to mind is some sort of >>> > stutter due to >>> > > power/clock gating (or gfx off/etc). But that typically >>> > affects the >>> > > display/gpu side not the CPU side. >>> > > >>> > > Felix: Any known issues with Raven and ROCm interacting >>> > over memory >>> > > bus performance? >>> > > >>> > > Tom >>> > > >>> > > On Mon, Mar 11, 2019 at 12:56 PM Lauri Ehrenpreis >>> > <<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a> <mailto:<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a>> >>> > > <mailto:<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a> <mailto:<a href="mailto:laurioma@gmail.com" target="_blank" moz-do-not-send="true">laurioma@gmail.com</a>>>> >>> > wrote: >>> > > >>> > > Hi! >>> > > >>> > > The 100x memory slowdown is hard to belive indeed. I >>> > attached the >>> > > test program with my first e-mail which depends only on >>> > > rocm-opencl-dev package. Would you mind compiling it >>> > and checking >>> > > if it slows down memory for you as well? >>> > > >>> > > steps: >>> > > 1) g++ cl_slow_test.cpp -o cl_slow_test -I >>> > > /opt/rocm/opencl/include/ -L >>> > /opt/rocm/opencl/lib/x86_64/ -lOpenCL >>> > > 2) logout from desktop env and disconnect >>> > hdmi/diplayport etc >>> > > 3) log in over ssh >>> > > 4) run the program ./cl_slow_test 1 >>> > > >>> > > For me it reproduced even without step 2 as well but >>> > less >>> > > reliably. moving mouse for example could make the >>> > memory speed >>> > > fast again. >>> > > >>> > > -- >>> > > Lauri >>> > > >>> > > >>> > > >>> > > On Mon, Mar 11, 2019 at 6:33 PM Tom St Denis >>> > <<a href="mailto:tstdenis82@gmail.com" target="_blank" moz-do-not-send="true">tstdenis82@gmail.com</a> <mailto:<a href="mailto:tstdenis82@gmail.com" target="_blank" moz-do-not-send="true">tstdenis82@gmail.com</a>> >>> > > <mailto:<a href="mailto:tstdenis82@gmail.com" target="_blank" moz-do-not-send="true">tstdenis82@gmail.com</a> >>> > <mailto:<a href="mailto:tstdenis82@gmail.com" target="_blank" moz-do-not-send="true">tstdenis82@gmail.com</a>>>> wrote: >>> > > >>> > > Hi Lauri, >>> > > >>> > > There's really no connection between the two >>> > other than they >>> > > run in the same package. I too run a 2400G (as my >>> > > workstation) and I got the same ~6.6GB/sec >>> > transfer rate but >>> > > without a CL app running ... The only logical >>> > reason is your >>> > > CL app is bottlenecking the APUs memory bus but >>> > you claim >>> > > "simply opening a context is enough" so >>> > something else is >>> > > going on. >>> > > >>> > > Your last reply though says "with it running in the >>> > > background" so it's entirely possible the CPU >>> > isn't busy but >>> > > the package memory controller (shared between >>> > both the CPU and >>> > > GPU) is busy. For instance running xonotic in a >>> > 1080p window >>> > > on my 4K display reduced the memory test to >>> > 5.8GB/sec and >>> > > that's hardly a heavy memory bound GPU app. >>> > > >>> > > The only other possible connection is the GPU is >>> > generating so >>> > > much heat that it's throttling the package which >>> > is also >>> > > unlikely if you have a proper HSF attached (I >>> > use the ones >>> > > that came in the retail boxes). >>> > > >>> > > Cheers, >>> > > Tom >>> > > >>> > > > _______________________________________________ > amd-gfx mailing list > <a href="mailto:amd-gfx@lists.freedesktop.org" target="_blank" moz-do-not-send="true"> amd-gfx@lists.freedesktop.org</a> > <a href="https://lists.freedesktop.org/mailman/listinfo/amd-gfx" rel="noreferrer" target="_blank" moz-do-not-send="true"> https://lists.freedesktop.org/mailman/listinfo/amd-gfx</a> </blockquote> </div> </blockquote> </body> </html>