<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Xiangliang</p>
<p><br>
</p>
<p>please BUG() when register access occured in RUNTIME and IRQ context, e.g.:</p>
<p><br>
</p>
<p><font size="2"><span style="font-size:10pt">if (amdgpu_sriov_runtime(adev)) {</span></font></p>
<p><font size="2"><span style="font-size:10pt"><br>
</span></font></p>
<p><font size="2"><span style="font-size:10pt">}<br>
return amdgpu_virt_kiq_wreg(adev, reg, v);</span></font><br>
</p>
<p><br>
</p>
<p>with above addressed, Reviewed-by: Monk Liu <monk.liu@amd.com> <br>
</p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>发件人:</b> amd-gfx <amd-gfx-bounces@lists.freedesktop.org> 代表 Liu, Monk <Monk.Liu@amd.com><br>
<b>发送时间:</b> 2017年1月11日 22:54:52<br>
<b>收件人:</b> Christian König; Yu, Xiangliang; amd-gfx@lists.freedesktop.org<br>
<b>主题:</b> RE: [V3 04/11] drm/amdgpu/virt: use kiq to access registers</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">We should BUG and not access register at all if found during RUNTIME && IRQ_context<br>
By far.<br>
<br>
BR Monk<br>
-----Original Message-----<br>
From: Christian König [<a href="mailto:deathsimple@vodafone.de">mailto:deathsimple@vodafone.de</a>]
<br>
Sent: Wednesday, January 11, 2017 10:44 PM<br>
To: Liu, Monk <Monk.Liu@amd.com>; Yu, Xiangliang <Xiangliang.Yu@amd.com>; amd-gfx@lists.freedesktop.org<br>
Subject: Re: [V3 04/11] drm/amdgpu/virt: use kiq to access registers<br>
<br>
> do you think that's okay ?<br>
Fine with me, but I'm not sure if the hypervisor doesn't gets a hickup when a client writes to some registers from interrupt context.<br>
<br>
Christian.<br>
<br>
Am 11.01.2017 um 15:38 schrieb Liu, Monk:<br>
> Yeah, make sense<br>
><br>
> I think before we have a full version of KIQ reg accessing without context switch, we should totally disable KIQ reg read/write in IRQ context, do you think that's okay ?<br>
><br>
> BR Monk<br>
><br>
> -----Original Message-----<br>
> From: Christian König [<a href="mailto:deathsimple@vodafone.de">mailto:deathsimple@vodafone.de</a>]<br>
> Sent: Wednesday, January 11, 2017 10:16 PM<br>
> To: Liu, Monk <Monk.Liu@amd.com>; Yu, Xiangliang <br>
> <Xiangliang.Yu@amd.com>; amd-gfx@lists.freedesktop.org<br>
> Subject: Re: [V3 04/11] drm/amdgpu/virt: use kiq to access registers<br>
><br>
>> Because if we are in interrupt , we are forbid to do schedule, and use kiq to read register will invoke fence_wait() ...<br>
> That won't work. Locking a mutext like it is done in the write path can cause scheduling as well.<br>
><br>
> If we need to push writes through the KIQ in interrupt context we need to change the code to use a spinlock with disabled interrupts instead of a mutex.<br>
><br>
> Otherwise the whole thing can easily deadlock when an interrupt happens to come while the lock is taken.<br>
><br>
> Please also test the patchset with lockdep enabled, that should yield a bunch of error messages when you try to do something like this.<br>
><br>
> Regards,<br>
> Christian.<br>
><br>
> Am 11.01.2017 um 14:51 schrieb Liu, Monk:<br>
>> Because if we are in interrupt , we are forbid to do schedule, and use kiq to read register will invoke fence_wait() ...<br>
>><br>
>> If you think that's odds, we can use BUG_ON() to stop driver running if we need read registers in SRIOV case.<br>
>><br>
>> And to fully support register reading, we need to implement a new <br>
>> method to use KIQ access register without any schedule() chance, but <br>
>> that's overhead for current stage, and we don't observe registers <br>
>> reading in IRQ context now on 3D apps. (kfd observed such case, and <br>
>> @Shaoyun dispatches a kernel thread/tasklet in IRQ to do the register <br>
>> dumping, instead of directly reading them via KIQ )<br>
>> <br>
>> BR Monk<br>
>><br>
>> -----Original Message-----<br>
>> From: Christian König [<a href="mailto:deathsimple@vodafone.de">mailto:deathsimple@vodafone.de</a>]<br>
>> Sent: Wednesday, January 11, 2017 9:38 PM<br>
>> To: Yu, Xiangliang <Xiangliang.Yu@amd.com>; <br>
>> amd-gfx@lists.freedesktop.org<br>
>> Cc: Liu, Monk <Monk.Liu@amd.com><br>
>> Subject: Re: [V3 04/11] drm/amdgpu/virt: use kiq to access registers<br>
>><br>
>> Am 11.01.2017 um 14:18 schrieb Xiangliang Yu:<br>
>>> For virtualization, it is must for driver to use KIQ to access <br>
>>> registers when it is out of GPU full access mode.<br>
>>><br>
>>> Signed-off-by: Xiangliang Yu <Xiangliang.Yu@amd.com><br>
>>> Signed-off-by: Monk Liu <Monk.Liu@amd.com><br>
>>> ---<br>
>>> drivers/gpu/drm/amd/amdgpu/Makefile | 2 +-<br>
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 6 +++<br>
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 82 ++++++++++++++++++++++++++++++<br>
>>> drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h | 5 ++<br>
>>> drivers/gpu/drm/amd/amdgpu/vi.c | 3 ++<br>
>>> 5 files changed, 97 insertions(+), 1 deletion(-)<br>
>>> create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c<br>
>>><br>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/Makefile<br>
>>> b/drivers/gpu/drm/amd/amdgpu/Makefile<br>
>>> index 4185b03..0b8e470 100644<br>
>>> --- a/drivers/gpu/drm/amd/amdgpu/Makefile<br>
>>> +++ b/drivers/gpu/drm/amd/amdgpu/Makefile<br>
>>> @@ -30,7 +30,7 @@ amdgpu-y += amdgpu_device.o amdgpu_kms.o \<br>
>>> atombios_encoders.o amdgpu_sa.o atombios_i2c.o \<br>
>>> amdgpu_prime.o amdgpu_vm.o amdgpu_ib.o amdgpu_pll.o \<br>
>>> amdgpu_ucode.o amdgpu_bo_list.o amdgpu_ctx.o amdgpu_sync.o \<br>
>>> - amdgpu_gtt_mgr.o amdgpu_vram_mgr.o<br>
>>> + amdgpu_gtt_mgr.o amdgpu_vram_mgr.o amdgpu_virt.o<br>
>>> <br>
>>> # add asic specific block<br>
>>> amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o kv_smc.o <br>
>>> kv_dpm.o \ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br>
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br>
>>> index f82919d..9a2fd3e 100644<br>
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br>
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br>
>>> @@ -95,6 +95,9 @@ uint32_t amdgpu_mm_rreg(struct amdgpu_device *adev, uint32_t reg,<br>
>>> {<br>
>>> uint32_t ret;<br>
>>> <br>
>>> + if (amdgpu_sriov_runtime(adev) && !in_interrupt())<br>
>>> + return amdgpu_virt_kiq_rreg(adev, reg);<br>
>>> +<br>
>> Mhm, why is the in_interrupt() necessary here?<br>
>><br>
>>> if ((reg * 4) < adev->rmmio_size && !always_indirect)<br>
>>> ret = readl(((void __iomem *)adev->rmmio) + (reg * 4));<br>
>>> else {<br>
>>> @@ -114,6 +117,9 @@ void amdgpu_mm_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v,<br>
>>> {<br>
>>> trace_amdgpu_mm_wreg(adev->pdev->device, reg, v);<br>
>>> <br>
>>> + if (amdgpu_sriov_runtime(adev))<br>
>>> + return amdgpu_virt_kiq_wreg(adev, reg, v);<br>
>>> +<br>
>> And why it isn't necessary here? What happens when we write a register from an interrupt routine?<br>
>><br>
>> Christian.<br>
>><br>
>>> if ((reg * 4) < adev->rmmio_size && !always_indirect)<br>
>>> writel(v, ((void __iomem *)adev->rmmio) + (reg * 4));<br>
>>> else {<br>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c<br>
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c<br>
>>> new file mode 100644<br>
>>> index 0000000..7861b6b<br>
>>> --- /dev/null<br>
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c<br>
>>> @@ -0,0 +1,82 @@<br>
>>> +/*<br>
>>> + * Copyright 2017 Advanced Micro Devices, Inc.<br>
>>> + *<br>
>>> + * Permission is hereby granted, free of charge, to any person <br>
>>> +obtaining a<br>
>>> + * copy of this software and associated documentation files (the <br>
>>> +"Software"),<br>
>>> + * to deal in the Software without restriction, including without <br>
>>> +limitation<br>
>>> + * the rights to use, copy, modify, merge, publish, distribute, <br>
>>> +sublicense,<br>
>>> + * and/or sell copies of the Software, and to permit persons to <br>
>>> +whom the<br>
>>> + * Software is furnished to do so, subject to the following conditions:<br>
>>> + *<br>
>>> + * The above copyright notice and this permission notice shall be <br>
>>> +included in<br>
>>> + * all copies or substantial portions of the Software.<br>
>>> + *<br>
>>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, <br>
>>> +EXPRESS OR<br>
>>> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF <br>
>>> +MERCHANTABILITY,<br>
>>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO <br>
>>> +EVENT SHALL<br>
>>> + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, <br>
>>> +DAMAGES OR<br>
>>> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR <br>
>>> +OTHERWISE,<br>
>>> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE <br>
>>> +USE OR<br>
>>> + * OTHER DEALINGS IN THE SOFTWARE.<br>
>>> + */<br>
>>> +<br>
>>> +#include "amdgpu.h"<br>
>>> +#include "amdgpu_virt.h"<br>
>>> +<br>
>>> +void amdgpu_virt_init_setting(struct amdgpu_device *adev) {<br>
>>> + mutex_init(&adev->virt.lock);<br>
>>> +}<br>
>>> +<br>
>>> +uint32_t amdgpu_virt_kiq_rreg(struct amdgpu_device *adev, uint32_t<br>
>>> +reg) {<br>
>>> + signed long r;<br>
>>> + uint32_t val;<br>
>>> + struct fence *f;<br>
>>> + struct amdgpu_kiq *kiq = &adev->gfx.kiq;<br>
>>> + struct amdgpu_ring *ring = &kiq->ring;<br>
>>> +<br>
>>> + BUG_ON(!ring->funcs->emit_rreg);<br>
>>> +<br>
>>> + mutex_lock(&adev->virt.lock);<br>
>>> + amdgpu_ring_alloc(ring, 32);<br>
>>> + amdgpu_ring_emit_hdp_flush(ring);<br>
>>> + amdgpu_ring_emit_rreg(ring, reg);<br>
>>> + amdgpu_ring_emit_hdp_invalidate(ring);<br>
>>> + amdgpu_fence_emit(ring, &f);<br>
>>> + amdgpu_ring_commit(ring);<br>
>>> + mutex_unlock(&adev->virt.lock);<br>
>>> +<br>
>>> + r = fence_wait(f, false);<br>
>>> + if (r)<br>
>>> + DRM_ERROR("wait for kiq fence error: %ld.\n", r);<br>
>>> + fence_put(f);<br>
>>> +<br>
>>> + val = adev->wb.wb[adev->virt.reg_val_offs];<br>
>>> +<br>
>>> + return val;<br>
>>> +}<br>
>>> +<br>
>>> +void amdgpu_virt_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, <br>
>>> +uint32_t v) {<br>
>>> + signed long r;<br>
>>> + struct fence *f;<br>
>>> + struct amdgpu_kiq *kiq = &adev->gfx.kiq;<br>
>>> + struct amdgpu_ring *ring = &kiq->ring;<br>
>>> +<br>
>>> + BUG_ON(!ring->funcs->emit_wreg);<br>
>>> +<br>
>>> + mutex_lock(&adev->virt.lock);<br>
>>> + amdgpu_ring_alloc(ring, 32);<br>
>>> + amdgpu_ring_emit_hdp_flush(ring);<br>
>>> + amdgpu_ring_emit_wreg(ring, reg, v);<br>
>>> + amdgpu_ring_emit_hdp_invalidate(ring);<br>
>>> + amdgpu_fence_emit(ring, &f);<br>
>>> + amdgpu_ring_commit(ring);<br>
>>> + mutex_unlock(&adev->virt.lock);<br>
>>> +<br>
>>> + r = fence_wait(f, false);<br>
>>> + if (r)<br>
>>> + DRM_ERROR("wait for kiq fence error: %ld.\n", r);<br>
>>> + fence_put(f);<br>
>>> +}<br>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h<br>
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h<br>
>>> index 3d38a9f..2869980 100644<br>
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h<br>
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h<br>
>>> @@ -33,6 +33,7 @@<br>
>>> struct amdgpu_virt {<br>
>>> uint32_t caps;<br>
>>> uint32_t reg_val_offs;<br>
>>> + struct mutex lock;<br>
>>> };<br>
>>> <br>
>>> #define amdgpu_sriov_enabled(adev) \ @@ -59,4 +60,8 @@ static <br>
>>> inline bool is_virtual_machine(void)<br>
>>> #endif<br>
>>> }<br>
>>> <br>
>>> +void amdgpu_virt_init_setting(struct amdgpu_device *adev); uint32_t <br>
>>> +amdgpu_virt_kiq_rreg(struct amdgpu_device *adev, uint32_t reg); <br>
>>> +void amdgpu_virt_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, <br>
>>> +uint32_t v);<br>
>>> +<br>
>>> #endif<br>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c <br>
>>> b/drivers/gpu/drm/amd/amdgpu/vi.c index 7350a8f..dc0d4fa 100644<br>
>>> --- a/drivers/gpu/drm/amd/amdgpu/vi.c<br>
>>> +++ b/drivers/gpu/drm/amd/amdgpu/vi.c<br>
>>> @@ -892,6 +892,9 @@ static int vi_common_early_init(void *handle)<br>
>>> (amdgpu_ip_block_mask & (1 << AMD_IP_BLOCK_TYPE_SMC)))<br>
>>> smc_enabled = true;<br>
>>> <br>
>>> + if (amdgpu_sriov_vf(adev))<br>
>>> + amdgpu_virt_init_setting(adev);<br>
>>> +<br>
>>> adev->rev_id = vi_get_rev_id(adev);<br>
>>> adev->external_rev_id = 0xFF;<br>
>>> switch (adev->asic_type) {<br>
>> _______________________________________________<br>
>> amd-gfx mailing list<br>
>> amd-gfx@lists.freedesktop.org<br>
>> <a href="https://lists.freedesktop.org/mailman/listinfo/amd-gfx">https://lists.freedesktop.org/mailman/listinfo/amd-gfx</a><br>
><br>
> _______________________________________________<br>
> amd-gfx mailing list<br>
> amd-gfx@lists.freedesktop.org<br>
> <a href="https://lists.freedesktop.org/mailman/listinfo/amd-gfx">https://lists.freedesktop.org/mailman/listinfo/amd-gfx</a><br>
<br>
<br>
_______________________________________________<br>
amd-gfx mailing list<br>
amd-gfx@lists.freedesktop.org<br>
<a href="https://lists.freedesktop.org/mailman/listinfo/amd-gfx">https://lists.freedesktop.org/mailman/listinfo/amd-gfx</a><br>
</div>
</span></font>
</body>
</html>