[PATCH v5 01/13] mm: add zone device coherent type memory support
Sierra Guiza, Alejandro (Alex)
alex.sierra at amd.com
Wed Jun 22 23:16:20 UTC 2022
On 6/21/2022 11:16 AM, David Hildenbrand wrote:
> On 21.06.22 18:08, Sierra Guiza, Alejandro (Alex) wrote:
>> On 6/21/2022 7:25 AM, David Hildenbrand wrote:
>>> On 21.06.22 13:55, Alistair Popple wrote:
>>>> David Hildenbrand<david at redhat.com> writes:
>>>>
>>>>> On 21.06.22 13:25, Felix Kuehling wrote:
>>>>>> Am 6/17/22 um 23:19 schrieb David Hildenbrand:
>>>>>>> On 17.06.22 21:27, Sierra Guiza, Alejandro (Alex) wrote:
>>>>>>>> On 6/17/2022 12:33 PM, David Hildenbrand wrote:
>>>>>>>>> On 17.06.22 19:20, Sierra Guiza, Alejandro (Alex) wrote:
>>>>>>>>>> On 6/17/2022 4:40 AM, David Hildenbrand wrote:
>>>>>>>>>>> On 31.05.22 22:00, Alex Sierra wrote:
>>>>>>>>>>>> Device memory that is cache coherent from device and CPU point of view.
>>>>>>>>>>>> This is used on platforms that have an advanced system bus (like CAPI
>>>>>>>>>>>> or CXL). Any page of a process can be migrated to such memory. However,
>>>>>>>>>>>> no one should be allowed to pin such memory so that it can always be
>>>>>>>>>>>> evicted.
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Alex Sierra<alex.sierra at amd.com>
>>>>>>>>>>>> Acked-by: Felix Kuehling<Felix.Kuehling at amd.com>
>>>>>>>>>>>> Reviewed-by: Alistair Popple<apopple at nvidia.com>
>>>>>>>>>>>> [hch: rebased ontop of the refcount changes,
>>>>>>>>>>>> removed is_dev_private_or_coherent_page]
>>>>>>>>>>>> Signed-off-by: Christoph Hellwig<hch at lst.de>
>>>>>>>>>>>> ---
>>>>>>>>>>>> include/linux/memremap.h | 19 +++++++++++++++++++
>>>>>>>>>>>> mm/memcontrol.c | 7 ++++---
>>>>>>>>>>>> mm/memory-failure.c | 8 ++++++--
>>>>>>>>>>>> mm/memremap.c | 10 ++++++++++
>>>>>>>>>>>> mm/migrate_device.c | 16 +++++++---------
>>>>>>>>>>>> mm/rmap.c | 5 +++--
>>>>>>>>>>>> 6 files changed, 49 insertions(+), 16 deletions(-)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/include/linux/memremap.h b/include/linux/memremap.h
>>>>>>>>>>>> index 8af304f6b504..9f752ebed613 100644
>>>>>>>>>>>> --- a/include/linux/memremap.h
>>>>>>>>>>>> +++ b/include/linux/memremap.h
>>>>>>>>>>>> @@ -41,6 +41,13 @@ struct vmem_altmap {
>>>>>>>>>>>> * A more complete discussion of unaddressable memory may be found in
>>>>>>>>>>>> * include/linux/hmm.h and Documentation/vm/hmm.rst.
>>>>>>>>>>>> *
>>>>>>>>>>>> + * MEMORY_DEVICE_COHERENT:
>>>>>>>>>>>> + * Device memory that is cache coherent from device and CPU point of view. This
>>>>>>>>>>>> + * is used on platforms that have an advanced system bus (like CAPI or CXL). A
>>>>>>>>>>>> + * driver can hotplug the device memory using ZONE_DEVICE and with that memory
>>>>>>>>>>>> + * type. Any page of a process can be migrated to such memory. However no one
>>>>>>>>>>> Any page might not be right, I'm pretty sure. ... just thinking about special pages
>>>>>>>>>>> like vdso, shared zeropage, ... pinned pages ...
>>>>>>>>> Well, you cannot migrate long term pages, that's what I meant :)
>>>>>>>>>
>>>>>>>>>>>> + * should be allowed to pin such memory so that it can always be evicted.
>>>>>>>>>>>> + *
>>>>>>>>>>>> * MEMORY_DEVICE_FS_DAX:
>>>>>>>>>>>> * Host memory that has similar access semantics as System RAM i.e. DMA
>>>>>>>>>>>> * coherent and supports page pinning. In support of coordinating page
>>>>>>>>>>>> @@ -61,6 +68,7 @@ struct vmem_altmap {
>>>>>>>>>>>> enum memory_type {
>>>>>>>>>>>> /* 0 is reserved to catch uninitialized type fields */
>>>>>>>>>>>> MEMORY_DEVICE_PRIVATE = 1,
>>>>>>>>>>>> + MEMORY_DEVICE_COHERENT,
>>>>>>>>>>>> MEMORY_DEVICE_FS_DAX,
>>>>>>>>>>>> MEMORY_DEVICE_GENERIC,
>>>>>>>>>>>> MEMORY_DEVICE_PCI_P2PDMA,
>>>>>>>>>>>> @@ -143,6 +151,17 @@ static inline bool folio_is_device_private(const struct folio *folio)
>>>>>>>>>>> In general, this LGTM, and it should be correct with PageAnonExclusive I think.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> However, where exactly is pinning forbidden?
>>>>>>>>>> Long-term pinning is forbidden since it would interfere with the device
>>>>>>>>>> memory manager owning the
>>>>>>>>>> device-coherent pages (e.g. evictions in TTM). However, normal pinning
>>>>>>>>>> is allowed on this device type.
>>>>>>>>> I don't see updates to folio_is_pinnable() in this patch.
>>>>>>>> Device coherent type pages should return true here, as they are pinnable
>>>>>>>> pages.
>>>>>>> That function is only called for long-term pinnings in try_grab_folio().
>>>>>>>
>>>>>>>>> So wouldn't try_grab_folio() simply pin these pages? What am I missing?
>>>>>>>> As far as I understand this return NULL for long term pin pages.
>>>>>>>> Otherwise they get refcount incremented.
>>>>>>> I don't follow.
>>>>>>>
>>>>>>> You're saying
>>>>>>>
>>>>>>> a) folio_is_pinnable() returns true for device coherent pages
>>>>>>>
>>>>>>> and that
>>>>>>>
>>>>>>> b) device coherent pages don't get long-term pinned
>>>>>>>
>>>>>>>
>>>>>>> Yet, the code says
>>>>>>>
>>>>>>> struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags)
>>>>>>> {
>>>>>>> if (flags & FOLL_GET)
>>>>>>> return try_get_folio(page, refs);
>>>>>>> else if (flags & FOLL_PIN) {
>>>>>>> struct folio *folio;
>>>>>>>
>>>>>>> /*
>>>>>>> * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a
>>>>>>> * right zone, so fail and let the caller fall back to the slow
>>>>>>> * path.
>>>>>>> */
>>>>>>> if (unlikely((flags & FOLL_LONGTERM) &&
>>>>>>> !is_pinnable_page(page)))
>>>>>>> return NULL;
>>>>>>> ...
>>>>>>> return folio;
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> What prevents these pages from getting long-term pinned as stated in this patch?
>>>>>> Long-term pinning is handled by __gup_longterm_locked, which migrates
>>>>>> pages returned by __get_user_pages_locked that cannot be long-term
>>>>>> pinned. try_grab_folio is OK to grab the pages. Anything that can't be
>>>>>> long-term pinned will be migrated afterwards, and
>>>>>> __get_user_pages_locked will be retried. The migration of
>>>>>> DEVICE_COHERENT pages was implemented by Alistair in patch 5/13
>>>>>> ("mm/gup: migrate device coherent pages when pinning instead of failing").
>>>>> Thanks.
>>>>>
>>>>> __gup_longterm_locked()->check_and_migrate_movable_pages()
>>>>>
>>>>> Which checks folio_is_pinnable() and doesn't do anything if set.
>>>>>
>>>>> Sorry to be dense here, but I don't see how what's stated in this patch
>>>>> works without adjusting folio_is_pinnable().
>>>> Ugh, I think you might be right about try_grab_folio().
>>>>
>>>> We didn't update folio_is_pinnable() to include device coherent pages
>>>> because device coherent pages are pinnable. It is really just
>>>> FOLL_LONGTERM that we want to prevent here.
>>>>
>>>> For normal PUP that is done by my change in
>>>> check_and_migrate_movable_pages() which migrates pages being pinned with
>>>> FOLL_LONGTERM. But I think I incorrectly assumed we would take the
>>>> pte_devmap() path in gup_pte_range(), which we don't for coherent pages.
>>>> So I think the check in try_grab_folio() needs to be:
>>> I think I said it already (and I might be wrong without reading the
>>> code), but folio_is_pinnable() is *only* called for long-term pinnings.
>>>
>>> It should actually be called folio_is_longterm_pinnable().
>>>
>>> That's where that check should go, no?
>> David, I think you're right. We didn't catch this since the LONGTERM gup
>> test we added to hmm-test only calls to pin_user_pages. Apparently
>> try_grab_folio is called only from fast callers (ex.
>> pin_user_pages_fast/get_user_pages_fast). I have added a conditional
>> similar to what Alistair has proposed to return null on LONGTERM &&
>> (coherent_pages || folio_is_pinnable) at try_grab_folio. Also a new gup
>> test was added with LONGTERM set that calls pin_user_pages_fast.
>> Returning null under this condition it does causes the migration from
>> dev to system memory.
>>
> Why can't coherent memory simply put its checks into
> folio_is_pinnable()? I don't get it why we have to do things differently
> here.
>
>> Actually, Im having different problems with a call to PageAnonExclusive
>> from try_to_migrate_one during page fault from a HMM test that first
>> migrate pages to device private and forks to mark as COW these pages.
>> Apparently is catching the first BUG VM_BUG_ON_PGFLAGS(!PageAnon(page),
>> page)
> With or without this series? A backtrace would be great.
Here's the back trace. This happens in a hmm-test added in this patch
series. However, I have tried to isolate this BUG by just adding the COW
test with private device memory only. This is only present as follows.
Allocate anonymous mem->Migrate to private device memory->fork->try to
access to parent's anonymous memory (which will suppose to trigger a
page fault and migration to system mem). Just for the record, if the
child is terminated before the parent's memory is accessed, this problem
is not present.
patch name for this test: tools: add selftests to hmm for COW in device
memory
[ 528.727237] BUG: unable to handle page fault for address:
ffffea1fffffffc0
[ 528.739585] #PF: supervisor read access in kernel mode
[ 528.745324] #PF: error_code(0x0000) - not-present page
[ 528.751062] PGD 44eaf2067 P4D 44eaf2067 PUD 0
[ 528.756026] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 528.760890] CPU: 120 PID: 18275 Comm: hmm-tests Not tainted
5.19.0-rc3-kfd-alex #257
[ 528.769542] Hardware name: AMD Corporation BardPeak/BardPeak, BIOS
RTY1002BDS 09/17/2021
[ 528.778579] RIP: 0010:try_to_migrate_one+0x21a/0x1000
[ 528.784225] Code: f6 48 89 c8 48 2b 05 45 d1 6a 01 48 c1 f8 06 48 29
c3 48 8b 45 a8 48 c1 e3 06 48 01 cb f6 41 18 01 48 89 85 50 ff ff ff 74
0b <4c> 8b 33 49 c1 ee 11 41 83 e6 01 48 8b bd 48 ff ff ff e8 3f 99 02
[ 528.805194] RSP: 0000:ffffc90003cdfaa0 EFLAGS: 00010202
[ 528.811027] RAX: 00007ffff7ff4000 RBX: ffffea1fffffffc0 RCX:
ffffeaffffffffc0
[ 528.818995] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffffc90003cdfaf8
[ 528.826962] RBP: ffffc90003cdfb70 R08: 0000000000000000 R09:
0000000000000000
[ 528.834930] R10: ffffc90003cdf910 R11: 0000000000000002 R12:
ffff888194450540
[ 528.842899] R13: ffff888160d057c0 R14: 0000000000000000 R15:
03ffffffffffffff
[ 528.850865] FS: 00007ffff7fdb740(0000) GS:ffff8883b0600000(0000)
knlGS:0000000000000000
[ 528.859891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 528.866308] CR2: ffffea1fffffffc0 CR3: 00000001562b4003 CR4:
0000000000770ee0
[ 528.874275] PKRU: 55555554
[ 528.877286] Call Trace:
[ 528.880016] <TASK>
[ 528.882356] ? lock_is_held_type+0xdf/0x130
[ 528.887033] rmap_walk_anon+0x167/0x410
[ 528.891316] try_to_migrate+0x90/0xd0
[ 528.895405] ? try_to_unmap_one+0xe10/0xe10
[ 528.900074] ? anon_vma_ctor+0x50/0x50
[ 528.904260] ? put_anon_vma+0x10/0x10
[ 528.908347] ? invalid_mkclean_vma+0x20/0x20
[ 528.913114] migrate_vma_setup+0x5f4/0x750
[ 528.917691] dmirror_devmem_fault+0x8c/0x250 [test_hmm]
[ 528.923532] do_swap_page+0xac0/0xe50
[ 528.927623] ? __lock_acquire+0x4b2/0x1ac0
[ 528.932199] __handle_mm_fault+0x949/0x1440
[ 528.936876] handle_mm_fault+0x13f/0x3e0
[ 528.941256] do_user_addr_fault+0x215/0x740
[ 528.945928] exc_page_fault+0x75/0x280
[ 528.950115] asm_exc_page_fault+0x27/0x30
[ 528.954593] RIP: 0033:0x40366b
[ 528.958001] Code: 00 48 89 85 d8 fe ff ff eb 2a 48 8b 85 d0 fe ff ff
48 8d 14 85 00 00 00 00 48 8b 85 d8 fe ff ff 48 01 d0 48 8b 95 d0 fe ff
ff <89> 10 48 83 85 d0 fe ff ff 01 48 8b 85 40 ff ff ff 48 c1 e8 02 48
[ 528.978973] RSP: 002b:00007fffffffe280 EFLAGS: 00010206
[ 528.984806] RAX: 00007ffff7ff4000 RBX: 0000000000000000 RCX:
0000000000000000
[ 528.992774] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
00007ffff77ee968
[ 529.000742] RBP: 00007fffffffe430 R08: 00007ffff7fdb740 R09:
0000000000000000
[ 529.008709] R10: 00007ffff7fdba10 R11: 0000000000000246 R12:
0000000000400e30
[ 529.016675] R13: 00007fffffffe630 R14: 0000000000000000 R15:
0000000000000000
[ 529.024638] </TASK>
[ 529.027074] Modules linked in: test_hmm xt_conntrack xt_MASQUERADE
nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 br_netfilter ip6table_filter ip6_tables iptable_filter
k10temp ip_tables x_tables i2c_piix4 [last unloaded: test_hmm]
[ 529.053595] CR2: ffffea1fffffffc0
[ 529.057296] ---[ end trace 0000000000000000 ]---
[ 529.197816] RIP: 0010:try_to_migrate_one+0x21a/0x1000
[ 529.197823] Code: f6 48 89 c8 48 2b 05 45 d1 6a 01 48 c1 f8 06 48 29
c3 48 8b 45 a8 48 c1 e3 06 48 01 cb f6 41 18 01 48 89 85 50 ff ff ff 74
0b <4c> 8b 33 49 c1 ee 11 41 83 e6 01 48 8b bd 48 ff ff ff e8 3f 99 02
[ 529.197826] RSP: 0000:ffffc90003cdfaa0 EFLAGS: 00010202
[ 529.197828] RAX: 00007ffff7ff4000 RBX: ffffea1fffffffc0 RCX:
ffffeaffffffffc0
[ 529.197830] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffffc90003cdfaf8
[ 529.197831] RBP: ffffc90003cdfb70 R08: 0000000000000000 R09:
0000000000000000
[ 529.197832] R10: ffffc90003cdf910 R11: 0000000000000002 R12:
ffff888194450540
[ 529.197833] R13: ffff888160d057c0 R14: 0000000000000000 R15:
03ffffffffffffff
[ 529.197835] FS: 00007ffff7fdb740(0000) GS:ffff8883b0600000(0000)
knlGS:0000000000000000
[ 529.197837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 529.197839] CR2: ffffea1fffffffc0 CR3: 00000001562b4003 CR4:
0000000000770ee0
[ 529.197840] PKRU: 55555554
[ 529.197841] note: hmm-tests[18275] exited with preempt_count 1
Regards,
Alex Sierra
>
More information about the amd-gfx
mailing list