[PATCH 1/1] [RFC] drm/ttm: Don't init dma32_zone on 64-bit systems

[Kuehling, Felix]

On 2019-02-22 8:45 a.m., Thomas Hellstrom wrote:
> On Fri, 2019-02-22 at 07:10 +0000, Koenig, Christian wrote:
>> Am 21.02.19 um 22:02 schrieb Thomas Hellstrom:
>>> Hi,
>>>
>>> On Thu, 2019-02-21 at 20:24 +0000, Kuehling, Felix wrote:
>>>> On 2019-02-21 12:34 p.m., Thomas Hellstrom wrote:
>>>>> On Thu, 2019-02-21 at 16:57 +0000, Kuehling, Felix wrote:
>>>>>> On 2019-02-21 2:59 a.m., Koenig, Christian wrote:
>>>>>>> On x86 with HIGHMEM there is no dma32 zone. Why do we need
>>>>>>> one
>>>>>>> on
>>>>>>>>> x86_64? Can we make x86_64 more like HIGHMEM instead?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>         Felix
>>>>>>>>>
>>>>>>>> IIRC with x86, the kernel zone is always smaller than any
>>>>>>>> dma32
>>>>>>>> zone,
>>>>>>>> so we'd always exhaust the kernel zone before dma32
>>>>>>>> anyway.
>>>>>>>>
>>>>>>>> Not sure why we have dma32 on x86 without highmem,
>>>>>>>> though.
>>>>>>>> sounds
>>>>>>>> superflous but harmless.
>>>>>>> Well DMA32 denotes memory which is accessible by devices
>>>>>>> who
>>>>>>> can
>>>>>>> only do
>>>>>>> 32bit addressing. And IIRC we can actually do DMA32 to
>>>>>>> highmem
>>>>>>> since
>>>>>>> something like 2.4.*.
>>>>>>>
>>>>>>> Because of this it is actually irrelevant if you have
>>>>>>> highmem
>>>>>>> or
>>>>>>> not,
>>>>>>> what matters for DMA32 is if you have an IOMMU or not.
>>>>>> Are you saying we should have a dma32_zone even on x86 with
>>>>>> HIGHMEM?
>>>>>>
>>>>>>
>>>>>>> So even on x86_64 you actually do need the DMA32 zone if
>>>>>>> you
>>>>>>> don't
>>>>>>> have
>>>>>>> an IOMMU which remaps all memory for devices which can't
>>>>>>> directly
>>>>>>> address it.
>>>>>> Why is DMA32 special in this way? For example AMD GFX8 GPUs
>>>>>> support
>>>>>> 40-bit DMA. But we don't have a special zone for that.
>>>>> If you're running on a non-IOMMU system with physical memory
>>>>> addresses
>>>>>> 40 bits, and tell the DMA subsystem that you need to restrict
>>>>>> to
>>>>>> 40
>>>>> bits, it will probably start using bounce buffers for streaming
>>>>> DMA
>>>>> (which won't work with most graphics drivers), or for
>>>>> dma_alloc_coherent(), it will probably use memory from the
>>>>> DMA32
>>>>> zone.
>>>> OK, then why is it not needed when CONFIG_HIGHMEM is defined?
>>>>
>>>> I found that there is a CONFIG_ZONE_DMA32 parameter. Maybe we
>>>> should
>>>> use
>>>> that to decide whether to account for the DMA32 zone in TTM. It
>>>> is
>>>> set
>>>> on x86_64 and a number of other 64-bit architectures.
>>>>
>>>>
>>>>>> How common is it to have devices that need DMA32 on an x86_64
>>>>>> system?
>>>>> IIRC All devices using dma_alloc_coherent() allocate DMA32
>>>>> memory
>>>>> unless they explicitly set the dma coherent mask to something
>>>>> larger.
>>>>> Like Christian says, if an IOMMU is present and enabled, the
>>>>> need
>>>>> for
>>>>> the DMA32 zone goes away. In theory at least.
>>>> Thanks. I read up a bit on DMA32 and memory zones in general. I
>>>> found
>>>> that there is a lowmem_reserve_ratio feature that protects
>>>> against
>>>> normal page allocations overflowing into lowmem zones. There is
>>>> some
>>>> documentation in Documentation/scsctl/vm.txt (search for
>>>> lowmem_reserve_ratio). The protected amount of memory in each
>>>> zone
>>>> can
>>>> be seen in /proc/zoneinfo.
>>>>
>>>> With that, can we conclude that we don't need to count
>>>> ttm_mem_global_alloc against the dma32 zone.
>>> Yes, it indeed looks like that.
>>> But then I would suggest removing the DMA32 zone entirely.
>> We still need it for the pages we allocate, but we should just stop
>> accounting all the housekeeping to it.
> Why is that? Can't we just account all pages in the kernel zone, and
> leave it up to the kernel to make sure there are still DMA32 pages
> left?

ttm_page_alloc and ttm_page_alloc_dma support allocating from DMA32 
explicitly (setting GFP_DMA32). Such allocations could exhaust DMA32 
memory, which TTM should prevent by limiting its DMA32 usage. This would 
still be counted against the dma32 zone by ttm_mem_global_alloc_page.

I'll send out a new patch that counts general kernel allocations against 
the kernel zone only. I hope this would be acceptable.

Regards,
   Felix


>
> /Thomas
>