[PATCH v2] drm/xe: Support for mmap-ing mmio regions
Levi, Ilia
ilia.levi at intel.com
Mon Jun 23 10:45:19 UTC 2025
On 10/06/2025 17:21, Matthew Brost wrote:
> On Tue, Jun 10, 2025 at 04:57:51PM +0300, Levi, Ilia wrote:
>> On 09/06/2025 18:46, Matthew Brost wrote:
>>> On Mon, Jun 09, 2025 at 12:59:38PM +0300, Ilia Levi wrote:
>>>> Allow the driver to expose hardware register spaces to userspace
>>>> through GEM objects with fake mmap offsets. This can be useful
>>>> for userspace-firmware communication, debugging, etc.
>>>>
>>>> v2: Minor doc fix (CI)
>>>>
>>>> Signed-off-by: Ilia Levi <ilia.levi at intel.com>
>>>> ---
>>>> drivers/gpu/drm/xe/xe_device_types.h | 14 +++
>>>> drivers/gpu/drm/xe/xe_mmio.c | 142 +++++++++++++++++++++++++++
>>>> drivers/gpu/drm/xe/xe_mmio.h | 4 +
>>>> 3 files changed, 160 insertions(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/xe/xe_device_types.h b/drivers/gpu/drm/xe/xe_device_types.h
>>>> index ac27389ccb8b..78542de0d48d 100644
>>>> --- a/drivers/gpu/drm/xe/xe_device_types.h
>>>> +++ b/drivers/gpu/drm/xe/xe_device_types.h
>>>> @@ -10,6 +10,7 @@
>>>>
>>>> #include <drm/drm_device.h>
>>>> #include <drm/drm_file.h>
>>>> +#include <drm/drm_gem.h>
>>>> #include <drm/drm_pagemap.h>
>>>> #include <drm/ttm/ttm_device.h>
>>>>
>>>> @@ -161,6 +162,19 @@ struct xe_mmio {
>>>> u32 adj_offset;
>>>> };
>>>>
>>>> +/**
>>>> + * struct xe_mmio_gem - GEM wrapper for xe_mmio
>>>> + *
>>>> + * A GEM object for exposing xe_mmio instance to userspace via mmap.
>>>> + */
>>>> +struct xe_mmio_gem {
>>>> + /** @base: GEM object base */
>>>> + struct drm_gem_object base;
>>>> +
>>>> + /** @mmio: The MMIO region to expose */
>>>> + struct xe_mmio mmio;
>>> Any reason not this is not a pointer to xe_mmio?
>>
>> Yes, I'd like to keep xe_mmio a lightweight object that could even be defined on the stack.
>> Here's an example of how this could be used:
>>
>> int foo_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
>> {
>> struct xe_mmio mmio = {0};
>> struct xe_mmio_gem *gem;
>>
>> xe_mmio_init(&mmio, xe_device_get_root_tile(xe), xe->mmio.regs +
>> SOME_HW_BLOCK_PCI_OFFSET, SOME_HW_BLOCK_SIZE);
>>
>> gem = xe_mmio_gem_create(&mmio, file);
>> / * store the gem somewhere, return drm_vma_node_offset_addr(&gem->base.vma_node) to the user */
>> }
>>
>> - Ilia
>>
>>
>>>> +};
>>>> +
>>>> /**
>>>> * struct xe_tile - hardware tile structure
>>>> *
>>>> diff --git a/drivers/gpu/drm/xe/xe_mmio.c b/drivers/gpu/drm/xe/xe_mmio.c
>>>> index 7357458bc0d2..6bfa915a9602 100644
>>>> --- a/drivers/gpu/drm/xe/xe_mmio.c
>>>> +++ b/drivers/gpu/drm/xe/xe_mmio.c
>>>> @@ -408,3 +408,145 @@ int xe_mmio_wait32_not(struct xe_mmio *mmio, struct xe_reg reg, u32 mask, u32 va
>>>> {
>>>> return __xe_mmio_wait32(mmio, reg, mask, val, timeout_us, out_val, atomic, false);
>>>> }
>>>> +
>>>> +/**
>>>> + * DOC: Exposing MMIO regions to userspace
>>>> + *
>>>> + * In certain cases, the driver may allow userspace to mmap a portion of the hardware registers.
>>>> + *
>>>> + * This can be done as follows:
>>>> + * 1. Define an xe_mmio instance that represents this portion.
>>>> + * 2. Call xe_mmio_gem_create() to create a GEM object with an mmap-able fake offset.
>>>> + * 3. Use drm_vma_node_offset_addr() on the created GEM object to retrieve the fake offset.
>>>> + * 4. Provide the fake offset to userspace.
>>>> + * 5. Userspace can call mmap with the fake offset. The length provided to mmap
>>>> + * must match the size of the xe_mmio instance.
>>>> + * 6. When the region is no longer needed, call xe_mmio_gem_destroy() to release the GEM object.
>>>> + *
>>>> + * Limitations: The exposed xe_mmio must be page-aligned with regards to its BAR offset and size.
>>>> + *
>>>> + * WARNING: Exposing MMIO regions to userspace can have security and stability implications.
>>>> + * Make sure not to expose any sensitive registers.
>>> This is no secondary issue, it is a primary one. There is no way we can
>>> expose entire PCIe MMIO register space to a non-root user (a root user
>>> itself can just mmap the PCIe MMIO bar bypassing the XeKMD if it wants
>>> btw).
>>>
>>> What is the requirement for this patch? I know on other PCIe devices
>>> I've worked on we had similar issues where we needed to expose a subset
>>> of the PCIe MMIO register space to user space for kernel bypass
>>> submission - everything we needed to expose in that case was 4k aligned
>>> + tied to a single process via a KMD config before mapping to user
>>> space. I suspect to do anything safely here, somethine similar needs to
>>> be done.
>>>
>>> Matt
>>
>> As you've figured out - the idea is of course to expose small chunks (say 1 page)
>> that are allocated solely to a specific user. One use-case is direct submission in
>> upcoming platforms. I am checking 4K alignment in the patch. Not sure what
>> you mean about KMD config - xe_mmio_gem_create accepts a drm_file *.
> On the previous device I worked on we'd config some non-user exposed
> MMIO registers to tie the exposed user registers to specifc user process
> (e.g. we'd setup the PASID of the exposed submission queue). Not sure if
> something like is required on the upcoming device(s).
>
>> So the driver performs exposure of an MMIO region on behalf of the user and
>> only this user will be able to perform mmap. Do you think anything else is required
>> here?
> No, I think it makes sense now but personally I'd drop the reuse of
> xe_mmio and keep that as a heavy weight kernel object only used in
> xe_device, xe_tile, xe_gt, etc...
>
> Can you just pass in size / MMIO address (result of xe_mmio_phys_addr)
> to xe_mmio_gem_create? I think that would avoid a bit confusion.
>
> Matt
The usage of "small" xe_mmio's is actually something we do want in upcoming platforms.
But I agree that copying the xe_mmio into xe_mmio_gem is somewhat not intuitive (hence your
question about it not being a pointer before). How about leaving xe_mmio_gem_create signature as is, but storing only xe_mmio_phys_addr(mmio) in the xe_mmio_gem struct? I'll send out a revision with this change - hopefully it will make this less confusing.
- Ilia
>
>> - Ilia
>>
>>>> + */
>>>> +
>>>> +static void xe_mmio_gem_free(struct drm_gem_object *);
>>>> +static int xe_mmio_gem_mmap(struct drm_gem_object *, struct vm_area_struct *);
>>>> +
>>>> +static const struct vm_operations_struct vm_ops = {
>>>> + .open = drm_gem_vm_open,
>>>> + .close = drm_gem_vm_close,
>>>> +};
>>>> +
>>>> +static const struct drm_gem_object_funcs xe_mmio_gem_funcs = {
>>>> + .free = xe_mmio_gem_free,
>>>> + .mmap = xe_mmio_gem_mmap,
>>>> + .vm_ops = &vm_ops,
>>>> +};
>>>> +
>>>> +static inline struct xe_mmio_gem *to_xe_mmio_gem(struct drm_gem_object *obj)
>>>> +{
>>>> + return container_of(obj, struct xe_mmio_gem, base);
>>>> +}
>>>> +
>>>> +static inline phys_addr_t xe_mmio_phys_addr(struct xe_mmio *mmio)
>>>> +{
>>>> + struct xe_device *xe = tile_to_xe(mmio->tile);
>>>> +
>>>> + /*
>>>> + * All MMIO instances are currently on PCI BAR 0, so we can do the trick below.
>>>> + * In the future we may want to store the physical address in struct xe_mmio.
>>>> + */
>>>> + return pci_resource_start(to_pci_dev(xe->drm.dev), GTTMMADR_BAR) +
>>>> + (uintptr_t)(mmio->regs - xe->mmio.regs);
>>>> +}
>>>> +
>>>> +/**
>>>> + * xe_mmio_gem_create - Expose an MMIO region to userspace
>>>> + * @mmio: xe_mmio instance
>>>> + * @file: DRM file descriptor
>>>> + *
>>>> + * This function creates a GEM object with an mmap-able fake offset that wraps
>>>> + * the provided xe_mmio instance.
>>>> + *
>>>> + * See: "Exposing MMIO regions to userspace"
>>>> + */
>>>> +struct xe_mmio_gem *
>>>> +xe_mmio_gem_create(struct xe_mmio *mmio, struct drm_file *file)
>>>> +{
>>>> + struct xe_device *xe = tile_to_xe(mmio->tile);
>>>> + size_t size = mmio->regs_size;
>>>> + struct xe_mmio_gem *obj;
>>>> + struct drm_gem_object *base;
>>>> + int err;
>>>> +
>>>> + if ((xe_mmio_phys_addr(mmio) % PAGE_SIZE != 0) || (size % PAGE_SIZE != 0))
>>>> + return ERR_PTR(-EINVAL);
>>>> +
>>>> + obj = kzalloc(sizeof(*obj), GFP_KERNEL);
>>>> + if (!obj)
>>>> + return ERR_PTR(-ENOMEM);
>>>> +
>>>> + base = &obj->base;
>>>> + base->funcs = &xe_mmio_gem_funcs;
>>>> + obj->mmio = *mmio;
>>>> +
>>>> + drm_gem_private_object_init(&xe->drm, base, size);
>>>> +
>>>> + err = drm_gem_create_mmap_offset(base);
>>>> + if (err)
>>>> + goto free_gem;
>>>> +
>>>> + err = drm_vma_node_allow(&base->vma_node, file);
>>>> + if (err)
>>>> + goto free_gem;
>>>> +
>>>> + return obj;
>>>> +
>>>> +free_gem:
>>>> + xe_mmio_gem_free(base);
>>>> + return ERR_PTR(err);
>>>> +}
>>>> +
>>>> +static void xe_mmio_gem_free(struct drm_gem_object *base)
>>>> +{
>>>> + struct xe_mmio_gem *obj = to_xe_mmio_gem(base);
>>>> +
>>>> + drm_gem_object_release(base);
>>>> + kfree(obj);
>>>> +}
>>>> +
>>>> +/**
>>>> + * xe_mmio_gem_destroy - Destroy the GEM object wrapping xe_mmio
>>>> + * @gem: the GEM object to destroy
>>>> + *
>>>> + * This function releases resources associated with the GEM object created by
>>>> + * xe_mmio_gem_create().
>>>> + *
>>>> + * See: "Exposing MMIO regions to userspace"
>>>> + */
>>>> +void xe_mmio_gem_destroy(struct xe_mmio_gem *gem)
>>>> +{
>>>> + xe_mmio_gem_free(&gem->base);
>>>> +}
>>>> +
>>>> +static int xe_mmio_gem_mmap(struct drm_gem_object *base, struct vm_area_struct *vma)
>>>> +{
>>>> + struct xe_mmio_gem *obj = to_xe_mmio_gem(base);
>>>> + struct xe_mmio *mmio = &obj->mmio;
>>>> +
>>>> + if (vma->vm_end - vma->vm_start != base->size)
>>>> + return -EINVAL;
>>>> +
>>>> + /*
>>>> + * Set vm_pgoff (used as a fake buffer offset by DRM) to 0 and map the
>>>> + * whole buffer from the start.
>>>> + */
>>>> + vma->vm_pgoff = 0;
>>>> + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>>>> +
>>>> + vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP |
>>>> + VM_DONTCOPY | VM_NORESERVE);
>>>> +
>>>> + return remap_pfn_range(vma, vma->vm_start, xe_mmio_phys_addr(mmio) >> PAGE_SHIFT,
>>>> + base->size, vma->vm_page_prot);
>>>> +}
>>>> diff --git a/drivers/gpu/drm/xe/xe_mmio.h b/drivers/gpu/drm/xe/xe_mmio.h
>>>> index c151ba569003..2990bbcef24d 100644
>>>> --- a/drivers/gpu/drm/xe/xe_mmio.h
>>>> +++ b/drivers/gpu/drm/xe/xe_mmio.h
>>>> @@ -8,6 +8,7 @@
>>>>
>>>> #include "xe_gt_types.h"
>>>>
>>>> +struct drm_file;
>>>> struct xe_device;
>>>> struct xe_reg;
>>>>
>>>> @@ -42,4 +43,7 @@ static inline struct xe_mmio *xe_root_tile_mmio(struct xe_device *xe)
>>>> return &xe->tiles[0].mmio;
>>>> }
>>>>
>>>> +struct xe_mmio_gem *xe_mmio_gem_create(struct xe_mmio *mmio, struct drm_file *file);
>>>> +void xe_mmio_gem_destroy(struct xe_mmio_gem *gem);
>>>> +
>>>> #endif
>>>> --
>>>> 2.43.0
>>>>
More information about the Intel-xe
mailing list