[RFC 0/4] Some (drm_sched_|dma_)fence lifetime issues

[Tvrtko Ursulin]

On 28/04/2025 14:15, Christian König wrote:
> On 4/24/25 09:07, Tvrtko Ursulin wrote:
>>
>> On 23/04/2025 14:12, Christian König wrote:
>>> On 4/18/25 18:42, Tvrtko Ursulin wrote:
>>>> Hi all,
>>>>
>>>> Recently I mentioned to Danilo about some fence lifetime issues so here is a
>>>> rough series, more than anything intended to start the discussion.
>>>>
>>>> Most of the problem statement can be found in the first patch but to briefly
>>>> summarise - because sched fence can outlive the scheduler, we can trivially
>>>> engineer an use after free with xe and possibly other drivers. All that is
>>>> needed is to convert a syncobj into a sync file behind drivers back, and I don't
>>>> see what the driver can do about it.
>>>
>>>
>>> Yeah that topic again :) The problem here is that this is not a bug, it is a feature!
>>>
>>> IIRC it was Alex who pointed that issue out on the very first fence patch set, and we already discussed what to do back then.
>>>
>>> The problem with grabbing module references for fences is that you get trivially into circle references and so basically always preventing the module from unloading.
>>
>> Where "always" is only "while there are active objects from that module", no?
> 
> 
> The problem is that dma_fences stay around after they are signaled. And basically all drivers keep some dma_fence around for their resource management. E.g. amdgpu for the VMIDs.
> 
> This means that some dma_fence is referenced by the module and the module referenced by some dma_fence. E.g. you are never able to unload the module.

Are you thinking truly never or for as long someone has a reference?

For example while userspace has a reference to dma_fence via sync_file 
fence owning module would not unloadable. One would have to terminate 
the process, which granted wouldn't be easy to see which process 
prevents the unload, before driver could be unloaded.

For the foreign fences kept around in kernel space, that would be 
solvable by some periodic house keeping at worst.

Also, about the use cases for module unload. Since you and Brost 
especially seem to be expressing a hard no to module references, what 
are the use cases you are concerned about?

>>> The decision was made to postpone this and live with the potential use after free on module unload until somebody has time to fix it. Well that was +10 years ago :)
>>>
>>> I discussed this with Sima again last year and we came to the conclusion that the easiest way forward would be to decouple the dma_fence implementation from the driver or component issuing the fence.
>>>
>>> I then came up with the following steps to allow this:
>>> 1. Decouple the lock used for protecting the dma_fence callback list from the caller.
>>> 2. Stop calling enable_signaling with the lock held.
>>> 3. Nuke all those kmem_cache implementations and force drivers to always allocate fences using kvmalloc().
>>> 4. Nuke the release callback (or maybe move it directly after signaling) and set fence->ops to NULL after signaling the fence.
>>>
>>> I already send patches out for #1 and #2, but don't have enough time to actually finish the work.
>>>
>>> If you want take a look at nuking all those kmem_cache implementations for allocating the fence memory. I think that can be completed completely separate to everything else.
>>
>> So enabling dma fence "revoke" so to say.
>>
>> Just to check we are on the same page, it is not just about the module references, but also use after frees which can happen even if module is still loaded but any memory reachable via dma fence entry points has been freed.
> 
> 
> Yeah, that came much later when people started to use the scheduler dynamically. Basically the sched pointer in the drm_sched_fence implementation becomes invalid as soon as the fence signals.
> 
>>
>> In that case, as Matt has already asked, if you could dig up your unfinished work it would be interesting to see.
> 
> 
> This is what I already send out: https://gitlab.freedesktop.org/ckoenig/linux-drm/-/commits/dma-fence-rework-enable-signaling
> 
> A bunch of the cleanup patches in that branch have already been applied, only the last one is missing IIRC.
> 
> And here is a WIP patch to decouple the lock I wrote halve a year ago or so: https://gitlab.freedesktop.org/ckoenig/linux-drm/-/commits/dma-fence-rework-locking

Thanks!

My concern here is that to me it appears the whole premise is to leave 
fences dangling in memory and somehow make them safe to be accessed by 
importers.

For starters this can create permanent memory leaks. Or at least for the 
same window of duration as would the exporters be not unloadable with 
the reference counting alternative. So we would not a strong argument 
for why poorly bound memory leaks are better than poorly bound 
unloadable modules.

It is also a question how to "revoke" fences safely (race free). It 
sounds hard to me. It does not seem you got to this last problem in the 
above branches so I don't know if you had some elegant ideas for that.

Maybe first to ask if anyone is aware of a precedent where something in 
the kernel already uses this design pattern?

Regards,

Tvrtko