Fence, timeline and android sync points

[Jerome Glisse]

On Tue, Aug 12, 2014 at 06:13:41PM -0400, Jerome Glisse wrote:
> Hi,
> 
> So i want over the whole fence and sync point stuff as it's becoming a pressing
> issue. I think we first need to agree on what is the problem we want to solve
> and what would be the requirements to solve it.
> 
> Problem :
>   Explicit synchronization btw different hardware block over a buffer object.
> 
> Requirements :
>   Share common infrastructure.
>   Allow optimal hardware command stream scheduling accross hardware block.
>   Allow android sync point to be implemented on top of it.
>   Handle/acknowledge exception (like good old gpu lockup).
>   Minimize driver changes.
> 
> Glossary :
>   hardware timeline: timeline bound to a specific hardware block.
>   pipeline timeline: timeline bound to a userspace rendering pipeline, each
>                      point on that timeline can be a composite of several
>                      different hardware pipeline point.
>   pipeline: abstract object representing userspace application graphic pipeline
>             of each of the application graphic operations.
>   fence: specific point in a timeline where synchronization needs to happen.
> 
> 
> So now, current include/linux/fence.h implementation is i believe missing the
> objective by confusing hardware and pipeline timeline and by bolting fence to
> buffer object while what is really needed is true and proper timeline for both
> hardware and pipeline. But before going further down that road let me look at
> things and explain how i see them.
> 
> Current ttm fence have one and a sole purpose, allow synchronization for buffer
> object move even thought some driver like radeon slightly abuse it and use them
> for things like lockup detection.
> 
> The new fence want to expose an api that would allow some implementation of a
> timeline. For that it introduces callback and some hard requirement on what the
> driver have to expose :
>   enable_signaling
>   [signaled]
>   wait
> 
> Each of those have to do work inside the driver to which the fence belongs and
> each of those can be call more or less from unexpected (with restriction like
> outside irq) context. So we end up with thing like :
> 
>  Process 1              Process 2                   Process 3
>  I_A_schedule(fence0)
>                         CI_A_F_B_signaled(fence0)
>                                                     I_A_signal(fence0)
>                                                     CI_B_F_A_callback(fence0)
>                         CI_A_F_B_wait(fence0)
> Lexique:
> I_x  in driver x (I_A == in driver A)
> CI_x_F_y call in driver X from driver Y (CI_A_F_B call in driver A from driver B)
> 
> So this is an happy mess everyone call everyone and this bound to get messy.
> Yes i know there is all kind of requirement on what happen once a fence is
> signaled. But those requirement only looks like they are trying to atone any
> mess that can happen from the whole callback dance.
> 
> While i was too seduced by the whole callback idea long time ago, i think it is
> a highly dangerous path to take where the combinatorial of what could happen
> are bound to explode with the increase in the number of players.
> 
> 
> So now back to how to solve the problem we are trying to address. First i want
> to make an observation, almost all GPU that exist today have a command ring
> on to which userspace command buffer are executed and inside the command ring
> you can do something like :
> 
>   if (condition) execute_command_buffer else skip_command_buffer
> 
> where condition is a simple expression (memory_address cop value)) with cop one
> of the generic comparison (==, <, >, <=, >=). I think it is a safe assumption
> that any gpu that slightly matter can do that. Those who can not should fix
> there command ring processor.
> 
> 
> With that in mind, i think proper solution is implementing timeline and having
> fence be a timeline object with a way simpler api. For each hardware timeline
> driver provide a system memory address at which the lastest signaled fence
> sequence number can be read. Each fence object is uniquely associated with
> both a hardware and a pipeline timeline. Each pipeline timeline have a wait
> queue.
> 
> When scheduling something that require synchronization on a hardware timeline
> a fence is created and associated with the pipeline timeline and hardware
> timeline. Other hardware block that need to wait on a fence can use there
> command ring conditional execution to directly check the fence sequence from
> the other hw block so you do optimistic scheduling. If optimistic scheduling
> fails (which would be reported by hw block specific solution and hidden) then
> things can fallback to software cpu wait inside what could be considered the
> kernel thread of the pipeline timeline.
> 
> 
> From api point of view there is no inter-driver call. All the driver needs to
> do is wakeup the pipeline timeline wait_queue when things are signaled or
> when things go sideway (gpu lockup).
> 
> 
> So how to implement that with current driver ? Well easy. Currently we assume
> implicit synchronization so all we need is an implicit pipeline timeline per
> userspace process (note this do not prevent inter process synchronization).
> Everytime a command buffer is submitted it is added to the implicit timeline
> with the simple fence object :
> 
> struct fence {
>   struct list_head   list_hwtimeline;
>   struct list_head   list_pipetimeline;
>   struct hw_timeline *hw_timeline;
>   uint64_t           seq_num;
>   work_t             timedout_work;
>   void               *csdata;
> };
> 
> So with set of helper function call by each of the driver command execution
> ioctl you have the implicit timeline that is properly populated and each
> dirver command execution get the dependency from the implicit timeline.
> 
> 
> Of course to take full advantages of all flexibilities this could offer we
> would need to allow userspace to create pipeline timeline and to schedule
> against the pipeline timeline of there choice. We could create file for
> each of the pipeline timeline and have file operation to wait/query
> progress.
> 
> Note that the gpu lockup are considered exceptional event, the implicit
> timeline will probably want to continue on other job on other hardware
> block but the explicit one probably will want to decide wether to continue
> or abort or retry without the fault hw block.
> 
> 
> I realize i am late to the party and that i should have taken a serious
> look at all this long time ago. I apologize for that and if you consider
> this is to late then just ignore me modulo the big warning the crazyness
> that callback will introduce an how bad things bound to happen. I am not
> saying that bad things can not happen with what i propose just that
> because everything happen inside the process context that is the one
> asking/requiring synchronization there will be not interprocess kernel
> callback (a callback that was registered by one process and that is call
> inside another process time slice because fence signaling is happening
> inside this other process time slice).
> 
> 
> Pseudo code for explicitness :
> 
> drm_cs_ioctl_wrapper(struct drm_device *dev, void *data, struct file *filp)
> {
>    struct fence *dependency[16], *fence;
>    int m;
> 
>    m = timeline_schedule(filp->implicit_pipeline, dev->hw_pipeline,
>                          dependency, 16, &fence);
>    if (m < 0)
>      return m;
>    if (m >= 16) {
>        // alloc m and recall;
>    }
>    dev->cs_ioctl(dev, data, filp, dev->implicit_pipeline, dependency, fence);
> }
> 
> int timeline_schedule(ptimeline, hwtimeline, timeout,
>                        dependency, mdep, **fence)
> {
>    // allocate fence set hw_timeline and init work
>    // build up list of dependency by looking at list of pending fence in
>    // timeline
> }
> 
> 
> 
> // If device driver schedule job hopping for all dependency to be signaled then
> // it must also call this function with csdata being a copy of what needs to be
> // executed once all dependency are signaled
> void timeline_missed_schedule(timeline, fence, void *csdata)
> {
>    INITWORK(fence->work, timeline_missed_schedule_worker)
>    fence->csdata = csdata;
>    schedule_delayed_work(fence->work, default_timeout)
> }
> 
> void timeline_missed_schedule_worker(work)
> {
>    driver = driver_from_fence_hwtimeline(fence)
> 
>    // Make sure that each of the hwtimeline dependency will fire irq by
>    // calling a driver function.
>    timeline_wait_for_fence_dependency(fence);
>    driver->execute_cs(driver, fence);
> }
> 
> // This function is call by driver code that signal fence (could be call from
> // interrupt context). It is responsabilities of device driver to call that
> // function.
> void timeline_signal(hwtimeline)
> {
>   for_each_fence(fence, hwtimeline->fences, list_hwtimeline) {
>     wakeup(fence->pipetimeline->wait_queue);
>   }
> }

Btw as extra note, because of implicit timeline any shared object schedule on a
hw timeline must add a fence to all the implicit timeline where this object exist.

Also there is no need to have a fence pointer per object.

> 
> 
> Cheers,
> Jérôme