First version of host1x intro

[Terje Bergström]

Hi,

I created a base for host1x introduction text, and pasted it into
https://gitorious.org/linux-tegra-drm/pages/Host1xIntroduction. For
convenience, I also copy it below.

As I've worked with all of this for so long, I cannot know what areas
are most interesting to you, so I just tried to put in the basics and
scope it to the features we've been discussing so far. Please point out
the features that you'd like more information on so I can add details.

2D is still totally missing from here. Everything is treated as generic
host1x clients. libdrm is touched on only briefly.

The text is written in LaTeX, and converted with pandoc. I beg for
forgiveness for any formatting oddities.

Hardware introduction
=====================

HOST1X is a front-end to a list of client units which deal with graphics
and multimedia. The most important features are channels for serializing
and offloading programming of the client units, and sync points for
synchronizing client units with each other, or with CPU.

Channels
--------

Channel is a push buffer containing HOST1X opcodes. The push buffer
boundaries are defined with `HOST1X_CHANNEL_DMASTART_0` and
`HOST1X_CHANNEL_DMAEND_0`. `HOST1X_CHANNEL_DMAGET_0` indicates the next
position within the boundaries that is going to be processes, and
`HOST1X_CHANNEL_DMAPUT_0` indicates the position of last valid opcode.
Whenever `HOST1X_CHANNEL_DMAPUT_0` and `HOST1X_CHANNEL_DMAGET_0` differ,
command DMA will copy commands from push buffer to a command FIFO.

If command DMA sees opcode GATHER, it will a memory area to command
FIFO. The number of words is indicated in GATHER opcode, and the base
address is read from the following word. GATHERs are not recursive.

HOST1X command processor goes through the FIFO and executes opcodes.
Each channel has some stored state, such as the client unit this channel
is talking to. The most important opcodes are:

-   SETCL for changing the target client unit

-   IMM, INCR, NONINCR, MASK write values to registers of client unit

-   GATHER instructs command DMA to fetch from another memory area

-   RESTART instructs command DMA to start over from beginning of push
    buffer

Channel class can also be HOST1X itself. Register writes to HOST1X will
invoke host class methods. The most important use is
`NV_CLASS_HOST_WAIT_SYNCPT_0`, which freezes a channel until sync point
reaches a threshold value.

Synchronization
---------------

A sync point is a 32-bit register in HOST1X. There are 32 sync points in
Tegra2 and Tegra3. HOST1X can be programmed to assert an interrupt when
a value higher than a pre-determined threshold is written to sync
pointer register. Each channel can also be frozen waiting for a
threshold to be reached.

Sync points are initialized to zero at boot-up, and treated as
monotonously incrementing counter with wrapping. CPU can increment a
sync point by writing the sync point id (0-31 in Tegra2 and Tegra3) to
register `HOST1X_SYNC_SYNCPT_CPU_INCR_0`. Client units all have sync
point increment method at offset 0, and the command streams request
client units to increment sync point using that. The parameters for the
increment method are condition and sync point id. Condition could be
`OP_DONE` telling to increment sync point when previous operations are
done, or `RD_DONE` indicating that client unit has finished all reads
from buffers.

Software
========

There are three components involved with programming HOST1X and its
client units. Linux kernel contains the drivers tegradrm and host1x.
User space library libdrm is added functionality to communicate with the
tegradrm, which communicates with host1x driver.

This text discusses only pieces relevant to HOST1X and its client units,
excluding the part about frame buffer and display controller
programming.

libdrm
======

libdrm communicates with tegradrm kernel driver to allocate buffers,
create and send command streams, synchronize.

TODO

tegradrm
========

tegradrm contains functionality to allocate buffers, and open channels.
The only channel available at the moment is 2D channel, which is handled
by the 2D driver inside tegradrm.

Command stream management and synchronization is passed on from 2D
driver to host1x driver. The 2D driver inside tegradrm processes the
requests from user space, and calls relevant calls in host1x.

host1x driver
=============

At bootup, host1x initializes hardware. It clears sync points, and
registers interrupt handlers.

Sync points
-----------

Each sync point register is treated as a range. The range minimum is a
shadow copy of the sync point register, and the maximum tracks how many
increments we expect to be done. A fence is a pair (sync point id,
threshold value) indicating completion of an event of interest to
software.

Due to wrapping, software does pre-checking for each sync point wait,
whether done via HOST1X channel, or CPU. Each wait is potentially for an
already expired fence. Any wait whose threshold value lies outside the
range ]min, max] is treated as already expired and will not be sent to
HOST1X hardware.

The sync point CPU wait is handled by registering the threshold value as
an event to the interrupt code, and waiting for completion of that
event.

Interrupt management
--------------------

HOST1X has two kinds of interrupt: generic and sync point. Generic
interrupts are not interesting in this scope, so this text focuses on
sync point threshold interrupts.

Interrupt code manages a sorted list of events, and their sync point
threshold values. The earliest event is kept first in the list.

`nvhost_intr_add_action()` adds an action to the event list. If the
event list was empty, HOST1X interrupt is programmed to assert interrupt
when that threshold is reached.

When an interrupt is asserted, the event list is processed. Each event
that has had its threshold passes will be moved to a completed list, and
removed from the event list. Submit complete is treated specially to
optimize for the fact that processing the event is heavy, so we call it
only once even though we have completed multiple submit complete events.

`action_submit_complete()` handles all clean-up for completed jobs.

`action_wakeup()` and `action_wakeup_interruptible()` wake up a thread
waiting for a particular sync point threshold.

After the list of events is processed, the value of the head of the list
is written to HOST1X as the next interrupt threshold.

Job management
--------------

Each command stream sent from user space to kernel is treated as a job.
User space indicates how many sync point increments that stream
generates, and which sync point register it’s using. It also indicates
the buffers involved with the command stream, and the locations in
command stream where the buffers are referred to. Last but not least,
user space indicates locations of sync point waits and thresholds.

First action taken is taking a reference to all buffers in the command
stream. This includes the command stream buffers themselves, but also
the target buffers. We also map each buffer to target hardware to get a
device virtual address.

After this, relocation information is processed. Each reference to
target buffers in command stream are replaced with device virtual
addresses. The relocation information contains the reference to target
buffer, and to command stream to be able to do this.

After relocation, each wait is checked against expiration. Any wait
whose threshold has already expired will be converted to a no-wait by
writing `0x00000000` over the word. This will essentially turn any
expired wait into a wait for sync point register 0, value 0, and thus we
keep sync point 0 reserved for this purpose and never change it from
value 0.

In upstream kernel without IOMMU support we also check the contents of
the command stream for any accesses to memory that are not taken care of
by relocation information.

Next, the number of sync point increments is checked and the id of the
sync point. The sync point maximum value is incremented by the number of
increments, and thus kernel ends up with a fence indicating when that
job has completed. Then each command stream is added to push buffer. In
case of IOMMU support, GATHER opcodes referring to the command streams
are added to the channel push buffer. If IOMMU isn’t supported, the
contents of the GATHER is copied. The fence is added to interrupt event
list as a submit complete action, and at this point the job is
submitted.

When a fence for a job is reached, `action_submit_complete()` will call
`nvhost_cdma_update()`. It goes through list of jobs in channel, and
frees the resources associated for all jobs whose fence has been
reached.

At submit time, we also start a timer for each job. If the timer times
out, the job is removed from the channel, and the sync point increments
that haven’t been done, will be done by the host1x driver. This prevents
channel from remaining stuck in case a command stream is formed
incorrectly and cannot be completed.