[Nouveau] SERIALIZE command

[Marcin Kościelnicki]

> Oops... I thought that each subchannel is always bound to the same object,
> and I was actually looking at only the kernel driver ;-) I have been trying
> to understand the behavior of GPU operations, using Gallium demo
> applications. To do so, I was dumping the commands sent from the user space
> (Nouveau/Gallium libdrm), by mapping the pushbuf to the kernel virtual
> address  in the kernel driver, to see what is going on. I was also
> inserting another command in the kernel driver, like in
> nouveau_gem_ioctl_pushbuf(), to see what happens. Since several Gallium
> demo applications seem to use the subchannels in the same manner, I
> misunderstood that they are constantly bound to the same objects... I
> should have looked into the libdrm code in detail. Now I see them, thanks.
> By the way, when I was dumping the commands, I sometimes saw 0x00000000
> without any data. Do you have any idea what this value means? (just a NOP?)

Plain 0 in a pushbuffer is just a nop, yes. It's inserted as a placeholder for 
unused relocations sometimes.

The known pushbuf commands are roughly:

- bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 0: submit methods, increasing 
kind. method is in bits 2-12, subchan in 13-15, method count in 18-28. 
Subsequent <method count> words in FIFO get pushed into sequential methods 
starting from the given one.
- bits 0-1 = 0, bits 29-31 = 1: JMP to data & 0x1ffffffc
- bits 0-1 = 0, bits 29-31 = 2, bits 16-17 = 0: submit methods, non-increasing 
kind. method is in bits 2-12, subchan in 13-15, method count in 18-28. 
Subsequent <method count> words in FIFO get pushed into that method. [NV10+]
- bits 0-1 = 2: CALL to data & 0xfffffffc [NV11+]
- bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 2: RET [NV11+]
- bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 1: conditional. disables method 
submission if bits 4-15 of this command AND contents of PFIFO reg 2058 == 0. 
Used for selecting individual cards in SLI configurations for part of the 
command stream [NV40+]
- bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 3: big non-increasing method 
submission. like the other non-increasing method submission, but method count 
is now taken from next word after the command and can be much larger. Used for 
pushing big amounts of data through the PFIFO, like nv50-style index buffers. 
[NV50+... I think.]

There are likely more, but I don't recall any right now.

> > PFIFO_CACHE_ERROR is an error that happens when the PFIFO puller
> > encounters some problems. One of the causes is submitting a command on a
> > subchannel which
> > isn't actually bound to anything, which is what happens here.
> >
> > Software objects are fake objects where you don't actually bind any
> > object to
> > a subchannel, and instead trap the PFIFO_CACHE_ERROR this causes on
> > command submission and do something on the host side. We use this for
> > syncing with vblank on nv50. These don't support the SERIALIZE command
> > because they're not
> > real objects.
>
> So I was sending those commands with unbounded subchannels, that is why I
> got the PFIFO_CACHE_ERROR interrupts.

Yup. Or maybe you sent them on out software object, and nouveau didn't 
recognise it.

As I already mentioned, there are two kinds of objects you can bind to 
subchannels... software and PGRAPH objects. There are likely more for video 
dec, but meh.

For software objects, all methods >= 0x100 cause PFIFO_CACHE_ERRORs, which 
driver can trap and emulate, or not.

For PGRAPH objects, all methods >= 0x100 are passed on to PGRAPH and are 
handled there, perhaps causing PGRAPH_ILLEGAL_MTHD error if they happen to not 
exist.

Methods < 0x100 are handled directly by the PFIFO, and are independent of the 
current object. There are a couple of known ones...

> > Basically, there exist only 4 types of real objects you can submit
> > commands on
> > and bind to subchans:
> >
> > - m2mf [5039]: memory copies
> > - eng2d [502d]: 2d engine
> > - tesla [5097, 8297, 8397, 8597]: 3d engine
> > - turing [50c0, 85c0]: computation engine: CUDA and the like
> >
> > Very likely there are more types, using different engine than PGRAPH, but
> > we
> > don't know anything about them yet. At the very least there should be the
> > video decoding engine.
>
> Got it.
> Regarding turing, you mean that Nouveau now supports CUDA?

Kernel allows you to create the relevant object, but other than that there's 
no support. You can indeed run your gpu code under nouveau if you're happy 
with writing it in nv50 machine code and submitting via the pushbuffer 
interface. Also our current memory managment is totally unfit for GPGPU 
applications and will need to be redone before there's any hope of seeing CUDA 
or OpenCL support in nouveau.

Also, if you have any more questions and don't hate IRC, you can come to 
#nouveau channel on freenode. Makes it easier to answer.

> Thanks,
> - Shinpei

Marcin Kościelnicki