Keeping the Screen Turned off While Getting Inputs

[Vladimir Dergachev]

On Sun, 27 Aug 2023, Ahmad Nouralizadeh wrote:

> > The framebuffer that is displayed on the monitor is always in video card
> > memory. There is a piece of hardware (CRTC) that continuously pulls data
> > from the framebuffer and transmits it to the monitor.
> 
> So the framebuffer memory should normally be in the kernel (Perhaps in special cases could be mapped in the userspace?!). IIUC, XServer works on the app GUI data in the userspace and sends it to the
> kernel to finally arrive at the framebuffer. Correct? Does it use some kind of ioctl()?
> 
>

Not necessarily - for very old cards you would issue a special command to 
transfer data or paint a line.

Modern video cards (and video drivers) usually work like this - the 
graphics card exposes several regions that work like memory over PCIe bus 
- i.e. the CPU can access them by issuing a "mov" command to an address 
outside main CPU memory (assuming the graphics card is a physical PCIe 
card).

One of the regions is the entire video card memory that includes the 
framebuffer. This way you can transfer data by simplying copying it to the 
memory mapped region.

This however is slow, even with modern CPUs, because of limitations of 
PCIe bandwidth and because the CPUs are not well suited to the task.

Instead a second memory region contains "registers" - special memory 
locations that, when written, make magic happen. Magic is an appropriate 
word here because the function of those registers is entirely arbitrary - 
their function is picked by hardware designers and there aren't any strict
constraints to force a particular structure.

For example, one register could contain starting x coordinate, another 
starting y, another ending x, another ending y, one more contain a color, 
and finally a special register that, when written, will draw a line in the 
framebuffer from start to end using that color.

This is much faster than using a CPU because only a few values are 
transferred - rest of the work is done by the video card.

And this is how video cards used to work a few decades back, and partially 
still do. However, for modern needs this is still too slow.

So one more feature of video cards is that they have "PCIe bus master" - 
the ability to access main CPU memory directly and retrieve (or write) 
data there.

So instead of transferring data to the framebuffer (for example) by having 
the CPU write there, the CPU will write to video card registers the 
addresses (plural) of memory regions to transfer and then trigger the 
transfer by writing a special register. The video card will do the work.

The transfer to the framebuffer is not very interesting, but what you can 
do is PCI bus master to registers instead. This is usually done by a 
dedicated unit, so it is not exactly like writing to the registers, but 
this makes for a good simplified explanation.

So now you have a memory region in main memory where CPU has assembled 
data like "address of register of starting X", "value of starting X", 
"register address for color of starting point", "value of color" and so 
on, finishing "address of trigger register", "Trigger !".

And this now looks like instructions for a very, very weird VLIW (very 
long instruction word) processor.

The OpenGL driver now works by taking OpenGL commands and compiling them 
to sequences of these weird GPU instructions that are placed into memory 
buffer. When enough of these accumulate, the video card is given the 
trigger to go and execute them, and something gets painted.

If you need to paint a picture, another buffer is allocated, picture data 
is written into it, and then a special command is created instructing to 
pull data from that buffer.

Now, over the past few decades the video cards evolved to be slightly less 
weird VLIW processors - they are getting rid of dedicated commands like 
draw a line from X to Y, in favor of commands like "compute  dot product 
between arrays of 4-dimensional vectors".

They still have the weird multi-tier PCIe bus master, and multiple caches
used to access multiple types of memory: framebuffer memory, texture 
memory, main memory and a few others. And a weird quirks that make doing 
interesting programming with GPUs tricky.

So now, if you start some OpenGL app on Linux and look into /proc/XXX/maps 
you should be able to find several memory regions that have been mapped by 
the graphics driver. Some of those is real memory, some are registers and 
are entirely virtual - there isn't any physical DRAM backing them.

These aren't all the regions exposed by video card, because if multiple 
apps write to video card register directly it will lock up hard, freezing 
PCIe bus. Instead, this is arbitrated by the kernel driver.

best

Vladimir Dergachev