drm framebuffer/scanout buffer transform

[Dan Walkes]

Hi everyone,

I’m working on a project to implement a color transformation of all
pixels in the framebuffer.  I’d like to add this transform at a layer
where video from all sources (mesa, X, directfb) can be modified with
a color transformation.  I’ve researched linux driver, mesa and
xserver source and based on the architecture described at:
http://intellinuxgraphics.org/documentation/presentations/Graphics_architecture.pdf
it looks like drm is the correct place to add my transform as it is
the common layer immediately above the hardware which manages video
frame buffer memory.

I’d ideally like to add a transform whenever video is ready to
transfer from the frame buffer to the scanout buffer and be displayed
on the screen.  I’m having trouble finding a set of locations where
this actually occurs and I’m wondering if someone on this list can
point me in the right direction.

I’ve setup systemtap scripts to analyze the interactions with the drm
driver and video driver.  In my case I’ve started with a virtualbox
system and vmwgfx, although I’d like to eventually support intel
platforms.

I’ve found drm_mode_dirtyfb_ioctl is one ioctl which seems to be
invoked whenever the display changes (as I move the mouse across the
screen, start applications, etc) however I don’t see this method when
I attempt to play a video with mplayer.  Based on my research I had
expected video to use overlay buffers, however I also don’t see calls
to overlay ioctls like vmw_overlay_ioctl, vmw_stream_*_ioctl.

I see references to the scanout buffer in drm_irq and intel_display
within drm source, as well as in the documentation at
http://www.kernel.org/doc/htmldocs/drm/ch02s05.html but I’m having a
hard time figuring out where the scanout buffer actually *is*.  If I
could understand the relationship between the framebuffer and scanout
buffer better I could probably use this to figure out where I’d need
to add my transform.

I’m still trying to come up to speed with the entire video
architecture and have a long way to go but I was wondering if someone
on this list can provide any pointers or even just provide their
opinion about whether what I’m trying to do is even possible.

Any input is much appreciated.

Thanks for your time.