[cairo] Cairo in 3 Dimensions

[Bill Spitzak]

Esben Stien wrote:
 > Any plans to add a 3'rd dimension to cairo?

I feel the only "vital" missing element is the ability to provide a 
perspective transformation to the source surface. This would change the 
current 3x2 matrix into a 3x3 one. Notice that it is not a 4x4 matrix, 
it does not input or output z information. With this addition a Cairo 
program would be physically capable of producing a texture-mapped 3d scene.

Everything else is optional, and here are my opinions:

3D transforms:

It seems nice to provide a perspective transformation of the user's 
points as they build the path. This could be done by expanding the ctm 
to be a 3x3 matrix. However there is a serious problem in that, as 
defined, Cairo transforms the current font and line stroking pen by the 
ctm. Thus the correct result of drawing a 3D path is as though that path 
was drawn on a flat plane and the resulting plane rotated, then lines 
and text would clearly show that they were on a flat plane that is not 
parallel to the camera, and further away ones would be thinner. This 
does not sound that useful to a program trying to draw and label a 3D 
line drawing. More importantly it sounds very difficult and slow to 
implement, the only plausable method I see is to draw it flat on a 
surface and then perspective transform the result, which will also 
defeat all the hinting and attempts to turn off antialiasing. And that 
won't work unless the Z is equal in the drawing.

At first I thought the solution was to make Cairo have a different 
non-3D CTM for fonts and lines, but on modern machines there is no need 
to have the perspective calculations done by the server. It is instead 
quite practical for the program itself to do the 3D transformation and 
thus leave Cairo entirely 2D, and this serves the purpose of having the 
two different CTMs. A library to assist in drawing a 3D wireframe on 
Cairo might be useful, however.

Z buffer:

Another 3D thing is depth (z) compositing. Basically if something is 
drawn with a z greater than is already there it is invisible (or is 
UNDER composited?). You may think this required 3d transformations, but 
it doesn't. All that is needed is to add an extra coordinate to the 
path, which is the 1/z value (1/z is used because it will linearly 
interpolate). Problems are that antialising is impossible except by 
supersampling, and unless you are compositing with OVER a 100% opaque 
surface, it will not produce the correct picture unless you sort the 
objects from back-to-front and divide them at intersections, but doing 
that eliminates the need to do z buffer entirely! But sending 1/z to the 
points may be useful for shaders, see below:

Lighting/shading:

I think it is pretty clear that the future is programmable shading, and 
any attempt to add literal lights or shading models to Cairo is a waste 
of time. Other than an API to actually set the shader (which will likely 
be surface-specific for quite awhile), the only thing needed for Cairo 
is to be able to add a vector of linearly-interpolated values to each 
point in the path. I would do this by extending the "color" api to list 
a whole set of numbers and bind colors to points in the path. So two 
changes here: allow an arbitrary set of numbers to be provided with the 
color, and store it with each point in the path. Also the enumerate-path 
api needs to be changed so it can return an arbitrary set of numbers 
with each point. It may be a good idea to make these changes even if 
nothing else is done.

Filling a path would linearly-interpolate triangles and send the vector 
to the shader (user has no control over the triangazation and it is 
assummed they have assigned values so it does not matter). Stroking a 
path probably should linearly *extrapolate* the numbers to get the 
corners of the polygon that is filled. Cairo would always set the x and 
y of the vector, and would set u,v,1/w (1/w is for perspecitve textures) 
to map the source pattern transform if those values were not provided 
with a point, and set the rgba to the current color if those values were 
not provided with a point (in effect Cairo would have a hardcoded 
"vertex shader").

The antialiasing and mask would *not* be sent to the shader, instead 
Cairo would use the coverage of the shape to mix whatever was output by 
the shader with the current value of the buffer.

If it was possible to read/write more values than rgba to the buffers, 
then even z-depth compositing could be done by this.

CONCLUSIONS:

So things that *should* be done soon:

1. I think it is absolutely necessary to add 3d transforms to the source 
pattern.

2. Specify the "color" with an arbitrary-length vector of numbers. Cairo 
can throw away all the numbers after rgba for now, the purpose is to fix 
the API.

3. Might as well store the colors with the points and linear-interpolate 
those colors. Lots of people want this.

Things that I think should *not* be done:

1. Don't change the CTM from the current 6-number version.

2. Do not do Z buffer.

Things that can be done in the future:

1. Add surface-dependent api's to set the shader. The current api for 
setting the compositing operation would just turn into this internally, 
so the last one done is the one that is in effect.

2. Allow you to specify an arbitrary number of channels in each surface, 
so that shaders can get more than rgba from them, and output more than 
rgba to them.