[Fontconfig] Greatly improved subpixel filtering in Xft

[Anders Kaseorg]

Keith Packard wrote:
> The current filter was designed to be used in conjunction with well hinted 
> text, that's for sure.  I wonder if you might compare the results of 
> correctly hinted glyphs with your new filter; I'm concerned that extending 
> the filter beyond a single pixel will cause those edges to be fuzzier.

We might disagree about what correct hinting is--I happen to think that 
Freetype's medium autohinting (shown in my screenshots) gives the best 
fonts I've ever seen on a computer screen. I'm sure it's a matter of 
opinion.

If by correct hinting you mean Freetype's full hinting, here are some 
screenshots of that under the two filters:
   <http://www.kaseorg.com/~anders/subpixel-before-fullhinting.png>
   <http://www.kaseorg.com/~anders/subpixel-after-fullhinting.png>
Again, my filter seems to give reduced color artifacts, and doesn't look 
significantly blurrier. Is that what you wanted, or should I compare 
something else?

A closeup of full hinting with the current filter shows that subpixel 
antialiasing only gets applied to diagonal and curved strokes; the rest 
is black and white. I think that's wrong, because a row of white, black, 
white gets shown as "RGB___RGB", and the red and blue parts are off 
center by 1/3 of a pixel. Indeed, if I look really closely at the 
screen, I can see fringes of blue and red surrounding the vertical black 
lines. My filter borders those lines with very light orange and cyan to 
maintain balance. In xmag, that looks as if it would blur things, but on 
the actual screen, it results in sharp black lines.

However, with subpixel rendering (and good filtering :-)) there isn't 
much of an advantage to snapping things to whole-pixel boundaries (at 
least horizontally), as full hinting currently does. It would make more 
sense to hint to pixel boundaries vertically and subpixel boundaries 
horizontally. Is it possible to make Freetype do that?

Anders