[HarfBuzz] regression test suite

Evan Martin evan at chromium.org
Sun Jan 31 15:00:40 PST 2010


One thing I've really learned from working on WebKit is the utility of
a regression test suite.  Because WebKit is covered by so many tests,
developers are unafraid to make major changes, like large
refactorings, knowing that if they changed any behavior other than
what they intended the test suite will point it out at them.  (See for
example commit e70f45e in the harfbuzz-ng tree as the sort of thing a
test suite would help prevent.)

I made an attempt of sketching out what a test suite for harfbuzz-ng
might look like.
Conceptually, you need:
1) a program that takes a font+text and outputs the result after shaping is done
2) a test runner that archives these results and quickly runs new code
against the previous results

Here's the code:

- The shaper dumper (outputs JSON):
  http://github.com/martine/HarfBuzz-ng/commit/f2cb3c3f1933ded5fc95debc55aa6c937ccdf261

- A test suite runner (just a sketch, but shows the idea):
  http://github.com/martine/HarfBuzz-ng/commit/04bc97769a54905ec196ae9b34845d96ff71e3d9

- An example entry in the test suite (contains the output of the shaper dumper):
  http://github.com/martine/HarfBuzz-ng/commit/d6de8cb3c51f71768a5edf1024f889a74b22e38a

One issue I still haven't quite figured out is how to handle the fact
that shaping results are font-dependent.
My current thinking is we should store the font name and hash of the
font file in the test entry and then have the test runner just skip
any fonts that you don't have available.  In practice, I would expect
that most interesting fonts to test against (including bitstream, the
Microsoft web fonts, Libertine) are freely available if not free
software, so I don't think it's too high a burden to say that if you
want to run all of the test suite you need to install all of the
fonts.

Behdad, what do you think?



More information about the HarfBuzz mailing list