[poppler] Poppler reg tests

[Carlos Garcia Campos]

Hi all,

as you all probably know, our current regression test suite are some
custom shell scripts and a lot of pdf files that Albert has collected
for some years. During the last few days I have been working on
improving the regression test, converting Albert's scripts into a
small python program to allow everybody to run their own tests. We are
not going to upload pdfs to the repo, so we still depend on Albert
to make sure a change doesn't actually introduce regressions. We will
only provide the program so that anybody could use it with their own
pdfs.

So, how does it work?

usage: poppler-regtests [options ...] command [command-options ...] tests ...

Common options:

 --utils-dir: The directory where poppler utils are. The script is
 thought to be used from the git repo, so by default it's ../utils.

 -b, --backends: The backends to tests, for now only cairo, splash,
 text and postscript backends are supported. You can provide a comma
 separated list of backends to test. By default it uses all backends.

Commands:

 create-refs: This is the command to crate the reference files.

 run-tests: This command runs the tests, comparing with references
 previously created with create-refs command

create-refs command:

For every pdf file to test a new directory is created in the
references dir with the name of the pdf file, creating also other
directories found in the tests dir, for example, for the given tests
dir:

./tests/foo.pdf
./tests/annots/annot-file.pdf
./tests/forms/form-file.pdf

the following dirs will be created in references dir:

./refs/foo.pdf/
./refs/annots/annot-file.pdf/
./refs/forms/form-file.pdf/

The references directory is /tmp/refs by default, but any other dir can
be used with --refs-dir command line option.

Reference files are stored in the references dir of every document
prefixed with the backend-name. The following references file can be
created:

 - Checksum files: A file for each backend containing the md5 sum of
 every backend dependent file.
 
 - Backend dependent files: A png file for every document page for
 cairo and splash backends, a single ps file for postscript backend
 and a single txt file for text backend. These can be removed to save
 disk once checksums files are created with the option
 --checksums-only. If you have enough disk it's recommended to keep
 backend dependent files, so that they can be used later by run-tests
 command to generate diffs for failing tests.

 - Crashed files: A file with the extension .crashed indicating the
 test crashed while creating reference images.

 - Failed files: A file with the extension .failed indicating the test
 failed to run. It didn't crash, but the command returned an error
 status. The file contains the error status returned by the command
 used to create the reference files.

 - stderr files: A file with extension .stderr indicating the command
 used to create the reference files wrote something to stderr.

For example:

$ ls -R refs
refs/forms:
forms-scribus.pdf

refs/forms/forms-scribus.pdf:
cairo.md5  postscript.md5  splash.md5  text.md5

refs/sample.pdf:
cairo.crashed  cairo.stderr  postscript.crashed  postscript.ps
postscript.stderr  splash.crashed  splash.stderr  text.md5

Reference files are not regenerated unless --force command line option
is used. You can update your reference files just adding a new file to
the tests dir and running create-refs again.


run-tests command:

This command creates reference files in the output dir and compares
the results with the reference files in references dir.

A test can produce the following results:

 - PASS: Test passed as expected.

 - FAIL: Test was expected to pass but it failed, differences were
   found in checksum files.

 - CRASH: Test was expected to pass but it crashed.
 
 - DOES NOT CRASH: Test was expected to crash, but it didn't. It's not
   possible to know whether test passed or not.
   
 - DOES NOT FAIL: Test was expected to fail with an exit error status,
   but it didn't. It's not possible to know whether test passed or
   not.
   
 - FAIL (status error <status>): Test was expected to pass but it
   failed, the test program returned with an exit status error.
   
 - PASS (Expected crash): Test was expected to crash and it
   crashed. Even though the test crashed, it's considered to pass
   since it's not a regression.
   
 - PASS (Expected fail with status error <status>): Test was expected
   to fail and it failed. Even though the test failed to run, it's
   considered to pass since it's not a regression.
           
Result files for passing test are automatically removed from output
dir unless --keep-results command line option is used. Result files of
tests that failed are not removed. If backend specific reference files
were not removed from references dir, --create-diffs command line
option can be used to create diff files for tests that fail. The
format of the diff file depends on the backend, and it's not currently
supported by postscript backend.

At the end a summary of the test results is printed, something like:

Total 16 tests
15 tests passed (93.75%)
1 tests failed (6.25%): ./tests/annots/annots_move.PDF (splash)
4 tests have stderr output (25.00%): ./tests/sample.pdf (splash),./tests/sample.pdf (postscript),./tests/sample.pdf (text),./tests/sample.pdf (cairo)
Tests run in 10 seconds

The equivalent to what Albert did with his scripts would be something
like:

./poppler-regtest --backends=splash,cairo create-refs --refs-dir=./refs --checksums-only ./tests/
./poppler-regtest --backends=splash,cairo run-tests --refs-dir=./refs --out-dir=./out ./tests/

I think that's all the important stuff. Sorry for the long mail, but
I think it'll make it easier for you to test and review the patch. For
more information use poppler-regtst --help. I'll add all this info to
the README file once the patch is reviewed.

There are still things to do, like adding an option to provide documents
to skip, html backend support, and I'm sure there are a lot of bugs
too. So, feel free to comment, or even better provide patches :-)

-- 
Carlos Garcia Campos
PGP key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x523E6462
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-initial-poppler-regressions-test-program.patch
Type: application/octet-stream
Size: 40626 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/poppler/attachments/20110912/04413f15/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/poppler/attachments/20110912/04413f15/attachment-0001.pgp>