X Integration test suite

[Chase Douglas]

On 08/29/2012 03:36 PM, Peter Hutterer wrote:
> On Wed, Aug 29, 2012 at 01:14:35PM -0700, Chase Douglas wrote:
>> On 08/28/2012 03:57 AM, Peter Hutterer wrote:
>>> One of the things I've spent quite a bit of time on over the last weeks
>>> is a test suite. Chase tried to get some integration tests into the X
>>> server repo a while ago but I think a standalone repo is best (for now
>>> anyway).
>>>
>>> I've pushed the current set of tests to
>>> http://cgit.freedesktop.org/~whot/xorg-integration-tests/, with a
>>> lengthier explanation here:
>>> http://who-t.blogspot.com.au/2012/08/xorg-integration-test-suite.html
>>>
>>> Long story short, the xorg integration tests (XIT) are built on
>>> googletest and xorg-gtest, i.e. written in C++. Most of them write out a
>>> config, fire up a server, and then either check the log or query the
>>> server for state. A few tests use evemu devices to send events.
>>>
>>> Input testing is relatively simple with these tests since we can emulate
>>> virtually anything, but I'm not sure yet if such test cases can scale to
>>> output tests as well. Or even if there are tests that can be fully
>>> automatic without anyone staring at the screen. I know mesa already has
>>> such tests.
>>>
>>> Any feedback is appreciated, I intend to talk about this a bit more at
>>> XDC, but in the meantime you can see if it is useful. Right now, the
>>> bigger issues I'm facing are the build system and scalability if we end
>>> up with a ton of tests. And the ifdef hell that already started with
>>> tests covering different X server versions, RHEL support, etc. Any
>>> epiphanies on how to avoid a train wreck would be appreciated.
>>
>> Overall, I'm quite pleased with how things have progressed :).
>>
>> As for the run time issue, I assume you are mostly hitting it during
>> device tests. Could you find a way to set up all the input config
>> blocks you need into the same context, and then start the tests
>> against a single server?
>
> most of the tests (so far) require a specific InputDevice section that may
> or may not be CorePointer, or require some xorg.conf option to toggle the
> defaults. there are plenty of normal bug tests that don't need that, though
> then we do run the chance of having state-dependent tests. instead of
> starting with a clean slate for all of them.

For those that require specific and different InputDevice sections, you 
could use MatchProductName (or whatever it is) on the name of the 
device, and use different named devices for each test.

Normal tests should be run on the same server instance. If there is any 
state dependency, that's also the sign of a bug. In the end, we'd want 
to be running all of the test in a sequential order, and then in a 
random order. The sequential order will be a stable test suite, the 
random order will give us clues if some server state gets messed up as 
we go.

-- Chase