X Integration test suite

[Matt Dew]

On 08/30/2012 06:13 AM, Peter Hutterer wrote:
> On Wed, Aug 29, 2012 at 04:22:33PM -0700, Chase Douglas wrote:
>> On 08/29/2012 03:36 PM, Peter Hutterer wrote:
>>> On Wed, Aug 29, 2012 at 01:14:35PM -0700, Chase Douglas wrote:
>>>> On 08/28/2012 03:57 AM, Peter Hutterer wrote:
>>>>> One of the things I've spent quite a bit of time on over the last weeks
>>>>> is a test suite. Chase tried to get some integration tests into the X
>>>>> server repo a while ago but I think a standalone repo is best (for now
>>>>> anyway).
>>>>>
>>>>> I've pushed the current set of tests to
>>>>> http://cgit.freedesktop.org/~whot/xorg-integration-tests/, with a
>>>>> lengthier explanation here:
>>>>> http://who-t.blogspot.com.au/2012/08/xorg-integration-test-suite.html
>>>>>
>>>>> Long story short, the xorg integration tests (XIT) are built on
>>>>> googletest and xorg-gtest, i.e. written in C++. Most of them write out a
>>>>> config, fire up a server, and then either check the log or query the
>>>>> server for state. A few tests use evemu devices to send events.
>>>>>
>>>>> Input testing is relatively simple with these tests since we can emulate
>>>>> virtually anything, but I'm not sure yet if such test cases can scale to
>>>>> output tests as well. Or even if there are tests that can be fully
>>>>> automatic without anyone staring at the screen. I know mesa already has
>>>>> such tests.
>>>>>
>>>>> Any feedback is appreciated, I intend to talk about this a bit more at
>>>>> XDC, but in the meantime you can see if it is useful. Right now, the
>>>>> bigger issues I'm facing are the build system and scalability if we end
>>>>> up with a ton of tests. And the ifdef hell that already started with
>>>>> tests covering different X server versions, RHEL support, etc. Any
>>>>> epiphanies on how to avoid a train wreck would be appreciated.
>>>>
>>>> Overall, I'm quite pleased with how things have progressed :).
>>>>
>>>> As for the run time issue, I assume you are mostly hitting it during
>>>> device tests. Could you find a way to set up all the input config
>>>> blocks you need into the same context, and then start the tests
>>>> against a single server?
>>>
>>> most of the tests (so far) require a specific InputDevice section that may
>>> or may not be CorePointer, or require some xorg.conf option to toggle the
>>> defaults. there are plenty of normal bug tests that don't need that, though
>>> then we do run the chance of having state-dependent tests. instead of
>>> starting with a clean slate for all of them.
>>
>> For those that require specific and different InputDevice sections,
>> you could use MatchProductName (or whatever it is) on the name of
>> the device, and use different named devices for each test.
>>
>> Normal tests should be run on the same server instance. If there is
>> any state dependency, that's also the sign of a bug. In the end,
>> we'd want to be running all of the test in a sequential order, and
>> then in a random order. The sequential order will be a stable test
>> suite, the random order will give us clues if some server state gets
>> messed up as we go.
>
> the problem with random orders are that they're hard to reproduce. It
> doesn't help me when a random-order test fails if I can't run the tests in
> exactly the same order to reproduce. so we'd have to store or print the seed
> somewhere.

Generate an output file (xml, json,  whatever) than can be fed straight 
back in that contains seed, and all other pertinent info.


>
> Cheers,
>     Peter
> _______________________________________________
> xorg-devel at lists.x.org: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: http://lists.x.org/mailman/listinfo/xorg-devel
>