[Piglit] Ideas for Wine Test Suite and Mesa Workflow

[Emil Velikov]

On 18 February 2016 at 22:02, Stefan Dösinger <stefandoesinger at gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Am 2016-02-18 um 19:59 schrieb Emil Velikov:
>> Bth I'm not sure how many developers (on mesa side) actively
>> monitor the wine tests, although at least one of the Wine devs
>> (Stefan Dösinger I believe) does a form of tracking.
> I do, or rather did, performance tracking. Henri and Matteo use r600g as
> their main development platform I believe.
>
> I have changed jobs recently, so I don't have nearly as much time for
> Wine as I used to, but I try to follow it and provide some patches and
> advice. I want to re-start my performance tracking once I have shipped
> the computers that did this job to my new place.
>
Congratulations on the new job. Hope the graphs are to be read from
right to left PTS bug will be sorted by the time you got your setup
sorted. Or perhaps it already is ?

>> Obviously you won't need any of the root or sysfs requirements,
>> although I strongly recommend that wine tests provides distinction
>>  between these four states - pass, skip, timeout and fail.
> Wine has pretty much these states.
>
>> How would one detect between the states depends on you/wine devs.
>> Having distinction between separate tests (i.e. one can run as
>> many tests, independently, as possible) and tests and subtests
>> would also be nice to have.
> This is where things get tricky. Wine's d3d tests are in huge files that
> contain lots of independent tests, but because they're in one file
> (and thus one make target) you cannot run them independently. If you
> simply use make's return value you cannot evaluate them independently
> either.
>
If subtests are interdependent, then splitting/running them
independently does not make much sense.
Some reporting "subtest X passed/failed/etc" would be great through.
Not too sure how the runner interprets/handles those currently.

> However, the test output can be used to break the results up using
> scripts. Wine's testbot (an automated system that runs these tests on
> Windows) has scripts that do that.

One option would be to get those in the wine repo. Alternatively
looking at intel-gpu-tools might give some other ideas. From a very
quick look one could even tweak tools/make_makefiles in order to
generate the list (test command line) that is to be fed to piglit ?

-Emil