[RFC] [tests] Running the wayland tests against compositor-headless

[Sam Spilsbury]

On Mon, Apr 22, 2013 at 3:48 PM, Eoff, Ullysses A
<ullysses.a.eoff at intel.com> wrote:
>> -----Original Message-----
>> From: wayland-devel-bounces+ullysses.a.eoff=intel.com at lists.freedesktop.org
>> [mailto:wayland-devel-
>> bounces+ullysses.a.eoff=intel.com at lists.freedesktop.org] On Behalf Of Eoff,
>> Ullysses A
>> Sent: Monday, April 22, 2013 1:24 PM
>> To: Sam Spilsbury; wayland
>> Subject: RE: [RFC] [tests] Running the wayland tests against compositor-headless
>>
>> Changing the tests to run headless is probably not necessary...
>>
>
> To clarify... I only meant this in the context of the weston-tests-env script , i.e.
> it does not need to be changed in order for you to automate 'make check' to
> run them on headless ;)
>

Ah interesting. I guess the point of the tests is to verify the
backends themselves as opposed to testing the core compositor (well,
it tests that as an incident, but I guess its not the point?). Does it
make sense to have also acceptance tests which really only verify the
core behavior of weston and not the backends themselves?

> ...but if it's a problem that the tests themselves can't run on the headless backend,
> then changing those in a way to work on headless, too, might be useful.

So at the moment - they aren't - well, its possible to run the two
library based ones. 3 of the client based ones are easy to enable, as
for the other three, two of them protocol support from the backend,
the other one (xwayland) doesn't really make sense on headless.

The broader reason I'm looking into this is because I want to see if
we can eventually export the weston-test module as part of the
installation as well as the headless and no-op backends so that
applications can run integration tests against weston to verify that
they handle input, shell surface changes etc correctly. I'm looking
into doing some GSoC work on a project where I suspect that the
wayland backend will not be run all the time, and it will be good to
have integration tests in place to ensure that it doesn't break
accidentally because developers may not be able to run it regularly.

So - shall I create and submit some patches to get the tests working
on headless? I suppose this is a desirable thing right?

Best,

Sam

>
>> You can create your own custom "check" script if weston-test-env
>> does not suite your needs.  To use it with "make check", just set/export the
>> LOG_COMPILER environment variable to refer to that script before running
>> "make check".  "make check" just invokes your LOG_COMPILER script with the
>> current test as an argument and then uses your scripts exitcode to determine
>> pass/fail status.
>>
>> See <http://www.gnu.org/software/automake/manual/html_node/Parallel-
>> Test-Harness.html>
>>
>> I use a custom python script in this manner so that I can run the tests on both
>> Weston drm and x11 backends per test with only one invocation of "make
>> check".
>> My custom script also allows me to parse the test output/results into a standard,
>> normalized
>> Xml format that is recognized by our continuous integration server.
>>
>> U. Artie Eoff
>>
>> > -----Original Message-----
>> > From: wayland-devel-
>> bounces+ullysses.a.eoff=intel.com at lists.freedesktop.org
>> > [mailto:wayland-devel-
>> > bounces+ullysses.a.eoff=intel.com at lists.freedesktop.org] On Behalf Of Sam
>> > Spilsbury
>> > Sent: Monday, April 22, 2013 11:50 AM
>> > To: wayland
>> > Subject: [RFC] [tests] Running the wayland tests against compositor-headless
>> >
>> > Hi,
>> >
>> > I'm not sure if this is a work-item that's already in-progress by
>> > somebody. I noticed that weston has a "headless" compositor backend
>> > and a "no-op" renderer, which effectively do not depend on having
>> > OpenGL or *a* windowing system available at the time that weston is
>> > run.
>> >
>> > It would be really great if we could run the tests against this
>> > backend, because then it means that they can be run on foreign
>> > machines like continuous integration servers with ease.
>> >
>> > I've tried quickly changing the backend over in weston-test-env -
>> > currently most of the tests are failing because client->pointer is not
>> > initialized. Upon initializing that, most of them pass except those
>> > that depend on client->keyboard . I suspect that this will involve
>> > having to create a wl_keyboard with a set of dummy xkb data for
>> > layouts, though I'm not sure how to do that yet.
>> >
>> > My question to the list is - does it make sense to submit a patch to
>> > change the tests over to use the headless backend? Or is the
>> > preference to keep them using the wayland or x11 backend for now.
>> >
>> > Best,
>> >
>> > Sam.
>> >
>> > --
>> > Sam Spilsbury
>> > _______________________________________________
>> > wayland-devel mailing list
>> > wayland-devel at lists.freedesktop.org
>> > http://lists.freedesktop.org/mailman/listinfo/wayland-devel
>> _______________________________________________
>> wayland-devel mailing list
>> wayland-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel



--
Sam Spilsbury