[PATCH weston] Added simple unit/integration test framework and corresponding test program.

[Daniel Stone]

Hi,
Seems Pekka and I did an entirely unintentional tag-team, as this kind
of high-level goal summary was exactly what I had in mind ...

On 6 May 2015 at 12:57, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> On Tue, 05 May 2015 15:54:34 -0700
> "Jon A. Cruz" <jonc at osg.samsung.com> wrote:
>> On 05/05/2015 09:40 AM, Jon A. Cruz wrote:
>> > I'll move forward with merging things together. At the moment there is
>> > some general testing work that others have outstanding, so I'll hold off
>> > on modifying the existing files. Most likely some will just take
>> > migrating appropriate functionality to the separated files.
>> >
>> > One thing I have pending on docs are quick diagrams of the current code
>> > organization and a proposed end state. Those should be sent out soon for
>> > additional info and review.
>>
>> I've submitted a slightly updated patch that mainly adds the mentioned
>> docs. To help avoid the need to run doxygen to generate them I'm
>> attaching the diagrams directly here.
>
> Hi Jon,
>
> nice pics!
>
> My take-away with these is that you are separating weston-specific
> tests from compositor-agnostic tests. I think that is excellent.

Yes!

> So far you have been looking at:
> - unit tests 'ish (vertex-clip-test.c)
> - generic client tests (subsurface-test.c)
> - weston-specific client tests (devices-test.c)
>
> Unit tests are just plain executables. Client tests require Weston to
> be running with desktop-shell.
>
> If you look at tests/weston-tests-env script, you see more test types
> not mentioned above:
> - weston plugin tests (surface-global-test.c)
> - client tests requiring ivi-shell (ivi-shell-app-test.c)
> - weston plugin tests requiring ivi-shell (ivi_layout-internal-test.c)
> - weston plugin tests requiring ivi-shell and a client
>   (ivi_layout-test.c + ivi_layout-test-plugin.c)
>
> So we have lots of different types of tests, each requiring a specific
> runtime setup: run weston or not, which shell, is it a plugin or a
> client, does the client need weston test extension...
>
> Getting all these organized somehow would be cool. What do I mean by
> that? I'm not sure. The current system just feels quite messy. Bryce
> was trying to bring some order in weston-tests-env, but I think the
> attempt I last looked went slightly overboard from another side.

What I would like to see in terms of organisation is a set of simple C
helpers that allows tests to set these environments up very easily:
start a compositor instance with a certain set of modules or config
fragments, spawn a test client (perhaps in a separate thread), etc.
And no shell scripts required: the harder we make it to gdb something,
the more we're just shooting ourselves in the foot.

> This is not a wish for some sort of new "test specification language"
> with which you can express oceans and the Moon, but a call for some
> better structure. ;-)

Agreed.

> So, maybe take this as more like a note, that we have many kinds of
> tests with different runtimes.
>
> I do have one wish: making it easy to run a test in gdb. That has
> problems like choosing which process you want to attach gdb to, since
> usually there are at least the compositor and the test client. Also,
> how do you get gdb in there; start the process in gdb, or maybe add a
> raise(SIGSTOP) on some conditions so gdb can attach "from outside".
>
> That's just a brain dump of the moment. :-)

And this. But I think that comes with more helpers, not more
copy-and-wasted shell scripts.

Here's a few things that I would really like to see, which is partly
coloured by having worked quite a bit with intel-gpu-tools recently.

Self-contained executables: as above, I want to kill off the shell
scripts to the extent that we can (perhaps one shell fragment you can
source to set up to get paths).

Differentation between failures and unmet requirements: if we want to
test something which isn't available on the target system, then that's
a prerequisite failure, not a failure of the test itself.

Better output: just a pass/fail/skip by default with no other
messages, with (optionally?) showing all output on fail/skip, but
nothing at all on pass. See Bryce's 9/10 for what I mean: seeing that
the buffer was successfully allocated is useless when the test passes,
but helpful when you're trying to debug a failure. So this implies
some kind of buffering in the core, but also infrastructure to handle
signals and assertion failures which will dump all buffered output, a
la igt. Machine-readable is a must here, in order to support ...

A real test runner/harness: run all tests by default, but allow
inclusion/exclusion of specific patterns, print the list of available
tests, print a concise summary, allow comparison of runs to runs.
Piglit does quite well here, with its output optionally taking a
previous run to compare to, showing you the current status
(pass/fail/skip/crash), as well as differences between runs (newly
passing, regressed to fail, etc).

As little duplication as possible: no copy-and-wasted shell scripts,
no having to put every test in 4 different lists.

Same implementation language for tests: C.

No onerous dependencies for the harness: Python 2 is available
basically everywhere. Python 3 is getting there. Perl is usually
available, but I can't tell you how much I dislike it. C also works.
More exotic languages ... no thanks. The more difficult we make it to
run on targets, the less it will get run.

Coherence with test frameworks around us: people testing
Wayland/Weston are likely to be testing DRM/KMS, (E)GL(ES), GStreamer,
and the various toolkits. The closer we are to what these projects
implement - especially in terms of compatibility with their automated
run output - the more impact our framework will have.

I posit that igt + Piglit is a really good starting point for this.
Others might disagree. Honestly I don't care too much, as long as we
get something which not only meets these requirements, but also isn't
onerous to maintain. XTS is a perfect example of what happens when you
build up a huge infrastructure that ends up getting abandoned and
becoming a net cost, as the cost of maintaining it outweighs the
benefits the tests bring.

It's blindingly obvious from our discussions on IRC and the style
review I did earlier that we have very differing views on how testing
should work. But what we do agree on is that testing (all of unit,
integration and functional) is very important, and that without a
great infrastructure to support this, we're shooting ourselves in the
foot. I think the best way to go about this is to work with our
existing tests - including Bryce's headless work - and piece-by-piece
build up a framework which supports that. My main worry right now is
that by designing a grand framework upfront and moulding the tests to
fit that, that we might end up with a bit of a mismatch between that.
But ultimately, as long as we get a framework which makes it as easy
to write tests as possible, and can eventually fulfill all these
goals, then I'm sure we'll all be happy.

Thanks for your work on these. Looking forward to see how it all develops.

Cheers,
Daniel