Weston test suite

Tue Aug 25 00:16:17 PDT 2015

On Mon, 24 Aug 2015 17:04:50 -0700
"Jon A. Cruz" <jonc at osg.samsung.com> wrote:

> On 08/24/2015 02:50 AM, Pekka Paalanen wrote:
> > On Sat, 22 Aug 2015 00:00:58 -0700
> > Bryce Harrington <bryce at osg.samsung.com> wrote:
> > 
> >> On Fri, Aug 21, 2015 at 10:35:34AM +0300, Pekka Paalanen wrote:
> >>> On Thu, 20 Aug 2015 11:05:39 -0700
> >>> Bryce Harrington <bryce at osg.samsung.com> wrote:
> >>>
> > 
> >>>> (I wish we had some tests with coverage of this chunk of code, so I could
> >>>> give my usual plug for adding test cases; if we did, it'd probably
> >>>> pretty easy to add a couple test cases here and give us a bit extra
> >>>> confidence.)
> >>>
> >>> It's a bit tough in this case, I'm not sure if we can affect what
> >>> outputs the kernel exposes, so it'd be tied to the test machine.
> >>
> >> No, the way you'd do it is to construct a "mock kernel", and then
> >> whereever the functionality expects data from the kernel (or is making
> >> kernel calls), you abstract that so you can slip your mock in instead.
> >> Sort of the same idea as how we replaced the graphics card / monitor
> >> with synthetic ones in the headless testing.
> >>
> >> I know it probably sounds like a pain in the ass (and it is), but if
> >> you're serious about wanting stronger automated testing of Weston, then
> >> that's the sort of stuff that needs done.  And it's certainly
> >> understandable to want to put off spending time writing tests, but it's
> >> the same thing with documentation and look how horribly undocumented
> >> Wayland is - unless you document and write tests as you go, it ain't
> >> gonna get done.
> > 
> > Hi,
> > 
> > it's actually good to bring this up right now. We have the old test
> > suite which is painful to work with, and we have the new test suite
> > which is seriously lacking to become on par even with the old suite in
> > terms of helpers.
> > 
> > Jon, what are your plans in continuing with the test suite?
> > 
> > I haven't seen a new patch series in quite a while, and I don't see any
> > pending in Patchwork either.
> 
> 
> Yes,
> 
> The other project I work on just finished pushing out their latest
> version, so Wayland was getting refocused for my attention.
> 
> Patches for both base Wayland testing and explicit Weston testing
> fixtures should be coming once things are updated to apply to the latest
> code.

Hi Jon,

nice to hear!

I can't promise landing them before 1.9.90, but I'm definitely
interested in getting them in sooner than later.

> Also, if we are talking about actual mocking actual system calls, that
> can be done but does take a little thought.

Let's do our immediate targets first: getting the new testing framework
up to par and superior to the old one and convert all old tests, and
eventually nuke the old framework in Weston.

> If we can pinpoint a few target cases for testing to cover, then getting
> the needed support in place can flow a lot quicker. At many other groups
> I've been the main person to deliver fixtures and mocks to allow other
> people to write tests against them. Ideally we do the heavy lifting once
> and then allow individual developers to easily test a much wider range
> of functionality and scenarios.

I fear that mocking KMS ioctls is going to be a rabbit hole. At least
it needs very careful planning on what we do not want to mock. For
instance, we likely do not want to mock any driver-specific ioctls,
which means that our related tests cannot do anything that touches
driver-specific ioctls. Driver-specific ioctls are usually called from
within Mesa drivers, which means through GBM, EGL and GL APIs. To avoid
driver-specific ioctls while using EGL and GL, we must essentially use
software rendering. Whether that actually is available depends on Mesa,
so there might be some work to be done in Mesa as a pre-requisite. Oh,
there is of course the DRM + Pixman-renderer option in Weston, that
would be likely the best place to start looking at it.

I suspect that planning the kernel mocking now would push the immediate
targets mentioned above too far into the future.

I'd consider adding e.g. screenshot-based tests a lot more important
than those interactions with the kernel that we could mock. We hope to
have testing running on real hardware for the kernel stuff where mocking
is not feasible.

> Pragmatically, If I don't make testing easy, then I end up the only one
> writing tests. Don't want that case :-)

Indeed.

Btw. I CC'd you on https://bugs.freedesktop.org/show_bug.cgi?id=83985
where we have been talking about testing things. I wonder how you'd
handle some tests being specific to a backend, and some tests being
generic such that a user (the person doing 'make check') could run it
against any backend he chooses? If we need such things... perhaps
backend-specificity comes mostly from configuration and options, so
maybe we might be able to unify the options so much that the
differences don't matter.

And then screenshot-based tests are likely something we want to run
once for each possible renderer for a chosen backend.

Hmm... I just got an idea.

We've been adding renderer support to headless-backend.so so that we
could run screenshot-based tests headless. What if... we kept headless
backend without real renderers (only mocked renderer, one capable of
initialising EGL_WL_bind_wayland_display though), ran Weston/wayland on
top of Weston/headless, and test clients on the Weston/wayland instance?

It might require some work on the wayland-backend, but it would also
leave all testing related mocks and hacks into the headless backend
code, without leaking it into gl-renderer or such. And it would stress
the wayland-backend which I imagine could use more use.

Well, that's just a wild idea, I'm not sure what it would entail. What
do you all think?

Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 811 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freedesktop.org/archives/wayland-devel/attachments/20150825/7602d0d1/attachment.sig>