How to test Weston gfx (Re: Weston test suite)

Tue Sep 1 00:13:12 PDT 2015

On Mon, 31 Aug 2015 13:47:47 -0500
Derek Foreman <derekf at osg.samsung.com> wrote:

> On 31/08/15 03:58 AM, Pekka Paalanen wrote:
> > On Tue, 25 Aug 2015 10:43:22 +0300 Pekka Paalanen 
> > <ppaalanen at gmail.com> wrote:
> > 
> >> On Tue, 25 Aug 2015 10:16:17 +0300 Pekka Paalanen 
> >> <ppaalanen at gmail.com> wrote:
> >> 
> >>> Btw. I CC'd you on 
> >>> https://bugs.freedesktop.org/show_bug.cgi?id=83985 where we 
> >>> have been talking about testing things. I wonder how you'd 
> >>> handle some tests being specific to a backend, and some tests 
> >>> being generic such that a user (the person doing 'make check') 
> >>> could run it against any backend he chooses? If we need such 
> >>> things... perhaps backend-specificity comes mostly from 
> >>> configuration and options, so maybe we might be able to unify 
> >>> the options so much that the differences don't matter.
> >>> 
> >>> And then screenshot-based tests are likely something we want
> >>> to run once for each possible renderer for a chosen backend.
> >>> 
> >>> 
> >>> Hmm... I just got an idea.
> >>> 
> >>> We've been adding renderer support to headless-backend.so so 
> >>> that we could run screenshot-based tests headless. What if... 
> >>> we kept headless backend without real renderers (only mocked 
> >>> renderer, one capable of initialising 
> >>> EGL_WL_bind_wayland_display though), ran Weston/wayland on top 
> >>> of Weston/headless, and test clients on the Weston/wayland 
> >>> instance?
> >>> 
> >>> It might require some work on the wayland-backend, but it
> >>> would also leave all testing related mocks and hacks into the 
> >>> headless backend code, without leaking it into gl-renderer or 
> >>> such. And it would stress the wayland-backend which I imagine 
> >>> could use more use.
> >>> 
> >>> Well, that's just a wild idea, I'm not sure what it would 
> >>> entail. What do you all think?
> >> 
> >> Wait... using Weston/headless to host Weston/wayland would 
> >> immediately enable software GL rendering, wouldn't it? The EGL 
> >> Wayland platform in Mesa today supports software-GL on wl_shm. 
> >> This means we should be able to test Weston's gl-renderer on 
> >> llvmpipe by just running it with the wayland-backend on another 
> >> Weston instace.
> > 
> > Hi all,
> > 
> > I think this needs a bit of discussion. Should we be adding more 
> > and more renderer features into the headless backend, or should we 
> > move to running Weston/wayland on top of Weston/headless for 'make 
> > check'?
> > 
> > It is a blocker question for things like 
> > https://bugs.freedesktop.org/show_bug.cgi?id=83985 which Dawid is 
> > working on.
> > 
> > Derek, I recall you had plans to control Weston's repaint loop from
> > tests. How would that be affected if we run nested instead of 
> > directly on headless?
> 
> Right, I'll try to get a new revision of those patches out shortly
> after the release...
> 
> I can't immediately think of a reason why nesting would cause
> problems.  I think it should be ok.
> 
> > The reason why I like the nested approach now is that while it 
> > allows immediately to run things with llvmpipe, we would not need 
> > to modify gl-renderer to e.g. render into an FBO just for testing. 
> > We could have the headless backend even initialize a render node 
> > and support EGL_WL_bind_wayland_display. Code only related to 
> > testing would be limited into the headless backend with its 
> > built-in mock-renderer.
> 
> Well, I think the clock control stuff would have to take place in the
> nested instance, so there's a limit to how much we can
> compartmentalize the test stuff to the headless backend?

That's likely true. Also, we do not have to run *everything* with the
nested arrangement, we should be able to choose the arrangement per
test.

Everything that particularly tests Weston's rendering features would
benefit from the nested arrangement, but other things might be better
without, if only for test execution speed.

> Screenshots would be taken on the headless backend though - the test
> clients would have to connect to both the nested and the parent
> compositor?

No, I don't think so. The whole point is that headless would not need a
functioning renderer at all. Therefore screenshots would be taken on
the nested Weston/wayland instance, precisely because it has the fully
functional renderers. In this case we're not really interested in what
pixels the headless receives, but what pixels the nested paints.

Yeah, this means that the work already done to add real renderers to
headless would be thrown away or at least unused.

Ideally test clients would not need to connect to the headless instance
at all. Headless would only function as a platform for Weston/wayland
to run on, similar how Weston/x11 uses the X server or Weston/DRM uses
DRM/KMS/GBM. I would like to limit the scope of the nested approach to
things we can test by not connecting test clients to the headless
instance.

Testing the wayland-backend specifically would be another matter.

> > What do you think?
> > 
> > Hmm, and we could pull in fullscreen-shell into the mix and use it 
> > on headless. The wayland backend seems to already have code for
> > it. This would avoid running weston-desktop-shell or the VKB app
> > for the headless instance.
> 
> I guess if I was trying very hard to find a reason to dislike this it
> would be that changes to fullscreen-shell could break all sorts of
> tests that don't intentionally test fullscreen-shell...

I could argue that to be a good thing, because currently we have zero
test coverage on fullscreen-shell. :-) Which is even less than we have
on the wayland-backend, because at least in theory you can run 'make
check BACKEND=wayland-backend.so', though I suspect no-one really does.

But, if we are testing the nested compositor, I can't really imagine
what problems in fullscreen-shell could break random tests, aside from
everything failing at the same time.

> Really though, I think it seems like a reasonable idea.  And I think
> fullscreen-shell is fairly mature.
> 
> Having gl-renderer not have to worry about whether it's in "test mode"
> or not sounds like the lesser of two evils.

Ok, cool.

Anyone else have an opinion?

Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 811 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freedesktop.org/archives/wayland-devel/attachments/20150901/e3d3b2e3/attachment.sig>