Weston testing (Re: New to wayland, need help pls.)

[Srivardhan M S]

Pardon me for the delayed response. I was on a vacation.

Thank you very much for the reply. As I told I am new to Wayland (indeed
new to Display server). I am going through the documentation of the Wayland
which is available in the website.

My understanding of Wayland is very little, so I won't be able to develop
the test suite myself. I will now start going through the code and logs to
understand how Wayland actually works. If any of you can guide me
initially, I would be happy to develop.

Thank-you,
Sri


On Thu, Apr 17, 2014 at 12:23 PM, Pekka Paalanen <ppaalanen at gmail.com>wrote:

> On Wed, 16 Apr 2014 20:41:50 +0000
> "Bryce W. Harrington" <b.harrington at samsung.com> wrote:
>
> > On Wed, Apr 16, 2014 at 08:51:21AM +0300, Pekka Paalanen wrote:
> > > On Tue, 15 Apr 2014 19:08:12 +0000
> > > "Bryce W. Harrington" <b.harrington at samsung.com> wrote:
> > >
> > > > Maybe adding more tests?  Any particular features recently added that
> > > > could benefit from tests added?
> > >
> > > Yes! That is a good one. Either adding completely new tests, or maybe
> > > working with Marek Chalupa in converting all possible FAIL_TEST cases
> to
> > > normal TEST cases, as permitted by Marek's
> > >
> http://lists.freedesktop.org/archives/wayland-devel/2014-April/014190.html
> > >
> > > I can't list any features off-hand that would need new tests which
> would
> > > be easy to implement.
> >
> > Would there be value to just basic unit tests on some of the established
> > API's?  E.g. like take xdg-shell's API, check passing in negative or
> > zero dimentions, crazy long strings for string params, stuff like that.
> > A lot of the protocol's code is generated so it's unclear to me whether
> > this level of testing would be useful or not, but if it is, it tends to
> > be pretty newbie-friendly.
>
> Even when the wrappers are generated, argument validity checks are
> written manually, so yes, there would be value in writing those.
>
> Another easy bit is testing object destruction orders for somehow
> related objects. Like if you create a wl_viewport for wl_surface, and
> then the wl_surface is destroyed, the wl_viewport should be inert and
> using it should only be ignored in the compositor, not cause crashes or
> protocol errors. "Inert" is a specific term in the specification
> language for exactly that.
>
> The sub-surface tests do extensive destroy order testing, but I think
> it might lack testing the "ignore" part, and I'm sure there would be
> other places to add more such tests.
>
> Also, every defined protocol error in the protocol specifications should
> be excercised by a test, that should be easy, too.
>
> However, I'm not sure if xdg_shell particularly is a good target at the
> moment, as I think there might still be changes coming?
>
> > > But I guess we should at some point hook up a
> > > real renderer to the headless backend, and add infrastructure for the
> > > test framework to do screencaptures, so we could test also rendering
> > > and things like window stacking and surface/buffer transformations.
> >
> > That sounds keen, and could open up a lot of testing potential.  Shame
> > it's not already in place!  Could you elaborate on how you think this
> > could be created?  I'd like to explore this further myself.
>
> Well, there is the test protocol extension, you could add
> client-initiated screenshooting feature to it, perhaps on a designated
> sub-rectangle.
>
> Easiest for headless would probably be to start by plugging in the
> pixman renderer, and defining a hardcoded single-resolution output.
>
> Since tests also run the desktop-shell client which draws "random"
> things like takes colors from configs and paints the clock, maybe it
> could on init check if the test extension is advertised by the
> compositor. If the test extension is there, desktop-shell client would
> use only hardcoded configs, and paint a hardcoded time in the clock,
> etc. so that the rendered output should always be the same.
>
> Window initial position randomization in shell.c might be a problem.
>
> You'd also need to save the screenshots into image files, so a
> developer can check the result.
>
> > > Pixman's fuzzer test might be a good inspiration on how to actually do
> > > the tests once the infrastructure is up:
> > > http://lists.freedesktop.org/archives/pixman/2014-April/003229.html
> > > Provided we can guarantee pixel-perfect rendering, which might not
> > > always be true, so a checksum-based checking might not always work.
> >
> > I take it this is similar to the Cairo testsuite's perceptual diff
> > stuff?  (Checksumming two bitmaps pixel by pixel)
>
> I suppose, though I haven't looked at Cairo.
>
> > What we're seeing in Cairo is that sometimes pixel differences indicate
> > issues, but a lot of time it's ignorable, and deciding which is which is
> > fairly hard.  But used in situations where you are 100% sure exactly
> > what the images should look like, this can be quite effective.
>
> Yes, Pixman does the same separation. Some tests require bit-perfect
> results, others are known to have a little variance.
>
> I suppose with a Pixman renderer, we can have the guarantees Pixman
> has since we know which operations we use (or do we?), but with a
> gl-renderer, or say, rpi-renderer, I don't think we can require
> bit-perfect reproduction.
>
> We can already choose the backend to use when running 'make check', we
> could have the headless backend expose pixman renderer by default and
> software rendered gl-renderer as an option.
>
> > In practice, just having all the different rendering tests can be a
> > handy workload for exposing crashes.  I suspect coupling it with some
> > timing measurements it might also help expose performance regressions.
>
> I don't think we can make automated performance testing too reliable. I
> wouldn't want a build-bot for some distribution to fail 'make check'
> just because it runs in a virtual machine on some heavily loaded
> hardware.
>
> So performance testing would be a separate case.
>
> > > But I think that is getting quite heavy for just introductory work.
> >
> > For something like this, 80% of the work is just deciding on a good
> > design.  Most of the implementation work is going to be basic coding.
> > There's going to be a few hard tasks (like attaching a snapshot renderer
> > functionality to the headless backend), but even those are likely to get
> > owners if they're well enough described.
>
> Right. We need someone to put in the infrastructure. :-)
>
>
> Thanks,
> pq
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/wayland-devel/attachments/20140421/dbebf441/attachment-0001.html>