<div dir="ltr">Pardon me for the delayed response. I was on a vacation.<div> </div><div>Thank you very much for the reply. As I told I am new to Wayland (indeed new to Display server). I am going through the documentation of the Wayland which is available in the website. </div> <div> </div><div>My understanding of Wayland is very little, so I won't be able to develop the test suite myself. I will now start going through the code and logs to understand how Wayland actually works. If any of you can guide me initially, I would be happy to develop.</div> <div> </div><div>Thank-you,</div><div>Sri</div></div><div class="gmail_extra"> <div class="gmail_quote">On Thu, Apr 17, 2014 at 12:23 PM, Pekka Paalanen <<a href="mailto:ppaalanen@gmail.com" target="_blank">ppaalanen@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Wed, 16 Apr 2014 20:41:50 +0000 "Bryce W. Harrington" <<a href="mailto:b.harrington@samsung.com">b.harrington@samsung.com</a>> wrote: > On Wed, Apr 16, 2014 at 08:51:21AM +0300, Pekka Paalanen wrote: > > On Tue, 15 Apr 2014 19:08:12 +0000 > > "Bryce W. Harrington" <<a href="mailto:b.harrington@samsung.com">b.harrington@samsung.com</a>> wrote: > > > > > Maybe adding more tests? Any particular features recently added that > > > could benefit from tests added? > > > > Yes! That is a good one. Either adding completely new tests, or maybe > > working with Marek Chalupa in converting all possible FAIL_TEST cases to > > normal TEST cases, as permitted by Marek's > > <a href="http://lists.freedesktop.org/archives/wayland-devel/2014-April/014190.html" target="_blank">http://lists.freedesktop.org/archives/wayland-devel/2014-April/014190.html</a> > > > > I can't list any features off-hand that would need new tests which would > > be easy to implement. > > Would there be value to just basic unit tests on some of the established > API's? E.g. like take xdg-shell's API, check passing in negative or > zero dimentions, crazy long strings for string params, stuff like that. > A lot of the protocol's code is generated so it's unclear to me whether > this level of testing would be useful or not, but if it is, it tends to > be pretty newbie-friendly. Even when the wrappers are generated, argument validity checks are written manually, so yes, there would be value in writing those. Another easy bit is testing object destruction orders for somehow related objects. Like if you create a wl_viewport for wl_surface, and then the wl_surface is destroyed, the wl_viewport should be inert and using it should only be ignored in the compositor, not cause crashes or protocol errors. "Inert" is a specific term in the specification language for exactly that. The sub-surface tests do extensive destroy order testing, but I think it might lack testing the "ignore" part, and I'm sure there would be other places to add more such tests. Also, every defined protocol error in the protocol specifications should be excercised by a test, that should be easy, too. However, I'm not sure if xdg_shell particularly is a good target at the moment, as I think there might still be changes coming? > > But I guess we should at some point hook up a > > real renderer to the headless backend, and add infrastructure for the > > test framework to do screencaptures, so we could test also rendering > > and things like window stacking and surface/buffer transformations. > > That sounds keen, and could open up a lot of testing potential. Shame > it's not already in place! Could you elaborate on how you think this > could be created? I'd like to explore this further myself. Well, there is the test protocol extension, you could add client-initiated screenshooting feature to it, perhaps on a designated sub-rectangle. Easiest for headless would probably be to start by plugging in the pixman renderer, and defining a hardcoded single-resolution output. Since tests also run the desktop-shell client which draws "random" things like takes colors from configs and paints the clock, maybe it could on init check if the test extension is advertised by the compositor. If the test extension is there, desktop-shell client would use only hardcoded configs, and paint a hardcoded time in the clock, etc. so that the rendered output should always be the same. Window initial position randomization in shell.c might be a problem. You'd also need to save the screenshots into image files, so a developer can check the result. > > Pixman's fuzzer test might be a good inspiration on how to actually do > > the tests once the infrastructure is up: > > <a href="http://lists.freedesktop.org/archives/pixman/2014-April/003229.html" target="_blank">http://lists.freedesktop.org/archives/pixman/2014-April/003229.html</a> > > Provided we can guarantee pixel-perfect rendering, which might not > > always be true, so a checksum-based checking might not always work. > > I take it this is similar to the Cairo testsuite's perceptual diff > stuff? (Checksumming two bitmaps pixel by pixel) I suppose, though I haven't looked at Cairo. > What we're seeing in Cairo is that sometimes pixel differences indicate > issues, but a lot of time it's ignorable, and deciding which is which is > fairly hard. But used in situations where you are 100% sure exactly > what the images should look like, this can be quite effective. Yes, Pixman does the same separation. Some tests require bit-perfect results, others are known to have a little variance. I suppose with a Pixman renderer, we can have the guarantees Pixman has since we know which operations we use (or do we?), but with a gl-renderer, or say, rpi-renderer, I don't think we can require bit-perfect reproduction. We can already choose the backend to use when running 'make check', we could have the headless backend expose pixman renderer by default and software rendered gl-renderer as an option. > In practice, just having all the different rendering tests can be a > handy workload for exposing crashes. I suspect coupling it with some > timing measurements it might also help expose performance regressions. I don't think we can make automated performance testing too reliable. I wouldn't want a build-bot for some distribution to fail 'make check' just because it runs in a virtual machine on some heavily loaded hardware. So performance testing would be a separate case. > > But I think that is getting quite heavy for just introductory work. > > For something like this, 80% of the work is just deciding on a good > design. Most of the implementation work is going to be basic coding. > There's going to be a few hard tasks (like attaching a snapshot renderer > functionality to the headless backend), but even those are likely to get > owners if they're well enough described. Right. We need someone to put in the infrastructure. :-) Thanks, pq </blockquote></div> </div>