Litmus, a proposal

[Bjoern Michaelsen]

Hi all,

when the topic of getting testcases for LibreOffice into Ubuntus Checkbox, I
finally got myself a Litmus account. I defered that before as "yet another
login" I did not want to use. Having a look at it now, I find it quite simple
and useful -- it is very well suited to be the tool to get our job done -- if
we fix some issues with it.

The first and most important one is: it is behind its own login -- and that
prevents the casual bypasser to get involved and interested. IMHO, we
absolutely need to change that. The testcases should be browsable without a
login to get people interested and see what to expect. However, unfortunately
for people to report back we will need logins, but not yet another one. Litmus
should support Open-ID so people can reuse their existing logins. Askbot showed
us how important this lowering of barrier might be. Maybe we could work
together with the Moziila folks on this? Maybe we could even throw a limited
amount of dedicated funds in that direction?

Next thing is to get testcases in. Thinking about this, I guess, the people
best suited to sketch up a initial draft of a testcase are the developers
(these drafts actually can be oneliners). They know best the features and
bugfixes they are working on and can hint testers to torture that area.
Therefore I would propose to create two additional "test groups" in Litmus:
"release features 3.6" and "bugfixes 3.5".

== release features 3.6 ==

The release features testcases are about new features introduced with a new
major release. It will be the task of the QA team to skim thorugh the commit
logs on master to identify devs working on features to kindly remind them of
the possibility to write testcases for these, so that we actually test and QA
with focus on the areas we touched in the release. In the end this is in the interest of the developer too, as:

a) he gets his feature tested
b) gets visibility to other groups besides QA too: l10n, UX etc.
c) he gets his feature tested _early_ and isnt overwhelmed with a plethora
   of complains late at beta release
d) we can manage expectations better: if the testers expected something
   different from the feature description than they are getting, we can
   mitigate disappointments for the endusers by better describing the feature

== bugfixes 3.5 ==

Once we split off the release branch, we are already very careful with the
changes to the code. We require one sign-offs for a 3.X branch and three
sign-offs for a 3.X.Y branch.

To communicate what is going on on the release branch I suggest that requests
for changes to be cherry-picked for the release branch to be required contain a
broad testcase for the code area touched in words that is parsable by the
common enduser (e.g. "please test copy-paste in Calc"). If the fix has no
suitable enduser testing (e.g. a buildbreaker fix) it should state just that
quickly, but explicitly: 

 "No testcase for this request as this change is not enduser visible."

Creating such draft testcase hints shouldnt put too much extra load on
development, but give QA and testers valuable starting points to pick up. Of
course, it would be even more awesome if devs write not only a draft hint, but
a full testcase. Thats will never be a requirement, but it might actually be in
the interest of the developer.

Once the next major comes around, we can recycle the testcases which are
generic enough and keep using them.

tl;dr:

I am proposing:
- We need anonymous access to Litmus ASAP
- We need Open-ID logins to Litmus ASAP
- there is a simple way to request testing for devs: write a draft test case
- QA should kindly nag feature developers into drafting testcases along the way
- release branch commits should come with a draft testcase describing the
  affected area by default

Im sure everybody hates me now. ;) Im still looking forward for your opinions!

Best,

Bjoern