[Libreoffice-qa] QA ideas for 3.5 WAS Re: [Libreoffice] Improving the QA and Release for 3.5

[Petr Mladek]

Cor Nouws píše v Út 29. 11. 2011 v 15:34 +0100:
> Petr Mladek wrote (29-11-11 11:54)
> > Rimas implemented a nice feature yesterday. We are now able to mark some
> > test cases as "language" and/or "OS" dependent. This option affects when
> > the test will be marked as "done" in test run. This way we could get
> > easier structure and optimize the testing efforts.
> 
> Ah cool. That reads to me as a feature that will prevent more people 
> doing the same test without good reason.

Exactly.

> > Another problem is that we still have only few testcases. We need to
> > create more. We still could do it during the test phase.
> >
> > Any helping hands are welcome.
> 
> I saw your note in an earlier mail yes. But did not really think about 
> it, sorry.
> Is it a good idea if for the next 7/8 weeks we focus on as much testing 
> as possible and that we after that try to encourage people to help with 
> writing tests cases?

I would encourage people to do both. Testing is important. On the other
hand, the written tests help to avoid duplication. Also they are
inspiration for people that have less experience. Note most of the test
make sense to duplicate on Windows, Linux, and MAC.

I would write it the way that we are looking for testers. Any test make
sense. We want to do the existing tests from Litmus. They are also
inspiration for people that are in doubts. On the other hand, if anyone
does an interesting test that is missing in Litmus, we kindly ask them
to document it there. The description need not be perfect, we could
always improve it later.

I think that people do not need to do this only during the bug-hunting
session. They could do it at any time.

We should mention links to the documentation, .e.g.
http://wiki.documentfoundation.org/QA/Testing/Regression_Tests
http://wiki.documentfoundation.org/QA/Testing/Test_Case
http://wiki.documentfoundation.org/QA/Testing/Test_Cases_Contribution


> >> Cor Nouws wrote (19-11-11 16:56)
> >> - With the first beta, two day's organised bug-hunting:
> >> - Friday 9 and Saturday 10 December
> >> - the beta-0, makes sure that reliable builds are there for all
> >> - mail all people on l10n lists and ask to spread the news
> >> - mail to discuss&  users
> >> - people available on irc for guidance
> >
> > It sounds good. Rainer is just afraid that we do not get enough people
> > attention. He tried to do such events in the past and nobody appeared.
> > Well, I am not sure how Rainer announced it. You might know better
> > channels.
> 
> I share his experience from years past that it is not easy.
> On the other hand:
> - all we get on testing early is important;
> - we might help l10n people/groups to present it. Announcing the first 
> beta of the next great LibreOffice (-> features..) and a bug hunt 
> session, sure will get some media attention. Which is good for the 
> overall project and good for the l10n teams...
> Of course, needs some preparation, but should not be too difficult.
> I have some hours available this and next week to help.

Sounds good to me.

> >> - during the months, a two day's competition:
> >> - Saturday 14 and Sunday 15 January
> >> - a prize for the three that post most bug-weight in those two day's
> >> - bug weight: points/issue:
> >> 1 for normal, 1,5 for major, 2,5 for every blocker issue
> >> - severity needs to be confirmed by QA specialists (in max one week)
> >> - prizes are e.g. sweater and mug.
> >> - QA specialists cannot win a prize
> >> (they should just get the sweater and mug without that)
> >
> > I am not sure about this action. I am afraid that we could get too many
> > bug reports and it will be hard to sort them, ... Note that we currently
> > have only very few people doing the bug triage.
> 
> That is a possible trap, yes.
> Maybe there are some simple way's to make it better (will think about 
> that later, but of course, hints welcome..)
> 
> So best is the we start with the first step, and in the mean while think 
> about if a competition can be managed in a pretty fair way.

Yup, I like this step by step approach.

> > Another problem is who would judge the results :-)
> 
> I thought you would like to volunteer ;-)

Please not! It would be my nightmare. I would be stressed that I would
not be objective :-)

> > Another question is how to instruct people during the bug-hunting
> > session. We do not have enough test cases. One possibility would be to
> > ask them to create them as described at
> > http://wiki.documentfoundation.org/QA/Testing/Test_Case
> > Otherwise, it would end in a chaotic testing. Well, it would be better
> > than nothing.
> 
> Chaotic tests give great results - if people can write down the results 
> in a structure way of course ;-)
> I can draft a page to get just the people in the bug-hunt sessions on 
> the right track.

That would be great if you could draft something.

Best Regards,
Petr