[Libreoffice-qa] QA ideas for 3.5 WAS Re: [Libreoffice] Improving the QA and Release for 3.5

[Cor Nouws]

Hi Petr,

Petr Mladek wrote (29-11-11 11:54)

> I have added QA and devel mailing lists into CC. We should not cook such
> an important event in background.

Agreed - that is why I started at the list initially of course ;-)

> Cor Nouws píše v Út 29. 11. 2011 v 00:06 +0100:

>> Is it OK for you if I proceed more or less as outlined below?
>> I have some additional remarks that I'll take into account.
>> I think I'll explain clearly to the people the way to Litmus for if they
>> want to do the structural tests.
>> I saw some work on changed naming. Is that already in place?
>
> Heh, we are going to do one more restructuralization in Litmus.
>
> Rimas implemented a nice feature yesterday. We are now able to mark some
> test cases as "language" and/or "OS" dependent. This option affects when
> the test will be marked as "done" in test run. This way we could get
> easier structure and optimize the testing efforts.

Ah cool. That reads to me as a feature that will prevent more people 
doing the same test without good reason.

> Yi Fan is creating documentation at
> http://wiki.documentfoundation.org/QA/Testing/Test_Cases_Organization
> Also he is going to update the existing test cases accordingly. He wants
> to finish this until Dec 6 or so.
>
> Another problem is that we still have only few testcases. We need to
> create more. We still could do it during the test phase.
>
> Any helping hands are welcome.

I saw your note in an earlier mail yes. But did not really think about 
it, sorry.
Is it a good idea if for the next 7/8 weeks we focus on as much testing 
as possible and that we after that try to encourage people to help with 
writing tests cases?

>> Cor Nouws wrote (19-11-11 16:56)
>> Goal is (as discussed before) get more people's focus on early testing,
>> rather then with the first RC, and also more people testing :-)
>>
>> I have draft the following schedule with special events:
>>
>> - With the first beta, two day's organised bug-hunting:
>> - Friday 9 and Saturday 10 December
>> - the beta-0, makes sure that reliable builds are there for all
>> - mail all people on l10n lists and ask to spread the news
>> - mail to discuss&  users
>> - people available on irc for guidance
>
> It sounds good. Rainer is just afraid that we do not get enough people
> attention. He tried to do such events in the past and nobody appeared.
> Well, I am not sure how Rainer announced it. You might know better
> channels.

I share his experience from years past that it is not easy.
On the other hand:
- all we get on testing early is important;
- we might help l10n people/groups to present it. Announcing the first 
beta of the next great LibreOffice (-> features..) and a bug hunt 
session, sure will get some media attention. Which is good for the 
overall project and good for the l10n teams...
Of course, needs some preparation, but should not be too difficult.
I have some hours available this and next week to help.

>> - during the months, a two day's competition:
>> - Saturday 14 and Sunday 15 January
>> - a prize for the three that post most bug-weight in those two day's
>> - bug weight: points/issue:
>> 1 for normal, 1,5 for major, 2,5 for every blocker issue
>> - severity needs to be confirmed by QA specialists (in max one week)
>> - prizes are e.g. sweater and mug.
>> - QA specialists cannot win a prize
>> (they should just get the sweater and mug without that)
>
> I am not sure about this action. I am afraid that we could get too many
> bug reports and it will be hard to sort them, ... Note that we currently
> have only very few people doing the bug triage.

That is a possible trap, yes.
Maybe there are some simple way's to make it better (will think about 
that later, but of course, hints welcome..)

So best is the we start with the first step, and in the mean while think 
about if a competition can be managed in a pretty fair way.

> Another problem is who would judge the results :-)

I thought you would like to volunteer ;-)

>> - with the first RC, two day's organised bug-hunting
>> - Friday 20 and Saturday 21 January
>> - ... (see above for more)
>
> Same as for the first bug hunting session.

Same for me ;-)

> When I think about it, we have a chicken and egg problem. I am afraid
> that we do not have enough active QA people that would help to organize
> such event. On the other hand, such action could bring new people or get
> them from the hiding corner.

Agree. So what might help is find some support in organising, I think I 
can do that.

> I would try a well announced "bug-hunting" session. If we get enough
> people there, I would skip the competition because it is hard to
> organize.

Might be a good choice - as written before, we can decide later.

> Another question is how to instruct people during the bug-hunting
> session. We do not have enough test cases. One possibility would be to
> ask them to create them as described at
> http://wiki.documentfoundation.org/QA/Testing/Test_Case
> Otherwise, it would end in a chaotic testing. Well, it would be better
> than nothing.

Chaotic tests give great results - if people can write down the results 
in a structure way of course ;-)
I can draft a page to get just the people in the bug-hunt sessions on 
the right track.

Does that sound OK, fair enough to continue?
If so, I'll do some more bits tomorrow at latest.

Thanks for your comments!
Cheers,

-- 
  - Cor
  - http://nl.libreoffice.org