[Libreoffice] Number of days for testing RC

Petr Mladek pmladek at suse.cz
Mon Dec 20 09:09:13 PST 2010


Hi Jean,

I am sorry for the late repply. I have been busy with LO-3.3-rc2.

Jean-Baptiste Faure píše v Čt 16. 12. 2010 v 06:05 +0100:
> Le 13/12/2010 15:04, Petr Mladek a écrit :
> > This is why I proposed 7 days instead 5 days. 7 days have the Weekend
> > included by definition :-)
> Not really if the period start just before the WeekEnd. If a new RC is
> released on Friday for QA-test, most of the testers will not be able to
> start their tests before Monday.

Fair enough. We will try to do not release on Friday. Alternatively,
would it help if we pre-announce the upcoming rc2 few days before it is
available?

It might always be delayed because of technical problems. Though, I
think that we could predict it pretty well two days before it is
published.

> > Hmm, I think that OOo-3.3 release is not a good example:
> Yes, from RC cycle point of view, it is an example that LibO should not
> follow.
> The process is not sufficiently clear:

Yeah, we are still setting the processes up.

>  what QA-team must do when a stopper is found and known?

The current solution is described at
http://wiki.documentfoundation.org/Release_Criteria#Blocker_Bug_Nomination

>  If a new RC is build immediately, a second
> stopper can be fixed only on this new RC so it is not interesting to
> continue to test the old RC. It is more true because the new RC can
> introduce regression to the old one.

You are right that any fix could bring regression. Though, I think that
it always makes sense to continue with testing:

	+ there might be more blocker bugs; they are usually
          independent; it is better to find all blockers as soon as
          possible
	+ the fix for the stable branches should be tested and reviewed
          before they are committed; it should reduce the number of
          regressions to minimum; the question is if we really need to
          repeat the whole QA for each rc; I think that it should depend
          on the number and complexity of the fixes; this information
          should be mentioned in the announce mail
	+ it takes few days to produce new rc (time needed to fix the
          bug, do some testing, build in Windows, Linux, MAC; upload to
          mirrors)


> So I think we need a very clear QA test process: known dates for RC,

I am not sure if we want a regular schedule, e.g. do the rc release
every Thursday and every second Thursday. I think that such approach
would wok well for alpha/beta builds.

In case of release candidates, I think that it is better to do them
according to the current state. I mean to release them when all blockers
are fixed or when there are more that 2 weeks without release.

Note that we plan to upload also daily builds in the future. They might
be used to verify fixes even before the official release.

> when it is useful to search blockers and when we can stop to search
> stoppers until a new RC is released.

I think that it makes sense only when the application does not start at
all, so that the blocker bug blocks the testing completely.

> > There is an idea of more micro bug fix releases. The published minor
> > release should be well usable for 99% of normal users. The micro
> > releases could be done every next month or so. They would include just
> > safe and reviewed bug fixes and should need not full QA. The second or
> > third micro release should be well usable for 99% experienced users.
> Ok for me. The only condition I see to make the end-users happy, is that
> the status of each version released must be very clear for the end-user.
> Individual users or small organizations may want use always the last
> micro-release while bigger organizations like administrations will have
> a longer cycle of update.

Yes, we need to make it clear.

> >>> Also I think that it does not make sense that every national team would
> >>> do the whole testing. IMHO, 95% of the functionality is language
> >>> independent, so most of the testing can be shared and distributed.
> >> Right. For OOo, NL QA-team do only Release Sanity Test. But the current
> >> test period on OOo 3.3.0 shows that the last blockers have been found by
> >> the Community and these blockers could not be found by automatic or TCM
> >> tests; they have been found by users who have done tests on their
> >> documents they use in the real life.
> >> To do that we need time.
> > I see. Well, I think that even the released OOo version included pretty
> > annoying bugs and some of them would be considered as blockers. Such
> > bugs were usually fixed in the micro release x.y.1.
> >
> > We just need to define the right compromise that would be acceptable for
> > all sides developers, testers, and users

sure

> > Just to get better picture. How much time would you need to finish all
> > known manual tests?
> For OOo and Release Sanity Test, time needed is about 2 to 4 hours to
> complete all the 25 tests in the scenario.

Okay, this sounds doable within the week if you can plan it.

> But, as said the last blockers are out of the scope of these tests and
> have been found by test-cases and files from the real life.

I guess that many of these bugs were found by normal users that used OOo
in their daily work. I think that it actually supports the idea for more
frequent releases and tested/reviewed fixes. 

The manual testing makes sense and is really helpful, definitely.
Though, it could not verify the complete functionality within days or
even weeks.


Best Regards,
Petr



More information about the LibreOffice mailing list