[Libreoffice-qa] Minutes - QA related - TSC call 2012-01-26

[Petr Mladek]

Hi,

first, I am sorry with the late reply. I am a bit late in reading mails.

Pedro píše v So 28. 01. 2012 v 08:30 -0800:
> More intensive testing is not the problem. Your TSC summary shows there are
> still 80 regressions. At a rate of 1 fix per day that would take over 2.5
> months.

I agree that regressions are bad and might discourage some users. On the
other hand If am not sure about the priorities. Some regressions are are
in rarely used functionality. Bugs in more widely used/visible
functionality might have higher priority.


> The problem is that there is a new release each month (sometimes 2 releases
> in the same month)
> http://wiki.documentfoundation.org/ReleasePlan
> 
> It is reasonable to expect that these new builds will have more features
> (especially the 3.6.x) so new bugs and new regressions will be added...

We have a release with new features only every 6 months. The many builds
are pure bugfix releases. We allow only safe fixes for the bug-fix
releases. All fixes are reviewed by second person. We ask QA for extra
testing if a more risky fix needs to go into the bugfix release.

So, the full QA should be needed only during the beta and rc phase for
the .0 release. Minimal testing should be needed for the bug fix
releases.

Of course, we had some regressions even in bug fix releases but I do not
remember anything super critical. IMHO, the stability of every 3.X
release was improving with every bug fix release.


> And since for each pre-release period, which happens monthly, the priority
> are the blockers, regressions will tend to accumulate and overall quality to
> decrease as mentioned by Nino on this topic (I agree with him)
> http://nabble.documentfoundation.org/Libreoffice-qa-preparing-QA-talk-for-FOSDEM-tp3691881p3691881.html

I think that it is not easy to compare.

LO-3.3.0 release was based on OOo-3.3.1. The code base already absolved
several months of bugfixing. You would need to compare 3.4.3 or 3.4.4
with it.

LO-3.4 was our first real release with features, refactoring, and other
big changes. It was not ideal because we merged many features too close
before feature branch. The first betas were not much usable. Anyway, we
fixed many things in the bug fix releases and we are still going to fix
many annoying bugs in 3.4.6.

LO-3.5 is our second real release. We have added features regularly
during the past months. They were tested using daily builds... I think
that the overal feeling about 3.5.0 is much better than it was with
3.4.0.

Note that we might have more bugs reported, more regressions detected.
Though, it might be also because we have more users that report bugs,
more people doing bug triage and correctly marking regressions. So, it
is possible we have even less number of bugs but we know about them in
time (in compare with 3.4).

Note that .0 release will always have annoying bugs if we stay with the
time based release. We just need to teach users and set the right
expectation.

The important thing is the comparison between the different .0 releases,
and different .1 bug fix releases. 

Note that you would need to compare 3.3.0 with 3.4.3, 3.3.1 with
3.4.4, ... because of the history of the 3.3 release.

I have the feeling that 3.5.0 is better than 3.4.0, so the trend might
be positive.

I agree that the situation is not ideal. We need more people doing QA,
triaging bugs, ... I think that we still need to improve the wiki pages
that might help volunteers to start. We need to propagate the existing
ones, ...

Also many QA processes might need optimization and improvement. It might
help if we decide on what functionality is core, what is important and
what is less important.

Also I wonder what approach you use when triaging bugs, if you take them
by subject, or initial severity, or creation date, ... IMHO, we need to
teach users to set the initial severity reasonable, so it helps to
prioritize even the triage.

Another thing is how much time we need to spend with reproducing the
bugs. The information is sometimes very useful. On the other hand, it is
very time consuming. We might want to use NEEDINFO flag more often.

Well, I am not sure where now is the best time to communicate this. It
might be long discussion and it might be better to do it later when 3.5
is out and in a reasonable state.

In each case, I have a good feeling about the current trend.

Thanks a lot for your contribution. You do a lot of work and it is much
appreciated.


Best Regards,
Petr