[Libreoffice-qa] Regressions in Open Source projects ...

[Bjoern Michaelsen]

Hi Rainer, all,

On Fri, Mar 16, 2012 at 07:21:15AM +0100, Rainer Bielefeld wrote:
> a) Regression "Bug 47096 EDITING: Alt + Drag and Drop for complete
> rows impossible in particular documents" frequently forces me to use
> 3.4.5.

I had a short look at this one as a exemplary post-mortem (the bug seems to be
assumed fixed from the developer side). What we have here is a bug that was
introduced on the release branch (as it wasnt in the beta), which is not good.
But such things can and will happen, the question is how to handle them
quickly. So what can we do better here?

A casual (actually I have no idea about this code domain) digging through the
history shows that this regression was likely introduced by:

 http://cgit.freedesktop.org/libreoffice/core/commit/?id=a00c5917a8deb465cc1322e140000a81340f9d69

because that last touched related code areas (so this was neither adding nor
removing a regression, but exchanging one for another). So how can we improve
detecting such things on the release branch?

 git log libreoffice-3.5.0.2..libreoffice-3-5-1

tells me that there have been 213 commits between 3.5.0 final and 3.5.1 final, which suggest a rather limited scope of areas touched. IMHO it would make sense to focus manual testing for the touched areas of code. Doing a:

for d in `find . -maxdepth 1 -mindepth 1 -type d`; do echo `git log --pretty=oneline libreoffice-3.5.0.2..HEAD $d|wc -l` $d;done|sort -n|tail

shows the following stats:

5 ./svx (GUI)
7 ./officecfg (configuration/settings)
7 ./oox (docx filters etc.)
7 ./writerfilter (Writer)
9 ./solenv (build/installation)
12 ./scp2 (installation)
13 ./extensions (various)
15 ./connectivity (Base)
22 ./sc (Calc)
39 ./sw (Writer)

If we could coordinate those interested in a bit more systematic manual testing
to look at these areas for micro releases, to prevent "fixed a regression,
introduced a regression scenarios", that would be awesome. Since the number of
changes in one specific area of code in rather limited, we then can:

- pinpoint the change (and the developer) that causes the trouble quickly
- reject fixes that cause another regression
- consciously make a choice to include a fix that causes a minor regression,
  when it fixes a major one and there is no good alternative

Divide and conquer is the way to go here. With the 22 commits in Calc for
example this should be something one volunteer should be able to comfortably
have an eye on over the ~one month period between both releases. Writer (most
likely because of Michael Stahls regression fixing rampage) currently might
need two people volunteering. And obviously, the number of commits will likely
slow down now after 3.5.1.

So dear QA-list members:
  Anyone interested in picking this up for one application?
  Who would like to do it for Writer?
  Who for Calc?

> b) Regression "Bug 45385 EDITING: Copy Paste formula to different
> document adds source document filename to references" (what even
> isn't accepted as a bug, but will be a showstopper for me to use
> LibO if no solution will be found) frequently forces me to use
> 3.4.5.

So far this is not a bug, but intended behaviour. As that I wouldnt consider it
a regression or an indication of QA problems. It indicates something else
though: we had a failure to communicate there, the intended change of behaviour
would have needed to be communicated clearly and before the actual change was
made. This is something we need to improve upon, and we need to find a good way
to make this information accessable. This could be done on the dev ML, but
given the noise on it a bug filed for this would be even better. I am open to
other ways to improve communication there, if proposed.

> c) 3.4 has no working e-mail mail merge, so I have to use 3.5 for this.

3.4 is history and we dont work like that anymore.
 
> d) No LibO version shows line width in Chart legend correctly
> because of "Bug 31551 Lines in legend not shown with current width",
> so I have to use OOo 3.1.1 for some needs (inherited problem, 3.3
> has the same problem like LibO). But unfortunately OOo 3.1.1 is very
> unstable with my master documents edited with LibO.

Another pre-3.5 regression from when our workflow was suboptimal. Yes, we got
to fix those, but it is even more important that we make sure now to not add
regressions. Preventing a regression is so much easier than fixing one. Thus
preventing regressions helps us find the resources to fix regressions that did
slide in while fixing bugs and adding features.

> May be you understand why I have some empathy for users fulminating
> on mailing lists and why I never recommended LibO to friends without
> many reservations?

I totally have for users using 3.4. But empathy shouldnt make us stray from
rational decisionmaking.
 
> But we should use our time to find and eliminate these problems
> instead of whining, thanks to all pest control helpers.

Indeed!

So: Im looking forward to voluteers interested in adopting an area of code for
regression testing between minor -- just reply to this mail!

Best,

Bjoern