[Libreoffice-qa] Regression triage

[Matthew Francis]

Hi All,

Some of you may know regressions are something I have a bit of an interest
in ;)

Our handling of them is slowly improving, but one thing we aren't doing
very well at the moment is systematically evaluating all confirmed bugs for
regressions. When the origin of a bug hasn't been investigated in detail,
it will often languish unloved for want of developer attention, whereas
when we can find exactly when and how a bug was introduced, it can be
escalated to a particular developer, or at least be advertised as an easier
fix than one a developer has to investigate from scratch. In addition, the
closer in time to when it was introduced that a regression is investigated,
the better chance it has of being fixed easily.

So, what can we do better?

As I see it, a two pronged approach is needed.


1) Make it easier to identify regressions in the first place

It's been suggested before, including most recently in conversation on IRC
with Norbert Thiebaud (shm_get), that a Linux bibisect repository of
released versions would be useful, and I agree with this. If every regular
QA contributor kept a copy of this, it would be quick and easy to identify
whether a bug was a regression, and confirm to the first approximation when
it was introduced.
(Snapshots of this repository could be made available in the usual manner;
in terms of size it should be small enough to be a live repository, but
then there is the issue that at least two release branches are active at
any one time. To deal with this, we could represent the release branches as
branches in this repository too, but that could be challenging to work with
for the less git savvy, so possibly the simplest solution is just to keep
rewriting it as a linear sequence and issue a new snapshot every once in a
while... ? Either that, or wait until a branch is EOL before releasing the
next one - but that would be quite a long time behind real time)

Following this of course, we still need to identify an individual commit
where possible. The growing set of "max" bibisect repositories should make
this easier too, but that is still a longer and more involved process than
what should be the easy task of identifying which are regressions in the
first place, and I don't suggest that whoever confirms a bug must leap to
that stage in the first instance.


2) Increase the visibility of the work to be done

By this I mean arranging things so that the work that needs doing can be
seen from a simple bugzilla search.
To make this happen, some small changes to how we triage bugs are probably
needed, and for that we should have a discussion on the semantics of what a
regression is and isn't.

In the broadest terms, if there is a previous version of LibreOffice which
doesn't contain a bug, then it can be considered a regression, but there is
a catch here. Some such bugs aren't regressions in the sense of "something
which used to work but now doesn't", but are rather bugs which were
introduced together with a new feature which didn't exist before.
Currently, we don't categorise these as regressions, but as I see it they
should probably be investigated in the same way. Sometimes a developer will
comment on such a bug that "This was obviously introduced with such and
such a feature", but from the bugs I've seen this does not by itself
generally translate into a quick resolution. Just as mentioned in (1)
above, properly identifying the commit which introduced the feature (with
the bug) creates more opportunities for the bug to be fixed by the original
developer or an interested newcomer.

Given this, we should either:
a) Just label these as regressions too, or
b) Give them their own keyword (such as "newFeatureBug"? Bit of a mouthful
though...)

Then, we can have a simple bugzilla search for bugs which are both NEW (or
REOPENED), and not set to one of
- Version: Inherited from OOo
- Keywords: regression
- (in the case of (b)) Whatever piece of metadata identifies "new feature
bugs"

And that will serve as our canonical in-tray for ensuring that initial
regression triage is done right - any bug in the results of that search
will need attention by definition, with no exceptions.


Any thoughts on the above would be appreciated

Regards
Matthew Francis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/libreoffice-qa/attachments/20150403/1e3ea778/attachment.html>