[Libreoffice-qa] Regressions in Open Source projects ...

[Bjoern Michaelsen]

Hi,

On Fri, Mar 16, 2012 at 08:30:50PM +0100, Markus Mohrhard wrote:
> The correct commit is
> http://cgit.freedesktop.org/libreoffice/core/commit/?h=libreoffice-3-5-0&id=49a1737b7d3d61304e749c6c164165b8bf68790e
> 
> While is not good that this commit introduced a new regression it was
> fixing several problems and I was testing copy/paste (the only know
> bug problem area at this point) and did not find any problems.

Proofs my "no domain knowledge" claim quite nicely. ;)

> No it is not that easy. This commit fixed several crashes and problems
> in the copy/paste code. We will from time to time get such regressions
> since it is nearly impossible to find all affected code paths in our
> highly coupled code. This has been introduced somewhere in one of the
> betas and has not been marked as serious for a long time.

Right. Still I think we can do better there. I am thinking about getting
communication channels in place between development and testers to filter the
out the 40.000ft view on changes. We "only" has 213 changes between 3.5.0 and
3.5.1 so a hint like "we tweaked on Copy/Paste in Calc, please torture that a
bit to see if you find any regressions". Incidentally, we could make that a
drive-by exercise in adding testcases to Litmus (ideally with a reference to
the bugs fixed). ;)

> IMHO the way to go is automated testing and adding a test case for
> fixed bugs im possible. I fear that most of the time not even devs
> know all affected features so how should a normal user or qa person
> know what to test.

I dont think I can agree here. Getting automated tests is good and nice, but it
will never cover the whole story. Esp. things like Cut-n-Paste, which might be
platformdependant and only show trouble when used in a real environment and not
some cuddly headless testcase. The same goes for many other topics:
 - pretty much all of UI
 - printing
 - window management interaction
 - ...

> And some more general remarks.
> I think it was always known that the .0 release will still contain
> some bugs and maybe the .1 release too. I think we are doing a great
> job in the 3-5 release in fixing the serious bugs so that 3.5.2 is
> already more stable than 3.4.5 or 3.4.6 (at least in calc). For
> example my query for calc bugs shows no serious calc bug that is open
> and that can be fixed for 3.5.2.

Totally agree: 3.5 is an awesome series of our product.

> What is annoying me is the steady complains about every changed
> feature and fixed bug as making it worse. We sometimes need to change
> the behavior a bit to fix bugs.

Yes, that is true and development should not lose inertia for this. But we need
to get this communicated out of development if such changes of behaviour
happen, that _is_ essential. A non-developer will absolutely get lost on the
noisy and messy bazaar that is the development mailinglist. We need to find a
way to provide "the outside world" with a 40.000ft view of what is going on
without burdening ourselves too much.

> Instead of immediately screaming I would love to see some more constructive
> critisism especially here on the qa-list instead of indirectly saying that
> the devs ruined the release with this change. I will now stop here.

If you think the qa-list is troublesome, I urge you to talk to a selection of
random endusers. The qa-list is tame in comparison. But a lot of that is
mitgated by simply having that documented (one line in the release notes, or
better yet a bug on bugzilla). An enduser that went through the trouble of
filing a bug is not exactly happy when his work (bugfiling) is swept away with
"yeah, we actually intended that, but never bothered to tell anyone" (his
perspective). Keep in mind that for every good enduser who filed a bug, there
are 100 who do not, but rant about it on twitter instead. 

Having such things documented (release notes or whatever) helps in multiple ways:
- Nobody reads the release notes, but at least the reporter of the bug knows he
  _could_ have looked there
- bugwranglers have it a lot easier to manage reports, when they can refer to
  the release notes

Having too many "false positives" i.e. bug reports about intended changes
wastes a lot of time: the time of the reporter (reporters are possible future
libreoffice-qa volunteers), the time of the bugwranglers, the time of the
developers. And it might make us miss the important bugs in the mud. This is
why getting this documented early is so important.

Also, we need to get more serious testing on master going and raise the
expectations on its quality. Master has a decent quality now, but I feel we are
still suffering from the quality of the 3.4 days. The perception that master is
ok to be broken is dangerous as it might lead to severe miscommunication:
- development does an intended change
- testers even see and ignore it assuming it only to be transitional
  (as sideeffect of the low expectations on master)
- upon release, we get a sudden "hey, why wasnt that fixed?" confrontation
  for a lot of things at once

The confrontation itself is not the problem. The problem is that it happens to
late and en bloc.

Best,

Bjoern