[Libreoffice-qa] Regressions in Open Source projects ...

[Petr Mladek]

Hi,

it is a bit late but I haven't found time to read this until today.

Bjoern Michaelsen píše v Pá 16. 03. 2012 v 22:37 +0100:
> On Fri, Mar 16, 2012 at 08:30:50PM +0100, Markus Mohrhard wrote:
> > No it is not that easy. This commit fixed several crashes and problems
> > in the copy/paste code. We will from time to time get such regressions
> > since it is nearly impossible to find all affected code paths in our
> > highly coupled code. This has been introduced somewhere in one of the
> > betas and has not been marked as serious for a long time.

I agree with Marcus that often it is not easy to say what functionality
is affected. Various changes might have many side effects.


> Right. Still I think we can do better there. I am thinking about
>  getting communication channels in place between development and
>  testers to filter the out the 40.000ft view on changes. We "only" has
>  213 changes between 3.5.0 and 3.5.1 so a hint like "we tweaked on
>  Copy/Paste in Calc, please torture that a bit to see if you find any
>  regressions". Incidentally, we could make that a drive-by exercise in
>  adding testcases to Litmus (ideally with a reference to 
> the bugs fixed). ;)

I am a bit scared by adding another channel and layer. Developers
already have to provide commit message, update bugzilla, ask for review
on the mailing list. 

It might be enough to write good commit messages and do not forget to
mention the related bug numbers. I think that we are quite good with the
bug numbers but we could do more user friendly commit messages. They are
sometimes too technical, so normal user or QA does not have any idea
what functionality is affected.

Developers are already pretty overloaded. I doubt that they have time to
write detailed testcases in Litmus. It does not make sense to write one
line in Litmus when it is already mentioned in the commit log.

I suggest that QA volunteers follow commit messages (would be my week
summary still useful here?) and check affected areas. They might ask the
developer when in doubts. It will teach developers to write better
commit messages, ...

The same is valid for features. We already have release notes in the
wiki page. QA guys could watch it and play with new features. They might
just document in Litmus what they did.


Also we must not set wrong expectations. If we force developers to
provide a lot of information for QA, they would think that their changes
are heavily tested and become less careful about their changes.

It would be great to ask for testing if we have many volunteers and
people asking what need testing but I do not see such people.



> > IMHO the way to go is automated testing and adding a test case for
> > fixed bugs im possible. I fear that most of the time not even devs
> > know all affected features so how should a normal user or qa person
> > know what to test.

I agree here.

> I dont think I can agree here. Getting automated tests is good and nice, but it
> will never cover the whole story. Esp. things like Cut-n-Paste, which might be
> platformdependant and only show trouble when used in a real environment and not
> some cuddly headless testcase. The same goes for many other topics:
>  - pretty much all of UI
>  - printing
>  - window management interaction
>  - ...

Yes, some things can't be tested easily automatically but automatic
testing is really important. Once you have a test, you could run it
quickly for each release and you are sure about the state.

Manual testing is time consuming, need a lot of volunteers and nobody is
sure that they will test everything.


> > What is annoying me is the steady complains about every changed
> > feature and fixed bug as making it worse. We sometimes need to change
> > the behavior a bit to fix bugs.

I see your pain.

> > Instead of immediately screaming I would love to see some more constructive
> > critisism especially here on the qa-list instead of indirectly saying that
> > the devs ruined the release with this change. I will now stop here.

Some users communicate this way. People sorting and fixing bugs
sometimes get depressed when looking into bugzilla. In each case, we
could not make happy everyone. It is pity that we do not hear from the
many other happy users. 

> Having such things documented (release notes or whatever) helps in multiple ways:
> - Nobody reads the release notes, but at least the reporter of the bug knows he
>   _could_ have looked there
> - bugwranglers have it a lot easier to manage reports, when they can refer to
>   the release notes

I am not sure how were the problematic changes communicated in the
release notes. I agree that it is the right place to explain such
things in advance.

> Also, we need to get more serious testing on master going and raise
> the expectations on its quality. Master has a decent quality now, but
> I feel we are still suffering from the quality of the 3.4 days. The
> perception that master is ok to be broken is dangerous.

I think that many people are still focusing on 3.5. I know that it is
not ideal but given the resources, number of 3.5 bugs, ...


Best Regards,
Petr





Best Regards,
Petr