[systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Martin Pitt]

Jóhann B. Guðmundsson [2016-02-18 10:08 +0000]:
> Will failed tests or false positives start auto creating issues on github?
> If so is that really the smart thing do to here?

IMHO not. The idea is that raising the red flag on the PR should be
enough, and that the point of the CI tests is to not land regressions
in master in the first place. issues should mostly be filed against
releases or master, not PRs (which can always potentially break
stuff).

> The only test system that should be hooked into upstream should just be
> upstreams own not some downstreams right.

That's what we do today, but these only run the unit tests, which have
a very low coverage of all of systemd's functionality. E. g. I found
about five regressions in networkd alone last year with these tests,
two instances of "sometimes the boot breaks" [1][2], and the
regression with handling block devices (#2537). It's so much easier if
these get raised right away in the PR that introduces the regression
than having to bisect and investigate days or weeks after it landed
and the damage is already done.

Note that these builds are done with zero downstream patches -- it's
just the source as it comes out of the PR. Of course it has to run
*on* some OS, and that currently happens to be Ubuntu 16.04 for
"ubuntu-ci" and Ubuntu 14.04 for semaphoreci.

I'd actually turn it the other way around and claim that if Fedora,
Arch, etc. have downstream tests, then please trigger them too. After
all, failures of them don't block anything (right now, anyway), and
having the extra information in the PRs can only be useful?

Martin

[1] http://lists.freedesktop.org/archives/systemd-devel/2015-February/028640.html
[2] https://github.com/systemd/systemd/issues/1505

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)