[systemd-devel] New "ubuntu-ci" integration tests are being added to PRs

[Martin Pitt]

Jóhann B. Guðmundsson [2016-02-18 11:35 +0000]:
> Arguably not if upstream test do not detect these things then they should be
> fixed to do so

The upstream tests are mostly unit tests which cover isolated logic.
Of course having more coverage there would be great, but that's not
the point here. systemd is not an isolated project, but it lives in an
operating system, controlling services like dbus, gdm, NetworkManager, etc.

So saying "fix them to do so" in an email is easy, but did you
actually think about *how* this should look like? We do have some QEMU
tests upstream which cover a little bit of "does this VM boot", but
not much else -- there is no integration testing with other
components. So for doing proper integration testing you need to take
an actual OS, with actual packages, and check that they still work. If
that's Fedora or Debian or whatnot is not that important in the end,
and ideally we test them all.

But it's certainly not practical for the upstream systemd tests to
pull and build glib, X/wayland, gdm, NetworkManager, evemu-tools, etc.
and construct their own synthetic OS into a VM. That would be a
huge maintenance burden, take awfully long, and this synthetic
mini-distro isn't actually being used by any any real person.

So, is it possible that these tests will fail because of a downstream
issue? Absolutely, yes, and I'm sure they will do that at some point.
I'm looking at those failures, and they are a problem that needs
fixing anyway (and if it's a downstream issue, I'm no less on the hook
for that than now). But experience has shown that almost all test
failures which aren't due to actual regressions are due to either bugs
in the tests or infrastructure failure -- and both of these will apply
equally to so-called "upstream tests".

So integrating these into PRs instead of running them every other day
on master makes it much simpler and cheaper to identify the cause of a
regrssion. You can also always compare against other PRs or a
test against master.

Martin
-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)