[Telepathy] RFC: flattening Telepathy projects into fewer source trees

[Simon McVittie]

On 21/10/13 19:33, David Edmundson wrote:
> I don't fully understand the reason behind the problem you are having
> with having to release the spec, then release glib and only then merge
> the dev branches in each CM.

The problem is that it's quite "chicken and egg": we shouldn't merge
things that aren't ready to support long-term, but we can't really know
it's ready until we've tried implementing it service- and client-side
and seen the flaws. So a developer wanting to add a new feature has to
branch at least 3 projects, get review on all of them, iterate on
changes if necessary, and merge everything at once.

Further, our policy of not having 'master' of component A depend on an
unreleased version of component B means you have to do actual releases
of several components in order to finish merging. I think that's a
valuable policy - it ensures that 'master' is always releasable - but in
practice, it means people who aren't comfortable with the release
process can't merge non-trivial features, and it takes a significant
time even for someone who knows the release process.

If you're hacking on two features at the same time (perhaps developing
one while waiting for review feedback on the other), you now need to
juggle 6 builds... not particularly straightforward, and not exactly
welcoming to new developers.

> Why can't you just merge every dev branch at the same time and release
> all the repositories all at the same time? It would have the same
> drawbacks as having everything in the same repository.

OK, so suppose I'm adding a feature. Connection_Interface_Hats, say. I
have to:

* merge it to telepathy-spec
* release telepathy-spec 0.27.a
* copy that into telepathy-glib
* maybe merge a branch adding high-level API to telepathy-glib
* release telepathy-glib 0.23.b
* it's only now that I can merge a branch into Empathy
* it's also only now that I can merge a branch into whichever CM wants
  it (usually telepathy-gabble)

Based on experience, this takes much more work than it ought to -
particularly if one of the releases doesn't go according to plan (e.g.
smoke-testing a release fails). Please try implementing a new interface
throughout the stack if you want to experience this first-hand :-)

If I was a new contributor and not comfortable with making releases (or
if the maintainers weren't comfortable with me making releases), the
process would take even longer.

> It seems very backwards from the current trend of libraries which is
> to split things.

I don't think either "split all the things" or "merge all the things" is
correct - there's an optimum level of splitting/merging somewhere in
between. It's interesting to see projects' level of granularity
oscillate over time.

Historically, the Qt/KDE world had a few enormous repositories (at least
for "core" projects), whereas GNOME was more modular and loosely-coupled
(e.g. the libsoup http library isn't part of GLib). At the moment, KDE
seems to have a trend towards splitting off lots of small repositories,
whereas the trend in GNOME is to accumulate more functionality into GIO
and Gtk. I think both are reacting to historical weaknesses by heading
in the other direction - perhaps they'll settle on a good middle point,
or perhaps they'll overshoot their targets and go too far the other way.
Ask again in a few years :-)

I should perhaps point out that I'm proposing to keep the same number of
shared libraries and pkg-config modules we have now, just build more of
them from the same source tree, with a --disable-foo option for partial
builds (a bit like the way Qt 4 builds QtCore and QtGui separately, or
the way Telepathy-Qt builds the main library and optional Farsight and
Farstream glue libraries).

> In KDE in particular we already have > 100 git
> repositories all being tagged at the same time.

Would you consider a distribution that shipped non-synced versions of
those 100 repositories (with explicit dependencies satisfied, but
versions not otherwise in sync) to be acceptable, or seriously broken?
If the answer is "seriously broken, you should use version 5.2 of
everything, consistently" then I'm not sure there's a lot of point in
separating them. If the answer is "acceptable" then, wow, you must have
remarkably good QA to be able to support that.

At the moment Telepathy aims for the "very good QA" answer, but with the
number of contributors who are currently active, adding major features
while preserving compatibility for arbitrary versions takes more time
than we really have. We need to conserve "core maintainer" time because,
to put it bluntly, we don't have enough; and I think the new developer
experience needs to become much easier, because at the moment, *I* find
it awkward to add features (and I've been doing this for 7 years, much
of it full-time).

>> telepathy-spec and telepathy-glib
> I don't like the "glib is important, Qt is a second class citizen"
> mentality. Granted that does reflect the current situation, but I
> don't want it being constrained to being like this forever.

Yeah, I've opposed this merge in the past myself - if anything I have a
stronger "breaking compatibility is unacceptable" instinct than the rest
of the Telepathy maintainers. However, if we don't get more design
agility somehow, I think there's a serious risk that Telepathy will just
stagnate, and that doesn't benefit anyone.

The high-level goal isn't the API, after all: the high-level goal is to
enable high-quality applications, and the API needs to support that.
Obviously, if the API breaks too often, that hurts the applications by
forcing everyone to rewrite - but if the API constrains the applications
by making it take forever to add features, that's also bad.

Telepathy 1.0 has taken over 4 years so far[1] and still isn't ready. A
substantial part of the problem with doing Telepathy 1.0, which I'm
trying to address via batches of "matching" snapshots, has been that
whenever we break API in the spec or in a library, we have to prepare
patches for a dozen components, and merge them all at once, otherwise
things don't remain buildable.

To some extent, the API in telepathy-spec is already irrelevant: what
matters is the API in the client libraries (telepathy-glib and
Telepathy-Qt), and the API in services. In addition to the "official"
interfaces, we have several interfaces that never quite made it into
telepathy-spec, most notably CD.I.Messages, which has been implemented
in a "draft" state by MC and Telepathy-Qt for several years, but never
made it into the spec. I'm trying to consolidate that, but processes
that take up time without direct benefit are just not sustainable.

Having one "primary" library API in which we can prototype things seems
a bit better than treating *both* of the library APIs like second-class
citizens, as we do at the moment :-) At the moment, the GLib API is the
only one in which it's practical to write services; even after
Telepathy-Qt gets service support, the GLib API is going to be the one
used by Mission Control (given the amount of time I've spent developing
MC, I don't think reimplementing it is desirable or practical), by the
Logger, and by Telepathy-Farstream.

I've recently been looking into how we can make telepathy-glib support
GObject-Introspection enough to write connection managers (connection
managers written in Python/JavaScript/etc.), which would be great for
prototyping. It's tricky because of implementation details and
dbus-glib, unfortunately, but I've made some progress on it.

[1] based on the date at which I opened
<https://bugs.freedesktop.org/show_bug.cgi?id=23148>

> Farstream I'm not so convinced about, TelepathyQt uses
> TelepathyFarstream but it does not use TelepathyGlib.

It does, indirectly: telepathy-farstream requires telepathy-glib.
Neither telepathy-farstream nor telepathy-glib is exposed in the Qt API
(so in principle they could be replaced with a Qt reimplementation,
although I think that would be a massive waste of effort), but it's
required behind the scenes.

>> telepathy-glib and "major" connection managers
...
> It makes some connection manager's appear more important than others.

In practical terms, some CMs are already more important (or at least,
better-quality) than others: Gabble > {Salut, Haze, Rakia} > {Idle,
Ring} > anything that used telepathy-python. telepathy-glib started as a
fork of telepathy-gabble, if you look back far enough in its git history.

I might be biased on this because we've been doing a lot of cleanup work
recently in preparation for Telepathy 1.0. A lot of that was removing
deprecated telepathy-glib API calls from all the CMs, which should have
happened several years ago, but didn't, because it involved patching
half a dozen parallel projects and people lost interest before they got
round all of them. If some large subset of the CMs were in-tree with
telepathy-glib, the same person that deprecated the function could have
fixed all the internal uses in an earlier commit to the same branch.

> It becomes very hard to ever replace a connection manager (and we have
> dropped sunshine and butterfly in the past) as you end up with a
> connection manager in the main repo that everyone has to build (or
> packagers will build at least) as well as something newer.

We can turn CMs into "disabled unless explicitly enabled", or delete
them. Packagers in sensible distributions can still ship separate binary
packages from one big source package.

> Whilst we're on the subject of repositories, telepathy-logger-qt is
> inside KDE repositories and not on Freedesktop.org. This makes little
> sense, and is something I'd like to fix.
> Maybe I should merge TpQt if you go that way with logger+TpGlib.

Yes, I think it makes sense to turn telepathy-logger-qt into an
optionally-built subdirectory of TpQt. It seems better to do that than
to introduce a new /telepathy/telepathy-logger-qt.git on fd.o.

    S