ABI guarantees around runtimes

Thu Apr 15 16:37:57 UTC 2021

Hi all,

A recent discussion around some security updates for the Freedesktop
SDK highlighted an unresolved issue in how runtimes/platforms are
currently published and updated for Flatpak.

Most distributions, platforms, libraries, etc - even those
conscientious upstreams with ABI compatibility guarantees, conservative
enterprise long-term support distros, etc - only guarantee ABI
backwards compatibility. Depending on what the upstream policy is, it's
totally acceptable to either upgrade to new versions within a stable
series or backport patches, that add symbols/enum values/etc if that
new interface or code addresses a serious bug or security
vulnerability. Within a compatible/stable series, there is implicitly a
>= relationship between the version of the library that can be used to
run the program, compared to the library that was used to build it.

Conversely, Flatpak has a "flat" purely equality versioning
relationship using the branch as a string. A SDK branch 20.08 is
assumed to provide a certain set of symbols, and any application built
against it can depend on runtime branch 20.08 and expect that same set
of symbols to be provided. Ensuring that a newer updated platform never
contains symbols that weren't present in the first/original version is
a "forwards compatible" ABI guarantee which is implicitly required due
to the versioning being based on equality alone. You can validly
install a newly-compiled app, and an older release of the runtime on
the same branch, and expect it to work.

This forward compatibility requirement is particularly onerous for
runtime maintainers because updates or patches which are widely
produced and deployed across the Linux community, and are perfectly
acceptable to most parties, are permitted to add symbols. Suppressing
those new symbols/interfaces in the SDK might require linker script
surgery, reworking code, etc, so that the interfaces never change. It's
essentially infeasible, particularly in complex and fast-moving
upstream projects, and would require a far larger team than the
Freedesktop SDK can muster. I don't believe the effort is particularly
worthwhile in any case. So, can we stop?

In Flatpak's "flat" branch space with implied backwards and forwards
compatible interfaces, changes to the SDK that add symbols pose two
problems in a Flatpak universe, one I think slightly hypothetical, and
one very real (in as much as: we've been bitten by it).

1. The now fairly hypothetical problem is just that you can build your
application against a new version of the SDK, embedding a dependency on
some of the new symbols, and distribute/install it to a machine with an
older version of the platform, meaning Flatpak will be happy but it
will not work in practice. Earlier versions of GNOME Software fully
"trusted" the branch - in as much as an application could be installed
without runtimes being modified/updated provided the branch matched.
However the Flatpak CLI, and subsequently the Flatpak Transaction API
for several years now, always unconditionally includes any pending
dependent runtime/platform updates in the same operation as installing
an application. So, although it's possible to arrange manually, anyone
using the higher-level interfaces in Flatpak will always get the newest
contemporary runtime/SDK/etc release for the branch their app uses, so
you will usually get a new runtime. Exporting apps onto a USB for
sideloading also bundles the latest runtime along with the app. So, I
think this is very unlikely to cause any problems in practice - in
certain cases, runtime updates have been issued that do indeed add
symbols to some libraries, and the world didn't end.

2. The far trickier problem comes from derived runtimes. GL drivers and
some other binaries which appear in extension points are often built at
the same time as the runtime, or they could be rebuilt after a symbol
has been added to the runtime, acquiring an implicit dependency on the
new release. In the case there is a derived runtime (eg GNOME or KDE)
that re-uses binaries or recipes from the Freedesktop SDK, you can have
the situation that a new base runtime upgrades a core library, so the
extensions might implicitly depend on some new symbols or interfaces,
but the /derived/ runtime has not been updated/rebuilt to include these
symbols. This means these extensions are "from the future" compared to
the derived runtime, and they could at best fail to load, or prevent
anything from working. Although typically the extensions have a limited
interface surface (a handful of libraries rather than the dozens in the
whole platform), it only takes once glibc itself versions all symbols
and so it's very easy to trip over this situation even with relatively
minor platform updates.

It was originally tempting to say: it's fine we'll just rely on the
Flatpak behaviour that runtimes are always upgraded at the same time as
apps are installed/upgraded, then temporal/causal ordering will remove
our forwards compatibility constraint. However, I don't have an answer
for the "shared extension" case where maintainer action from the
derived runtime is necessary to prevent a window of time where, even if
Flatpak upgrades everything, you could have a new extension that's
incompatible with an older (derived) runtime.

I had also made noises (and I think it was discussed at GUADEC a couple
of years ago) about some kind of versioning scheme where you could use
the content-addressible nature of ostree to release the platform under
a number of branch names, creating an inexpensive simulation of the >=
semantic - ie that you could revise the SDK/platform version, and
publish it under the older compatible branch names where appropriate.
As an example here, the SDK 20.08 branch could be released with a point
releases (20.08.02), and when the platform was published, it would be
released as 20.08.01 and 20.08.00 as well - so any older app would see
the new platform as a (backwards compatible) upgrade, but no
misrepresentation would be made about forwards compatibility.

The extensions points would _have_ to be versioned specifically with
the point release of the platform, with no backwards-compatible
releasing, to avoid the "interfaces from the future" problem for
derived runtimes.

There are two problems I can see with this approach. The first is
around tooling - although flatpak-builder allows the SDK and the
platform to be specified separately by an app, what it doesn't allow is
for the SDK to influence the version of the platform that the
applications end up being run against. So, my Flatpak specifying SDK
20.08 would always end up needing Platform 20.08, not 20.08.02 which is
which is what I would want it to specify if that was the latest SDK
release - without having to go back every time and increment the branch
name in every app manifest.

The second is that developer tools also frequently use a full SDK as
their runtime, meaning that any solution that works for build time but
not runtime is also not sufficient. It's relatively easy for us to say,
flathub has a new flatpak-builder that supports some build-time
binding/indirection of the runtime branch, but it's harder to see how
to solve that at runtime in a backwards-compatible way. Maybe they just
"float" and use the unversioned 20.08 name which is always the latest
release? Does that work?

I think this also bloats the number of branches lying around on
Flathub, but if it was just for runtimes it might be tolerable. Does
anyone have any bright ideas here?

As well as easing the burden on Freedesktop SDK, to encourage timely
updates and community participation, my other goal is that I would like
to build some consensus (and if needed tooling) around what the "best
practice" is for versioning and updating runtimes, so that Flathub can
start to consider a policy for accepting runtime uploads from other
first-party publishers.

Cheers,
Rob