Is anyone using p2p ostree support?

[Matthew Leeds]

On Mon, Feb 17, 2020 at 11:56 PM Alexander Larsson
<alexander.larsson at gmail.com> wrote:
>
> I'm at a point where I'm starting to reconsider the support for the
> p2p ostree setup in flatpak. We keep running into issues with it,
> which means we haven't been able to enable it by default on flathub.
> And it vastly complicates the code as well as the behaviour of flatpak
> on the network, turning a simple natural workflow into something that
> is super-hard to reason about because information comes from several,
> partial and untrustworthy sources.
>

I certainly agree that the P2P (LAN/USB) support in Flatpak adds complexity and
technical debt, and it would be great to find ways to reduce that. But I'm not
aware of any current issues that prevent us from pushing it out to everyone on
Flathub; what did you have in mind?

> I know endless uses p2p as a way to do sideloading of apps via usb and
> external disks. Similarly there has been interest from redhat in
> shipping flatpak apps with the OS install media for install/update
> during OS install. However, we could still support such use of p2p,
> but in a less abstract way.
>

Indeed, there are Endless users which use USB updates. We have support in our
fork of GNOME Software to copy apps to USB drives by calling out to "flatpak
create-usb", and the install/update side of it we get "for free" since it's
built into Flatpak (although we also populate a "USB" category with what is
found on the drive). I don't have numbers on how many people are using the
feature, but it is an important one since it's the only currently
supported way to
update apps or the OS without Internet.

> I imagine we could do a regular flatpak install/update operation with
> a manual option to use a different url for a particular remote. This
> would move the p2p installation source choice to the
> FlatpakTransaction level rather than deep in the guts of ostree pull.
> That sounds a lot more supportable.
>

That does sound easier to support, and in the USB case it sounds like a
reasonable requirement to put on users of the CLI. But in the case of GNOME
Software I guess we would need to make some changes to pass the USB remote
location to the FlatpakTransaction (I don't know if that would require new
API). GNOME Software already has support for an app being available from
multiple sources, in which case there's a drop down box in the top right to let
you choose, so perhaps we could re-use that and populate it with repos found on
USB drives (more generally, mounted filesystems). If we were to select the USB
source when the computer is offline (or online and the USB provides the most
recent version) then we might be able to preserve the ease-of-use of the
current setup, where the user just clicks "Install" and doesn't have to do
anything special to use the USB source. However, it's hard to see how such a
scheme would work for LAN updates (more on that below).

> However, to make sure we're not regressing something I'm interested in
> knowing exactly who uses p2p, and how. Rob, could you (or someone else
> from endless) write up a short document on exactly how p2p is use at
> endless and what future plans for its use you have?

LAN updates were sadly never put into production, and are even disabled by
default on recent versions of Endless. So I can nearly guarantee no one on
Endless is using those, and it seems quite unlikely anyone in the wider world
of Flatpak/Ostree users is using them either, since I could only ever get it
working barely enough for a customer demo, and never had time to fix the
remaining bugs with it. For example, I was able to install an app over the LAN
but not if its runtime was not already installed, and that wasn't ever
investigated. The LAN code paths also have performance issues which haven't
been addressed. In other news, the component of eos-updater that creates Avahi
records advertising available refs on the network (eos-updater-avahi) was never
upstreamed into Ostree, so it would be very difficult for anyone not using
Endless to use LAN updates.

However, the problem intended to be solved by LAN updates still exists, and
there are some potential large deployments of Endless computers in offline or
mostly offline environments for which USB updates are not a practical solution
(due to the manual work involved) and where LAN updates would make a lot of
sense (since the computers would be on the same LAN e.g. in a school). If and
when such deployments materialize, we will need some solution, and if it's not
using the existing Ostree API, then I think it would involve re-inventing a lot
of the same technology. It's hard for me to see how you'd implement that in a
scheme like "flatpak install --actually-install-from=$uri ..." since you can't
expect the user to know the location of repos on their network and what they
provide (the problem solved by OstreeRepoFinderAvahi). They could use "ostree
find-remotes ..." to find it but that would be an unfortunate degradation of
the UX. In the GNOME Software case, presumably you don't want a dropdown with
20 entries when there are that many updated computers on the network.

As you said on IRC, another major issue with the current implementation is that
the repo metadata itself (on the ostree-metadata branch) can come from LAN
peers and is not guaranteed to be in sync with everything else. If what we have
is newer than what peers have, we use the newer version and it might be out of
sync with the refs provided by peers (see the comments in
_flatpak_dir_get_remote_state()). Maybe we should instead have Ostree build a
"fake" ostree-metadata which combines information from the available peers that
provide the latest version of each ref, which would itself be unsigned but
Ostree would verify the signatures on each copy from which it was built. That
would also require changes to ostree_pull_from_remotes_async() since you
wouldn't want to automatically pull from a more outdated peer if the pull from
the one with the latest version failed. But that would mean that complexity
would be removed from Flatpak, which could behave just as if the metadata came
directly from Flathub.

It might be reasonable to not support fully P2P and instead have a single
computer taken from a classroom into town where there's Internet, updated
there, and brought back. Then we would need a way for users to configure that
one as an update source, which is at best a UX degradation from the status quo
where it works transparently. But it would at least mean "flatpak install
flathub ..." could continue to work without any extra options, if Flatpak
checks that system-wide configured update source.

If you have an alternative implementation to propose that would be interesting
to hear, but I'm inclined to think it would make more sense to fix the issues
with what we've got (e.g. https://github.com/flatpak/flatpak/issues/1862).

Best,

Matthew Leeds