Flathub, initial proposal

[Alexander Larsson]

Flatpak and the surrounding software projects
(ostree/gnome-software/appstream/etc) are now in a pretty good
state. We need to work on the portal implementation and port
applications to that to make the sandboxing aspects work, but for the
pure distribution usecase things mostly work.

However, there are two big missing pieces in the full flatpak story:

The App Store Problem
=====================

When a user logs in to a newly installed Linux laptop and launches the
flatpak supporting software installer app (e.g. gnome-software) it
should be full of all the major FOSS software from the start. Right
now this only works if the user (or the admin/distro) manually found
all the various upstream repositories and added them. This is neither
a good experience, or very scalable.

Instead we want to have a single curated entity where you can get most
major FOSS software. We also want that entity to have some degree of
trust, in that it does some kind of verification (manual and
automatic) of the software in the repositories.

The Build/Distribution Problem
==============================

flatpak-builder makes it reasonably easy to package an application as
a flatpak, including whatever dependencies it needs. However, once you
have a working build of your application things get more tricky. How
do you make an official, signed build of it? How do you build on
multiple arches? How do you set up and maintain a public repository?
What about bandwidth costs? Mirroring, etc?

All these are solvable. GPG is daunting but tutorials exists,
scripting the release builds on some machine is doable, and all you
need for distribution is a dumb web server somewhere. However, this is
not exactly easy, and its not work that an upstream developer is
interested in. There should be a simpler way to do this, that everyone
could reuse.

I imagine a web service that you create an account on, somewhat similar
to say github or the ubuntu PPA service. Your account gets access to a
per-user ostree repository, and you can create projects and upload
builder json manifests for them. These get automatically build on
several arches and you can pull the builds for testing. Once the build
is working you can click a button to push them to a stable repository
that you can tell your users to pull from.

These per-user repos would be similar to github repos. I.e. many repos
with little guarantees, rather than the vet:ed single access global
app store. I.e. you can easily fork some existing app and throw a
patch in it to let someone test your new feature.

The proposal: Flathub
=====================

Conceptually these are two different problem, with different
audiences. For instance, one can easily imagine the app store working
without the shared build system. Or the build system being used without
a centralized app store.

We discussed this quite a bit at the Flatpak BOF at guadec however,
and came to the conclusion that an initial attempt at this would best
handle these as a single thing. Partly this is because it gives nice
synergy effects, and that many people would probably want to use
both. But it also lets us give certain guarantees for the app-store,
which are important to the organization that hosts the service.

For instance, having everything locally build from source, with all
the sources available, makes it possible to fulfill licensing
requirements (e.g. the GPL), and makes it possible to verify that the
software is free software. We hope that the Gnome Foundation (or some
similar organization) would help us host and maintain the system
during an incubation phase, and that means we can only support free
software projects.

So, the proposal is to create a web service that combines these two.
In many ways it is similar to github, but focusing on ostree instead
of git.  The minimal viable product for this has the following
features:

 * Register and authenticate as a user.
 * Create and edit projects.
 * Assign members to projects
 * Create OSTree repositories for a user
 * Create OSTree repositories for a project
 * Maintain member permissions for project repos.
 * Create and/or register GPG keys for users/projects/repos
   These would be used automatically for signing builds
   in the repos.
 * Trigger a build in a repository by uploading a json file
   or a tarball with json file + dependent files.
 * Migrate a commit from one repo/branch to another.
   For instance, you may have a testing repo where you build
   things. Once tested you can then take the latest build
   from that and put (+re-sign) in the stable repo that you
   give to users. (This is the build-commit-from flatpak
   command.)
 * A well known Flathub project+repo that contains the curated
   "Linux upstream app store".

Bring the features!
===================

This MVP should be pretty simple to get going. Once that is working we
can start doing more complex things. Here is a mostly unsorted list of
things we probably want:

A way to request that a project or build should be available in the
app store. This should put the build in a review queue that the
curators of the app store can view and work on.

Automatic scanning/review of builds. For instance, this could ensure
that apps have valid appdata, desktop files, icons etc. It could also
do some simple license scanning, CVE scanning, etc. The sky is the
limit. This is very useful for automatic-review of the app store
review queue, but is also nice feedback for normal builds.

We want to keep all downloaded source for all builds forever in a
directory that is shared by every build, both to avoid multiple
downloads, but also so that we can always rebuild things, and fulfill
license requirements for GPL:ed software.

Also, we need to keep the json manifest and required files for all
builds done ever, so that they are reproducible. Maybe these can be
stored in the ostree repository itself?

Set up mirroring/CDN for the repos that get heavy traffic, to handle
heavy network loads and to get better download performance.

Collect download statistics and make available to users/projects.

Automatically scan and hardlink common object files across repos to
keep down disk requirements, while still allowing users full access to
do whatever they want in their repos.

A REST API and a command line tool to make it easier to submit builds,
etc.

A web app-store frontend equivalent to gnome-software. This lets you
browse the appdata files, rate/comment on apps and install them (it
could generate key-value files that would open in gnome-software for
installation).

Also, there is the question of what runtimes we should allow on the
site. They would need to be on the build servers to be able to build
things. Should we allow people to add runtimes themselves? Should we
redistribute the runtime on the appstore (probably, to get the
mirroring, etc).

Have some way to donate/pay for applications. Here be dragons...

Allow pre-built (i.e. not built on Flathub) flatpaks in the
app-store. Dragons abound here too, but useful for larger projects
like libreoffice that do their own builds.

Support for user-private repositories. This seems useful, but there is
no support for authentication in ostree atm, so would have to be added.

Build on weaker architectures, like ARM, using the userspace emulation
stuff that bastien has been working on.

Going forward
=============

I don't really have a lot of experience building services like this,
and I'm pretty busy with the flatpak core stuff. So I would love if
people who are interested in working on this could chime in.

One very important issue is where to host this initially. I want to
bring this up with the Gnome Foundation as a start, but we may also
want to talk to e.g. Digital Ocean for hosting. They seem to be
sponsoring a lot of free software stuff.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
       alexl at redhat.com            alexander.larsson at gmail.com 
He's a lonely guerilla sorceror with a passion for fast cars. She's a 
ditzy communist bounty hunter living on borrowed time. They fight crime!