Please test flatpak git master

[Alexander Larsson]

Hi,

The last month or so I've been working on the repo format in flatpak
and ostree, with the goal of improving repository scalability. 

The problem is that as repos get more and more apps the summary file
gets bigger and bigger, and essentially every single operation that
flatpak does requires an up-to-date summary file from the remote.
Additionally, as the number of apps grows, it is more likely that in
some time period one of them is rebuilt, and that will cause a new
version of the summary file, which then everyone needs to download.

After some analysis I came to the conclusion that the main issues
where:
 * All the deltas of the entire repo is listed in the summary, and
   you really only need the ones for the ref you're updating.
 * The size depends on the number of refs, and for each architecture
   a repo supports, the number of refs grows. But users only ever
   need refs for one arch.
 * The only way to update the summary is to download the new one from
   scratch, even though in practice there is very little that changes
   between two versions.

So, with current flatpak master, which requires ostree master, we now
generate a new summary format in addition to the old one.

If you run build-update-repo on a repo it would now look something like
this:
 repo/summary
 repo/delta-indexes
 repo/summary.idx
 repo/summaries/086bcef85.gz
 repo/summaries/21d7eb857.gz
 repo/summaries/21d7eb857-086bcef85.delta


The summary are the same for backwards compat (only difference is that
we can configure flatpak to not put all arches into it).

The delta-indexes directory contains indexes of the deltas so you don't
need to download all of them.

Then the summary.idx contains an index of a set of subsummaries that
are stored, one for each arch. The subsummaries are indexed by sha256
in the summaries subdirectory. A client typically only needs to
download the one for the right arch. 

Additionally the sub-summaries have deltas, which allows you to go from
a previous version to the current by downloading the much smaller delta
file.

This solves all the scalability issues listed above, and allows us to
add new architectures to flathub without hurting x86-64 users. However,
this is all new code and it needs a bit more testing than a typical
update. So, if anyone who is maintaining a repo is interested in
experimenting with this I would very much appreciate that, so that we
can get it solid before it hits users.


I expect to do a release of this pretty soon, but it is currently
blocking on getting an ostree release out first.