Flathub, initial proposal

[Alexander Larsson]

On mån, 2016-09-26 at 10:18 -0400, Owen Taylor wrote:
> I think in any context it's probably good to start off with some
> estimates of what the requirements are; data points that one could
> use:
> 
>  - How much disk space do the nightly builds of gimp (etc.) take up?
> How much does that increase per day?
>  - How much disk space does build.gnome.org use for its git mirrors
> of all modules it builds (checked - 31G)

Here are some calculations I did on the existing app builds I have.
There are three repos, the stable gnome apps which are unlimited depth,
but less commits (only stable builds) and each build are likely to be
less simimar than the build before it. Then we have the unstable gnome
app (nightly) builds which are limited to the last 12 successful builds
which are likely to have more shared since these have few changes
between builds. And finally the nightly builds repo (gimp, et al). The
nightly builds is x86_64 only, but the others are 4 different arches.

                     apps   builds   repo size  avg build size
gnome-apps stable:   24     474      38 GB      82 MB
gnome-apps unstable: 30     1222     60 GB      50 MB
nightly builds:  
   6      59       3.6GB      62 MB

Total average build size: 59 MB

This is a pretty decent variation of apps, architectures, level of file sharing etc, so its probably an OK estimate.

Also of note is that using static deltas for increases the size of the repo but makes downloads faster. The repos above generally have from-empty and from-previous-commit deltas for the HEAD of each branch.

Another interesting thing about ostree repo is that its pretty easy to detect identical objects in two different repos and replace hardlink them, which will let us drop disk use even if the repos are separated.

Of course, there are other uses of disk space too. During builds we use a lot of space for the actual build, and if we want caching between builds (flatpak-builder cache and/or ccache) we need to save per-app build caches. And then there is the mirrored sources, although that is not likely to require so much space.

> The bottleneck on GNOME infrastructure is most specifically backup -
> we use space on a file server from Fedora to do our backup - any
> signficant expansion of that set would need to be negotiated with
> Fedora.
> 
> Beyond the prototyping stage "flathub" is probably better off not
> relying on traditional "mirror tree of files" for backup, but either
> setting up mirroring as part of the application infrastructure or
> using a filesystem with built-in replication. 
> 

Additionally, apart from the backup for "safety" we may need mirroring
for distributing downloads across the world too. I think OSTree has
some built in support for mirroring, but I don't know the details.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
       alexl at redhat.com            alexander.larsson at gmail.com 
He's an immortal misogynist waffle chef with a passion for fast cars. 
She's a bloodthirsty punk soap star from a family of eight older 
brothers. They fight crime!