Sizes data in flatpaks

[Dan Nicholson]

In the ostree meetup the other day we talked about accurate progress
for flatpaks when doing object pulls. At Endless we've been making our
OS commits with the ostree sizes metadata and using that for progress
in our updater.

The advantage is that the client has the entire object listing for the
commit and can know exactly how many objects will be downloaded and
what their sizes will be ahead of time. The downside is that the sizes
metadata balloons the commit object. The metadata is represented as
one long byte array for each object. The byte array is the sha256
checksum, compressed object size as a variable length integer,
uncompressed object size as a variable length integer and the object
type. So, there's a minimum of 35 bytes per object. Our OS commit
objects are large as a result.

Anyways, I wanted to see how this would affect flatpak, so I wrote a
script (attached) to fetch refs then recommit them with the sizes
metadata. Here are some results for some popular refs.

Ref                                              Objects  Current
With Sizes    Change
---------------------------------------------  ---------  ---------
------------  --------
runtime/org.freedesktop.Platform/x86_64/19.08      11266  2.6 KiB
456.4 KiB     +172.1x
runtime/org.gnome.Platform/x86_64/3.38             20092  2.4 KiB
810.2 KiB     +336.2x
app/org.gimp.GIMP/x86_64/stable                    10297  1.4 KiB
415.2 KiB     +287.0x
app/org.mozilla.firefox/x86_64/stable              20220  1.5 KiB
814.4 KiB     +526.8x
app/com.spotify.Client/x86_64/stable               21214  1.8 KiB
854.2 KiB     +481.8x

It definitely makes a big difference, although none of these are over
1MB like our OS commits are.

--
Dan Nicholson  |  +1.206.437.0833  |  Endless
-------------- next part --------------
A non-text attachment was scrubbed...
Name: add-commit-sizes
Type: application/octet-stream
Size: 5906 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/flatpak/attachments/20201009/b6eb7629/attachment.obj>