[Fontconfig] Next steps for a reproducible Fontconfig?

[Alexander Larsson]

On Thu, Jan 10, 2019 at 9:41 PM Keith Packard <keithp at keithp.com> wrote:
>
> Alexander Larsson <alexander.larsson at gmail.com> writes:
>
> > I'd like to repeat that this is not really flatpak specific as such.
> > The issue can happen in multiple cases like nfs mounts, multi-boot
> > systems, docker containers, etc.
>
> Sure, any place where path names are not the same would cause the same
> issue. I think your other examples are unlikely to exhibit this in
> practice though -- when NFS is used for system-level sharing, the normal
> configuration tooling goes to extreme measures to ensure that pathnames
> are the same across all systems.

Yeah, practice seems to indicate that this is not presently a problem.
But as the world moves more and more towards containerization in many
places and other tools like snappy also target the desktop it is imho
important to at least consider it.

> > Flatpak has an additional weakness here, which is that we don't store
> > mtimes (to maximize content sharing abilities the mtime is not part of
> > the content addressing). This means the mtime can't be used to detect
> > a stale cache, so we use the uuid to detect such changes.
>
> I don't understand this -- the only way UUID could be used to trigger a
> stale cache is by being missing. Otherwise, I thought mtimes were still
> the only method used to know if the cache file was out of date?

Lemme describe the expected mode of operation for fontconfig in a flatpak:

First of all, /usr comes from the runtime, and it is immutable. We
only ever modify the runtime by creating a new version of it in a
separate place and pointing new sandbox instances at the new
directory, and /usr is mounted read-only in the sandbox. Since all the
files in the runtime are maintained by ostree they all have a mtime of
0.

The runtime contains /usr/share/fonts with uuid files. These uuid
files are re-generated each time the runtime is rebuilt, so an updated
runtime will get a new uuid. It also contains a pre-generated font
caches for the fonts in the runtime based on the uuid.

In the sandbox $XDG_CACHE_HOME is set to ~/.var/app/$APPID/cache, and
this is the first writable cache directory in the fontconfig config,
so all caches will be per-app. However, if you update the app (say
from 1.0 to 2.0) the same cache dir will be used, just not between two
completely different apps).

In the normal case we will never need to generate any caches for the
runtime fonts. However, in weird cases (maybe someone pulled in a
fontconfig that has a different font cache format or whatever) the
cache generated will be based on the uuid, so if you update the
runtime you will get a new uuid and the old cache will not be used (if
we did we could not detect it was stale using the mtime).

Host fonts are mounted in /run/host/[user-]fonts, and host caches in
/run/host/[user-]fonts-cache, and the runtime has a config snippet to
add these. If the host has uuid for the dirs, then we can pick up the
host caches from the host cache dirs. If not, then the app will on the
first run re-generate the caches for /run/host/fonts and store it in
the per-app cache dir.

These directories are just bind-mounts of the host stuff, so normally
they have proper mtimes and we can detect stale caches fine, its just
slow the first time you start the app. However, on an ostree-using
base system (such as fedora silverblue) the mtime issue may actually
cause us to use stale caches here.

> > I realize this is not what the reproducible builds project wants, but
> > it is what the UUID was added for.
>
> The UUID files also violate one of the principle design goals I adopted
> for fontconfig when cache files were moved to a separate directory --
> never touch the font directories.

I think an optional approach fits with this. I.e. when it makes sense,
allow the font dirs to carry some identity, but don't touch them by
default.

> I also fought with fontconfig for about a week when the release
> including them was installed on my machine as firefox would spin
> whenever it found a directory with no fonts. At the time, I felt injured
> by this change.

Hmm, why was it doing that though? Doesn't seem like it would have to.

> Hrm. Could some combination of mtime checking and content addressing
> work? Consider a .uuid file generated from a hash of the directory
> contents. If the .uuid file is older than the directory, you could
> regenerate it reasonably quickly by hashing the directory contents. That
> would be faster than re-scanning all of the fonts in the directory at
> least.

You can't trust directory mtimes in this way. A file in the directory
can be updated without modifying the directory mtime. That is only
modified when you create or remove files.

> > I'm willing to make *some* changes to flatpak, but I'm not sure this
> > is the right approach. First of all it just looks at a tiny subset of
> > the problem (only flatpak, and only one directory).
>
> Changing flatpak to *always* mount host directories at the same place in
> the container should solve the problem for all directories.

There are two issues with this.

First of all, the runtime is like a chroot, and its supposed to look
like a "regular" system, but now certain paths are "reserved" and has
to be handled specially. I could accept this for a limited set of
paths though, although all existing flatpak runtimes would have to be
modified.

However, the second problem is that it puts demands on the *host*, as
it now has to match the layout of the runtimes so the pathnames can
match identically. The distro may store fonts in a different location,
and as long as you configure flatpak for your distro with
--with-system-fonts-dir= then the current setup will work. But if we
enforce it to /usr/share/fonts that will break. I'm not sure if this
is currently a practical issue for existing distros. I know distros
put the font caches in different places, but i'm not sure about fonts.
Maybe "weird" setups like nix will run into issues. You might know
this better.

> > Also, we'll be guaranteeing that caches for /usr/share/fonts and
> > /usr/share/fonts-minimal don't conflict, but there is no guarantee
> > that different versions of /usr/share/fonts-minimal don't conflict.
>
> I don't understand this -- the cache for /usr/share/fonts-minimal live inside the flatpak environment, and should be per-flatpak?

Yes, but if you update the runtime and /usr/share/fonts-minimal
changed in the new version (but has same mtime), then the stale cache
file in the users homedir will still be used.

> > Here is my proposal:
> >
> > Make the uuid *generation* optional and manual. Then, when we create
> > the flatpak runtime we run fc-cache --make-uuid (or something) to
> > generate the uuid files. Then fontconfig would never confuse the
> > sandboxed /usr/share/fonts with any other, and since we would get a
> > new uuid each time we regenerated the runtime it would correctly pick
> > up stale caches when we update the runtime (even with no mtime
> > change).
>
> Hrm. This is a tempting solution -- normal users would never see .uuid
> files at all.
>
> However, it means that new directories created within the flatpak while
> the system is running would not get .uuid files and might then have
> cache names which collide with the outer system.

In practice, this is not an issue, because the flatpak directories are
immutable.

> How about making it a font configuration per-'dir' option instead? This
> way, uuid files would be automatically added to all 'internal'
> directories and never to external ones.

Initially I imagined uuid files would work somewhat like this. I.e.
you put a .uuid file in the top /usr/share/fonts directory, and all
the subdirectories would hash based on uuid + relative path. This way
you don't have to have all these uuid files all over the place.
But, as you say we could do that but inject the uuid file from the
outside in a config file. Then all we have to do is rewrite such a
config file in flatpak.

However, flatpak will never parse the entire xml fontconfig file
format (which isn't even really stable over time), so such a config
would have to be external in a simpler config format.
For example, we could have a /etc/fonts/uuids file which is a simple list like:

/usr/share/fonts b81b806a-fb12-4a31-b458-181b1be0ec23

And then flatpak could read this and generate one for the sandbox that said:

/run/host/fonts b81b806a-fb12-4a31-b458-181b1be0ec23

This, in combination with the use of uuid + relative path would allow
the sandbox to use host caches.

However, the /etc/fonts/uuids file would still not be reproducible, so
i'm not sure this is better than using uuids for the sandboxed dirs
only, and then mapping the paths for picking up the host caches.

> > We still wouldn't have a way to reuse host caches which were mounted
> > in a different way, but if we assume all conflicting directories use
> > uuids (like they would in the flatpak case), then we could solve this
> > in a pretty simple way by a config file saying "treat all instances of
> > /run/host/fonts as /usr/share/fonts", and I could make flatpak
> > generate such a file.
>
> I've already got a patch series which solves this problem -- you can map
> paths to cache keys on a per-'dir' element basis.

Currently the runtime contains a static conf.d snippet that says:
        <dir>/run/host/fonts</dir>
        <dir>/run/host/user-fonts</dir>

We would have to turn that into a dynamic snippet. But that would be a
problem for pre-existing flatpak binaries which doesn't do this.
Could we instead have a way to modify a previously added dir element?
Or maybe it could handle duplicate <dir> tags such that they keep the
original order, but update the mapping? Then we can have both the
static snippet an a dynamic one for later flatpak versions.

> Here's an alternative proposal:
>
>         Add a per 'dir' element 'salt' value, which is stirred into the
>         path name when generating the cache key. You'd generate this
>         randomly when the flatpak was created so that all cache keys
>         would not collide with entries using a different (or absent)
>         salt value.
>
> With this, and my path->key mapping series, we would be able to access
> the existing cache files for external fonts (via the mapping mechanism), as
> well as avoid collisions between internal and external font paths within
> the cache. And we wouldn't have .uuid files (see above).

I like this in theory. But i don't want to rewrite the entire runtime
font config xml file (its just to fragile). I much prefer if such a
salt could be added by just dropping a file somewhere. I.e. a
fonts.conf snippet that tweaked the salt of a previously defined dir
element.

So, counter proposal:

Flatpak generates at startup a file like this in /run/host/fontconf.xml

<cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
<cache-as path="/home/alex/.fonts">/run/host/user-fonts></cache-as>

In the runtime we create at build-time a /etc/fonts/conf.d/ file:

<salt id="randomdata">/usr/share/fonts</salt>
# Duplicate this with static versions for old flatpak versions
<cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
# This will (if it exists) override the above with the live values
<include ignore_missing="yes">/run/host/fontconf.xml</include>

We are basically in full control of existing flatpak runtimes, so this
could easily be added to all of them an propagated out. However, the
new flatpak version that creates the fontconf.xml file will not reach
distros in a while, so the above duplicates the "most likely to be
right" value for /usr/share/fonts for older flatpak versions to pick
up.