[Fontconfig] Next steps for a reproducible Fontconfig?

[Keith Packard]

Alexander Larsson <alexander.larsson at gmail.com> writes:

>> I also fought with fontconfig for about a week when the release
>> including them was installed on my machine as firefox would spin
>> whenever it found a directory with no fonts. At the time, I felt injured
>> by this change.
>
> Hmm, why was it doing that though? Doesn't seem like it would have to.

A .uuid file was added and removed to every directory in the font tree
which contained no fonts or sub-directories; (a) the directory mtime was
changed, causing the system to re-scan the fonts and then
re-create/re-delete the .uuid file (goto a). Somehow this would
eventually stablize (I'm not entirely sure how). It was a 'surprising'
problem that happened only once in a while and it took several days of
searching to locate as the only symptom was that firefox would hang for
'a while' and then start working. Once I ran strace on the process while
hung, it was pretty easy to track down, but having fontconfig affecting
directory mtimes was not what I expected...

> You can't trust directory mtimes in this way. A file in the directory
> can be updated without modifying the directory mtime. That is only
> modified when you create or remove files.

Fontconfig already assumes that all 'interesting' changes result in
directory mtime changes -- it doesn't expect files to be changed in
place. If you have a system doing this, fontconfig will fail.

> However, the second problem is that it puts demands on the *host*, as
> it now has to match the layout of the runtimes so the pathnames can
> match identically.

Well, I was thinking that the runtime would dynamically adapt to the
host environment and change the configuration inside to match the
host.

Is it that flatpak treats fonts very specially, or are there other bits
of host data injected into the flatpak runtime in a similar fashion?

Essentially, I was imagining the flatpak runtime system would discover
the set of fontconfig directories exposed by the host and inject those
into the runtime using matching paths. Right now, you inject two fixed
paths (/usr/share/fonts and ~/.fonts), but if we really want to deal
with systems that put fonts in other places, then presumably flatpak
will need to adapt anyways.

Once you've discovered where fonts are being stored in the host, having
those get mounted in a matching path in the runtime doesn't seem like a
huge step to me. It's a fairly simple matter of changing the mount
target path and injecting an xml fragment file into /etc/fonts/conf.d

> Maybe "weird" setups like nix will run into issues. You might know
> this better.

Good point; as new distros start experimenting with different filesystem
layouts, we will need to be more cautious about assuming fixed paths
of any kind.

> Yes, but if you update the runtime and /usr/share/fonts-minimal
> changed in the new version (but has same mtime), then the stale cache
> file in the users homedir will still be used.

Ah. Thanks for explaining this. It seems like the only way this could
happen is if the cache file within the flatpak was stale and a
replacement generated and written to the user's homedir (as the only
writable location available).

I think that's just a bug in the flatpak generation -- the cache file
within the flatpak should always be up-to-date as (I assume) the
fontconfig library provided in the flatpak would have been used to
generate that cache file. And, a future flatpak shipped without that bug
would have a correct cache file, which would presumably be used in
preference to the one stored in the home directory?

> Initially I imagined uuid files would work somewhat like this. I.e.
> you put a .uuid file in the top /usr/share/fonts directory, and all
> the subdirectories would hash based on uuid + relative path.

That sounds like the same as my 'salt' idea, except you're storing it in
the font directories rather than the font configuration.

> However, flatpak will never parse the entire xml fontconfig file
> format (which isn't even really stable over time), so such a config
> would have to be external in a simpler config format.

The fontconfig xml format is quite stable and is designed to be
manipulated by tools that do not understand the full contents, hence
using XML and providing a suitable DTD.

However, in practice it's become far easier to use XML snippets instead,
so let's figure out how to do that instead.

> For example, we could have a /etc/fonts/uuids file which is a simple list like:
>
> /usr/share/fonts b81b806a-fb12-4a31-b458-181b1be0ec23
>
> And then flatpak could read this and generate one for the sandbox that said:
>
> /run/host/fonts b81b806a-fb12-4a31-b458-181b1be0ec23

Yeah, moving the 'salt' out of the font directory and into the font
configuration seems like a good direction here. And placing it in a
separate file (although probably in XML format) seems like a better idea
than merging it into the existing <dir> elements.

> However, the /etc/fonts/uuids file would still not be reproducible, so
> i'm not sure this is better than using uuids for the sandboxed dirs
> only, and then mapping the paths for picking up the host caches.

Right, the 'host' installation cannot include any UUID values to
ensure that builds are reproducible. That doesn't invalidate this
approach though; we can add UUID values for any directories within the
sandbox and those will not collide with the host entries.

> Currently the runtime contains a static conf.d snippet that says:
>         <dir>/run/host/fonts</dir>
>         <dir>/run/host/user-fonts</dir>
>
> We would have to turn that into a dynamic snippet. But that would be a
> problem for pre-existing flatpak binaries which doesn't do this.

Any existing flatpaks will presumably include an existing fontconfig
which will not use font caches for the host which will not have .uuid
files in each directory anyways. As long as those existing flatpaks
"work" by re-generating the cache at first start, that seems fine to
me. Re-create those flatpaks with new fontconfig bits and things will
work better.

> Flatpak generates at startup a file like this in /run/host/fontconf.xml
>
> <cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
> <cache-as path="/home/alex/.fonts">/run/host/user-fonts></cache-as>
>
> In the runtime we create at build-time a /etc/fonts/conf.d/ file:
>
> <salt id="randomdata">/usr/share/fonts</salt>
> # Duplicate this with static versions for old flatpak versions
> <cache-as path="/usr/share/fonts">/run/host/fonts></cache-as>
> # This will (if it exists) override the above with the live values
> <include ignore_missing="yes">/run/host/fontconf.xml</include>

I think you'll want two separate files -- the salt is "constant" for the
flatpak, while the cache-as values depend on the host
environment. Otherwise, this looks good to me.

If we can convince you to mount the host font paths inside the runtime
where they had been in the host, then we don't even need that part. But
I can easily see where you'd end up with a pile of magic conditionals to
avoid having some critical part of the flatpak file system smashed by
fonts. Plus, we've already got that part of the solution mostly
implemented, and we all know how horrible it is to throw away code (just
kidding).

-- 
-keith
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/fontconfig/attachments/20190111/20faa235/attachment.sig>