[Fontconfig] Application startup performance

[Nick Alcock]

On 14 Jan 2016, Keith Packard stated:

> Nick Alcock <nick.alcock at oracle.com> writes:
>> I think automated out-of-date cache detection and cache updating must
>> stay for $HOME. It's only system fonts that are problematic for me
>> (though even for $HOME we need to avoid mega-statting, as now, and we
>> *also* need to be robust against filesystems with coarse-granularity
>> timestamps. Both of these behaviours currently appear to be broken.)
>
> There used to be an explicit 'sleep' when updating caches to avoid
> problems with such file systems; perhaps that's insufficient?

Not sure. All I know for sure is that my entire NFS-mounted system font
directory (~10,000 fonts) gets every font statted on every startup. Even
with a boosted attribute cache, this is... very slow.

A stat() of these files (and indeed of their containing directory) is a
tad peculiar, on both NFS client and server:

Access: 2016-01-08 13:36:46.992954790 +0000
Modify: 2006-07-03 21:16:43.000000000 +0100
Change: 2009-06-27 23:04:25.200485375 +0100

What's happened here is that the underlying fs and NFS server and client
combination are both capable of nanosecond timestamps, but when the font
file was last modified, the filesystem wasn't capable of that and was
keeping times at a 1s granularity. This does mean that clever tricks
like looking to see if the microseconds are set and determining that oh,
they're all zeroes, this fs is not microsecond-capable will not work
well.

However... the *earliest* change time of anything in
/var/cache/fontconfig is

Access: 2013-04-28 19:59:55.728033866 +0100
Modify: 2012-09-23 10:10:47.513425940 +0100
Change: 2012-09-23 10:10:47.513425940 +0100

so clearly the cache validation code should be able to get away with a
timestamp check on the directory: determining that it is not necessary
to re-stat() everything does not even require *month* granularity of
timestamps, let alone microsecond granularity! :)

Digging into this some more, the underlying problem is that
FcDirCacheMapFd() unconditionally calls FcCacheDirsValid(), which more
or less always calls FcDirScanOnly(), which does a readdir() on the
directory (already pointless) and an FcFileIsDir() on every file in
it... which devolves to an FcStat() even though this also is pointless
because we just readdir()ed it, which got the d_type... and then we
threw it away only to have to re-stat() it again.

A repair of *that* (incidentally dropping a qsort() -- in a path as hot
as this! -- whose declared intent is only 'to make things prettier' and
whose other declared intent, to prevent Heisenbugs, is probably
relatively minor and can be reintroduced under a debug flag later on --
I don't know if it has any actual effect on semantics but I doubt it),
though incomplete and ugly, helps a lot. my set of fonts is restricted
right now because I'm sick of waiting for half a minute for programs to
start, but even with this, we see the time required for fontconfig font
scanning dropping from 2.865s to 0.05s.

This patch will probably need changing in small ways if we care about
any systems on which dentry.d_type does not exist (probably we do: this
is just a proof of concept):

>From ebb2e08e2077fc2f24b82a2aded9b682318ae7da Mon Sep 17 00:00:00 2001
From: Nick Alcock <nick.alcock at oracle.com>
Date: Thu, 14 Jan 2016 14:10:05 +0000
Subject: [PATCH] Speedup of startup on systems for which dentry.d_type exists.

The speedup can be radical, reducing time taken to validate the
caches by 100x when a lot of fonts are being served over NFS.

We also throw away the sort of font filenames.  Yes, it reduces
Heisenbugs, but it also makes it impossible to track ancillary things
like dentry.d_type and introduces an expensive operation into one
of the hottest paths in fontconfig for purely precautionary ends.
---
 src/fcdir.c | 54 ++++++++++--------------------------------------------
 1 file changed, 10 insertions(+), 44 deletions(-)

diff --git a/src/fcdir.c b/src/fcdir.c
index f4807dd..7bedf42 100644
--- a/src/fcdir.c
+++ b/src/fcdir.c
@@ -190,15 +190,6 @@ FcFileScan (FcFontSet	    *set,
     return FcFileScanConfig (set, dirs, blanks, file, FcConfigGetCurrent ());
 }
 
-/*
- * Strcmp helper that takes pointers to pointers, copied from qsort(3) manpage
- */
-static int
-cmpstringp(const void *p1, const void *p2)
-{
-    return strcmp(* (char **) p1, * (char **) p2);
-}
-
 FcBool
 FcDirScanConfig (FcFontSet	*set,
 		 FcStrSet	*dirs,
@@ -210,11 +201,9 @@ FcDirScanConfig (FcFontSet	*set,
 {
     DIR			*d;
     struct dirent	*e;
-    FcStrSet		*files;
     FcChar8		*file;
     FcChar8		*base;
     FcBool		ret = FcTrue;
-    int			i;
 
     if (!force)
 	return FcFalse;
@@ -248,48 +237,25 @@ FcDirScanConfig (FcFontSet	*set,
 	goto bail;
     }
 
-    files = FcStrSetCreate ();
-    if (!files)
-    {
-	ret = FcFalse;
-	goto bail1;
-    }
     while ((e = readdir (d)))
     {
 	if (e->d_name[0] != '.' && strlen (e->d_name) < FC_MAX_FILE_LEN)
 	{
 	    strcpy ((char *) base, (char *) e->d_name);
-	    if (!FcStrSetAdd (files, file)) {
-		ret = FcFalse;
-		goto bail2;
-	    }
-	}
-    }
-
-    /*
-     * Sort files to make things prettier
-     */
-    qsort(files->strs, files->num, sizeof(FcChar8 *), cmpstringp);
 
-    /*
-     * Scan file files to build font patterns
-     */
-    for (i = 0; i < files->num; i++)
-    {
-	if (scanOnly)
-	{
-	    if (FcFileIsDir (files->strs[i]))
-		FcFileScanConfig (NULL, dirs, NULL, files->strs[i], config);
-	}
-	else
-	{
-	    FcFileScanConfig (set, dirs, blanks, files->strs[i], config);
+	    if (scanOnly)
+	    {
+		if (e->d_type == DT_DIR ||
+			(e->d_type == DT_UNKNOWN && FcFileIsDir (file)))
+		    FcFileScanConfig (NULL, dirs, NULL, file, config);
+	    }
+	    else
+	    {
+		FcFileScanConfig (set, dirs, blanks, file, config);
+	    }
 	}
     }
 
-bail2:
-    FcStrSetDestroy (files);
-bail1:
     closedir (d);
 bail:
     if (file)
-- 
2.7.0.198.g6dd47b6