XDG_CONFIG_DIRS an /usr/local/etc/xdg

[Peter White]

On Mon, Sep 20, 2021 at 10:20:05AM +0200, David Faure wrote:
> On dimanche 19 septembre 2021 16:57:22 CEST Peter White wrote:
> > On Sun, Sep 19, 2021 at 12:19:33PM +0200, David Faure wrote:
> > > On jeudi 16 septembre 2021 18:48:41 CEST Peter White wrote:
> > > > But, /etc should be off limits for software in /usr/local, right?
> > > 
> > > I don't think this assessment is correct.
> > > 
> > > For instance, I certainly expect KDE software installed in any prefix, to
> > > respect the global settings in /etc/xdg/kdeglobals
> > 
> > Why would software need to read that file? 
> 
> Well, that's the point of config files, to be read by software, isn't it?

Well, duh, but not by *other* software. Why would any software have to
read files it otherwise has no business with? kdeglobals is a file
related directly to KDE. Plus, every app then also needs to *parse* that
file. That sounds like an invitation for a whole lot of duplicated
effort when, if those values are intended to be used, they should be
available by other means, i.e. provide a library that abstracts that
away.

The way you describe it, it would be OK for any app to just parse the
config of any other. That just feels wrong, because app A should have no
business snooping in /etc/xdg/B/Brc. If app B wants to make such
information available to others it should export it instead of requiring those to
parse the file.

> > Granted, I know virtually
> > noting about KDE, but shouldn't there be a facility that makes those
> > values available by other means, i.e. environment variables, or
> > a settings daemon or whatever? 
> 
> You want one environment variable per setting in that file? That doesn't 
> scale. A settings daemon might be what gnome does, it doesn't mean that all 
> other free desktops want to have such an architecture. Surely reading files is 
> still allowed in 2021?

Yes, it very much is, just not files of other applications. You seem to
taking an "illegal" shortcut here, by expecting apps to do things that
are otherwise frowned upon.

> > And BTW, shouldn't that be /usr/{,local/}share/kdeglobals [1]?
> 
> No, config files don't go to XDG_DATA_DIRS.
> 
> > 	> share/config/ ... A special case is "kdeglobals": this file is
> > 	> 
> > 	> 		    read by all KDE applications.
> > 
> > and then XDG_DATA_DIRS is the relevant env var which already has the
> > correct default, as you point out below. Now, I don't know why "config"
> > files would go anywhere other than ${PREFIX}/etc but that is apparently
> > what KDE deems the right place.
> 
> The above documentation is really outdated, it says "kde 3" in many places,
> it predates the XDG base dir spec (at least, its use by KDE).

Then provide a better reference? That's what my search came up with. ;)

> > Anyhow, if one really needs to make /etc/xdg/kdeglobals available to
> > apps in /usr/local, then that is one special case that applies to KDE
> > only
> 
> I don't believe so. As I said, everyone agrees that /usr is available to apps 
> in /usr/local (since that's the default value for XDG_DATA_DIRS)
> so why not do the same with config dirs?

And again, *you* talk about /usr and *I* talk about /etc. Compare that
to PATH. There is no equivalent for /etc, because there doesn't need to
be.

> > and that is the only case I can think of right now. 
> 
> There are lots of other files in /etc/xdg...
> For instance /etc/xdg/user-dirs.conf which is not KDE specific at all.

And that also has nothing to do with the config of the app itself. It
provides information about the environment.

> > > XDG_CONFIG_DIRS was modeled very much after XDG_DATA_DIRS,
> > > where I would have tons of other examples like: apps in /usr/local or
> > > anywhere else should still see /usr/share, for e.g. /usr/share/mime which
> > > has the mimetype definitions.
> > 
> > That is not the same as /etc. The well known behaviour, prior to XDG,
> > should not be broken for desktops and, as pointed out above, use the
> > share/.. hierarchy then or whatever, since this seems very much like a
> > KDE quirk to me and should not be baked into a standard that is supposed
> > to agnostic of the environment.
> 
> Please stop saying this is a KDE quirk. It's the XDG base spec that defines 
> /etc/xdg to be the default location for systemwide config files,

And I welcome that effort very much to reduce clutter in /etc. BTW, KDE
should then also put kdeglobals in its own sub/base directory, i.e.
/etc/xdg/kde/kdeglobals. That would make it more consistent with how
XDG_CONFIG_HOME works, which XDG_CONFIG_DIRS is supposed to mirror
system-wide.

> > > And yes, the intent is definitely that they should be read at runtime,
> > 
> > Yes, to find some global settings, maybe, but to find its *own* rc-file?
> 
> An app's own configuration is combination of the system wide defaults
> (in /etc/xdg), its own installed defaults (in $PREFIX/etc/xdg),

Now we are getting somewhere...

> and the user's 
> preferences (in $XDG_CONFIG_HOME). This assumes the XDG_CONFIG_DIRS is
> set to contain the first two values, as is customary when using a custom 
> prefix since the exact same thing has to be done with XDG_DATA_DIRS.

Why would XDG_CONFIG_DIRS need to contain ${PREFIX}/etc/xdg for that?
The app pretty much already knows where it is supposed to find *its own*
system-wide config. The location of which *should* be in
${PREFIX}/etc/xdg/<appname>/, yes, but one does not need to check
XDG_CONFIG_DIRS for that at runtime. This is very much the reason for me
starting this thread. App developers seem to think that they need to
find *their own* configs this way, which invites ambiguity and probably
unpredictable/unintended behaviour as I tried to point out with my worst
case example.

It would make sense, though, to query XDG_CONFIG_DIRS at *compile time*,
to see if the target prefers something different from the default
/etc/xdg and just prepend ${PREFIX} to be more consistent with that.

And the more I think about it *and* seeing that no-one seems to ever set
XDG_CONFIG_DIRS, which then defaults to /etc/xdg, this variable could
just as well be renamed to the singular version XDG_CONFIG_DIR and only
contain *one* value, either set explicitly or the current default. The
whole reason for XDG base directory spec was reduction of clutter, after
all. So why would you allow to have *multiple* base directories in the
first place? There is only one XDG_CONFIG_HOME as well.

And please don't come talking about ../share again. ;) That is a
different story entirely.

> > > so that [advanced] users can install things in custom prefixes and make
> > > things work by setting a few env vars.
> > 
> > That is not something for "advanced" users, /usr/local is *the* very
> > well known default for make.
> 
> I know, that's what I'm saying.
> By custom I mean really custom (e.g. /opt or $HOME/install or whatever).

/opt has other implications and is usually reserved for apps one cannot
or does not want to build locally, i.e. Firefox. One requirement is that
*all* files need to reside there including the system-wide defaults.
Now, if Firefox then needs *additional* information to better fit into
its environment, *then* it can go and query XDG_CONFIG_DIR(S) but not
for its *own* config. BTW, /etc/opt is missing from the default, because
that is where additional configs for /opt are supposed to go, according
to Filesystem Hierarchy Standard.

> > And the expectation is that everything
> > needed to run software installed by 'make install' is available *inside*
> > said prefix. 
> 
> Agreed. Hence /usr/local/etc/xdg.

But that breaks, as soon as there are multiple locations specified in
XDG_CONFIG_DIRS, see my worst case.

> > Just look at the default PATH on any distro. It will contain
> > /usr/local/bin and that will take precedence over /usr/bin. So one does
> > not need to be "advanced" to run 'make && sudo make install' and then
> > just run the locally built software by entering its binary's name
> > without any further ado.
> 
> Agreed, hence my suggestion.

That is not the same. Again, PATH works on a *file* basis and the first
match found wins, ignoring all others. If that were the case with
XDG_CONFIG_DIRS, I would most certainly agree with you, but it works
differently, so no. ;) Again, see my worst case.

> > > What it seems to me, is that /usr/local/etc/xdg should simply be added to
> > > the default value for XDG_CONFIG_DIRS.
> > 
> > Again, that breaks prior art.
> 
> Bugfixing is about changing behaviour indeed, but for the better.

And yet it got worse. ;) There was no bug to fix in the first place. And
XDG should stick to its premise and not invite clutter through the back
door by means of multiple system wide base directories. There should
really only be one: /etc/xdg by default. Mirroring that in any prefix is
as easy as prepending it. Don't make things more complicated for the
sake of it. ;)

> > Picture this: Software "foobar" installed in /usr/local finds *file*
> > foobarrc in /usr/local/etc/xdg/foobar with *information*/setting A but
> > not B in there, it will go on and read /etc/xdg/foobar/foobarrc and use
> > setting B from there, if a distro provided version is also installed.
> 
> That's the point of layering config files, yes.

Yes, layering *user* config on top of the system-wide one but not
layering one (up-to-date) version of the apps own config on top of an
outdated one. There is a reason for etc being present in /usr/local. ;)

> You could add ~/.config/foobarrc to this sentence, it's a 3-layers view of the 
> configuration.

Yes, if you want to get pedantic. I wanted to stick to the relevant
stuff. ;)

> > In turn, running the distro version would also prefer setting A in
> > /usr/local/etc/xdg/foobar/foobarrc despite there being A with a
> > *different* value in /etc/xdg/foobar/foobarrc. Setting A in the most
> > certainly made for a newer version of foobarrc in /usr/local could just
> > as well lead to a DoS of the distro version of foobar, because at the
> > time it was packaged the newer, now legal, value of A was illegal.
> > And, since setting B was deprecated in the meantime the local version
> > will also deny service.
> > How is that for a worst case? ;)
> 
> You can cook up a similar scenario with /usr/share and /usr/local/share,

I invite you to try. ;)

> and yet I don't remember anyone ever having a problem with the distro version 
> of an application picking up a wrong version of a file in /usr/local/share.

Because you don't seem to get the difference. That is, in fact, data
meant to be shareable. And /usr/local/share is *meant* to override
distro versions but also on a file basis. So, you are comparing apples
to oranges here. I did notice a difference in the XDG spec compared to
FHS, though. XDG also talks about information here, or at least it looks
that way, where the FHS clearly only goes down to the file level. So the
reason, no-one complained (so far) is that non-XDG software simply
ignores all others after it has found a matching file in the most
important directory. That is the *expected* behaviour, and quite
frankly, I believe XDG should not change that by expecting apps to go on
and read other files, because that makes deprecation unnecessarily hard.
With FHS you would only need one file to mask the deprecated one. With
the seemingly intended behaviour of XDG_DATA_DIRS there is no other way
than having code that deals with that.

"Everthing is a file", remember? But XDG wants to get more granular and
breaks many assumptions that come with that mantra. So at the very least
the spec should elaborate on how to deal with that.

> I think the idea was that one would not install the same app into /usr and 
> /usr/local. It was rather about making things work out of the box in both 
> prefixes, for different apps.

Well, if that is the case, then it is a huge oversight, which I hinted
at already.

> > > It's inconsistent that XDG_DATA_DIRS
> > > defaults to /usr/local/share:/usr/share while XDG_CONFIG_DIRS defaults to
> > > only /etc/xdg instead of /usr/local/etc/xdg:/etc/xdg
> > 
> > And there might be good reasons for that, as just outlined above, which
> > is why I would very much like some input from the original authors on
> > this topic.
> 
> Good luck. At least I'm pretty sure Waldo doesn't read this list.

Well, then what is it there for, if one cannot get information directly
from the horse's mouth? But thanks for the hint. I will poke the authors
to have a look at this thread then.

> But one thing is for sure: if there's a reason, it's not the fact that config 
> files would be layered, because *that's the exact reason why XDG_CONFIG_DIRS is 
> a set of dirs and not just one.*
(emphasis added)

When in reality there is only one? Layering of config files is pretty
much a very old and most certainly prior art. Look at how shells
inititalize themselves. They do exactly that. User config trumps system
wide config, but the order of reading the config files is from least
important, i.e. /etc/bashrc, to most important, i.e. ~/.bashrc, simply
overwriting values as they come, so the most important setting wins,
because it is guaranteed to be the last read.

> And I remember that at the time, there was a 
> strong use case for user groups sharing similar defaults, so the whole point 
> was to set things up like /opt/admins/etc/xdg:/etc/xdg for some people

That should be /etc/opt/admins/etc/xdg:etc/xdg, just as an aside. ;)
And since you only point to non-prefixed examples, I am more and more
inclined to *not* set XDG_CONFIG_DIRS to include any prefix whatsoever.
One can take XDG_CONFIG_DIRS as a template for ${PREFIX}, maybe, and at
runtime just prepend ${PREFIX} when walking through XDG_CONFIG_DIRS.
If that is acceptable for the spec I would appreciate an amendment to
that effect. Even if one then sets:

XDG_CONFIG_DIRS=/usr/local/etc/xdg:/etc/xdg

the last match would win since the first would in reality point to
/usr/local/usr/local/etc/xdg and result in a miss. For everything else,
*aside* from its own config, use the prestine value of XDG_CONFIG_DIRS.

But I still fail to see why an app would even need to check that value
to find its own config files.

> Because /etc/xdg isn't /usr/etc/xdg, the lack of /usr/local/etc/xdg
> was overlooked.

Which I was suspecting from the beginning. If that is the case, then the
spec needs amendments or at least some guidance on how to deal with it
properly. First and foremost I would discourage app devs from abusing it
to find their own config files, whose location they should know anyways.

But I would also encourage a rethink of the whole reading of files in
/etc/xdg that are not directly related to an app. As I said, that is
frowned upon where I come from. There should be a specified way of
getting such information.
At first glance $XDG_RUNTIME_DIR/xdg seems like a good fit.  But that
should not be all. There must be a unified way of putting information
there and, more importantly, reading it without having to resort to
custom parsers for different origins of information. Think sysfs. So for
kdeglobals that could be something like this:

	> $ cat ${XDG_RUNTIME_DIR}/xdg/kde/kdeglobals/key_A
	> value_A

> > The main question still remains without a clear answer: Who or what
> > should query XDG_CONFIG_DIRS and to what end? Regular software: should
> > it really use it to find and then read its very own config file(s), the
> > location of which being known at compile time anyways? Or is this for
> > getting *other* information about the environment which said software
> > has no other means of getting, like kdeglobals, for instance?
> 
> Regular software also uses libraries, which also have to find their own config 
> files. "Known at compile time" leads to non-relocatable installations, which 
> is rather restrictive for some use cases.

Name one. ;) Looking at the software I build locally to be more
up-to-date, i.e. mpv, that is no problem. It even changes the docs to
account for prefix. And relocatable is most certainly not what anyone
expects from software installed by make or even the distro package
manager. Everything has its proper place and there is simply no need for
relocation after installation. That is the beauty of it all in a
UNIX-ish world. Plus, there is /opt.

> Here's a definitive answer from the KDE side at least: *yes*, "regular 
> software" uses XDG_CONFIG_DIRS and XDG_CONFIG_HOME to find all the config 
> files it's looking for (both in the application code and in the library code).

Still incomplete: I get the environment stuff, but not the *its own
config* part. ;) Because for that one only needs to make sure to
put/expect it in the right location and save that path at
compile/installation time. Everything else is just making things overly
complex for no apparent gain or reason. Every LOC not written to deal
with situations that are very unlikely to arise is a win. There are
plenty possibilities to deal with those outside the code, or as Eric S.
Raymond wrote in "The Art of UNIX programming": "Put the intelligence
into the data (structure) not the code." I am paraphrasing, but that's
the gist of it.

Just to be extra clear: I don't want to bash on XDG, I very much like
the intention and the (partial) results that have come of it, i.e. less
dotfile clutter in my $HOME or $XDG_DATA_HOME etc., I even bent my
shell's config to use $XDG_CONFIG_HOME. But I am also sad that some
things apparently have been overlooked. So there is still room for
improvement and I hope the authors agree, seeing that the current
version still starts with 0 (0.8).


Best,
PW