Request for clarification on menu/file spec

[Mike Hearn]

On Tue, 07 Feb 2006 15:56:54 -0800, Waldo Bastian wrote:
> If there is concensus that that is the right long term direction and that the 
> benefits outweigh the disadvantages then I guess we should go that way. I 
> would like to hear some more cheers of support for that direction first 
> though.

-1 from me (autopackage maintainer).

I long ago came to the conclusion that the prefixes system UNIX uses sucks
in fundamental ways. The /usr vs /usr/local decision which autopackage
gets so much flak for is really the least of our worries:

 * Global prefixes like /usr are not namespace managed. LANANA tried to
   fix this and everybody ignored them. How can I install a program called
   WonderWidget when some random dead-since-2002 project on SourceForge
   may have already claimed that name?

 * Every prefix except /usr is pretty much broken on 99% of distros. It
   was nice to see YOPER and SUSE work on fixing this, but bugs filed 
   against Gentoo, Red Hat and so on were all either ignored or closed 
   as WONTFIX. Adding new prefixes is so painful, and so prone to making
   mistakes (for instance Stanislavs list seems to miss
   BONOBO_ACTIVATION_PATH and DBUS include paths), that it's really not
   a scalable solution.

If I had total freedom to implement a solution, I think I would do
something like this:

 * Every user-interesting program is installed to /software/$X as its
   prefix, where $X is a name generated by the system not selected by
   the software authors. It could be totally random, a GUID or something,
   but for optimal command line usability it'd be better to have it 
   related to the software name/version.

   /software is not meant to be end-user visible like /Applications is
   on MacOS, it's a system internal thing.

 * What is currently in /usr is put into /system

 * /usr is set up to be the union mount of /system and /software. This can
   be done with unionfs today for instance. Exactly what is mounted into
   /usr is managed by a simple daemon that whacks a file notification
   watch on /software, and as directories are added/deleted from here they
   appear and disappear from /usr

This has a few advantages over the current scheme, which I'll explain
in a sec, but obviously it requires quite significant changes to the
way the distro is set up. But as we're tossing around alternatives and
the current system is badly flawed why not investigate "extreme"
solutions?

OK so the advantages are like this.

Firstly, whilst you can still get namespace conflicts in /usr, they
are not fatal. Today if I wish to install two programs with the same
name either due to accidental collision or because I wish to test a
CVS build but keep my stable build around too then I must choose one
to be the "primary" install, and then the other one is installed to
some other prefix and ignored by the desktop. This is because
/usr/share/inkscape can only be owned by one program at once.

But in the union mount scheme, if I had two versions of Inkscape
installed at once, they would (because they have been made binary
relocatable using our spiffy APIs, natch) resolve
$PREFIX/share/inkscape to /software/inkscape-0.43cvs/share/inksape and
/software/inkscape-0.42/share/inkscape respectively. So with simple
modification they would be able to run side-by-side.

If we want to avoid that modification entirely, then it is simple to
modify the OS so /proc/self/prefix is a directory that resolves to
wherever the program is installed to, then you can
./configure --prefix=/proc/self/prefix. Pick your poison. More and
more apps are getting relocatable these days as they're ported to Windows
anyhow.

OK, so we still have the problem of /usr/bin which is presumably in
the PATH conflicting. Union mounts don't die when faced with this, the
earlier mount overrides the later mount. So by fiddling the settings
in my simple daemon, I can control which package "owns" the inkscape
command/manpage. And of course my little daemon is free to add symlinks with
more precise names similar to how it chooses a name for the /software/$X
directories.

OK but we STILL have the problem of conflicting namespaces for things
like menu entries. Nobody has to do this, but in practice, everybody
calls their menu entry wonder-widget.desktop even though the desktop
doesn't care what it's called. So we can solve this EITHER by having
desktop-file-install randomize the names ensuring no conflicts. OR we
can modify the desktops so they can scan every directory under
/software and don't care about duplicated names.

Alright, where are we up to so far? We have:

 * Either introduced /proc/self/prefix OR made each app binary
   relocatable. We can now easily have multiple programs with
   the same name installed to different directories at once

 * Union them mounted over the distros provided /system directory
   to form /usr, which is there for backwards compatibility and
   so the various $FOO_PATHs don't get too long.

 * Written a little daemon with GUI that lets us order the priority of
   apps that may fight over namespace. Our desktop understands how to
   show multiple apps from the /software directory in the menus,
   maybe KDE/GNOME get smart enough to show the version number when
   two Inkscapes are in the menus at once!

But we're not done yet.

Every shipping OS today has a software database as part of its design.
The nature of this database varies but it always exists, and it exists
because filesystems (which are databases themselves) can in SQLish
terms only query on one column. So we can choose to make the filesystem
so it queries EITHER files that are logically related in packages OR
files of different types but not both. Every OS needs the ability to
query both though, so we introduce an extra database to solve this.

On Windows, the filing system groups files by package (in Program Files)
and the database which lets you query things like file associations is
called the registry.

On MacOS/X the filing system groups files by package/bundle, and the
database has no specific name but is held in memory by LaunchServices,
and is compiled from Info.plist files in the bundles on startup.

On Linux we do things the other way around, the filing system lets us
rapidly query for file associations, named help pages etc, but it doesn't
let us see which files are related to each other by package. For that we
have an RPM database, a DPKG database, and so on.

There's no real need for this these days, because we have union
mounts, which let us make the filing system the database for both
columns. In other words, we can do what we do today and query for a
filetype icon by
/usr/share/icons/hicolor/24/mime/x-application-wonderwidget.png AND we
can also query by package /software/wonderwidget-1.0/*

(NB: This sort of thinking is inspired by Hans Reisers papers on namespace
 unification and design. More here: http://www.namesys.com/whitepaper.html)

An RPMDB may still be useful for managing the stuff in /system, this
is up to the distro vendor. But there's no need to tell it about the
type of user-installed third party software which triggered this thread.
This means that uninstall becomes literally "rm -rf /software/inkscape":
the simple daemon will notice that the directory has vanished and remove
the union mount, the desktop will be informed that /usr/share/applications
has changed, so menu entries will disappear, and the system cleanly unwinds.

So by doing all this we solve many problems that hurt Linux today:

 * There is no namespace management, causing nasty accidents as names
   conflict
 * You can't easily install two versions of the same program at once
   and expect the desktop to work right
 * Installing 3rd party software is awkward, because people want it in their
   RPM/DPKG/FooixManager database, but implementing this is too hard
 * It solves the endless arguments over /usr vs /usr/local vs /opt vs ~/.local
 * It lets admins easily merge apps mounted over the network or from extra
   hard disks into the system, that would otherwise appear in the FS as /appserver
 * In future it may allow for MacOS X style drag/drop software management, which
   nicely solves the problem of menus being self-organising (so I can never find
   where the menu entry appeared after I install a program).

OK, so this is pretty crazy I know, and you could argue that maybe freedesktop.org
should not be trying to impose such surgery upon distros. But IMHO xdg-list has
historically done what was right, rather than what was the path of least resistance.
It'd be worth at least discussing this sort of change.

thanks -mike