continued: Common-VFS proposal

[Sean Middleditch]

On Wed, 2005-01-19 at 11:09 -0500, Zack Rusin wrote:

> "Corba for IPC (DBUS can't be used here, because this requires 
> heavy-weight IPC)" is something I'm considering putting in my list of 
> signatures.

Heh.  I actually started drafting a D-BUS based VFS system based on the
earlier discussions of the gnome-vfs/kioslaves bridging.  The D-BUS
documentation stopped me cold, so I'll start back up on that again once
D-BUS and its docs are a little closer to 1.0.

> 
> But again, I very much appreciate your effort but as of now you just 
> don't have enough knowledge to implement it. You are of course more 
> than welcome to try and experiment but as it stands you could never 
> realistically expect that we would even consider using it. The most 
> realistic thing you can hope for is that people who either were or are 
> actually involved in creating VFS layers and desktop environments get 
> together on one of the conferences and nail this out. It's not going to 
> happen anytime soon though.

Honestly, the problem is *really* simple.  It's just a matter of doing
the legwork.  If we have someone who is willing to do that work (and
Norbert is being a real trooper here!) we shouldn't call them ignorant
and tell them to not bother, but instead help out with guidance.

That being said, Norbert:

First, what *is* with this "D-BUS won't work" deal?  That makes no
sense, and you provide no explanation.  D-BUS is absolutely *perfect*
for our needs.  Certainly something *more* heavyweight - like CORBA - is
going to be worse.  VFS operations are very common, they need to be
fast.

The model I had in mind looked something like this:

A client library is provided that offers basic VFS operations: open,
read, write, rename, move, setattr, getattr, setaccess, getaccess, etc.
The idea is to *not* model POSIX but instead model what real
applications do.  There's no "move" function in POSIX (rename is very
different than a real move operation), but apps want to be able to move
files.  Therefor, move should be provided an actual function.

File names will be URIs.  I'm also thinking about offering a second
syntax for things like virtual locations (i..e, floppy:// or fonts:// or
whatever - those are *NOT* real URIs, they shouldn't pretend to be) and
for filters (lots of uses - use your imagination).

Basically, virtual locations would simply map to a some other filename.
trash: would map to ~/.trash/, so trash:file.doc would just be
reinterpreted as ~/.trash/file.doc.  Filters are a little more complex
and I planned on leaving them for post-1.0.  (You'll see why they're
complex in a bit.)

Now, for local files (file:// URIs) the library would just call the
normal libc functions in most cases and provide code for things like
move that wrapped the usual POSIX API.  The reasoning here is that local
file access should be just as fast as when using the POSIX API.

When a remote URI is accessed a daemon would be contacted.  I didn't
plan on using the D-BUS bus for this; the bus doesn't buy us anything
but a performance degradation as data is passed between three process
address spaces.  Instead I was going to use the session bus for
negotiation and such and use a direct connection (which D-BUS supports)
to a per-session daemon (started and found using the bus and service
activation).

The daemon would cache credentials for URIs.  Mostly it would just load
a bunch of plugins and marshal any file requests to those plugins, and
return the responses.  Basically what you'd expect.

The plugin API would provide a vtable of operations.  Each entry could
be set to "default", "disable", or over-ridden with an actual function.
The default vtable would mostly be "disable" entries but with any
logically composite operations (like move) having a default
implementation (in move's case, it'd be comprised of copy-and-delete).
What that does is allow plugins to be written that implement only the
most basic of features and still support more complicated operations.
For protocols that have more advanced support for those operations, the
plugin can include code using those features.  (i.e., an ftp plugin
could just call the ftp mv command.)

A plugin would list the vtable with a size, allowing the daemon to add
new methods to the vtable without breaking binary compatibility with the
plugins - if a method past the end of a plugin's vtable is required the
default is automatically chosen.

The entire library and daemon suite would be designed for as low
resource requirements as possible and keep as clean and UNIXy an API as
possible.  The idea was to basically push this as a general, modern UNIX
VFS API, with my ego letting me believe that in 5 or 10 years this might
become a de facto standard across all UNIXes - which of course requires
the same of D-BUS, which I also hope happens.  ;-)

The big problem is the event handling, especially when you realize that
you're going to end up needing asynchronous I/O and probably some kind
of monitoring as well.  Using D-BUS mostly solves this as it is already
possible to integrate with any mainloop, has glib support to do so, and
Qt integration is definitely possible.  A lot of CLI apps and such
probably just wouldn't use any asynchronous operations.  The only
problem comes from having local files handled in the library/application
itself - local files would have distinct descriptors while all remote
connections would be handled over one connection to the daemon;
basically it's a problem of knowing which file descriptors to pass to
select()/poll()/epoll/kqueue.  One solution is to provide an API to
obtain a list of all descriptors in use by the VFS library.  Another is
to simply state that each open file has on descriptor, and in the case
of remote files use a separate daemon connection for each file (yuck).
The final is to just document the problem clearly and punt to the apps.
Since the problem only happens in asynchronous operations anyhow, which
happen mostly in GUI mainloops, each of which would have its own
simplified wrapping around the VFS, I'm leaning towards the final
solution.

The bridging of gnome-vfs and kioslaves just is *not* the correct
solution here.  The first is the simple design - I'm a picky bastard,
and to me, design and cleanliness are pretty important.  ;-)  A clean
implementation of a VFS is definitely preferred in that department.

Second is the usefulness of a gnome-vfs/kioslaves bridge.  That only
helps GNOME and KDE apps.  GnuStep, Rox, CLI apps, and so on *will not*
depend on an architecture built on GNOME or KDE libraries like that.
Even if there were no technical reason not to, there's political
reasons.  Yay humanity.  ;-)