[Fwd: Re: virtual filesystem ideas]

[Alexander Larsson]

This was sent in private by mistake. Forwarding to the list.

-----Forwarded Message-----
From: Alexander Larsson <alexl at redhat.com>
To: Ken Deeter <ktdeeter at alumni.princeton.edu>
Subject: Re: virtual filesystem ideas
Date: 22 Sep 2003 12:43:30 +0200

On Mon, 2003-09-22 at 10:53, Ken Deeter wrote:
> > 
> > I don't like this proposal. All the message passing looks slow, due to
> > context switches and unnecessary work. The way observers are used seems
> > to mean every single vfs operation will end up with a whole bunch of
> > out-of-process messages back and forth. 
> >
> 
> Well first of all, I didn't mean for it to be a complete proposal. I
> realize there were a lot of things left out, but to me it seems the
> plumbing might be the most important part. As for performance, I'm not sure
> how d-bus or any of the other things perform in this kind of case, so
> I'm afraid I can't address that point as of yet. I only meant to take advantage
> of the flexibility that a system like d-bus does allow.
> 
> I should mention that I didn't mean for every operation to be required
> to have an observer. You could always have an option to pass in a null
> observer which would make no message passing happen at all.

So, can you give an example when the observer would be used?

> > The way this is set up doesn't look much like a vfs to me, more like a
> > download manager of some sort, which you tell to download files in the
> > background. It mentions status dialogs and all sorts of things that
> > should happen on a higher level. VFS operations are on a much lower
> > level (read(), readdir(), write(), seek(), rename() etc) and if they
> > return an error it is only the app that knows what to do. 
> > (i.e. We don't
> > want status dialogs while reading a configuration file on statup.)
> 
> I'm not sure that's really the case. For example, in kde, if you pass a
> sftp:// URI to kate, then you do get a little status box that shows the
> accessing of the remote file. Now whether that is user-friendly or not,
> well it looks like the desktops disagree at this point. I don't think my
> design forces you to pop up dialog boxes if you don't want to.

In the ideal world, apps would use the vfs for all file access. Not all
file access is of the form "user asks the app to load a file". Much of
it is things that happen automatically, in the background, or as small
parts of everyday program work. When loading an icon for display you
can't have the vfs automatically poping up dialogs for you. And in the
case you want to have status while loading, chances are that you don't
want a standard dialog, but instead want to use some specific UI feature
of your app (throbber, status bar in window, etc).

> I don't think anything in what I said meant to ignore the basic vfs
> operations. Presumably, that would happen between the client library and
> the vfs server. Whether that is efficient or not I do not know. But it
> can't be that much worse than the other userland fs's out there, which
> presumably have a app -> kernel -> user vfs -> kernel -> app kind of
> data path. If you wan't out-of-proc vfs then you're gonna run into it at
> some point.

It is worse with dbus. For the case above the kernel switches are not
context switches. With dbus everything will go through the dbus daemon,
so you'll get: app -> dbus daemon -> vfs server -> dbus daemon -> app.
That is 4 context switches as opposed to the 2 that are "needed" for
out-of-process communication.

> Also, 'only the app knows what to do in an error': I'm not sure that
> with all the different kinds of situations that come up with network
> fs's and such, that every app should have to deal with every kind of
> error. Instead of every app implementing dialog boxes for every kind of
> network error possible, I would think the more sensible thing to do
> would be able to capture these error in a central location and have some
> consistent way of dealing with them. The app can override this behaviour
> if it wants to, but doing it that way seems better than every app having
> to deal with every possible situation on its own. An app should receive
> simple indiciations of an error, like how fopen() returns a NULL. If it
> wants to know more, then it can find out, but it if it doesnt, then we
> can avoid the case where the app seems like its doing nothing because no
> dialog or anything pops up, or that it just pops up a box saying "could
> not open file" for any one of a 100 different errors.

Sure, it is nice if there is some common code for doing some thing. But
there are so many different ways an error can be handled. Sometimes the
app expects an operation to fail, for instance when searching for a file
through a set of paths by calling open in each path. And even when a
failure is really a failure its almost always not interesting what the
lowlevel error was, but instead the i/o error needs to be propagated a
few levels up in the application where it can be either handled by doing
something different of a dialog with a much higher level description can
be presented to the user. 

> > The proposal even mentions that the most important thing a vfs is for
> > (reading data) isn't really specified in the proposal.
> > 
> 
> Again, it wasn't meant to be a complete proposal. Notice i deliberately
> just said 'ideas' instead of 'proposal' or 'full out solution'.

Yeah, maybe I was harsh on you due to this, but I just think the dbus
direction is wrong, and will cause us problems, while not solving really
many issues.

> As for reading data, well it the main debate in my head right now is
> whether the client app and vfs backend should communicate directly, or
> whether it should come through the vfs server. Maybe the vfs backends
> don't even have to be out-of-proc from the server, though something in
> me tells me that it would be much easier to maintain if they were. in-proc
> backends means API/ABI stability requirements, especially w/ 3rd party backends,
> where as if they are just other services on the D-BUS then they only need to
> implement a consistent messaging interface.

The gnome-vfs backends are in-process, but I'm moving some of them
out-of-process, since then cached data, authentication and connections
can be shared, and the backends get a much more well-specified
environment to run in. I think a common vfs system has to be out-of-proc
to have any chance to work, but that may mean bad performance for local
files, slowing down things like the file manager...

> > It seems like the proposal is all about plumbing, and there is no
> > specifics about how vfs operations are supposed to work, or any mention
> > about the issues related to vfs implementations: asynchronity,
> > cancellation, performance, API, threads vs processes, hanging NFS
> > mounts, how uris look, filename encoding, how vfs backend chaining works
> > (i.e. reading the contents of a zip file from a smb mount), seekability,
> > streams, etc.
> > 
> 
> well since you ask, i'll describe my ideas:
> 
> asynchronicity: this part of it is essentially delegated to d-bus. Since
> d-bus has seemed to figure out how to work async message passing into
> both glib and qt at the same time, I don't think we should re-invent the
> wheel.
> 
> cancellation: presumably this would be part of the interface of the vfs
> server. Since it would know about the various transactions going on at
> the moment, it should know how to cancel one.

This ties in with asyncronicity. There had to be a way to refer to an
outstanding asynchronous job, so you can cancel it (or a group of jobs).
Gnome-vfs has a whole scheduling subsystem to handle this, including I/O
priorities and group cancellation.

> performance: I don't know. My original ideas concentrated more on flexible
> design. If you are going to say that there is too much message passing,
> at this point, that is as arbitrary as me saying this will perform well.
> 
> API: I think the plumbing or general architecture is a bit more high
> level and important to think about before you start working out API
> details. Maybe thats just me. I think once we figure out what kinds of
> features we want, then we start talking API. But most of the time its
> fairly obvious is an architecture is going to support a particular
> feature well, so I don't see the harm in discussing architectures before
> we work out details.
> 
> how uri's look: since there even seems to be RFC's regarding this issue,
> it would probably be wise to stick to the protocol:// type URI's

Thats not enough though. Both kde and gnome have extensions to the
availible RFCs.

> backend chaining: I would delegate this functionality to the vfs server.
> Hopefully the client side wouldn't have to worry too much about how
> exactly to set up various chains. It should just worry about URI's,
> and whether they represent something that is a file, or a directory,
> or maybe even both.

I don't care much about how the stacking is implemented, but how is
chaining exposed. gnome-vfs uses '#' to chain uris, libferris treats
e.g. zips as directories.

> filename-encoding: I didn't address this at all. It would seem to me like
> the practical thing to do would be to make all transactions based on
> unicode.. but perhaps a local filesystem backend could tell if it was
> supposed to be writing files in some other encoding, it would do the
> appropriate translation. Or perhaps, each backend could have some
> method to ask which encodings it can support, and the vfs server
> would contain the necessary intelligence in dealing with filenames.

Unfortunately you can't use unicode or any other encoding for the
low-level filenames, since that doesn't allow you to express the set of
all allowable filenames (and thus, you can't e.g. rename a non-utf8
filename to a utf8 one to fix it, because the vfs doesn't allow you to
express the current name). Furthermore its often impossible to know what
encoding is on the disk (e.g. for a remote ftp site), so you can't
convert to/from utf8. How this is handled differs between KDE and Gnome,
and gnome even lets you configure it manually using env-vars. The
solution has to be a way to ask the vfs for "display filenames", that
can be used in the UI (being unicode or at least a well defined
encoding).


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
                   alexl at redhat.com    alla at lysator.liu.se 
He's an immortal zombie farmboy in drag. She's a bloodthirsty streetsmart 
research scientist with a knack for trouble. They fight crime! 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
                   alexl at redhat.com    alla at lysator.liu.se 
He's a Nobel prize-winning coffee-fuelled dwarf living undercover at Ringling 
Bros. Circus. She's a virginal cigar-chomping hooker married to the Mob. They 
fight crime!