virtual filesystem ideas

[Ken Deeter]

> 
> I don't like this proposal. All the message passing looks slow, due to
> context switches and unnecessary work. The way observers are used seems
> to mean every single vfs operation will end up with a whole bunch of
> out-of-process messages back and forth. 
>

Well first of all, I didn't mean for it to be a complete proposal. I
realize there were a lot of things left out, but to me it seems the
plumbing might be the most important part. As for performance, I'm not sure
how d-bus or any of the other things perform in this kind of case, so
I'm afraid I can't address that point as of yet. I only meant to take advantage
of the flexibility that a system like d-bus does allow.

I should mention that I didn't mean for every operation to be required
to have an observer. You could always have an option to pass in a null
observer which would make no message passing happen at all.
 
> The way this is set up doesn't look much like a vfs to me, more like a
> download manager of some sort, which you tell to download files in the
> background. It mentions status dialogs and all sorts of things that
> should happen on a higher level. VFS operations are on a much lower
> level (read(), readdir(), write(), seek(), rename() etc) and if they
> return an error it is only the app that knows what to do. 
> (i.e. We don't
> want status dialogs while reading a configuration file on statup.)

I'm not sure that's really the case. For example, in kde, if you pass a
sftp:// URI to kate, then you do get a little status box that shows the
accessing of the remote file. Now whether that is user-friendly or not,
well it looks like the desktops disagree at this point. I don't think my
design forces you to pop up dialog boxes if you don't want to.

I don't think anything in what I said meant to ignore the basic vfs
operations. Presumably, that would happen between the client library and
the vfs server. Whether that is efficient or not I do not know. But it
can't be that much worse than the other userland fs's out there, which
presumably have a app -> kernel -> user vfs -> kernel -> app kind of
data path. If you wan't out-of-proc vfs then you're gonna run into it at
some point.

Also, 'only the app knows what to do in an error': I'm not sure that
with all the different kinds of situations that come up with network
fs's and such, that every app should have to deal with every kind of
error. Instead of every app implementing dialog boxes for every kind of
network error possible, I would think the more sensible thing to do
would be able to capture these error in a central location and have some
consistent way of dealing with them. The app can override this behaviour
if it wants to, but doing it that way seems better than every app having
to deal with every possible situation on its own. An app should receive
simple indiciations of an error, like how fopen() returns a NULL. If it
wants to know more, then it can find out, but it if it doesnt, then we
can avoid the case where the app seems like its doing nothing because no
dialog or anything pops up, or that it just pops up a box saying "could
not open file" for any one of a 100 different errors.

> The proposal even mentions that the most important thing a vfs is for
> (reading data) isn't really specified in the proposal.
> 

Again, it wasn't meant to be a complete proposal. Notice i deliberately
just said 'ideas' instead of 'proposal' or 'full out solution'.

As for reading data, well it the main debate in my head right now is
whether the client app and vfs backend should communicate directly, or
whether it should come through the vfs server. Maybe the vfs backends
don't even have to be out-of-proc from the server, though something in
me tells me that it would be much easier to maintain if they were. in-proc
backends means API/ABI stability requirements, especially w/ 3rd party backends,
where as if they are just other services on the D-BUS then they only need to
implement a consistent messaging interface.


> It seems like the proposal is all about plumbing, and there is no
> specifics about how vfs operations are supposed to work, or any mention
> about the issues related to vfs implementations: asynchronity,
> cancellation, performance, API, threads vs processes, hanging NFS
> mounts, how uris look, filename encoding, how vfs backend chaining works
> (i.e. reading the contents of a zip file from a smb mount), seekability,
> streams, etc.
> 

well since you ask, i'll describe my ideas:

asynchronicity: this part of it is essentially delegated to d-bus. Since
d-bus has seemed to figure out how to work async message passing into
both glib and qt at the same time, I don't think we should re-invent the
wheel.

cancellation: presumably this would be part of the interface of the vfs
server. Since it would know about the various transactions going on at
the moment, it should know how to cancel one.

performance: I don't know. My original ideas concentrated more on flexible
design. If you are going to say that there is too much message passing,
at this point, that is as arbitrary as me saying this will perform well.

API: I think the plumbing or general architecture is a bit more high
level and important to think about before you start working out API
details. Maybe thats just me. I think once we figure out what kinds of
features we want, then we start talking API. But most of the time its
fairly obvious is an architecture is going to support a particular
feature well, so I don't see the harm in discussing architectures before
we work out details.

how uri's look: since there even seems to be RFC's regarding this issue,
it would probably be wise to stick to the protocol:// type URI's

backend chaining: I would delegate this functionality to the vfs server.
Hopefully the client side wouldn't have to worry too much about how
exactly to set up various chains. It should just worry about URI's,
and whether they represent something that is a file, or a directory,
or maybe even both.

filename-encoding: I didn't address this at all. It would seem to me like
the practical thing to do would be to make all transactions based on
unicode.. but perhaps a local filesystem backend could tell if it was
supposed to be writing files in some other encoding, it would do the
appropriate translation. Or perhaps, each backend could have some
method to ask which encodings it can support, and the vfs server
would contain the necessary intelligence in dealing with filenames.

seekability: I imagined something where the client lib would ask the vfs
server if a particular URI was seekable. The vfs server could then
either look in its cached table of vfs attributes, or ask the backends
themselves and pass the result back.

streams: this would probably be something like the seekability issue,
perhaps a way for an app to ask the vfs whether a URI can be opened as
a stream?


Again, I didn't by any means to have every single issue worked out, so I
understand your disappointment if thats what you expected. If that is
what is expected on this list, then my posting was definitely premature.

It seems like there are a lot of separate interests with regards to this
topic so I didn't want to go too far in hashing it all out.

------

 +----------------------------------------------+
(  Ken Deeter (Kentarou SHINOHARA)             (
 )                                              )
(  "If only God were alive to see this.. "     (
 )                             -Homer Simpson   )
+----------------------------------------------+