A common VFS and a Common conf-system [Part II]

[Sean Middleditch]

On Wed, 2005-03-09 at 22:04 +0900, nbs_r at gazeta.pl wrote:
> While I fully agree with load/store D-VFS semantics, a POSIX-compliant 
> fall-back for _local_ document copies could be very useful. What I mean, 
> is we should not restrict operations that can be done with already 
> downloaded documents. If an application wants to "mmap" or "seek" the 
> local document copy by itself D-VFS should not interfere with this. What 
> I humbly propose is:
> 
>    1. "Load" checks a document out of the D-VFS backend. If necessary 
> (i.e. the backend is not a "file://" protocol) the document is copied to 
> a local directory(*1). The file may (but does not have to) be 
> automatically mmaped to the shared memory if the application requests 
> that. Otherwise, the path of the local copy is made available to the 
> application.
>    2. The application may open the local copy and use any POSIX 
> operations on it. The D-VFS specification should not introduce any 
> constraints other than reflecting the access rights of the remote copy 
> in the attributes of the local one.
>    3. "Store" closes the local file and checks the document in the D-VFS 
> backend. Assuming we do not provide caching ability the local file can 
> be deleted. However if an architecture with a daemon is to be used 
> documents caching implementation can be quite easy and reliable.

Downloading whole copies of remote files is not a good idea.  Remote
files could potentially be HUGE.  One of the more interesting use cases
for mmaping documents is to be efficient.

A solution like FUSE can emulate mmap'ing; a plain library cannot
(feasibly).  An operation whose entire point of existence isn't possible
in D-VFS, even if the API can be emulated, makes little sense to
provide.

If an applications *really* wants to mmap a file, it can open and cache
the file itself.  I would strongly urge application authors not to do
this, though.  Large files would have slow access speeds, modification
of mmap'd files would not be capable of working *anything* like it does
normally with POSIX, and on many systems it could outright fail due to
disk space limitations even in /tmp (think of dumb terminals that share
a wee bit of disk space on the central server and rely on D-VFS/NFS to
access home directories/documents).

>    Why using local copy/reference:
> - This method provides a robust and efficient fall-back for "file://" 
> backend.
> - Porting applications that strongly depend on POSIX API will be easier.
> - It is necessary for document caching.
> - POSIX API can be used _only_ for local copies, therefore it does not 
> clutter the (remote) D-VFS semantics.
> 
> *1) A bidirectional path<->URI mapping convention will be necessary.
> 
> A fall-back utility for D-VFS-unaware programs:
> Combining the above concept with a wrapper utility we can enable 
> POSIX-only applications to benefit from D-VFS. There are two 
> possibilities that comes to my mind:

For Linux, use FUSE.

>    1. Explicit file download. The sample wrapper command invocation 
> could be like this:
> $ dvfswrap [-R] [URI_cache_pattern] URI program [program_args] {}
> ("{}" is a "find"-like substitution pattern). The "-R" option and 
> URI_cache_pattern explicitly provide dependent files for the "program".
>    2. On-demand file download. The wrapper could intercept open/close 
> function calls and download all the necessary files. In this case a user 
> does not have to specify the dependent files so the command invocation 
> could look like:
> $ dvfswrap URI program [program_args] {}
>    In both cases the wrapper utility should intercept termination of the 
> program and "store" the changed document(s) if necessary. As command 
> line programs often run from scripts, they will particularly benefit 
> from document caching.
>    While this is an optional functionality, (IMHO) it would be useful to 
> provide a coherent way to wrap the POSIX-only programs.
> 
> What is unrelated to the above but still quite important (and I did not 
> find it in the wiki) is a possibility to query the D-VFS backend about 
> the implemented interfaces/operations. Obvious choices for such 
> interfaces are:

That should be there.  It's definitely a necessity in certain contexts.
I want it to be very, very, very rarely needed.  If we add all sorts of
APIs to D-VFS that only work on one or two backends, then apps will
start relying on those features, and you end up defeating one of the two
primary purposes for using D-VFS.  If apps don't depend on those
features, there's no reason to have them, since they are unnecessary.

> - CD/DVD burn/erase interface,

I don't believe this is necessary in D-VFS.  I'd rather see a "standard"
CD/DVD burning library.  The entire idea of how burning a disk works is
fundamentally different than the random-access nature of any other
protocol, and isn't really a good fit.

> - Check-in/check-out/diff/... for revision control systems,

That is essentially a "read" and "write" operation, since D-VFS is based
on document-APIs and not byte range APIs (although the later will be
available as they're necessary evils).  Added to locking, you could
easily turn CVS/Subversion/etc into a virtual file system.  Revision
control is one of the "we need more information" items - what other
kinds of revisioning file systems are out there, what should the API
really look like, etc.  I think revisioning file systems are a really
good thing, but it's better to just punt those features for 1.0 if we
can't be reasonably sure that we can get them Right(tm).

> - Backend options/logs, etc.
> As hardcoding all the possible operations is impossible, a D-VFS backend 
> should advertise non-standard interfaces to the client application. Then 
> the application can simply add backend-specific GUI controls to the 
> menu/panel. I am not sure if it needs to be specially addressed in the 
> D-VFS specification or D-BUS will make it for free.

That's not a good thing at all, IMO.  I'd much rather see any and all
interfaces defined clearly by D-VFS.  If some backend comes along and
manages to invent some radically new operations that no-one else has
thought of before, they should get D-VFS updated.  If nothing else, thin
of what happens when two backends both offer a very similar feature but
invent their own API/interface to use it.  By requiring that D-VFS
support it we make sure that that doesn't happen.  Since D-VFS should be
upgradable to a new minor revision without breaking old applications
(API/ABI/protocol stability is absolutely vital to me) this shouldn't be
a problem.

> 
> Regards,
> Andrzej
> 
> 
> 
> _______________________________________________
> xdg mailing list
> xdg at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/xdg
> 
-- 
Sean Middleditch <elanthis at awesomeplay.com>