A common VFS and a Common conf-system (Was: namespacing)

[Sean Middleditch]

On Tue, 2005-03-01 at 01:05 +0000, Jamie McCracken wrote:

>If you want something thats fast, easy to use and KISS that could take 
>advantage of this then look at FUSE (http://fuse.sourceforge.net/) - it 
>rocks!.

Unfortunately, it's Linux only, and has an incredibly brain-dead design
where you have to utilize "round-trips" through the kernel for something
that essentially happens entirely in user-space, and forces a superbly
poor API (POSIX) onto all its file systems, including those where POSIX
makes absolutely no sense whatsoever.

A real solution must be portable, and by that extension completely in
user space, and should have an extensible API designed around
Desktop-oriented file access.

Several excellent design summaries have been posted to the list on this
topic already - search them up.

In all honesty, I don't think it is at all a complex project to get
working; at least not the basics.  Especially not when you have working
code from GVS and KIO to pull from.  The absolute most essential part is
getting the API right.

Heck, here is a super-quick proposal just to break ice (maybe this
should go on the Wiki somewhere?).

Target profile for Desktop VFS 1.0:
 * Read whole documents including meta-data
 * Read just document meta-data
 * Delete files and folders
 * Move files and folders between locations on the same server
 * Create folders
 * Write whole files, create or replace
 * Append to files
 * Set meta-data

I'm going to use the prefix vfs_/VFS_.  However, that is pretty common,
and could be in wide use, especially internally to many apps.  Since
this is the Desktop VFS, perhaps the prefix dvfs_ should be used
instead.  The whole thing could be called D-VFS, and then we get cool
points for the naming conformity with D-BUS.  ;-)

Any operations defined below that are not supported by a backend will
returned not-supported errors.  There should most definitely be a
function to query the capabilities of a particular backend, although I
did not include one below.

Also note that I didn't define the data items in the vfs_metadata_t
type, although doing so is very important.  I'll get to those later.

The following are some incomplete types (pseudocode, I guess):
 vfs_uri_t: opaque type representing a URI
 vfs_meta_data_t: opaque type representing an associate array of
meta-data items
 vfs_error_t: integer for storing VFS errors, VFS_EOK = 0 is success
 vfs_handle_t: opaque handle for a pending operation; reference counted.

Following are pseudo-codeish function prototypes.  They are by no means
the whole list of function prototypes, or even all of the important
ones, because it's really late and I don't have time to write them all.
If nobody has told me to just shut the hell up by tomorrow, I'll post
more and irritate you all again with my annoying pointless commentary.

 /** Initializes the VFS library.  This may be called multiple times,
  *  to ensure that apps with multiple libraries using the VFS library
  *  will work.  This function might connect to a VFS daemon or do
  *  other initialization work.  (RFC: should this be implicit in the
  *  other VFS calls?  That makes the API look simpler, but could make
  *  error handling more complicated, potentially.)
  */
 vfs_error_t vfs_initialize (void);

 /** Gets a file-descriptor (socket) which is to be polled as part of
  *  the application's main event loop.
  */
 int vfs_get_poll_fd (void);

 /** Gets the poll flags for the file descriptor returned by
  *  vfs_get_poll_fd.  Will always have POLLIN set, may have
  *  POLLOUT set.  (RFC: would it be better to define our own
  *  flags here instead of reusing those from the poll syscall?
  *  Re-using them makes coding poll-based event loops easier,
  *  but really, how many apps code their own event-loops these
  *  days?
  */
 int vfs_get_poll_flags (void);

 /** Decrements the count on a handle, and frees the resources of the
  *  handle if the count reaches zero.
  */
 vfs_error_t vfs_release (vfs_handle_t* handle);

 /** Increments the count on a handle.
  *  (RFC: stupid function name, but it's late and I'm not thinking:
  *  what's a better name for this?)
  */
 vfs_error_t vfs_reserve (vfs_handle_t* handle);

 /** Reads the file contents and meta-data from the given URI.  This
  *  will read in the _whole_ file.  It will allocate the buffer
  *  space and set the size parameter to the file size; the caller must
  *  free the buffer.  This is not a recommended function, but is a
  *  convenience for apps where it makes sense.  Please use
  *  vfs_begin_read instead if possible.  (RFC: this is a convenience
  *  function, mostly intended for apps without mainloops like most
  *  CLI apps.  Should this be part of the main library, or part of a
  *  very clearly defined and equally standard helper library?
  *  Functions like this would be needed for every operation in the
  *  main library.)
  */
 vfs_error_t vfs_read_atomic (vfs_uri_t* uri, vfs_meta_data_t* metadata,
char** buffer, size_t* size);

 /** Begins a read.  The callback will be invoked once after the
  *  meta-data is loaded, periodically as chunks of the file data is
  *  read, and once when the operation is complete or canceled.
  *
  *  vfs_read_callback_t is the callback type, defined as
  *    void (*vfs_read_callback_t)(vfs_handle_t* handle, vfs_metadata_t*
  *      data, char* chunk, size_t chunk_size, vfs_read_state_t state,
  *      void* userdata);
  *  vfs_read_state_t is an enum of VFS_READ_STATE_BEGIN,
  *     VFS_READ_STATE_DATA, VFS_READ_STATE_EOF, VFS_READ_STATE_CANCEL,
  *     VFS_READ_STATE_ERROR.
  *
  *  The metadata will be freed after the last call to the callback, so
  *  any data that app wants to keep must be copied from it by then.
  *  The buffer is reused for each call, so its contents must be copied.
  *
  *  The handle begins with a reference count of two, unless the handle
  *  parameter is NULL, in which case it starts with a count of one.
  *  After the last call to the callback, the reference count is
  *  reduce by one.
  *
  *  Note that something like a progress meter can be done by querying
  *  the meta-data for the file-size and, if its known, comparing the
  *  total bytes read so far to that total size.
  */
 vfs_error_t vfs_read_begin (vfs_uri_t* uri, vfs_handle_t** handle,
vfs_read_callback_t callback, void* userdata);

 /** Very similar to vfs_read_begin, except that we only read the
  *  meta-data, and not the file contents.  The callback is only called
  *  once - either when the meta-data is read, or when the operation is
  *  canceled.
  *
  *  vfs_read_metadata_callback_t is defined as:
  *    void (*vfs_read_metadata_callback_t)(vfs_handle_t* handle,
  *      vfs_metadata_t* metadata, vfs_read_state_t state, void*
  *      userdata);
  */
 vfs_error_t vfs_read_metadata_begin (vfs_uri_t* uri, vfs_handle_t**
handle, vfs_read_metadata_callback_t callback, void* userdata);

 /** Cancels the operating pending on a VFS file handle.  This also
  *  calls vfs_release on the handle.
  */
 vfs_error_t vfs_cancel (vfs_handle_t* handle);

 
 And that's as far as I'm writing tonight.  Heck, those alone would
allow reading of any file from any VFS you could imagine.  Need to add
the functions for reading folders and then you'd be able to implement a
read-only file manage, at the very least.  Adding functions for writing
files will be a snap.  Folder operations aren't much harder.  Then just
write the atomic versions of each.

 Yes, there are more advanced things that can be added beyond the basic
profile above.  Rollback and versioning, for example, for backends that
have versioning capabilities like WebDAV or a theoretically CVS/SVN/Arch
backend.  Those could probably be added to a 1.0 API.  Remember, you are
perfectly allowed to add new functions later to 1.1, 1.2, etc. - you
just can't *EVER* change the API (or ABI) once 1.0 is hit, so getting
things right before 1.0 is absolutely positively in all ways essential.

 I also think the API above should be easy to wrap in scripting
languages.  It's been a long time since I've done any wrapping in any
script languages other than the ones I've written, though, so I could be
totally wrong there - please correct me if so.

 If somebody wants to post this in the Wiki, or tell me where to post
it, great.  Probably better there than several posts on the mailing
list.
>
>jamie.
>_______________________________________________
>xdg mailing list
>xdg at lists.freedesktop.org
>http://lists.freedesktop.org/mailman/listinfo/xdg
>
-- 
Sean Middleditch <elanthis at awesomeplay.com>