dvfs api and toolkits

[Sean Middleditch]

On Thu, 2005-04-07 at 18:42 +0200, P. Kaluza wrote:
> Hi List,
> 
> Sean Middleditch wrote:
> 
> >Some backends might have relatively efficient support for seeking, and
> >it might be possible to design the API in such a way that it can
> >transparently support either native seek support or local-copy
> >emulation.  Doing that could get tricky, though, in some cases - what if
> >you open a large file on a backend that doesn't handle seeking but just
> >stream it in?  Should the system try to create the temporary copy
> >anyhow, or should the API require the developer to explicitly state, "I
> >want this file to be seekable" ?
> >  
> >
> I'd say definitely the latter. It should be quite acceptable to API 
> users to have to specify a SEEK_NEEDED flag on opening. Then the daemon 
> can fork the incoming stream from the first byte. (I think most 
> browsers' http backend work this way too.)

When in seek mode, should we automatically download the whole file, or
switch to a mode where we download bits as they're needed?  If the app
wants to open a 5GB movie file and only ever seeks/reads 400MB worth of
its data at the beginning of the file, should the bandwidth and disk
space be consumed for that whole 5GB?  If we stop the download to save
space/bandwidth we may have to start it up again from the beginning in
filesystems that don't support seek to begin with.

How many file systems are there that don't support seeking (or features
necessary to reasonably emulate it without downloading the whole file),
anyhow?  The only big one I can think of off hand is FTP - how often
would users directly work with files on FTP these days?  Most FTP sites
I know of are file download archives or upload spaces, not general work
space storage areas.

For file systems like HTTP, we can't actually seek in a file.  We have
to reopen the file at a new location.  Would there be a problem with the
file possibly changing under us?  Could we potentially detect that?
There is no way to provide the classic POSIX file semantics with HTTP at
all, since there's no concept of "open files" or inodes at the HTTP
layer, and seeking is pretty heavily based on those semantics.  If
WebDAV is used, should the file be automatically locked if seek mode is
used to work around this problem?  Would automatic locking pose any
other problems?  What about when locking isn't available?

I'm still not convinced that seeking should be in D-VFS at all.  If an
application really needs seeking (it could check the file size and know
if it can fit the whole thing in memory or not) it could do the download
and cache on its own.  We could even provide helper APIs for doing that
at the higher level wrapper level.  Putting it in D-VFS proper might
lead too many developers to rely on it being available and trustworthy,
which we just can't guarantee.  I don't want to make it difficult for an
application to do what it needs to do, but I also don't want to make it
easier to do the wrong thing than it is to do the right thing.

> 
> However, the point made about caching (for nautilus thumbnailer + 
> opening app) is valid too - but maybe more of a corner case, especially 
> in the near-to-medium future, where nautilus maybe uses DVFS, but most 
> called apps won't yet. So if this is implemented, please make it 
> configureable.

I'd like to know if it's an actual problem first.  What does gnome-vfs
and KIOSlaves do right now in this case?  Do they cache?  If not, is
there an actual noticeable performance problem here that they'd like
solved?  If they do cache, what are their behaviors in regards to cache
lifetimes for backends that don't natively support caching (like FTP) ?

> 
> 
> Another thing i was just thinking is this: there are _already_ two file 
> system semantics under unix, namely POSIX and AFS/CODA. We don't like 
> posix, but it probably would make sense to try and stay compatible to 
> the afs/coda semantic, if it roughly fits your model. It basically works 
> like this: open document for writing, write around in it (write out 
> completely or append in D-VFS' case), then close, after which the actual 
> writeback to the server occurs (also some basic version control in 
> Coda's case, but thats an area   D-VFS could tackle later, if at all). 
> So basically an atomic-write-on-close. So in the case of DAV, you would 
> MKACTIVITY, stream the file to the server (in the chunks sent by the 
> app), and then, on closing, move the file atomically over the original. 
> Does this sound sensible ?

That sounds ideal.

> 
> Bye,
>   Philipp
> 
> _______________________________________________
> xdg mailing list
> xdg at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/xdg
> 
-- 
Sean Middleditch <elanthis at awesomeplay.com>