continued: Common-VFS proposal

[Sean Middleditch]

On Thu, 2005-01-20 at 15:01 +0100, nf2 wrote:
> Hi!
> 
> Sean Middleditch wrote:
> 
> >That being said, Norbert:
> >
> >First, what *is* with this "D-BUS won't work" deal?  That makes no
> >sense, and you provide no explanation.  D-BUS is absolutely *perfect*
> >for our needs.  Certainly something *more* heavyweight - like CORBA - is
> >going to be worse.  VFS operations are very common, they need to be
> >fast.
> >  
> >
> Ok. I'll put questionmarks behind this statement. The reason why i wrote 
> this is that
> i thought that DBUS was not designed to transport gigabytes of data. But 
> i really don't know which "wire protocol" for method calls (DBUS vs 
> ORBIT) is leaner. The other reason was that ORBIT provides a nice 

D-BUS is just as capable of transferring large amounts of data as CORBA.

> framework - as far as i understand it provides a multi-threaded server 
> for putting protocol handlers in the VFS daemon. This is used by 

Multi-threaded doesn't really do much for us.  Among several other
reasons, do keep in mind that the protocol between the app and daemon
will *not* be the bottle-neck - the connection between your machine and
the remote server will be.

Also, the IPC daemon being threaded doesn't at all affect the
performance of the VFS daemon.  If multi-threaded *did* become
necessary, you'd want to multi-thread the VFS daemon itself, which won't
be affected by the IPC mechanism you use.

As my proposal said, I actually don't even want to use an IPC
daemon/bus, so using ORBit would actually be slower.  You need a point-
to-point protocol, and D-BUS provides an API and library to do just
that, in addition to, basically, not being ORBit.  You are never going
to get anyone but GNOME to rely on a framework that uses ORBit.  And in
GNOME 3 ORBit might very well just be gone, replaced entirely by D-
BUS.  ;-)

> 
> >The big problem is the event handling, especially when you realize that
> >you're going to end up needing asynchronous I/O and probably some kind
> >of monitoring as well.  Using D-BUS mostly solves this as it is already
> >possible to integrate with any mainloop, has glib support to do so, and
> >Qt integration is definitely possible.  A lot of CLI apps and such
> >probably just wouldn't use any asynchronous operations.  The only
> >problem comes from having local files handled in the library/application
> >itself - local files would have distinct descriptors while all remote
> >connections would be handled over one connection to the daemon;
> >  
> >
> I don't know if the daemon is necessary for all network connections*). 
> For sftp for instance that might not be necessary (if there is no 
> connection limit). That's one of the things i like about Gnome-VFS. 
> Certain modules go into the daemon and others stay in the client process 
> (As threads in async mode AFAIK - please correct me!)

I'd want a daemon for sftp so that individual user apps don't need to
access my SSH key.  I could use SELinux to ensure that only the VFS
daemon and ssh can access the key, making sure that random apps I
install can still use the VFS without being able to ever steal my keys.

> 
> cat /etc/gnome-vfs-2.0/modules/default-modules.conf
> cat /etc/gnome-vfs-2.0/modules/smb-module.conf (look at the [daemon] flag)
> 
> *) Although that might be nice, because it would be easier to disconnect 
> them on demand.

Most in-app modules of gnome-vfs are things like the trash:// handler,
which I addressed.

> 
> >basically it's a problem of knowing which file descriptors to pass to
> >select()/poll()/epoll/kqueue.  One solution is to provide an API to
> >obtain a list of all descriptors in use by the VFS library.  Another is
> >to simply state that each open file has on descriptor, and in the case
> >of remote files use a separate daemon connection for each file (yuck).
> >The final is to just document the problem clearly and punt to the apps.
> >Since the problem only happens in asynchronous operations anyhow, which
> >happen mostly in GUI mainloops, each of which would have its own
> >simplified wrapping around the VFS, I'm leaning towards the final
> >solution.
> >  
> >
> The common main loop... I think commercial os'es have that... ;-)

Not even.  ;-)

The importance isn't using a common main loop, but making sure the
library can be added to any main loop.  Most apps that actually *need* a
main loop don't write their own anyhow, they use one from GNOME or Qt or
whatever.

For CLI apps my library design can just be used in synchronous mode, the
same way most CLI apps already do remote I/O, so there's no need to even
*have* a main loop.

> 
> >The bridging of gnome-vfs and kioslaves just is *not* the correct
> >solution here.  The first is the simple design - I'm a picky bastard,
> >and to me, design and cleanliness are pretty important.  ;-)  A clean
> >implementation of a VFS is definitely preferred in that department.
> >  
> >
> You are right here: The KIO->Gnome-VFS bridge is quite ugly from design. 
> But it would be a first step. In the future - with a common main loop 
> and after improving the Gnome-VFS client API with things KIO requires - 
> KIO could perhaps call directly into the async Gnome-VFS API without 
> going via the internal KIO-slave IPC.
> 
> What concerns writing another VFS i'm a bit scared that the result might 
> look 90% like KIO or Gnome-VFS. A VFS is just a VFS.

Not true politically, and politics matter. 

> 
> >Second is the usefulness of a gnome-vfs/kioslaves bridge.  That only
> >helps GNOME and KDE apps.  GnuStep, Rox, CLI apps, and so on *will not*
> >depend on an architecture built on GNOME or KDE libraries like that.
> >Even if there were no technical reason not to, there's political
> >reasons.  Yay humanity.  ;-)
> >
> >  
> >
> The dependencies of Gnome-VFS are not that many (mainly orbit, glib, 
> dbus). Probably you won't get that a lot cheaper.

Kill ORBit (redundant with D-BUS) and glib.  Not hard to do.  ;-)

Dependencies can be important.  What happens when you want to run that
Common VFS-using app on your custom-designed embedded machine?  Why use
a VFS framework that needs several other libraries when you can make a
VFS framework that works equally well that doesn't need those libraries,
and possibly even end up smaller despite some code duplication here and
there?