Update on DeviceKit

Wed May 7 00:40:04 PDT 2008

Hi,

As I've mentioned quite a few times over the past year, I've been
wanting to work on the next major version of HAL as there are several
reasons warranting a rewrite:

 - It's a huge kitchen sink that hasn't seen any real rewrites
   except for the 0.4->0.5 transition very early on. This means
    - weird constraints; e.g. methods can only return an integer
    - introspection is dodgy
    - lots of handwritten marshaling
    - tends to be a dumping ground for lots of random stuff
 - Because it does a lot, no single developer has a 100% overview
   of the code base
   - slow release process; people blocking on each other
 - Too abstract / generic
   - probably too tied to Linux
   - exports lots of useless information
   - does a lot of things but none of them particular well
     - "Jack of all trades, Master of none"
   - probably somewhat over engineered; definitely difficult for
     people to understand; lots of concepts
 - Inefficient
   - Just doesn't work very well on big iron boxes / doesn't scale
     - If you've dealt with these socalled Enterprise Linux releases
       you've probably seen a ton of bugs where it takes ~1 hour for
       hald to start up
     - Ditto for embedded systems
 - Huge overlap with underlying components
   - Can't speak for other OS'es here but on Linux, there's just too
     much in common with udev. Both can run jobs when events happen;
     both have the concept of a device database.
     - Not only is this inefficient; it's downright confusing to
       developers in what they should choose.

Apart from all that, I still believe the _idea_ of having privileged
mechanisms exporting useful API is the right one if one wants to make
progress on the Linux desktop. And certainly, just the last year with
the introduction of ConsoleKit, PolicyKit and D-Bus system bus
activation shows there's a real trend going in this direction. Many
people, even the distro people, are agreeing that the way forward is the
model where you have a policy-less privileged mechanism that can be
controlled by an unprivileged GUI policy agent. 

So I guess, to sum up my view, is that we've been pretty successful with
HAL and the other Project Utopia-ish stuff at least on the _conceptual_
level. More importantly, I think this whole way of doing things have
helped (but HAL etc. definitely isn't the _only_ factor here) bridge the
gaps not only between the desktop camps (e.g. GNOME and KDE) but also
between the kernel and desktop developers. 

So I've been having all these thoughts when I've reflected on the
project over the past year or so. And then I realized that we actually
have much better tools now than we had back in September 2003 when I
started working on HAL. Anyway, these are some of the thoughts I've been
having for the past year or so. I've tried to think really hard about
how to create a better architecture. 

How can we make this scale, how can we avoid blocking on people like me
too busy (or lazy) to do a release? How can we create mechanisms that
are truly useful for desktop applications? So I've been talking to many
people about this problem, in particular when I was traveling in January
with Kay Sievers and Lennart Poettering for three weeks. I think Kay and
I went over, like, five different system architectures; some of them
really wacky.

For the past 2-3 months, on and off (because stuff like RHEL and Fedora,
other open source commitments and RL never stops), I've actually managed
to find some time to sit down and prototype some ideas. The main idea is
to try and stop pretending we're an abstraction layer. We never was and
we never will be (the kernel and X11 is doing that just fine) [1].

However, one *useful* aspect is the merging of information from several
sources and making this process easy. The other main point is to focus
on the few subsystems that really matters. For the desktop this includes
system-wide power management, USB and storage devices. The third point
is to provide a migration path away from HAL to this new-fangled thing
including ensuring it's parallel-installable with HAL.

So I have three new projects right now. Neither are ready for a release
for a few months, but in the interest of keeping people in the loop (and
attracting contributors!) I'm announcing them now as they're slowly
getting more and more mature.

---------
DeviceKit
---------

This is a simple system service that a) can enumerate devices; b) emits
signals when devices are added removed; c) provides a way to merge
device information / quirks onto devices. And that's it. A device is
identified by a) the native OS-specific path (on Linux a sysfs path); b)
an optional UNIX device file; and c) key/value pairs describing the
device. It's a very simple service, a bit like what HAL is today without
all the probers/callouts

 http://hal.freedesktop.org/docs/DeviceKit/
 http://gitweb.freedesktop.org/?p=users/david/DeviceKit.git;a=summary

Status: the last point, merging device information, isn't done yet. This
is for things like classifying devices as audio players, cameras etc.
Ideas welcome.

Here's the TODO list

http://gitweb.freedesktop.org/?p=users/david/DeviceKit.git;a=blob;h=1f2b1c337e7ca5d0fe0ec5cbff2c6e3b4220b2fa;hb=e67cf1ad360471ae6c6ebf04f0d1f9648f322a93;f=doc/TODO

---------------
DeviceKit-disks
---------------

This is a system service to keep track of block devices. It already does
quite a few things already

 - Enumerate and manipulate block devices including;
   - Filesystems / Signatures
     - create, modify label, delete, mount, unmount
   - Partitions and Partition Tables
     - create, modify, delete
   - Linux MD Software RAID (both arrays and components)
     - add/remove components, start, stop
   - LUKS devices
     - create, lock, unlock, change passphrase
   - Change notifications
 - Secure Erase
   - (Still need to actually implement > 1 pass erases)
 - Retrieve SMART data
   - includes storing a history in a database

so it's already a lot more useful than what we have in HAL [2]. Here's
the API and source code

 http://hal.freedesktop.org/docs/DeviceKit-disks/
 http://gitweb.freedesktop.org/?p=users/david/DeviceKit-disks.git;a=summary

Here's the TODO list

http://gitweb.freedesktop.org/?p=users/david/DeviceKit-disks.git;a=blob;h=00a3469335e78f4b25605ca484bcf97e31081be5;hb=61fd98cfa4478e4a564b5cf225e4100f002ee650;f=doc/TODO

------------------
gnome-disk-utility
------------------

This is, in some way, a graphical frontend to DeviceKit-disks. It's
supposed to be a user-friendly tool for managing disks.

 http://gitweb.freedesktop.org/?p=users/david/gnome-disk-utility.git;a=summary

Here are some screenshots

 http://people.freedesktop.org/~david/gdu-attr-maxtor.png
 http://people.freedesktop.org/~david/gdu-raid5.png
 http://people.freedesktop.org/~david/gdu-more-raid-progress.png
 http://people.freedesktop.org/~david/gdu-smart-and-failing.png
 http://people.freedesktop.org/~david/gdu-luks-easy.png

Here's the TODO list

http://gitweb.freedesktop.org/?p=users/david/gnome-disk-utility.git;a=blob;h=1e7abfcf24947de142f6cc3661ae0852613ad239;hb=79f00c48cd9e37ba09b76fbf95e113297da0a94c;f=doc/TODO

----------
NEXT STEPS
----------

Also, we need subsystem daemons for USB and for power management. For
PM, I have some code lying around that I never put in git. I'm going to
try and clean it up over the next week or so and post it here. 

For other subsystems, such as Firewire, audio and input my answer for
this is to, for notifications and enumeration, rely on the core
DeviceKit daemon and values exported by the OS kernel (e.g. sysfs on
Linux). In effect, this is not any different from using HAL today.

I'm expecting to do a release of (at least) these three projects over
the next month or two. Especially for the disks stuff, lots of
dependencies needs to have a release as well. The goal, my goal at
least, is to get this in a workable state for Fedora 10 / GNOME 2.24 and
get as much as GNOME moved over to new mechanisms. I hope by Fedora 11 /
GNOME 2.26 to have most of the OS moved to new mechanisms. 

(Not implying that this thing should revolve around Fedora or GNOME (it
really shouldn't), just saying that's where I'm going to spend my time
(because that's what I'm paid to do) and thought it would be useful for
other people to know that.)

There's some administrative steps as well. The repositories currently
used will be moved (g-d-u will move to GNOME SVN); we probably need new
mailing lists etc. I'll post more updates when we get to that.

So what happens with HAL? Applications will continue to use it and we
will still do bug-fix releases when necessary. But no patches for new
features will be accepted. I expect most distros to keep shipping it
(Fedora certainly will) as some apps will take some time to get ported
over. And the enterprise releases will need to support it for many
years. So it's not like we're going to ignore it, I just won't accept
patches for new features. I hope that's understandable.

-------------
BONUS PREVIEW
-------------

I have Fedora 9 packages of DeviceKit, DeviceKit-disks and
gnome-disk-utility and the dependencies they need. Here are the SRPM's

 http://people.freedesktop.org/~david/DK/F9/src/

and here are RPM's for i386

 http://people.freedesktop.org/~david/DK/F9/i386/

Warning: this will replace some F9 packages. You need to reboot
(technically you don't but...) after installing all packages. Then you
can start gnome-disk-utility.

(The usual disclaimer goes here: this is unreleased alpha-quality
software that may very well eat your disks, delete your data and kill
your dog. If it breaks you get to keep both pieces. E.g. use at your own
risk etc. etc.)

Note that there's some hard-to-grok dependencies on specific versions of
Linux, udev, mdadm and device-mapper with specific patchsets. I suggest
to wait until there are new releases of these before tinkering with it;
see the doc/TODO file in DeviceKit-disks for the details.

I actually have a Fedora 9 live CD with this stuff. I can upload it to
my page at p.fd.o if people not using F9 wants to try out g-d-u and get
a feeling of how things work. Let me know.

      David

[1] : I still blame Havoc for the name HAL.

[2] : In fact, I've been wanting to do mkfs and partitioning stuff in
HAL for a long long time. But then I realized there was no good
authorization framework so I went off and wrote PolicyKit.