Speed up DeviceKit device probing on really slow devices

[David Zeuthen]

On Tue, 2010-01-19 at 14:54 -0800, Linus Torvalds wrote:
> 
> On Tue, 19 Jan 2010, David Zeuthen wrote:
> > 
> > As Kay said on the util-linux-ng list, the plan is to use blkid for
> this
> > in the future - so at some point udisks-part-id will go away.
> 
> I hope that blkid will be less of a pig by then. It looks like the 
> partition probing of blkid is (once more) overly complete at the
> expense of simplicity and performance. 

I think Karel (and everyone else who's working in this area) is
committed to making things not suck - even for really slow devices like
your embrodiery machine (I'm curious - did you try to timing it on OS X
and Windows?).

Anyway, there are huge maintenance (and probably performance) benefits
to having all the common probing/guessing code in a single project (e.g.
blkid).

Keep in mind we also additional probing for e.g. MD components and (in
the future) LVM2 PV's and other complicated things.

> After looking at blkid IO patterns, it was a pleasure to see 
> udisks-part-id just care about the normal case.

Well, udisks-part-id only cares about commonly used partition tables:
MBR, GPT and Apple Partitioning Format (and you might argue that APM is
not commonly used anymore). I think Karel's plan with blkid is to also
care about all the weird partitioning formats that the kernel also cares
about - dunno how much of a performance impact that's going to be.

> > (see e.g.
> http://people.freedesktop.org/~david/gdu-edit-partition.png 
> > for how this information is used)
> 
> So that's a bit scary, if the detection program needs to do all of
> that. I  already object to the fact that the kernel took time to settle the
> device  and scan the partitions (which defaults to a horrible five-second
> settling  time - although fixing it to be 1s was trivial), but then that
> partition scan was ignored, and when I started this whole saga, blkid +
> DeviceKit took a total of one _minute_ to scan that embroidery device.

Well, the school of thought here is to collect as a lot of data (but not
all - there's a fine line) about the device and store it in the udev
database. And also update the data in real time. This is used e.g. to
maintain the /dev/disk/ symlinks but it is also used for disk management
programs and to decide what icons to put on your desktop, in your file
chooser and so on. And it's more than just filesystem type or partition
information - it also includes VPD as collected by e.g. /lib/udev/ata_id
and /lib/udev/scsi_id and other tools.

So, yeah, there's a lot of IO and processing going on when devices are
attached and when there are changes - e.g. when the last non-read-only
opener closes the device, udev synthesizes an udev change event which is
propagated up the stack.

Is it worth to doing all this? For desktop workloads, the answer is:
definitely. It means that the desktop UI, /dev/disk symlinks and your
disk management programs are kept in sync even when you modify the
device using (unmodified) command-line tools like fdisk, e2label, mkfs,
mdadm, lvm and so on. It's generally the behavior that people,
especially OS hackers, expect.

> Now it's down to 15 seconds, which is better. But it's still an
> annoyingly long time.
> 
> Mentally do this: imagine inserting a USB device, and then count
> slowly to fifteen to simulate how long it is before the desktop reacts
> to your device insertion. Doesn't feel so great.

No one saying that the current experience with slow devices is great -
but it looks like the slow device case is getting fixed (thanks to you).
For the record, with even the slowest of the slowest USB keys, the
desktop icon appears instantly for me. FWIW, I'm pretty nazi about these
things - which is one of the reasons that we poll storage devices every
two seconds. Polling the device less often (to save battery), say every
ten seconds, or not at all - is simple going to result in a poor user
experience (and we have an interface to inhibit polling altogether too)

It might be nice if usb-storage's delay_use wasn't five seconds - IIRC
Ubuntu has shipped with 0 seconds without any problems. Maybe it's time
to bring that upstream, I don't know.

Also, it's not ideal that both the kernel and user space is looking for
partition tables - maybe the default could be to use the in-kernel
partitioner (so things like "init=/bin/bash root=/dev/sda2" works -
never underestimate simplicity etc.) but most distros could turn it off
early on and do partitioning in user space based on whatever
partitioning scheme that blkid detects. I mean, we already have an
interface for adding/removing partitions (BLK_PG_{ADD|DEL}_PART) - or,
brr, we could use the kpartx/dm-linear - though that's more complexity
than I like.
 
     David