[systemd-devel] Improving module loading

[Greg KH]

On Tue, Dec 23, 2014 at 11:25:23AM +0000, Hoyer, Marko (ADITG/SW2) wrote:
> > What do you mean by this?  What is limiting this?  What is your limit?
> > How large are these kernel modules that you are having a hard time to
> > build into your kernel image?
> - As far as I remember, we have special fastboot aware partitions on the emmc that are available very fast. But those are very limited in size. But with this point I'm pretty much not sure. This is something I got told.
> 
> - targeted kernel size: 2-3MB packed
> 
> - Kernel modules:
> 	- we have heavy graphics drivers (~800kb, stripped), they are needed half the way at startup
> 	- video processing unit drivers (don't know the size), they are needed half the way at startup
> 	- wireless & bluetooth, they are needed very late
> 	- usb subsystem, conventionally needed very late (but this finally depends on the concrete product)
> 	- hot plug mass storage handling, conventionally needed very late (but this finally depends on the concrete product)
> 	- audio driver, in most of our products needed very late
> 	- some drivers for INC communication (partly needed very early -> we compiled in them, partly needed later -> we have them as module)
> 
> All in all I'd guess we are getting twice the size if we would compile in all the stuff.

All of those should dynamically be loaded when the hardware is found by
the system, so I don't see why you are trying to load them "by hand".

> > > - Loading the image is a kind of monolithic block in terms of time
> > > where you can hardly do things in parallel
> > 
> > How long does loading a tiny kernel image actually take?
> 
> I don't know exact numbers, sorry. I guess something between 50-200ms
> plus time for unpacking. But this loading and unpacking job is
> important since it is located directly on the critical path.
> 

Exact numbers matter :)

> > > - We are strongly following the idea from Umut (loading things not
> > > before they are actually needed) to get up early services very early
> > > (e.g. rendering a camera on a display in less than 2secs after power
> > > on)
> > 
> > Ah, IVI, you all have some really strange hardware configurations :(
> 
> Yes IVI. Since we are developing our hardware as well as our software
> (different department), I'm interested in getting more infos about
> what is strange about IVI hardware configuration in general. Maybe we
> can improve things to a certain extent. Could you go more into
> details?

Traditionally IVI systems are _very_ underpowered, use old processors,
have tiny storage systems on very slow interfaces, very little memory,
huge numbers of IPC calls at startup due to legacy userspace programs
written originally for other operating systems, and expect to get
high-performance results out of bad hardware decisions.

> > There is no reason you have to do a "cold reset" to get your boot times
> > down, there is the fun "resume from a system image" solution that
> > others have done that can get that camera up and displayed in
> > milliseconds.
> > 
> 
> I'm interested in this point.
> - Are you talking about "Save To RAM", "Save to Disk", or a hybrid combination of both?
> - Or do you have something completely different in mind?

A number of devices in the past have done a "save system image" to
flash, and then when starting up, just load the system image into memory
and jump into it, everything up and running with no "startup time"
needed other than the initial memory load.

When updating the system with new software, just be sure to write out a
new "initial system state" image at the same time, and you should be
fine for your next boot.

This idea / implementation has been around for a very long time, and has
shipped in lots of devices, solving the "image in x seconds from power
on" problem quite easily.

> I personally thought about such a solution as well. I'm by now not fully convinced since we have really hard timing requirements (partly motivated by law). So I see two different principal ways for a "resume" solution:
> - either the resume solution is robust enough to guarantee to come up properly every boot up
> 	- achieved for instance by a static system image that brings the system into a static state very fast, from which on a kind of conventional boot is going on then ...

This is the easiest to do.

> - or the boot up after an actual "cold reset" is fast enough to at least guarantee the really important timing requirements in case the resume is not coming up properly

If you have good hardware, do this, but odds are, your hardware can't do
this, otherwise we wouldn't be having this whole conversation :)

> > > - Some modules do time / CPU consuming things in init(), which would
> > > delay the entry time into userspace
> > 
> > Then fix them, that's the best thing about Linux, you have the source
> > to not accept problems like this!  And no module should do expensive
> > things in init(), we have been fixing issues like that for a while now.
> > 
> 
> This would be properly the cleanest solution. In a long term
> perspective we are of course going this way and we are trying to get
> suppliers to go this way with us as well. But finally, we have to
> bring up now products at a fixed date. So it sometimes is easier, and
> more stable to work around suboptimal things.

As you have the source to everything, this shouldn't be an issue.

> For instance:
> - refactoring a driver that is doing lots of CPU intensive things in init()

No driver should be doing this, if it does, push back on the supplier of
it as unacceptable.

> vs.
> - taking the module as it is and using the time by loading things from emmc in parallel

Work around crappy code?  No, push back, as it obviously doesn't work
properly.

> > > 	-> deferred init calls are not really a solution because they
> > cannot
> > > be controlled in the needed granularity
> > 
> > We have loads of granularity there, how much more do you need?
> > 
> > > So finally it is actually a trade of between compiling things in and
> > spending the overhead of module loading to gain the flexibility to load
> > things later.
> > 
> > That's fine, but you will run into the kernel lock that prevents
> > modules loading at the same time for some critical sections, if your
> > I/O issues don't limit you already.
> > 
> > There are lots of areas you can work on to speed up boot times other
> > than worrying about multithreaded kernel module loading.  I really
> > doubt this is going to be the final solution for your problems.
> 
> It is of course not. The initial intention to develop something new
> here on top of kmod or systemd-modules-load was not to load kernel
> modules in parallel. We found that for lots of our modules we can
> actually gain some benefit by loading things in parallel so we decided
> to include the threaded approach as well.
> 
> The initial motivation to develop something new here was to get rid of
> using the "udevd" / "udevadm trigger" approach during startup. This
> "setting up the system hardware completely in one early phase during
> startup" approach is not going well with our timing requirements. So
> we are setting up our static hardware piece by piece exactly at this
> point in time when it is needed using our tool. Besides the actually
> module loading, this tool provides a mechanism for synchronization,
> and is doing some addition setup stuff on top. The threaded loading is
> just a feature.
> 
> Some numbers to the above mentioned arguments:
> - in an idle system, systemd-udev-trigger.service takes by about
> 150-200ms just to get the complete device tree to send out "add"
> uevents again (udevd was deactivated while doing this measure)

This all depends on the size of your device tree.

> - the processing of the resulting uevents by udevd takes 1-2 seconds
> (with the default rule set) again in an idle system

That's a very long time, something is wrong there, do you have too many
rules with lots of userspace processes being called out?

> - in a general solution, we'd need to wait for udevd to completely
> settle until we can start using the devices

As others have pointed out, you should never have to wait for 'udevd to
settle' if so, something is wrong with your kernel drivers, or your
system design.

best of luck,

greg k-h