[pulseaudio-discuss] Distribution kernels and glitch-free (Packagers, read this!)

Mon Feb 23 12:18:29 PST 2009

I just wanted to clarify that Ubuntu (like other distros) has
different kernel flavors. Very roughly summarized, here are the main
differences I could discern between the different kernel flavors on
Ubuntu. This is based on Jaunty (9.04) which is currently in late
alpha/early beta, but I have also checked this stuff on earlier
kernels, and if memory serves, nothing has changed.

* The -generic kernel flavor is the one that's installed if you
download an Ubuntu Desktop ISO. This kernel's config regards
preemption and timing as follows:

CONFIG_HZ=250
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_NO_HZ=y

Also, on a 32-bit system, PAE is disabled (so all of the consequences
thereof hold; max. 4GB addressable physical memory, etc.) This is
irrelevant for a 64-bit system.

* The -server kernel flavor is the one that's installed if you
download the Ubuntu Server ISO. This one corresponds more to what you
are all referring to when you say the Ubuntu kernel has "no
preemption":

CONFIG_HZ=100
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_NO_HZ=y

Notably, the -server kernel _enables_ PAE on 32-bit systems. You have
additional memory latency/overhead for all physical memory accesses
due to the number of page tables. This can worsen the already bad
situation with latency here.

* Once upon a time (old releases, e.g. 7.04 and below?) there was the
-lowlatency kernel flavor. This one went all the way and gave us
CONFIG_HZ=1000 and a fully preemptible kernel (not only voluntary, but
forced preemption). I'm not sure of the rationale for discontinuing
this kernel flavor, but it would be silly to say it's only because of
mirror disk space or something; their disk space consumption has gone
way up since then.

I think a valid question to be asked is:

for $(every proprietary kernel module); do
      (a) Is it designed with forced preemption in mind, or does it
break with it enabled?
      (b) Is it designed with voluntary preemption points? Or, if not,
then it follows that the _only_ means of preempting it is to turn on
forced preemption; see (a).
done

If every driver you want to use has at least (b), then a kernel with
voluntary preemption should be enough for low latencies, even if we
assume the worst case (that the driver likes to hog resources for an
extended time without letting go). We'll be optimists and assume that
their voluntary preemption points will not be so coarse as to fail to
preempt long operations.

If you have a driver where (b) is false -- it is not designed with any
(or sufficient) voluntary preemption points -- then we must turn to a
kernel that is fully preemptible; and in that case, we must ensure
that they all can obtain (a).

If we run into a situation where we must support a driver that obtains
neither (a) nor (b), AND it likes to run a critical section for a very
long time, then you're sort of "out of luck" and you must disable
glitch-free, if you want to continue using your broken drivers with
audio.

So, in order from most preferable to least:

1. All your drivers have good voluntary preemption points that
minimize resource hogging, and they're compatible with forced
preemption. Use VOLUNTARY preemption; latency will be good enough, and
you'll still get good throughput on many-SMP, cache-coherent systems.
You can turn glitch-free on, and should expect few or no dropouts.
2. One or more drivers have bad or non-existent voluntary preemption,
but they work fine with forced preemption. Use FORCED preemption;
latency will still be good, but you will get preemptions when it's not
really necessary, which can substantially slow down many-SMP systems.
For instance, even on desktops nowadays we have 8 threads with the
Core i7; triggering an IPI on all of these at once is expensive, and
serializes your operating system each time. However, you can still
keep glitch-free on!
3. One or more drivers have bad or non-existent voluntary preemption,
and bad things happen when you load them with forced preemption. Use
NO preemption; you have lost all guarantees of good latency timing, so
glitch-free should probably be turned off, unless the driver is
inherently good at not requiring intensive critical sections.

With this line of reasoning, you don't have to give up support for
your devices; either way, you still get to load all the broken drivers
you want. But choosing option 3 is actually justifiable considering
the situation. This assumes, of course, that we have no means of
improving these broken drivers (which is almost always true with
proprietary drivers, BTW).

I think distributions should give users the flexibility to pick any of
these three options, and should ship three kernels corresponding to
each. I also think that you should scale the CONFIG_HZ option linearly
with the preemption method, since it makes little sense to try and
interrupt the system 1000 times a second (CONFIG_HZ=1000) if you have
no preemption mechanism. So CONFIG_HZ=100 <-> no preemption;
CONFIG_HZ=250 (or 300?) <-> voluntary; CONFIG_HZ=1000 <-> forced
preemption. That's my take on it.

HTH,

Sean

On Mon, Feb 23, 2009 at 1:58 PM, Mark Greenwood <fatgerman at ntlworld.com> wrote:
> On Monday 23 February 2009 17:42:44 Colin Guthrie wrote:
>> 'Twas brillig, and Lennart Poettering at 23/02/09 16:31 did gyre and gimble:
>> > On Mon, 23.02.09 10:14, Colin Guthrie (gmane at colin.guthr.ie) wrote:
>> >
>> >> 'Twas brillig, and Lennart Poettering at 23/02/09 02:05 did gyre and gimble:
>> >>> OpenSUSE apparently does not enable it. The same is true for
>> >>> Ubuntu. Same for Debian. Note sure about Mandriva.
>> >> Here is a (not quite current) grep from my running kernel (it's .27 as
>> >> .28/.29 both break my intel graphics :( at the moment):
>> >>
>> >> $ zgrep -E "HZ|PREEMP" /proc/config.gz
>> >> CONFIG_PREEMPT_NOTIFIERS=y
>> >> CONFIG_NO_HZ=y
>> >> # CONFIG_PREEMPT_NONE is not set
>> >> CONFIG_PREEMPT_VOLUNTARY=y
>> >> # CONFIG_PREEMPT is not set
>> >> # CONFIG_HZ_100 is not set
>> >> # CONFIG_HZ_250 is not set
>> >> # CONFIG_HZ_300 is not set
>> >> CONFIG_HZ_1000=y
>> >> CONFIG_HZ=1000
>> >> CONFIG_MACHZ_WDT=m
>> >>
>> >> I'll ask our kernel team why PREEMPT is not on as I've no idea of the
>> >> consequences. My guess is that it relates to graphics drivers...
>> >
>> > The Fedora kernels set PREEMPT_VOLUNTARY as well as it seems.
>>
>> Yeah, I asked this of one of our Kernel guys. This is his response,
>> which seems pretty sensible and balanced to me :)
>>
>>
>>
>> Well, Lennart actually answered your question  ;-)
>>
>> Full Preempt used to break several drivers, including graphic, wireless, ...
>>
>
> I've been running a homespun full preempt (and indeed, RT) kernel on Mandriva while using several "dodgy" drivers (nvidia, atheros) for a couple of years without any trouble.
>
> On the other hand Ubuntu's non-preempt 300Hz kernel is nothing but trouble... go figure??
>
> Mark
> _______________________________________________
> pulseaudio-discuss mailing list
> pulseaudio-discuss at mail.0pointer.de
> https://tango.0pointer.de/mailman/listinfo/pulseaudio-discuss
>