CPUfreq and dm-crypt performance problems

From lxadm | Linux administration tips, tutorials, HOWTOs and articles
Jump to: navigation, search

Is using CPU frequency scaling on a server a good idea? After all, some servers don’t do very much at night, so why shouldn’t we scale CPU frequency down when servers are idle?

Unfortunately, simple tests revealed serious performance problems when reading from a drive encrypted with dm-crypt / LUKS – here is why (with a simple solution).


First, let’s use “ondemand” governor – reads will be only about ~8 MB/s.

# echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# cat /proc/cpuinfo | grep MHz
cpu MHz : 349.991
cpu MHz : 349.991

# echo 3 > /proc/sys/vm/drop_caches

# dd if=/dev/san2_data/test of=/dev/null bs=64k count=1000
1000+0 records in
1000+0 records out
65536000 bytes (66 MB) copied, 8.64089 seconds, 7.6 MB/s

During the test, the CPU was using the lowest frequency possible. However, starting “cat /dev/zero | bzip2 >/dev/null” for a second or two was causing the CPU to switch to the full speed and fast reading.

Now, let’s use performance governor (or, full CPU speed) – we get almost 60 MB/s:

# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# cat /proc/cpuinfo | grep MHz
cpu MHz : 2799.930
cpu MHz : 2799.930

# echo 3 > /proc/sys/vm/drop_caches

# dd if=/dev/san2_data/test of=/dev/null bs=64k count=1000
1000+0 records in
1000+0 records out
65536000 bytes (66 MB) copied, 1.14821 seconds, 57.1 MB/s

Why is it happening? Does a bell ring somewhere?

“ondemand” CPUfreq governor has a few tunables, described in Documentation/cpu-freq. One of them is up_threshold:

up_threshold: defines what the average CPU usaged between the samplings
of 'sampling_rate' needs to be for the kernel to make a decision on
whether it should increase the frequency. For example when it is set
to its default value of '80' it means that between the checking
intervals the CPU needs to be on average more than 80% in use to then
decide that the CPU frequency needs to be increased.

While reading from that dm-crypt device, the following CPU usage was shown on yet another machine (the numbers were consistent across a few tests):

187 MHz -> ~75% system CPU time in top
375 MHz -> ~66% system CPU time in top
562 MHz -> ~94% system CPU time in top
750 MHz -> ~85% system CPU time in top
937 MHz -> ~85% system CPU time in top
1125 MHz -> ~83% system CPU time in top
1312 MHz -> ~70% system CPU time in top
1500 MHz -> ~70% system CPU time in top

dm-crypt won’t eat the whole CPU power.

With default 80 as up_threshold for “ondemand” governor, no wonder the frequency didn’t scale up.

So a “fix” for this problem was a matter of lowering up_threshold. One more thing as a final note. Some CPUs don’t support lowering their voltage automatically, and the only way to lower CPU frequency is by using throttling. However, throttling won’t save any energy on an idle machine. It’s designed only to cool down a busy CPU by forcing it to HALT from time to time.