CPUfreq and dm-crypt performance problems
Is using CPU frequency scaling on a server a good idea? After all, some servers don’t do very much at night, so why shouldn’t we scale CPU frequency down when servers are idle?
Unfortunately, simple tests revealed serious performance problems when reading from a drive encrypted with dm-crypt / LUKS – here is why (with a simple solution).
First, let’s use “ondemand” governor – reads will be only about ~8 MB/s.
# echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # cat /proc/cpuinfo | grep MHz cpu MHz : 349.991 cpu MHz : 349.991 # echo 3 > /proc/sys/vm/drop_caches # dd if=/dev/san2_data/test of=/dev/null bs=64k count=1000 1000+0 records in 1000+0 records out 65536000 bytes (66 MB) copied, 8.64089 seconds, 7.6 MB/s
During the test, the CPU was using the lowest frequency possible. However, starting “cat /dev/zero | bzip2 >/dev/null” for a second or two was causing the CPU to switch to the full speed and fast reading.
Now, let’s use performance governor (or, full CPU speed) – we get almost 60 MB/s:
# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # cat /proc/cpuinfo | grep MHz cpu MHz : 2799.930 cpu MHz : 2799.930 # echo 3 > /proc/sys/vm/drop_caches # dd if=/dev/san2_data/test of=/dev/null bs=64k count=1000 1000+0 records in 1000+0 records out 65536000 bytes (66 MB) copied, 1.14821 seconds, 57.1 MB/s
Why is it happening? Does a bell ring somewhere?
“ondemand” CPUfreq governor has a few tunables, described in Documentation/cpu-freq. One of them is up_threshold:
up_threshold: defines what the average CPU usaged between the samplings of 'sampling_rate' needs to be for the kernel to make a decision on whether it should increase the frequency. For example when it is set to its default value of '80' it means that between the checking intervals the CPU needs to be on average more than 80% in use to then decide that the CPU frequency needs to be increased.
While reading from that dm-crypt device, the following CPU usage was shown on yet another machine (the numbers were consistent across a few tests):
187 MHz -> ~75% system CPU time in top 375 MHz -> ~66% system CPU time in top 562 MHz -> ~94% system CPU time in top 750 MHz -> ~85% system CPU time in top 937 MHz -> ~85% system CPU time in top 1125 MHz -> ~83% system CPU time in top 1312 MHz -> ~70% system CPU time in top 1500 MHz -> ~70% system CPU time in top
dm-crypt won’t eat the whole CPU power.
With default 80 as up_threshold for “ondemand” governor, no wonder the frequency didn’t scale up.
So a “fix” for this problem was a matter of lowering up_threshold. One more thing as a final note. Some CPUs don’t support lowering their voltage automatically, and the only way to lower CPU frequency is by using throttling. However, throttling won’t save any energy on an idle machine. It’s designed only to cool down a busy CPU by forcing it to HALT from time to time.