Setting cpu affinity for programs with large number of threads

From lxadm | Linux administration tips, tutorials, HOWTOs and articles
Jump to: navigation, search

If a process on your server is creating a large number of threads and is using a lot of CPU time, it can be that kernel will be more busy migrating the processes between CPUs than doing any real work.

Real life example, as shown by "top" - around 60% of CPU time is used by the kernel:

%Cpu(s):  1.5 us, 60.1 sy, 20.2 ni, 17.1 id,  0.6 wa,  0.0 hi,  0.5 si,  0.0 st


After pining each thread to a single CPU, kernel usage went down by over 25%:

%Cpu(s):  4.5 us, 33.2 sy, 18.8 ni, 42.6 id,  0.0 wa,  0.0 hi,  0.9 si,  0.0 st


The code below - make sure to adjust grep /usr/sbin/pmtad below to the processes you want to set the CPU affinity with taskset:

#!/bin/bash

# Number of CPUs we have
CPUS=$(grep -c MHz /proc/cpuinfo)

# pmtad process threads
PROCESS_THREADS=$(ps -eLf | grep /usr/sbin/pmtad | grep -v grep | awk '{print $4}')

# change the CPU
# CPUs for taskset start from 0 and go up to CPUS-1
function change_cpu() {
        CPU=$((CPU+1))
        # if we "overflow", get back to 0 for the assigned CPU
        if [ $CPU -eq $CPUS ] ; then
            CPU=0
        fi
}

CPU=0 # start from CPU=0
for PROCESS_THREAD in $PROCESS_THREADS ; do
    taskset -c -p $CPU $PROCESS_THREAD
    # uncomment to set all CPUs (default behaviour)
    #taskset -c -p 0-$((CPUS-1)) $PROCESS_THREAD
    change_cpu
done

Depending on how fast new threads are created (and exit), you may need to run such a script i.e. every 30 mins.