Executing local programs and scripts remotely

From lxadm | Linux administration tips, tutorials, HOWTOs and articles
Jump to: navigation, search

Theory[edit]

If you want to execute a local command (be it a script or a program) remotely, you can upload it via SCP first, and then execute it via SSH. But why make two connections, when one is enough?

To execute a local script or a program on a remote machine, do something like below:


cat /some/script | ssh user@server "cat > /tmp/script ; chmod 755 /tmp/script ; /tmp/script --arguments"

or

cat /usr/bin/program | ssh user@server "cat > /tmp/program ; chmod 755 /tmp/program ; /tmp/program --arguments" 


It will copy the command, and execute it using just one SSH connection.

You can also remove the command after its execution (just add rm -f as the last command), "randomize" its name, etc.

It's useful for remote monitoring, with nagios, etc.


If the script doesn't need any arguments, this simple command will suffice:

cat /some/script | ssh user@server


Example usage with nagios[edit]

Nagios already has a plugin which enables checking services using SSH. Unfortunately, one of the prerequisites is that the plugin you are about to execute needs to be installed on the remote server.


Example plugin[edit]

This nagios plugin will upload and execute any nagios check on a remote machine, using SSH.

  • user nagios needs to be able to login to the remote machine using keys,
  • if the plugin already exists on the remote side, it won't be uploaded,
  • the plugin will be uploaded only once; all subsequent times it will use already uploaded plugin,
  • the plugins from your nagios server need to be exacutable on the remote server (they won't run if the remote architecture is different, i.e. your nagios server is a x86 box, and the remote machine runs on a MIPS CPU, or, when your glibc versions differ a lot) - if it's the case, you have to install correct plugins on the remote servers,
  • use only on the remote servers you trust; if you want to use it on the remote servers you don't trust, change the REMOTEDIR to the path only accessible by the nagios user.

Save the below script as check_remote to your nagios plugin directory.

#!/bin/bash

# Executes a local nagios plugin on a remote server.

# Some variables you are likely to change
SSH_USER=checkuser                      # user for SSH logins, default user running nagios
SSH_KEY=~nagios/.ssh/checkuser.rsa      # SSH key, can be empty

PLUGINDIR=/usr/lib/nagios/plugins       # nagios plugin dir
REMOTEDIR=/tmp                          # remote dir for file uploads

# Strips off hostname (first two arguments, $1 and $2), and leaves options only
ARGS=$@
ARGS=${ARGS##*$2}                               # plugin arguments

# Display help if no arguments given, or paths wrong
if [ ! "$1" -o ! "$2" -o "$1" == "--help" -o "$1" == "-h" ] ; then
cat <<EOF

Executes a local nagios plugin on a remote server.

Usage:

    $0 <hostname> <nagios_check_plugin> [plugin arguments]

EOF
exit 0
fi

# Fetch the SSH key; if file exists, set SSH options to use this key
[ -f "$SSH_KEY" ] && SSH_OPTIONS="-i "$SSH_KEY""


# Fetch the user, if exists, it is used, otherwise, use the current user
[ ! "$SSH_USER" ] && SSH_USER=$USERNAME

# Calculate md5sum of the plugin - we want to be sure that we run it remotely, and not something else
MD5_LOCAL=$(md5sum $PLUGINDIR/$2 | cut -f1 -d " ")

# Our plugin executed remotely

# We login to the remote server, check if the plugin is there - if yes, we run it.
# If not, we "cat" the plugin, and pipe it via ssh to the remote server, and execute it.

cat $PLUGINDIR/$2 | ssh $SSH_OPTIONS $SSH_USER@$1 \
    "# if the plugin exists on the remote side in $PLUGINDIR, execute it
    if [ -x $PLUGINDIR/$2 ] ; then
        $PLUGINDIR/$2 $ARGS
        EXITCODE=\$?
        exit \$EXITCODE

    # if it doesn't exist in $PLUGINDIR, is it in $REMOTEDIR?
    elif [ -x $REMOTEDIR/nagios-$2 ] ; then

    # is it the same as the plugin on the server? if yes, execute it
        MD5_REMOTE=\$(md5sum < $REMOTEDIR/nagios-$2)
            if [ \"$MD5_LOCAL\" == \"\$MD5_REMOTE\" ] ; then
                $REMOTEDIR/nagios-$2 $ARGS
                EXITCODE=\$?
                exit \$EXITCODE
            else
                UPLOAD=1
        fi
    fi

    # it's not in $PLUGINDIR nor $REMOTEDIR, upload it
    if [ ! -x \"$REMOTEDIR/nagios-$2\" -a ! -x \"$PLUGINDIR/$2\" -o \"\$UPLOAD\" ] ; then
        rm -f $REMOTEDIR/nagios-$2 # remove if it exists; someone might made a symlink to some other file
        cat > $REMOTEDIR/nagios-$2 
        chmod 700 $REMOTEDIR/nagios-$2
        $REMOTEDIR/nagios-$2 $ARGS
        EXITCODE=\$?
        exit \$EXITCODE
    fi"

EXITCODE=$?

# Note that the remote command should not return exit code 255,
# it is used by ssh if it fails to connect
if [ $EXITCODE -eq 255 ] ; then
    echo "CRITICAL - Unable to connect ($1)"
    exit 2
else
    exit $EXITCODE
fi

Example nagios command definition[edit]

These two commands check system load and free space on remote servers.

define command{
        command_name    check_remote_load
        command_line    $USER1$/check_remote $HOSTADDRESS$ check_load -w $ARG1$ -c $ARG2$
        }

define command{
        command_name    check_remote_disk
        command_line    $USER1$/check_remote $HOSTADDRESS$ check_disk -w $ARG1$ -c $ARG2$
        }


Example nagios service definition[edit]

Two example service definitions - for checking system load and free space on remote servers.

define service {
        use                             generic-service
#       host_name                       host1, host2
        service_description             System load
        is_volatile                     0
        check_period                    24x7
        max_check_attempts              3
        normal_check_interval           60
        retry_check_interval            30
        contact_groups                  localhost-admins
        notification_interval           240
        notification_period             24x7
        notification_options            w,u,c,r
        check_command                   check_remote_load!5.0,4.0,3.0!10.0,6.0,4.0
}

define service{
        use                             generic-service
#       host_name                       host1, host2
        service_description             Free Space
        is_volatile                     0
        check_period                    24x7
        max_check_attempts              3
        normal_check_interval           60
        retry_check_interval            30
        contact_groups                  localhost-admins
        notification_interval           240
        notification_period             24x7
        notification_options            w,u,c,r
        check_command                   check_remote_disk!20%!10%
        }

Keywords: SSH, SCP, Linux, UNIX, nagios, remote, local