Capistrano for the sysadmin

There's always been a fine line between a good sysadmin and a programmer. Capistrano is software built for Ruby on Rails to make deploys across multiple servers easy, however, the potential is much greater than just code deployment. I have recently grown in love with how easily I can run commands across groups of servers, and setting up Capistrano is not a difficult task. If you've ever configured password-less SSH access, you can get this up and running without much difficulty.

Have a centralized ssh-key

# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/

You have an SSH key, now you just need to install it on each server you would like to manage. One of the things I like to do, is setup an rsync share to pull down important config files. This can also be easily accomplished using an NFS mount. Either way after you have password-less SSH access to all the servers from one location, this will be your controlling server. You won't need to login to the other boxes individually from this point forward.

Security note: in this example, I'm using root logins. I recommend setting PermitRootLogin to without-password if you want to use root access, however, I really do not recommend using root directly, as it becomes a liability issue when you have more than one admin. The use of a normal user account plus sudo is highly recommended.

Install the software

Depending on what OS you're on, these steps may be different, but the end goal is the same: get RubyGems installed. On CentOS/RHEL the following works:

# yum install -y ruby ruby-devel ruby-rdocs ruby-ri

Since the version of ruby included on CentOS/RHEL is a bit dated, we have to use an older version of RubyGems (1.3.5).

# wget
# tar xzf rubygems-1.3.5.tgz
# cd rubygems-1.3.5/
# ruby setup.rb
# gem install capistrano

Configuring Capistrano

At this point, a Rails developer would cd into their code directory, run capify and configure Capistrano's deploy scripts to their needs. Going over code deployments is an entirely different process, which I'll save for another time. As a sysadmin, we can skip running capify and just make our own cap file. Think of a cap file as a basic config file, except written in Ruby. You get the power of a scripting language built right into your config file. This truly gives you endless options and the ability to tie into anything you'd like (think database, or ldap). Here I'll show you a basic "cap file" that will do what sysadmins want: make managing multiple servers easy.

# cd /etc/manage
# ls -l 
-rw-r--r-- 1 root root 2739 Jul 20 18:14 capfile
drwxr-xr-x 2 root root 4096 Jul 19 12:15 roles/

# cat capfile
def getrole(role)
        f = open('/etc/manage/roles/' + role)

role(:web) { getrole('web') }
role(:db) { getrole('db') }

desc "Show Memory"
task :show_mem do
  run "grep -i memtotal /proc/meminfo"

What I like to do is create a directory (/etc/manage/), then populate this with central management type configurations. In this case, it would be my cap file, and a directory called roles. The roles directory just contains files with lists of servers, for example:

# cat /etc/manage/roles/web

So this would be the resolvable names of all my web servers. The cap file uses a basic function called getrole() to read the file declared by role inside the roles directo One would be to cd into /etc/manage and the other would be to declare the cap file from the command line with the -f parameter. The first thing you can look at, is the output of -T.

# cap -T
cap invoke     # Invoke a single command on the remote servers.
cap shell      # Begin an interactive Capistrano session.
cap show_mem   # Show Memory

This will show you all the different tasks you have available. There are two built in ones (invoke, shell) and one we defined in our cap file (show_mem). This is a very basic example, but will suite a wide range of needs. The most useful, in my opinion (at least at this stage), is the shell. This is where you have the power to run commands across "roles" of servers.

# cap shell
cap> df -h
 ** [out :: web1] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web1] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web1] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: web2] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web2] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web2] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: web3] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: web3] /dev/cciss/c0d0p1      63G  8.9G   51G  15% /
 ** [out :: web3] tmpfs                 7.9G     0  7.9G   0% /dev/shm
 ** [out :: db1] Filesystem            Size  Used Avail Use% Mounted on
 ** [out :: db1] /dev/cciss/c0d0p1     128G  2.6G  119G   3% /
 ** [out :: db1] tmpfs                 2.0G     0  2.0G   0% /dev/shm
cap> with web
cap> grep admin /etc/passwd
 ** [out :: web1] admin:x:500:500:Admin user:/home/admin:/bin/bash
 ** [out :: web2] admin:x:500:500:Admin user:/home/admin:/bin/bash
 ** [out :: web3] admin:x:500:500:Admin user:/home/admin:/bin/bash
cap> with db
cap> !show_mem 
 ** [out :: db1] MemTotal:      3985972 kB

When you first enter the shell, it will be set to execute tasks or commands across ALL roles. Using the with directive, you can specify which role you want to use for the subsequent commands. Finally, adding a bang (!) in front, tells Capistrano to execute a task rather than standard shell commands. Hopefully I've given you enough information to apply Capistrano to your own environments. Think about commands you often execute, create a new task and you will find yourself saving a lot of time. Even troubleshooting and adding new functionality across servers becomes much quicker when you can access everything from one location.