Sometimes when you're trying to automate all the things, you have to revisit old technologies that you have long forgotten. In the networking world, SNMP is king for monitoring and metric gathering. I haven't seen a lot of information out there about setting up SNMP traps in a way that allows you to do what you want. Sure there's plenty of Network Management Systems out there, but I don't need all the bells and whistles.

I recently spent some time figuring out how to audit Juniper gear for configuration changes, get notified of flapping ports on the network, and check for failed SSH login attempts. I considered reinventing the wheel here, but came across snmptt. While the project is on Sourceforge, and uses CVS, it still looks actively maintained. I looked into the configuration, and there are a ton of options with plenty of flexibility.

Quick primer on net-snmp

If you've never used the net-snmp tools, here's a quick guide to getting started. First, make sure you have the MIBs downloaded for your device and placed into /etc/snmp/mibs. Second, add the following to /etc/snmp/snmp.conf:

mibdirs +/etc/snmp/mibs
mibs +ALL

This sets up net-snmp to work with your custom MIBs, feel free to puruse their documentation if you want to know more.

Get your mibs in the snmptt format

In order to get snmptt configured with your MIBs, they include a useful tool called snmpttconvertmib. It's quick enough to use like so:

mkdir /tmp/snmptt_configs
for mibfile in /etc/snmp/mibs/*.mib; do
    snmpttconvertmib --in $mibfile \
                     --out /tmp/snmptt_configs/$(basename $mibfile) \
                     --net_snmp_perl
done
cat /tmp/snmptt_configs/* > /etc/snmp/snmptt_vendor.conf

This takes all of those mib files and lets you translate them into simple text strings. The conversion isn't perfect, but it's easy to go in and tweak the ones you care about and execute actions.

Intercept the traps and send them to snmptt

I'm not going to go over setting up snmptt in daemon mode, you can follow their documentation, or if you're on CentOS yum install snmptt to get init scripts. Once you do have your daemon running, you have to set up snmptrapd to execute snmptthandler like so:

# cat /etc/snmp/snmptrapd.conf
traphandle default /usr/sbin/snmptthandler
disableAuthorization yes

To send SNMP traps to snmptrapd from a Juniper device, you need a configuration like this:

    set snmp trap-group default version v2
    set snmp trap-group default categories chassis
    set snmp trap-group default categories link
    set snmp trap-group default categories routing
    set snmp trap-group default categories startup
    set snmp trap-group default categories configuration
    set snmp trap-group default categories chassis-cluster
    set snmp trap-group default targets $snmptrapd_ip

    set event-options policy login-traps events sshd_login_failed
    set event-options policy login-traps events login_failed
    set event-options policy login-traps events login_pam_authentication_error
    set event-options policy login-traps then raise-trap

A couple tips about snmptt

If you're receiving a lot of traps, I recommend setting threads_enable = 1 in your snmptt.ini file. This allows EXEC statements to be launched in a new thread, preventing a build up of traps while you wait for the previous EXEC. I also recommend net_snmp_perl_enable = 1 which will convert integers to their proper names if you have net-snmp set up properly. In the example below, this is what converts a 2 into the word "cli," which is way more informative to us humans.

Do something useful with your traps

You can store all these traps in MySQL, which I highly recommend for analytics and historical reporting. (Protip: use DATETIME rather than VARCHAR to query by time easily). But what if you have a bunch of Juniper gear and happen to use Slack? I find it useful to know when people are committing changes to the network, especially since we have a handful of engineers all working on different tasks, and I can't keep up with all the network changes.

First you have to know which trap you really care about. For me it's jnxCmCfgChange, which gets triggered on configuration changes. Looking through my previously converted mib to snmptt config, I found the event and changed it to look like this:

EVENT jnxCmCfgChange .1.3.6.1.4.1.2636.4.5.0.1 "Status Events" Normal
FORMAT Configuration change by $4 (reason: $5, from: $3)
EXEC send_slack_notification "$A - Configuration change by $4 (reason: $5, from: $3)"
..snip (took out description for simplicity)..

In my snmptt log, anytime this gets fired I get a line like this:

10.1.10.1 - Configuration change by afeid (from: cli, reason: look a change!)

You may have noticed the EXEC statement. That launches an arbitrary command and can use the same string format as the FORMAT string. So for me, I created a dumb little script to post a message to a slack channel like so:

#!/usr/bin/env ruby
require 'uri'
require 'net/https'

TOKEN='yourtokenhere'
CHANNEL='#yourchannelhere'
USERNAME='SNMP Traps'

def usage
  puts "#{$0} message"
end

text = ARGV.join(' ')
usage if text.nil?

uri = URI.parse("https://slack.com/")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true

params = URI.encode("token=#{TOKEN}&channel=#{CHANNEL}" +
                    "&username=#{USERNAME}&text=#{text}")
request = Net::HTTP::Post.new("/api/chat.postMessage?#{params}")
http.request(request)

The end result:

Now this is a very simple use case, there are plenty of powerful things you can do like create alerts in nagios using passive checks or trigger a pager when an interface goes down. What do you do with your SNMP traps? I'd be interesting in finding out.