Sometimes when you're trying to automate all the things, you have to revisit old technologies that you have long forgotten. In the networking world, SNMP is king for monitoring and metric gathering. I haven't seen a lot of information out there about setting up SNMP traps in a way that allows you to do what you want. Sure there's plenty of Network Management Systems out there, but I don't need all the bells and whistles.
I recently spent some time figuring out how to audit Juniper gear for configuration changes, get notified of flapping ports on the network, and check for failed SSH login attempts. I considered reinventing the wheel here, but came across snmptt. While the project is on Sourceforge, and uses CVS, it still looks actively maintained. I looked into the configuration, and there are a ton of options with plenty of flexibility.
Quick primer on net-snmp
If you've never used the net-snmp tools, here's a quick guide to getting started. First, make sure you have the MIBs downloaded for your device and placed into /etc/snmp/mibs
. Second, add the following to /etc/snmp/snmp.conf
:
mibdirs +/etc/snmp/mibs mibs +ALL
This sets up net-snmp
to work with your custom MIBs, feel free to puruse their documentation if you want to know more.
Get your mibs in the snmptt format
In order to get snmptt
configured with your MIBs, they include a useful tool called snmpttconvertmib
. It's quick enough to use like so:
mkdir /tmp/snmptt_configs
for mibfile in /etc/snmp/mibs/*.mib; do
snmpttconvertmib --in $mibfile \
--out /tmp/snmptt_configs/$(basename $mibfile) \
--net_snmp_perl
done
cat /tmp/snmptt_configs/* > /etc/snmp/snmptt_vendor.conf
This takes all of those mib files and lets you translate them into simple text strings. The conversion isn't perfect, but it's easy to go in and tweak the ones you care about and execute actions.
Intercept the traps and send them to snmptt
I'm not going to go over setting up snmptt
in daemon mode, you can follow their documentation, or if you're on CentOS yum install snmptt
to get init scripts. Once you do have your daemon running, you have to set up snmptrapd
to execute snmptthandler
like so:
# cat /etc/snmp/snmptrapd.conf traphandle default /usr/sbin/snmptthandler disableAuthorization yes
To send SNMP traps to snmptrapd
from a Juniper device, you need a configuration like this:
set snmp trap-group default version v2 set snmp trap-group default categories chassis set snmp trap-group default categories link set snmp trap-group default categories routing set snmp trap-group default categories startup set snmp trap-group default categories configuration set snmp trap-group default categories chassis-cluster set snmp trap-group default targets $snmptrapd_ip set event-options policy login-traps events sshd_login_failed set event-options policy login-traps events login_failed set event-options policy login-traps events login_pam_authentication_error set event-options policy login-traps then raise-trap
A couple tips about snmptt
If you're receiving a lot of traps, I recommend setting threads_enable = 1
in your snmptt.ini
file. This allows EXEC statements to be launched in a new thread, preventing a build up of traps while you wait for the previous EXEC. I also recommend net_snmp_perl_enable = 1
which will convert integers to their proper names if you have net-snmp
set up properly. In the example below, this is what converts a 2 into the word "cli," which is way more informative to us humans.
Do something useful with your traps
You can store all these traps in MySQL, which I highly recommend for analytics and historical reporting. (Protip: use DATETIME rather than VARCHAR to query by time easily). But what if you have a bunch of Juniper gear and happen to use Slack? I find it useful to know when people are committing changes to the network, especially since we have a handful of engineers all working on different tasks, and I can't keep up with all the network changes.
First you have to know which trap you really care about. For me it's jnxCmCfgChange
, which gets triggered on configuration changes. Looking through my previously converted mib to snmptt config, I found the event and changed it to look like this:
EVENT jnxCmCfgChange .1.3.6.1.4.1.2636.4.5.0.1 "Status Events" Normal FORMAT Configuration change by $4 (reason: $5, from: $3) EXEC send_slack_notification "$A - Configuration change by $4 (reason: $5, from: $3)" ..snip (took out description for simplicity)..
In my snmptt
log, anytime this gets fired I get a line like this:
10.1.10.1 - Configuration change by afeid (from: cli, reason: look a change!)
You may have noticed the EXEC statement. That launches an arbitrary command and can use the same string format as the FORMAT string. So for me, I created a dumb little script to post a message to a slack channel like so:
#!/usr/bin/env ruby
require 'uri'
require 'net/https'
TOKEN='yourtokenhere'
CHANNEL='#yourchannelhere'
USERNAME='SNMP Traps'
def usage
puts "#{$0} message"
end
text = ARGV.join(' ')
usage if text.nil?
uri = URI.parse("https://slack.com/")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
params = URI.encode("token=#{TOKEN}&channel=#{CHANNEL}" +
"&username=#{USERNAME}&text=#{text}")
request = Net::HTTP::Post.new("/api/chat.postMessage?#{params}")
http.request(request)
The end result:
Now this is a very simple use case, there are plenty of powerful things you can do like create alerts in nagios using passive checks or trigger a pager when an interface goes down. What do you do with your SNMP traps? I'd be interesting in finding out.
Attachment | Size |
---|---|
![]() | 7 KB |