If you have compiled an SNMP support into the Zabbix server, you should already have the complete SNMP suite installed, which contains the SNMP daemon, the SNMP trap daemon, and a bunch of utilities, such as snmpwalk and snmptrap.

If it turns out that you don't actually have the SNMP suite installed, the following command should take care of the matter:

# yum install net-snmp net-snmp-utils

Just as the Zabbix server has a bunch of daemon processes that listen on the TCP port 10051 for incoming connections (from agents, proxies, and nodes), snmptrapd is the daemon process that listens on the UDP port 162 for incoming traps coming from remote SNMP agents.

Once installed, snmptrapd reads its configuration options from an snmptrapd.conf file, which can be usually found in the /etc/snmp/ directory. The bare minimum configuration for snmptrapd requires only the definition of a community string in the case of versions 1 and 2 of SNMP, which is as follows:

authCommunity log public

Alternatively, the definition of a user and a privacy level in the case of SNMP Version 3 is as follows:

createUser -e ENGINEID user MD5 auth DES priv

With this minimal configuration, snmptrapd will limit itself to log the trap to syslog. While it could be possible to extract this information and send it to Zabbix, it's easier to tell snmptrapd how it should handle the traps. While the daemon has no processing capabilities of its own, it can execute any command or application by either using the trapHandle directive or leveraging its embedded perl functionality. The latter is more efficient as the daemon won't have to fork a new process and wait for its execution to finish, so it's the recommended one if you plan to receive a significant number of traps. Just add the following line to snmptrapd.conf:

perl do "/usr/local/bin/zabbix_trap_receiver.pl";

Once it is restarted, the snmptrapd daemon will execute the perl script of your choice to process every trap received. As you can probably imagine, your job doesn't end here—you still need to define how to handle the traps in your script and find a way to send the resulting work over to your Zabbix server. We'll discuss both of these aspects in the following section.

The perl script included in the Zabbix distribution works as a translator from an SNMP trap format to a Zabbix item measurement. For every trap received, it will format it according to the rules defined in the script and will output the result in a log file. The Zabbix server will, in turn, monitor the said log file and process every new line as an SNMP trap item, basically matching the content of the line to any trap item defined for the relevant host. Let's see how it all works by looking at the perl script itself and illustrating its logic:

#!/usr/bin/perl

#
# Zabbix
# Copyright (C) 2001-2013 Zabbix SIA
#
#########################################
#### ABOUT ZABBIX SNMP TRAP RECEIVER ####
#########################################


# This is an embedded perl SNMP trapper receiver designed for
# sending data to the server.
# The receiver will pass the received SNMP traps to Zabbix server
# or proxy running on the
# same machine. Please configure the server/proxy accordingly.
#
# Read more about using embedded perl with Net-SNMP:
#       http://net-snmp.sourceforge.net/wiki/index.php/Tut:Extending_snmpd_using_perl

This first section contains just the licensing information and a brief description of the script. Nothing that's worth mentioning, except a simple reminder—check that your perl executable is correctly referenced in the first line, or change it accordingly. The following section is more interesting, and if you are happy with the script's default formatting of SNMP traps, it may also be the only section that you will ever need to customize:

#################################################
#### ZABBIX SNMP TRAP RECEIVER CONFIGURATION ####
#################################################

$SNMPTrapperFile = '/tmp/zabbix_traps.tmp';
$DateTimeFormat = '%H:%M:%S %Y/%m/%d';

Just set $SNMPTrapperFile to the path of the file that you wish the script to log its trap to, and set the SNMPTrapperFile option in your zabbix_server.conf file to the same value. While you are at it, also set StartSNMPTrapper to 1 in zabbix_server.conf so that the server will start monitoring the said file.

$DateTimeFormat, on the other hand, should match the format of the actual SNMP traps you receive from the remote agents. Most of the time, the default value is correct, but take the time to check it and change it as needed.

The following section contains the actual logic of the script. Notice how the bulk of the logic is contained in a subroutine called zabbix_receiver. This subroutine will be called and executed towards the end of the script but is worth examining in detail:

###################################
#### ZABBIX SNMP TRAP RECEIVER ####
###################################
use Fcntl qw(O_WRONLY O_APPEND O_CREAT);
use POSIX qw(strftime);
sub zabbix_receiver
{
        my (%pdu_info) = %{$_[0]};
        my (@varbinds) = @{$_[1]};

The snmptrapd daemon will execute the script and pass the trap that it just received. The script will, in turn, call its subroutine, which will immediately distribute the trap information into two lists—the first argument is assigned to the %pdu_info hash and the second one to the @varbinds array:

# open the output file
unless (sysopen(OUTPUT_FILE, $SNMPTrapperFile,O_WRONLY|O_APPEND|O_CREAT, 0666))
  {
    print STDERR "Cannot open [$SNMPTrapperFile]:$!
";
    return NETSNMPTRAPD_HANDLER_FAIL;
  }

Here, the script will open the output file or fail graciously if it somehow cannot. The next step consists of extracting the hostname (or IP address) of the agent that sent the trap. This information is stored in the %pdu_info hash we defined previously:

# get the host name
my $hostname = $pdu_info{'receivedfrom'} || 'unknown';
if ($hostname ne 'unknown') {
  $hostname =~ /[(.*?)].*/;
  $hostname = $1 || 'unknown';
}

Now, we are ready to build the actual SNMP trap notification message. The first part of the output will be used by Zabbix to recognize the presence of a new trap (by looking for the ZBXTRAP string and knowing which of the monitored hosts the trap refers to). Keep in mind that the IP address or hostname set here must match the SNMP address value in the host configuration as set using the Zabbix frontend. This value must be set even if it's identical to the main IP/hostname for a given host. Once the Zabbix server has identified the correct host, it will discard this part of the trap notification:

# print trap header
#       timestamp must be placed at the beginning of the first line (can be omitted)
#       the first line must include the header "ZBXTRAP [IP/DNS address] "
#       * IP/DNS address is the used to find the corresponding SNMP trap items
#       * this header will be cut during processing (will not appear in the item value)
printf OUTPUT_FILE "%s ZBXTRAP %s
",
strftime($DateTimeFormat, localtime), $hostname;

After the notification header, the script will output the rest of the trap as received by the SNMP agent:

# print the PDU info
print OUTPUT_FILE "PDU INFO:
";
foreach my $key(keys(%pdu_info))
{
  printf OUTPUT_FILE "  %-30s %s
", $key,
  $pdu_info{$key};
}

The printf statement in the previous code will circle over the %pdu_info hash and output every key-value pair:

# print the variable bindings:
print OUTPUT_FILE "VARBINDS:
";
foreach my $x (@varbinds)
{

  printf OUTPUT_FILE "  %-30s type=%-2d value=%s
", $x->[0], $x->[2], $x->[1];
}
close (OUTPUT_FILE);
return NETSNMPTRAPD_HANDLER_OK;
}

The second printf statement, printf OUTPUT_FILE " %-30s type=%-2d value=%s ", $x->[0], $x->[2], $x->[1];, will output the contents of the @varbinds array one by one. This array is the one that contains the actual values reported by the trap. Once done, the log file is closed and the execution of the subroutine ends with an exit message:

NetSNMP::TrapReceiver::register("all", &zabbix_receiver) or
        die "failed to register Zabbix SNMP trap receiver
";
print STDOUT "Loaded Zabbix SNMP trap receiver
";

The last few lines of the script set the zabbix_receiver subroutine as the actual trap handler and give feedback about its correct setup. Once the trap handler starts populating the zabbix_traps.log log file, you need to define the corresponding Zabbix items.

As you've already seen, the first part of the log line is used by the Zabbix trap receiver to match a trap with its corresponding host. The second part is matched to the aforesaid host's SNMP trap item's RegExp definitions, and its contents are added to every matching item's history of values. This means that if you wish to have a startup trap item for a given host, you'll need to configure an SNMP trap item with an snmptrap["coldStart"] key, as shown in the following screenshot:

From now on, you'll be able to see the contents of the trap in the item's data history.