A monitoring system is, by definition, all about connecting and communicating with other systems. On the one hand, it needs to connect to its monitored objects in order to take measurements and evaluate the service status. On the other hand, it needs to be able to communicate the collected information outside of itself so that system administrators can act on the data and an alarm is raised. In the previous chapters of the book, we focused mainly on the first part of the equation, namely collecting data, and always assumed that the second part, exposing data and warnings, would involve sending a series of messages to human operators. While this is certainly the most common setup, the one that will be at the core of every Zabbix deployment, it's also true that it can prove to be quite limited in a large, complex IT environment.
Every managing system has a specific, detailed view of its environment that is directly dependent on the function it must perform. Identity management systems know all about users, passwords, and permissions, while inventory systems keep track of hardware and software configurations and deployment. Trouble ticketing systems keep track of current issues with users, while monitoring systems keep track of the availability status and performance metrics of anything they can connect to. As many of these systems actually share some common data among them, whether it is user information, connection configuration, or anything else, it is vital that as much of this data as possible is allowed to pass from one system to the next without constant manual intervention on the part of the administrators.
It will be impossible for Zabbix or any monitoring system to come with an out-of-the-box integration with any other arbitrary system in the world. Its open source nature, clear protocol, and powerful APIs make it relatively easy to integrate your monitoring system with any other IT management tools you have deployed in your environment. This can be the subject of a book in itself, but we will try to get you started on the path of Zabbix integration by looking at one such integration possibility.
In this chapter, you will see a concrete example of integration between Zabbix and WhatsApp™ and an example of integration between Zabbix and Request Tracker (RT). I don't think there is any need to explain what WhatsApp is as it is a widely known messaging system that now supports encryption and even phone calls using VoIP.
Request Tracker is the open source trouble ticket management system from Best Practical (http://bestpractical.com/rt/). By the end of the chapter, you will be able to do the following:
There won't be any new concepts or Zabbix functionality explained here as we'll explore some of the real-world applications made possible by the features we have already learned about in the rest of the book.
WhatsApp is so widely used that it does not require any kind of presentation. More interesting is that, on the other hand, the libraries that have been developed using the WhatsApp communication protocol. In this example, we are going to use a Python library, Yowsup, that will enable us to interact with WhatsApp. Anyway, during the year, we had quite a few libraries that were developed around this service. Yowsup has been used to create an unofficial WhatsApp client for Nokia N9 through the Wazapp project, which was in use by more than 200K users. Another fully featured unofficial client for Blackberry 10 has been created using Yowsup, which is a robust component that we can use for our integration.
Let's have a look at the requirements:
python-dateutil
protobuf
, pycrypto
, and python-axolotl-curve25519
yowsup-cli
: argparse
, readline
, and pillow
(to send images)Then, we can start installing the required packages as root with yum
:
$ yum install python python-dateutil python-argparse
Yum
, as usual, will resolve all the dependencies for us; now we can finally start downloading Yowsup. You need to decide whether you prefer to clone the Git repository or download directly archive packages of master
. In this example, we will download the package:
# wget https://github.com/tgalal/yowsup/archive/master.zip
Once the zip archive has been saved, we can extract it using:
# unzip master.zip
This will expand the zip archive by reproducing the Git directory structure. Now, we can step into the main directory:
# cd ./yowsup-master
And from there, we can build the project. To build the software, you need to have installed even python-devel
. You can install it with:
# yum install -y python-devel
Now, you can finally build the project using:
# python setup.py install
setup.py
will resolve all the dependencies for us, avoiding the use of pip
, which installs all the dependencies and the required packages manually.
Now that we are finally ready to configure our package, the first thing to do is create the configuration file. The configuration needs to be in the following form:
# cat ./yowsup.config cc= phone= id= password=
The field cc
must be filled with the country code.
The phone
field is composed of country code + area code + phone number. Please remember that the country code must be provided without + or 00 leading.
The ID field is used in registration calls and for logging in. WhatsApp has recently deprecated using IMEI/MAC to generate the account's password in updated versions of their clients. Use the --v1
switch to try it anyway. Typically, this field should contain the phone's IMEI if your account is set up on a Nokia or Android device or the MAC address of the phone's WLAN for iOS devices. If you are not trying to use existing credentials, you can leave this field blank or remove it.
Finally, password
has the login password. You will get this password when you register the phone number using yowsup-cli
. If you are registering a number, you need to leave this field blank and populate it once you have your password.
Now it's time to register our client, thus enabling it to send messages.
First of all, we need a phone number to sacrifice; this phone number will then be used for this application. Here, it is important to have a real mobile number where we can receive SMS.
To register our client, we need to properly fill in the configuration file as previously explained. We need then to populate id
and phone number
in our yowsup.config
configuration file. We can let the other fields remain empty for now.
Once this is done, we can run the following command:
# ./yowsup-cli registration -c ./yowsup.config -r sms INFO:yowsup.common.http.warequest:{"status":"sent","length":6,"method":"sms","retry_after":1805} status: sent retry_after: 1805 length: 6 method: sms #
Once this is done, we should receive an SMS in our phone in the NNN-NNN form. We need to use this command as follows:
# ./yowsup-cli registration -c ./yowsup.config -R 117-741 INFO:yowsup.common.http.warequest:{"status":"ok","login":"41076XXXXXX","pw":"w3cp6Vb7UAUlKG6/xhx/1K4hA=","type":"existing","expiration":1465119599,"kind":"free","price":"u20ac 0,89","cost":"0.89","currency":"EUR","price_expiration":1439763526} status: ok kind: free pw: w3cp6Vb7UAUlKG6/xhx/1K4hA= price: € 0,89 price_expiration: 1439763526 currency: EUR cost: 0.89 expiration: 1465119599 login: 41076XXXXXXX type: existing #
Now, we have received the password encoded in BASE64. The password is specified in the field as pw: w3cp6Vb7UAUlKG6/xhx/1K4hA=
. We need to include this password in our yowsup.config
configuration file.
Finally, we have everything ready to be used. The first thing that we can try to do is send a message. Now, for all the time required for those tests, we can use the new yousup
account. From there, we can run the following:
# $HOME/yowsup-master/yowsup-cli demos -c ./yowsup.config -s 4176XXXXX "Test message form cli" WARNING:yowsup.stacks.yowstack:Implicit declaration of parallel layers in a tuple is deprecated, pass a YowParallelLayer instead INFO:yowsup.demos.sendclient.layer:Message sent Yowsdown
We can now send another message and test whether the messages are getting delivered. Then, we can run the following from yowsup
:
# $HOME/yowsup-master/yowsup-cli demos -c ./yowsup.config -s 4176XXXXX "Test message form cli. 2nd test" WARNING:yowsup.stacks.yowstack:Implicit declaration of parallel layers in a tuple is deprecated, pass a YowParallelLayer instead INFO:yowsup.demos.sendclient.layer:Message sent
Now, we can see the result on our phone or directly online using WhatsApp web. The result is the following:
Now, let's see what the options used are. First of all, we've used the demo client.
Before proceeding any further, it makes sense to restrict access to yowsup
to Zabbix and the relative Zabbix server account.
To do that, we need to create a user ad hoc, for example, yowsup
. Then from root, we can run the following command:
# useradd yowsup
Create the relative password that executes the following command from root:
# passwd yowsup Changing password for user yowsup. New password: Retype new password: passwd: all authentication tokens updated successfully.
Now it is time to edit the sudoers and allow the account using your Zabbix server to execute the required command. Then, we need to run the following from root:
#visudo -f /etc/sudoers.d/Zabbix
We need to add the following content:
zabbix ALL=(ALL) NOPASSWD: /usr/bin/sudo -l zabbix ALL=(ALL) NOPASSWD: /home/yowsup/yowsup-master/yowsup-cli *
Now, we can test whether the account with which Zabbix will be able to run all the required commands can become our Zabbix account. Then, type the following command:
$ sudo -l
The output must contain the following section:
User zabbix may run the following commands on this host: (ALL) NOPASSWD: /usr/bin/sudo –l (ALL) NOPASSWD: /home/yowsup/yowsup-master/yowsup-cli *
Now, the last thing to do is transfer all the files and data to our new yowsup
account by running the following command as root:
# cp -r -a yowsup-master /home/yowsup/ # chown –R yowsup:yowsup /home/yowsup/*
Test whether everything works as expected by running the following command from the Zabbix account:
$ sudo -u yowsup /home/yowsup/yowsup-master/yowsup-cli Available commands: =================== demos, version, registration
Now, if you don't get the same output, it is better to check your configuration. Now, as a final test, we can send a message from the Zabbix account, and then, from Zabbix, you can run:
$ sudo -u yowsup /home/yowsup/yowsup-master/yowsup-cli demos -c /home/yowsup/yowsup-master/yowsup.config -s 4176XXXXXX "Test message form zabbix 1st test" WARNING:yowsup.stacks.yowstack:Implicit declaration of parallel layers in a tuple is deprecated, pass a YowParallelLayer instead INFO:yowsup.demos.sendclient.layer:Message sent Yowsdown
To confirm that everything works as expected, you should see the message arrive at your terminal or WhatsApp web, as shown in the following screenshot:
Here, you see that the message has been sent by me as I have saved the number that I use to send messages from Zabbix under my name.
Now that we've secured and locked down our setup by taking care to grant Zabbix the required privilege to send messages, but to avoid reading the configuration password file, it is time to think about a real scenario of usage. Now, you've tasted the basic functionality of this software, but in a real scenario, we need to consider that the messages need to be delivered to a team or a group of people that might change from time to time following the nightshift plan, the weekly support shift plan, and so on. To solve this problem, we can simply create a WhatsApp group. Luckily, the software provides us with the functionality to create a group and add/remove people from a group, among many other functions.
Here, we will see how we can create a group called zabbix_alert
in this example. From the yowsup
account, we can then run the following command:
# cd yowsup-master && ./yowsup-cli demos -c yowsup.config --yowsup
This command starts the Yowsup command-line client. It is actually an interactive shell that allows us to send extended commands to WhatsApp. The following is the welcome message:
Yowsup Cli client ================== Type /help for available commands
Now, if we type /help
, we can have an idea of the power of this shell; let's do it:
[offline]:/help ---------------------------------------------- /profile setPicture [path] Set profile picture /profile setStatus [text] Set status text /account delete Delete your account /group info [group_jid] Get group info /group picture [group_jid] [path] Set group picture /group invite [group_jid] [jids] Invite to group. Jids are a comma separated list /group leave [group_jid] Leave a group you belong to /group setSubject [group_jid] [subject] Change group subject /group demote [group_jid] [jids] Remove admin of a group. Jids are a comma separated list /group promote [group_jid] [jids] Promote admin of a group. Jids are a comma separated list /group kick [group_jid] [jids] Kick from group. Jids are a comma separated list /help Print this message /seq Send init seq /contacts sync [contacts] Sync contacts, contacts should be comma separated phone numbers, with no spaces /keys set Send prekeys /keys get [jids] Get shared keys /image send [number] [path] Send and image /presence available Set presence as available /presence subscribe [contact] Subscribe to contact's presence updates /presence unsubscribe [contact] Unsubscribe from contact's presence updates /presence name [name] Set presence name /presence unavailable Set presence as unavailable /ping Ping server /L Quick login /state paused [jid] Send paused state /state typing [jid] Send typing state /contact picture [jid] Get profile picture for contact /contact picturePreview [jid] Get profile picture preview for contact /contact lastseen [jid] Get lastseen for contact /groups create [subject] [jids] Create a new group with the specified subject and participants. Jids are a comma separated list. Use '-' to keep group without participants but you. /groups list List all groups you belong to /disconnect Disconnect /login [username] [b64password] Login to WhatsApp /ib clean [dirtyType] Send clean dirty /message broadcast [numbers] [content] Broadcast message. numbers should comma separated phone numbers /message send [number] [content] Send message to a friend ---------------------------------------------- [offline]:
As you can quickly spot, this is a very complete client as it allows us to operate against all the possible options that the messaging service provides us.
Now, before being able to create a group, we need to log in. Note that the shell provides you your status. In this case, we are still [offline]
. We can use the quick login as we've specified in our configuration file after the –c
option. Then, we can simply run this command:
[offline]:/L Auth: Logged in! [connected]:
Now, the status is changed to [connected]
, and we can finally send commands. Now we will create the group with group create, followed by group name and a comma separated list of phone numbers we would like to add; in this example, it is only one number, but you can add all the numbers you wish to add here in a comma-separated list:
[connected]:/groups create zabbix_alert 4176XXXXXX
[connected]:INFO:yowsup.layers.protocol_groups.layer:Group create success Iq: ID: 1 Type: result from: g.us Notification: Notification From: [email protected] Type: w:gp2 Participant: [email protected] Creator: 39340XXXXXXX @s.whatsapp.net Create type: new Creation timestamp: 1436940409 Subject: zabbix_alert Subject owner: [email protected] Subject timestamp: 1436940409 Participants: {[email protected]': 'admin', '[email protected]': None} [connected]:
The result of this command is shown in the following screenshot:
Here, the group JID and the group identifier are:
From: [email protected]
The JID is composed of the phone number that creates the group, followed by a unique identifier. Now we are ready to send the first message to this group using a command line. We can run the following command:
# ./yowsup-cli demos -c ./yowsup.config -s [email protected] "Test message to zabbix_alert group"
The result is shown in the following screenshot:
Now, as the final step, it make sense to have more than one group administrator as it is safer to have someone human that can manage the emergency by adding a newcomer who is not included in the automated process, and so on.
To add one more group administrator, we need to log in and access the interactive shell with:
# ./yowsup-cli demos -c ./yowsup.config --yowsup Yowsup Cli client ================== Type /help for available commands [offline]:/L Auth: Logged in! [connected]:
Now, we can run our command, which will be a group command. Then promote the group JID and the list of numbers that we want promote to admin. Here is just one number:
[connected]:/group promote [email protected] 4176XXXXXX [connected]:INFO:yowsup.layers.protocol_groups.layer:Group promote participants success [connected]:
The result is shown in the following screenshot:
Now, I can personally add and remove contacts from this group.
Now we are finally ready to integrate Zabbix with our WhatsApp gateway. First of all, we need to create the appropriate script to use the command line by using the proper sudo
command. The script needs to be placed at the AlertScript
location that we can retrieve from here:
grep AlertScript /etc/zabbix/zabbix_server.conf ### Option: AlertScriptsPath # AlertScriptsPath=${datadir}/zabbix/alertscripts AlertScriptsPath=/usr/lib/zabbix/alertscripts
Then, we can create our script in the /usr/lib/zabbix/alertscripts
directory.
We can create a script called whatsapp.sh
with the following content:
$ cat /usr/lib/zabbix/alertscripts/whatsapp.sh #!/bin/bash BASEDIR=/home/yowsup/yowsup-master sudo -u yowsup $BASEDIR/yowsup-cli demos -c $BASEDIR/yowsup.config -s $1 "$2 $3"
Now it's time to create a new notification method that will use our brand-new script. To create a new media type, you need to navigate to Administration | Media type | Create media type and fill in the form, as shown in the following screenshot:
Now, we need to create the action that will use our new media type. Let's then go on to Configuration | Actions, select Trigger in the drop-down menu, and click on Create action, as shown in the following screenshot:
Then, we need to define in the Operations tab to whom we would like to send this message. Here, we've decided to send the message to the entire Zabbix administrators group, as shown in the following screenshot:
Now, we need to populate all the media fields of all the accounts that would like to receive alerts and that are part of this example of the Zabbix administrators group.
Then, we need to go to Administration | Users, select the user, and then add media as whatsapp. Then, we need to write the phone number without + or 00 in front of the country code, as shown in the following screenshot:
Here, of course, we can select which severity can be sent out using this media.
Now, we can act in two different ways—either send the messages to all the accounts listed in a group in the media section, or use the WhatsApp group. Then, in our case, we can define just a group with one account or even only an account that uses the group [email protected]
(that we created a few pages ago) as media.
We can debug and see the flow of messages sent to our media by looking at the actions and monitoring them after navigating to Administration | Audit and selecting Action log. There, we can see all the actions that are triggered. In the following screenshot, you see an event, which I've caused, to test whether everything works as expected. In the next screenshot, you can see the event caused by a temporary iptables rule that has been properly tracked:
I've also slightly changed our whatsapp.sh
script in order to properly track how it is called:
$ cat /usr/lib/zabbix/alertscripts/whatsapp.sh
#!/bin/bash
BASEDIR=/home/yowsup/yowsup-master
echo "sudo -u yowsup $BASEDIR/yowsup-cli demos -c $BASEDIR/yowsup.config -s $1 "$2 $3"" >> /var/log/whatsapp.log
sudo -u yowsup $BASEDIR/yowsup-cli demos -c $BASEDIR/yowsup.config -s $1 "$2 $3"
As you can see in the highlighted line, I've added a sort of log. Now, let's see how our script has been called:
$ tail -n 12 /var/log/whatsapp.log sudo -u yowsup /home/yowsup/yowsup-master/yowsup-cli demos -c /home/yowsup/yowsup-master/yowsup.config -s 4176XXXXXXX "OK: More than 100 items having missing data for more than 10 minutes Trigger: More than 100 items having missing data for more than 10 minutes Trigger status: OK Trigger severity: Warning Trigger URL: Item values: 1. Zabbix queue over 10m (Zabbix server:zabbix[queue,10m]): 0 2. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN* 3. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN* Original event ID: 1060"
As you can see in the following screenshot, the command has been called properly, and even if the message is written in multiple lines, it has been delivered properly. Now, for our end-to-end test, we can close our check with the message received:
This integration can be really useful, especially nowadays that people have smartphones always connected to the network. Here, there are some things to take into account. First of all, we need to decide whether we want to send an alarm to a specific group, or people individually. If we want to alarm the group, we need to use the group JID and then 39340XXXXXXX-1436940409
.
The same message has also been delivered to the Zabbix_alert
group as, within the Zabbix administrator group we previously configured, the WhatsApp group JID is the default WhatsApp media for Admin (Zabbix administrator).
The following screenshot displays the result:
Now, we can move on and see how to integrate Zabbix with RT.