Chapter 10. Integrating Zabbix

A monitoring system is, by definition, all about connecting and communicating with other systems. On the one hand, it needs to connect to its monitored objects in order to take measurements and evaluate the service status. On the other hand, it needs to be able to communicate the collected information outside of itself so that system administrators can act on the data and an alarm is raised. In the previous chapters of the book, we focused mainly on the first part of the equation, namely collecting data, and always assumed that the second part, exposing data and warnings, would involve sending a series of messages to human operators. While this is certainly the most common setup, the one that will be at the core of every Zabbix deployment, it's also true that it can prove to be quite limited in a large, complex IT environment.

Every managing system has a specific, detailed view of its environment that is directly dependent on the function it must perform. Identity management systems know all about users, passwords, and permissions, while inventory systems keep track of hardware and software configurations and deployment. Trouble ticketing systems keep track of current issues with users, while monitoring systems keep track of the availability status and performance metrics of anything they can connect to. As many of these systems actually share some common data among them, whether it is user information, connection configuration, or anything else, it is vital that as much of this data as possible is allowed to pass from one system to the next without constant manual intervention on the part of the administrators.

It will be impossible for Zabbix or any monitoring system to come with an out-of-the-box integration with any other arbitrary system in the world. Its open source nature, clear protocol, and powerful APIs make it relatively easy to integrate your monitoring system with any other IT management tools you have deployed in your environment. This can be the subject of a book in itself, but we will try to get you started on the path of Zabbix integration by looking at one such integration possibility.

In this chapter, you will see a concrete example of integration between Zabbix and WhatsApp™ and an example of integration between Zabbix and Request Tracker (RT). I don't think there is any need to explain what WhatsApp is as it is a widely known messaging system that now supports encryption and even phone calls using VoIP.

Request Tracker is the open source trouble ticket management system from Best Practical (http://bestpractical.com/rt/). By the end of the chapter, you will be able to do the following:

  • Route an alert directly to your Unix system administrator and support via WhatsApp
  • Integrate custom media with Zabbix
  • Relay an alert to a trouble ticketing system
  • Keep track of which tickets relate to which events
  • Update event acknowledgments based on the status of a ticket
  • Automatically close specific tickets when a trigger returns to an OK state

There won't be any new concepts or Zabbix functionality explained here as we'll explore some of the real-world applications made possible by the features we have already learned about in the rest of the book.

Stepping into WhatsApp

WhatsApp is so widely used that it does not require any kind of presentation. More interesting is that, on the other hand, the libraries that have been developed using the WhatsApp communication protocol. In this example, we are going to use a Python library, Yowsup, that will enable us to interact with WhatsApp. Anyway, during the year, we had quite a few libraries that were developed around this service. Yowsup has been used to create an unofficial WhatsApp client for Nokia N9 through the Wazapp project, which was in use by more than 200K users. Another fully featured unofficial client for Blackberry 10 has been created using Yowsup, which is a robust component that we can use for our integration.

Let's have a look at the requirements:

  • Python 2.6+, or Python 3.0 +
  • Python packages: python-dateutil
  • Python packages for end-to-end encryption: protobuf, pycrypto, and python-axolotl-curve25519
  • Python packages for yowsup-cli: argparse, readline, and pillow (to send images)

Then, we can start installing the required packages as root with yum:

$ yum install python python-dateutil python-argparse

Yum, as usual, will resolve all the dependencies for us; now we can finally start downloading Yowsup. You need to decide whether you prefer to clone the Git repository or download directly archive packages of master. In this example, we will download the package:

# wget https://github.com/tgalal/yowsup/archive/master.zip

Once the zip archive has been saved, we can extract it using:

# unzip master.zip

This will expand the zip archive by reproducing the Git directory structure. Now, we can step into the main directory:

# cd ./yowsup-master

And from there, we can build the project. To build the software, you need to have installed even python-devel. You can install it with:

# yum install -y python-devel

Now, you can finally build the project using:

# python setup.py install

setup.py will resolve all the dependencies for us, avoiding the use of pip, which installs all the dependencies and the required packages manually.

Getting ready to send messages

Now that we are finally ready to configure our package, the first thing to do is create the configuration file. The configuration needs to be in the following form:

# cat ./yowsup.config
cc=
phone=
id=
password=

The field cc must be filled with the country code.

The phone field is composed of country code + area code + phone number. Please remember that the country code must be provided without + or 00 leading.

The ID field is used in registration calls and for logging in. WhatsApp has recently deprecated using IMEI/MAC to generate the account's password in updated versions of their clients. Use the --v1 switch to try it anyway. Typically, this field should contain the phone's IMEI if your account is set up on a Nokia or Android device or the MAC address of the phone's WLAN for iOS devices. If you are not trying to use existing credentials, you can leave this field blank or remove it.

Finally, password has the login password. You will get this password when you register the phone number using yowsup-cli. If you are registering a number, you need to leave this field blank and populate it once you have your password.

Tip

It is recommended that you set a permission 600 to the configuration file, and since the command line will be used by our Zabbix server account, you can enforce the security with a sudo role provided only to your Zabbix account. Only then the Zabbix server will be able to send out messages.

Registering the yowsup client

Now it's time to register our client, thus enabling it to send messages.

First of all, we need a phone number to sacrifice; this phone number will then be used for this application. Here, it is important to have a real mobile number where we can receive SMS.

To register our client, we need to properly fill in the configuration file as previously explained. We need then to populate id and phone number in our yowsup.config configuration file. We can let the other fields remain empty for now.

Once this is done, we can run the following command:

# ./yowsup-cli registration -c ./yowsup.config -r sms
INFO:yowsup.common.http.warequest:{"status":"sent","length":6,"method":"sms","retry_after":1805}
status: sent
retry_after: 1805
length: 6
method: sms
#

Once this is done, we should receive an SMS in our phone in the NNN-NNN form. We need to use this command as follows:

# ./yowsup-cli registration -c ./yowsup.config -R 117-741
INFO:yowsup.common.http.warequest:{"status":"ok","login":"41076XXXXXX","pw":"w3cp6Vb7UAUlKG6/xhx/1K4hA=","type":"existing","expiration":1465119599,"kind":"free","price":"u20ac 0,89","cost":"0.89","currency":"EUR","price_expiration":1439763526}

status: ok
kind: free
pw: w3cp6Vb7UAUlKG6/xhx/1K4hA=
price: € 0,89
price_expiration: 1439763526
currency: EUR
cost: 0.89
expiration: 1465119599
login: 41076XXXXXXX
type: existing
#

Now, we have received the password encoded in BASE64. The password is specified in the field as pw: w3cp6Vb7UAUlKG6/xhx/1K4hA=. We need to include this password in our yowsup.config configuration file.

Sending the first WhatsApp message

Finally, we have everything ready to be used. The first thing that we can try to do is send a message. Now, for all the time required for those tests, we can use the new yousup account. From there, we can run the following:

# $HOME/yowsup-master/yowsup-cli demos -c ./yowsup.config -s 4176XXXXX "Test message form cli"
WARNING:yowsup.stacks.yowstack:Implicit declaration of parallel layers in a tuple is deprecated, pass a YowParallelLayer instead
INFO:yowsup.demos.sendclient.layer:Message sent

Yowsdown

We can now send another message and test whether the messages are getting delivered. Then, we can run the following from yowsup:

# $HOME/yowsup-master/yowsup-cli demos -c ./yowsup.config -s 4176XXXXX "Test message form cli. 2nd test"
WARNING:yowsup.stacks.yowstack:Implicit declaration of parallel layers in a tuple is deprecated, pass a YowParallelLayer instead
INFO:yowsup.demos.sendclient.layer:Message sent

Now, we can see the result on our phone or directly online using WhatsApp web. The result is the following:

Sending the first WhatsApp message

Now, let's see what the options used are. First of all, we've used the demo client.

Securing the yowsup setup

Before proceeding any further, it makes sense to restrict access to yowsup to Zabbix and the relative Zabbix server account.

To do that, we need to create a user ad hoc, for example, yowsup. Then from root, we can run the following command:

# useradd yowsup

Create the relative password that executes the following command from root:

# passwd yowsup
Changing password for user yowsup.
New password:
Retype new password:
passwd: all authentication tokens updated successfully. 

Now it is time to edit the sudoers and allow the account using your Zabbix server to execute the required command. Then, we need to run the following from root:

#visudo -f /etc/sudoers.d/Zabbix

We need to add the following content:

zabbix ALL=(ALL) NOPASSWD: /usr/bin/sudo -l
zabbix ALL=(ALL) NOPASSWD: /home/yowsup/yowsup-master/yowsup-cli *

Now, we can test whether the account with which Zabbix will be able to run all the required commands can become our Zabbix account. Then, type the following command:

$ sudo -l

The output must contain the following section:

User zabbix may run the following commands on this host:
    (ALL) NOPASSWD: /usr/bin/sudo –l
    (ALL) NOPASSWD: /home/yowsup/yowsup-master/yowsup-cli *

Now, the last thing to do is transfer all the files and data to our new yowsup account by running the following command as root:

# cp -r -a yowsup-master /home/yowsup/
# chown –R yowsup:yowsup /home/yowsup/*

Tip

Yowsup stores all the history at $HOME/.yowsup/ just in case you are relocating a preexistent setup. This is something you need to consider.

Test whether everything works as expected by running the following command from the Zabbix account:

$ sudo -u yowsup  /home/yowsup/yowsup-master/yowsup-cli
Available commands:
===================
demos, version, registration

Now, if you don't get the same output, it is better to check your configuration. Now, as a final test, we can send a message from the Zabbix account, and then, from Zabbix, you can run:

$ sudo -u yowsup /home/yowsup/yowsup-master/yowsup-cli demos -c /home/yowsup/yowsup-master/yowsup.config -s 4176XXXXXX "Test message form zabbix 1st test"
WARNING:yowsup.stacks.yowstack:Implicit declaration of parallel layers in a tuple is deprecated, pass a YowParallelLayer instead
INFO:yowsup.demos.sendclient.layer:Message sent

Yowsdown

To confirm that everything works as expected, you should see the message arrive at your terminal or WhatsApp web, as shown in the following screenshot:

Securing the yowsup setup

Here, you see that the message has been sent by me as I have saved the number that I use to send messages from Zabbix under my name.

Creating our first Zabbix alert group

Now that we've secured and locked down our setup by taking care to grant Zabbix the required privilege to send messages, but to avoid reading the configuration password file, it is time to think about a real scenario of usage. Now, you've tasted the basic functionality of this software, but in a real scenario, we need to consider that the messages need to be delivered to a team or a group of people that might change from time to time following the nightshift plan, the weekly support shift plan, and so on. To solve this problem, we can simply create a WhatsApp group. Luckily, the software provides us with the functionality to create a group and add/remove people from a group, among many other functions.

Here, we will see how we can create a group called zabbix_alert in this example. From the yowsup account, we can then run the following command:

# cd yowsup-master && ./yowsup-cli demos -c yowsup.config  --yowsup

This command starts the Yowsup command-line client. It is actually an interactive shell that allows us to send extended commands to WhatsApp. The following is the welcome message:

Yowsup Cli client
==================
Type /help for available commands

Now, if we type /help, we can have an idea of the power of this shell; let's do it:

[offline]:/help
----------------------------------------------
/profile  setPicture  [path]                Set profile picture
/profile  setStatus   [text]                Set status text
/account  delete                            Delete your account
/group    info        [group_jid]           Get group info
/group    picture     [group_jid] [path]    Set group picture
/group    invite      [group_jid] [jids]    Invite to group. Jids are a comma separated list
/group    leave       [group_jid]           Leave a group you belong to
/group    setSubject  [group_jid] [subject] Change group subject
/group    demote      [group_jid] [jids]    Remove admin of a group. Jids are a comma separated list
/group    promote     [group_jid] [jids]    Promote admin of a group. Jids are a comma separated list
/group    kick        [group_jid] [jids]    Kick from group. Jids are a comma separated list
/help                                       Print this message
/seq                                        Send init seq
/contacts  sync       [contacts]            Sync contacts, contacts should be comma separated phone numbers, with no spaces
/keys      set                              Send prekeys
/keys      get        [jids]                Get shared keys
/image     send       [number] [path]       Send and image
/presence  available                        Set presence as available
/presence  subscribe  [contact]             Subscribe to contact's presence updates
/presence  unsubscribe [contact]            Unsubscribe from contact's presence updates
/presence  name        [name]               Set presence name
/presence  unavailable                      Set presence as unavailable
/ping                                       Ping server
/L                                          Quick login
/state      paused  [jid]                   Send paused state
/state      typing  [jid]                   Send typing state
/contact    picture [jid]                   Get profile picture for contact
/contact       picturePreview [jid]          Get profile picture preview for contact
/contact       lastseen       [jid]          Get lastseen for contact
/groups        create [subject] [jids]       Create a new group with the specified subject and participants. Jids are a comma separated list. Use '-' to keep group without participants but you.
/groups        list                          List all groups you belong to
/disconnect                                  Disconnect
/login        [username] [b64password]       Login to WhatsApp
/ib           clean      [dirtyType]         Send clean dirty
/message      broadcast [numbers] [content]  Broadcast message. numbers should comma separated phone numbers
/message       send [number] [content]       Send message to a friend
----------------------------------------------
[offline]:

As you can quickly spot, this is a very complete client as it allows us to operate against all the possible options that the messaging service provides us.

Now, before being able to create a group, we need to log in. Note that the shell provides you your status. In this case, we are still [offline]. We can use the quick login as we've specified in our configuration file after the –c option. Then, we can simply run this command:

[offline]:/L
Auth: Logged in!
[connected]:

Now, the status is changed to [connected], and we can finally send commands. Now we will create the group with group create, followed by group name and a comma separated list of phone numbers we would like to add; in this example, it is only one number, but you can add all the numbers you wish to add here in a comma-separated list:

[connected]:/groups create zabbix_alert 4176XXXXXX

The following is the output:

[connected]:INFO:yowsup.layers.protocol_groups.layer:Group create success
Iq:
ID: 1
Type: result
from: g.us

Notification: Notification
From: [email protected]
Type: w:gp2
Participant: [email protected]
Creator: 39340XXXXXXX @s.whatsapp.net
Create type: new
Creation timestamp: 1436940409
Subject: zabbix_alert
Subject owner: [email protected]
Subject timestamp: 1436940409
Participants: {[email protected]': 'admin', '[email protected]': None}

[connected]:

The result of this command is shown in the following screenshot:

Creating our first Zabbix alert group

Here, the group JID and the group identifier are:

The JID is composed of the phone number that creates the group, followed by a unique identifier. Now we are ready to send the first message to this group using a command line. We can run the following command:

# ./yowsup-cli demos -c ./yowsup.config -s [email protected] "Test message to zabbix_alert group"

The result is shown in the following screenshot:

Creating our first Zabbix alert group

Now, as the final step, it make sense to have more than one group administrator as it is safer to have someone human that can manage the emergency by adding a newcomer who is not included in the automated process, and so on.

To add one more group administrator, we need to log in and access the interactive shell with:

# ./yowsup-cli demos -c ./yowsup.config --yowsup
Yowsup Cli client
==================
Type /help for available commands

[offline]:/L
Auth: Logged in!
[connected]:

Now, we can run our command, which will be a group command. Then promote the group JID and the list of numbers that we want promote to admin. Here is just one number:

[connected]:/group promote [email protected] 4176XXXXXX
[connected]:INFO:yowsup.layers.protocol_groups.layer:Group promote participants success

[connected]:

The result is shown in the following screenshot:

Creating our first Zabbix alert group

Now, I can personally add and remove contacts from this group.

Integrating yowsup with Zabbix

Now we are finally ready to integrate Zabbix with our WhatsApp gateway. First of all, we need to create the appropriate script to use the command line by using the proper sudo command. The script needs to be placed at the AlertScript location that we can retrieve from here:

grep AlertScript /etc/zabbix/zabbix_server.conf
### Option: AlertScriptsPath
# AlertScriptsPath=${datadir}/zabbix/alertscripts
AlertScriptsPath=/usr/lib/zabbix/alertscripts 

Then, we can create our script in the /usr/lib/zabbix/alertscripts directory.

We can create a script called whatsapp.sh with the following content:

$ cat /usr/lib/zabbix/alertscripts/whatsapp.sh
#!/bin/bash
BASEDIR=/home/yowsup/yowsup-master
sudo -u yowsup $BASEDIR/yowsup-cli demos -c $BASEDIR/yowsup.config -s $1 "$2 $3"

Now it's time to create a new notification method that will use our brand-new script. To create a new media type, you need to navigate to Administration | Media type | Create media type and fill in the form, as shown in the following screenshot:

Integrating yowsup with Zabbix

Now, we need to create the action that will use our new media type. Let's then go on to Configuration | Actions, select Trigger in the drop-down menu, and click on Create action, as shown in the following screenshot:

Integrating yowsup with Zabbix

Then, we need to define in the Operations tab to whom we would like to send this message. Here, we've decided to send the message to the entire Zabbix administrators group, as shown in the following screenshot:

Integrating yowsup with Zabbix

Now, we need to populate all the media fields of all the accounts that would like to receive alerts and that are part of this example of the Zabbix administrators group.

Then, we need to go to Administration | Users, select the user, and then add media as whatsapp. Then, we need to write the phone number without + or 00 in front of the country code, as shown in the following screenshot:

Integrating yowsup with Zabbix

Here, of course, we can select which severity can be sent out using this media.

Now, we can act in two different ways—either send the messages to all the accounts listed in a group in the media section, or use the WhatsApp group. Then, in our case, we can define just a group with one account or even only an account that uses the group [email protected] (that we created a few pages ago) as media.

We can debug and see the flow of messages sent to our media by looking at the actions and monitoring them after navigating to Administration | Audit and selecting Action log. There, we can see all the actions that are triggered. In the following screenshot, you see an event, which I've caused, to test whether everything works as expected. In the next screenshot, you can see the event caused by a temporary iptables rule that has been properly tracked:

Integrating yowsup with Zabbix

I've also slightly changed our whatsapp.sh script in order to properly track how it is called:

$ cat /usr/lib/zabbix/alertscripts/whatsapp.sh
#!/bin/bash
BASEDIR=/home/yowsup/yowsup-master
echo "sudo -u yowsup $BASEDIR/yowsup-cli demos -c $BASEDIR/yowsup.config -s $1 "$2 $3"" >> /var/log/whatsapp.log
sudo -u yowsup $BASEDIR/yowsup-cli demos -c $BASEDIR/yowsup.config -s $1 "$2 $3"

As you can see in the highlighted line, I've added a sort of log. Now, let's see how our script has been called:

$ tail -n 12 /var/log/whatsapp.log
sudo -u yowsup /home/yowsup/yowsup-master/yowsup-cli demos -c /home/yowsup/yowsup-master/yowsup.config -s 4176XXXXXXX "OK: More than 100 items having missing data for more than 10 minutes Trigger: More than 100 items having missing data for more than 10 minutes
Trigger status: OK
Trigger severity: Warning
Trigger URL:

Item values:

1. Zabbix queue over 10m (Zabbix server:zabbix[queue,10m]): 0
2. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN*
3. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN*

Original event ID: 1060"

As you can see in the following screenshot, the command has been called properly, and even if the message is written in multiple lines, it has been delivered properly. Now, for our end-to-end test, we can close our check with the message received:

Integrating yowsup with Zabbix

This integration can be really useful, especially nowadays that people have smartphones always connected to the network. Here, there are some things to take into account. First of all, we need to decide whether we want to send an alarm to a specific group, or people individually. If we want to alarm the group, we need to use the group JID and then 39340XXXXXXX-1436940409.

The same message has also been delivered to the Zabbix_alert group as, within the Zabbix administrator group we previously configured, the WhatsApp group JID is the default WhatsApp media for Admin (Zabbix administrator).

The following screenshot displays the result:

Integrating yowsup with Zabbix

Now, we can move on and see how to integrate Zabbix with RT.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset