Zabbix provides a way to group similar items in a host in order to navigate them more easily when looking at the latest monitoring data. These item containers, called applications, come in very handy when the number of items in a host is quite consistent. If you group all CPU-monitoring items together under a label, say CPU, all filesystem-related items under filesystems, and so on, you could find the data you are looking for more easily. Applications are just labels tied to a specific template or host and are just used to categorize items. This makes them simple and lightweight. But it also means that they are not really used elsewhere in the Zabbix system.

Still, it's sometimes useful to look at the trigger status or event history, not just by the host but by the application too. A report of all network-related problems regardless of the host, host group, or specific trigger, could be very useful for some groups in your IT department. The same goes for a report on filesystem events, database problems, and so on.

Let's see how you can build a script that will export all events related to a specific application into a .csv file. The setup is basically the same as the previous examples:

#!/usr/bin/python
from pyzabbix import ZabbixAPI
import sys
import csv
from datetime import datetime
appname = sys.argv[1]
timeformat="%d/%m/%y %H:%M"
zh = ZabbixAPI("http://locahost/zabbix")
zh.login(user="Admin", password="zabbix")

As you can see, the application name is taken from the command line, while the API's URL and credentials are just examples. When you use your own, you can also consider using an external configuration file for more flexibility. Since events are recorded using a Unix timestamp, you'll need to convert it to a readable string later on. The timeformat variable will let you define your preferred format. Speaking of formats, the .csv module will let you define the output format of your report with more flexibility than a series of manual prints.

Now, you can proceed to extract all applications that share the name you passed on the command line:

 applications = zh.application.get(output="shorten", filter={"name": [appname]})

Once we have the list of applications, you can get the list of items that belong to the said application:

items = zh.item.get(output="count", applicationids=[x['applicationid'] for x in applications])

From there, you still need to extract all the triggers that contain the given items before moving on to the actual events:

triggers = zh.trigger.get(output="refer", itemids=[x['itemid'] for x in items]) 

Now, you can finally get the list of events that are related to the application you are interested in:

events = zh.event.get(triggerids=[j['triggerid'] for j in triggers])

Here, only the event IDs are extracted. You didn't ask for a specific time period, so it's possible that a great number of events will be extracted. For every event, we'll also need to extract all related hosts, triggers, and items. To do that, let's first define a couple of helper functions to get the host, item, and trigger names:

def gethostname(hostid=''):
     return zh.host.get(hostids=hostid, output=['host'])[0]['host']

def getitemname(itemid=''):
     return zh.item.get(itemids=itemid, output=['name'])[0]['name']

def gettriggername(triggerid=''):
      return zh.trigger.get(triggerids=triggerid, expandDescription="1", output=['description'])[0]['description']

Finally, you can define an empty eventstable table and fill it with event information based on what you have extracted until now:

eventstable = []
triggervalues = ['OK', 'problem', 'unknown']
for e in events:
     eventid = e['eventid']
     event = zh.event.get(eventids=eventid, 
                      selectHosts="refer", 
                      selectItems="refer",
                      selectTriggers="refer",
                      output="extend")
     host = gethostname(event[0]['hosts'][0]['hostid'])
     item = getitemname(event[0]['items'][0]['itemid'])
     trigger = gettriggername(event[0]['triggers'][0]['triggerid'])
     clock = datetime.fromtimestamp(int(event[0]['clock'])).strftime(timeformat)
     value = triggervalues[int(event[0]['value'])]
     eventstable.append({"Host": host, 
                            "Item": item, 
                            "Trigger": trigger, 
                            "Status": value,
                            "Time" : clock
                          })

Now that you have all the events' details, you can create the output .csv file:

filename = "events_" + appname + "_" + datetime.now().strftime("%Y%m%d%H%M")
fieldnames = ['Host', 'Item', 'Trigger', 'Status', 'Time']
outfile = open(filename, 'w')
csvwriter = csv.DictWriter(outfile, delimiter=';', fieldnames=fieldnames)
csvwriter.writerow(dict((h,h) for h in fieldnames))
for row in eventstable:
     csvwriter.writerow(row)
outfile.close()

The report's filename will be automatically generated based on the application you want to focus on and the time of execution. Since every event in the eventstable array is dict, a fieldnames array is needed to tell the csv.DictWriter module in what order the fields should be written. Next, a column header is written before finally looping over the eventstable array and writing out the information we want.

There are a number of ways that you can expand on this script in order to get even more useful data. Here are a few suggestions, but the list is limited only by your imagination:

At this point in the book, you should be familiar with Zabbix's powerful data visualization possibilities. On the frontend, you can create and visualize many kinds of graphs, maps, and charts that can help you to analyze and understand item history data, changes in the trigger status over time, IT services availability, and so on. Just as any other Zabbix capabilities, all of the visualization functions are also exposed through the API. You can certainly write programs to create, modify, or visualize screens, graphs, and maps, but unless you are building a custom frontend, it's quite unlikely that you'll ever need to do so.

On the other hand, it may be interesting to extract and visualize data that is otherwise too dispersed and hard to analyze through the frontend. A good example of such data is trigger dependency. You may recall from Chapter 6, Managing Alerts, that a trigger can depend on one or more other triggers such that it won't change to a PROBLEM state if the trigger it depends on is already in a PROBLEM state.

As useful as this feature is, there's no easy way to see at a glance the triggers that depend on other triggers, if those triggers, in turn, depend on other triggers, and so on. The good news is that with the help of the Graphviz package and a couple of lines of Python code, you can create a handy visualization of all trigger dependencies.

The Graphviz suite of programs

Graphviz (http://www.graphviz.org) is a suite of graph visualization software utilities that enables you to create arbitrary complex graphs from specially formatted text data. The suite provides you with many features for data visualization and can become quite complex to use, but it is quite simple to create a basic, functional setup that you can later build on.

If you do not have it installed on your system, Graphviz is just a yum install command away if you are on a Red Hat Enterprise Linux system:

# yum install 'graphviz*'

The program you will use to create your graphs is called dot. Dot takes a graph text description and generates the corresponding image in a number of formats. A graph description looks similar to this:

digraph G {
     main → node1 → node2;
     main → node3;
     main → end;
     node2 → node4;
     node2 → node5;
     node3 → node4;
     node4 → end;
}

Put the preceding graph in a graph.gv file and run the following command:

$ dot -Tpng graph.gv -o graph.png

You will obtain a PNG image file that will look somewhat similar to the following diagram:

As you can see, it should be fairly simple to create a trigger-dependency graph once we have extracted the right information through the API. Let's see how we can do it.

Creating a trigger dependency graph

The following is a Python script that will extract data about trigger dependencies and output a dot language graph description that you can later feed into the dot program itself:

#!/usr/bin/python 
from pyzabbix import ZabbixAPI 
zh = ZabbixAPI("https://127.0.0.1/zabbix") 
zh.login(user="Admin", password="zabbix") 

def gettriggername(triggerid=''): 
     return zh.trigger.get(triggerids=triggerid, output=['description'])[0]['description'] 

In the first part, there are no surprises. A Zabbix API session is initiated and a simple helper function, identical to the one shown before, is defined:

tr = zh.trigger.get(selectDependencies="refer", expandData="1", output="refer") 
dependencies = [(t['dependencies'], t['host'], t['triggerid']) for t in tr if t['dependencies'] != [] ] 

The next two lines extract all triggers and their dependencies and then create a list, filtering out triggers that don't have any dependencies:

outfile = open('trigdeps.gv', 'w') 
outfile.write('digraph TrigDeps {
') 
outfile.write('graph[rankdir=LR]
') 
outfile.write('node[fontsize=10]
') 

Here, the first few lines of the graph are written out to the output file, thus setting up the graph direction from left to right and the font size for the nodes' labels:

for (deps, triggerhost, triggerid) in dependencies: 
     triggername = gettriggername(triggerid) 
      
     for d in deps: 
          depname = gettriggername(d['triggerid']) 
          dephost = d['host'] 
          edge = '"{}:\n{}" -> "{}:\n{}";'.format(dephost, depname, triggerhost, triggername) 
          outfile.write(edge + '
') 

This is the core of the script. The double for loop is necessary because a single trigger may have more than one dependency and you want to map out all of them. For every dependency-trigger relationship found, an edge is defined in the graph file:

outfile.write('}
') 
outfile.close() 

Once the script reaches the end of the list, there is nothing more to do except close the graph description and close the output file.

Execute the script:

$ chmod +x triggerdeps.py
$ ./triggerdeps.py

You will get a trigdeps.gv file that will look somewhat similar to this:

digraph TrigDeps { 
graph[rankdir=LR] 
node[fontsize=10] 
"Template IPMI Intel SR1630:
Power" -> "Template IPMI Intel SR1630:
Baseboard Temp Critical [{ITEM.VALUE}]"; 
"Template IPMI Intel SR1630:
Baseboard Temp Critical [{ITEM.VALUE}]" -> "Template IPMI Intel SR1630:
Baseboard Temp Non-Critical [{ITEM.VALUE}]"; 
"Template IPMI Intel SR1630:
Power" -> "Template IPMI Intel SR1630:
Baseboard Temp Non-Critical [{ITEM.VALUE}]"; 
"Template IPMI Intel SR1630:
Power" -> "Template IPMI Intel SR1630:
BB +1.05V PCH Critical [{ITEM.VALUE}]"; 
"Template IPMI Intel SR1630:
BB +1.05V PCH Critical [{ITEM.VALUE}]" -> "Template IPMI Intel SR1630:
BB +1.05V PCH Non-Critical [{ITEM.VALUE}]"; 
"Template IPMI Intel SR1630:
Power" -> "Template IPMI Intel SR1630:
BB +1.05V PCH Non-Critical [{ITEM.VALUE}]"; 
"Template IPMI Intel SR1630:
Power" -> "Template IPMI Intel SR1630:
BB +1.1V P1 Vccp Critical [{ITEM.VALUE}]";
}

Just run it through the dot program in order to obtain your dependencies graphs:

$ dot -Tpng trigdeps.gv -o trigdeps.png 

The resulting diagram will probably be quite big; the following is a close up of a sample resulting image:

From improving the layout and the node shapes, to integrating the graph generating part into Python with its graphviz bindings, once again, there are many ways to improve the script. Moreover, you could feed the image back to a Zabbix map using the API, or you could invert the process and define trigger dependencies based on an external definition.

Now, it is interesting to see how starting with a Graphviz dot file, we can generate a Zabbix map in an automated way. Here, the automation is quite interesting as Zabbix is affected by certain boring issues, such as:

Those are already a good set of points to think about an automated way to speed up a long and slow process. Graphviz provides us with a good tool to be used in this case to generate an image and transform it into Zabbix's API calls. What we need to do is:

We can now, finally, start coding lines in Python; the following example is similar to the one presented by Volker Fröhlich. Anyway, the code here has been changed and fixed (it did not work well with Zabbix 2.4).

As the first thing, we need to import the ZabbixApi and networkx libraries:

import networkx as nx
from pyzabbix import ZabbixAPI

Then, we can define the Graphviz DOT file to use as a source; here, we can generate our DOT file by exporting data from Zabbix itself, which involves taking care to populate all the relations between the nodes. In this example, we are using a simple line of code:

dot_file="/tmp/example.dot"

In the next lines, we will define our username, password, map dimension, and the relative map name:

username="Admin"
password="zabbix"
width = 800
height = 600
mapname = "my_network"

What follows here is a static map to define the element type:

ELEMENT_TYPE_HOST = 0
ELEMENT_TYPE_MAP = 1
ELEMENT_TYPE_TRIGGER = 2
ELEMENT_TYPE_HOSTGROUP = 3
ELEMENT_TYPE_IMAGE = 4
ADVANCED_LABELS = 1
LABEL_TYPE_LABEL = 0

Then, we can define the icons to use and the relative color code:

icons = {
    "router": 23,
    "cloud": 26,
    "desktop": 27,
    "laptop": 28,
    "server": 29,
    "sat": 30,
    "tux": 31,
    "default": 40,
}
colors = {
    "purple": "FF00FF",
    "green": "00FF00",
    "default": "00FF00",
}

Now, we will define certain functions: the first one is to manage the login and the second one is to define a host lookup:

def api_connect():
    zapi = ZabbixAPI("http://127.0.0.1/zabbix/")
    zapi.login(username, password)
    return zapi

def host_lookup(hostname):
    hostid = zapi.host.get({"filter": {"host": hostname}})
    if hostid:
        return str(hostid[0]['hostid'])

The next thing to do is read our dot file and start converting it into a graph:

G=nx.read_dot(dot_file)

Then, we can finally open our graph:

pos = nx.graphviz_layout(G)

Then, the next thing to do, as the graph is already generated, is find the maximum coordinates of the layout. This will enable us to scale our predefined map output size better:

positionlist=list(pos.values())
maxpos=map(max, zip(*positionlist))
for host, coordinates in pos.iteritems():
   pos[host] = [int(coordinates[0]*width/maxpos[0]*0.95-coordinates[0]*0.1), int((height-coordinates[1]*height/maxpos[1])*0.95+coordinates[1]*0.1)]
nx.set_node_attributes(G,'coordinates',pos)

Then, we need to retrieve selementids as they are required for links and even for the node data coordinates:

selementids = dict(enumerate(G.nodes_iter(), start=1))
selementids = dict((v,k) for k,v in selementids.iteritems())
nx.set_node_attributes(G,'selementid',selementids)
nx.set_node_attributes(G,'selementid',selementids)

Now, we will define the map on Zabbix, the name, and the relative map size:

map_params = {
    "name": mapname,
    "label_type": 0,
    "width": width,
    "height": height
}
element_params=[]
link_params=[]

Finally, we can connect to our Zabbix server:

zapi = api_connect()

Then, prepare all the node information and the coordinates, and then set the icon to use:

for node, data in G.nodes_iter(data=True):
    # Generic part
    map_element = {}
    map_element.update({
            "selementid": data['selementid'],
            "x": data['coordinates'][0],
            "y": data['coordinates'][1],
            "use_iconmap": 0,
            })

Check, whether we have the hostname:

    if "hostname" in data:
        map_element.update({
                "elementtype": ELEMENT_TYPE_HOST,
                "elementid": host_lookup(data['hostname'].strip('"')),
                "iconid_off": icons['server'],
                })
    else:
        map_element.update({
            "elementtype": ELEMENT_TYPE_IMAGE,
            "elementid": 0,
        })

We will now set labels for the images:

    if "label" in data:
        map_element.update({
            "label": data['label'].strip('"')
        })
    if "zbximage" in data:
        map_element.update({
            "iconid_off": icons[data['zbximage'].strip('"')],
        })
    elif "hostname" not in data and "zbximage" not in data:
        map_element.update({
            "iconid_off": icons['default'],
        })

    element_params.append(map_element)

Now, we need to scan all the edges to create the element links based on the element. We've identified selementids:

nodenum = nx.get_node_attributes(G,'selementid')
for nodea, nodeb, data in G.edges_iter(data=True):
    link = {}
    link.update({
        "selementid1": nodenum[nodea],
        "selementid2": nerodenum[nodeb],
        })

    if "color" in data:
        color =  colors[data['color'].strip('"')]
        link.update({
            "color": color
        })
    else:
        link.update({
            "color": colors['default']
        })

    if "label" in data:
        label =  data['label'].strip('"')
        link.update({
            "label": label,
        })

    link_params.append(link)

# Join the prepared information
map_params["selements"] = element_params
map_params["links"] = link_params

Now, we have populated all map_params. We need to call Zabbix's API with that data:

map=zapi.map.create(map_params)

The program is now complete, and we can let it run! In a real-case scenario, the time spent to design a topology of more than 2,500 hosts is only 2-3 seconds!

We can test the software here that has been proposed against a dot file that contains 24 hosts:

[root@localhost]# time ./Generate_MyMap.py
real    0m0.005s
user    0m0.002s
sys     0m0.003s

As you can see, our software is really quick… but let's check what has been generated. In the next picture, you can see the map generated automatically in 0.005 seconds:

The goal of this example was to see how we can easily automate complex and long tasks using the Zabbix API. The same method proposed here is really useful when you have to create or do the initial startup. Also, nowadays, there are quite a few tools that can provide you with the data of the host already monitored, for example, Cisco Prime or other vendor-specific management tools from where you can extract a considerable amount of data, convert it into .dot, and populate the Zabbix hosts, maps, and so on.