Zabbix has a bunch of processes internally, and we've already covered a few; we enabled IPMI and VMware pollers, as well as SNMP trappers. For several of these, we're also able to configure how many processes to start. How can we know whether one process is enough or maybe we should have a hundred of them? We'll discuss general guidelines per type a bit later, but a very important thing to know is how busy the currently running processes are. There're internal items for this purpose as well. For these items, the general syntax is as follows:

zabbix[process,<type>,<mode>,<state>] 

The parameters in the preceding code are explained as follows:

• The first parameter, process, is a fixed keyword.
• The second parameter, type, is the process type, as in poller, trapper, and so on.
• The third parameter, mode, could be one of these:
  • avg: This is the average rate across all processes of the specified type.
  • count: This is the number of processes of the specified type.
  • max: This is the maximum rate across the processes of the specified type.
  • min: This is the minimum rate across the processes of the specified type.
  • A number: This is the rate for an individual process of the specified type. For example, there're five pollers running by default. With a process number specified here, we could monitor poller 1 or poller 3. Note that this is the internal process number, not the system PID.

We talked about rate here; this is the amount of time a target process or processes spent in a state, specified by the fourth parameter. It could either be busy or idle.

Should we monitor the busy rate or the idle one? In most cases, the average busy time for all processes of a specific type is monitored. Why busy? Just by convention, as when this monitoring got implemented, the first templates monitored busy rate. Additionally, when debugging a specific issue, it could be helpful to monitor the busy rate for individual processes. Unfortunately, there's no way to query such values directly from the server; we would have to add an item in the frontend and then wait for it to start working. There's no built-in LLD for process types or the number of them; we would have to create such items manually or automate them using XML importing or the Zabbix API.

To see how this works, let's monitor the average busy rate for all poller processes:

1. Go to Configuration | Hosts, click on Items next to A test host, and then on Create item. Fill in these values:

• • Name: Zabbix $4 process $2 rate
  • Type: Zabbix internal
  • Key: zabbix[process,poller,avg,busy]
  • Type of information: Numeric (float)
  • Units: %
  • New application: Zabbix process busy rates

We used positional variables in the item name again; if we wanted to monitor another process, it would be easy to clone this item and change the process name in the item key only.

2. When done, click on the Add button at the bottom. Check this item in the Latest data page:

Most likely, our small Zabbix instance isn't very busy polling values. By default, there're five pollers started, and they're dealing with the current load without any issues.

As an exercise, monitor a few more process types; maybe trapper and unreachable pollers. Check the Zabbix manual section on internal items for the exact process names to be used in this item.

After adding a few more items, you'll probably observe that there're a lot of internal processes. We discussed creating such items automatically using XML importing or the API, but then there were also all of the caches we could and should monitor. Zabbix tries to help here a bit and ships with default internal monitoring templates. In the search box in the upper-right corner, enter app zabbix and hit the Enter key. Look at the Templates block:

While the agent template is quite simple and not of much interest at this time, the server and proxy templates cover quite a lot, with 31 and 42 items respectively. These templates will allow out-of-the-box monitoring of internal process busy rates, cache usage, queues, values processed, and a few other things. It's highly recommended to use these templates in all Zabbix installations.

These templates might still be missing a few interesting items, such as the expected NVPS item we created earlier. It's suggested to create a separate template with such missing things instead of modifying the default template. Such an approach will allow easier upgrades, as new versions could add more processes, caches, and have other improvements to the default templates. If we leave the default templates intact, we can import a new XML file, tell Zabbix to add all missing things, update existing things, and remove whatever isn't in the XML, and we'll have an up-to-date default template. If we had it modified, it could be a lot of manual work to update it.