Network interfaces on servers seem simple to monitor, but they tend to get more complicated as the environment size increases and time goes by. Back in the day, we had eth0, and everybody was happy. Well, not everybody—people needed more interfaces, so we had eth1, eth2, and so on. It would already be a challenge to manually match the existing interfaces to Zabbix items so that all interfaces are properly monitored. Then, Linux-based systems changed the interface naming scheme, and now, you could have enp0s25 or something similar, or a totally different interface name. That would not be easy to manage on a large number of different systems. Interface names on Windows are even more fun—they could include the name of the vendor, driver, antivirus software, firewall software, and a bunch of other things. In the past, people have even written VB scripts to sort of create fake eth0 interfaces on Windows systems.
Luckily, LLD should solve all that by providing a built-in way to automatically discover all the interfaces and monitor the desired items on each interface. This is supported on the majority of the platforms that the Zabbix agent runs on, including Linux, Windows, FreeBSD, OpenBSD, NetBSD, Solaris, AIX, and HP-UX.
Let's see how we can discover all the interfaces automatically on our monitored systems. Navigate to Configuration | Templates and click on Discovery next to C_Template_Linux. This is the section that lists the LLD rules—currently, we have none. Before we create a rule, it might be helpful to understand what an LLD rule is and what other entities supplement it.
A Discovery rule is a configuration entity that tells Zabbix what it should discover. In the case of network interfaces, an LLD rule would return a list of all interfaces. Assuming our system has interfaces called ETH0 and ETH1, the LLD rule would just return a list of them:
Then, the LLD rule contains prototypes. In the first place, prototypes for items would be required, although LLD allows us to add trigger and custom graph prototypes as well. What actually are prototypes? We discussed templates in Chapter 8, Simplifying Complex Configurations with Templates. You can think of LLD prototypes as mini-templates. Instead of affecting the whole host, they affect items or triggers, or custom graphs on a host. For example, an item prototype for network interface discovery could tell Zabbix to monitor incoming network traffic on all discovered interfaces the same way.
Getting back to creating an LLD rule, in the empty list of LLD rules, click on Create discovery rule in the upper-right corner and then fill in the following details:
- Name: Interface discovery
- Key: net.if.discovery
- Update interval: 120s
When done, click on Add. The discovery rule is added, although it won't do much useful work for now. The key we used, net.if.discovery, is supposed to return all the interfaces on the system. As you probably spotted, the properties of an LLD rule look quite similar to item properties—there's an update interval, and there are flexible intervals. Overall, the built-in agent LLD rules are actually items. Later, we will look at the details of how they operate.
A discovery rule returns macros. In the same way as before, it might be safer to think about them as variables, although we will again refer to them as macros here. These macros return various properties of the discovered entities. In the case of the network interface discovery by the Zabbix agent, these macros return interface names. LLD macros always use the syntax of {#NAME}, that is, the name wrapped in curly braces and prefixed with a hash mark. The macros can be later used in prototypes to create items for each discovered interface. The built-in LLD rule keys return a fixed set of such macros, and we will discuss each set whenever we look at the specific discovery method, such as network interfaces first, and the filesystem and others later. We have an LLD rule now, but it just reveals the interfaces. Nothing is done about them without the prototypes. To derive any benefit from the previous step, let's create some prototypes. Still in the LLD rule list, click on Item prototype in the ITEMS column next to Interface discovery. Then, click on the Create item prototype button, and fill in the following:
- Name: Incoming traffic on $1
- Key: net.if.in[{#IFNAME}]
- Units: Bps
- Preprocessing step (in the Preprocessing tab): Change per second
The fields can be seen in the following screenshot:
Our prototype here uses a discovery macro in the item key parameters. Actually, this is required. These macros will be replaced with different values when creating the final items, so the resulting item keys will be different. We could create item prototypes without using LLD macros in the key parameters, but the resulting discovery would fail as it would attempt to create one item per LLD macro.
When done with the configuration, click on the Add button at the bottom. Let's see whether this item prototype now works as intended. We set the interval in our LLD rule to a low value—120 seconds. As we cannot force items and discovery rules to run manually, this will allow us to play with various configuration changes and see the results much sooner. Wait for a few minutes and go to Configuration | Hosts. Then, click on Discovery next to A test host. Something's not right—in the Info column, there's a red error icon. Move your mouse cursor over it to see what the error message is:
It's complaining that an item that would have to be created based on the LLD item prototype already exists. That is correct; we created an item exactly like that earlier, when we manually added items for interface monitoring.
As is always the case, item uniqueness is determined by the item key, including all the parameters. Unfortunately, there is no way to merge manually configured items with LLD-generated ones. There is also no easy way to keep the collected history. We could change the item key either for the existing item or for the item prototype slightly and keep the manually added item for historic purposes and then remove it later when the new, LLD-generated item has collected sufficient historical data.
In this case, we could apply a small hack to the existing item key.
Navigate to Configuration | Templates, and click on Items next to C_Template_Linux. Click on Incoming traffic on interface enp0s3 in the Name column. In the properties, make the following changes:
- Name: Incoming traffic on interface $1 (manual)
- Key: net.if.in[enp0s3,]
That is, add (manual) to the name and a trailing comma inside the square brackets. The first change was not strictly required, but it will allow us to identify these items. The second change does not change anything functionally—the item will still collect exactly the same information. We changed the item key, though. Even a small change like this results in the key being different, and the discovery rule should be able to create those items now. When done, click on Update. Now, make the same changes to the outgoing network traffic item and the loopback interface item.
With the item keys changed, we could also monitor outgoing traffic automatically:
- Go to Configuration | Templates, click on Discovery next to C_Template_Linux, and then Item prototype next to Interface discovery.
- Click on Incoming traffic on {#IFNAME} and then on the Clone button.
- Change Incoming to Outgoing in the Name field, and change the Key field to read net.if.out[{#IFNAME}].
- When done, click on the Add button at the bottom.
- Allow a few minutes to pass and then head back to Configuration | Hosts.
- Click on Discovery next to A test host. The error icon should be gone. If not, track down any other items mentioned here and make the same changes to them.
- Once there are no errors listed in this section, navigate to Configuration | Hosts, click on Items next to A test host, and then click on the Discovered sub-filter under DISCOVERY. There should be several new items, and they should all be prefixed with the LLD rule name, Interface discovery, as shown in the following screenshot:
Clicking on the discovery rule name will open the list of prototypes in the LLD rule.
Our first discovery rule seems to be working nicely now; all interfaces on the system have been discovered and network traffic is being monitored on them. If we wanted to monitor other parameters on each interface, we would add more prototypes, using the discovery macro in the item key parameters so that the created items have unique keys.