The discovery does not just look a bit like an item in the frontend; it also operates in the same way underneath. The magic happens based on the content of a specific item value. All the things that are discovered are encoded in a JSON structure. The easiest way to see what's returned is to use zabbix_get and query a Zabbix agent. On A test host, run the following command:

 $ zabbix_get -s 127.0.0.1 -k net.if.discovery

Here, net.if.discovery is just an item key, not different from other item keys. This will return a small string, similar to the following:

{data:[{{#IFNAME}:enp0s3},{{#IFNAME}:enp0s8},{{#IFNAME}:lo}]} 

While it's mostly understandable, it would be even better with some formatting. The easiest way is to use Perl or Python tools. The Python method would be as follows:

$ zabbix_get -s 127.0.0.1 -k net.if.discovery | python -mjson.tool 

The Perl method would be one of these:

$ zabbix_get -s 127.0.0.1 -k net.if.discovery | json_pp
$ zabbix_get -s 127.0.0.1 -k net.if.discovery | json_xs 

The latter method should be faster, but requires the JSON::XS Perl module. For our purposes, performance should not be a concern, so choose whichever method works for you. The output will be similar to this:

{ 
    data : [ 
       { 
          {#IFNAME} : enp0s3 
       }, 
       { 
          {#IFNAME} : enp0s8 
       }, 
       { 
          {#IFNAME} : lo 
       } 
    ] 
} 

The number of interfaces and their names might differ, but we can see that for each discovered interface, we are returning one macro—the interface name. The key for filesystem discovery is similar—vfs.fs.discovery. We can now run this:

$ zabbix_get -s 127.0.0.1 -k vfs.fs.discovery | json_pp  

This would most likely return lots and lots of entries. Here's a snippet:

{ 
    data : [ 
        { 
            {#FSNAME} : /dev/pts, 
            {#FSTYPE} : devpts 
        }, 
        { 
            {#FSNAME} : /, 
            {#FSTYPE} : xfs 
        }, 
        { 
            {#FSNAME} : /proc, 
            {#FSTYPE} : proc 
        }, 
        { 
            {#FSNAME} : /sys, 
            {#FSTYPE} : sysfs 
... 

Two things can be seen here:

• It definitely returns way more than we would want to monitor
• It returns two values for each filesystem—name and type

While we could filter according to filesystem name, some monitored systems could have the root filesystem only, some could have separate /home, and so on. The best way would be to filter by filesystem type. In this example, we only want to monitor filesystems of type xfs.

With this knowledge in hand, let's navigate to Configuration | Templates, click on Discovery next to C_Template_Linux, and then click on Create discovery rule. Fill in the following values:

• Name: Filesystem discovery
• Key: vfs.fs.discovery
• Update interval: 2m

The same as with network interface discovery, we set the update interval to 2m or 120s. The default in the form, 30 seconds, is very low and should not be used. Discovery can be resource intensive, and, if possible, should be run on an hourly basis or so. Now, switch to the Filters tab, and fill in these values:

• Macro: {#FSTYPE}
• Regular expression: matches ^xfs$

When done, click on the Add button at the bottom. We have the discovery now, but no prototypes. Click on Item prototypes next to Filesystem discovery, and then click on Create item prototype. Fill in the following values:

• Name: Free space on {#FSNAME}
• Key: vfs.fs.size[{#FSNAME},free]

When done, click on the Add button at the bottom. We now expect the discovery to get the list of all filesystems and discard most of those, except the ones precisely with the type xfs, and then create a free disk space item for each of them. We filter by one LLD macro, {#FSTYPE}, but use another {#FSNAME} in the actual item configuration.

After a couple of minutes have passed, navigate to Configuration | Hosts and click on Items next to A test host. You can select Discovered items from the Discovery selection box in the filter if you like. For each filesystem of type xfs, there should be a free disk space item, as shown in the following screenshot:

With more prototypes, we could also monitor total space, inode statistics, and other data. We could have triggers as required on all of these filesystems.

As this discovery returns multiple macros, it might be desirable to filter by multiple macros at the same time. For example, we might want to exclude the /boot filesystem from monitoring. Similar to the type of calculation in action conditions, as discussed in Chapter 7, Acting upon Monitored Conditions, we can choose between the automatic options of And, Or, and And/Or and there's also the Custom expression option. This should allow us to create discovery logic of varying complexity.