Zabbix has another small trick to help with debugging. Run top and see which mode gives you a more stable and longer list of Zabbix processes—one of sorting by processor usage (hitting Shift + P) or memory usage (hitting Shift + M) might.
Press C and notice how the Zabbix processes have updated their command line to show which exact internal process it is and what is it doing as we can see here:
zabbix_server: poller #1 [got 0 values in 0.000005 sec, idle 1 sec]
zabbix_server: poller #4 [got 1 values in 0.000089 sec, idle 1 sec]
zabbix_server: poller #5 [got 0 values in 0.000004 sec, idle 1 sec]
Follow their status and see how the task and the time it takes change for some of the processes. We could also have output that could be redirected or filtered through other commands:
# top -c -b | grep zabbix_server
The -c option tells it to show the command line, the same thing we achieved by hitting C before. The -b option tells top to run in batch mode without accepting input and just outputting the results. We could also specify -n 1 to run it only once or specify any other number as needed.
It might be more convenient to use ps:
# ps -f -C zabbix_server
The -f flag enables full output, which includes the command line. The -C flag filters by the executable name:
zabbix 21969 21962 0 18:43 ? 00:00:00 zabbix_server: poller #1 [got 0 values in 0.000006 sec, idle 1 sec]
zabbix 21970 21962 0 18:43 ? 00:00:00 zabbix_server: poller #2 [got 0 values in 0.000008 sec, idle 1 sec]
zabbix 21971 21962 0 18:43 ? 00:00:00 zabbix_server: poller #3 [got 0 values in 0.000004 sec, idle 1 sec]
The full format prints out some extra columns—if all we needed was the PID and the command line, we could limit columns in the output with the -o flag, like this:
# ps -o pid=,command= -C zabbix_server
21975 zabbix_server: trapper #1 [processed data in 0.000150 sec, waiting for connection]
21976 zabbix_server: trapper #2 [processed data in 0.001312 sec, waiting for connection]
And to see a dynamic list that shows the current status, we can use the watch command:
# watch -n 1 'ps -o pid=,command= -C zabbix_server'
This list will be updated every second. Note that the interval parameter, -n, also accepts decimals, so to update twice every second, we could use -n 0.5.
This is also the method to find out which PID corresponds to which process type if startup messages are not available in the log file—we can see the process type and PID in the output of top or ps.