Prior to the introduction of Mod_MultiCPU, gmond was able to produce only a single CPU-related value for each of the various CPU metrics that it reported. If the hardware architecture supported multiple CPUs, gmond reported only the overall usage rather than the usage for each individual CPU. The Mod_MultiCPU module is capable of detecting how many CPUs exist on the system and constructs the series of metric definitions for each one (Table A-1). Through the configuration of Mod_MultiCPU, all CPU-related metrics can be reported for each CPU detected on the system.

The Mod_GStatus module (Table A-2) started out as a metric module debugging tool when the modular interface was first introduced into gmond. The purpose of this module was to detect and report all of the metric gathering, data packet sends, and receives as gmond was running. In other words, as gmond is capable of monitoring every aspect of the system, why shouldn’t gmond also monitor itself? Some of the metrics that Mod_GStatus reports are the number of metadata packets sent and received and the number of value packets sent and received, as well as the overall totals. It also tracks any failures in the system. If gmond is incapable of sending or receiving a packet of any kind, Mod_GStatus will report these failures as well.

The Multidisk module (Table A-3) was introduced as one of the new metric gathering modules for many of the same reasons as Mod_MultiCPU. In previous versions of gmond, the metrics that reported disk space added up the totals for all disks and reported this value for total disk space and used space. The Multidisk module provided a way to report disk usage metrics for each individual disk rather than a total of all disks on the system.

The memcached module (Table A-4) introduced a way to take a closer look at what is actually happening under the hood of the memory management system. The standard memory metrics only report overall memory usage and totals for the system and do not provide any further details about memory management. This module dives a little deeper into how the memory is being used and can help to point out memory inefficiencies.

The TcpConn metric module (Table A-5) provides a way to look at TCP network connections in an effort to detect problems or misconfiguration. By monitoring the TCP connection activity on the system, this module can help point out issues that may affect network latency or the inability to send or receive data in an efficient manner. This module also introduced a new pattern for how to write Python metric modules that include threading and caching. Because the TcpConn module relies heavily on the netstat Linux utility to acquire TCP metric data and the fact that gmond is not a multithreaded daemon, the module doesn’t want to cause any delays in the gmond gathering process due to latency in calling an external process utility. In order to avoid any kind of latency issues, the TcpConn module starts up its own gathering thread, which is then free to invoke the netstat utility as required. By invoking the netstat utility within a thread, the module is able to gather the TCP connection related values without having to worry about delaying the gmond gathering process. As the metrics are being gathered from within the thread, these values are stored in a shared cache that can be accessed quickly whenever gmond asks the module for its metric values. Introducing threads through Python is actually a very convenient way to make a single-threaded gmond daemon act as if it were multithreaded.

Most statsd instances are fairly similar to configure for submitting metrics to Ganglia. The important considerations should be how that data is represented in your Ganglia instance. For example, statsd and its clones don’t have any particular notion of “host,” so each statsd instance is tied to submitting metrics that will be associated with a specific Ganglia host.

-G (ganglia host) -g (ganglia port) -S (spoof string)

--transport="ganglia" --ganglia_host="localhost" --ganglia_port=8649
--ganglia_spoof_host="statd:statd"

VDED has some of the same constraints as statsd, except that it is not constrained to submit all values as if they belonged to a single host. The optional “spoof” parameter for submitting metrics to VDED allows different spoof arguments to be associated with different tracked metrics. It is a good idea to remember that VDED aggregation is limited to the cluster of which the receiving gmond instance is a member.

The command-line arguments for VDED, which are managed in /etc/vded/config on RHEL installations, is configured using the following switches:

--ghost=(ganglia host) --gport=(ganglia port) --gspoof=(default spoof)

The rrdcached package can be installed on Debian-based distributions (Debian, Ubuntu, Linux, Mint, etc.) by using apt:

For Red Hat/RHEL-based distributions (Red Hat/RHEL, Fedora, CentOS, etc.), rrdcached can be installed via the rrdtool package, which was probably installed already for gmetad to function properly:

gmetad can be configured to use rrdcached by setting the RRDCACHED_ADDRESS variable in the configuration file included by gmetad’s init script. For Red Hat distributions, this is /etc/sysconfig/gmetad, and for Debian distributions, it is /etc/default/gmetad. For local sockets, the format unix:/PATH/TO/SOCKET should be used to specify the address parameter.

Along with the gmetad configuration change (which will require a restart of any running gmetad processes), it is also recommended that a change be made to the Ganglia web frontend, which will force the frontend to also use rrdcached for forming graphs. Enabling rrdcached support in the web frontend is done by setting the configuration variable $conf['rrdcached_socket'] to the value of gmetad’s RRDCACHED_ADDRESS.

There are a number of useful operations that can be performed by using telnet, netcat, or socat (depending on whether you have a network or Unix socket set up as the control socket). For example, a FLUSHALL command forces the rrdcached daemon to flush all RRD data to disk as soon as it can:

There are several things that can go awry with an rrdcached Ganglia installation, primarily because an extra layer of complexity has been added.