gmond installation is straightforward, and the libraries it depends upon are installed by default on most modern Linux distributions (as of this writing, those libraries are libconfuse, pkgconfig, PCRE, and APR). Ganglia packages are available for most Linux distributions, so if you are using the package manager shipped with your distribution (which is the suggested approach), resolving the dependencies should not be problematic.

The Ganglia components are available in a prepackaged binary format for most Linux distributions. We’ll cover the two most popular types here: .deb- and .rpm-based systems.

user@host:# sudo apt-get install ganglia-monitor

user@host:$ yum search ganglia-gmond

user@host:# sudo rpm -Uvh 
http://mirror.ancl.hawaii.edu/linux/epel/5/i386/epel-release-5-4.noarch.rpm

user@host:# sudo rpm -Uvh 
http://mirror.chpc.utah.edu/pub/epel/6/i386/epel-release-6-7.noarch.rpm

user@host:# sudo yum install ganglia-gmond

gmond compiles and runs fine on Mac OS X; however, at the time of this writing, there are no prepackaged binaries available. OS X users must therefore build Ganglia from source. Refer to the following instructions, which work for the latest Mac OS X Lion. For other versions of Mac OS X, the dependencies might vary. Please refer to Ganglia’s website for further information.

Several dependencies must be satisfied before building and installing Ganglia on OS X. These are, in the order they should be installed:

Xcode is a collection of development tools, and an Integrated Development Environment (IDE) for OS X. You will find Xcode at Apple’s developer tools website for download or on the MAC OS X installation disc.

MacPorts is a collection of build instructions for popular open source software for OS X. It is architecturally identical to the venerable FreeBSD Ports system. To install MacPorts, download the installation disk image from the MacPorts website. MacPorts for MAC OS X Lion is here. If you’re using Snow Leopard, the download is located here. For older versions, please refer here for documentation and download links.

Once MacPorts is installed and working properly, use it to install both libconfuse and pkconfig:

$ sudo port install libconfuse pkgconfig pcre apr

After satisfying the previously listed requirements, you are ready to proceed with the installation. Please download the latest Ganglia source release.

Change to the directory where the source file has been downloaded. Uncompress the tar-gzip file you have just downloaded:

$ tar -xvzf ganglia-major.minor.release.tar.gz

On Mac OS X 10.5+, you need to apply a patch so that gmond builds successfully. For further details on the patch, please visit the website. Download the patch file, copy it to the root of the build directory, and run the patch:

$ cd ganglia-major.minor.release 
$ patch -p0 < patch-file

Assuming that you installed MacPorts under the default installation directory (/opt/local), export MacPorts’ bin directory to your PATH and run the configure script, specifying the location of lib/ and include/ as options:

$ export PATH=$PATH:/opt/local/bin 
$ ./configure LDFLAGS="-L/opt/local/lib" CPPFLAGS="-I/opt/local/include"

Compile and install Ganglia:

$ make 
$ sudo make install

Convenient binary packages for Solaris are distributed in the OpenCSW collection. Follow the standard procedure to install the OpenCSW. Run the pkgutil tool on Solaris, and then use the tool to install the package:

$ pkgutil
$ CSWgangliaagent

The default location for the configuration files on Solaris (OpenCSW) is /etc/opt/csw/ganglia. You can now start and stop all the Ganglia processes using the normal SMF utility on Solaris, such as:

$ svcadm enable cswgmond

Because Ganglia is an open source project, it is possible to compile a runnable binary executable of the gmond agent on virtually any platform with a C compiler.

The Ganglia projects uses the autotools build system to detect the tools available on most Linux and UNIX-like environments and build the binaries.

The autotools build system is likely to have support for many other platforms that are not explicitly documented in this book. Please start by reading the INSTALL file in the source tree, and also look online for tips about Ganglia or generic tips about using autotools projects in your environment.

The requirements for installing gmetad on Linux are nearly the same as gmond, except for the addition of RRDtool, which is required to store and display time-series data collected from other gmetad or gmond sources.

Once again, you are encouraged to take advantage of the prepackaged binaries available in the repository of your Linux distribution; we provide instructions for the two most popular formats next.

user@host:# sudo apt-get install gmetad

user@host:# sudo yum install ganglia-gmetad

There are only two functional differences between building gmond and gmetad on OS X. First, gmetad has one additional software dependency (RRDtool), and second, you must include the --with-gmetad option to the configure script, because only gmond is built by the default Makefile.

Following is the list of requirements that must be satisfied before you can build gmetad on Mac OS X:

Refer to OS X for instructions on installing Xcode and MacPorts. Once you have those sorted out, install the following packages to satisfy the requirements:

$ sudo port install libconfuse pkgconfig pcre apr rrdtool

Once those packages have been installed, proceed with the Ganglia installation by downloading the latest Ganglia version.

Uncompress and extract the tarball you have just downloaded:

$ tar -xvzf ganglia-major.minor.release.tar.gz

Successfully building Ganglia 3.1.2 on OS X 10.5 requires that you apply the patch detailed here. Download the patch file and copy it to the root of the extracted Ganglia source tree, then apply it:

$ cd ganglia-major.minor.release
$ patch -p0 < patch-file

Assuming that you installed MacPorts under the default installation directory (/opt/local). Export MacPorts’ /bin directory to your PATH, and run the configure script, specifying the location of lib/ and include/ as options

$ export PATH=$PATH:/opt/local/bin 
$ ./configure --with-gmetad LDFLAGS="-L/opt/local/lib" CPPFLAGS="-I/opt/local/include"

Compile and install Ganglia:

$ make
$ sudo make install

Convenient binary packages for Solaris are distributed in the OpenCSW collection. Follow the standard procedure to install the OpenCSW. Run the pkgutil tool on Solaris, and then use the tool to install the package:

$ pkgutil
$ CSWgangliagmetad

The default location for the configuration files on Solaris (OpenCSW) is /etc/opt/csw/ganglia. You can now start and stop all the Ganglia processes using the normal SMF utility on Solaris, as in:

$ svcadm enable cswgmetad

As of Ganglia 3.4.0, the web interface is a separate distribution tarball maintained in a separate source code repository. The release cycle and version numbers of gweb are no longer in lockstep with the release cycle and version numbers of the Ganglia gmond and the gmetad daemon.

Ganglia developers support gweb 3.4.0 with all versions of gmond/gmetad version 3.1.x and higher. Future versions of gweb may require a later version of gmond/gmetad. It’s recommended to check the installation documentation for exact details whenever installing or upgrading gweb.

The frontend, as already mentioned, is a web application. This book covers gweb versions 3.4.x and later, which may not be available to all distributions, requiring more work to get it installed. Before proceeding, please review the requirements to install gweb:

If you are installing from the repositories, the installation is pretty straightforward. Requirements will be automatically satisfied, and within a few commands you should be able to play with the web interface.

root@host:# apt-get install apache2 php5 php5-json

root@host:# grep ^extension=json.so /etc/php5/conf.d/json.ini

root@host:# echo 'extension=json.so' >> /etc/php5/conf.d/json.ini

root@host:# tar -xvzf ganglia-web-major.minor.release.tar.gz
root@host:# cd ganglia-web-major.minor.release

# Location where gweb should be installed to
DESTDIR = /var/www/html/ganglia2
APACHE_USER = www-data
...

root@host:# make install

root@host:# yum install httpd php

extension=json.ini

root@host:# tar -xvzf ganglia-web-major.minor.release.tar.gz
root@host:# cd ganglia-web-major.minor.release

# Location where gweb should be installed to
DESTDIR = /var/www/html/ganglia2
APACHE_USER = apache
...

root@host:# make install

If you need to install gweb on Mac OS X, you have to follow a slightly different approach than if you were installing in Linux. Again, there isn’t any binary package for Mac OS X, leaving you with the option of downloading the source from the website. Before downloading, you have to make sure that your Mac OS X has shipped with a few of the requirements. That’s what this section is about.

First off, an HTTP server is required, and chances are good that your Mac OS X installation was shipped with Apache Web Server. You can also install it via MacPorts, but this approach is not covered here. It is your choice. In order to verify your Apache installation, go to System Preferences → Sharing. Turn Web Services on if it is off. Make sure it’s running by typing http://localhost on your browser. You should see a test page. You can also load Apache via Terminal by typing:

$ sudo launchctl load -w /System/Library/LaunchDaemons/org.apache.httpd.plist

PHP is also required to run gweb. PHP is shipped with Mac OS X, but it’s not enabled by default. To enable, edit the httpd.conf file and uncomment the line that loads the php5_module.

$ cd /etc/apache2
$ sudo vim httpd.conf

Search for the following line, uncomment (strip the #) it, and save the file:

# LoadModule php5_module libexec/apache2/libphp5.so

Restart Apache:

$ sudo launchctl unload -w /System/Library/LaunchDaemons/org.apache.httpd.plist
$ sudo launchctl load -w /System/Library/LaunchDaemons/org.apache.httpd.plist

Now that you have satisfied the requirements, it’s time to download and install gweb 2. Please download the latest release. Once you have finished, change to the directory where the file is located and extract its content. Next, cd to the extraction directory:

$ tar -xvzf ganglia-web-major.minor.release.tar.gz
$ cd ganglia-web-major.minor.release

This next step really depends on how Apache Web Server is set up on your system. You need to find out where Apache serves its pages from or, more specifically, its DocumentRoot. Of course, the following location isn’t the only possibility, but for clarity’s sake, we will work with the default settings. So here, we’re using /Library/WebServer/Documents:

$ grep -i documentroot /etc/apache2/httpd.conf

Edit the Makefile found in the tarball. Insert the location of your Apache’s DocumentRoot and the name of the user that Apache runs. On Mac OS X Lion, the settings are:

# Location where gweb should be installed
DESTDIR = /Library/WebServer/Documents/ganglia2
APACHE_USER = _www
...

This means that gweb will be available to the user here. You can change this to whichever name you want. Finally, run:

$ sudo make install

If no errors are shown, Ganglia Web is successfully installed. Read the next sections to configure Ganglia prior to running it for the first time.

Convenient binary packages for Solaris are distributed in the OpenCSW collection. Follow the standard procedure to install the OpenCSW. Run the pkgutil tool on Solaris, and then use the tool to install the package:

$ pkgutil
$ CSWgangliaweb

The default location for the configuration files on Solaris (OpenCSW) is /etc/opt/csw/ganglia. You can now start and stop all the Ganglia processes using the normal SMF utility on Solaris, as in:

$ svcadm enable cswapache

gmond’s default topology is a multicast mode, meaning that all nodes in the cluster both send and receive metrics, and every node maintains an in-memory database—stored as a hash table—containing the metrics of all nodes in the cluster. This topology is illustrated in Figure 2-1.

Figure 2-1. Default multicast topology

Of particular importance in this diagram is the disparate nature of the gmond daemon. Internally, gmond’s sending and receiving halves are not linked (a fact that is emphasized in Figure 2-1 by the dashed vertical line). gmond does not talk to itself—it only talks to the network. Any local data captured by the metric modules are transmitted directly to the network by the sender, and the receiver’s internal database contains only metric data gleaned from the network.

This topology is adequate for most environments, but in some cases it is desirable to specify a few specific listeners rather than allowing every node to receive (and thereby waste CPU cycles to process) metrics from every other node. More detail about this architecture is provided in Chapter 3.

The use of “deaf” nodes, as illustrated in Figure 2-2, eliminates the processing overhead associated with large clusters. The deaf and mute parameters exist to allow some gmond nodes to act as special-purpose aggregators and relays for other gmond nodes. Mute means that the node does not transmit; it will not even collect information about itself but will aggregate the metric data from other gmond daemons in the cluster. Deaf means that the node does not receive any metrics from the network; it will not listen to state information from multicast peers, but if it is not muted, it will continue sending out its own metrics for any other node that does listen.

Figure 2-2. Deaf/mute multicast topology

The use of multicast is not required in any topology. The deaf/mute topology can be implemented using UDP unicast, which may be desirable when multicast is not practical or preferred (see Figure 2-3).

Figure 2-3. UDP unicast topology

Further, it is possible to mix and match the deaf/mute, and default topologies to create a system architecture that better suits your environment. The only topological requirements are:

In practice, however, nodes not configured with any multicast connectivity do not need to be deaf; it can be useful to configure such nodes to send metrics to themselves using the address 127.0.0.1 so that they will keep a record of their own metrics locally. This makes it possible to make a TCP probe of any gmond for an XML about its own agent state while troubleshooting.

You can generate a default configuration file for gmond by running the following command:

user@host:$ gmond -t

The configuration file is composed of sections, enclosed in curly braces, that fall roughly into two logical categories. The sections in the first category deal with host and cluster configuration; those in the second category deal with the specifics of metrics collection and scheduling.

All section names and attributes are case insensitive. The following attributes, for example, are all equivalent:

name NAME Name NaMe

Some configuration sections are optional; others are required. Some may be defined in the configuration file multiple times; others must appear only once. Some sections may contain subsections.

The include directive can be used to break up the gmond.conf file into multiple files for environments with large complex configurations. The include directive supports the use of typeglobs. For example, the line:

include ('/etc/ganglia/conf.d/*.conf')

would instruct gmond to load all files in /etc/ganglia/conf.d/ that ended in “.conf”.

The configuration file is parsed using libconfuse, a third-party API for configuration files. The normal rules of libconfuse file format apply. In particular, boolean values can be set using yes, true, and on for a positive value and their opposites, no, false, and off for a negative value. Boolean values are not handled in a case-sensitive manner.

There are eight sections that deal with the configuration of the host itself.

globals {
  daemonize = yes
  setuid = yes
  user = nobody
  debug_level = 0
  max_udp_msg_len = 1472
  mute = no
  deaf = no
  allow_extra_data = yes
  host_dmax = 86400 /*secs. Expires (removes from web interface) hosts in 1 day */
  host_tmax = 20 /*secs */
  cleanup_threshold = 300 /*secs */
  gexec = no
  send_metadata_interval = 0 /*secs */
}

cluster {
  name = "unspecified"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

host {
  location = "unspecified"
}

udp_send_channel {
  #bind_hostname = yes 
  mcast_join = 239.2.11.71
  port = 8649
  ttl = 1
}

tcp_accept_channel {
   port = 8649
}

acl {
    default = "deny"
    access {
      ip = 192.168.0.0
      mask = 24
      action = "allow"
    }
    access {
      ip = ::ff:1.2.3.0
      mask = 120
      action = "deny"
    }
}

#sflow {
# udp_port = 6343
# accept_vm_metrics = yes
# accept_jvm_metrics = yes
# multiple_jvm_instances = no
# accept_http_metrics = yes
# multiple_http_instances = no
# accept_memcache_metrics = yes
# multiple_memcache_instances = no
#}

modules {
     module {
       name = "example_module"
       language = "C/C++"
       enabled = yes
       path = "modexample.so"
       params = "An extra raw parameter"
       param RandomMax {
         value = 75
       }
       param ConstantValue {
         value = 25
       }
    }
}

collection_group {
  collect_once = yes
  time_threshold = 1200
  metric {
    name = "cpu_num"
    title = "CPU Count"
  }
}
collection_group {
  collect_every = 20
  time_threshold = 90
  /* CPU status */
  metric {
    name = "cpu_user"
    value_threshold = "1.0"
    title = "CPU User"
  }
  metric {
    name = "cpu_system"
    value_threshold = "1.0"
    title = "CPU System"
  }
}

The simplest topology is a single gmetad process polling one or more gmond instances, as illustrated in Figure 2-4.

Figure 2-4. Basic gmetad topology

Redundancy/high availability is a common requirement and is easily implemented. Figure 2-5 shows an example in which two (redundant) gmetads poll multiple gmonds in the same cluster. The gmetads will poll node2 only if they are unable to poll node1 successfully. Both gmetads are always polling (active-active clustering).

Figure 2-5. gmetad topology for high availability (active-active)

gmetad is not limited to polling gmond: a gmetad can poll another gmetad to create a hierarchy of gmetads. This concept is illustrated in Figure 2-6.

Figure 2-6. gmetad hierarchical topology

gmetad, by default, writes all metric data directly to RRD files on the filesystem, as illustrated in Figure 2-4.

In large installations in which there is an IO constraint, rrdcached acts as a buffer between gmetad and the RRD files, as illustrated in Figure 2-7.

Figure 2-7. gmetad with rrdcached

The gmetad.conf configuration file is composed of single-line attributes and their corresponding values. Attribute names are case insensitive, but their values are not. The following attributes, for example, are all equivalent:

name NAME Name NaMe

Most attributes are optional; others are required. Some may be defined in the configuration file multiple times; others must appear only once.

data_source "my cluster" 10 localhost my.machine.edu:8649 1.2.3.5:8655
data_source "my grid" 50 1.3.4.7:8655 grid.org:8651 grid-backup.org:8651
data_source "another source" 1.3.4.8:8655  1.3.4.8

/

/cluster1

/cluster1/host1

/cluster1?filter=summary

Although gweb itself requires no configuration to speak of, some web server configuration is necessary to get gweb up and running. Any web server with PHP support will do the job, and although web server configuration is beyond of the scope of this book, the Apache Web Server is such a common choice that it has been included an example of a virtual host configuration for Apache. Assuming that gweb is installed in /var/www/html/ganglia2 on a host that resolves to myganglia.example.org, the following configuration should get you started with Apache:

NameVirtualHost *.80

<VirtualHost *:80>
   ServerName myganglia.example.org
   ServerAlias myganglia
   
   DocumentRoot /var/www/html/ganglia2

   # Other directives here
</VirtualHost>

This is, of course, a simplistic example. For further reading on the subject, we recommend further reading here.

gweb is configured by way of the conf.php file. In fact, this file overrides and extends the default configuration set in conf_default.php. conf.php is located in the web root directory. This file is well documented, and as of this writing there are more than 80 options, so it won’t cover them all, but it will cover some of the more important ones, and make note of some option categories—just so you’re aware they’re there.

The file, as its name suggests, is itself a PHP script made up of variable assignments. Unlike the other configuration files, assignments might span multiple lines. Attribute names are themselves keys in gweb’s $conf data structure, so they are case sensitive, and look like PHP array assignments. The following line, for example, informs gweb of the location of the RRDtool binary:

$conf['rrdtool'] = "/usr/bin/rrdtool";

All attributes in the file are required, and some may be defined multiple times; others must appear only once. Some values are derived from other values. For example, the rrds attribute is derived from gmetad_root:

$conf['rrds'] = "${conf['gmetad_root']}/rrds";