In a classic Ganglia deployment, gmond agents are installed on each host in a cluster; see gmond: Big Bang in a Few Bytes. Each gmond agent performs three tasks:

In an sFlow deployment, sFlow agents replace gmond agents on all the hosts within the cluster; see Figure 8-1. The sFlow agents in each cluster send metrics as unicast messages to a single gmond instance that tracks cluster state and responds to requests from gmetad. As you will see in Configuring gmond to Receive sFlow, deploying gmond as an sFlow collector requires minimal configuration and eliminates dependencies because functionality associated with generating and transmitting metrics is disabled.

Figure 8-1. sFlow, gmond, and gmetad

The sFlow agents export standard groups of metrics; see the section Standard sFlow Metrics. However, supplementing sFlow’s standard metrics with additional custom metrics can be accomplished using gmetric; see Custom Metrics Using gmetric.

A single gmond instance monitoring each cluster does represent a single point of failure. If this is a concern, there are a number of strategies for making the deployment fault tolerant. A second gmond instance can be added to each cluster and the sFlow agents can be configured to send metrics to both the primary and secondary gmond instances. Alternatively, a virtual IP address can be assigned to gmond and used as the destination for sFlow messages and gmetad requests. The virtual IP address can be handed to a secondary system in the event that the primary system fails. Running gmond instances on virtual machines makes it easy to quickly bring up replacements in the event of a failure.

An entire cluster should be homogeneously monitored using sFlow agents or gmond agents—mixing sFlow and gmond agents within a single cluster is not recommended. However, you can adopt different measurement technologies within a grid, selecting the best strategy for monitoring each cluster. Using sFlow as the agent technology works best for commodity web, memcache, virtual server, and Java clusters where sFlow’s standard metrics provide good coverage. For specialized environments, gmond’s extensibility and extensive library of modules are likely to be a better option.

Current approaches to server performance monitoring are highly fragmented. Each operating system, server vendor, and application developer creates specific agents and software for performance monitoring, none of which interoperate. Standardizing the metrics simplifies monitoring by decoupling agents from performance monitoring applications, allowing measurements to be made once and shared among different monitoring applications.

The bulk of a typical gmond configuration file, gmond.conf, is devoted to the metrics that gmond exports for the local host. When gmond is configured as a pure sFlow collector, most configuration settings can be eliminated, resulting in a simple configuration file:

/* Configuration settings for a pure sFlow receiver */
/* Delete all udp_send_channel, modules, collection_group and include sections */

globals {
  daemonize = yes
  setuid = yes
  user = nobody
  debug_level = 0
  max_udp_msg_len = 1472
  mute = yes       /* don't send metrics */ 
  deaf = no        /* listen for metrics */
  allow_extra_data = yes
  host_dmax = 0
  host_tmax = 20
  cleanup_threshold = 300
  gexec = no
  send_metadata_interval = 0
}

cluster {
  name = "unspecified"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

host {
  location = "unspecified"
}

/* channel to receive gmetric messages */
/* eliminate mcast_join - sFlow is a unicast protocol */
udp_recv_channel {
  port = 8649
}

/* channel to service requests for XML data from gmetad */
tcp_accept_channel {
  port = 8649
}

/* channel to receive sFlow */
/* 6343 is the default sFlow port, an explicit sFlow    */
/* configuration section is needed to override default  */
udp_recv_channel {
  port = 6343
}

/* Optional sFlow settings */
#sflow {
# udp_port = 6343
# accept_vm_metrics = yes
# accept_jvm_metrics = yes
# multiple_jvm_instances = no
# accept_http_metrics = yes
# multiple_http_instances = no
# accept_memcache_metrics = yes
# multiple_memcache_instances = no
#}

/* end of configuration file */
/* Delete all modules, collection_group and include sections */

In the Ganglia architecture, each cluster is monitored by a separate gmond process. If more than one cluster is to be monitored, then it is possible to run multiple gmond processes on a single server, each with its own configuration file. For example, if sFlow agents on the first cluster are sending to port 6343, then sFlow agents on the second cluster should be configured to send to a different port, say 6344. The following settings, in a separate configuration file, configure the second gmond instance to listen on the nonstandard port:

...

/* channel to receive gmetric messages */
udp_recv_channel {
  port = 8650
}

/* channel to service requests for XML data from gmetad */
tcp_accept_channel {
  port = 8650
}

/* channel to receive sFlow */
udp_recv_channel {
  port = 6344
}

/* Change sFlow channel to non-standard port 6344 */
sflow {
  udp_port = 6344
}

Another alternative is to assign multiple IP addresses to a single server, one per cluster that is to be monitored. In this case, multiple gmond instances are created, each associated with a distinct IP address, and the sFlow agents in each cluster are configured to send metrics to the associated IP address. For example, the following settings configures the gmond instance to listen for sFlow on a specific IP address:

/* channel to receive sFlow */
udp_recv_channel {
  port = 6343
  bind = <IP address>
}

For more information on configuring Ganglia clusters, see Chapter 2.

The Host sFlow agent is an open source implementation of the sFlow standard for server monitoring. The Host sFlow agent provides “scalable, multi-vendor, multi-OS performance monitoring with minimal impact on the systems being monitored.

The following example shows how to install and configure the Host sFlow daemon (hsflowd) on a Linux server in order to illustrate how to send sFlow metrics to Ganglia gmond. The Host sFlow website should be consulted for detailed instructions on installing and configuring Host sFlow on other platforms.

First, install the hsflowd software:

rpm -Uvh hsflowd_XXX.rpm

Alternatively, hsflowd can be installed from sources:

tar -xzf hsflowd-X.XX.tar.gz
cd hsflowd-X.XX
make
make install
make schedule

Next, edit the hsflowd configuration file, /etc/hsflowd.conf:

sflow{
  DNSSD = off
  polling = 20
  sampling = 512
  sampling.http = 100
  sampling.memcache = 400
  collector{
   ip = <gmond IP address>
   udpport = <gmond udp_rcv_channel port>
  }
}

Finally, start the hsflowd daemon:

service hsflowd start

Within a few minutes, metrics for the server should start appearing in Ganglia. If metrics fail to appear, follow the directions in Troubleshooting.

Host sFlow Subagents

The Host sFlow website maintains a list of related projects implementing subagents that extend sFlow monitoring to HTTP, memcache, and Java applications running on the server. These subagents require minimal additional configuration because they share configuration settings with hsflowd (Figure 8-2).

Figure 8-2. Host sFlow subagents

The hsflowd daemon writes configuration information that it receives via DNS-SD or through its configuration file to the /etc/hsflowd.auto file. Other sFlow subagents running on the host automatically detect changes to the /etc/hsflowd.auto file and apply the configuration settings.

The sFlow protocol allows each subagent to operate autonomously and send an independent stream of metrics to the sFlow collector. Distributing monitoring among the agents eliminates dependencies and synchronization challenges that would increase the complexity of the agents.

Note

Extending the functionality of Host sFlow using subagents differs from gmond’s use of modules (see the section Extending gmond with Modules), but is very similar in approach to adding custom metrics using gmetric (see the section Extending gmond with gmetric). In fact, gmetric can easily be used to add custom metrics to the standard metrics exported using sFlow (see Custom Metrics Using gmetric).

Each of the sFlow subagents is responsible for exporting a different set of metrics. At the time of writing, the following subagents are available:

hsflowd

The Host sFlow daemon exports standard host metrics (see Server Metrics), and can also export per-virtual-machine statistics (see Hypervisor Metrics) when run on a Hyper-V, Xen, XenServer, or KVM hypervisor. In addition, network traffic monitoring using iptables/ULOG is supported on Linux platforms.

Open vSwitch

The Open vSwitch is the default switch in XenServer 6.0, the Xen Cloud Platform, and also supports Xen, KVM, Proxmox VE, and VirtualBox. It has also been integrated into many virtual management systems including OpenStack, openQRM, and OpenNebula. Enabling the built-in sFlow monitoring on the virtual switch offers the same visibility as sFlow on physical switches, providing a unified, end-to-end view of network performance across the physical and virtual infrastructure. The sflowovsd daemon ships with Host sFlow and automatically configures sFlow monitoring on the Open vSwitch using the ovs-vsctl command. Similar integrated sFlow support is available for the Microsoft extensible virtual switch that is part of the upcoming Windows Server 2012 version of Hyper-V.

jmx-sflow-agent

The Java sFlow agent is invoked on the Java command line, easily adding monitoring to existing Java applications and exporting standard metrics (see Java Virtual Machine Metrics).

tomcat-sflow-valve

This is a Tomcat Valve that can be loaded in an Apache Tomcat web server to export Java metrics (see Java Virtual Machine Metrics) and HTTP metrics (see HTTP Metrics).

mod-sflow

This is a module for the Apache web server that exports HTTP metrics (see HTTP Metrics).

nginx-sflow-module

This is a module for the NGINX web server that exports HTTP metrics (see HTTP Metrics).

sflow/memcached

This project tracks the latest memcache release and includes an embedded sFlow subagent that exports memcache metrics (see memcache Metrics).

gmetric -S 10.0.0.1:server1 -n Current_Users -v `who |wc -l` -t int32 -u current_users

#!/bin/bash
# Read configuration settings from hsflowd.auto
while IFS="=" read name value
do
  case "$name" in
    agentIP)
       SPOOF_IP=$value
       ;;
    hostname)
       SPOOF_HOSTNAME=$value
       ;;
    collector)
       set $value
       HOST=$1
       PORT=$(($2-6343+8649))
       ;;
    *)
       ;;
  esac
done < /etc/hsflowd.auto

# Export one or more custom metrics using gmetric.py
/usr/local/bin/gmetric.py
 --host $HOST
 --port $PORT
 --spoof $SPOOF_IP:$SPOOF_HOSTNAME
 --name Current_Users
 --value `/usr/bin/who |/usr/bin/wc -l`
 --type int32
 --units current_users

Most problems with sFlow deployments occur because the sFlow datagrams are dropped somewhere between the sFlow agent and gmond. The following steps will help identify where measurements are being dropped.

tcpdump -p udp port 6343

tcpdump -p src host 10.0.0.237 and udp port 6343

telnet localhost 8649

tcpdump -p -s 0 -w - udp port 6343 | sflowtool

startDatagram =================================
datagramSourceIP 10.1.4.2
datagramSize 432
unixSecondsUTC 1339651142
datagramVersion 5
agentSubId 100000
agent 10.1.4.2
packetSequenceNo 464
sysUpTime 9244000
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 464
sourceId 2:1
counterBlock_tag 0:2001
adaptor_0_ifIndex 1
adaptor_0_MACs 1
adaptor_0_MAC_0 000000000000
adaptor_1_ifIndex 2
adaptor_1_MACs 1
adaptor_1_MAC_0 eedb257595e5
counterBlock_tag 0:2005
disk_total 64427231232
disk_free 51740361728
disk_partition_max_used 35.43
disk_reads 20349
disk_bytes_read 403682304
disk_read_time 181676
disk_writes 16994
disk_bytes_written 144289792
disk_write_time 1130328
counterBlock_tag 0:2004
mem_total 1073741824
mem_free 723337216
mem_shared 0
mem_buffers 17854464
mem_cached 192057344
swap_total 2181029888
swap_free 2181029888
page_in 201324
page_out 70454
swap_in 0
swap_out 0
counterBlock_tag 0:2003
cpu_load_one 0.000
cpu_load_five 0.000
cpu_load_fifteen 0.000
cpu_proc_run 1
cpu_proc_total 96
cpu_num 1
cpu_speed 3200
cpu_uptime 9298
cpu_user 5590
cpu_nice 5570
cpu_system 16120
cpu_idle 9204610
cpu_wio 66020
cpuintr 0
cpu_sintr 0
cpuinterrupts 231288
cpu_contexts 370635
counterBlock_tag 0:2006
nio_bytes_in 2514529
nio_pkts_in 7316
nio_errs_in 0
nio_drops_in 0
nio_bytes_out 1799289
nio_pkts_out 8199
nio_errs_out 0
nio_drops_out 0
counterBlock_tag 0:2000
hostname virtual-vm
UUID 7c270fa3830347a9b6aef60bac8cd16f
machine_type 3
os_name 2
os_release 2.6.18-308.1.1.el5xen
endSample   ----------------------
endDatagram   =================================

Ganglia is reporting an increase in HTTP traffic to your web servers—how do you know whether you are the target of a denial of service attack, or a marketing promotion has just gone viral? You are seeing an increase in cache misses to your memcached cluster—how can you fix the problem? If you are using sFlow agents to generate the metrics, you have the data to answer these types of question at your fingertips.

The metrics that Ganglia displays are only part of the information contained in an sFlow stream. For example, the mod_sflow agent running in Apache servers also randomly samples HTTP operations, sending records that include attributes such as URL, user-agent, response time, client socket, and bytes transferred; see Table 8-5. Accessing the sampled HTTP operations allows you to dig deeper into a trend and identify the source of the increased traffic.

The following sflowtool output shows the HTTP counters and operation samples reported by mod_sflow:

startDatagram =================================
datagramSourceIP 10.0.0.150
datagramSize 132
unixSecondsUTC 1339652714
datagramVersion 5
agentSubId 0
agent 10.0.0.150
packetSequenceNo 30526
sysUpTime 3981481944
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 19510
sourceId 3:80
counterBlock_tag 0:2002
parent_dsClass 2
parent_dsIndex 1
counterBlock_tag 0:2201
http_method_option_count 0
http_method_get_count 55755
http_method_head_count 4
http_method_post_count 1359
http_method_put_count 0
http_method_delete_count 0
http_method_trace_count 0
http_methd_connect_count 7
http_method_other_count 0
http_status_1XX_count 0
http_status_2XX_count 54577
http_status_3XX_count 2211
http_status_4XX_count 314
http_status_5XX_count 23
http_status_other_count 0
endSample   ----------------------
endDatagram   =================================
startDatagram =================================
datagramSourceIP 10.0.0.150
datagramSize 264
unixSecondsUTC 1339652714
datagramVersion 5
agentSubId 0
agent 10.0.0.150
packetSequenceNo 30527
sysUpTime 3981483944
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 28550
sourceId 3:80
meanSkipCount 2
samplePool 57126
dropEvents 2
inputPort 0
outputPort 1073741823
flowBlock_tag 0:2100
extendedType socket4
socket4_ip_protocol 6
socket4_local_ip 10.0.0.150
socket4_remote_ip 10.0.0.70
socket4_local_port 80
socket4_remote_port 63729
flowBlock_tag 0:2206
flowSampleType http
http_method 2
http_protocol 1001
http_uri /index.php
http_host 10.0.0.150
http_useragent curl/7.21.4
http_request_bytes 0
http_bytes 0
http_duration_uS 1329
http_status 403
endSample   ----------------------
endDatagram   =================================

The following example demonstrates how the sflowtool output can be used to generate additional metrics. The Perl script uses sflowtool to decode the HTTP request data and calculates average response time over a minute, printing out the results:

#!/usr/bin/perl -w
use strict;
use POSIX;

my $total_time = 0;
my $total_requests = 0;
my $now = time();
my $start = $now - ($now % 60);

open(PS, "/usr/local/bin/sflowtool|") || die "Failed: $!
";
while( <PS> ) { 
 
  my ($attr,$value) = split;
 
  # process attribute
  if('startDatagram' eq $attr) {
    $now = time();
    if($now - $start >= 60) {
      if($total_requests > 0) {
        printf "%d %d
", $start, int($total_time/$total_requests);
      }
      $total_time = 0;
      $total_requests = 0;
      $start = $now - ($now % 60);
    }
  }
  if('http_duration_uS' eq $attr) {
    $total_time += $value;
    $total_requests++;
  }
}

The output of the script follows. The first column is the time (in seconds since the epoch) and the second column is the average HTTP response time (in microseconds):

[pp@pcentos ~]$ ./http_response_time.pl
1339653360 2912326
1339653420 3002692
1339653480 1358454
1339653540 3983638

This example demonstrated how sampled transactions can be used to generate new metrics. The metrics can be fed back to Ganglia using gmetric; see Custom Metrics Using gmetric.

You don’t have to write your own analysis scripts. The sFlow data can be converted into different forms for compatibility with existing tools. For example, the following command uses sflowtool to convert the binary sFlow HTTP operation data into ASCII CLF so that the operations can be visually inspected or exported to a web log analyzer such as Webalizer:

sflowtool -H | rotatelogs log/http_log

You can also use protocol reporting tools such as Wireshark with sFlow, using sflowtool to convert sFlow into the standard PCAP format:

wireshark -k -i <(sflowtool -t)

In addition, there are a broad range of native sFlow analysis options that provide complementary functionality to Ganglia’s. The sFlow.org website contains a list of open source and commercial sFlow analysis tools—including Ganglia, of course!

When using multiple sFlow analysis tools, each tool needs to receive copies of the sFlow packets. There are two main approaches to replicating sFlow:

Finally, the case study Tagged, Inc. demonstrates how Ganglia and sFlow are used in a large deployment, illustrating the techniques described in this chapter and providing examples that demonstrate the importance of performance monitoring to the operations and developer teams.