Small teams and start-ups often don’t have the resources to set up their own cluster. An in-house cluster is a fixed cost of initial investment. It requires effort to set up and servers and switches as well as maintaining a Hadoop and Hive installation.

On the other hand, Elastic MapReduce comes with a variable cost, plus the installation and maintenance is Amazon’s responsibility. This is a huge benefit for teams that can’t or don’t want to invest in their own clusters, and even for larger teams that need a test bed to try out new tools and ideas without affecting their production clusters.

An Amazon cluster is comprised of one or more instances. Instances come in various sizes, with different RAM, compute power, disk drive, platform, and I/O performance. It can be hard to determine what size would work the best for your use case. With EMR, it’s easy to start with small instance sizes, monitor performance with tools like Ganglia, and then experiment with different instance sizes to find the best balance of cost versus performance.

Before using Amazon EMR, you need to set up an Amazon Web Services (AWS) account. The Amazon EMR Getting Started Guide provides instructions on how to sign up for an AWS account.

You will also need to create an Amazon S3 bucket for storing your input data and retrieving the output results of your Hive processing.

When you set up your AWS account, make sure that all your Amazon EC2 instances, key pairs, security groups, and EMR jobflows are located in the same region to avoid cross-region transfer costs. Try to locate your Amazon S3 buckets and EMR jobflows in the same availability zone for better performance.

Although Amazon EMR supports several versions of Hadoop and Hive, only some combinations of versions of Hadoop and Hive are supported. See the Amazon EMR documentation to find out the supported version combinations of Hadoop and Hive.

Amazon provides multiple ways to bring up, terminate, and modify a Hive cluster. Currently, there are three ways you can manage your EMR Hive cluster:

It is common to use more than one way to manage Hive clusters.

Here is an example that uses the Ruby elastic-mapreduce CLI to start up a single-node Amazon EMR cluster with Hive configured. It also sets up the cluster for interactive use, rather than for running a job and exiting. This cluster would be ideal for learning Hive:

elastic-mapreduce --create --alive --name "Test Hive" --hive-interactive

If you also want Pig available, add the --pig-interface option.

Next you would log in to this cluster as described in the Amazon EMR documentation.

Typically, the Hive Thrift server (see Chapter 16) listens for connections on port 10000. However, in the Amazon Hive installation, this port number depends on the version of Hive being used. This change was implemented in order to allow users to install and support concurrent versions of Hive. Consequently, Hive v0.5.X operates on port 10000, Hive v0.7.X on 10001, and Hive v0.7.1 on 10002. These port numbers are expected to change as newer versions of Hive get ported to Amazon EMR.

Each Amazon cluster has one or more nodes. Each of these nodes can fit into one of the following three instance groups:

The task instance group is convenient when you want to increase cluster capacity during hours of peak demand and bring it back to normal afterwards. It is also useful when using spot instances (discussed below) for lower costs without risking the loss of data when a node gets removed from the cluster.

If you are running a cluster with just a single node, the node would be a master node and a core node at the same time.

You will often want to deploy your own configuration files when launching an EMR cluster. The most common files to customize are hive-site.xml, .hiverc, hadoop-env.sh. Amazon provides a way to override these configuration files.

elastic-mapreduce --create --alive --name "Test Hive" --hive-interactive 
--hive-site=s3n://example.hive.oreilly.com/conf/hive_site.xml

s3cmd get s3n://example.hive.oreilly.com/conf/.hiverc ~/.hiverc

--bootstrap-action
  s3n://elasticmapreduce/bootstrap-actions/configurations/latest/memory-intensive

An EMR cluster comes with a MySQL server installed on the master node of the cluster. By default, EMR Hive uses this MySQL server as its metastore. However, all data stored on the cluster nodes are deleted once you terminate your cluster. This includes the data stored on the master node metastore, as well! This is usually unacceptable because you would like to retain your table schemas, etc., in a persistent metastore.

You can use one of the following methods to work around this limitation:

Note that this metastore is not shared amongst different versions of Hive running on your EMR cluster. Suppose you spin up a cluster with both Hive v0.5 and v0.7.1 installed. When you create a table using Hive v0.5, you won’t be able to access this table using Hive v0.7.1. If you would like to share the metadata between different Hive versions, you will have to use an external persistent metastore.

HDFS and S3 have their own distinct roles in an EMR cluster. All the data stored on the cluster nodes is deleted once the cluster is terminated. Since HDFS is formed by ephemeral storage of the nodes in the core instance group, the data stored on HDFS is lost after cluster termination.

S3, on the other hand, provides a persistent storage for data associated with the EMR cluster. Therefore, the input data to the cluster should be stored on S3 and the final results obtained from Hive processing should be persisted to S3, as well.

However, S3 is an expensive storage alternative to HDFS. Therefore, intermediate results of processing should be stored in HDFS, with only the final results saved to S3 that need to persist.

Please note that as a side effect of using S3 as a source for input data, you lose the Hadoop data locality optimization, which may be significant. If this optimization is crucial for your analysis, you should consider importing “hot” data from S3 onto HDFS before processing it. This initial overhead will allow you to make use of Hadoop’s data locality optimization in your subsequent processing.

You should upload all your bootstrap scripts, configuration scripts (e.g., hive-site.xml and .hiverc), resources (e.g., files that need to go in the distributed cache, UDF or streaming JARs), etc., onto S3. Since EMR Hive and Hadoop installations natively understand S3 paths, it is straightforward to work with these files in subsequent Hadoop jobs.

For example, you can add the following lines in .hiverc without any errors:

ADD FILE s3n://example.hive.oreilly.com/files/my_file.txt;
ADD JAR s3n://example.hive.oreilly.com/jars/udfs.jar;
CREATE TEMPORARY FUNCTION my_count AS 'com.oreilly.hive.example.MyCount';

Amazon EMR saves the log files to the S3 location pointed to by the log-uri field. These include logs from bootstrap actions of the cluster and the logs from running daemon processes on the various cluster nodes. The log-uri field can be set in the credentials.json file found in the installation directory of the elastic-mapreduce Ruby client. It can also be specified or overridden explicitly when spinning up the cluster using elastic-mapreduce by using the --log-uri flag. However, if this field is not set, those logs will not be available on S3.

If your workflow is configured to terminate if your job encounters an error, any logs on the cluster will be lost after the cluster termination. If your log-uri field is set, these logs will be available at the specified location on S3 even after the cluster has been terminated. They can be an essential aid in debugging the issues that caused the failure.

However, if you store logs on S3, remember to purge unwanted logs on a frequent basis to save yourself from unnecessary storage costs!

Spot instances allows users to bid on unused Amazon capacity to get instances at cheaper rates compared to on-demand prices. Amazon’s online documentation describes them in more detail.

Depending on your use case, you might want instances in all three instance groups to be spot instances. In this case, your entire cluster could terminate at any stage during the workflow, resulting in a loss of intermediate data. If it’s “cheap” to repeat the calculation, this might not be a serious issue. An alternative is to persist intermediate data periodically to S3, as long as your jobs can start again from those snapshots.

Another option is to only include the nodes in the task instance group as spot nodes. If these spot nodes get taken out of the cluster because of unavailability or because the spot prices increased, the workflow will continue with the master and core nodes, but with no data loss. When spot nodes get added to the cluster again, MapReduce tasks can be delegated to them, speeding up the workflow.

Using the elastic-mapreduce Ruby client, spot instances can be ordered by using the --bid-price option along with a bid price. The following example shows a command to create a cluster with one master node, two core nodes and two spot nodes (in the task instance group) with a bid price of 10 cents:

elastic-mapreduce --create --alive --hive-interactive 
--name "Test Spot Instances" 
--instance-group master --instance-type m1.large 
--instance-count 1 --instance-group core 
--instance-type m1.small --instance-count 2 --instance-group task 
--instance-type m1.small --instance-count 2 --bid-price 0.10

If you are spinning up a similar cluster using the Java SDK, use the following InstanceGroupConfig variables for master, core, and task instance groups:

InstanceGroupConfig masterConfig = new InstanceGroupConfig()
.withInstanceCount(1)
.withInstanceRole("MASTER")
.withInstanceType("m1.large");
InstanceGroupConfig coreConfig = new InstanceGroupConfig()
.withInstanceCount(2)
.withInstanceRole("CORE")
.withInstanceType("m1.small");
InstanceGroupConfig taskConfig = new InstanceGroupConfig()
.withInstanceCount(2)
.withInstanceRole("TASK")
.withInstanceType("m1.small")
.withMarket("SPOT")
.withBidPrice("0.05");

The Hadoop JobTracker and NameNode User Interfaces are accessible on port 9100 and 9101 respectively in the EMRmaster node. You can use ssh tunneling or a dynamic SOCKS proxy to view them.

In order to be able to view these from a browser on your client machine (outside of the Amazon network), you need to modify the Elastic MapReduce master security group via your AWS Web Console. Add a new custom TCP rule to allow inbound connections from your client machine’s IP address on ports 9100 and 9101.

An elastic alternative to EMR is to bring up several Amazon EC2 nodes and install Hadoop and Hive on a custom Amazon Machine Image (AMI). This approach gives you more control over the version and configuration of Hive and Hadoop. For example, you can experiment with new releases of tools before they are made available through EMR.

The drawback of this approach is that customizations available through EMR may not be available in the Apache Hive release. As an example, the S3 filesystem is not fully supported on Apache Hive [see JIRA HIVE-2318]. There is also an optimization for reducing start-up time for Amazon S3 queries, which is only available in EMR Hive. This optimization is enabled by adding the following snippet in your hive-site.xml:

<property>
  <name>hive.optimize.s3.query</name>
  <value>true</value>
  <description> Improves Hive query performance for Amazon S3 queries
  by reducing their start up time </description>
</property>

Alternatively, you can run the following command on your Hive CLI:

set hive.optimize.s3.query=true;

Another example is a command that allows the user to recover partitions if they exist in the correct directory structure on HDFS or S3. This is convenient when an external process is populating the contents of the Hive table in appropriate partitions. In order to track these partitions in the metastore, one could run the following command, where emr_table is the name of the table:

ALTER TABLE emr_table RECOVER PARTITIONS;

Here is the statement that creates the table, for your reference:

CREATE EXTERNAL TABLE emr_table(id INT, value STRING)
PARTITIONED BY (dt STRING)
LOCATION 's3n://example.hive.oreilly.com/tables/emr_table';

Amazon EMR provides an elastic, scalable, easy-to-set-up way to bring up a cluster with Hadoop and Hive ready to run queries as soon as it boots. It works well with data stored on S3. While much of the configuration is done for you, it is flexible enough to allow users to have their own custom configurations.