Running a Jar File on Amazon Elastic MapReduce: A Step by Step Guide

In this post I am describing the steps of running hadoop jar files on Amazon Elastic MapReduce (EMR).

1- Install EMR Command Line Tool as described here.

2- Upload the dataset file (dataset.txt) and the jar file (myjar.jar) to Amazon S3.
     https://console.aws.amazon.com/s3/home

3-  cd to the installation directory of the command line tool

4- Launch the EMR Cluster

./elastic-mapreduce --create --alive --name "cluster name" --hadoop-version 1.0.3 --num-instances 10 --instance-type m1.large --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop --args "-m","mapred.reduce.child.java.opts=-Xmx2g"

The command above starts an EMR cluster using hadoop version 1.0.3. The cluster consists of 10 nodes (1 master and 9 slaves). The 10 nodes are all m1.large EC2 instances. I am also using a bootstrap action to enlarge the JVM heap size of the reducer (more details: in a previous post here).
The job Id (e.g. j-4CYDKHIUAQL7) will be printed out after running the command above. 

5- Wait till the cluster is started and bootstrapped. You can use the command

./elastic-mapreduce --list j-4CYDKHIUAQL7

to pull the state of the cluster. An output like this:

j-4CYDKHIUAQL7      STARTING                                                       cluster_name

means that the cluster is still starting. Wait till STARTING is replaced with WAITING.

6- Use the following command to get the public DNS address of the master node

./elastic-mapreduce --describe j-4CYDKHIUAQL7 | grep MasterPublicDnsName | cut -d'"' -f4

7- Log in to the master node to copy the dataset from S3 to HDFS installed on the cluster. You might decide to run the Jar on the dataset on S3. That is perfectly fine if a single iteration is carried out over the dataset. But, when multiple iterations over the dataset are expected, performance-wise its better to copy the dataset to HDFS only once before starting the job.

ssh -A hadoop@ec2-xx-xx-xxx-xx.compute-1.amazonaws.com

8- Set the configurations of s3cmd (a tool to 'get' & 'put' files from/to S3). s3cmd will be installed on the master EMR node. You just need to set its configurations

hadoop@ip-xx-xx-xxx-xxx:~$ s3cmd --configure

The two required configurations are the access key and the secret key of your AWS account. The other configurations can be set to their default values.

9- Copy the dataset from S3 to the local storage of the master node

hadoop@ip-xx-xx-xxx-xxx:~$ s3cmd get s3://mys3bucket/dataset.txt dataset.txt

10- Copy the dataset from the local storage to HDFS.

hadoop@ip-xx-xx-xxx-xxx:~$ hadoop dfs -copyFromLocal dataset.txt /dataset.txt

11- Now you can log out of the master node. Back to the command line tool of EMR and run the jar file:

./elastic-mapreduce -j j-4CYDKHIUAQL7 --jar s3n://mys3bucket/myjar.jar --args "/dataset.txt"

Here I am passed the path of the dataset on HDFS as the first argument to the jar file.

12- You can monitor the job execution via EMR web console or even using the command in step 5.

That is all about it!