Configuring the Reducer Heap Size on Elastic MapReduce

I was running mahout MatrixMultiplicationJob on Amazon Elastic MapReduce (EMR) to multiply two matrices of sizes (150k x 8.2m) and (100 x 150k  ). My cluster consisted of 1 master node and 19 core nodes. All of the 20 nodes were m1.large ec2 instances each o which had 7.5GB of memory according to EC2 Docs. Despite the fairly suitable memory size (as I thought), I got an out of memory error in the reduce task.

Task attempt_201209140440_0004_m_000024_0 failed to report status for 601 seconds. Killing!
Error: java.lang.OutOfMemoryError: Java heap space
 at org.apache.mahout.math.map.OpenIntDoubleHashMap.rehash(OpenIntDoubleHashMap.java:434)
 at org.apache.mahout.math.map.OpenIntDoubleHashMap.put(OpenIntDoubleHashMap.java:387)
 at org.apache.mahout.math.RandomAccessSparseVector.setQuick(RandomAccessSparseVector.java:139)
 at org.apache.mahout.math.AbstractVector.assign(AbstractVector.java:560)
 at org.apache.mahout.math.hadoop.MatrixMultiplicationJob$MatrixMultiplicationReducer.reduce(MatrixMultiplicationJob.java:161)
 at org.apache.mahout.math.hadoop.MatrixMultiplicationJob$MatrixMultiplicationReducer.reduce(MatrixMultiplicationJob.java:147)
 at org.apache.hadoop.mapred.Task$OldCombinerRunner.combine(Task.java:1436)
 at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2815)
 at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2753)

Then, I tried to increase the JVM Heap size of the reduce tasks. EMR allows users to use a set of Bootstrap Actions to overwrite the configurations and the environment argument of EMR. Bootstrap Actions can be specified in the --create command (more about EMR commands here). The format of a bootstrap action should be like this:

 --bootstrap-action "s3://myawsbucket/FileName" --args "arg1","arg2"   

FileName refers to a script file that is executed on the cluster to do whatever changes. The required changed should be specified via the arguments of that script.

To alter any of Hadoop configurations, the script file that should be used is:

s3://elasticmapreduce/bootstrap-actions/configure-hadoop 

and the arguments should be formated as "-configFile","parameter=newValue". configFile determines Hadoop configurations file that contains the default value of the parameter being altered. Here are Hadoop configuration files and the corresponding configFile:

File Name	configFile
hadoop-site.xml	s
hadoop-default.xml	d
core-site.xml	c
hdfs-site.xml	h
mapred-site.xml	m

I wanted to increase the heap size of the reducer JVM. So, I had to alter the configuration parameter mapred.reduce.child.java.opts to set the maximum heap size to a larger value (-Xmx2g). That parameter is one of the configurations in mapred-site.xml. So, by bootstrap command was:

 --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop --args "-m","mapred.reduce.child.java.opts=-Xmx2g"

The full --create command was:

 ./elastic-mapreduce --create --alive --name "GCSS Eval-mem" --hadoop-version 1.0.3 --num-instances 20 --instance-type m1.large --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop --args "-m","mapred.reduce.child.java.opts=-Xmx2g"