How to Run Hadoop MapReduce Program on Ubuntu 16.04

In this blog, I will show you how to run a MapReduce program. MapReduce is one of the core part of Apache Hadoop, it is the processing layer of Apache Hadoop. So before I show you how to run a MapReduce program, let me briefly explain you MapReduce.

MapReduce is a system for parallel processing of large data sets. MapReduce reduces the data into results and creates a summary of the data. A mapreduce program has two parts - mapper and reducer. After the mapper finishes its work then only reducers start.

Mapper : It maps input key/value pairs to a set of intermediate key/value pairs.

Reducer : It reduces a set of intermediate values which share a key to a smaller set of values.

Basically, in the wordcount mapreduce program, we provide input file(s) - any text file, as input. When the mapreduce program starts, below are the processes it goes through:

Splitting : It splits the each line in the input file into words.

Mapping : It forms a key value pair, where word is the key and 1 is the value assigned to each key.

Shuffling : Common key value pairs get grouped together.

Reducing : The values of similar keys are added together.

Running MapReduce Program

A MapReduce program is written in Java. And mostly Eclipse IDE is used for programming by the developers. So in this blog, I will show you how to export a mapreduce program into a jar file from Eclipse IDE and run it on a Hadoop cluster.

My MapReduce program is there in my Eclipse IDE.

Now to run this MapReduce program on a hadoop cluster, we will export the project as a jar file. Select File option in eclipse ide and click on Export. In Java option, select Jar file and click on Next.

Select the Wordcount project and give the path and name for the jar file, I am keeping it wordcount.jar, Click on Next twice.

Now Click on Browse and select the main class and finally click on Finish to make the jar file. In case you get any warning as below, just click OK.

Check whether your Hadoop cluster is up and working or not.

Command: jps

hadoop@hadoop-VirtualBox:~$ jps

3008 NodeManager

3924 Jps

2885 ResourceManager

2505 DataNode

3082 JobHistoryServer

2716 SecondaryNameNode

2383 NameNode

hadoop@hadoop-VirtualBox:~$

We have our input file onto HDFS for wordcount program.

hadoop@hadoop-VirtualBox:~$ hdfs dfs -put input /

hadoop@hadoop-VirtualBox:~$ hdfs dfs -cat /input

This is my first mapreduce test

This is wordcount program

hadoop@hadoop-VirtualBox:~$

Now run the wordcount.jar file using below command.

Note: Since we selected the main class while exporting wordcount.jar , so no need to mention the main class in the command.

Command: hadoop jar wordcount.jar /input /output

hadoop@hadoop-VirtualBox:~$ hadoop jar wordcount.jar /input /output

16/11/27 22:52:20 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:
8032

16/11/27 22:52:22 WARN mapreduce.JobResourceUploader: Hadoop command-line option 
parsing not performed. Implement the Tool interface and execute your application 
with ToolRunner to remedy this.

16/11/27 22:52:27 INFO input.FileInputFormat: Total input paths to process : 1

16/11/27 22:52:28 INFO mapreduce.JobSubmitter: number of splits:1

16/11/27 22:52:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_14802
67251741_0001

16/11/27 22:52:32 INFO impl.YarnClientImpl: Submitted application application_14802
67251741_0001

16/11/27 22:52:33 INFO mapreduce.Job: The url to track the job: http://hadoop-Virtu
alBox:8088/proxy/application_1480267251741_0001/

16/11/27 22:52:33 INFO mapreduce.Job: Running job: job_1480267251741_0001

16/11/27 22:53:20 INFO mapreduce.Job: Job job_1480267251741_0001 running in uber mo
de : false

16/11/27 22:53:20 INFO mapreduce.Job:  map 0% reduce 0%

16/11/27 22:53:45 INFO mapreduce.Job:  map 100% reduce 0%

16/11/27 22:54:13 INFO mapreduce.Job:  map 100% reduce 100%

16/11/27 22:54:15 INFO mapreduce.Job: Job job_1480267251741_0001 completed 
successfully

16/11/27 22:54:16 INFO mapreduce.Job: Counters: 49

          File System Counters

                    FILE: Number of bytes read=124

                    FILE: Number of bytes written=237911

                    FILE: Number of read operations=0

                    FILE: Number of large read operations=0

                    FILE: Number of write operations=0

                    HDFS: Number of bytes read=150

                    HDFS: Number of bytes written=66

                    HDFS: Number of read operations=6

                    HDFS: Number of large read operations=0

                    HDFS: Number of write operations=2

          Job Counters

                    Launched map tasks=1

                    Launched reduce tasks=1

                    Data-local map tasks=1

                    Total time spent by all maps in occupied slots (ms)=21062

                    Total time spent by all reduces in occupied slots (ms)=25271

                    Total time spent by all map tasks (ms)=21062

                    Total time spent by all reduce tasks (ms)=25271

                    Total vcore-milliseconds taken by all map tasks=21062

                    Total vcore-milliseconds taken by all reduce tasks=25271

                    Total megabyte-milliseconds taken by all map tasks=21567488

                    Total megabyte-milliseconds taken by all reduce tasks=25877504

          Map-Reduce Framework

                    Map input records=2

                    Map output records=10

                    Map output bytes=98

                    Map output materialized bytes=124

                    Input split bytes=92

                    Combine input records=0

                    Combine output records=0

                    Reduce input groups=8

                    Reduce shuffle bytes=124

                    Reduce input records=10

                    Reduce output records=8

                    Spilled Records=20

                    Shuffled Maps =1

                    Failed Shuffles=0

                    Merged Map outputs=1

                    GC time elapsed (ms)=564

                    CPU time spent (ms)=4300

                    Physical memory (bytes) snapshot=330784768

                    Virtual memory (bytes) snapshot=3804205056

                    Total committed heap usage (bytes)=211812352

          Shuffle Errors

                    BAD_ID=0

                    CONNECTION=0

                    IO_ERROR=0

                    WRONG_LENGTH=0

                    WRONG_MAP=0

                    WRONG_REDUCE=0

          File Input Format Counters

                    Bytes Read=58

          File Output Format Counters

                    Bytes Written=66

hadoop@hadoop-VirtualBox:~$

After the program runs successfully, go to HDFS and check the part file inside output directory.

Below is the output of wordcount program.

hadoop@hadoop-VirtualBox:~$ hdfs dfs -cat /output/part-r-00000

 This    2
 first     1
 is        2
 mapreduce   1
 my      1
 program        1
 test     1
 wordcount     1
 hadoop@hadoop-VirtualBox:~$

Conclusion

This example here is in Java, you can write a MapReduce program in python as well. We successfully ran a Hadoop MapReduce program on a Hadoop Cluster on Ubuntu 16.04. The steps to run a Mapreduce Program on other Linux environments remains the same.  Make sure that before running the program, you Hadoop cluster should be up and running, also your input file should be present in HDFS.

Filed Under : CLUSTER

Tagged With : Hadoop