MapReduce Program Inverted Index:

Step 1: Create a Project in Eclipse

(i) Open the Eclipse and create a new Java Project After typing name click finish.

 New->Project->Java Project


(ii)  Then Right Click on Project and create a new package and then create a new class in that package.

Project->New->Package
Package->New->Class


(iii) Once Class is created then right click on the project (or) package scroll down to Build path and select configure Build path

Project->Build Path->Configure Build Path
Libraries->Add External Jars

Eclipse build path


(iv) Then browse to the hadoop/share directory and add all the supportive jars in MapReduce,Common and Yarn Folders

Add archieves

 


Step-2: In Class,

Paste the Following Java Code in the Class

package mapreduce; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.lib.input.FileSplit; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer;

public class WordCountDriver { public class WordcountDriver extends Configured implements Tool { public int run(String[] args) throws Exception { if (args.length != 2) { System.out.println("Usage: [input] [output]"); System.exit(-1); }

Job job = Job.getInstance(getConf()); job.setJobName("wordcount"); job.setJarByClass(WordcountDriver.class); /* Field separator for reducer output*/ job.getConfiguration().set("mapreduce.output.textoutputformat.separator", " | "); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class);

job.setMapperClass(WordcountMapper.class); job.setCombinerClass(WordcountReducer.class); job.setReducerClass(WordcountReducer.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class);

Path inputFilePath = new Path(args[0]); Path outputFilePath = new Path(args[1]); /* This line is to accept input recursively */ FileInputFormat.setInputDirRecursive(job, true); FileInputFormat.addInputPath(job, inputFilePath); FileOutputFormat.setOutputPath(job, outputFilePath); /* Delete output filepath if already exists */ FileSystem fs = FileSystem.newInstance(getConf());

if (fs.exists(outputFilePath)) { fs.delete(outputFilePath, true); } return job.waitForCompletion(true) ? 0 : 1; } public static void main(String[] args) throws Exception { WordcountDriver wordcountDriver = new WordcountDriver();

int res = ToolRunner.run(wordcountDriver, args); System.exit(res); } } public class WordcountMapper extends Mapper<LongWritable, Text, Text, Text> {

private Text word = new Text(); private Text filename = new Text(); private boolean caseSensitive = false; @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

String filenameStr = ((FileSplit) context.getInputSplit()).getPath().getName(); filename = new Text(filenameStr); String line = value.toString(); if (!caseSensitive) { line = line.toLowerCase(); }

StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, filename); } }

@Override protected void setup(Context context) throws IOException, InterruptedException { Configuration conf = context.getConfiguration(); this.caseSensitive = conf.getBoolean("wordcount.case.sensitive",false); } } public class WordcountReducer extends Reducer<Text, Text, Text, Text> {

@Override public void reduce(final Text key, final Iterable<Text> values, final Context context) throws IOException, InterruptedException { StringBuilder stringBuilder = new StringBuilder(); for (Text value : values) { stringBuilder.append(value.toString());

if (values.iterator().hasNext()) { stringBuilder.append(" -> "); } } context.write(key, new Text(stringBuilder.toString())); } } }

 


Step-3: Create a Jar

(i) Then Rectify any errors if shown in eclipse otherwise right click on project and click export.

 Project->Export->Jar Archive.

(ii) Then Select Java Archive Format and choose the destination of the jar.

Jar creation

 


Step-4: Now load the two input files into HDFS.

    input_invert1:
            Welcome to Geoinsyssoft, Chennai
    input_invert2:
            Welcome to Geoinsyssoft Leading Big Data Training Institute in Chennai.

hdfs dfs -put /home/geouser/data/input_inverted1 /home/geouser/data/input_inverted2 /input_inverted

geouser@geouser:~$ hdfs dfs -put -f /home/hduser/hivedata/input_invert1 /home/hduser/hivedata/input_invert2 /input_inverted


Step-5: Now Run the Jar File

hadoop jar /home/geouser/jars/inverted_index.jar mapreduce.WordCountDriver /input_inverted/* /output_inverted

geouser@geouser:~$ hadoop jar /home/hduser/jars/inverted_index.jar mapreduce.WordCountDriver /input_inverted/* /output_invert
16/09/08 17:14:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
16/09/08 17:14:53 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/09/08 17:14:56 INFO input.FileInputFormat: Total input paths to process : 2
16/09/08 17:14:56 INFO mapreduce.JobSubmitter: number of splits:2
16/09/08 17:14:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1473328512783_0001
16/09/08 17:14:58 INFO impl.YarnClientImpl: Submitted application application_1473328512783_0001
16/09/08 17:14:59 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1473328512783_0001/
16/09/08 17:14:59 INFO mapreduce.Job: Running job: job_1473328512783_0001
16/09/08 17:15:29 INFO mapreduce.Job: Job job_1473328512783_0001 running in uber mode : false
16/09/08 17:15:29 INFO mapreduce.Job:  map 0% reduce 0%
16/09/08 17:16:06 INFO mapreduce.Job:  map 50% reduce 0%
16/09/08 17:16:13 INFO mapreduce.Job:  map 100% reduce 0%
16/09/08 17:16:47 INFO mapreduce.Job:  map 100% reduce 100%
16/09/08 17:16:49 INFO mapreduce.Job: Job job_1473328512783_0001 completed successfully
16/09/08 17:16:50 INFO mapreduce.Job: Counters: 49

File System Counters
FILE: Number of bytes read=335
FILE: Number of bytes written=349656
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=335
HDFS: Number of bytes written=320
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=97908
Total time spent by all reduces in occupied slots (ms)=19823
Total time spent by all map tasks (ms)=97908
Total time spent by all reduce tasks (ms)=19823
Total vcore-seconds taken by all map tasks=97908
Total vcore-seconds taken by all reduce tasks=19823
Total megabyte-seconds taken by all map tasks=100257792
Total megabyte-seconds taken by all reduce tasks=20298752

Map-Reduce Framework
Map input records=2
Map output records=14
Map output bytes=301
Map output materialized bytes=341
Input split bytes=230
Combine input records=14
Combine output records=14
Reduce input groups=12
Reduce shuffle bytes=341
Reduce input records=14
Reduce output records=12
Spilled Records=28
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=887
CPU time spent (ms)=5430
Physical memory (bytes) snapshot=606748672
Virtual memory (bytes) snapshot=1668476928
Total committed heap usage (bytes)=484704256
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=105
File Output Format Counters
Bytes Written=320

 

Step-6: Now view the results of the MapReduce Process

hdfs dfs -cat /output_invert/part-r-00000

geouser@geouser:~$ hdfs dfs -cat /output_invert/part-r-00000
16/09/08 17:19:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
big | input_invert2
chennai | input_invert1
chennai. | input_invert2
data | input_invert2
geoinsyssoft | input_invert2
geoinsyssoft, | input_invert1
in | input_invert2
institute | input_invert2
leading | input_invert2
to | input_invert2 -> input_invert1
training | input_invert2
welcome | input_invert2 -> input_invert1

Leave a Reply

Your email address will not be published. Required fields are marked *