org.apache.tez.mapreduce.examples

Class TestOrderedWordCount

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.tez.mapreduce.examples.TestOrderedWordCount

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class TestOrderedWordCount
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

An MRR job built on top of word count to return words sorted by their frequency of occurrence. Use -DUSE_TEZ_SESSION=true to run jobs in a session mode. If multiple input/outputs are provided, this job will process each pair as a separate DAG in a sequential manner. Use -DINTER_JOB_SLEEP_INTERVAL= where N is the sleep interval in seconds between the sequential DAGs.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TestOrderedWordCount.IntSumReducer
static class	TestOrderedWordCount.MyOrderByNoOpReducer Shuffle ensures ordering based on count of employees per department hence the final reducer is a no-op and just emits the department name with the employee count per department.
static class	TestOrderedWordCount.TokenizerMapper

Constructor Summary

Constructors

Constructor and Description
TestOrderedWordCount()

Method Summary

Modifier and Type	Method and Description
org.apache.tez.dag.api.DAG	createDAG(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> commonLocalResources, org.apache.hadoop.fs.Path stagingDir, int dagIndex, String inputPath, String outputPath, boolean generateSplitsInClient, boolean useMRSettings, int intermediateNumReduceTasks, int maxDataLengthThroughIPC, int exceedDataLimit)
static void	main(String[] args)
int	run(String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Constructor Detail

TestOrderedWordCount

public TestOrderedWordCount()

Method Detail

createDAG

public org.apache.tez.dag.api.DAG createDAG(org.apache.hadoop.fs.FileSystem fs,
                                            org.apache.hadoop.conf.Configuration conf,
                                            Map<String,org.apache.hadoop.yarn.api.records.LocalResource> commonLocalResources,
                                            org.apache.hadoop.fs.Path stagingDir,
                                            int dagIndex,
                                            String inputPath,
                                            String outputPath,
                                            boolean generateSplitsInClient,
                                            boolean useMRSettings,
                                            int intermediateNumReduceTasks,
                                            int maxDataLengthThroughIPC,
                                            int exceedDataLimit)
                                     throws Exception

Throws:

Exception

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception