org.apache.hadoop.mapreduce.v2.app

Class MRAppMaster

java.lang.Object

org.apache.hadoop.yarn.service.AbstractService

org.apache.hadoop.yarn.service.CompositeService

org.apache.hadoop.mapreduce.v2.app.MRAppMaster

All Implemented Interfaces:

org.apache.hadoop.yarn.service.Service

public class MRAppMaster
extends org.apache.hadoop.yarn.service.CompositeService

The Map-Reduce Application Master. The state machine is encapsulated in the implementation of Job interface. All state changes happens via Job interface. Each event results in a Finite State Transition in Job. MR AppMaster is the composition of loosely coupled services. The services interact with each other via events. The components resembles the Actors model. The component acts on received event and send out the events to other components. This keeps it highly concurrent with no or minimal synchronization needs. The events are dispatched by a central Dispatch mechanism. All components register to the Dispatcher. The information is shared across different components using AppContext.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.service.CompositeService

org.apache.hadoop.yarn.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.yarn.service.Service

org.apache.hadoop.yarn.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.security.UserGroupInformation	currentUser
protected MRAppMetrics	metrics
static int	SHUTDOWN_HOOK_PRIORITY Priority of the MRAppMaster shutdown hook.

Constructor Summary

Constructors

Constructor and Description
MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.Clock clock, long appSubmitTime)
MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, long appSubmitTime)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addIfService(Object object)
void	cleanupStagingDir() clean up staging directories for the job.
protected ClientService	createClientService(AppContext context)
protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent>	createCommitterEventHandler(AppContext context, org.apache.hadoop.mapreduce.OutputCommitter committer)
protected ContainerAllocator	createContainerAllocator(ClientService clientService, AppContext context)
protected ContainerLauncher	createContainerLauncher(AppContext context)
protected org.apache.hadoop.yarn.event.Dispatcher	createDispatcher()
protected Job	createJob(org.apache.hadoop.conf.Configuration conf, JobStateInternal forcedState, String diagnostic) Create and initialize (but don't start) a single job.
protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent>	createJobFinishEventHandler() create an event handler that handles the job finish event.
protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>	createJobHistoryHandler(AppContext context)
protected Speculator	createSpeculator(org.apache.hadoop.conf.Configuration conf, AppContext context)
protected org.apache.hadoop.yarn.service.AbstractService	createStagingDirCleaningService()
protected TaskAttemptListener	createTaskAttemptListener(AppContext context)
protected void	downloadTokensAndSetupUGI(org.apache.hadoop.conf.Configuration conf) Obtain the tokens needed by the job and put them in the UGI
List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo>	getAllAMInfos()
org.apache.hadoop.yarn.api.records.ApplicationId	getAppID()
org.apache.hadoop.yarn.api.records.ApplicationAttemptId	getAttemptID()
org.apache.hadoop.mapreduce.OutputCommitter	getCommitter()
Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo>	getCompletedTaskFromPreviousRun()
ContainerAllocator	getContainerAllocator()
ContainerLauncher	getContainerLauncher()
AppContext	getContext()
protected org.apache.hadoop.security.Credentials	getCredentials()
org.apache.hadoop.yarn.event.Dispatcher	getDispatcher()
protected org.apache.hadoop.fs.FileSystem	getFileSystem(org.apache.hadoop.conf.Configuration conf) Create the default file System for this job.
org.apache.hadoop.mapreduce.v2.api.records.JobId	getJobId()
protected RMHeartbeatHandler	getRMHeartbeatHandler()
int	getStartCount()
TaskAttemptListener	getTaskAttemptListener()
void	init(org.apache.hadoop.conf.Configuration conf)
protected static void	initAndStartAppMaster(MRAppMaster appMaster, org.apache.hadoop.yarn.conf.YarnConfiguration conf, String jobUserName)
boolean	isNewApiCommitter()
protected boolean	keepJobFiles(org.apache.hadoop.mapred.JobConf conf)
static void	main(String[] args)
void	shutDownJob()
void	start()
protected void	startJobs() This can be overridden to instantiate multiple jobs and create a workflow.
protected void	sysexit() Exit call.

Methods inherited from class org.apache.hadoop.yarn.service.CompositeService

addService, getServices, removeService, stop

Methods inherited from class org.apache.hadoop.yarn.service.AbstractService

getConfig, getName, getServiceState, getStartTime, register, unregister

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SHUTDOWN_HOOK_PRIORITY

public static final int SHUTDOWN_HOOK_PRIORITY

Priority of the MRAppMaster shutdown hook.

See Also:

Constant Field Values

metrics

protected final MRAppMetrics metrics

currentUser

protected org.apache.hadoop.security.UserGroupInformation currentUser

Constructor Detail

MRAppMaster

public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
           org.apache.hadoop.yarn.api.records.ContainerId containerId,
           String nmHost,
           int nmPort,
           int nmHttpPort,
           long appSubmitTime)

MRAppMaster

public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
           org.apache.hadoop.yarn.api.records.ContainerId containerId,
           String nmHost,
           int nmPort,
           int nmHttpPort,
           org.apache.hadoop.yarn.Clock clock,
           long appSubmitTime)

Method Detail

init

public void init(org.apache.hadoop.conf.Configuration conf)

Specified by:

init in interface org.apache.hadoop.yarn.service.Service

Overrides:

init in class org.apache.hadoop.yarn.service.CompositeService

createDispatcher

protected org.apache.hadoop.yarn.event.Dispatcher createDispatcher()

keepJobFiles

protected boolean keepJobFiles(org.apache.hadoop.mapred.JobConf conf)

getFileSystem

protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException

Create the default file System for this job.

Parameters:

conf - the conf object

Returns:

the default filesystem for this job

Throws:

IOException

getCredentials

protected org.apache.hadoop.security.Credentials getCredentials()

cleanupStagingDir

public void cleanupStagingDir()
                       throws IOException

clean up staging directories for the job.

Throws:

IOException

sysexit

protected void sysexit()

Exit call. Just in a function call to enable testing.

shutDownJob

public void shutDownJob()

createJobFinishEventHandler

protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent> createJobFinishEventHandler()

create an event handler that handles the job finish event.

Returns:

the job finish event handler.

createJob

protected Job createJob(org.apache.hadoop.conf.Configuration conf,
            JobStateInternal forcedState,
            String diagnostic)

Create and initialize (but don't start) a single job.

Parameters:

forcedState - a state to force the job into or null for normal operation.

diagnostic - a diagnostic message to include with the job.

downloadTokensAndSetupUGI

protected void downloadTokensAndSetupUGI(org.apache.hadoop.conf.Configuration conf)

Obtain the tokens needed by the job and put them in the UGI

Parameters:

conf -

addIfService

protected void addIfService(Object object)

createJobHistoryHandler

protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent> createJobHistoryHandler(AppContext context)

createStagingDirCleaningService

protected org.apache.hadoop.yarn.service.AbstractService createStagingDirCleaningService()

createSpeculator

protected Speculator createSpeculator(org.apache.hadoop.conf.Configuration conf,
                          AppContext context)

createTaskAttemptListener

protected TaskAttemptListener createTaskAttemptListener(AppContext context)

createCommitterEventHandler

protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent> createCommitterEventHandler(AppContext context,
                                                                                    org.apache.hadoop.mapreduce.OutputCommitter committer)

createContainerAllocator

protected ContainerAllocator createContainerAllocator(ClientService clientService,
                                          AppContext context)

getRMHeartbeatHandler

protected RMHeartbeatHandler getRMHeartbeatHandler()

createContainerLauncher

protected ContainerLauncher createContainerLauncher(AppContext context)

createClientService

protected ClientService createClientService(AppContext context)

getAppID

public org.apache.hadoop.yarn.api.records.ApplicationId getAppID()

getAttemptID

public org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptID()

getJobId

public org.apache.hadoop.mapreduce.v2.api.records.JobId getJobId()

getCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getCommitter()

isNewApiCommitter

public boolean isNewApiCommitter()

getStartCount

public int getStartCount()

getContext

public AppContext getContext()

getDispatcher

public org.apache.hadoop.yarn.event.Dispatcher getDispatcher()

getCompletedTaskFromPreviousRun

public Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo> getCompletedTaskFromPreviousRun()

getAllAMInfos

public List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo> getAllAMInfos()

getContainerAllocator

public ContainerAllocator getContainerAllocator()

getContainerLauncher

public ContainerLauncher getContainerLauncher()

getTaskAttemptListener

public TaskAttemptListener getTaskAttemptListener()

start

public void start()

Specified by:

start in interface org.apache.hadoop.yarn.service.Service

Overrides:

start in class org.apache.hadoop.yarn.service.CompositeService

startJobs

protected void startJobs()

This can be overridden to instantiate multiple jobs and create a workflow. TODO: Rework the design to actually support this. Currently much of the job stuff has been moved to init() above to support uberization (MR-1220). In a typical workflow, one presumably would want to uberize only a subset of the jobs (the "small" ones), which is awkward with the current design.

main

public static void main(String[] args)

initAndStartAppMaster

protected static void initAndStartAppMaster(MRAppMaster appMaster,
                         org.apache.hadoop.yarn.conf.YarnConfiguration conf,
                         String jobUserName)
                                     throws IOException,
                                            InterruptedException

Throws:

IOException

InterruptedException