org.apache.tez.dag.app

Class DAGAppMaster

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.tez.dag.app.DAGAppMaster

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

public class DAGAppMaster
extends org.apache.hadoop.service.AbstractService

The Tez DAG Application Master. The state machine is encapsulated in the implementation of Job interface. All state changes happens via Job interface. Each event results in a Finite State Transition in Job. Tez DAG AppMaster is the composition of loosely coupled services. The services interact with each other via events. The components resembles the Actors model. The component acts on received event and send out the events to other components. This keeps it highly concurrent with no or minimal synchronization needs. The events are dispatched by a central Dispatch mechanism. All components register to the Dispatcher. The information is shared across different components using AppContext.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	DAGAppMaster.DAGAppMasterShutdownHandler

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	isLastAMRetry
static int	SHUTDOWN_HOOK_PRIORITY Priority of the DAGAppMaster shutdown hook.
protected DAGAppMaster.DAGAppMasterShutdownHandler	shutdownHandler

Constructor Summary

Constructors

Constructor and Description
DAGAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.util.Clock clock, long appSubmitTime, boolean isSession, String workingDirectory, String clientVersion)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addIfService(Object object, boolean addDispatcher)
protected void	addIfServiceDependency(Object object, Object dependency)
protected ContainerHeartbeatHandler	createContainerHeartbeatHandler(AppContext context, org.apache.hadoop.conf.Configuration conf)
protected ContainerLauncher	createContainerLauncher(AppContext context)
protected ContainerSignatureMatcher	createContainerSignatureMatcher()
protected DAG	createDAG(org.apache.tez.dag.api.records.DAGProtos.DAGPlan dagPB)
protected org.apache.hadoop.yarn.event.Dispatcher	createDispatcher()
protected TaskAttemptListener	createTaskAttemptListener(AppContext context, TaskHeartbeatHandler thh, ContainerHeartbeatHandler chh)
protected TaskHeartbeatHandler	createTaskHeartbeatHandler(AppContext context, org.apache.hadoop.conf.Configuration conf)
org.apache.tez.common.security.ACLManager	getACLManager()
org.apache.hadoop.yarn.api.records.ContainerId	getAppContainerId()
org.apache.hadoop.yarn.api.records.ApplicationId	getAppID()
String	getAppNMHost()
int	getAppNMHttpPort()
int	getAppNMPort()
org.apache.hadoop.yarn.api.records.ApplicationAttemptId	getAttemptID()
ContainerLauncher	getContainerLauncher()
AppContext	getContext()
List<String>	getDiagnostics()
org.apache.hadoop.yarn.event.Dispatcher	getDispatcher()
float	getProgress()
int	getRpcPort()
int	getStartCount()
DAGAppMasterState	getState()
TaskAttemptListener	getTaskAttemptListener()
protected TaskSchedulerEventHandler	getTaskSchedulerEventHandler()
protected void	handle(DAGAppMasterEvent event)
static void	initAndStartAppMaster(DAGAppMaster appMaster, String jobUserName)
boolean	isSession()
static void	main(String[] args)
void	serviceInit(org.apache.hadoop.conf.Configuration conf)
void	serviceStart()
void	serviceStop()
void	setCurrentDAG(DAG currentDAG)
void	shutdownTezAM()
String	submitDAGToAppMaster(org.apache.tez.dag.api.records.DAGProtos.DAGPlan dagPlan, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalResources)
protected void	sysexit() Exit call.
void	tryKillDAG(DAG dag)

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SHUTDOWN_HOOK_PRIORITY

public static final int SHUTDOWN_HOOK_PRIORITY

Priority of the DAGAppMaster shutdown hook.

See Also:

Constant Field Values

shutdownHandler

protected DAGAppMaster.DAGAppMasterShutdownHandler shutdownHandler

isLastAMRetry

protected boolean isLastAMRetry

Constructor Detail

DAGAppMaster

public DAGAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId,
            org.apache.hadoop.yarn.api.records.ContainerId containerId,
            String nmHost,
            int nmPort,
            int nmHttpPort,
            org.apache.hadoop.yarn.util.Clock clock,
            long appSubmitTime,
            boolean isSession,
            String workingDirectory,
            String clientVersion)

Method Detail

serviceInit

public void serviceInit(org.apache.hadoop.conf.Configuration conf)
                 throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

createContainerSignatureMatcher

protected ContainerSignatureMatcher createContainerSignatureMatcher()

createDispatcher

protected org.apache.hadoop.yarn.event.Dispatcher createDispatcher()

sysexit

protected void sysexit()

Exit call. Just in a function call to enable testing.

getTaskSchedulerEventHandler

protected TaskSchedulerEventHandler getTaskSchedulerEventHandler()

handle

protected void handle(DAGAppMasterEvent event)

setCurrentDAG

public void setCurrentDAG(DAG currentDAG)

getACLManager

public org.apache.tez.common.security.ACLManager getACLManager()

createDAG

protected DAG createDAG(org.apache.tez.dag.api.records.DAGProtos.DAGPlan dagPB)

addIfService

protected void addIfService(Object object,
                boolean addDispatcher)

addIfServiceDependency

protected void addIfServiceDependency(Object object,
                          Object dependency)

createTaskAttemptListener

protected TaskAttemptListener createTaskAttemptListener(AppContext context,
                                            TaskHeartbeatHandler thh,
                                            ContainerHeartbeatHandler chh)

createTaskHeartbeatHandler

protected TaskHeartbeatHandler createTaskHeartbeatHandler(AppContext context,
                                              org.apache.hadoop.conf.Configuration conf)

createContainerHeartbeatHandler

protected ContainerHeartbeatHandler createContainerHeartbeatHandler(AppContext context,
                                                        org.apache.hadoop.conf.Configuration conf)

createContainerLauncher

protected ContainerLauncher createContainerLauncher(AppContext context)
                                             throws UnknownHostException

Throws:

UnknownHostException

getAppID

public org.apache.hadoop.yarn.api.records.ApplicationId getAppID()

getAttemptID

public org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptID()

getStartCount

public int getStartCount()

getContext

public AppContext getContext()

getDispatcher

public org.apache.hadoop.yarn.event.Dispatcher getDispatcher()

getContainerLauncher

public ContainerLauncher getContainerLauncher()

getTaskAttemptListener

public TaskAttemptListener getTaskAttemptListener()

getAppContainerId

public org.apache.hadoop.yarn.api.records.ContainerId getAppContainerId()

getAppNMHost

public String getAppNMHost()

getAppNMPort

public int getAppNMPort()

getAppNMHttpPort

public int getAppNMHttpPort()

getRpcPort

public int getRpcPort()

getState

public DAGAppMasterState getState()

getDiagnostics

public List<String> getDiagnostics()

getProgress

public float getProgress()

shutdownTezAM

public void shutdownTezAM()

submitDAGToAppMaster

public String submitDAGToAppMaster(org.apache.tez.dag.api.records.DAGProtos.DAGPlan dagPlan,
                          Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalResources)
                            throws org.apache.tez.dag.api.TezException

Throws:

org.apache.tez.dag.api.TezException

tryKillDAG

public void tryKillDAG(DAG dag)

serviceStart

public void serviceStart()
                  throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

public void serviceStop()
                 throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

isSession

public boolean isSession()

main

public static void main(String[] args)

initAndStartAppMaster

public static void initAndStartAppMaster(DAGAppMaster appMaster,
                         String jobUserName)
                                  throws IOException,
                                         InterruptedException

Throws:

IOException

InterruptedException