org.apache.tez.serviceplugins.api

Class TaskCommunicator

java.lang.Object

org.apache.tez.serviceplugins.api.TaskCommunicator

All Implemented Interfaces:

org.apache.tez.common.ServicePluginLifecycle

Direct Known Subclasses:

TezTaskCommunicatorImpl

public abstract class TaskCommunicator
extends Object
implements org.apache.tez.common.ServicePluginLifecycle

This class represents the API for a custom TaskCommunicator which can be run within the Tez AM. This is used to communicate with running services, potentially launching tasks, and getting updates from running tasks.

The plugin is initialized with an instance of TaskCommunicatorContext - which provides a mechanism to notify the system about allocation decisions and resources to the Tez framework. If setting up a heartbeat between the task and the AM, the framework is responsible for error checking of this heartbeat mechanism, handling lost or duplicate responses.

Constructor Summary

Constructors

Constructor and Description
TaskCommunicator(TaskCommunicatorContext taskCommunicatorContext)

Method Summary

Modifier and Type	Method and Description
abstract void	dagComplete(int dagIdentifier) Indicates the current running dag is complete.
abstract InetSocketAddress	getAddress() Return the address, if any, that the service listens on
String	getCompletedLogsUrl(org.apache.tez.dag.records.TezTaskAttemptID attemptID, org.apache.hadoop.yarn.api.records.NodeId containerNodeId) Return a URL that can be used as a link to the logs for a completed attempt.
TaskCommunicatorContext	getContext() Get the TaskCommunicatorContext associated with this instance of the scheduler, which is used to communicate with the rest of the system
String	getInProgressLogsUrl(org.apache.tez.dag.records.TezTaskAttemptID attemptID, org.apache.hadoop.yarn.api.records.NodeId containerNodeId) Return a URL that can be used as a link to the logs for a running attempt.
abstract Object	getMetaInfo() Share meta-information such as host:port information where the Task Communicator may be listening.
void	initialize() An entry point for initialization.
abstract void	onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate) Receive notifications on vertex state changes.
abstract void	registerContainerEnd(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.tez.serviceplugins.api.ContainerEndReason endReason, String diagnostics) Register the end of a container.
abstract void	registerRunningContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, String hostname, int port) Register a new container.
abstract void	registerRunningTaskAttempt(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.tez.runtime.api.impl.TaskSpec taskSpec, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalResources, org.apache.hadoop.security.Credentials credentials, boolean credentialsChanged, int priority) Register a task attempt to execute on a container
void	shutdown() Stop the service.
void	start() An entry point for starting the service.
abstract void	unregisterRunningTaskAttempt(org.apache.tez.dag.records.TezTaskAttemptID taskAttemptID, org.apache.tez.serviceplugins.api.TaskAttemptEndReason endReason, String diagnostics) Register the completion of a task.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TaskCommunicator

public TaskCommunicator(TaskCommunicatorContext taskCommunicatorContext)

Method Detail

getContext

public TaskCommunicatorContext getContext()

Get the TaskCommunicatorContext associated with this instance of the scheduler, which is used to communicate with the rest of the system

Returns:

an instance of TaskCommunicatorContext

initialize

public void initialize()
                throws Exception

An entry point for initialization. Order of service setup. Constructor, initialize(), start() - when starting a service.

Specified by:

initialize in interface org.apache.tez.common.ServicePluginLifecycle

Throws:

Exception

start

public void start()
           throws Exception

An entry point for starting the service. Order of service setup. Constructor, initialize(), start() - when starting a service.

Specified by:

start in interface org.apache.tez.common.ServicePluginLifecycle

Throws:

Exception

shutdown

public void shutdown()
              throws Exception

Stop the service. This could be invoked at any point, when the service is no longer required - including in case of errors.

Specified by:

shutdown in interface org.apache.tez.common.ServicePluginLifecycle

Throws:

Exception

registerRunningContainer

public abstract void registerRunningContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                              String hostname,
                                              int port)
                                       throws org.apache.tez.serviceplugins.api.ServicePluginException

Register a new container.

Parameters:

containerId - the associated containerId

hostname - the hostname on which the container runs

port - the port for the service which is running the container

Throws:

ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

registerContainerEnd

public abstract void registerContainerEnd(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                          org.apache.tez.serviceplugins.api.ContainerEndReason endReason,
                                          @Nullable
                                          String diagnostics)
                                   throws org.apache.tez.serviceplugins.api.ServicePluginException

Register the end of a container. This can be caused by preemption, the container completing successfully, etc.

Parameters:

containerId - the associated containerId

endReason - the end reason for the container completing

diagnostics - diagnostics associated with the container end

Throws:

ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

registerRunningTaskAttempt

public abstract void registerRunningTaskAttempt(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                                org.apache.tez.runtime.api.impl.TaskSpec taskSpec,
                                                Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalResources,
                                                org.apache.hadoop.security.Credentials credentials,
                                                boolean credentialsChanged,
                                                int priority)
                                         throws org.apache.tez.serviceplugins.api.ServicePluginException

Register a task attempt to execute on a container

Parameters:

containerId - the containerId on which this task needs to run

taskSpec - the task specifications for the task to be executed

additionalResources - additional local resources which may be required to run this task on the container

credentials - the credentials required to run this task

credentialsChanged - whether the credentials are different from the original credentials associated with this container

priority - the priority of the task being executed

Throws:

ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

unregisterRunningTaskAttempt

public abstract void unregisterRunningTaskAttempt(org.apache.tez.dag.records.TezTaskAttemptID taskAttemptID,
                                                  org.apache.tez.serviceplugins.api.TaskAttemptEndReason endReason,
                                                  @Nullable
                                                  String diagnostics)
                                           throws org.apache.tez.serviceplugins.api.ServicePluginException

Register the completion of a task. This may be a result of preemption, the container dying, the node dying, the task completing to success

Parameters:

taskAttemptID - the task attempt which has completed / needs to be completed

endReason - the endReason for the task attempt.

diagnostics - diagnostics associated with the task end

Throws:

ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

getAddress

public abstract InetSocketAddress getAddress()
                                      throws org.apache.tez.serviceplugins.api.ServicePluginException

Return the address, if any, that the service listens on

Returns:

the address

Throws:

ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

onVertexStateUpdated

public abstract void onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate)
                                   throws org.apache.tez.serviceplugins.api.ServicePluginException

Receive notifications on vertex state changes.

State changes will be received based on the registration via InputInitializerContext.registerForVertexStateUpdates(String, java.util.Set). Notifications will be received for all registered state changes, and not just for the latest state update. They will be in order in which the state change occurred.

Extensive processing should not be performed via this method call. Instead this should just be used as a notification mechanism.
This method may be invoked concurrently with other invocations into the TaskCommunicator and multi-threading/concurrency implications must be considered.

Parameters:

stateUpdate - an event indicating the name of the vertex, and it's updated state. Additional information may be available for specific events, Look at the type hierarchy for VertexStateUpdate

Throws:

ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

dagComplete

public abstract void dagComplete(int dagIdentifier)
                          throws org.apache.tez.serviceplugins.api.ServicePluginException

Indicates the current running dag is complete. The TaskCommunicatorContext can be used to query information about the current dag during the duration of the dagComplete invocation.

After this, the contents returned from querying the context may change at any point - due to the next dag being submitted.

Parameters:

dagIdentifier - the unique numerical identifier for the DAG in the specified execution context.

Throws:

ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

getMetaInfo

public abstract Object getMetaInfo()
                            throws org.apache.tez.serviceplugins.api.ServicePluginException

Share meta-information such as host:port information where the Task Communicator may be listening. Primarily for use by compatible launchers to learn this information.

Returns:

meta info for the task communicator

Throws:

ServicePluginException - when the service runs into a fatal error which it cannot handle. This will cause the app to shutdown.

getInProgressLogsUrl

public String getInProgressLogsUrl(org.apache.tez.dag.records.TezTaskAttemptID attemptID,
                                   org.apache.hadoop.yarn.api.records.NodeId containerNodeId)

Return a URL that can be used as a link to the logs for a running attempt.

Parameters:

attemptID - Attempt ID for which the log link should be provided

containerNodeId - Node Id on which the attempt is meant to have run

Returns:

URL to logs for the attempt

getCompletedLogsUrl

public String getCompletedLogsUrl(org.apache.tez.dag.records.TezTaskAttemptID attemptID,
                                  org.apache.hadoop.yarn.api.records.NodeId containerNodeId)

Return a URL that can be used as a link to the logs for a completed attempt.

Parameters:

attemptID - Attempt ID for which the log link should be provided

containerNodeId - Node Id on which the attempt is meant to have run

Returns:

URL to logs for the attempt