Class ContainerManagerImpl
java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.service.CompositeService
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
- All Implemented Interfaces:
Closeable,AutoCloseable,org.apache.hadoop.service.Service,org.apache.hadoop.service.ServiceStateChangeListener,org.apache.hadoop.yarn.api.ContainerManagementProtocol,org.apache.hadoop.yarn.event.EventHandler<ContainerManagerEvent>,ContainerManager
public class ContainerManagerImpl
extends org.apache.hadoop.service.CompositeService
implements ContainerManager
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService
org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHookNested classes/interfaces inherited from interface org.apache.hadoop.service.Service
org.apache.hadoop.service.Service.STATE -
Field Summary
FieldsModifier and TypeFieldDescriptionprotected booleanprotected final Contextprotected LocalDirsHandlerServicestatic final Stringprotected final NodeStatusUpdaterFields inherited from class org.apache.hadoop.service.CompositeService
STOP_ONLY_STARTED_SERVICES -
Constructor Summary
ConstructorsConstructorDescriptionContainerManagerImpl(Context context, ContainerExecutor exec, DeletionService deletionContext, NodeStatusUpdater nodeStatusUpdater, NodeManagerMetrics metrics, LocalDirsHandlerService dirsHandler) -
Method Summary
Modifier and TypeMethodDescriptionprotected voidauthorizeGetAndStopContainerRequest(org.apache.hadoop.yarn.api.records.ContainerId containerId, Container container, boolean stopRequest, org.apache.hadoop.yarn.security.NMTokenIdentifier identifier, String remoteUser) protected voidauthorizeStartAndResourceIncreaseRequest(org.apache.hadoop.yarn.security.NMTokenIdentifier nmTokenIdentifier, org.apache.hadoop.yarn.security.ContainerTokenIdentifier containerTokenIdentifier, boolean startRequest) protected voidauthorizeUser(org.apache.hadoop.security.UserGroupInformation remoteUgi, org.apache.hadoop.yarn.security.NMTokenIdentifier nmTokenIdentifier) voidvoidorg.apache.hadoop.yarn.api.protocolrecords.CommitResponsecommitLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId) Commit last reInitialization after which no rollback will be possible.protected voidcreateAMRMProxyService(org.apache.hadoop.conf.Configuration conf) protected org.apache.hadoop.yarn.event.EventHandler<ApplicationEvent>protected org.apache.hadoop.yarn.event.AsyncDispatcherprotected ContainerSchedulercreateContainerScheduler(Context cntxt) protected AbstractContainersLaunchercreateContainersLauncher(Context ctxt, ContainerExecutor exec) protected ContainersMonitorprotected LogHandlercreateLogHandler(org.apache.hadoop.conf.Configuration conf, Context context, DeletionService deletionService) protected NMTimelinePublisherprotected ResourceLocalizationServicecreateResourceLocalizationService(ContainerExecutor exec, DeletionService deletionContext, Context nmContext, NodeManagerMetrics nmMetrics) protected SharedCacheUploadServiceorg.apache.hadoop.yarn.api.protocolrecords.GetContainerStatusesResponsegetContainerStatuses(org.apache.hadoop.yarn.api.protocolrecords.GetContainerStatusesRequest request) Get a list of container statuses running on this NodeManagerprotected org.apache.hadoop.yarn.api.records.ContainerStatusgetContainerStatusInternal(org.apache.hadoop.yarn.api.records.ContainerId containerID, org.apache.hadoop.yarn.security.NMTokenIdentifier nmTokenIdentifier, String remoteUser) org.apache.hadoop.yarn.event.AsyncDispatcherorg.apache.hadoop.yarn.api.protocolrecords.GetLocalizationStatusesResponsegetLocalizationStatuses(org.apache.hadoop.yarn.api.protocolrecords.GetLocalizationStatusesRequest request) org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatusprotected org.apache.hadoop.security.UserGroupInformationvoidhandle(ContainerManagerEvent event) voidorg.apache.hadoop.yarn.api.protocolrecords.IncreaseContainersResourceResponseincreaseContainersResource(org.apache.hadoop.yarn.api.protocolrecords.IncreaseContainersResourceRequest requests) Deprecated.protected booleanorg.apache.hadoop.yarn.api.protocolrecords.ResourceLocalizationResponselocalize(org.apache.hadoop.yarn.api.protocolrecords.ResourceLocalizationRequest request) protected voidrecoverActiveContainer(Application app, org.apache.hadoop.yarn.api.records.ContainerLaunchContext launchContext, org.apache.hadoop.yarn.security.ContainerTokenIdentifier token, NMStateStoreService.RecoveredContainerState rcs) Recover a running container.org.apache.hadoop.yarn.api.protocolrecords.ReInitializeContainerResponsereInitializeContainer(org.apache.hadoop.yarn.api.protocolrecords.ReInitializeContainerRequest request) voidreInitializeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.ContainerLaunchContext reInitLaunchContext, boolean autoCommit) ReInitialize a container using a new Launch Context.org.apache.hadoop.yarn.api.protocolrecords.RestartContainerResponserestartContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) org.apache.hadoop.yarn.api.protocolrecords.RollbackResponserollbackLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId) Rollback the last reInitialization, if possible.protected org.apache.hadoop.yarn.security.NMTokenIdentifierselectNMTokenIdentifier(org.apache.hadoop.security.UserGroupInformation remoteUgi) voidserviceInit(org.apache.hadoop.conf.Configuration conf) protected voidvoidprotected voidsetAMRMProxyService(AMRMProxyService amrmProxyService) org.apache.hadoop.yarn.api.protocolrecords.SignalContainerResponsesignalToContainer(org.apache.hadoop.yarn.api.protocolrecords.SignalContainerRequest request) protected voidstartContainerInternal(org.apache.hadoop.yarn.security.ContainerTokenIdentifier containerTokenIdentifier, org.apache.hadoop.yarn.api.protocolrecords.StartContainerRequest request, String remoteUser) org.apache.hadoop.yarn.api.protocolrecords.StartContainersResponsestartContainers(org.apache.hadoop.yarn.api.protocolrecords.StartContainersRequest requests) Start a list of containers on this NodeManager.voidstateChanged(org.apache.hadoop.service.Service service) protected voidstopContainerInternal(org.apache.hadoop.yarn.api.records.ContainerId containerID, String remoteUser) org.apache.hadoop.yarn.api.protocolrecords.StopContainersResponsestopContainers(org.apache.hadoop.yarn.api.protocolrecords.StopContainersRequest requests) Stop a list of containers running on this NodeManager.org.apache.hadoop.yarn.api.protocolrecords.ContainerUpdateResponseupdateContainer(org.apache.hadoop.yarn.api.protocolrecords.ContainerUpdateRequest request) Update resource of a list of containers on this NodeManager.protected voidupdateNMTokenIdentifier(org.apache.hadoop.yarn.security.NMTokenIdentifier nmTokenIdentifier) voidupdateQueuingLimit(org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit queuingLimit) protected org.apache.hadoop.yarn.security.ContainerTokenIdentifierverifyAndGetContainerTokenIdentifier(org.apache.hadoop.yarn.api.records.Token token, org.apache.hadoop.yarn.security.ContainerTokenIdentifier containerTokenIdentifier) Methods inherited from class org.apache.hadoop.service.CompositeService
addIfService, addService, getServices, removeServiceMethods inherited from class org.apache.hadoop.service.AbstractService
close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop
-
Field Details
-
INVALID_NMTOKEN_MSG
- See Also:
-
context
-
nodeStatusUpdater
-
dirsHandler
-
amrmProxyEnabled
protected boolean amrmProxyEnabled
-
-
Constructor Details
-
ContainerManagerImpl
public ContainerManagerImpl(Context context, ContainerExecutor exec, DeletionService deletionContext, NodeStatusUpdater nodeStatusUpdater, NodeManagerMetrics metrics, LocalDirsHandlerService dirsHandler)
-
-
Method Details
-
serviceInit
- Overrides:
serviceInitin classorg.apache.hadoop.service.CompositeService- Throws:
Exception
-
createContainerManagerDispatcher
protected org.apache.hadoop.yarn.event.AsyncDispatcher createContainerManagerDispatcher() -
createAMRMProxyService
protected void createAMRMProxyService(org.apache.hadoop.conf.Configuration conf) -
createContainerScheduler
-
createContainersMonitor
-
recoverActiveContainer
protected void recoverActiveContainer(Application app, org.apache.hadoop.yarn.api.records.ContainerLaunchContext launchContext, org.apache.hadoop.yarn.security.ContainerTokenIdentifier token, NMStateStoreService.RecoveredContainerState rcs) throws IOException Recover a running container.- Throws:
IOException
-
createLogHandler
protected LogHandler createLogHandler(org.apache.hadoop.conf.Configuration conf, Context context, DeletionService deletionService) -
getContainersMonitor
- Specified by:
getContainersMonitorin interfaceContainerManager
-
createResourceLocalizationService
protected ResourceLocalizationService createResourceLocalizationService(ContainerExecutor exec, DeletionService deletionContext, Context nmContext, NodeManagerMetrics nmMetrics) -
createNMTimelinePublisher
-
createContainersLauncher
-
createApplicationEventDispatcher
protected org.apache.hadoop.yarn.event.EventHandler<ApplicationEvent> createApplicationEventDispatcher() -
serviceStart
- Overrides:
serviceStartin classorg.apache.hadoop.service.CompositeService- Throws:
Exception
-
serviceStop
- Overrides:
serviceStopin classorg.apache.hadoop.service.CompositeService- Throws:
Exception
-
cleanUpApplicationsOnNMShutDown
public void cleanUpApplicationsOnNMShutDown() -
cleanupContainersOnNMResync
public void cleanupContainersOnNMResync() -
getRemoteUgi
protected org.apache.hadoop.security.UserGroupInformation getRemoteUgi() throws org.apache.hadoop.yarn.exceptions.YarnException- Throws:
org.apache.hadoop.yarn.exceptions.YarnException
-
selectNMTokenIdentifier
@Private @VisibleForTesting protected org.apache.hadoop.yarn.security.NMTokenIdentifier selectNMTokenIdentifier(org.apache.hadoop.security.UserGroupInformation remoteUgi) -
authorizeUser
protected void authorizeUser(org.apache.hadoop.security.UserGroupInformation remoteUgi, org.apache.hadoop.yarn.security.NMTokenIdentifier nmTokenIdentifier) throws org.apache.hadoop.yarn.exceptions.YarnException - Throws:
org.apache.hadoop.yarn.exceptions.YarnException
-
authorizeStartAndResourceIncreaseRequest
@Private @VisibleForTesting protected void authorizeStartAndResourceIncreaseRequest(org.apache.hadoop.yarn.security.NMTokenIdentifier nmTokenIdentifier, org.apache.hadoop.yarn.security.ContainerTokenIdentifier containerTokenIdentifier, boolean startRequest) throws org.apache.hadoop.yarn.exceptions.YarnException - Parameters:
containerTokenIdentifier- of the container whose resource is to be started or increased- Throws:
org.apache.hadoop.yarn.exceptions.YarnException
-
startContainers
public org.apache.hadoop.yarn.api.protocolrecords.StartContainersResponse startContainers(org.apache.hadoop.yarn.api.protocolrecords.StartContainersRequest requests) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException Start a list of containers on this NodeManager.- Specified by:
startContainersin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
startContainerInternal
protected void startContainerInternal(org.apache.hadoop.yarn.security.ContainerTokenIdentifier containerTokenIdentifier, org.apache.hadoop.yarn.api.protocolrecords.StartContainerRequest request, String remoteUser) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException - Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
verifyAndGetContainerTokenIdentifier
protected org.apache.hadoop.yarn.security.ContainerTokenIdentifier verifyAndGetContainerTokenIdentifier(org.apache.hadoop.yarn.api.records.Token token, org.apache.hadoop.yarn.security.ContainerTokenIdentifier containerTokenIdentifier) throws org.apache.hadoop.yarn.exceptions.YarnException, org.apache.hadoop.security.token.SecretManager.InvalidToken - Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionorg.apache.hadoop.security.token.SecretManager.InvalidToken
-
increaseContainersResource
@Deprecated public org.apache.hadoop.yarn.api.protocolrecords.IncreaseContainersResourceResponse increaseContainersResource(org.apache.hadoop.yarn.api.protocolrecords.IncreaseContainersResourceRequest requests) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException Deprecated.Increase resource of a list of containers on this NodeManager.- Specified by:
increaseContainersResourcein interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
updateContainer
public org.apache.hadoop.yarn.api.protocolrecords.ContainerUpdateResponse updateContainer(org.apache.hadoop.yarn.api.protocolrecords.ContainerUpdateRequest request) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException Update resource of a list of containers on this NodeManager.- Specified by:
updateContainerin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
updateNMTokenIdentifier
@Private @VisibleForTesting protected void updateNMTokenIdentifier(org.apache.hadoop.yarn.security.NMTokenIdentifier nmTokenIdentifier) throws org.apache.hadoop.security.token.SecretManager.InvalidToken - Throws:
org.apache.hadoop.security.token.SecretManager.InvalidToken
-
stopContainers
public org.apache.hadoop.yarn.api.protocolrecords.StopContainersResponse stopContainers(org.apache.hadoop.yarn.api.protocolrecords.StopContainersRequest requests) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException Stop a list of containers running on this NodeManager.- Specified by:
stopContainersin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
stopContainerInternal
protected void stopContainerInternal(org.apache.hadoop.yarn.api.records.ContainerId containerID, String remoteUser) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException - Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
getContainerStatuses
public org.apache.hadoop.yarn.api.protocolrecords.GetContainerStatusesResponse getContainerStatuses(org.apache.hadoop.yarn.api.protocolrecords.GetContainerStatusesRequest request) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException Get a list of container statuses running on this NodeManager- Specified by:
getContainerStatusesin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
getContainerStatusInternal
protected org.apache.hadoop.yarn.api.records.ContainerStatus getContainerStatusInternal(org.apache.hadoop.yarn.api.records.ContainerId containerID, org.apache.hadoop.yarn.security.NMTokenIdentifier nmTokenIdentifier, String remoteUser) throws org.apache.hadoop.yarn.exceptions.YarnException - Throws:
org.apache.hadoop.yarn.exceptions.YarnException
-
authorizeGetAndStopContainerRequest
@Private @VisibleForTesting protected void authorizeGetAndStopContainerRequest(org.apache.hadoop.yarn.api.records.ContainerId containerId, Container container, boolean stopRequest, org.apache.hadoop.yarn.security.NMTokenIdentifier identifier, String remoteUser) throws org.apache.hadoop.yarn.exceptions.YarnException - Throws:
org.apache.hadoop.yarn.exceptions.YarnException
-
handle
- Specified by:
handlein interfaceorg.apache.hadoop.yarn.event.EventHandler<ContainerManagerEvent>
-
stateChanged
public void stateChanged(org.apache.hadoop.service.Service service) - Specified by:
stateChangedin interfaceorg.apache.hadoop.service.ServiceStateChangeListener
-
getContext
-
getAuxServiceMetaData
-
getAMRMProxyService
-
setAMRMProxyService
-
isServiceStopped
protected boolean isServiceStopped() -
getOpportunisticContainersStatus
public org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus getOpportunisticContainersStatus()- Specified by:
getOpportunisticContainersStatusin interfaceContainerManager
-
updateQueuingLimit
public void updateQueuingLimit(org.apache.hadoop.yarn.server.api.records.ContainerQueuingLimit queuingLimit) - Specified by:
updateQueuingLimitin interfaceContainerManager
-
signalToContainer
public org.apache.hadoop.yarn.api.protocolrecords.SignalContainerResponse signalToContainer(org.apache.hadoop.yarn.api.protocolrecords.SignalContainerRequest request) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException - Specified by:
signalToContainerin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
localize
public org.apache.hadoop.yarn.api.protocolrecords.ResourceLocalizationResponse localize(org.apache.hadoop.yarn.api.protocolrecords.ResourceLocalizationRequest request) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException - Specified by:
localizein interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
reInitializeContainer
public org.apache.hadoop.yarn.api.protocolrecords.ReInitializeContainerResponse reInitializeContainer(org.apache.hadoop.yarn.api.protocolrecords.ReInitializeContainerRequest request) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException - Specified by:
reInitializeContainerin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
restartContainer
public org.apache.hadoop.yarn.api.protocolrecords.RestartContainerResponse restartContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException - Specified by:
restartContainerin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
reInitializeContainer
public void reInitializeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.ContainerLaunchContext reInitLaunchContext, boolean autoCommit) throws org.apache.hadoop.yarn.exceptions.YarnException ReInitialize a container using a new Launch Context. If the retryFailureContext is not provided, The container is terminated on Failure. NOTE: Auto-Commit is true by default. This also means that the rollback context is purged as soon as the command to start the new process is sent. (The Container moves to RUNNING state)- Parameters:
containerId- Container Id.autoCommit- Auto Commit flag.reInitLaunchContext- Target Launch Context.- Throws:
org.apache.hadoop.yarn.exceptions.YarnException- YARN Exception.
-
rollbackLastReInitialization
public org.apache.hadoop.yarn.api.protocolrecords.RollbackResponse rollbackLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws org.apache.hadoop.yarn.exceptions.YarnException Rollback the last reInitialization, if possible.- Specified by:
rollbackLastReInitializationin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Parameters:
containerId- Container ID.- Returns:
- Rollback Response.
- Throws:
org.apache.hadoop.yarn.exceptions.YarnException- YARN Exception.
-
commitLastReInitialization
public org.apache.hadoop.yarn.api.protocolrecords.CommitResponse commitLastReInitialization(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws org.apache.hadoop.yarn.exceptions.YarnException Commit last reInitialization after which no rollback will be possible.- Specified by:
commitLastReInitializationin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Parameters:
containerId- Container ID.- Returns:
- Commit Response.
- Throws:
org.apache.hadoop.yarn.exceptions.YarnException- YARN Exception.
-
getContainerScheduler
- Specified by:
getContainerSchedulerin interfaceContainerManager
-
handleCredentialUpdate
public void handleCredentialUpdate()- Specified by:
handleCredentialUpdatein interfaceContainerManager
-
getLocalizationStatuses
public org.apache.hadoop.yarn.api.protocolrecords.GetLocalizationStatusesResponse getLocalizationStatuses(org.apache.hadoop.yarn.api.protocolrecords.GetLocalizationStatusesRequest request) throws org.apache.hadoop.yarn.exceptions.YarnException, IOException - Specified by:
getLocalizationStatusesin interfaceorg.apache.hadoop.yarn.api.ContainerManagementProtocol- Throws:
org.apache.hadoop.yarn.exceptions.YarnExceptionIOException
-
getResourceLocalizationService
- Specified by:
getResourceLocalizationServicein interfaceContainerManager
-
getDispatcher
public org.apache.hadoop.yarn.event.AsyncDispatcher getDispatcher()
-