Package org.apache.hadoop.yarn.server.nodemanager.recovery

Class NMStateStoreService

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service
Direct Known Subclasses:
NMLeveldbStateStoreService, NMNullStateStoreService
@Private @Unstable public abstract class NMStateStoreService extends org.apache.hadoop.service.AbstractService

  • Constructor Details

    • NMStateStoreService

      public NMStateStoreService(String name)

  • Method Details

    • getNodeStatusUpdater

      protected NodeStatusUpdater getNodeStatusUpdater()

    • setNodeStatusUpdater

      public void setNodeStatusUpdater(NodeStatusUpdater nodeStatusUpdater)

    • serviceInit

      public void serviceInit(org.apache.hadoop.conf.Configuration conf) throws IOException
      Initialize the state storage
      Overrides:
      serviceInit in class org.apache.hadoop.service.AbstractService
      Throws:
      IOException

    • serviceStart

      public void serviceStart() throws IOException
      Start the state storage for use
      Overrides:
      serviceStart in class org.apache.hadoop.service.AbstractService
      Throws:
      IOException

    • serviceStop

      public void serviceStop() throws IOException
      Shutdown the state storage.
      Overrides:
      serviceStop in class org.apache.hadoop.service.AbstractService
      Throws:
      IOException

    • canRecover

      public boolean canRecover()

    • isNewlyCreated

      public boolean isNewlyCreated()

    • loadApplicationsState

      public abstract NMStateStoreService.RecoveredApplicationsState loadApplicationsState() throws IOException
      Load the state of applications.
      Returns:
      recovered state for applications.
      Throws:
      IOException - IO Exception.

    • storeApplication

      public abstract void storeApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.yarn.proto.YarnServerNodemanagerRecoveryProtos.ContainerManagerApplicationProto p) throws IOException
      Record the start of an application
      Parameters:
      appId - the application ID
      p - state to store for the application
      Throws:
      IOException

    • removeApplication

      public abstract void removeApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
      Remove records corresponding to an application
      Parameters:
      appId - the application ID
      Throws:
      IOException

    • getContainerStateIterator

      public abstract RecoveryIterator<NMStateStoreService.RecoveredContainerState> getContainerStateIterator() throws IOException
      get the Recovered Container State Iterator
      Returns:
      recovery iterator
      Throws:
      IOException

    • storeContainer

      public abstract void storeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, int containerVersion, long startTime, org.apache.hadoop.yarn.api.protocolrecords.StartContainerRequest startRequest) throws IOException
      Record a container start request
      Parameters:
      containerId - the container ID
      containerVersion - the container Version
      startTime - container start time
      startRequest - the container start request
      Throws:
      IOException

    • storeContainerQueued

      public abstract void storeContainerQueued(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws IOException
      Record that a container has been queued at the NM
      Parameters:
      containerId - the container ID
      Throws:
      IOException

    • storeContainerPaused

      public abstract void storeContainerPaused(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws IOException
      Record that a container has been paused at the NM.
      Parameters:
      containerId - the container ID.
      Throws:
      IOException - IO Exception.

    • removeContainerPaused

      public abstract void removeContainerPaused(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws IOException
      Record that a container has been resumed at the NM by removing the fact that it has be paused.
      Parameters:
      containerId - the container ID.
      Throws:
      IOException - IO Exception.

    • storeContainerLaunched

      public abstract void storeContainerLaunched(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws IOException
      Record that a container has been launched
      Parameters:
      containerId - the container ID
      Throws:
      IOException

    • storeContainerUpdateToken

      public abstract void storeContainerUpdateToken(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.security.ContainerTokenIdentifier containerTokenIdentifier) throws IOException
      Record that a container has been updated
      Parameters:
      containerId - the container ID
      containerTokenIdentifier - container token identifier
      Throws:
      IOException

    • storeContainerCompleted

      public abstract void storeContainerCompleted(org.apache.hadoop.yarn.api.records.ContainerId containerId, int exitCode) throws IOException
      Record that a container has completed
      Parameters:
      containerId - the container ID
      exitCode - the exit code from the container
      Throws:
      IOException

    • storeContainerKilled

      public abstract void storeContainerKilled(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws IOException
      Record a request to kill a container
      Parameters:
      containerId - the container ID
      Throws:
      IOException

    • storeContainerDiagnostics

      public abstract void storeContainerDiagnostics(org.apache.hadoop.yarn.api.records.ContainerId containerId, StringBuilder diagnostics) throws IOException
      Record diagnostics for a container
      Parameters:
      containerId - the container ID
      diagnostics - the container diagnostics
      Throws:
      IOException

    • storeContainerRemainingRetryAttempts

      public abstract void storeContainerRemainingRetryAttempts(org.apache.hadoop.yarn.api.records.ContainerId containerId, int remainingRetryAttempts) throws IOException
      Record remaining retry attempts for a container.
      Parameters:
      containerId - the container ID
      remainingRetryAttempts - the remain retry times when container fails to run
      Throws:
      IOException

    • storeContainerRestartTimes

      public abstract void storeContainerRestartTimes(org.apache.hadoop.yarn.api.records.ContainerId containerId, List<Long> restartTimes) throws IOException
      Record restart times for a container.
      Parameters:
      containerId -
      restartTimes -
      Throws:
      IOException

    • storeContainerWorkDir

      public abstract void storeContainerWorkDir(org.apache.hadoop.yarn.api.records.ContainerId containerId, String workDir) throws IOException
      Record working directory for a container.
      Parameters:
      containerId - the container ID
      workDir - the working directory
      Throws:
      IOException

    • storeContainerLogDir

      public abstract void storeContainerLogDir(org.apache.hadoop.yarn.api.records.ContainerId containerId, String logDir) throws IOException
      Record log directory for a container.
      Parameters:
      containerId - the container ID
      logDir - the log directory
      Throws:
      IOException

    • removeContainer

      public abstract void removeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws IOException
      Remove records corresponding to a container
      Parameters:
      containerId - the container ID
      Throws:
      IOException

    • loadLocalizationState

      public abstract NMStateStoreService.RecoveredLocalizationState loadLocalizationState() throws IOException
      Load the state of localized resources
      Returns:
      recovered localized resource state
      Throws:
      IOException

    • startResourceLocalization

      public abstract void startResourceLocalization(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.yarn.proto.YarnProtos.LocalResourceProto proto, org.apache.hadoop.fs.Path localPath) throws IOException
      Record the start of localization for a resource
      Parameters:
      user - the username or null if the resource is public
      appId - the application ID if the resource is app-specific or null
      proto - the resource request
      localPath - local filesystem path where the resource will be stored
      Throws:
      IOException

    • finishResourceLocalization

      public abstract void finishResourceLocalization(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.yarn.proto.YarnServerNodemanagerRecoveryProtos.LocalizedResourceProto proto) throws IOException
      Record the completion of a resource localization
      Parameters:
      user - the username or null if the resource is public
      appId - the application ID if the resource is app-specific or null
      proto - the serialized localized resource
      Throws:
      IOException

    • removeLocalizedResource

      public abstract void removeLocalizedResource(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.fs.Path localPath) throws IOException
      Remove records related to a resource localization
      Parameters:
      user - the username or null if the resource is public
      appId - the application ID if the resource is app-specific or null
      localPath - local filesystem path where the resource will be stored
      Throws:
      IOException

    • loadDeletionServiceState

      public abstract NMStateStoreService.RecoveredDeletionServiceState loadDeletionServiceState() throws IOException
      Load the state of the deletion service
      Returns:
      recovered deletion service state
      Throws:
      IOException

    • storeDeletionTask

      public abstract void storeDeletionTask(int taskId, org.apache.hadoop.yarn.proto.YarnServerNodemanagerRecoveryProtos.DeletionServiceDeleteTaskProto taskProto) throws IOException
      Record a deletion task
      Parameters:
      taskId - the deletion task ID
      taskProto - the deletion task protobuf
      Throws:
      IOException

    • removeDeletionTask

      public abstract void removeDeletionTask(int taskId) throws IOException
      Remove records corresponding to a deletion task
      Parameters:
      taskId - the deletion task ID
      Throws:
      IOException

    • loadNMTokensState

      public abstract NMStateStoreService.RecoveredNMTokensState loadNMTokensState() throws IOException
      Load the state of NM tokens
      Returns:
      recovered state of NM tokens
      Throws:
      IOException

    • storeNMTokenCurrentMasterKey

      public abstract void storeNMTokenCurrentMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) throws IOException
      Record the current NM token master key
      Parameters:
      key - the master key
      Throws:
      IOException

    • storeNMTokenPreviousMasterKey

      public abstract void storeNMTokenPreviousMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) throws IOException
      Record the previous NM token master key
      Parameters:
      key - the previous master key
      Throws:
      IOException

    • storeNMTokenApplicationMasterKey

      public abstract void storeNMTokenApplicationMasterKey(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt, org.apache.hadoop.yarn.server.api.records.MasterKey key) throws IOException
      Record a master key corresponding to an application
      Parameters:
      attempt - the application attempt ID
      key - the master key
      Throws:
      IOException

    • removeNMTokenApplicationMasterKey

      public abstract void removeNMTokenApplicationMasterKey(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt) throws IOException
      Remove a master key corresponding to an application
      Parameters:
      attempt - the application attempt ID
      Throws:
      IOException

    • loadContainerTokensState

      public abstract NMStateStoreService.RecoveredContainerTokensState loadContainerTokensState() throws IOException
      Load the state of container tokens
      Returns:
      recovered state of container tokens
      Throws:
      IOException

    • storeContainerTokenCurrentMasterKey

      public abstract void storeContainerTokenCurrentMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) throws IOException
      Record the current container token master key
      Parameters:
      key - the master key
      Throws:
      IOException

    • storeContainerTokenPreviousMasterKey

      public abstract void storeContainerTokenPreviousMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) throws IOException
      Record the previous container token master key
      Parameters:
      key - the previous master key
      Throws:
      IOException

    • storeContainerToken

      public abstract void storeContainerToken(org.apache.hadoop.yarn.api.records.ContainerId containerId, Long expirationTime) throws IOException
      Record the expiration time for a container token
      Parameters:
      containerId - the container ID
      expirationTime - the container token expiration time
      Throws:
      IOException

    • removeContainerToken

      public abstract void removeContainerToken(org.apache.hadoop.yarn.api.records.ContainerId containerId) throws IOException
      Remove records for a container token
      Parameters:
      containerId - the container ID
      Throws:
      IOException

    • loadLogDeleterState

      public abstract NMStateStoreService.RecoveredLogDeleterState loadLogDeleterState() throws IOException
      Load the state of log deleters
      Returns:
      recovered log deleter state
      Throws:
      IOException

    • storeLogDeleter

      public abstract void storeLogDeleter(org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.yarn.proto.YarnServerNodemanagerRecoveryProtos.LogDeleterProto proto) throws IOException
      Store the state of a log deleter
      Parameters:
      appId - the application ID for the log deleter
      proto - the serialized state of the log deleter
      Throws:
      IOException

    • removeLogDeleter

      public abstract void removeLogDeleter(org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
      Remove the state of a log deleter
      Parameters:
      appId - the application ID for the log deleter
      Throws:
      IOException

    • loadAMRMProxyState

      public abstract NMStateStoreService.RecoveredAMRMProxyState loadAMRMProxyState() throws IOException
      Load the state of AMRMProxy.
      Returns:
      recovered state of AMRMProxy
      Throws:
      IOException - if fails

    • storeAMRMProxyCurrentMasterKey

      public abstract void storeAMRMProxyCurrentMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) throws IOException
      Record the current AMRMProxyTokenSecretManager master key.
      Parameters:
      key - the current master key
      Throws:
      IOException - if fails

    • storeAMRMProxyNextMasterKey

      public abstract void storeAMRMProxyNextMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) throws IOException
      Record the next AMRMProxyTokenSecretManager master key.
      Parameters:
      key - the next master key
      Throws:
      IOException - if fails

    • storeAMRMProxyAppContextEntry

      public abstract void storeAMRMProxyAppContextEntry(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt, String key, byte[] data) throws IOException
      Add a context entry for an application attempt in AMRMProxyService.
      Parameters:
      attempt - app attempt ID
      key - key string
      data - state data to store
      Throws:
      IOException - if fails

    • removeAMRMProxyAppContextEntry

      public abstract void removeAMRMProxyAppContextEntry(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt, String key) throws IOException
      Remove a context entry for an application attempt in AMRMProxyService.
      Parameters:
      attempt - attempt ID
      key - key string
      Throws:
      IOException - if fails

    • removeAMRMProxyAppContext

      public abstract void removeAMRMProxyAppContext(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt) throws IOException
      Remove the entire context map for an application attempt in AMRMProxyService.
      Parameters:
      attempt - attempt ID
      Throws:
      IOException - if fails

    • storeAssignedResources

      public abstract void storeAssignedResources(Container container, String resourceType, List<Serializable> assignedResources) throws IOException
      Store the assigned resources to a container.
      Parameters:
      container - NMContainer
      resourceType - Resource Type
      assignedResources - Assigned resources
      Throws:
      IOException - if fails

    • releaseAssignedResources

      public void releaseAssignedResources(org.apache.hadoop.yarn.api.records.ContainerId containerId, String resourceType) throws IOException
      Delete the assigned resources of a container of specific resourceType.
      Parameters:
      containerId - Container Id
      resourceType - resource Type
      Throws:
      IOException - while releasing resources

    • initStorage

      protected abstract void initStorage(org.apache.hadoop.conf.Configuration conf) throws IOException
      Throws:
      IOException

    • startStorage

      protected abstract void startStorage() throws IOException
      Throws:
      IOException

    • closeStorage

      protected abstract void closeStorage() throws IOException
      Throws:
      IOException

    • updateContainerResourceMapping

      protected void updateContainerResourceMapping(Container container, String resourceType, List<Serializable> assignedResources)