Package org.apache.hadoop.yarn.server.nodemanager

Class DefaultContainerExecutor

java.lang.Object
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable
Direct Known Subclasses:
WindowsSecureContainerExecutor
public class DefaultContainerExecutor extends ContainerExecutor
The DefaultContainerExecuter class offers generic container execution services. Process execution is handled in a platform-independent way via ProcessBuilder.

  • Field Details

    • lfs

      protected final org.apache.hadoop.fs.FileContext lfs
      A FileContext for the local file system.

  • Constructor Details

    • DefaultContainerExecutor

      @VisibleForTesting public DefaultContainerExecutor()
      Default constructor for use in testing.

  • Method Details

    • copyFile

      protected void copyFile(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst, String owner) throws IOException
      Copy a file using the lfs FileContext.
      Parameters:
      src - the file to copy
      dst - where to copy the file
      owner - the owner of the new copy. Used only in secure Windows clusters
      Throws:
      IOException - when the copy fails
      See Also:

    • setScriptExecutable

      protected void setScriptExecutable(org.apache.hadoop.fs.Path script, String owner) throws IOException
      Make a file executable using the lfs FileContext.
      Parameters:
      script - the path to make executable
      owner - the new owner for the file. Used only in secure Windows clusters
      Throws:
      IOException - when the change mode operation fails
      See Also:

    • init

      public void init(Context nmContext) throws IOException
      Description copied from class: ContainerExecutor
      Run the executor initialization steps. Verify that the necessary configs and permissions are in place.
      Specified by:
      init in class ContainerExecutor
      Parameters:
      nmContext - Context of NM
      Throws:
      IOException - if initialization fails

    • startLocalizer

      public void startLocalizer(LocalizerStartContext ctx) throws IOException, InterruptedException
      Description copied from class: ContainerExecutor
      Prepare the environment for containers in this application to execute. 
       For $x in local.dirs
   create $x/$user/$appId
 Copy $nmLocal/appTokens -> $N/$user/$appId
 For $rsrc in private resources
   Copy $rsrc -> $N/$user/filecache/[idef]
 For $rsrc in job resources
   Copy $rsrc -> $N/$user/$appId/filecache/idef
      Specified by:
      startLocalizer in class ContainerExecutor
      Parameters:
      ctx - LocalizerStartContext that encapsulates necessary information for starting a localizer.
      Throws:
      IOException - for most application init failures
      InterruptedException - if application init thread is halted by NM

    • createContainerLocalizer

      @Private @VisibleForTesting protected ContainerLocalizer createContainerLocalizer(String user, String appId, String locId, String tokenFileName, List<String> localDirs, org.apache.hadoop.fs.FileContext localizerFc) throws IOException
      Create a new ContainerLocalizer instance.
      Parameters:
      user - the user who owns the job for which the localization is being run
      appId - the ID of the application for which the localization is being run
      locId - the ID of the container for which the localization is being run
      localDirs - a list of directories to use as destinations for the localization
      localizerFc - the FileContext to use when localizing files
      Returns:
      the new ContainerLocalizer instance
      Throws:
      IOException - if user or locId is null or if the container localizer has an initialization failure

    • launchContainer

      public int launchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
      Description copied from class: ContainerExecutor
      Launch the container on the node. This is a blocking call and returns only when the container exits.
      Specified by:
      launchContainer in class ContainerExecutor
      Parameters:
      ctx - Encapsulates information necessary for launching containers.
      Returns:
      the return status of the launch
      Throws:
      IOException - if the container launch fails
      org.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was found

    • relaunchContainer

      public int relaunchContainer(ContainerStartContext ctx) throws IOException, org.apache.hadoop.yarn.exceptions.ConfigurationException
      Description copied from class: ContainerExecutor
      Relaunch the container on the node. This is a blocking call and returns only when the container exits.
      Specified by:
      relaunchContainer in class ContainerExecutor
      Parameters:
      ctx - Encapsulates information necessary for relaunching containers.
      Returns:
      the return status of the relaunch
      Throws:
      IOException - if the container relaunch fails
      org.apache.hadoop.yarn.exceptions.ConfigurationException - if config error was found

    • buildCommandExecutor

      protected org.apache.hadoop.util.Shell.CommandExecutor buildCommandExecutor(String wrapperScriptPath, String containerIdStr, String user, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.yarn.api.records.Resource resource, File workDir, Map<String,String> environment, String[] numaCommands)
      Create a new Shell.ShellCommandExecutor using the parameters.
      Parameters:
      wrapperScriptPath - the path to the script to execute
      containerIdStr - the container ID
      user - the application owner's username
      pidFile - the path to the container's PID file
      resource - this parameter controls memory and CPU limits.
      workDir - If not-null, specifies the directory which should be set as the current working directory for the command. If null, the current working directory is not modified.
      environment - the container environment
      numaCommands - list of prefix numa commands
      Returns:
      the new Shell.ShellCommandExecutor
      See Also:
      • Shell.ShellCommandExecutor

    • getLocalWrapperScriptBuilder

      protected DefaultContainerExecutor.LocalWrapperScriptBuilder getLocalWrapperScriptBuilder(String containerIdStr, org.apache.hadoop.fs.Path containerWorkDir)
      Create a DefaultContainerExecutor.LocalWrapperScriptBuilder for the given container ID and path that is appropriate to the current platform.
      Parameters:
      containerIdStr - the container ID
      containerWorkDir - the container's working directory
      Returns:
      a new DefaultContainerExecutor.LocalWrapperScriptBuilder

    • signalContainer

      public boolean signalContainer(ContainerSignalContext ctx) throws IOException
      Description copied from class: ContainerExecutor
      Signal container with the specified signal.
      Specified by:
      signalContainer in class ContainerExecutor
      Parameters:
      ctx - Encapsulates information necessary for signaling containers.
      Returns:
      returns true if the operation succeeded
      Throws:
      IOException - if signaling the container fails

    • reapContainer

      public boolean reapContainer(ContainerReapContext ctx)
      No-op for reaping containers within the DefaultContainerExecutor.
      Specified by:
      reapContainer in class ContainerExecutor
      Parameters:
      ctx - Encapsulates information necessary for reaping containers.
      Returns:
      true given no operations are needed.

    • isContainerAlive

      public boolean isContainerAlive(ContainerLivenessContext ctx) throws IOException
      Description copied from class: ContainerExecutor
      Check if a container is alive.
      Specified by:
      isContainerAlive in class ContainerExecutor
      Parameters:
      ctx - Encapsulates information necessary for container liveness check.
      Returns:
      true if container is still alive
      Throws:
      IOException - if there is a failure while checking the container status

    • containerIsAlive

      @VisibleForTesting public static boolean containerIsAlive(String pid) throws IOException
      Returns true if the process with the specified pid is alive.
      Parameters:
      pid - String pid
      Returns:
      boolean true if the process is alive
      Throws:
      IOException - if the command to test process liveliness fails

    • killContainer

      protected void killContainer(String pid, ContainerExecutor.Signal signal) throws IOException
      Send a specified signal to the specified pid
      Parameters:
      pid - the pid of the process [group] to signal.
      signal - signal to send
      Throws:
      IOException - if the command to kill the process fails

    • deleteAsUser

      public void deleteAsUser(DeletionAsUserContext ctx) throws IOException, InterruptedException
      Description copied from class: ContainerExecutor
      Delete specified directories as a given user.
      Specified by:
      deleteAsUser in class ContainerExecutor
      Parameters:
      ctx - Encapsulates information necessary for deletion.
      Throws:
      IOException - if delete fails
      InterruptedException - if interrupted while waiting for the deletion operation to complete

    • symLink

      public void symLink(String target, String symlink) throws IOException
      Description copied from class: ContainerExecutor
      Create a symlink file which points to the target.
      Specified by:
      symLink in class ContainerExecutor
      Parameters:
      target - The target for symlink
      symlink - the symlink file
      Throws:
      IOException - Error when creating symlinks

    • getWorkingDir

      protected org.apache.hadoop.fs.Path getWorkingDir(List<String> localDirs, String user, String appId) throws IOException
      Return a randomly chosen application directory from a list of local storage directories. The probability of selecting a directory is proportional to its size.
      Parameters:
      localDirs - the target directories from which to select
      user - the user who owns the application
      appId - the application ID
      Returns:
      the selected directory
      Throws:
      IOException - if no application directories for the user can be found

    • createDir

      protected void createDir(org.apache.hadoop.fs.Path dirPath, org.apache.hadoop.fs.permission.FsPermission perms, boolean createParent, String user) throws IOException
      Use the lfs FileContext to create the target directory.
      Parameters:
      dirPath - the target directory
      perms - the target permissions for the target directory
      createParent - whether the parent directories should also be created
      user - the user as whom the target directory should be created. Used only on secure Windows hosts.
      Throws:
      IOException - if there's a failure performing a file operation
      See Also:

    • getLogDirPermissions

      @VisibleForTesting public String getLogDirPermissions()
      Return the default container log directory permissions.
      Returns:
      the default container log directory permissions

    • clearLogDirPermissions

      @VisibleForTesting public void clearLogDirPermissions()
      Clear the internal variable for repeatable testing.

    • execContainer

      public org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair execContainer(ContainerExecContext ctx) throws ContainerExecutionException
      Description copied from class: ContainerExecutor
      Perform interactive docker command into running container.
      Specified by:
      execContainer in class ContainerExecutor
      Parameters:
      ctx - Encapsulates information necessary for exec containers.
      Returns:
      the input/output stream of interactive docker shell.
      Throws:
      ContainerExecutionException

    • updateYarnSysFS

      public void updateYarnSysFS(Context ctx, String user, String appId, String spec) throws IOException
      Description copied from class: ContainerExecutor
      Update cluster information inside container.
      Specified by:
      updateYarnSysFS in class ContainerExecutor
      Parameters:
      ctx - ContainerRuntimeContext
      user - Owner of application
      appId - YARN application ID
      spec - Service Specification
      Throws:
      IOException - if there is a failure while writing spec to disk

    • reacquireContainer

      public int reacquireContainer(ContainerReacquisitionContext ctx) throws IOException, InterruptedException
      Description copied from class: ContainerExecutor
      Recover an already existing container. This is a blocking call and returns only when the container exits. Note that the container must have been activated prior to this call.
      Overrides:
      reacquireContainer in class ContainerExecutor
      Parameters:
      ctx - encapsulates information necessary to reacquire container
      Returns:
      The exit code of the pre-existing container
      Throws:
      IOException - if there is a failure while reacquiring the container
      InterruptedException - if interrupted while waiting to reacquire the container

    • postComplete

      public void postComplete(org.apache.hadoop.yarn.api.records.ContainerId containerId)
      clean up and release of resources.
      Parameters:
      containerId - containerId of running container

    • setNumaResourceAllocator

      @VisibleForTesting public void setNumaResourceAllocator(NumaResourceAllocator numaResourceAllocator)

    • setNumactl

      @VisibleForTesting public void setNumactl(String numactl)