Package org.apache.hadoop.yarn.server.resourcemanager.scheduler

Class SchedulerUtils

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils
@Private @Unstable public class SchedulerUtils extends Object
Utilities shared by schedulers.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    SchedulerUtils.MaxResourceValidationResult
    This class contains invalid resource information along with its resource request.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    COMPLETED_APPLICATION
     
    static final String
    EXPIRED_CONTAINER
     
    static final String
    LOST_CONTAINER
     
    static final String
    PREEMPTED_CONTAINER
     
    static final String
    RELEASED_CONTAINER
     
    static final String
    UNRESERVED_CONTAINER
     
    static final String
    UPDATED_CONTAINER
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    SchedulerUtils()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static boolean
    checkQueueLabelExpression(Set<String> queueLabels, String labelExpression, RMContext rmContext)
    Check queue label expression, check if node label in queue's node-label-expression existed in clusterNodeLabels if rmContext !
    static org.apache.hadoop.yarn.api.records.ContainerStatus
    createAbnormalContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, String diagnostics)
    Utility to create a ContainerStatus during exceptional circumstances.
    static org.apache.hadoop.yarn.api.records.ContainerStatus
    createKilledContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, String diagnostics)
    Utility to create a ContainerStatus for killed containers.
    static RMContainer
    createOpportunisticRmContainer(RMContext rmContext, org.apache.hadoop.yarn.api.records.Container container, boolean isRemotelyAllocated)
     
    static org.apache.hadoop.yarn.api.records.ContainerStatus
    createPreemptedContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, String diagnostics)
    Utility to create a ContainerStatus during exceptional circumstances.
    static void
    enforcePartitionExclusivity(org.apache.hadoop.yarn.api.records.ResourceRequest resReq, Set<String> enforcedPartitions, String appLabel)
    If RM should enforce partition exclusivity for enforced partition "x": 1) If request is "x" and app label is not "x", override request to app's label. 2) If app label is "x", ensure request is "x".
    static org.apache.hadoop.yarn.api.records.Resource
    getNormalizedResource(org.apache.hadoop.yarn.api.records.Resource ask, org.apache.hadoop.yarn.util.resource.ResourceCalculator resourceCalculator, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource incrementResource)
    Utility method to normalize a resource request, by ensuring that the requested memory is a multiple of increment resource and is not zero.
    static boolean
    hasPendingResourceRequest(org.apache.hadoop.yarn.util.resource.ResourceCalculator rc, ResourceUsage usage, String nodePartition, org.apache.hadoop.yarn.api.records.Resource cluster, SchedulingMode schedulingMode)
     
    static boolean
    isNodeHeartbeated(SchedulerNode node, long skipNodeInterval)
     
    static void
    normalizeAndValidateRequest(org.apache.hadoop.yarn.api.records.ResourceRequest resReq, org.apache.hadoop.yarn.api.records.Resource maximumAllocation, String queueName, boolean isRecovery, RMContext rmContext, org.apache.hadoop.yarn.api.records.QueueInfo queueInfo, boolean nodeLabelsEnabled)
     
    static void
    normalizeAndValidateRequest(org.apache.hadoop.yarn.api.records.ResourceRequest resReq, org.apache.hadoop.yarn.api.records.Resource maximumAllocation, String queueName, RMContext rmContext, org.apache.hadoop.yarn.api.records.QueueInfo queueInfo, boolean nodeLabelsEnabled)
     
    static void
    normalizeRequest(org.apache.hadoop.yarn.api.records.ResourceRequest ask, org.apache.hadoop.yarn.util.resource.ResourceCalculator resourceCalculator, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource)
    Utility method to normalize a resource request, by ensuring that the requested memory is a multiple of minMemory and is not zero.
    static org.apache.hadoop.yarn.security.AccessType
    toAccessType(org.apache.hadoop.yarn.api.records.QueueACL acl)
     
    static SchedulerUtils.MaxResourceValidationResult
    validateResourceRequestsAgainstQueueMaxResource(org.apache.hadoop.yarn.api.records.ResourceRequest resReq, org.apache.hadoop.yarn.api.records.Resource availableResource)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Constructor Details

    • SchedulerUtils

      public SchedulerUtils()

  • Method Details

    • createAbnormalContainerStatus

      public static org.apache.hadoop.yarn.api.records.ContainerStatus createAbnormalContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, String diagnostics)
      Utility to create a ContainerStatus during exceptional circumstances.
      Parameters:
      containerId - ContainerId of returned/released/lost container.
      diagnostics - diagnostic message
      Returns:
      ContainerStatus for an returned/released/lost container

    • createKilledContainerStatus

      public static org.apache.hadoop.yarn.api.records.ContainerStatus createKilledContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, String diagnostics)
      Utility to create a ContainerStatus for killed containers.
      Parameters:
      containerId - ContainerId of the killed container.
      diagnostics - diagnostic message
      Returns:
      ContainerStatus for a killed container

    • createPreemptedContainerStatus

      public static org.apache.hadoop.yarn.api.records.ContainerStatus createPreemptedContainerStatus(org.apache.hadoop.yarn.api.records.ContainerId containerId, String diagnostics)
      Utility to create a ContainerStatus during exceptional circumstances.
      Parameters:
      containerId - ContainerId of returned/released/lost container.
      diagnostics - diagnostic message
      Returns:
      ContainerStatus for an returned/released/lost container

    • normalizeRequest

      @VisibleForTesting public static void normalizeRequest(org.apache.hadoop.yarn.api.records.ResourceRequest ask, org.apache.hadoop.yarn.util.resource.ResourceCalculator resourceCalculator, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource)
      Utility method to normalize a resource request, by ensuring that the requested memory is a multiple of minMemory and is not zero.
      Parameters:
      ask - resource request.
      resourceCalculator - ResourceCalculator the resource calculator to use.
      minimumResource - minimum Resource.
      maximumResource - maximum Resource.

    • getNormalizedResource

      public static org.apache.hadoop.yarn.api.records.Resource getNormalizedResource(org.apache.hadoop.yarn.api.records.Resource ask, org.apache.hadoop.yarn.util.resource.ResourceCalculator resourceCalculator, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource incrementResource)
      Utility method to normalize a resource request, by ensuring that the requested memory is a multiple of increment resource and is not zero.
      Parameters:
      ask - resource request.
      resourceCalculator - ResourceCalculator the resource calculator to use.
      minimumResource - minimum Resource.
      maximumResource - maximum Resource.
      incrementResource - increment Resource.
      Returns:
      normalized resource

    • normalizeAndValidateRequest

      public static void normalizeAndValidateRequest(org.apache.hadoop.yarn.api.records.ResourceRequest resReq, org.apache.hadoop.yarn.api.records.Resource maximumAllocation, String queueName, boolean isRecovery, RMContext rmContext, org.apache.hadoop.yarn.api.records.QueueInfo queueInfo, boolean nodeLabelsEnabled) throws org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException
      Throws:
      org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException

    • normalizeAndValidateRequest

      public static void normalizeAndValidateRequest(org.apache.hadoop.yarn.api.records.ResourceRequest resReq, org.apache.hadoop.yarn.api.records.Resource maximumAllocation, String queueName, RMContext rmContext, org.apache.hadoop.yarn.api.records.QueueInfo queueInfo, boolean nodeLabelsEnabled) throws org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException
      Throws:
      org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException

    • enforcePartitionExclusivity

      public static void enforcePartitionExclusivity(org.apache.hadoop.yarn.api.records.ResourceRequest resReq, Set<String> enforcedPartitions, String appLabel)
      If RM should enforce partition exclusivity for enforced partition "x": 1) If request is "x" and app label is not "x", override request to app's label. 2) If app label is "x", ensure request is "x".
      Parameters:
      resReq - resource request
      enforcedPartitions - list of exclusive enforced partitions
      appLabel - app's node label expression

    • validateResourceRequestsAgainstQueueMaxResource

      public static SchedulerUtils.MaxResourceValidationResult validateResourceRequestsAgainstQueueMaxResource(org.apache.hadoop.yarn.api.records.ResourceRequest resReq, org.apache.hadoop.yarn.api.records.Resource availableResource) throws org.apache.hadoop.yarn.exceptions.SchedulerInvalidResourceRequestException
      Throws:
      org.apache.hadoop.yarn.exceptions.SchedulerInvalidResourceRequestException

    • checkQueueLabelExpression

      public static boolean checkQueueLabelExpression(Set<String> queueLabels, String labelExpression, RMContext rmContext)
      Check queue label expression, check if node label in queue's node-label-expression existed in clusterNodeLabels if rmContext != null.
      Parameters:
      queueLabels - queue Labels.
      labelExpression - label expression.
      rmContext - rmContext.
      Returns:
      true, if node label in queue's node-label-expression existed in clusterNodeLabels; otherwise false.

    • toAccessType

      public static org.apache.hadoop.yarn.security.AccessType toAccessType(org.apache.hadoop.yarn.api.records.QueueACL acl)

    • hasPendingResourceRequest

      @Private public static boolean hasPendingResourceRequest(org.apache.hadoop.yarn.util.resource.ResourceCalculator rc, ResourceUsage usage, String nodePartition, org.apache.hadoop.yarn.api.records.Resource cluster, SchedulingMode schedulingMode)

    • createOpportunisticRmContainer

      public static RMContainer createOpportunisticRmContainer(RMContext rmContext, org.apache.hadoop.yarn.api.records.Container container, boolean isRemotelyAllocated)

    • isNodeHeartbeated

      public static boolean isNodeHeartbeated(SchedulerNode node, long skipNodeInterval)