Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity

Class AbstractCSQueue

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue
All Implemented Interfaces:
CSQueue, Queue, SchedulerQueue<CSQueue>
Direct Known Subclasses:
AbstractLeafQueue, AbstractParentQueue
public abstract class AbstractCSQueue extends Object implements CSQueue
Provides implementation of CSQueue methods common for every queue class in Capacity Scheduler.

  • Field Details

  • Constructor Details

  • Method Details

    • setupConfigurableCapacities

      protected void setupConfigurableCapacities()
      Sets up capacity and weight values from configuration.

    • getQueuePath

      public String getQueuePath()
      Description copied from interface: CSQueue
      Get the full name of the queue, including the heirarchy.
      Specified by:
      getQueuePath in interface CSQueue
      Returns:
      the full name of the queue

    • getQueuePathObject

      public QueuePath getQueuePathObject()
      Description copied from interface: CSQueue
      Gets the queue path object.
      Specified by:
      getQueuePathObject in interface CSQueue
      Returns:
      the object of the queue

    • getCapacity

      public float getCapacity()
      Description copied from interface: CSQueue
      Get the configured capacity of the queue.
      Specified by:
      getCapacity in interface CSQueue
      Returns:
      configured queue capacity

    • getAbsoluteCapacity

      public float getAbsoluteCapacity()
      Description copied from interface: CSQueue
      Get capacity of the parent of the queue as a function of the cumulative capacity in the cluster.
      Specified by:
      getAbsoluteCapacity in interface CSQueue
      Returns:
      capacity of the parent of the queue as a function of the cumulative capacity in the cluster

    • getAbsoluteMaximumCapacity

      public float getAbsoluteMaximumCapacity()
      Description copied from interface: CSQueue
      Get maximum-capacity of the queue as a funciton of the cumulative capacity of the cluster.
      Specified by:
      getAbsoluteMaximumCapacity in interface CSQueue
      Returns:
      maximum-capacity of the queue as a funciton of the cumulative capacity of the cluster

    • getAbsoluteUsedCapacity

      public float getAbsoluteUsedCapacity()
      Description copied from interface: CSQueue
      Get the current absolute used capacity of the queue relative to the entire cluster.
      Specified by:
      getAbsoluteUsedCapacity in interface CSQueue
      Returns:
      queue absolute used capacity

    • getMaximumCapacity

      public float getMaximumCapacity()
      Description copied from interface: CSQueue
      Get the configured maximum-capacity of the queue.
      Specified by:
      getMaximumCapacity in interface CSQueue
      Returns:
      the configured maximum-capacity of the queue

    • getUsedCapacity

      public float getUsedCapacity()
      Description copied from interface: CSQueue
      Get the current used capacity of nodes without label(s) of the queue and it's children (if any).
      Specified by:
      getUsedCapacity in interface CSQueue
      Returns:
      queue used capacity

    • getUsedResources

      public org.apache.hadoop.yarn.api.records.Resource getUsedResources()
      Description copied from interface: CSQueue
      Get the currently utilized resources which allocated at nodes without any labels in the cluster by the queue and children (if any).
      Specified by:
      getUsedResources in interface CSQueue
      Returns:
      used resources by the queue and it's children

    • getNumContainers

      public int getNumContainers()

    • getState

      public org.apache.hadoop.yarn.api.records.QueueState getState()
      Description copied from interface: CSQueue
      Get the current run-state of the queue
      Specified by:
      getState in interface CSQueue
      Specified by:
      getState in interface SchedulerQueue<CSQueue>
      Returns:
      current run-state

    • getMetrics

      public CSQueueMetrics getMetrics()
      Description copied from interface: Queue
      Get the queue metrics
      Specified by:
      getMetrics in interface Queue
      Returns:
      the queue metrics

    • getQueueShortName

      public String getQueueShortName()
      Description copied from interface: CSQueue
      Get the queue's legacy name.
      Specified by:
      getQueueShortName in interface CSQueue
      Returns:
      the queue name

    • getQueueName

      public String getQueueName()
      Description copied from interface: CSQueue
      Get the queue's internal reference name.
      Specified by:
      getQueueName in interface CSQueue
      Specified by:
      getQueueName in interface Queue
      Returns:
      the queue name

    • getParent

      public CSQueue getParent()
      Description copied from interface: CSQueue
      Get the parent Queue.
      Specified by:
      getParent in interface CSQueue
      Specified by:
      getParent in interface SchedulerQueue<CSQueue>
      Returns:
      the parent queue

    • setParent

      public void setParent(CSQueue newParentQueue)
      Description copied from interface: CSQueue
      Set the parent Queue.
      Specified by:
      setParent in interface CSQueue
      Parameters:
      newParentQueue - new parent queue

    • getPrivilegedEntity

      public org.apache.hadoop.yarn.security.PrivilegedEntity getPrivilegedEntity()
      Specified by:
      getPrivilegedEntity in interface CSQueue

    • getQueueContext

      public CapacitySchedulerQueueContext getQueueContext()

    • getAccessibleNodeLabels

      public Set<String> getAccessibleNodeLabels()
      Description copied from interface: Queue
      Get labels can be accessed of this queue labels={*}, means this queue can access any label labels={ }, means this queue cannot access any label except node without label labels={a, b, c} means this queue can access a or b or c
      Specified by:
      getAccessibleNodeLabels in interface Queue
      Returns:
      labels

    • hasAccess

      public boolean hasAccess(org.apache.hadoop.yarn.api.records.QueueACL acl, org.apache.hadoop.security.UserGroupInformation user)
      Checks whether the user has the required permission to execute the action of QueueACL.
      Specified by:
      hasAccess in interface CSQueue
      Specified by:
      hasAccess in interface Queue
      Parameters:
      acl - the access type the user is checked for
      user - UGI of the user
      Returns:
      true, if the user has permission, false otherwise

    • getDefaultNodeLabelExpression

      public String getDefaultNodeLabelExpression()
      Description copied from interface: Queue
      Get default label expression of this queue. If label expression of ApplicationSubmissionContext and label expression of Resource Request not set, this will be used.
      Specified by:
      getDefaultNodeLabelExpression in interface Queue
      Returns:
      default label expression

    • setupQueueConfigs

      protected void setupQueueConfigs(org.apache.hadoop.yarn.api.records.Resource clusterResource) throws IOException
      Initialize queue properties that are based on configuration.
      Parameters:
      clusterResource - overall resource of the cluster
      Throws:
      IOException - if configuration is set in a way that is inconsistent

    • parseAndSetDynamicTemplates

      protected void parseAndSetDynamicTemplates()
      Set properties specific to dynamic queues.

    • setDynamicQueueACLProperties

      protected void setDynamicQueueACLProperties()

    • overrideCapacityVectorsForSpecialQueues

      protected void overrideCapacityVectorsForSpecialQueues(String label)

    • getMinimumAbsoluteResource

      protected org.apache.hadoop.yarn.api.records.Resource getMinimumAbsoluteResource(QueuePath queuePath, String label)

    • getMaximumAbsoluteResource

      protected org.apache.hadoop.yarn.api.records.Resource getMaximumAbsoluteResource(QueuePath queuePath, String label)

    • checkConfigTypeIsAbsoluteResource

      protected boolean checkConfigTypeIsAbsoluteResource(QueuePath queuePath, String label)

    • updateCapacityConfigType

      protected void updateCapacityConfigType()

    • updateConfigurableResourceLimits

      protected void updateConfigurableResourceLimits(org.apache.hadoop.yarn.api.records.Resource clusterResource)
      Initializes configured minimum and maximum capacity from configuration, if capacity is defined in ABSOLUTE node.
      Parameters:
      clusterResource - overall resource of the cluster

    • getCapacityConfigType

      public AbstractCSQueue.CapacityConfigType getCapacityConfigType()
      Description copied from interface: CSQueue
      Get CapacityConfigType as PERCENTAGE or ABSOLUTE_RESOURCE.
      Specified by:
      getCapacityConfigType in interface CSQueue
      Returns:
      CapacityConfigType

    • getEffectiveCapacity

      public org.apache.hadoop.yarn.api.records.Resource getEffectiveCapacity(String label)
      Description copied from interface: CSQueue
      Get effective capacity of queue. If min/max resource is configured, preference will be given to absolute configuration over normal capacity.
      Specified by:
      getEffectiveCapacity in interface CSQueue
      Parameters:
      label - partition
      Returns:
      effective queue capacity

    • getEffectiveCapacityDown

      public org.apache.hadoop.yarn.api.records.Resource getEffectiveCapacityDown(String label, org.apache.hadoop.yarn.api.records.Resource factor)
      Description copied from interface: CSQueue
      Get effective capacity of queue. If min/max resource is configured, preference will be given to absolute configuration over normal capacity. Also round down the result to normalizeDown.
      Specified by:
      getEffectiveCapacityDown in interface CSQueue
      Parameters:
      label - partition
      factor - factor to normalize down
      Returns:
      effective queue capacity

    • getEffectiveMaxCapacity

      public org.apache.hadoop.yarn.api.records.Resource getEffectiveMaxCapacity(String label)
      Description copied from interface: CSQueue
      Get effective max capacity of queue. If min/max resource is configured, preference will be given to absolute configuration over normal capacity.
      Specified by:
      getEffectiveMaxCapacity in interface CSQueue
      Parameters:
      label - partition
      Returns:
      effective max queue capacity

    • getEffectiveMaxCapacityDown

      public org.apache.hadoop.yarn.api.records.Resource getEffectiveMaxCapacityDown(String label, org.apache.hadoop.yarn.api.records.Resource factor)
      Description copied from interface: CSQueue
      Get effective max capacity of queue. If min/max resource is configured, preference will be given to absolute configuration over normal capacity. Also round down the result to normalizeDown.
      Specified by:
      getEffectiveMaxCapacityDown in interface CSQueue
      Parameters:
      label - partition
      factor - factor to normalize down
      Returns:
      effective max queue capacity

    • getConfiguredCapacityVector

      public QueueCapacityVector getConfiguredCapacityVector(String label)
      Description copied from interface: CSQueue
      Get configured capacity vector parsed from the capacity config of the queue.
      Specified by:
      getConfiguredCapacityVector in interface CSQueue
      Parameters:
      label - node label (partition)
      Returns:
      capacity resource vector

    • getConfiguredMaxCapacityVector

      public QueueCapacityVector getConfiguredMaxCapacityVector(String label)
      Description copied from interface: CSQueue
      Get configured maximum capacity vector parsed from the capacity config of the queue.
      Specified by:
      getConfiguredMaxCapacityVector in interface CSQueue
      Parameters:
      label - node label (partition)
      Returns:
      capacity resource vector

    • setConfiguredMinCapacityVector

      public void setConfiguredMinCapacityVector(String label, QueueCapacityVector minCapacityVector)
      Description copied from interface: CSQueue
      Sets the configured minimum capacity vector to a specific value.
      Specified by:
      setConfiguredMinCapacityVector in interface CSQueue
      Parameters:
      label - node label (partition)
      minCapacityVector - capacity vector

    • setConfiguredMaxCapacityVector

      public void setConfiguredMaxCapacityVector(String label, QueueCapacityVector maxCapacityVector)
      Description copied from interface: CSQueue
      Sets the configured maximum capacity vector to a specific value.
      Specified by:
      setConfiguredMaxCapacityVector in interface CSQueue
      Parameters:
      label - node label (partition)
      maxCapacityVector - capacity vector

    • getQueueInfo

      protected org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo()

    • getMaximumAllocation

      @Private public org.apache.hadoop.yarn.api.records.Resource getMaximumAllocation()
      Specified by:
      getMaximumAllocation in interface CSQueue

    • getMinimumAllocation

      @Private public org.apache.hadoop.yarn.api.records.Resource getMinimumAllocation()
      Specified by:
      getMinimumAllocation in interface CSQueue

    • releaseResource

      protected void releaseResource(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource resource, String nodePartition)
      Decrements resource usage of the queue and all related statistics and metrics that depends on it.
      Parameters:
      clusterResource - overall cluster resource
      resource - resource amount to decrement
      nodePartition - node label

    • isReservationsContinueLooking

      @Private public boolean isReservationsContinueLooking()
      Returns whether we should continue to look at all heart beating nodes even after the reservation limit was hit.
      Returns:
      true, continue to look at all heart beating nodes even after the reservation limit was hit. otherwise false.

    • getACLs

      @Private public Map<org.apache.hadoop.yarn.security.AccessType,org.apache.hadoop.security.authorize.AccessControlList> getACLs()

    • getPreemptionDisabled

      @Private public boolean getPreemptionDisabled()
      Description copied from interface: CSQueue
      Check whether disable_preemption property is set for this queue
      Specified by:
      getPreemptionDisabled in interface CSQueue
      Returns:
      true if disable_preemption is set, false if not

    • getIntraQueuePreemptionDisabled

      @Private public boolean getIntraQueuePreemptionDisabled()
      Description copied from interface: CSQueue
      Check whether intra-queue preemption is disabled for this queue
      Specified by:
      getIntraQueuePreemptionDisabled in interface CSQueue
      Returns:
      true if either intra-queue preemption or inter-queue preemption is disabled for this queue, false if neither is disabled.

    • getIntraQueuePreemptionDisabledInHierarchy

      @Private public boolean getIntraQueuePreemptionDisabledInHierarchy()
      Description copied from interface: CSQueue
      Determines whether or not the intra-queue preemption disabled switch is set at any level in this queue's hierarchy.
      Specified by:
      getIntraQueuePreemptionDisabledInHierarchy in interface CSQueue
      Returns:
      state of the intra-queue preemption switch at this queue level

    • getQueueCapacities

      @Private public QueueCapacities getQueueCapacities()
      Description copied from interface: CSQueue
      Get QueueCapacities of this queue
      Specified by:
      getQueueCapacities in interface CSQueue
      Returns:
      queueCapacities

    • getQueueResourceUsage

      @Private public ResourceUsage getQueueResourceUsage()
      Description copied from interface: CSQueue
      Get ResourceUsage of this queue
      Specified by:
      getQueueResourceUsage in interface CSQueue
      Returns:
      resourceUsage

    • getQueueResourceQuotas

      public QueueResourceQuotas getQueueResourceQuotas()
      Description copied from interface: CSQueue
      Get QueueResourceQuotas associated with each queue.
      Specified by:
      getQueueResourceQuotas in interface CSQueue
      Returns:
      QueueResourceQuotas

    • getReadLock

      public ReentrantReadWriteLock.ReadLock getReadLock()
      Description copied from interface: CSQueue
      Get readLock associated with the Queue.
      Specified by:
      getReadLock in interface CSQueue
      Returns:
      readLock of corresponding queue.

    • getWriteLock

      public ReentrantReadWriteLock.WriteLock getWriteLock()
      Description copied from interface: CSQueue
      Get writeLock associated with the Queue.
      Specified by:
      getWriteLock in interface CSQueue
      Returns:
      writeLock of corresponding queue.

    • getConfiguredNodeLabels

      public Set<String> getConfiguredNodeLabels()
      Specified by:
      getConfiguredNodeLabels in interface CSQueue

    • incReservedResource

      public void incReservedResource(String partition, org.apache.hadoop.yarn.api.records.Resource reservedRes)
      Description copied from interface: Queue
      Increment Reserved Capacity
      Specified by:
      incReservedResource in interface Queue
      Parameters:
      partition - asked by application
      reservedRes - reserved resource asked

    • decReservedResource

      public void decReservedResource(String partition, org.apache.hadoop.yarn.api.records.Resource reservedRes)
      Description copied from interface: Queue
      Decrement Reserved Capacity
      Specified by:
      decReservedResource in interface Queue
      Parameters:
      partition - asked by application
      reservedRes - reserved resource asked

    • incPendingResource

      public void incPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc)
      Description copied from interface: Queue
      When new outstanding resource is asked, calling this will increase pending resource in a queue.
      Specified by:
      incPendingResource in interface Queue
      Parameters:
      nodeLabel - asked by application
      resourceToInc - new resource asked

    • decPendingResource

      public void decPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec)
      Description copied from interface: CSQueue
      When an outstanding resource is fulfilled or canceled, calling this will decrease pending resource in a queue.
      Specified by:
      decPendingResource in interface CSQueue
      Specified by:
      decPendingResource in interface Queue
      Parameters:
      nodeLabel - asked by application
      resourceToDec - new resource asked

    • incUsedResource

      public void incUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc, SchedulerApplicationAttempt application)
      Description copied from interface: CSQueue
      When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
      Specified by:
      incUsedResource in interface CSQueue
      Parameters:
      nodeLabel - node label.
      resourceToInc - resource.
      application - application.

    • decUsedResource

      public void decUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec, SchedulerApplicationAttempt application)
      Description copied from interface: CSQueue
      When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
      Specified by:
      decUsedResource in interface CSQueue
      Parameters:
      nodeLabel - node label.
      resourceToDec - resource.
      application - application.

    • getDefaultApplicationPriority

      public org.apache.hadoop.yarn.api.records.Priority getDefaultApplicationPriority()
      Description copied from interface: Queue
      Get the Default Application Priority for this queue
      Specified by:
      getDefaultApplicationPriority in interface Queue
      Returns:
      default application priority

    • getNodeLabelsForQueue

      public Set<String> getNodeLabelsForQueue()
      Returns the union of all node labels that could be accessed by this queue based on accessible node labels and configured node labels properties.
      Specified by:
      getNodeLabelsForQueue in interface CSQueue
      Returns:
      node labels this queue has access to

    • getTotalKillableResource

      public org.apache.hadoop.yarn.api.records.Resource getTotalKillableResource(String partition)

    • getKillableContainers

      public Iterator<RMContainer> getKillableContainers(String partition)

    • assignContainers

      @VisibleForTesting public CSAssignment assignContainers(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerNode node, ResourceLimits resourceLimits, SchedulingMode schedulingMode)
      Specified by:
      assignContainers in interface CSQueue

    • accept

      public boolean accept(org.apache.hadoop.yarn.api.records.Resource cluster, ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode> request)
      Checks whether this queue could accept the container allocation request.
      Specified by:
      accept in interface CSQueue
      Parameters:
      cluster - overall cluster resource
      request - container allocation request
      Returns:
      true if queue could accept the container allocation request, false otherwise

    • validateSubmitApplication

      public void validateSubmitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, String userName, String queue) throws org.apache.hadoop.security.AccessControlException
      Description copied from interface: CSQueue
      Validate submitApplication api so that moveApplication do a pre-check.
      Specified by:
      validateSubmitApplication in interface CSQueue
      Parameters:
      applicationId - Application ID
      userName - User Name
      queue - Queue Name
      Throws:
      org.apache.hadoop.security.AccessControlException - if any acl violation is there.

    • updateQueueState

      public void updateQueueState(org.apache.hadoop.yarn.api.records.QueueState queueState)
      Description copied from interface: SchedulerQueue
      Update the queue state.
      Specified by:
      updateQueueState in interface SchedulerQueue<CSQueue>
      Parameters:
      queueState - the queue state

    • activateQueue

      public void activateQueue() throws org.apache.hadoop.yarn.exceptions.YarnException
      Sets the state of this queue to RUNNING.
      Specified by:
      activateQueue in interface SchedulerQueue<CSQueue>
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException - if its parent queue is not in RUNNING state

    • appFinished

      protected void appFinished()
      Stops this queue if no application is currently running on the queue.

    • getPriority

      public org.apache.hadoop.yarn.api.records.Priority getPriority()
      Description copied from interface: CSQueue
      Get priority of queue
      Specified by:
      getPriority in interface CSQueue
      Returns:
      queue priority

    • getUserWeights

      public UserWeights getUserWeights()
      Description copied from interface: CSQueue
      Get the UserWeights object that wraps a map of usernames and weights
      Specified by:
      getUserWeights in interface CSQueue
      Returns:
      The UserWeights object.

    • recoverDrainingState

      public void recoverDrainingState()
      Recursively sets the state of this queue and the state of its parent to DRAINING.

    • getMultiNodeSortingPolicyClassName

      public String getMultiNodeSortingPolicyClassName()
      Description copied from interface: CSQueue
      Get Multi Node scheduling policy name.
      Specified by:
      getMultiNodeSortingPolicyClassName in interface CSQueue
      Returns:
      policy name

    • setMultiNodeSortingPolicyClassName

      public void setMultiNodeSortingPolicyClassName(String policyName)

    • getMaximumApplicationLifetime

      public long getMaximumApplicationLifetime()
      Description copied from interface: CSQueue
      Get the maximum lifetime in seconds of an application which is submitted to this queue. Apps can set their own lifetime timeout up to this value.
      Specified by:
      getMaximumApplicationLifetime in interface CSQueue
      Returns:
      max lifetime in seconds

    • getDefaultApplicationLifetime

      public long getDefaultApplicationLifetime()
      Description copied from interface: CSQueue
      Get the default lifetime in seconds of an application which is submitted to this queue. If an app doesn't specify its own timeout when submitted, this value will be used.
      Specified by:
      getDefaultApplicationLifetime in interface CSQueue
      Returns:
      default app lifetime

    • getDefaultAppLifetimeWasSpecifiedInConfig

      public boolean getDefaultAppLifetimeWasSpecifiedInConfig()
      Description copied from interface: CSQueue
      Get the indicator of whether or not the default application lifetime was set by a config property or was calculated by the capacity scheduler.
      Specified by:
      getDefaultAppLifetimeWasSpecifiedInConfig in interface CSQueue
      Returns:
      indicator whether set or calculated

    • setMaxParallelApps

      public void setMaxParallelApps(int maxParallelApps)

    • getMaxParallelApps

      public int getMaxParallelApps()
      Description copied from interface: CSQueue
      Get the max-parallel-applications property of the queue
      Specified by:
      getMaxParallelApps in interface CSQueue
      Returns:
      max-parallel-applications

    • updateAbsoluteCapacities

      protected void updateAbsoluteCapacities()

    • isDynamicQueue

      public boolean isDynamicQueue()
      Description copied from interface: CSQueue
      Checks whether the queue is a dynamic queue (created dynamically in the fashion of auto queue creation v2).
      Specified by:
      isDynamicQueue in interface CSQueue
      Returns:
      true, if it is a dynamic queue, false otherwise

    • setDynamicQueue

      public void setDynamicQueue(boolean dynamicQueue)

    • getCapacityOrWeightString

      protected String getCapacityOrWeightString()

    • isEligibleForAutoDeletion

      public boolean isEligibleForAutoDeletion()
      Checks whether this queue is a dynamic queue and could be deleted.
      Returns:
      true if the dynamic queue could be deleted, false otherwise

    • isInactiveDynamicQueue

      public boolean isInactiveDynamicQueue()
      Checks whether this queue is a dynamic queue and there has not been an application submission on it for a configured period of time.
      Returns:
      true if queue has been idle for a configured period of time, false otherwise