Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity

Class AbstractLeafQueue

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue
All Implemented Interfaces:
CSQueue, Queue, SchedulerQueue<CSQueue>
Direct Known Subclasses:
AbstractAutoCreatedLeafQueue, LeafQueue
public class AbstractLeafQueue extends AbstractCSQueue

  • Field Details

    • maxApplications

      protected int maxApplications

    • maxApplicationsPerUser

      protected volatile int maxApplicationsPerUser

  • Constructor Details

  • Method Details

    • setupQueueConfigs

      protected void setupQueueConfigs(org.apache.hadoop.yarn.api.records.Resource clusterResource) throws IOException
      Description copied from class: AbstractCSQueue
      Initialize queue properties that are based on configuration.
      Overrides:
      setupQueueConfigs in class AbstractCSQueue
      Parameters:
      clusterResource - overall resource of the cluster
      Throws:
      IOException - if configuration is set in a way that is inconsistent

    • getMinimumAllocationFactor

      @Private public float getMinimumAllocationFactor()
      Used only by tests.
      Returns:
      minimumAllocationFactor.

    • getMaxAMResourcePerQueuePercent

      @Private public float getMaxAMResourcePerQueuePercent()
      Used only by tests.
      Returns:
      maxAMResourcePerQueuePercent.

    • getMaxApplications

      public int getMaxApplications()

    • getMaxApplicationsPerUser

      public int getMaxApplicationsPerUser()

    • getUsersManager

      public UsersManager getUsersManager()
      Returns:
      UsersManager instance.

    • getAbstractUsersManager

      public AbstractUsersManager getAbstractUsersManager()
      Description copied from interface: CSQueue
      Get the AbstractUsersManager for the queue.
      Returns:
      the AbstractUsersManager for the queue

    • getChildQueues

      public List<CSQueue> getChildQueues()
      Description copied from interface: CSQueue
      Get child queues
      Returns:
      child queues

    • getChildQueuesByTryLock

      public List<CSQueue> getChildQueuesByTryLock()
      Description copied from interface: CSQueue
      Get child queues By tryLock.
      Returns:
      child queues

    • getNumApplications

      public int getNumApplications()
      Description copied from interface: CSQueue
      Get the number of applications in the queue.
      Returns:
      number of applications

    • getNumPendingApplications

      public int getNumPendingApplications()

    • getNumActiveApplications

      public int getNumActiveApplications()

    • getNumPendingApplications

      @Private public int getNumPendingApplications(String user)

    • getNumActiveApplications

      @Private public int getNumActiveApplications(String user)

    • getUserLimit

      @Private public float getUserLimit()

    • getUserLimitFactor

      @Private public float getUserLimitFactor()

    • getQueueInfo

      public org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(boolean includeChildQueues, boolean recursive)
      Description copied from interface: Queue
      Get queue information
      Parameters:
      includeChildQueues - include child queues?
      recursive - recursively get child queue information?
      Returns:
      queue information

    • getQueueUserAclInfo

      public List<org.apache.hadoop.yarn.api.records.QueueUserACLInfo> getQueueUserAclInfo(org.apache.hadoop.security.UserGroupInformation user)
      Description copied from interface: Queue
      Get queue ACLs for given user.
      Parameters:
      user - username
      Returns:
      queue ACLs for user

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • getExtendedCapacityOrWeightString

      protected String getExtendedCapacityOrWeightString()

    • getUser

      @VisibleForTesting public UsersManager.User getUser(String userName)

    • getOrCreateUser

      @VisibleForTesting public UsersManager.User getOrCreateUser(String userName)

    • getPriorityACLs

      @Private public List<AppPriorityACLGroup> getPriorityACLs()

    • reinitialize

      public void reinitialize(CSQueue newlyParsedQueue, org.apache.hadoop.yarn.api.records.Resource clusterResource) throws IOException
      Description copied from interface: CSQueue
      Reinitialize the queue.
      Parameters:
      newlyParsedQueue - new queue to re-initalize from
      clusterResource - resources in the cluster
      Throws:
      IOException - an I/O exception has occurred.

    • submitApplicationAttempt

      public void submitApplicationAttempt(FiCaSchedulerApp application, String userName)
      Description copied from interface: CSQueue
      Submit an application attempt to the queue.
      Parameters:
      application - application whose attempt is being submitted.
      userName - userName who submitted the application.

    • submitApplicationAttempt

      public void submitApplicationAttempt(FiCaSchedulerApp application, String userName, boolean isMoveApp)
      Description copied from interface: CSQueue
      Submit an application attempt to the queue.
      Parameters:
      application - application whose attempt is being submitted
      userName - user who submitted the application attempt
      isMoveApp - is application being moved across the queue

    • submitApplication

      public void submitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, String userName, String queue) throws org.apache.hadoop.security.AccessControlException
      Description copied from interface: CSQueue
      Submit a new application to the queue.
      Parameters:
      applicationId - the applicationId of the application being submitted
      userName - user who submitted the application
      queue - queue to which the application is submitted
      Throws:
      org.apache.hadoop.security.AccessControlException - if any acl violation is there.

    • validateSubmitApplication

      public void validateSubmitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, String userName, String queue) throws org.apache.hadoop.security.AccessControlException
      Description copied from interface: CSQueue
      Validate submitApplication api so that moveApplication do a pre-check.
      Specified by:
      validateSubmitApplication in interface CSQueue
      Overrides:
      validateSubmitApplication in class AbstractCSQueue
      Parameters:
      applicationId - Application ID
      userName - User Name
      queue - Queue Name
      Throws:
      org.apache.hadoop.security.AccessControlException - if any acl violation is there.

    • getAMResourceLimit

      public org.apache.hadoop.yarn.api.records.Resource getAMResourceLimit()

    • getAMResourceLimitPerPartition

      public org.apache.hadoop.yarn.api.records.Resource getAMResourceLimitPerPartition(String nodePartition)

    • calculateAndGetAMResourceLimit

      @VisibleForTesting public org.apache.hadoop.yarn.api.records.Resource calculateAndGetAMResourceLimit()

    • getUserAMResourceLimit

      @VisibleForTesting public org.apache.hadoop.yarn.api.records.Resource getUserAMResourceLimit()

    • getUserAMResourceLimitPerPartition

      public org.apache.hadoop.yarn.api.records.Resource getUserAMResourceLimitPerPartition(String nodePartition, String userName)

    • calculateAndGetAMResourceLimitPerPartition

      public org.apache.hadoop.yarn.api.records.Resource calculateAndGetAMResourceLimitPerPartition(String nodePartition)

    • activateApplications

      protected void activateApplications()

    • finishApplication

      public void finishApplication(org.apache.hadoop.yarn.api.records.ApplicationId application, String user)
      Description copied from interface: CSQueue
      An application submitted to this queue has finished.
      Parameters:
      application - applicationId.
      user - user who submitted the application

    • finishApplicationAttempt

      public void finishApplicationAttempt(FiCaSchedulerApp application, String queue)
      Description copied from interface: CSQueue
      An application attempt submitted to this queue has finished.
      Parameters:
      application - application attempt.
      queue - queue.

    • assignContainers

      public CSAssignment assignContainers(org.apache.hadoop.yarn.api.records.Resource clusterResource, CandidateNodeSet<FiCaSchedulerNode> candidates, ResourceLimits currentResourceLimits, SchedulingMode schedulingMode)
      Description copied from interface: CSQueue
      Assign containers to applications in the queue or it's children (if any).
      Parameters:
      clusterResource - the resource of the cluster.
      candidates - CandidateNodeSet the nodes that are considered for the current placement.
      currentResourceLimits - how much overall resource of this queue can use.
      schedulingMode - Type of exclusive check when assign container on a NodeManager, see SchedulingMode.
      Returns:
      the assignment

    • accept

      public boolean accept(org.apache.hadoop.yarn.api.records.Resource cluster, ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode> request)
      Description copied from class: AbstractCSQueue
      Checks whether this queue could accept the container allocation request.
      Specified by:
      accept in interface CSQueue
      Overrides:
      accept in class AbstractCSQueue
      Parameters:
      cluster - overall cluster resource
      request - container allocation request
      Returns:
      true if queue could accept the container allocation request, false otherwise

    • apply

      public void apply(org.apache.hadoop.yarn.api.records.Resource cluster, ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode> request)

    • getHeadroom

      protected org.apache.hadoop.yarn.api.records.Resource getHeadroom(UsersManager.User user, org.apache.hadoop.yarn.api.records.Resource queueCurrentLimit, org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application)

    • getHeadroom

      protected org.apache.hadoop.yarn.api.records.Resource getHeadroom(UsersManager.User user, org.apache.hadoop.yarn.api.records.Resource queueCurrentLimit, org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, String partition)

    • getNodeLocalityDelay

      @Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class) public int getNodeLocalityDelay()

    • getRackLocalityAdditionalDelay

      @Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class) public int getRackLocalityAdditionalDelay()

    • getRackLocalityFullReset

      @Lock(org.apache.hadoop.yarn.server.utils.Lock.NoLock.class) public boolean getRackLocalityFullReset()

    • getResourceLimitForActiveUsers

      public org.apache.hadoop.yarn.api.records.Resource getResourceLimitForActiveUsers(String userName, org.apache.hadoop.yarn.api.records.Resource clusterResource, String nodePartition, SchedulingMode schedulingMode)
      Parameters:
      userName - Name of user who has submitted one/more app to given queue.
      clusterResource - total cluster resource
      nodePartition - partition name
      schedulingMode - scheduling mode RESPECT_PARTITION_EXCLUSIVITY/IGNORE_PARTITION_EXCLUSIVITY
      Returns:
      Computed User Limit

    • getResourceLimitForAllUsers

      public org.apache.hadoop.yarn.api.records.Resource getResourceLimitForAllUsers(String userName, org.apache.hadoop.yarn.api.records.Resource clusterResource, String nodePartition, SchedulingMode schedulingMode)
      Parameters:
      userName - Name of user who has submitted one/more app to given queue.
      clusterResource - total cluster resource
      nodePartition - partition name
      schedulingMode - scheduling mode RESPECT_PARTITION_EXCLUSIVITY/IGNORE_PARTITION_EXCLUSIVITY
      Returns:
      Computed User Limit

    • canAssignToUser

      @Private protected boolean canAssignToUser(org.apache.hadoop.yarn.api.records.Resource clusterResource, String userName, org.apache.hadoop.yarn.api.records.Resource limit, FiCaSchedulerApp application, String nodePartition, ResourceLimits currentResourceLimits)

    • parseAndSetDynamicTemplates

      protected void parseAndSetDynamicTemplates()
      Description copied from class: AbstractCSQueue
      Set properties specific to dynamic queues.
      Overrides:
      parseAndSetDynamicTemplates in class AbstractCSQueue

    • setDynamicQueueACLProperties

      protected void setDynamicQueueACLProperties()
      Overrides:
      setDynamicQueueACLProperties in class AbstractCSQueue

    • recalculateQueueUsageRatio

      public void recalculateQueueUsageRatio(org.apache.hadoop.yarn.api.records.Resource clusterResource, String nodePartition)
      Recalculate QueueUsage Ratio.
      Parameters:
      clusterResource - Total Cluster Resource
      nodePartition - Partition

    • completedContainer

      public void completedContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, FiCaSchedulerNode node, RMContainer rmContainer, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event, CSQueue childQueue, boolean sortQueues)
      Description copied from interface: CSQueue
      A container assigned to the queue has completed.
      Parameters:
      clusterResource - the resource of the cluster
      application - application to which the container was assigned
      node - node on which the container completed
      rmContainer - completed container, null if it was just a reservation
      containerStatus - ContainerStatus for the completed container
      event - event to be sent to the container
      childQueue - CSQueue to reinsert in childQueues
      sortQueues - indicates whether it should re-sort the queues

    • refreshAfterResourceCalculation

      public void refreshAfterResourceCalculation(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits resourceLimits)

    • updateClusterResource

      public void updateClusterResource(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits currentResourceLimits)
      Description copied from interface: CSQueue
      Update the cluster resource for queues as we add/remove nodes
      Parameters:
      clusterResource - the current cluster resource
      currentResourceLimits - the current ResourceLimits

    • updateClusterResourceLegacyMode

      public void updateClusterResourceLegacyMode(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits currentResourceLimits)

    • incUsedResource

      public void incUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc, SchedulerApplicationAttempt application)
      Description copied from interface: CSQueue
      When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
      Specified by:
      incUsedResource in interface CSQueue
      Overrides:
      incUsedResource in class AbstractCSQueue
      Parameters:
      nodeLabel - node label.
      resourceToInc - resource.
      application - application.

    • decUsedResource

      public void decUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec, SchedulerApplicationAttempt application)
      Description copied from interface: CSQueue
      When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
      Specified by:
      decUsedResource in interface CSQueue
      Overrides:
      decUsedResource in class AbstractCSQueue
      Parameters:
      nodeLabel - node label.
      resourceToDec - resource.
      application - application.

    • incAMUsedResource

      public void incAMUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc, SchedulerApplicationAttempt application)

    • decAMUsedResource

      public void decAMUsedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec, SchedulerApplicationAttempt application)

    • recoverContainer

      public void recoverContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, SchedulerApplicationAttempt attempt, RMContainer rmContainer)
      Description copied from interface: Queue
      Recover the state of the queue for a given container.
      Parameters:
      clusterResource - the resource of the cluster
      attempt - the application for which the container was allocated
      rmContainer - the container that was recovered.

    • getPendingApplications

      public Collection<FiCaSchedulerApp> getPendingApplications()
      Obtain (read-only) collection of pending applications.
      Returns:
      collection of pending applications.

    • getApplications

      public Collection<FiCaSchedulerApp> getApplications()
      Obtain (read-only) collection of active applications.
      Returns:
      collection of active applications.

    • getAllApplications

      public Collection<FiCaSchedulerApp> getAllApplications()
      Obtain (read-only) collection of all applications.
      Returns:
      collection of all applications.

    • getTotalPendingResourcesConsideringUserLimit

      public org.apache.hadoop.yarn.api.records.Resource getTotalPendingResourcesConsideringUserLimit(org.apache.hadoop.yarn.api.records.Resource clusterResources, String partition, boolean deductReservedFromPending)
      Get total pending resource considering user limit for the leaf queue. This will be used for calculating pending resources in the preemption monitor. Consider the headroom for each user in the queue. Total pending for the queue = sum(for each user(min((user's headroom), sum(user's pending requests)))) NOTE:
      Parameters:
      clusterResources - clusterResource
      partition - node partition
      deductReservedFromPending - When a container is reserved in CS, pending resource will not be deducted. This could lead to double accounting when doing preemption: In normal cases, we should deduct reserved resource from pending to avoid excessive preemption.
      Returns:
      Total pending resource considering user limit

    • collectSchedulerApplications

      public void collectSchedulerApplications(Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps)
      Description copied from interface: CSQueue
      Adds all applications in the queue and its subqueues to the given collection.
      Parameters:
      apps - the collection to add the applications to

    • attachContainer

      public void attachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, RMContainer rmContainer)
      Description copied from interface: CSQueue
      Attach a container to this queue
      Parameters:
      clusterResource - the current cluster resource
      application - application to which the container was assigned
      rmContainer - the container to attach

    • detachContainer

      public void detachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, RMContainer rmContainer)
      Description copied from interface: CSQueue
      Detach a container from this queue
      Parameters:
      clusterResource - the current cluster resource
      application - application to which the container was assigned
      rmContainer - the container to detach

    • getIgnoreExclusivityRMContainers

      public Map<String,TreeSet<RMContainer>> getIgnoreExclusivityRMContainers()
      Returns:
      all ignored partition exclusivity RMContainers in the LeafQueue, this will be used by preemption policy.

    • setCapacity

      public void setCapacity(float capacity)

    • setCapacity

      public void setCapacity(String nodeLabel, float capacity)

    • setAbsoluteCapacity

      public void setAbsoluteCapacity(float absoluteCapacity)

    • setAbsoluteCapacity

      public void setAbsoluteCapacity(String nodeLabel, float absoluteCapacity)

    • setMaxApplicationsPerUser

      public void setMaxApplicationsPerUser(int maxApplicationsPerUser)

    • setMaxApplications

      public void setMaxApplications(int maxApplications)

    • setMaxAMResourcePerQueuePercent

      public void setMaxAMResourcePerQueuePercent(float maxAMResourcePerQueuePercent)

    • getOrderingPolicy

      public OrderingPolicy<FiCaSchedulerApp> getOrderingPolicy()

    • getDefaultApplicationPriority

      public org.apache.hadoop.yarn.api.records.Priority getDefaultApplicationPriority()
      Description copied from interface: Queue
      Get the Default Application Priority for this queue
      Specified by:
      getDefaultApplicationPriority in interface Queue
      Overrides:
      getDefaultApplicationPriority in class AbstractCSQueue
      Returns:
      default application priority

    • updateApplicationPriority

      public void updateApplicationPriority(SchedulerApplication<FiCaSchedulerApp> app, org.apache.hadoop.yarn.api.records.Priority newAppPriority)

    • getPendingAppsOrderingPolicy

      public OrderingPolicy<FiCaSchedulerApp> getPendingAppsOrderingPolicy()

    • stopQueue

      public void stopQueue()
      Description copied from interface: SchedulerQueue
      Stop the queue.

    • getAllUsers

      public Set<String> getAllUsers()
      Get all valid users in this queue.
      Returns:
      user list

    • isEligibleForAutoDeletion

      public boolean isEligibleForAutoDeletion()
      Description copied from class: AbstractCSQueue
      Checks whether this queue is a dynamic queue and could be deleted.
      Overrides:
      isEligibleForAutoDeletion in class AbstractCSQueue
      Returns:
      true if the dynamic queue could be deleted, false otherwise