Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity

Interface CSQueue

All Superinterfaces:
Queue, SchedulerQueue<CSQueue>
All Known Implementing Classes:
AbstractAutoCreatedLeafQueue, AbstractCSQueue, AbstractLeafQueue, AbstractManagedParentQueue, AbstractParentQueue, AutoCreatedLeafQueue, LeafQueue, ManagedParentQueue, ParentQueue, PlanQueue, ReservationQueue
@Stable @Private public interface CSQueue extends SchedulerQueue<CSQueue>
CSQueue represents a node in the tree of hierarchical queues in the CapacityScheduler.

  • Method Details

    • getParent

      CSQueue getParent()
      Get the parent Queue.
      Specified by:
      getParent in interface SchedulerQueue<CSQueue>
      Returns:
      the parent queue

    • setParent

      void setParent(CSQueue newParentQueue)
      Set the parent Queue.
      Parameters:
      newParentQueue - new parent queue

    • getQueueName

      String getQueueName()
      Get the queue's internal reference name.
      Specified by:
      getQueueName in interface Queue
      Returns:
      the queue name

    • getQueueShortName

      String getQueueShortName()
      Get the queue's legacy name.
      Returns:
      the queue name

    • getQueuePath

      String getQueuePath()
      Get the full name of the queue, including the heirarchy.
      Returns:
      the full name of the queue

    • getQueuePathObject

      QueuePath getQueuePathObject()
      Gets the queue path object.
      Returns:
      the object of the queue

    • isDynamicQueue

      boolean isDynamicQueue()
      Checks whether the queue is a dynamic queue (created dynamically in the fashion of auto queue creation v2).
      Returns:
      true, if it is a dynamic queue, false otherwise

    • getPrivilegedEntity

      org.apache.hadoop.yarn.security.PrivilegedEntity getPrivilegedEntity()

    • getMaximumAllocation

      org.apache.hadoop.yarn.api.records.Resource getMaximumAllocation()

    • getMinimumAllocation

      org.apache.hadoop.yarn.api.records.Resource getMinimumAllocation()

    • getCapacity

      float getCapacity()
      Get the configured capacity of the queue.
      Returns:
      configured queue capacity

    • getAbsoluteCapacity

      float getAbsoluteCapacity()
      Get capacity of the parent of the queue as a function of the cumulative capacity in the cluster.
      Returns:
      capacity of the parent of the queue as a function of the cumulative capacity in the cluster

    • getMaximumCapacity

      float getMaximumCapacity()
      Get the configured maximum-capacity of the queue.
      Returns:
      the configured maximum-capacity of the queue

    • getAbsoluteMaximumCapacity

      float getAbsoluteMaximumCapacity()
      Get maximum-capacity of the queue as a funciton of the cumulative capacity of the cluster.
      Returns:
      maximum-capacity of the queue as a funciton of the cumulative capacity of the cluster

    • getAbsoluteUsedCapacity

      float getAbsoluteUsedCapacity()
      Get the current absolute used capacity of the queue relative to the entire cluster.
      Returns:
      queue absolute used capacity

    • getUsedCapacity

      float getUsedCapacity()
      Get the current used capacity of nodes without label(s) of the queue and it's children (if any).
      Returns:
      queue used capacity

    • getUsedResources

      org.apache.hadoop.yarn.api.records.Resource getUsedResources()
      Get the currently utilized resources which allocated at nodes without any labels in the cluster by the queue and children (if any).
      Returns:
      used resources by the queue and it's children

    • getState

      org.apache.hadoop.yarn.api.records.QueueState getState()
      Get the current run-state of the queue
      Specified by:
      getState in interface SchedulerQueue<CSQueue>
      Returns:
      current run-state

    • getMaxParallelApps

      int getMaxParallelApps()
      Get the max-parallel-applications property of the queue
      Returns:
      max-parallel-applications

    • getChildQueues

      List<CSQueue> getChildQueues()
      Get child queues
      Specified by:
      getChildQueues in interface SchedulerQueue<CSQueue>
      Returns:
      child queues

    • getChildQueuesByTryLock

      List<CSQueue> getChildQueuesByTryLock()
      Get child queues By tryLock.
      Returns:
      child queues

    • hasAccess

      boolean hasAccess(org.apache.hadoop.yarn.api.records.QueueACL acl, org.apache.hadoop.security.UserGroupInformation user)
      Check if the user has permission to perform the operation
      Specified by:
      hasAccess in interface Queue
      Parameters:
      acl - ACL
      user - user
      Returns:
      true if the user has the permission, false otherwise

    • submitApplication

      void submitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, String user, String queue) throws org.apache.hadoop.security.AccessControlException
      Submit a new application to the queue.
      Parameters:
      applicationId - the applicationId of the application being submitted
      user - user who submitted the application
      queue - queue to which the application is submitted
      Throws:
      org.apache.hadoop.security.AccessControlException - if any acl violation is there.

    • submitApplicationAttempt

      void submitApplicationAttempt(FiCaSchedulerApp application, String userName)
      Submit an application attempt to the queue.
      Parameters:
      application - application whose attempt is being submitted.
      userName - userName who submitted the application.

    • submitApplicationAttempt

      void submitApplicationAttempt(FiCaSchedulerApp application, String userName, boolean isMoveApp)
      Submit an application attempt to the queue.
      Parameters:
      application - application whose attempt is being submitted
      userName - user who submitted the application attempt
      isMoveApp - is application being moved across the queue

    • finishApplication

      void finishApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, String user)
      An application submitted to this queue has finished.
      Parameters:
      applicationId - applicationId.
      user - user who submitted the application

    • finishApplicationAttempt

      void finishApplicationAttempt(FiCaSchedulerApp application, String queue)
      An application attempt submitted to this queue has finished.
      Parameters:
      application - application attempt.
      queue - queue.

    • assignContainers

      CSAssignment assignContainers(org.apache.hadoop.yarn.api.records.Resource clusterResource, CandidateNodeSet<FiCaSchedulerNode> candidates, ResourceLimits resourceLimits, SchedulingMode schedulingMode)
      Assign containers to applications in the queue or it's children (if any).
      Parameters:
      clusterResource - the resource of the cluster.
      candidates - CandidateNodeSet the nodes that are considered for the current placement.
      resourceLimits - how much overall resource of this queue can use.
      schedulingMode - Type of exclusive check when assign container on a NodeManager, see SchedulingMode.
      Returns:
      the assignment

    • completedContainer

      void completedContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, FiCaSchedulerNode node, RMContainer container, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus, RMContainerEventType event, CSQueue childQueue, boolean sortQueues)
      A container assigned to the queue has completed.
      Parameters:
      clusterResource - the resource of the cluster
      application - application to which the container was assigned
      node - node on which the container completed
      container - completed container, null if it was just a reservation
      containerStatus - ContainerStatus for the completed container
      childQueue - CSQueue to reinsert in childQueues
      event - event to be sent to the container
      sortQueues - indicates whether it should re-sort the queues

    • getNumApplications

      int getNumApplications()
      Get the number of applications in the queue.
      Returns:
      number of applications

    • reinitialize

      void reinitialize(CSQueue newlyParsedQueue, org.apache.hadoop.yarn.api.records.Resource clusterResource) throws IOException
      Reinitialize the queue.
      Parameters:
      newlyParsedQueue - new queue to re-initalize from
      clusterResource - resources in the cluster
      Throws:
      IOException - an I/O exception has occurred.

    • refreshAfterResourceCalculation

      void refreshAfterResourceCalculation(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits resourceLimits)

    • updateClusterResource

      void updateClusterResource(org.apache.hadoop.yarn.api.records.Resource clusterResource, ResourceLimits resourceLimits)
      Update the cluster resource for queues as we add/remove nodes
      Parameters:
      clusterResource - the current cluster resource
      resourceLimits - the current ResourceLimits

    • getAbstractUsersManager

      AbstractUsersManager getAbstractUsersManager()
      Get the AbstractUsersManager for the queue.
      Specified by:
      getAbstractUsersManager in interface Queue
      Returns:
      the AbstractUsersManager for the queue

    • collectSchedulerApplications

      void collectSchedulerApplications(Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps)
      Adds all applications in the queue and its subqueues to the given collection.
      Parameters:
      apps - the collection to add the applications to

    • detachContainer

      void detachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, RMContainer container)
      Detach a container from this queue
      Parameters:
      clusterResource - the current cluster resource
      application - application to which the container was assigned
      container - the container to detach

    • attachContainer

      void attachContainer(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerApp application, RMContainer container)
      Attach a container to this queue
      Parameters:
      clusterResource - the current cluster resource
      application - application to which the container was assigned
      container - the container to attach

    • getPreemptionDisabled

      boolean getPreemptionDisabled()
      Check whether disable_preemption property is set for this queue
      Returns:
      true if disable_preemption is set, false if not

    • getIntraQueuePreemptionDisabled

      boolean getIntraQueuePreemptionDisabled()
      Check whether intra-queue preemption is disabled for this queue
      Returns:
      true if either intra-queue preemption or inter-queue preemption is disabled for this queue, false if neither is disabled.

    • getIntraQueuePreemptionDisabledInHierarchy

      boolean getIntraQueuePreemptionDisabledInHierarchy()
      Determines whether or not the intra-queue preemption disabled switch is set at any level in this queue's hierarchy.
      Returns:
      state of the intra-queue preemption switch at this queue level

    • getQueueCapacities

      QueueCapacities getQueueCapacities()
      Get QueueCapacities of this queue
      Returns:
      queueCapacities

    • getQueueResourceUsage

      ResourceUsage getQueueResourceUsage()
      Get ResourceUsage of this queue
      Returns:
      resourceUsage

    • incUsedResource

      void incUsedResource(String nodePartition, org.apache.hadoop.yarn.api.records.Resource resourceToInc, SchedulerApplicationAttempt application)
      When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
      Parameters:
      nodePartition - node label.
      resourceToInc - resource.
      application - application.

    • decUsedResource

      void decUsedResource(String nodePartition, org.apache.hadoop.yarn.api.records.Resource resourceToDec, SchedulerApplicationAttempt application)
      When partition of node updated, we will update queue's resource usage if it has container(s) running on that.
      Parameters:
      nodePartition - node label.
      resourceToDec - resource.
      application - application.

    • decPendingResource

      void decPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec)
      When an outstanding resource is fulfilled or canceled, calling this will decrease pending resource in a queue.
      Specified by:
      decPendingResource in interface Queue
      Parameters:
      nodeLabel - asked by application
      resourceToDec - new resource asked

    • getNodeLabelsForQueue

      Set<String> getNodeLabelsForQueue()
      Get valid Node Labels for this queue
      Returns:
      valid node labels

    • assignContainers

      @VisibleForTesting CSAssignment assignContainers(org.apache.hadoop.yarn.api.records.Resource clusterResource, FiCaSchedulerNode node, ResourceLimits resourceLimits, SchedulingMode schedulingMode)

    • accept

      boolean accept(org.apache.hadoop.yarn.api.records.Resource cluster, ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode> request)

    • apply

      void apply(org.apache.hadoop.yarn.api.records.Resource cluster, ResourceCommitRequest<FiCaSchedulerApp,FiCaSchedulerNode> request)

    • getReadLock

      ReentrantReadWriteLock.ReadLock getReadLock()
      Get readLock associated with the Queue.
      Returns:
      readLock of corresponding queue.

    • getWriteLock

      ReentrantReadWriteLock.WriteLock getWriteLock()
      Get writeLock associated with the Queue.
      Returns:
      writeLock of corresponding queue.

    • validateSubmitApplication

      void validateSubmitApplication(org.apache.hadoop.yarn.api.records.ApplicationId applicationId, String userName, String queue) throws org.apache.hadoop.security.AccessControlException
      Validate submitApplication api so that moveApplication do a pre-check.
      Parameters:
      applicationId - Application ID
      userName - User Name
      queue - Queue Name
      Throws:
      org.apache.hadoop.security.AccessControlException - if any acl violation is there.

    • getPriority

      org.apache.hadoop.yarn.api.records.Priority getPriority()
      Get priority of queue
      Returns:
      queue priority

    • getUserWeights

      UserWeights getUserWeights()
      Get the UserWeights object that wraps a map of usernames and weights
      Returns:
      The UserWeights object.

    • getQueueResourceQuotas

      QueueResourceQuotas getQueueResourceQuotas()
      Get QueueResourceQuotas associated with each queue.
      Returns:
      QueueResourceQuotas

    • getCapacityConfigType

      AbstractCSQueue.CapacityConfigType getCapacityConfigType()
      Get CapacityConfigType as PERCENTAGE or ABSOLUTE_RESOURCE.
      Returns:
      CapacityConfigType

    • getEffectiveCapacity

      org.apache.hadoop.yarn.api.records.Resource getEffectiveCapacity(String label)
      Get effective capacity of queue. If min/max resource is configured, preference will be given to absolute configuration over normal capacity.
      Parameters:
      label - partition
      Returns:
      effective queue capacity

    • getConfiguredCapacityVector

      QueueCapacityVector getConfiguredCapacityVector(String label)
      Get configured capacity vector parsed from the capacity config of the queue.
      Parameters:
      label - node label (partition)
      Returns:
      capacity resource vector

    • getConfiguredMaxCapacityVector

      QueueCapacityVector getConfiguredMaxCapacityVector(String label)
      Get configured maximum capacity vector parsed from the capacity config of the queue.
      Parameters:
      label - node label (partition)
      Returns:
      capacity resource vector

    • setConfiguredMinCapacityVector

      void setConfiguredMinCapacityVector(String label, QueueCapacityVector minCapacityVector)
      Sets the configured minimum capacity vector to a specific value.
      Parameters:
      label - node label (partition)
      minCapacityVector - capacity vector

    • setConfiguredMaxCapacityVector

      void setConfiguredMaxCapacityVector(String label, QueueCapacityVector maxCapacityVector)
      Sets the configured maximum capacity vector to a specific value.
      Parameters:
      label - node label (partition)
      maxCapacityVector - capacity vector

    • getConfiguredNodeLabels

      Set<String> getConfiguredNodeLabels()

    • getEffectiveCapacityDown

      org.apache.hadoop.yarn.api.records.Resource getEffectiveCapacityDown(String label, org.apache.hadoop.yarn.api.records.Resource factor)
      Get effective capacity of queue. If min/max resource is configured, preference will be given to absolute configuration over normal capacity. Also round down the result to normalizeDown.
      Parameters:
      label - partition
      factor - factor to normalize down
      Returns:
      effective queue capacity

    • getEffectiveMaxCapacity

      org.apache.hadoop.yarn.api.records.Resource getEffectiveMaxCapacity(String label)
      Get effective max capacity of queue. If min/max resource is configured, preference will be given to absolute configuration over normal capacity.
      Parameters:
      label - partition
      Returns:
      effective max queue capacity

    • getEffectiveMaxCapacityDown

      org.apache.hadoop.yarn.api.records.Resource getEffectiveMaxCapacityDown(String label, org.apache.hadoop.yarn.api.records.Resource factor)
      Get effective max capacity of queue. If min/max resource is configured, preference will be given to absolute configuration over normal capacity. Also round down the result to normalizeDown.
      Parameters:
      label - partition
      factor - factor to normalize down
      Returns:
      effective max queue capacity

    • getMultiNodeSortingPolicyClassName

      String getMultiNodeSortingPolicyClassName()
      Get Multi Node scheduling policy name.
      Returns:
      policy name

    • getMaximumApplicationLifetime

      long getMaximumApplicationLifetime()
      Get the maximum lifetime in seconds of an application which is submitted to this queue. Apps can set their own lifetime timeout up to this value.
      Returns:
      max lifetime in seconds

    • getDefaultApplicationLifetime

      long getDefaultApplicationLifetime()
      Get the default lifetime in seconds of an application which is submitted to this queue. If an app doesn't specify its own timeout when submitted, this value will be used.
      Returns:
      default app lifetime

    • getDefaultAppLifetimeWasSpecifiedInConfig

      boolean getDefaultAppLifetimeWasSpecifiedInConfig()
      Get the indicator of whether or not the default application lifetime was set by a config property or was calculated by the capacity scheduler.
      Returns:
      indicator whether set or calculated