Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair

Class FSQueue

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue
All Implemented Interfaces:
Schedulable, Queue
Direct Known Subclasses:
FSLeafQueue, FSParentQueue
@Private @Unstable public abstract class FSQueue extends Object implements Queue, Schedulable

  • Field Details

    • scheduler

      protected final FairScheduler scheduler

    • parent

      protected final FSParentQueue parent

    • recordFactory

      protected final org.apache.hadoop.yarn.factories.RecordFactory recordFactory

    • policy

      protected SchedulingPolicy policy

    • weights

      protected float weights

    • minShare

      protected org.apache.hadoop.yarn.api.records.Resource minShare

    • maxRunningApps

      protected int maxRunningApps

    • maxAMShare

      protected float maxAMShare

    • maxContainerAllocation

      protected org.apache.hadoop.yarn.api.records.Resource maxContainerAllocation

  • Constructor Details

  • Method Details

    • reinit

      public final void reinit(boolean recursive)
      Initialize a queue by setting its queue-specific properties and its metrics. This method is invoked when creating a new queue or reloading the allocation file. This method does not set policies for queues when reloading the allocation file since we need to either set all new policies or nothing, which is handled by method verifyAndSetPolicyFromConf(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfiguration).
      Parameters:
      recursive - whether child queues should be reinitialized recursively

    • getName

      public String getName()
      Description copied from interface: Schedulable
      Name of job/queue, used for debugging as well as for breaking ties in scheduling order deterministically.
      Specified by:
      getName in interface Schedulable
      Returns:
      Name of job/queue.

    • getQueueName

      public String getQueueName()
      Description copied from interface: Queue
      Get the queue name
      Specified by:
      getQueueName in interface Queue
      Returns:
      queue name

    • getPolicy

      public SchedulingPolicy getPolicy()

    • getParent

      public FSParentQueue getParent()

    • setPolicy

      public void setPolicy(SchedulingPolicy policy)

    • setWeights

      public void setWeights(float weights)

    • getWeight

      public float getWeight()
      Description copied from interface: Schedulable
      Job/queue weight in fair sharing. Weights are only meaningful when compared. A weight of 2.0f has twice the weight of a weight of 1.0f, which has twice the weight of a weight of 0.5f. A weight of 1.0f is considered unweighted or a neutral weight. A weight of 0 is no weight.
      Specified by:
      getWeight in interface Schedulable
      Returns:
      the weight

    • setMinShare

      public void setMinShare(org.apache.hadoop.yarn.api.records.Resource minShare)

    • getMinShare

      public org.apache.hadoop.yarn.api.records.Resource getMinShare()
      Description copied from interface: Schedulable
      Minimum Resource share assigned to the schedulable.
      Specified by:
      getMinShare in interface Schedulable
      Returns:
      Minimum Resource share.

    • setMaxShare

      public void setMaxShare(ConfigurableResource maxShare)

    • setMaxContainerAllocation

      public void setMaxContainerAllocation(org.apache.hadoop.yarn.api.records.Resource maxContainerAllocation)

    • getMaximumContainerAllocation

      public abstract org.apache.hadoop.yarn.api.records.Resource getMaximumContainerAllocation()

    • getMaxShare

      public org.apache.hadoop.yarn.api.records.Resource getMaxShare()
      Description copied from interface: Schedulable
      Maximum Resource share assigned to the schedulable.
      Specified by:
      getMaxShare in interface Schedulable
      Returns:
      Maximum Resource share.

    • getRawMaxShare

      public ConfigurableResource getRawMaxShare()

    • getReservedResource

      public org.apache.hadoop.yarn.api.records.Resource getReservedResource()

    • setMaxChildQueueResource

      public void setMaxChildQueueResource(ConfigurableResource maxChildShare)

    • getMaxChildQueueResource

      public ConfigurableResource getMaxChildQueueResource()

    • setMaxRunningApps

      public void setMaxRunningApps(int maxRunningApps)

    • getMaxRunningApps

      public int getMaxRunningApps()

    • getMaxAMShare

      @VisibleForTesting public float getMaxAMShare()

    • setMaxAMShare

      public void setMaxAMShare(float maxAMShare)

    • getStartTime

      public long getStartTime()
      Description copied from interface: Schedulable
      Start time for jobs in FIFO queues; meaningless for QueueSchedulables.
      Specified by:
      getStartTime in interface Schedulable
      Returns:
      Start time for jobs.

    • getPriority

      public org.apache.hadoop.yarn.api.records.Priority getPriority()
      Description copied from interface: Schedulable
      Job priority for jobs in FIFO queues; meaningless for QueueSchedulables.
      Specified by:
      getPriority in interface Schedulable
      Returns:
      Job priority.

    • getQueueInfo

      public org.apache.hadoop.yarn.api.records.QueueInfo getQueueInfo(boolean includeChildQueues, boolean recursive)
      Description copied from interface: Queue
      Get queue information
      Specified by:
      getQueueInfo in interface Queue
      Parameters:
      includeChildQueues - include child queues?
      recursive - recursively get child queue information?
      Returns:
      queue information

    • getQueueStatistics

      public org.apache.hadoop.yarn.api.records.QueueStatistics getQueueStatistics()

    • getMetrics

      public FSQueueMetrics getMetrics()
      Description copied from interface: Queue
      Get the queue metrics
      Specified by:
      getMetrics in interface Queue
      Returns:
      the queue metrics

    • getFairShare

      public org.apache.hadoop.yarn.api.records.Resource getFairShare()
      Get the fair share assigned to this Schedulable.
      Specified by:
      getFairShare in interface Schedulable
      Returns:
      the fair share assigned to this Schedulable.

    • setFairShare

      public void setFairShare(org.apache.hadoop.yarn.api.records.Resource fairShare)
      Description copied from interface: Schedulable
      Assign a fair share to this Schedulable.
      Specified by:
      setFairShare in interface Schedulable
      Parameters:
      fairShare - a fair share to this Schedulable.

    • getSteadyFairShare

      public org.apache.hadoop.yarn.api.records.Resource getSteadyFairShare()
      Get the steady fair share assigned to this Schedulable.
      Returns:
      the steady fair share assigned to this Schedulable.

    • hasAccess

      public boolean hasAccess(org.apache.hadoop.yarn.api.records.QueueACL acl, org.apache.hadoop.security.UserGroupInformation user)
      Specified by:
      hasAccess in interface Queue

    • isPreemptable

      public boolean isPreemptable()
      Description copied from interface: Schedulable
      Check whether the schedulable is preemptable.
      Specified by:
      isPreemptable in interface Schedulable
      Returns:
      true if the schedulable is preemptable; false otherwise

    • update

      public void update(org.apache.hadoop.yarn.api.records.Resource fairShare)
      Set the queue's fairshare and update the demand/fairshare of child queues/applications. To be called holding the scheduler writelock.
      Parameters:
      fairShare - queue's fairshare.

    • getChildQueues

      public abstract List<FSQueue> getChildQueues()
      Gets the children of this queue, if any.
      Returns:
      the children of this queue.

    • collectSchedulerApplications

      public abstract void collectSchedulerApplications(Collection<org.apache.hadoop.yarn.api.records.ApplicationAttemptId> apps)
      Adds all applications in the queue and its subqueues to the given collection.
      Parameters:
      apps - the collection to add the applications to

    • getNumRunnableApps

      public abstract int getNumRunnableApps()
      Return the number of apps for which containers can be allocated. Includes apps in subqueues.
      Returns:
      the number of apps.

    • isActive

      public boolean isActive()
      Returns true if queue has at least one app running.
      Returns:
      true, if queue has at least one app running; otherwise, false;

    • toString

      public String toString()
      Convenient toString implementation for debugging.
      Overrides:
      toString in class Object

    • getAccessibleNodeLabels

      public Set<String> getAccessibleNodeLabels()
      Description copied from interface: Queue
      Get labels can be accessed of this queue labels={*}, means this queue can access any label labels={ }, means this queue cannot access any label except node without label labels={a, b, c} means this queue can access a or b or c
      Specified by:
      getAccessibleNodeLabels in interface Queue
      Returns:
      labels

    • getDefaultNodeLabelExpression

      public String getDefaultNodeLabelExpression()
      Description copied from interface: Queue
      Get default label expression of this queue. If label expression of ApplicationSubmissionContext and label expression of Resource Request not set, this will be used.
      Specified by:
      getDefaultNodeLabelExpression in interface Queue
      Returns:
      default label expression

    • incPendingResource

      public void incPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc)
      Description copied from interface: Queue
      When new outstanding resource is asked, calling this will increase pending resource in a queue.
      Specified by:
      incPendingResource in interface Queue
      Parameters:
      nodeLabel - asked by application
      resourceToInc - new resource asked

    • decPendingResource

      public void decPendingResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec)
      Description copied from interface: Queue
      When an outstanding resource is fulfilled or canceled, calling this will decrease pending resource in a queue.
      Specified by:
      decPendingResource in interface Queue
      Parameters:
      nodeLabel - asked by application
      resourceToDec - new resource asked

    • incReservedResource

      public void incReservedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToInc)
      Description copied from interface: Queue
      Increment Reserved Capacity
      Specified by:
      incReservedResource in interface Queue
      Parameters:
      nodeLabel - asked by application
      resourceToInc - reserved resource asked

    • decReservedResource

      public void decReservedResource(String nodeLabel, org.apache.hadoop.yarn.api.records.Resource resourceToDec)
      Description copied from interface: Queue
      Decrement Reserved Capacity
      Specified by:
      decReservedResource in interface Queue
      Parameters:
      nodeLabel - asked by application
      resourceToDec - reserved resource asked

    • getResourceUsage

      public org.apache.hadoop.yarn.api.records.Resource getResourceUsage()
      Description copied from interface: Schedulable
      Get the aggregate amount of resources consumed by the schedulable.
      Specified by:
      getResourceUsage in interface Schedulable
      Returns:
      aggregate amount of resources.

    • incUsedResource

      public void incUsedResource(org.apache.hadoop.yarn.api.records.Resource res)
      Increase resource usage for this queue and all parent queues.
      Parameters:
      res - the resource to increase

    • decUsedResource

      protected void decUsedResource(org.apache.hadoop.yarn.api.records.Resource res)
      Decrease resource usage for this queue and all parent queues.
      Parameters:
      res - the resource to decrease

    • getDefaultApplicationPriority

      public org.apache.hadoop.yarn.api.records.Priority getDefaultApplicationPriority()
      Description copied from interface: Queue
      Get the Default Application Priority for this queue
      Specified by:
      getDefaultApplicationPriority in interface Queue
      Returns:
      default application priority

    • verifyAndSetPolicyFromConf

      public boolean verifyAndSetPolicyFromConf(AllocationConfiguration queueConf)
      Recursively check policies for queues in pre-order. Get queue policies from the allocation file instead of properties of FSQueue objects. Set the policy for current queue if there is no policy violation for its children. This method is invoked while reloading the allocation file.
      Parameters:
      queueConf - allocation configuration
      Returns:
      true if no policy violation and successfully set polices for queues; false otherwise

    • dumpState

      public String dumpState()
      Recursively dump states of all queues.
      Returns:
      a string which holds all queue states

    • dumpStateInternal

      protected abstract void dumpStateInternal(StringBuilder sb)
      Recursively dump states of all queues.
      Parameters:
      sb - the {code StringBuilder} which holds queue states

    • isDynamic

      public boolean isDynamic()

    • setDynamic

      public void setDynamic(boolean dynamic)

    • isEmpty

      public abstract boolean isEmpty()