Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair

Class SchedulingPolicy

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.SchedulingPolicy
Direct Known Subclasses:
DominantResourceFairnessPolicy, FairSharePolicy, FifoPolicy
@Public @Evolving public abstract class SchedulingPolicy extends Object
The SchedulingPolicy is used by the fair scheduler mainly to determine what a queue's fair share and steady fair share should be as well as calculating available headroom. This determines how resources can be shared between running applications within a queue.

Every queue has a policy, including parents and children. If a child queue doesn't specify one, it inherits the parent's policy. The policy for a child queue must be compatible with the policy of the parent queue; there are some combinations that aren't allowed. See isChildPolicyAllowed(SchedulingPolicy). The policy for a queue is specified by setting property schedulingPolicy in the fair scheduler configuration file. The default policy is FairSharePolicy if not specified.

  • Field Details

  • Constructor Details

    • SchedulingPolicy

      public SchedulingPolicy()

  • Method Details

    • getInstance

      public static SchedulingPolicy getInstance(Class<? extends SchedulingPolicy> clazz)
      Returns a SchedulingPolicy instance corresponding to the passed clazz.
      Parameters:
      clazz - a class that extends SchedulingPolicy
      Returns:
      a SchedulingPolicy instance

    • parse

      public static SchedulingPolicy parse(String policy) throws AllocationConfigurationException
      Returns SchedulingPolicy instance corresponding to the SchedulingPolicy passed as a string. The policy can be "fair" for FairSharePolicy, "fifo" for FifoPolicy, or "drf" for DominantResourceFairnessPolicy. For a custom SchedulingPolicys in the RM classpath, the policy should be canonical class name of the SchedulingPolicy.
      Parameters:
      policy - canonical class name or "drf" or "fair" or "fifo"
      Returns:
      a SchedulingPolicy instance parsed from given policy
      Throws:
      AllocationConfigurationException - for any errors.

    • initialize

      @Deprecated public void initialize(org.apache.hadoop.yarn.api.records.Resource clusterCapacity)
      Deprecated.
      Since it doesn't track cluster resource changes, replaced by initialize(FSContext).
      Initialize the scheduling policy with cluster resources.
      Parameters:
      clusterCapacity - cluster resources

    • initialize

      public void initialize(FSContext fsContext)
      Initialize the scheduling policy with a FSContext object, which has a pointer to the cluster resources among other information.
      Parameters:
      fsContext - a FSContext object which has a pointer to the cluster resources

    • getResourceCalculator

      public abstract org.apache.hadoop.yarn.util.resource.ResourceCalculator getResourceCalculator()
      The ResourceCalculator returned by this method should be used for any calculations involving resources.
      Returns:
      ResourceCalculator instance to use

    • getName

      public abstract String getName()
      Returns:
      returns the name of SchedulingPolicy

    • getComparator

      public abstract Comparator<Schedulable> getComparator()
      The comparator returned by this method is to be used for sorting the Schedulables in that queue.
      Returns:
      the comparator to sort by

    • computeShares

      public abstract void computeShares(Collection<? extends Schedulable> schedulables, org.apache.hadoop.yarn.api.records.Resource totalResources)
      Computes and updates the shares of Schedulables as per the SchedulingPolicy, to be used later for scheduling decisions. The shares computed are instantaneous and only consider queues with running applications.
      Parameters:
      schedulables - Schedulables whose shares are to be updated
      totalResources - Total Resources in the cluster

    • computeSteadyShares

      public abstract void computeSteadyShares(Collection<? extends FSQueue> queues, org.apache.hadoop.yarn.api.records.Resource totalResources)
      Computes and updates the steady shares of FSQueues as per the SchedulingPolicy. The steady share does not differentiate between queues with and without running applications under them. The steady share is not used for scheduling, it is displayed on the Web UI for better visibility.
      Parameters:
      queues - FSQueues whose shares are to be updated
      totalResources - Total Resources in the cluster

    • checkIfUsageOverFairShare

      public abstract boolean checkIfUsageOverFairShare(org.apache.hadoop.yarn.api.records.Resource usage, org.apache.hadoop.yarn.api.records.Resource fairShare)
      Check if the resource usage is over the fair share under this policy.
      Parameters:
      usage - Resource the resource usage
      fairShare - Resource the fair share
      Returns:
      true if check passes (is over) or false otherwise

    • getHeadroom

      public abstract org.apache.hadoop.yarn.api.records.Resource getHeadroom(org.apache.hadoop.yarn.api.records.Resource queueFairShare, org.apache.hadoop.yarn.api.records.Resource queueUsage, org.apache.hadoop.yarn.api.records.Resource maxAvailable)
      Get headroom by calculating the min of clusterAvailable and (queueFairShare - queueUsage) resources that are applicable to this policy. For eg if only memory then leave other resources such as CPU to same as clusterAvailable.
      Parameters:
      queueFairShare - fairshare in the queue
      queueUsage - resources used in the queue
      maxAvailable - available resource in cluster for this queue
      Returns:
      calculated headroom

    • isChildPolicyAllowed

      public boolean isChildPolicyAllowed(SchedulingPolicy childPolicy)
      Check whether the policy of a child queue is allowed.
      Parameters:
      childPolicy - the policy of child queue
      Returns:
      true if the child policy is allowed; false otherwise