Package org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator

Class LoadBasedGlobalPolicy

java.lang.Object
org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.GlobalPolicy
org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.LoadBasedGlobalPolicy
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable
public class LoadBasedGlobalPolicy extends GlobalPolicy
Load based policy that generates weighted policies by scaling the cluster load (based on pending) to a weight from 0.0 to 1.0.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static enum 
    LoadBasedGlobalPolicy.Scaling
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    LoadBasedGlobalPolicy()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo>
    getSubClustersMetricsInfo(Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,Map<Class,Object>> subClusterMetricsInfo)
    Get the ClusterMetric information of the subCluster.
    protected Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterIdInfo,Float>
    getTargetWeights(Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo> clusterMetrics)
    Get subCluster target weight.
    protected float
    getWeightByScaling(int maxPendingVal, int curPendingVal)
    Get weight information.
    protected org.apache.hadoop.yarn.server.federation.policies.manager.WeightedLocalityPolicyManager
    getWeightedLocalityPolicyManager(String queue, Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,Map<Class,Object>> subClusterMetricInfos)
    GPG can help update the policy of the queue.
    protected Map<Class<?>,String>
    registerPaths()
    Return a map of the object type and RM path to request it from - the framework will query these paths and provide the objects to the policy.
    void
    setConf(org.apache.hadoop.conf.Configuration conf)
     
    protected org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager
    updatePolicy(String queueName, Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,Map<Class,Object>> clusterInfo, org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager currentManager)
    Update the policy of the queue.

    Methods inherited from class org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.GlobalPolicy

    getConf

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • LoadBasedGlobalPolicy

      public LoadBasedGlobalPolicy()

  • Method Details

    • setConf

      public void setConf(org.apache.hadoop.conf.Configuration conf)
      Specified by:
      setConf in interface org.apache.hadoop.conf.Configurable
      Overrides:
      setConf in class GlobalPolicy

    • registerPaths

      protected Map<Class<?>,String> registerPaths()
      Description copied from class: GlobalPolicy
      Return a map of the object type and RM path to request it from - the framework will query these paths and provide the objects to the policy. Delegating this responsibility to the PolicyGenerator enables us to avoid duplicate calls to the same * endpoints as the GlobalPolicy is invoked once per queue.
      Overrides:
      registerPaths in class GlobalPolicy
      Returns:
      a map of the object type and RM path.

    • updatePolicy

      protected org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager updatePolicy(String queueName, Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,Map<Class,Object>> clusterInfo, org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager currentManager)
      Update the policy of the queue.
      Specified by:
      updatePolicy in class GlobalPolicy
      Parameters:
      queueName - name of the queue
      clusterInfo - subClusterId map to cluster information about the SubCluster used to make policy decisions
      currentManager - the FederationPolicyManager for the queue's existing policy the manager may be null, in which case the policy will need to be created.
      Returns:
      FederationPolicyManager.

    • getWeightedLocalityPolicyManager

      protected org.apache.hadoop.yarn.server.federation.policies.manager.WeightedLocalityPolicyManager getWeightedLocalityPolicyManager(String queue, Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,Map<Class,Object>> subClusterMetricInfos)
      GPG can help update the policy of the queue. We automatically generate the weight of the subCluster according to the clusterMetrics of the subCluster.
      Parameters:
      queue - queueName.
      subClusterMetricInfos - Metric information of the subCluster.
      Returns:
      WeightedLocalityPolicyManager.

    • getSubClustersMetricsInfo

      protected Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo> getSubClustersMetricsInfo(Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,Map<Class,Object>> subClusterMetricsInfo)
      Get the ClusterMetric information of the subCluster.
      Parameters:
      subClusterMetricsInfo - subCluster Metric Information.
      Returns:
      Mapping relationship between subCluster and Metric.

    • getTargetWeights

      @VisibleForTesting protected Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterIdInfo,Float> getTargetWeights(Map<org.apache.hadoop.yarn.server.federation.store.records.SubClusterId,org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo> clusterMetrics)
      Get subCluster target weight.
      Parameters:
      clusterMetrics - Metric of the subCluster.
      Returns:
      subCluster Weights.

    • getWeightByScaling

      protected float getWeightByScaling(int maxPendingVal, int curPendingVal)
      Get weight information. We will calculate the weight information according to different Scaling. NONE: No calculation is required, and the weight is 1 at this time. LINEAR: For linear computation, we will use (maxPendingVal - curPendingVal) / (maxPendingVal). QUADRATIC: Calculated using quadratic, We will calculate quadratic for maxPendingVal, curPendingVal, then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal). LOG(LOGARITHM): Calculated using logarithm, We will calculate logarithm for maxPendingVal, curPendingVal, then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal).
      Parameters:
      maxPendingVal - maxPending - minPending
      curPendingVal - pending - minPending
      Returns:
      Calculated weight information.