Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair

Class FairSchedulerConfiguration

java.lang.Object
org.apache.hadoop.conf.Configuration
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerConfiguration
All Implemented Interfaces:
Iterable<Map.Entry<String,String>>, org.apache.hadoop.io.Writable
@Private @Evolving public class FairSchedulerConfiguration extends org.apache.hadoop.conf.Configuration

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

    • RM_SCHEDULER_INCREMENT_ALLOCATION_MB

      @Deprecated public static final String RM_SCHEDULER_INCREMENT_ALLOCATION_MB
      Deprecated.
      The preferred way to configure the increment is by using the yarn.resource-types.{RESOURCE_NAME}.increment-allocation property, for memory: yarn.resource-types.memory-mb.increment-allocation
      Resource Increment request grant-able by the FairScheduler. This property is looked up in the yarn-site.xml.
      See Also:

    • DEFAULT_RM_SCHEDULER_INCREMENT_ALLOCATION_MB

      @Deprecated public static final int DEFAULT_RM_SCHEDULER_INCREMENT_ALLOCATION_MB
      Deprecated.
      See Also:

    • RM_SCHEDULER_INCREMENT_ALLOCATION_VCORES

      @Deprecated public static final String RM_SCHEDULER_INCREMENT_ALLOCATION_VCORES
      Deprecated.
      The preferred way to configure the increment is by using the yarn.resource-types.{RESOURCE_NAME}.increment-allocation property, for CPU: yarn.resource-types.vcores.increment-allocation
      Resource Increment request grant-able by the FairScheduler. This property is looked up in the yarn-site.xml.
      See Also:

    • DEFAULT_RM_SCHEDULER_INCREMENT_ALLOCATION_VCORES

      @Deprecated public static final int DEFAULT_RM_SCHEDULER_INCREMENT_ALLOCATION_VCORES
      Deprecated.
      See Also:

    • RM_SCHEDULER_RESERVATION_THRESHOLD_INCREMENT_MULTIPLE

      public static final String RM_SCHEDULER_RESERVATION_THRESHOLD_INCREMENT_MULTIPLE
      Threshold for container size for making a container reservation as a multiple of increment allocation. Only container sizes above this are allowed to reserve a node
      See Also:

    • DEFAULT_RM_SCHEDULER_RESERVATION_THRESHOLD_INCREMENT_MULTIPLE

      public static final float DEFAULT_RM_SCHEDULER_RESERVATION_THRESHOLD_INCREMENT_MULTIPLE
      See Also:

    • MIGRATION_MODE

      public static final String MIGRATION_MODE
      Used during FS->CS conversion. When enabled, background threads are not started. This property should NOT be used by end-users!
      See Also:

    • NO_TERMINAL_RULE_CHECK

      public static final String NO_TERMINAL_RULE_CHECK
      Disables checking whether a placement rule is terminal or not. Only used during migration mode. This property should NOT be used by end users!
      See Also:

    • ALLOCATION_FILE

      public static final String ALLOCATION_FILE
      See Also:

    • DEFAULT_ALLOCATION_FILE

      protected static final String DEFAULT_ALLOCATION_FILE
      See Also:

    • ALLOW_UNDECLARED_POOLS

      public static final String ALLOW_UNDECLARED_POOLS
      Whether pools can be created that were not specified in the FS configuration file
      See Also:

    • DEFAULT_ALLOW_UNDECLARED_POOLS

      public static final boolean DEFAULT_ALLOW_UNDECLARED_POOLS
      See Also:

    • USER_AS_DEFAULT_QUEUE

      public static final String USER_AS_DEFAULT_QUEUE
      Whether to use the user name as the queue name (instead of "default") if the request does not specify a queue.
      See Also:

    • DEFAULT_USER_AS_DEFAULT_QUEUE

      public static final boolean DEFAULT_USER_AS_DEFAULT_QUEUE
      See Also:

    • DEFAULT_LOCALITY_THRESHOLD

      protected static final float DEFAULT_LOCALITY_THRESHOLD
      See Also:

    • LOCALITY_THRESHOLD_NODE

      public static final String LOCALITY_THRESHOLD_NODE
      Cluster threshold for node locality.
      See Also:

    • DEFAULT_LOCALITY_THRESHOLD_NODE

      public static final float DEFAULT_LOCALITY_THRESHOLD_NODE
      See Also:

    • LOCALITY_THRESHOLD_RACK

      public static final String LOCALITY_THRESHOLD_RACK
      Cluster threshold for rack locality.
      See Also:

    • DEFAULT_LOCALITY_THRESHOLD_RACK

      public static final float DEFAULT_LOCALITY_THRESHOLD_RACK
      See Also:

    • LOCALITY_DELAY_NODE_MS

      @Deprecated protected static final String LOCALITY_DELAY_NODE_MS
      Deprecated.
      Continuous scheduling is known to cause locking issue inside Only used when CONTINUOUS_SCHEDULING_ENABLED is enabled
      Delay for node locality.
      See Also:

    • DEFAULT_LOCALITY_DELAY_NODE_MS

      @Deprecated protected static final long DEFAULT_LOCALITY_DELAY_NODE_MS
      Deprecated.
      See Also:

    • LOCALITY_DELAY_RACK_MS

      @Deprecated protected static final String LOCALITY_DELAY_RACK_MS
      Deprecated.
      Continuous scheduling is known to cause locking issue inside Only used when CONTINUOUS_SCHEDULING_ENABLED is enabled
      Delay for rack locality.
      See Also:

    • DEFAULT_LOCALITY_DELAY_RACK_MS

      @Deprecated protected static final long DEFAULT_LOCALITY_DELAY_RACK_MS
      Deprecated.
      See Also:

    • CONTINUOUS_SCHEDULING_ENABLED

      @Deprecated public static final String CONTINUOUS_SCHEDULING_ENABLED
      Deprecated.
      Continuous scheduling is known to cause locking issue inside the scheduler in larger cluster, more than 100 nodes, use ASSIGN_MULTIPLE to improve container allocation ramp up.
      Enable continuous scheduling or not.
      See Also:

    • DEFAULT_CONTINUOUS_SCHEDULING_ENABLED

      @Deprecated public static final boolean DEFAULT_CONTINUOUS_SCHEDULING_ENABLED
      Deprecated.
      See Also:

    • CONTINUOUS_SCHEDULING_SLEEP_MS

      @Deprecated public static final String CONTINUOUS_SCHEDULING_SLEEP_MS
      Deprecated.
      Continuous scheduling is known to cause locking issue inside Only used when CONTINUOUS_SCHEDULING_ENABLED is enabled
      Sleep time of each pass in continuous scheduling (5ms in default).
      See Also:

    • DEFAULT_CONTINUOUS_SCHEDULING_SLEEP_MS

      @Deprecated public static final int DEFAULT_CONTINUOUS_SCHEDULING_SLEEP_MS
      Deprecated.
      See Also:

    • PREEMPTION

      public static final String PREEMPTION
      Whether preemption is enabled.
      See Also:

    • DEFAULT_PREEMPTION

      public static final boolean DEFAULT_PREEMPTION
      See Also:

    • AM_PREEMPTION

      protected static final String AM_PREEMPTION
      See Also:

    • AM_PREEMPTION_PREFIX

      protected static final String AM_PREEMPTION_PREFIX
      See Also:

    • DEFAULT_AM_PREEMPTION

      protected static final boolean DEFAULT_AM_PREEMPTION
      See Also:

    • PREEMPTION_THRESHOLD

      protected static final String PREEMPTION_THRESHOLD
      See Also:

    • DEFAULT_PREEMPTION_THRESHOLD

      protected static final float DEFAULT_PREEMPTION_THRESHOLD
      See Also:

    • WAIT_TIME_BEFORE_KILL

      public static final String WAIT_TIME_BEFORE_KILL
      See Also:

    • DEFAULT_WAIT_TIME_BEFORE_KILL

      public static final int DEFAULT_WAIT_TIME_BEFORE_KILL
      See Also:

    • WAIT_TIME_BEFORE_NEXT_STARVATION_CHECK_MS

      public static final String WAIT_TIME_BEFORE_NEXT_STARVATION_CHECK_MS
      Configurable delay (ms) before an app's starvation is considered after it is identified. This is to give the scheduler enough time to allocate containers post preemption. This delay is added to the WAIT_TIME_BEFORE_KILL and enough heartbeats. This is intended to be a backdoor on production clusters, and hence intentionally not documented.
      See Also:

    • DEFAULT_WAIT_TIME_BEFORE_NEXT_STARVATION_CHECK_MS

      public static final long DEFAULT_WAIT_TIME_BEFORE_NEXT_STARVATION_CHECK_MS
      See Also:

    • ASSIGN_MULTIPLE

      public static final String ASSIGN_MULTIPLE
      Whether to assign multiple containers in one check-in.
      See Also:

    • DEFAULT_ASSIGN_MULTIPLE

      public static final boolean DEFAULT_ASSIGN_MULTIPLE
      See Also:

    • SIZE_BASED_WEIGHT

      public static final String SIZE_BASED_WEIGHT
      Whether to give more weight to apps requiring many resources.
      See Also:

    • DEFAULT_SIZE_BASED_WEIGHT

      public static final boolean DEFAULT_SIZE_BASED_WEIGHT
      See Also:

    • DYNAMIC_MAX_ASSIGN

      public static final String DYNAMIC_MAX_ASSIGN
      Maximum number of containers to assign on each check-in.
      See Also:

    • MAX_ASSIGN

      public static final String MAX_ASSIGN
      Specify exact number of containers to assign on each heartbeat, if dynamic max assign is turned off.
      See Also:

    • DEFAULT_MAX_ASSIGN

      public static final int DEFAULT_MAX_ASSIGN
      See Also:

    • UPDATE_INTERVAL_MS

      public static final String UPDATE_INTERVAL_MS
      The update interval for calculating resources in FairScheduler .
      See Also:

    • DEFAULT_UPDATE_INTERVAL_MS

      public static final int DEFAULT_UPDATE_INTERVAL_MS
      See Also:

    • RESERVABLE_NODES

      public static final String RESERVABLE_NODES
      Ratio of nodes available for an app to make an reservation on.
      See Also:

    • RESERVABLE_NODES_DEFAULT

      public static final float RESERVABLE_NODES_DEFAULT
      See Also:

  • Constructor Details

    • FairSchedulerConfiguration

      public FairSchedulerConfiguration()

    • FairSchedulerConfiguration

      public FairSchedulerConfiguration(org.apache.hadoop.conf.Configuration conf)

  • Method Details

    • getMinimumAllocation

      public org.apache.hadoop.yarn.api.records.Resource getMinimumAllocation()

    • getMaximumAllocation

      public org.apache.hadoop.yarn.api.records.Resource getMaximumAllocation()

    • getIncrementAllocation

      public org.apache.hadoop.yarn.api.records.Resource getIncrementAllocation()

    • getReservationThresholdIncrementMultiple

      public float getReservationThresholdIncrementMultiple()

    • getLocalityThresholdNode

      public float getLocalityThresholdNode()

    • getLocalityThresholdRack

      public float getLocalityThresholdRack()

    • isContinuousSchedulingEnabled

      @Deprecated public boolean isContinuousSchedulingEnabled()
      Deprecated.
      use ASSIGN_MULTIPLE to improve container allocation ramp up.
      Whether continuous scheduling is turned on.
      Returns:
      whether continuous scheduling is enabled

    • getContinuousSchedulingSleepMs

      @Deprecated public int getContinuousSchedulingSleepMs()
      Deprecated.
      linked to CONTINUOUS_SCHEDULING_ENABLED deprecation
      The sleep time of the continuous scheduler thread.
      Returns:
      sleep time in ms

    • getLocalityDelayNodeMs

      @Deprecated public long getLocalityDelayNodeMs()
      Deprecated.
      linked to CONTINUOUS_SCHEDULING_ENABLED deprecation
      Delay in milliseconds for locality fallback node to rack.
      Returns:
      delay in ms

    • getLocalityDelayRackMs

      @Deprecated public long getLocalityDelayRackMs()
      Deprecated.
      linked to CONTINUOUS_SCHEDULING_ENABLED deprecation
      Delay in milliseconds for locality fallback rack to other.
      Returns:
      delay in ms

    • getPreemptionEnabled

      public boolean getPreemptionEnabled()

    • getAMPreemptionEnabled

      public boolean getAMPreemptionEnabled(String queueName)

    • getPreemptionUtilizationThreshold

      public float getPreemptionUtilizationThreshold()

    • getAssignMultiple

      public boolean getAssignMultiple()

    • isMaxAssignDynamic

      public boolean isMaxAssignDynamic()

    • getMaxAssign

      public int getMaxAssign()

    • getSizeBasedWeight

      public boolean getSizeBasedWeight()

    • getWaitTimeBeforeNextStarvationCheck

      public long getWaitTimeBeforeNextStarvationCheck()

    • getWaitTimeBeforeKill

      public int getWaitTimeBeforeKill()

    • getUsePortForNodeName

      public boolean getUsePortForNodeName()

    • getReservableNodes

      public float getReservableNodes()

    • parseResourceConfigValue

      public static ConfigurableResource parseResourceConfigValue(String value) throws AllocationConfigurationException
      Parses a resource config value in one of three forms:
      1. Percentage: "50%" or "40% memory, 60% cpu"
      2. New style resources: "vcores=10, memory-mb=1024" or "vcores=60%, memory-mb=40%"
      3. Old style resources: "1024 mb, 10 vcores"
      In new style resources, any resource that is not specified will be set to Long.MAX_VALUE or 100%, as appropriate. Also, in the new style resources, units are not allowed. Units are assumed from the resource manager's settings for the resources when the value isn't a percentage.
      Parameters:
      value - the resource definition to parse
      Returns:
      a ConfigurableResource that represents the parsed value
      Throws:
      AllocationConfigurationException - if the raw value is not a valid resource definition

    • parseResourceConfigValue

      public static ConfigurableResource parseResourceConfigValue(String value, long missing) throws AllocationConfigurationException
      Parses a resource config value in one of three forms:
      1. Percentage: "50%" or "40% memory, 60% cpu"
      2. New style resources: "vcores=10, memory-mb=1024" or "vcores=60%, memory-mb=40%"
      3. Old style resources: "1024 mb, 10 vcores"
      In new style resources, any resource that is not specified will be set to missing or 0%, as appropriate. Also, in the new style resources, units are not allowed. Units are assumed from the resource manager's settings for the resources when the value isn't a percentage. The missing parameter is only used in the case of new style resources without percentages. With new style resources with percentages, any missing resources will be assumed to be 100% because percentages are only used with maximum resource limits.
      Parameters:
      value - the resource definition to parse
      missing - the value to use for any unspecified resources
      Returns:
      a ConfigurableResource that represents the parsed value
      Throws:
      AllocationConfigurationException - if the raw value is not a valid resource definition

    • getUpdateInterval

      public long getUpdateInterval()