Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.distributed

Class NodeQueueLoadMonitor

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.distributed.NodeQueueLoadMonitor
All Implemented Interfaces:
ClusterMonitor
public class NodeQueueLoadMonitor extends Object implements ClusterMonitor
The NodeQueueLoadMonitor keeps track of load metrics (such as queue length and total wait time) associated with Container Queues on the Node Manager. It uses this information to periodically sort the Nodes from least to most loaded.

  • Field Details

    • LOG

      protected static final org.slf4j.Logger LOG

    • numNodesForAnyAllocation

      protected int numNodesForAnyAllocation

    • sortedNodes

      protected final List<org.apache.hadoop.yarn.api.records.NodeId> sortedNodes

    • clusterNodes

      protected final Map<org.apache.hadoop.yarn.api.records.NodeId,ClusterNode> clusterNodes

    • nodeByHostName

      protected final Map<String,RMNode> nodeByHostName

    • nodeIdsByRack

      protected final Map<String,Set<org.apache.hadoop.yarn.api.records.NodeId>> nodeIdsByRack

    • comparator

      protected final NodeQueueLoadMonitor.LoadComparator comparator

    • thresholdCalculator

      protected QueueLimitCalculator thresholdCalculator

    • sortedNodesLock

      protected ReentrantReadWriteLock sortedNodesLock

    • clusterNodesLock

      protected ReentrantReadWriteLock clusterNodesLock

  • Constructor Details

  • Method Details

    • start

      public void start()

    • updateSortedNodes

      protected void updateSortedNodes()

    • getThresholdCalculator

      public QueueLimitCalculator getThresholdCalculator()

    • stop

      public void stop()

    • initThresholdCalculator

      public void initThresholdCalculator(float sigma, int limitMin, int limitMax)

    • addNode

      public void addNode(List<org.apache.hadoop.yarn.server.api.protocolrecords.NMContainerStatus> containerStatuses, RMNode rmNode)
      Specified by:
      addNode in interface ClusterMonitor

    • removeNode

      public void removeNode(RMNode removedRMNode)
      Specified by:
      removeNode in interface ClusterMonitor

    • onNodeRemoved

      protected void onNodeRemoved(ClusterNode node)
      Provide an integration point for extended class
      Parameters:
      node - the node removed

    • updateNode

      public void updateNode(RMNode rmNode)
      Specified by:
      updateNode in interface ClusterMonitor

    • onNewNodeAdded

      protected void onNewNodeAdded(RMNode rmNode, org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus status)

    • onExistingNodeUpdated

      protected void onExistingNodeUpdated(RMNode rmNode, ClusterNode clusterNode, org.apache.hadoop.yarn.server.api.records.OpportunisticContainersStatus status)

    • updateNodeResource

      public void updateNodeResource(RMNode rmNode, org.apache.hadoop.yarn.api.records.ResourceOption resourceOption)
      Specified by:
      updateNodeResource in interface ClusterMonitor

    • selectNodes

      public List<org.apache.hadoop.yarn.api.records.NodeId> selectNodes()
      Returns all Node Ids as ordered list from Least to Most Loaded.
      Returns:
      ordered list of nodes

    • selectLeastLoadedNodes

      public List<org.apache.hadoop.yarn.api.records.NodeId> selectLeastLoadedNodes(int k)
      Returns 'K' of the least Loaded Node Ids as ordered list.
      Parameters:
      k - max number of nodes to return
      Returns:
      ordered list of nodes

    • selectLocalNode

      public RMNode selectLocalNode(String hostName, Set<String> blacklist, org.apache.hadoop.yarn.api.records.Resource request)
      Selects the node as specified by hostName for resource allocation, unless the node has been blacklisted.
      Parameters:
      hostName - the hostname of the node for local resource allocation
      blacklist - the blacklisted nodes
      request - the requested resource
      Returns:
      the selected node, null if the node is full or is blacklisted

    • selectRackLocalNode

      public RMNode selectRackLocalNode(String rackName, Set<String> blacklist, org.apache.hadoop.yarn.api.records.Resource request)
      Selects a node from the rack as specified by rackName for resource allocation, excluding blacklisted nodes
      Parameters:
      rackName - the rack name for rack-local resource allocation
      blacklist - the blacklisted nodes
      request - the requested resource
      Returns:
      the selected node, null if no suitable nodes

    • selectAnyNode

      public RMNode selectAnyNode(Set<String> blacklist, org.apache.hadoop.yarn.api.records.Resource request)
      Selects a node from all ClusterNodes for resource allocation, excluding blacklisted nodes.
      Parameters:
      blacklist - the blacklisted nodes
      request - the requested resource
      Returns:
      the selected node, null if no suitable nodes

    • getCandidatesForSelectAnyNode

      protected List<org.apache.hadoop.yarn.api.records.NodeId> getCandidatesForSelectAnyNode()

    • removeFromNodeIdsByRack

      protected void removeFromNodeIdsByRack(RMNode removedNode)

    • addIntoNodeIdsByRack

      protected void addIntoNodeIdsByRack(RMNode addedNode)

    • sortNodes

      protected List<ClusterNode> sortNodes(boolean excludeFullNodes)