Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement

Class AppPlacementAllocator<N extends SchedulerNode>

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.AppPlacementAllocator<N>
Direct Known Subclasses:
LocalityAppPlacementAllocator, SingleConstraintAppPlacementAllocator
public abstract class AppPlacementAllocator<N extends SchedulerNode> extends Object

This class has the following functionality: 1) Keeps track of pending resource requests when following events happen: - New ResourceRequests are added to scheduler. - New containers get allocated. 2) Determines the order that the nodes given in the CandidateNodeSet will be used for allocating containers.

And different set of resource requests (E.g., resource requests with the same schedulerKey) can have one instance of AppPlacementAllocator, each AppPlacementAllocator can have different ways to order nodes depends on requests.

  • Field Details

    • appSchedulingInfo

      protected AppSchedulingInfo appSchedulingInfo

    • schedulerRequestKey

      protected org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey

    • rmContext

      protected RMContext rmContext

  • Constructor Details

    • AppPlacementAllocator

      public AppPlacementAllocator()

  • Method Details

    • getPreferredNodeIterator

      public Iterator<N> getPreferredNodeIterator(CandidateNodeSet<N> candidateNodeSet)
      Get iterator of preferred node depends on requirement and/or availability.
      Parameters:
      candidateNodeSet - input CandidateNodeSet
      Returns:
      iterator of preferred node

    • updatePendingAsk

      public abstract PendingAskUpdateResult updatePendingAsk(Collection<org.apache.hadoop.yarn.api.records.ResourceRequest> requests, boolean recoverPreemptedRequestForAContainer)
      Replace existing pending asks by the new requests
      Parameters:
      requests - new asks
      recoverPreemptedRequestForAContainer - if we're recovering resource requests for preempted container
      Returns:
      true if total pending resource changed

    • updatePendingAsk

      public abstract PendingAskUpdateResult updatePendingAsk(org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey, org.apache.hadoop.yarn.api.records.SchedulingRequest schedulingRequest, boolean recoverPreemptedRequestForAContainer)
      Replace existing pending asks by the new SchedulingRequest
      Parameters:
      schedulerRequestKey - scheduler request key
      schedulingRequest - new asks
      recoverPreemptedRequestForAContainer - if we're recovering resource requests for preempted container
      Returns:
      true if total pending resource changed

    • getResourceRequests

      public abstract Map<String,org.apache.hadoop.yarn.api.records.ResourceRequest> getResourceRequests()
      Get pending ResourceRequests by given schedulerRequestKey
      Returns:
      Map of resourceName to ResourceRequest

    • getPendingAsk

      public abstract PendingAsk getPendingAsk(String resourceName)
      Get pending ask for given resourceName. If there's no such pendingAsk, returns PendingAsk.ZERO
      Parameters:
      resourceName - resourceName
      Returns:
      PendingAsk

    • getOutstandingAsksCount

      public abstract int getOutstandingAsksCount(String resourceName)
      Get #pending-allocations for given resourceName. If there's no such pendingAsk, returns 0
      Parameters:
      resourceName - resourceName
      Returns:
      #pending-allocations

    • allocate

      public abstract ContainerRequest allocate(org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerKey, NodeType type, SchedulerNode node)
      Notify container allocated.
      Parameters:
      schedulerKey - SchedulerRequestKey for this ResourceRequest
      type - Type of the allocation
      node - Which node this container allocated on
      Returns:
      ContainerRequest which include resource requests associated with the container. This will be used by scheduler to recover requests. Please refer to ContainerRequest for more details.

    • canAllocate

      public abstract boolean canAllocate(NodeType type, SchedulerNode node)
      We can still have pending requirement for a given NodeType and node
      Parameters:
      type - Locality Type
      node - which node we will allocate on
      Returns:
      true if we has pending requirement

    • canDelayTo

      public abstract boolean canDelayTo(String resourceName)
      Can delay to give locality? TODO: This should be moved out of AppPlacementAllocator and should belong to specific delay scheduling policy impl. See YARN-7457 for more details.
      Parameters:
      resourceName - resourceName
      Returns:
      can/cannot

    • precheckNode

      public abstract boolean precheckNode(SchedulerNode schedulerNode, SchedulingMode schedulingMode, Optional<DiagnosticsCollector> dcOpt)
      Does this AppPlacementAllocator accept resources on given node?
      Parameters:
      schedulerNode - schedulerNode
      schedulingMode - schedulingMode
      dcOpt - optional diagnostics collector
      Returns:
      accepted/not

    • precheckNode

      public abstract boolean precheckNode(SchedulerNode schedulerNode, SchedulingMode schedulingMode)

    • getPrimaryRequestedNodePartition

      public abstract String getPrimaryRequestedNodePartition()
      It is possible that one request can accept multiple node partition, So this method returns primary node partition for pending resource / headroom calculation.
      Returns:
      primary requested node partition

    • getUniqueLocationAsks

      public abstract int getUniqueLocationAsks()
      Returns:
      number of unique location asks with #pending greater than 0, (like /rack1, host1, etc.). TODO: This should be moved out of AppPlacementAllocator and should belong to specific delay scheduling policy impl. See YARN-7457 for more details.

    • showRequests

      public abstract void showRequests()
      Print human-readable requests to LOG debug.

    • initialize

      public void initialize(AppSchedulingInfo appSchedulingInfo, org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey, RMContext rmContext)
      Initialize this allocator, this will be called by Factory automatically.
      Parameters:
      appSchedulingInfo - appSchedulingInfo
      schedulerRequestKey - schedulerRequestKey
      rmContext - rmContext

    • getSchedulingRequest

      public abstract org.apache.hadoop.yarn.api.records.SchedulingRequest getSchedulingRequest()
      Get pending SchedulingRequest.
      Returns:
      SchedulingRequest

    • getPlacementAttempt

      public int getPlacementAttempt()

    • incrementPlacementAttempt

      public void incrementPlacementAttempt()