Package org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement

Class LocalityAppPlacementAllocator<N extends SchedulerNode>

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.AppPlacementAllocator<N>
org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.LocalityAppPlacementAllocator<N>
public class LocalityAppPlacementAllocator<N extends SchedulerNode> extends AppPlacementAllocator<N>
This is an implementation of the AppPlacementAllocator that takes into account locality preferences (node, rack, any) when allocating containers.

  • Constructor Details

    • LocalityAppPlacementAllocator

      public LocalityAppPlacementAllocator()

  • Method Details

    • initialize

      public void initialize(AppSchedulingInfo appSchedulingInfo, org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey, RMContext rmContext)
      Description copied from class: AppPlacementAllocator
      Initialize this allocator, this will be called by Factory automatically.
      Overrides:
      initialize in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      appSchedulingInfo - appSchedulingInfo
      schedulerRequestKey - schedulerRequestKey
      rmContext - rmContext

    • updatePendingAsk

      public PendingAskUpdateResult updatePendingAsk(Collection<org.apache.hadoop.yarn.api.records.ResourceRequest> requests, boolean recoverPreemptedRequestForAContainer)
      Description copied from class: AppPlacementAllocator
      Replace existing pending asks by the new requests
      Specified by:
      updatePendingAsk in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      requests - new asks
      recoverPreemptedRequestForAContainer - if we're recovering resource requests for preempted container
      Returns:
      true if total pending resource changed

    • updatePendingAsk

      public PendingAskUpdateResult updatePendingAsk(org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerRequestKey, org.apache.hadoop.yarn.api.records.SchedulingRequest schedulingRequest, boolean recoverPreemptedRequestForAContainer) throws org.apache.hadoop.yarn.exceptions.SchedulerInvalidResourceRequestException
      Description copied from class: AppPlacementAllocator
      Replace existing pending asks by the new SchedulingRequest
      Specified by:
      updatePendingAsk in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      schedulerRequestKey - scheduler request key
      schedulingRequest - new asks
      recoverPreemptedRequestForAContainer - if we're recovering resource requests for preempted container
      Returns:
      true if total pending resource changed
      Throws:
      org.apache.hadoop.yarn.exceptions.SchedulerInvalidResourceRequestException

    • getResourceRequests

      public Map<String,org.apache.hadoop.yarn.api.records.ResourceRequest> getResourceRequests()
      Description copied from class: AppPlacementAllocator
      Get pending ResourceRequests by given schedulerRequestKey
      Specified by:
      getResourceRequests in class AppPlacementAllocator<N extends SchedulerNode>
      Returns:
      Map of resourceName to ResourceRequest

    • getPendingAsk

      public PendingAsk getPendingAsk(String resourceName)
      Description copied from class: AppPlacementAllocator
      Get pending ask for given resourceName. If there's no such pendingAsk, returns PendingAsk.ZERO
      Specified by:
      getPendingAsk in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      resourceName - resourceName
      Returns:
      PendingAsk

    • getOutstandingAsksCount

      public int getOutstandingAsksCount(String resourceName)
      Description copied from class: AppPlacementAllocator
      Get #pending-allocations for given resourceName. If there's no such pendingAsk, returns 0
      Specified by:
      getOutstandingAsksCount in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      resourceName - resourceName
      Returns:
      #pending-allocations

    • cloneResourceRequest

      public org.apache.hadoop.yarn.api.records.ResourceRequest cloneResourceRequest(org.apache.hadoop.yarn.api.records.ResourceRequest request)

    • canAllocate

      public boolean canAllocate(NodeType type, SchedulerNode node)
      Description copied from class: AppPlacementAllocator
      We can still have pending requirement for a given NodeType and node
      Specified by:
      canAllocate in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      type - Locality Type
      node - which node we will allocate on
      Returns:
      true if we has pending requirement

    • canDelayTo

      public boolean canDelayTo(String resourceName)
      Description copied from class: AppPlacementAllocator
      Can delay to give locality? TODO: This should be moved out of AppPlacementAllocator and should belong to specific delay scheduling policy impl. See YARN-7457 for more details.
      Specified by:
      canDelayTo in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      resourceName - resourceName
      Returns:
      can/cannot

    • precheckNode

      public boolean precheckNode(SchedulerNode schedulerNode, SchedulingMode schedulingMode, Optional<DiagnosticsCollector> dcOpt)
      Description copied from class: AppPlacementAllocator
      Does this AppPlacementAllocator accept resources on given node?
      Specified by:
      precheckNode in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      schedulerNode - schedulerNode
      schedulingMode - schedulingMode
      dcOpt - optional diagnostics collector
      Returns:
      accepted/not

    • precheckNode

      public boolean precheckNode(SchedulerNode schedulerNode, SchedulingMode schedulingMode)
      Specified by:
      precheckNode in class AppPlacementAllocator<N extends SchedulerNode>

    • getPrimaryRequestedNodePartition

      public String getPrimaryRequestedNodePartition()
      Description copied from class: AppPlacementAllocator
      It is possible that one request can accept multiple node partition, So this method returns primary node partition for pending resource / headroom calculation.
      Specified by:
      getPrimaryRequestedNodePartition in class AppPlacementAllocator<N extends SchedulerNode>
      Returns:
      primary requested node partition

    • getUniqueLocationAsks

      public int getUniqueLocationAsks()
      Specified by:
      getUniqueLocationAsks in class AppPlacementAllocator<N extends SchedulerNode>
      Returns:
      number of unique location asks with #pending greater than 0, (like /rack1, host1, etc.). TODO: This should be moved out of AppPlacementAllocator and should belong to specific delay scheduling policy impl. See YARN-7457 for more details.

    • showRequests

      public void showRequests()
      Description copied from class: AppPlacementAllocator
      Print human-readable requests to LOG debug.
      Specified by:
      showRequests in class AppPlacementAllocator<N extends SchedulerNode>

    • allocate

      public ContainerRequest allocate(org.apache.hadoop.yarn.server.scheduler.SchedulerRequestKey schedulerKey, NodeType type, SchedulerNode node)
      Description copied from class: AppPlacementAllocator
      Notify container allocated.
      Specified by:
      allocate in class AppPlacementAllocator<N extends SchedulerNode>
      Parameters:
      schedulerKey - SchedulerRequestKey for this ResourceRequest
      type - Type of the allocation
      node - Which node this container allocated on
      Returns:
      ContainerRequest which include resource requests associated with the container. This will be used by scheduler to recover requests. Please refer to ContainerRequest for more details.

    • getSchedulingRequest

      public org.apache.hadoop.yarn.api.records.SchedulingRequest getSchedulingRequest()
      Description copied from class: AppPlacementAllocator
      Get pending SchedulingRequest.
      Specified by:
      getSchedulingRequest in class AppPlacementAllocator<N extends SchedulerNode>
      Returns:
      SchedulingRequest