Package org.apache.hadoop.yarn.server.resourcemanager.reservation

Class AbstractSchedulerPlanFollower

java.lang.Object
org.apache.hadoop.yarn.server.resourcemanager.reservation.AbstractSchedulerPlanFollower
All Implemented Interfaces:
Runnable, PlanFollower
Direct Known Subclasses:
CapacitySchedulerPlanFollower, FairSchedulerPlanFollower
public abstract class AbstractSchedulerPlanFollower extends Object implements PlanFollower

  • Field Details

    • plans

      protected Collection<Plan> plans

    • scheduler

      protected YarnScheduler scheduler

    • clock

      protected org.apache.hadoop.yarn.util.Clock clock

  • Constructor Details

    • AbstractSchedulerPlanFollower

      public AbstractSchedulerPlanFollower()

  • Method Details

    • init

      public void init(org.apache.hadoop.yarn.util.Clock clock, ResourceScheduler sched, Collection<Plan> plans)
      Description copied from interface: PlanFollower
      Init function that configures the PlanFollower, by providing:
      Specified by:
      init in interface PlanFollower
      Parameters:
      clock - a reference to the system clock.
      sched - a reference to the underlying scheduler
      plans - references to the plans we should keep synchronized at every time tick.

    • run

      public void run()
      Specified by:
      run in interface Runnable

    • setPlans

      public void setPlans(Collection<Plan> plans)
      Description copied from interface: PlanFollower
      Setter for the list of plans.
      Specified by:
      setPlans in interface PlanFollower
      Parameters:
      plans - the collection of Plans we operate on at every time tick.

    • synchronizePlan

      public void synchronizePlan(Plan plan, boolean shouldReplan)
      Description copied from interface: PlanFollower
      The function performing the actual synchronization operation for a given Plan. This is normally invoked by the run method, but it can be invoked synchronously to avoid race conditions when a user's reservation request start time is imminent.
      Specified by:
      synchronizePlan in interface PlanFollower
      Parameters:
      plan - the Plan to synchronize
      shouldReplan - replan on reduction of plan capacity if true or proportionally scale down reservations if false

    • getReservationIdFromQueueName

      protected String getReservationIdFromQueueName(String resQueueName)

    • setQueueEntitlement

      protected void setQueueEntitlement(String planQueueName, String currResId, float targetCapacity, float maxCapacity) throws org.apache.hadoop.yarn.exceptions.YarnException
      Throws:
      org.apache.hadoop.yarn.exceptions.YarnException

    • getReservationQueueName

      protected String getReservationQueueName(String planQueueName, String reservationId)

    • cleanupExpiredQueues

      protected void cleanupExpiredQueues(String planQueueName, boolean shouldMove, Set<String> toRemove, String defReservationQueue)
      First sets entitlement of queues to zero to prevent new app submission. Then move all apps in the set of queues to the parent plan queue's default reservation queue if move is enabled. Finally cleanups the queue by killing any apps (if move is disabled or move failed) and removing the queue
      Parameters:
      planQueueName - the name of PlanQueue
      shouldMove - flag to indicate if any running apps should be moved or killed
      toRemove - the remnant apps to clean up
      defReservationQueue - the default ReservationQueue of the Plan

    • getReservedResources

      protected int getReservedResources(long now, Set<ReservationAllocation> currentReservations, Set<String> curReservationNames, org.apache.hadoop.yarn.api.records.Resource reservedResources)

    • sortByDelta

      protected List<ReservationAllocation> sortByDelta(List<ReservationAllocation> currentReservations, long now, Plan plan)
      Sort in the order from the least new amount of resources asked (likely negative) to the highest. This prevents "order-of-operation" errors related to exceeding 100% capacity temporarily.
      Parameters:
      currentReservations - the currently active reservations
      now - the current time
      plan - the Plan that is being considered
      Returns:
      the sorted list of ReservationAllocations

    • getPlanQueue

      protected abstract Queue getPlanQueue(String planQueueName)
      Get queue associated with reservable queue named.
      Parameters:
      planQueueName - name of the reservable queue
      Returns:
      queue associated with the reservable queue

    • getChildReservationQueues

      protected abstract List<? extends Queue> getChildReservationQueues(Queue planQueue)
      Get a list of reservation queues for this planQueue.
      Parameters:
      planQueue - the queue for the current Plan
      Returns:
      the queues corresponding to the reservations

    • addReservationQueue

      protected abstract void addReservationQueue(String planQueueName, Queue queue, String currResId)
      Add a new reservation queue for reservation currResId for this planQueue.
      Parameters:
      planQueueName - name of the reservable queue.
      queue - the queue for the current Plan.
      currResId - curr reservationId.

    • createDefaultReservationQueue

      protected abstract void createDefaultReservationQueue(String planQueueName, Queue queue, String defReservationQueue)
      Creates the default reservation queue for use when no reservation is used for applications submitted to this planQueue.
      Parameters:
      planQueueName - name of the reservable queue
      queue - the queue for the current Plan
      defReservationQueue - name of the default ReservationQueue

    • getPlanResources

      protected abstract org.apache.hadoop.yarn.api.records.Resource getPlanResources(Plan plan, Queue queue, org.apache.hadoop.yarn.api.records.Resource clusterResources)
      Get plan resources for this planQueue.
      Parameters:
      plan - the current Plan being considered
      queue - the queue for the current Plan
      clusterResources - the resources available in the cluster
      Returns:
      the resources allocated to the specified Plan

    • getReservationQueueResourceIfExists

      protected abstract org.apache.hadoop.yarn.api.records.Resource getReservationQueueResourceIfExists(Plan plan, org.apache.hadoop.yarn.api.records.ReservationId reservationId)
      Get reservation queue resources if it exists otherwise return null.
      Parameters:
      plan - the current Plan being considered
      reservationId - the identifier of the reservation
      Returns:
      the resources allocated to the specified reservation