Package org.apache.hadoop.mapreduce.v2.app.rm.preemption

Class CheckpointAMPreemptionPolicy

java.lang.Object
org.apache.hadoop.mapreduce.v2.app.rm.preemption.CheckpointAMPreemptionPolicy
All Implemented Interfaces:
AMPreemptionPolicy
public class CheckpointAMPreemptionPolicy extends Object implements AMPreemptionPolicy
This policy works in combination with an implementation of task checkpointing. It computes the tasks to be preempted in response to the RM request for preemption. For strict requests, it maps containers to corresponding tasks; for fungible requests, it attempts to pick the best containers to preempt (reducers in reverse allocation order). The TaskAttemptListener will interrogate this policy when handling a task heartbeat to check whether the task should be preempted or not. When handling fungible requests, the policy discount the RM ask by the amount of currently in-flight preemptions (i.e., tasks that are checkpointing). This class it is also used to maintain the list of checkpoints for existing tasks. Centralizing this functionality here, allows us to have visibility on preemption and checkpoints in a single location, thus coordinating preemption and checkpoint management decisions in a single policy.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.hadoop.mapreduce.v2.app.rm.preemption.AMPreemptionPolicy

    AMPreemptionPolicy.Context

  • Constructor Summary

    Constructors
    Constructor
    Description
    CheckpointAMPreemptionPolicy()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID
    getCheckpointID(org.apache.hadoop.mapreduce.v2.api.records.TaskId taskId)
    Method to retrieve the latest checkpoint for a given TaskId
    void
    handleCompletedContainer(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId attemptID)
    Callback informing the policy of containers exiting cleanly.
    void
    handleFailedContainer(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId attemptID)
    Callback informing the policy of containers exiting with a failure.
    void
    init(AppContext context)
     
    boolean
    isPreempted(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId yarnAttemptID)
    This method is invoked by components interested to learn whether a certain task is being preempted.
    void
    preempt(AMPreemptionPolicy.Context ctxt, org.apache.hadoop.yarn.api.records.PreemptionMessage preemptionRequests)
    Callback informing the policy of ResourceManager. requests for resources to return to the cluster.
    void
    reportSuccessfulPreemption(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId taskAttemptID)
    This method is used to report to the policy that a certain task has been successfully preempted (for bookeeping, counters, etc..)
    void
    setCheckpointID(org.apache.hadoop.mapreduce.v2.api.records.TaskId taskId, org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID cid)
    Method to store the latest CheckpointID for a given TaskId.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • CheckpointAMPreemptionPolicy

      public CheckpointAMPreemptionPolicy()

  • Method Details

    • init

      public void init(AppContext context)
      Specified by:
      init in interface AMPreemptionPolicy

    • preempt

      public void preempt(AMPreemptionPolicy.Context ctxt, org.apache.hadoop.yarn.api.records.PreemptionMessage preemptionRequests)
      Description copied from interface: AMPreemptionPolicy
      Callback informing the policy of ResourceManager. requests for resources to return to the cluster. The policy may take arbitrary action to satisfy requests by checkpointing task state, returning containers, or ignoring requests. The RM may elect to enforce these requests by forcibly killing containers not returned after some duration.
      Specified by:
      preempt in interface AMPreemptionPolicy
      Parameters:
      ctxt - Handle to the current state of running containers
      preemptionRequests - Request from RM for resources to return.

    • handleFailedContainer

      public void handleFailedContainer(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId attemptID)
      Description copied from interface: AMPreemptionPolicy
      Callback informing the policy of containers exiting with a failure. This allows the policy to implemnt cleanup/compensating actions.
      Specified by:
      handleFailedContainer in interface AMPreemptionPolicy
      Parameters:
      attemptID - Task attempt that failed

    • handleCompletedContainer

      public void handleCompletedContainer(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId attemptID)
      Description copied from interface: AMPreemptionPolicy
      Callback informing the policy of containers exiting cleanly. This is reported to the policy for bookeeping purposes.
      Specified by:
      handleCompletedContainer in interface AMPreemptionPolicy
      Parameters:
      attemptID - Task attempt that completed

    • isPreempted

      public boolean isPreempted(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId yarnAttemptID)
      Description copied from interface: AMPreemptionPolicy
      This method is invoked by components interested to learn whether a certain task is being preempted.
      Specified by:
      isPreempted in interface AMPreemptionPolicy
      Parameters:
      yarnAttemptID - Task attempt to query
      Returns:
      true if this attempt is being preempted

    • reportSuccessfulPreemption

      public void reportSuccessfulPreemption(org.apache.hadoop.mapreduce.v2.api.records.TaskAttemptId taskAttemptID)
      Description copied from interface: AMPreemptionPolicy
      This method is used to report to the policy that a certain task has been successfully preempted (for bookeeping, counters, etc..)
      Specified by:
      reportSuccessfulPreemption in interface AMPreemptionPolicy
      Parameters:
      taskAttemptID - Task attempt that preempted

    • getCheckpointID

      public org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID getCheckpointID(org.apache.hadoop.mapreduce.v2.api.records.TaskId taskId)
      Description copied from interface: AMPreemptionPolicy
      Method to retrieve the latest checkpoint for a given TaskId
      Specified by:
      getCheckpointID in interface AMPreemptionPolicy
      Parameters:
      taskId - TaskID
      Returns:
      CheckpointID associated with this task or null

    • setCheckpointID

      public void setCheckpointID(org.apache.hadoop.mapreduce.v2.api.records.TaskId taskId, org.apache.hadoop.mapreduce.checkpoint.TaskCheckpointID cid)
      Description copied from interface: AMPreemptionPolicy
      Method to store the latest CheckpointID for a given TaskId. Assigning a null is akin to remove all previous checkpoints for this task.
      Specified by:
      setCheckpointID in interface AMPreemptionPolicy
      Parameters:
      taskId - TaskID
      cid - Checkpoint to assign or null to remove it.