Package org.apache.hadoop.mapreduce.jobhistory

Class JobHistoryEventHandler

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service, org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>
public class JobHistoryEventHandler extends org.apache.hadoop.service.AbstractService implements org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>
The job history events get routed to this class. This class writes the Job history events to the DFS directly into a staging dir and then moved to a done-dir. JobHistory implementation is in this package to access package private classes.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    protected class 
    JobHistoryEventHandler.MetaInfo
     

    Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

    org.apache.hadoop.service.Service.STATE

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected org.apache.hadoop.yarn.event.AsyncDispatcher
    atsEventDispatcher
     
    protected Thread
    eventHandlingThread
     
    protected BlockingQueue<JobHistoryEvent>
    eventQueue
     
    protected static final Map<org.apache.hadoop.mapreduce.v2.api.records.JobId,JobHistoryEventHandler.MetaInfo>
    fileMap
     
    protected boolean
    forceJobCompletion
     
    protected boolean
    handleTimelineEvent
     
    protected org.apache.hadoop.yarn.client.api.TimelineClient
    timelineClient
     
    protected org.apache.hadoop.yarn.client.api.TimelineV2Client
    timelineV2Client
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    JobHistoryEventHandler(AppContext context, int startCount)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected void
    closeEventWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
     
    void
    closeWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId id)
    Close the event writer for this id
    protected org.apache.hadoop.yarn.event.AsyncDispatcher
    createDispatcher()
     
    protected org.apache.hadoop.mapreduce.jobhistory.EventWriter
    createEventWriter(org.apache.hadoop.fs.Path historyFilePath)
     
    void
    handle(JobHistoryEvent event)
     
    void
    handleEvent(JobHistoryEvent event)
     
    protected void
    moveTmpToDone(org.apache.hadoop.fs.Path tmpPath)
     
    protected boolean
    moveToDoneNow(org.apache.hadoop.fs.Path fromPath, org.apache.hadoop.fs.Path toPath)
     
    protected void
    processDoneFiles(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
     
    void
    processEventForJobSummary(org.apache.hadoop.mapreduce.jobhistory.HistoryEvent event, JobSummary summary, org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)
     
    protected void
    serviceInit(org.apache.hadoop.conf.Configuration conf)
     
    protected void
    serviceStart()
     
    protected void
    serviceStop()
     
    void
    setForcejobCompletion(boolean forceJobCompletion)
     
    protected void
    setupEventWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId, org.apache.hadoop.mapreduce.jobhistory.AMStartedEvent amStartedEvent)
    Create an event writer for the Job represented by the jobID.

    Methods inherited from class org.apache.hadoop.service.AbstractService

    close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

    • eventQueue

      protected BlockingQueue<JobHistoryEvent> eventQueue

    • handleTimelineEvent

      protected boolean handleTimelineEvent

    • atsEventDispatcher

      protected org.apache.hadoop.yarn.event.AsyncDispatcher atsEventDispatcher

    • eventHandlingThread

      protected Thread eventHandlingThread

    • fileMap

      protected static final Map<org.apache.hadoop.mapreduce.v2.api.records.JobId,JobHistoryEventHandler.MetaInfo> fileMap

    • forceJobCompletion

      protected volatile boolean forceJobCompletion

    • timelineClient

      @VisibleForTesting protected org.apache.hadoop.yarn.client.api.TimelineClient timelineClient

    • timelineV2Client

      @VisibleForTesting protected org.apache.hadoop.yarn.client.api.TimelineV2Client timelineV2Client

  • Constructor Details

    • JobHistoryEventHandler

      public JobHistoryEventHandler(AppContext context, int startCount)

  • Method Details

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
      Overrides:
      serviceInit in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • createDispatcher

      protected org.apache.hadoop.yarn.event.AsyncDispatcher createDispatcher()

    • serviceStart

      protected void serviceStart() throws Exception
      Overrides:
      serviceStart in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • serviceStop

      protected void serviceStop() throws Exception
      Overrides:
      serviceStop in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • createEventWriter

      protected org.apache.hadoop.mapreduce.jobhistory.EventWriter createEventWriter(org.apache.hadoop.fs.Path historyFilePath) throws IOException
      Throws:
      IOException

    • setupEventWriter

      protected void setupEventWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId, org.apache.hadoop.mapreduce.jobhistory.AMStartedEvent amStartedEvent) throws IOException
      Create an event writer for the Job represented by the jobID. Writes out the job configuration to the log directory. This should be the first call to history for a job
      Parameters:
      jobId - the jobId.
      amStartedEvent -
      Throws:
      IOException

    • closeWriter

      public void closeWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId id) throws IOException
      Close the event writer for this id
      Throws:
      IOException

    • handle

      public void handle(JobHistoryEvent event)
      Specified by:
      handle in interface org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>

    • handleEvent

      @Private public void handleEvent(JobHistoryEvent event)

    • processEventForJobSummary

      public void processEventForJobSummary(org.apache.hadoop.mapreduce.jobhistory.HistoryEvent event, JobSummary summary, org.apache.hadoop.mapreduce.v2.api.records.JobId jobId)

    • closeEventWriter

      protected void closeEventWriter(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId) throws IOException
      Throws:
      IOException

    • processDoneFiles

      protected void processDoneFiles(org.apache.hadoop.mapreduce.v2.api.records.JobId jobId) throws IOException
      Throws:
      IOException

    • moveTmpToDone

      protected void moveTmpToDone(org.apache.hadoop.fs.Path tmpPath) throws IOException
      Throws:
      IOException

    • moveToDoneNow

      protected boolean moveToDoneNow(org.apache.hadoop.fs.Path fromPath, org.apache.hadoop.fs.Path toPath) throws IOException
      Throws:
      IOException

    • setForcejobCompletion

      public void setForcejobCompletion(boolean forceJobCompletion)