Package org.apache.hadoop.yarn.server.timelineservice.collector

Class TimelineCollector

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.service.CompositeService
org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service
Direct Known Subclasses:
AppLevelTimelineCollector
@Private @Unstable public abstract class TimelineCollector extends org.apache.hadoop.service.CompositeService
Service that handles writes to the timeline service and writes them to the backing storage. Classes that extend this can add their own lifecycle management or customization of request handling.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    protected static class 
    TimelineCollector.AggregationStatusTable
     

    Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

    org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

    Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

    org.apache.hadoop.service.Service.STATE

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    SEPARATOR
     

    Fields inherited from class org.apache.hadoop.service.CompositeService

    STOP_ONLY_STARTED_SERVICES

  • Constructor Summary

    Constructors
    Constructor
    Description
    TimelineCollector(String name)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity
    aggregateEntities(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, String resultEntityId, String resultEntityType, boolean needsGroupIdInResult)
    Aggregate all metrics in given timeline entities with no predefined states.
    protected Map<String,TimelineCollector.AggregationStatusTable>
    getAggregationGroups()
     
    protected Set<String>
    getEntityTypesSkipAggregation()
    Method to decide the set of timeline entity types the collector should skip on aggregations.
    abstract TimelineCollectorContext
    getTimelineEntityContext()
     
    protected boolean
    isReadyToAggregate()
     
    org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse
    putDomain(org.apache.hadoop.yarn.api.records.timelineservice.TimelineDomain domain, org.apache.hadoop.security.UserGroupInformation callerUgi)
    Add or update an domain.
    org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse
    putEntities(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, org.apache.hadoop.security.UserGroupInformation callerUgi)
    Handles entity writes.
    void
    putEntitiesAsync(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, org.apache.hadoop.security.UserGroupInformation callerUgi)
    Handles entity writes in an asynchronous manner.
    protected void
    serviceInit(org.apache.hadoop.conf.Configuration conf)
     
    protected void
    serviceStart()
     
    protected void
    serviceStop()
     
    protected void
    setReadyToAggregate()
     
    protected void
    setWriter(TimelineWriter w)
     

    Methods inherited from class org.apache.hadoop.service.CompositeService

    addIfService, addService, getServices, removeService

    Methods inherited from class org.apache.hadoop.service.AbstractService

    close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

  • Constructor Details

    • TimelineCollector

      public TimelineCollector(String name)

  • Method Details

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
      Overrides:
      serviceInit in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • serviceStart

      protected void serviceStart() throws Exception
      Overrides:
      serviceStart in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • serviceStop

      protected void serviceStop() throws Exception
      Overrides:
      serviceStop in class org.apache.hadoop.service.CompositeService
      Throws:
      Exception

    • setWriter

      protected void setWriter(TimelineWriter w)

    • getAggregationGroups

      protected Map<String,TimelineCollector.AggregationStatusTable> getAggregationGroups()

    • setReadyToAggregate

      protected void setReadyToAggregate()

    • isReadyToAggregate

      protected boolean isReadyToAggregate()

    • getEntityTypesSkipAggregation

      protected Set<String> getEntityTypesSkipAggregation()
      Method to decide the set of timeline entity types the collector should skip on aggregations. Subclasses may want to override this method to customize their own behaviors.
      Returns:
      A set of strings consists of all types the collector should skip.

    • getTimelineEntityContext

      public abstract TimelineCollectorContext getTimelineEntityContext()

    • putEntities

      public org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse putEntities(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, org.apache.hadoop.security.UserGroupInformation callerUgi) throws IOException
      Handles entity writes. These writes are synchronous and are written to the backing storage without buffering/batching. If any entity already exists, it results in an update of the entity. This method should be reserved for selected critical entities and events. For normal voluminous writes one should use the async method putEntitiesAsync(TimelineEntities, UserGroupInformation).
      Parameters:
      entities - entities to post
      callerUgi - the caller UGI
      Returns:
      the response that contains the result of the post.
      Throws:
      IOException - if there is any exception encountered while putting entities.

    • putDomain

      public org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse putDomain(org.apache.hadoop.yarn.api.records.timelineservice.TimelineDomain domain, org.apache.hadoop.security.UserGroupInformation callerUgi) throws IOException
      Add or update an domain. If the domain already exists, only the owner and the admin can update it.
      Parameters:
      domain - domain to post
      callerUgi - the caller UGI
      Returns:
      the response that contains the result of the post.
      Throws:
      IOException - if there is any exception encountered while putting domain.

    • putEntitiesAsync

      public void putEntitiesAsync(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, org.apache.hadoop.security.UserGroupInformation callerUgi) throws IOException
      Handles entity writes in an asynchronous manner. The method returns as soon as validation is done. No promises are made on how quickly it will be written to the backing storage or if it will always be written to the backing storage. Multiple writes to the same entities may be batched and appropriate values updated and result in fewer writes to the backing storage.
      Parameters:
      entities - entities to post
      callerUgi - the caller UGI
      Throws:
      IOException - if there is any exception encounted while putting entities.

    • aggregateEntities

      public static org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity aggregateEntities(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities entities, String resultEntityId, String resultEntityType, boolean needsGroupIdInResult)
      Aggregate all metrics in given timeline entities with no predefined states.
      Parameters:
      entities - Entities to aggregate
      resultEntityId - Id of the result entity
      resultEntityType - Type of the result entity
      needsGroupIdInResult - Marks if we want the aggregation group id in each aggregated metrics.
      Returns:
      A timeline entity that contains all aggregated TimelineMetric.