Package org.apache.hadoop.yarn.server.timeline

Class RollingLevelDBTimelineStore

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.service.Service, TimelineReader, TimelineStore, TimelineWriter
@Private @Unstable public class RollingLevelDBTimelineStore extends org.apache.hadoop.service.AbstractService implements TimelineStore

An implementation of an application timeline store backed by leveldb.

There are three sections of the db, the start time section, the entity section, and the indexed entity section.

The start time section is used to retrieve the unique start time for a given entity. Its values each contain a start time while its keys are of the form: 

   START_TIME_LOOKUP_PREFIX + entity type + entity id

The entity section is ordered by entity type, then entity start time descending, then entity ID. There are four sub-sections of the entity section: events, primary filters, related entities, and other info. The event entries have event info serialized into their values. The other info entries have values corresponding to the values of the other info name/value map for the entry (note the names are contained in the key). All other entries have empty values. The key structure is as follows: 

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     DOMAIN_ID_COLUMN

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     EVENTS_COLUMN + reveventtimestamp + eventtype

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     PRIMARY_FILTERS_COLUMN + name + value

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     OTHER_INFO_COLUMN + name

   ENTITY_ENTRY_PREFIX + entity type + revstarttime + entity id +
     RELATED_ENTITIES_COLUMN + relatedentity type + relatedentity id

The indexed entity section contains a primary filter name and primary filter value as the prefix. Within a given name/value, entire entity entries are stored in the same format as described in the entity section above (below, "key" represents any one of the possible entity entry keys described above). 

   INDEXED_ENTRY_PREFIX + primaryfilter name + primaryfilter value +
     key

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

    org.apache.hadoop.service.Service.STATE

    Nested classes/interfaces inherited from interface org.apache.hadoop.yarn.server.timeline.TimelineReader

    TimelineReader.Field

    Nested classes/interfaces inherited from interface org.apache.hadoop.yarn.server.timeline.TimelineStore

    TimelineStore.SystemFilter

  • Field Summary

    Fields inherited from interface org.apache.hadoop.yarn.server.timeline.TimelineReader

    DEFAULT_LIMIT

  • Constructor Summary

    Constructors
    Constructor
    Description
    RollingLevelDBTimelineStore()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.hadoop.yarn.api.records.timeline.TimelineDomain
    getDomain(String domainId)
    This method retrieves the domain information for a given ID.
    org.apache.hadoop.yarn.api.records.timeline.TimelineDomains
    getDomains(String owner)
    This method retrieves all the domains that belong to a given owner.
    org.apache.hadoop.yarn.api.records.timeline.TimelineEntities
    getEntities(String entityType, Long limit, Long windowStart, Long windowEnd, String fromId, Long fromTs, NameValuePair primaryFilter, Collection<NameValuePair> secondaryFilters, EnumSet<TimelineReader.Field> fields, TimelineDataManager.CheckAcl checkAcl)
    This method retrieves a list of entity information, TimelineEntity, sorted by the starting timestamp for the entity, descending.
    org.apache.hadoop.yarn.api.records.timeline.TimelineEntity
    getEntity(String entityId, String entityType, EnumSet<TimelineReader.Field> fields)
    This method retrieves the entity information for a given entity.
    org.apache.hadoop.yarn.api.records.timeline.TimelineEvents
    getEntityTimelines(String entityType, SortedSet<String> entityIds, Long limit, Long windowStart, Long windowEnd, Set<String> eventType)
    This method retrieves the events for a list of entities all of the same entity type.
    void
    put(org.apache.hadoop.yarn.api.records.timeline.TimelineDomain domain)
    Store domain information to the timeline store.
    org.apache.hadoop.yarn.api.records.timeline.TimelinePutResponse
    put(org.apache.hadoop.yarn.api.records.timeline.TimelineEntities entities)
    Stores entity information to the timeline store.
    protected void
    serviceInit(org.apache.hadoop.conf.Configuration conf)
     
    protected void
    serviceStart()
     
    protected void
    serviceStop()
     

    Methods inherited from class org.apache.hadoop.service.AbstractService

    close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    Methods inherited from interface org.apache.hadoop.service.Service

    close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

  • Constructor Details

    • RollingLevelDBTimelineStore

      public RollingLevelDBTimelineStore()

  • Method Details

    • serviceInit

      protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
      Overrides:
      serviceInit in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • serviceStart

      protected void serviceStart() throws Exception
      Overrides:
      serviceStart in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • serviceStop

      protected void serviceStop() throws Exception
      Overrides:
      serviceStop in class org.apache.hadoop.service.AbstractService
      Throws:
      Exception

    • getEntity

      public org.apache.hadoop.yarn.api.records.timeline.TimelineEntity getEntity(String entityId, String entityType, EnumSet<TimelineReader.Field> fields) throws IOException
      Description copied from interface: TimelineReader
      This method retrieves the entity information for a given entity.
      Specified by:
      getEntity in interface TimelineReader
      Parameters:
      entityId - The entity whose information will be retrieved.
      entityType - The type of the entity.
      fields - Specifies which fields of the entity object to retrieve (see TimelineReader.Field). If the set of fields contains TimelineReader.Field.LAST_EVENT_ONLY and not TimelineReader.Field.EVENTS, the most recent event for each entity is retrieved. If null, retrieves all fields.
      Returns:
      An TimelineEntity object.
      Throws:
      IOException

    • getEntityTimelines

      public org.apache.hadoop.yarn.api.records.timeline.TimelineEvents getEntityTimelines(String entityType, SortedSet<String> entityIds, Long limit, Long windowStart, Long windowEnd, Set<String> eventType) throws IOException
      Description copied from interface: TimelineReader
      This method retrieves the events for a list of entities all of the same entity type. The events for each entity are sorted in order of their timestamps, descending.
      Specified by:
      getEntityTimelines in interface TimelineReader
      Parameters:
      entityType - The type of entities to retrieve events for.
      entityIds - The entity IDs to retrieve events for.
      limit - A limit on the number of events to return for each entity. If null, defaults to TimelineReader.DEFAULT_LIMIT events per entity.
      windowStart - If not null, retrieves only events later than the given time (exclusive)
      windowEnd - If not null, retrieves only events earlier than the given time (inclusive)
      eventType - Restricts the events returned to the given types. If null, events of all types will be returned.
      Returns:
      An TimelineEvents object.
      Throws:
      IOException

    • getEntities

      public org.apache.hadoop.yarn.api.records.timeline.TimelineEntities getEntities(String entityType, Long limit, Long windowStart, Long windowEnd, String fromId, Long fromTs, NameValuePair primaryFilter, Collection<NameValuePair> secondaryFilters, EnumSet<TimelineReader.Field> fields, TimelineDataManager.CheckAcl checkAcl) throws IOException
      Description copied from interface: TimelineReader
      This method retrieves a list of entity information, TimelineEntity, sorted by the starting timestamp for the entity, descending. The starting timestamp of an entity is a timestamp specified by the client. If it is not explicitly specified, it will be chosen by the store to be the earliest timestamp of the events received in the first put for the entity.
      Specified by:
      getEntities in interface TimelineReader
      Parameters:
      entityType - The type of entities to return (required).
      limit - A limit on the number of entities to return. If null, defaults to TimelineReader.DEFAULT_LIMIT.
      windowStart - The earliest start timestamp to retrieve (exclusive). If null, defaults to retrieving all entities until the limit is reached.
      windowEnd - The latest start timestamp to retrieve (inclusive). If null, defaults to Long.MAX_VALUE
      fromId - If fromId is not null, retrieve entities earlier than and including the specified ID. If no start time is found for the specified ID, an empty list of entities will be returned. The windowEnd parameter will take precedence if the start time of this entity falls later than windowEnd.
      fromTs - If fromTs is not null, ignore entities that were inserted into the store after the given timestamp. The entity's insert timestamp used for this comparison is the store's system time when the first put for the entity was received (not the entity's start time).
      primaryFilter - Retrieves only entities that have the specified primary filter. If null, retrieves all entities. This is an indexed retrieval, and no entities that do not match the filter are scanned.
      secondaryFilters - Retrieves only entities that have exact matches for all the specified filters in their primary filters or other info. This is not an indexed retrieval, so all entities are scanned but only those matching the filters are returned.
      fields - Specifies which fields of the entity object to retrieve (see TimelineReader.Field). If the set of fields contains TimelineReader.Field.LAST_EVENT_ONLY and not TimelineReader.Field.EVENTS, the most recent event for each entity is retrieved. If null, retrieves all fields.
      Returns:
      An TimelineEntities object.
      Throws:
      IOException

    • put

      public org.apache.hadoop.yarn.api.records.timeline.TimelinePutResponse put(org.apache.hadoop.yarn.api.records.timeline.TimelineEntities entities)
      Description copied from interface: TimelineWriter
      Stores entity information to the timeline store. Any errors occurring for individual put request objects will be reported in the response.
      Specified by:
      put in interface TimelineWriter
      Parameters:
      entities - a TimelineEntities object.
      Returns:
      a TimelinePutResponse object.

    • put

      public void put(org.apache.hadoop.yarn.api.records.timeline.TimelineDomain domain) throws IOException
      Description copied from interface: TimelineWriter
      Store domain information to the timeline store. If A domain of the same ID already exists in the timeline store, it will be COMPLETELY updated with the given domain.
      Specified by:
      put in interface TimelineWriter
      Parameters:
      domain - a TimelineDomain object
      Throws:
      IOException

    • getDomain

      public org.apache.hadoop.yarn.api.records.timeline.TimelineDomain getDomain(String domainId) throws IOException
      Description copied from interface: TimelineReader
      This method retrieves the domain information for a given ID.
      Specified by:
      getDomain in interface TimelineReader
      Returns:
      a TimelineDomain object.
      Throws:
      IOException

    • getDomains

      public org.apache.hadoop.yarn.api.records.timeline.TimelineDomains getDomains(String owner) throws IOException
      Description copied from interface: TimelineReader
      This method retrieves all the domains that belong to a given owner. The domains are sorted according to the created time firstly and the modified time secondly in descending order.
      Specified by:
      getDomains in interface TimelineReader
      Parameters:
      owner - the domain owner
      Returns:
      an TimelineDomains object.
      Throws:
      IOException