Package org.apache.hadoop.yarn.server.timelineservice.storage.reader

Class TimelineEntityReader

java.lang.Object
org.apache.hadoop.yarn.server.timelineservice.storage.reader.AbstractTimelineStorageReader
org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader
public abstract class TimelineEntityReader extends AbstractTimelineStorageReader
The base class for reading and deserializing timeline entities from the HBase storage. Different types can be defined for different types of the entities that are being requested.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.timelineservice.storage.reader.AbstractTimelineStorageReader

    AbstractTimelineStorageReader.FlowContext

  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    TimelineEntityReader(org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderContext ctxt, org.apache.hadoop.yarn.server.timelineservice.reader.TimelineDataToRetrieve toRetrieve)
    Instantiates a reader for single-entity reads.
    protected
    TimelineEntityReader(org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderContext ctxt, org.apache.hadoop.yarn.server.timelineservice.reader.TimelineEntityFilters entityFilters, org.apache.hadoop.yarn.server.timelineservice.reader.TimelineDataToRetrieve toRetrieve)
    Instantiates a reader for multiple-entity reads.

  • Method Summary

    Modifier and Type
    Method
    Description
    protected abstract org.apache.hadoop.hbase.filter.FilterList
    constructFilterListBasedOnFields(Set<String> cfsInFields)
    Creates a FilterList based on fields, confs and metrics to retrieve.
    protected abstract org.apache.hadoop.hbase.filter.FilterList
    constructFilterListBasedOnFilters()
    Creates a FilterList based on info, config and metric filters.
    protected <T extends org.apache.hadoop.yarn.server.timelineservice.storage.common.BaseTable<T>>
    byte[]
    createColQualifierPrefix(org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<T> colPrefix, String column)
     
    protected <T extends org.apache.hadoop.yarn.server.timelineservice.storage.common.BaseTable<T>>
    org.apache.hadoop.hbase.filter.FilterList
    createFiltersFromColumnQualifiers(org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<T> colPrefix, Set<String> columns)
    Create a filter list of qualifier filters based on passed set of columns.
    protected void
    createFiltersIfNull()
    Create a TimelineEntityFilters object with default values for filters.
    protected org.apache.hadoop.yarn.server.timelineservice.reader.TimelineDataToRetrieve
    getDataToRetrieve()
     
    protected org.apache.hadoop.yarn.server.timelineservice.reader.TimelineEntityFilters
    getFilters()
     
    protected abstract org.apache.hadoop.hbase.client.Result
    getResult(org.apache.hadoop.conf.Configuration hbaseConf, org.apache.hadoop.hbase.client.Connection conn, org.apache.hadoop.hbase.filter.FilterList filterList)
    Fetches a Result instance for a single-entity read.
    protected abstract org.apache.hadoop.hbase.client.ResultScanner
    getResults(org.apache.hadoop.conf.Configuration hbaseConf, org.apache.hadoop.hbase.client.Connection conn, org.apache.hadoop.hbase.filter.FilterList filterList)
    Fetches a ResultScanner for a multi-entity read.
    protected BaseTableRW<?>
    getTable()
    Returns the main table to be used by the entity reader.
    protected boolean
    hasField(EnumSet<org.apache.hadoop.yarn.server.timelineservice.storage.TimelineReader.Field> fieldsToRetrieve, org.apache.hadoop.yarn.server.timelineservice.storage.TimelineReader.Field requiredField)
    Check if we have a certain field amongst fields to retrieve.
    boolean
    isSingleEntityRead()
    Checks whether the reader has been created to fetch single entity or multiple entities.
    protected abstract org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity
    parseEntity(org.apache.hadoop.hbase.client.Result result)
    Parses the result retrieved from HBase backend and convert it into a TimelineEntity object.
    Set<org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity>
    readEntities(org.apache.hadoop.conf.Configuration hbaseConf, org.apache.hadoop.hbase.client.Connection conn)
    Reads and deserializes a set of timeline entities from the HBase storage.
    org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity
    readEntity(org.apache.hadoop.conf.Configuration hbaseConf, org.apache.hadoop.hbase.client.Connection conn)
    Reads and deserializes a single timeline entity from the HBase storage.
    protected static <T extends org.apache.hadoop.yarn.server.timelineservice.storage.common.BaseTable<T>>
    void
    readEvents(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity entity, org.apache.hadoop.hbase.client.Result result, org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<T> prefix)
    Read events from the entity table or the application table.
    protected void
    readMetrics(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity entity, org.apache.hadoop.hbase.client.Result result, org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<?> columnPrefix)
    Helper method for reading and deserializing TimelineMetric objects using the specified column prefix.
    protected <T extends org.apache.hadoop.yarn.server.timelineservice.storage.common.BaseTable<T>>
    void
    readRelationship(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity entity, org.apache.hadoop.hbase.client.Result result, org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<T> prefix, boolean isRelatedTo)
    Helper method for reading relationship.
    protected void
    setTable(BaseTableRW<?> baseTable)
     

    Methods inherited from class org.apache.hadoop.yarn.server.timelineservice.storage.reader.AbstractTimelineStorageReader

    augmentParams, defaultAugmentParams, getContext, lookupFlowContext, validateParams

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • TimelineEntityReader

      protected TimelineEntityReader(org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderContext ctxt, org.apache.hadoop.yarn.server.timelineservice.reader.TimelineEntityFilters entityFilters, org.apache.hadoop.yarn.server.timelineservice.reader.TimelineDataToRetrieve toRetrieve)
      Instantiates a reader for multiple-entity reads.
      Parameters:
      ctxt - Reader context which defines the scope in which query has to be made.
      entityFilters - Filters which limit the entities returned.
      toRetrieve - Data to retrieve for each entity.

    • TimelineEntityReader

      protected TimelineEntityReader(org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderContext ctxt, org.apache.hadoop.yarn.server.timelineservice.reader.TimelineDataToRetrieve toRetrieve)
      Instantiates a reader for single-entity reads.
      Parameters:
      ctxt - Reader context which defines the scope in which query has to be made.
      toRetrieve - Data to retrieve for each entity.

  • Method Details

    • constructFilterListBasedOnFields

      protected abstract org.apache.hadoop.hbase.filter.FilterList constructFilterListBasedOnFields(Set<String> cfsInFields) throws IOException
      Creates a FilterList based on fields, confs and metrics to retrieve. This filter list will be set in Scan/Get objects to trim down results fetched from HBase back-end storage. This is called only for multiple entity reads.
      Parameters:
      cfsInFields - column families in the fields
      Returns:
      a FilterList object.
      Throws:
      IOException - if any problem occurs while creating filter list.

    • constructFilterListBasedOnFilters

      protected abstract org.apache.hadoop.hbase.filter.FilterList constructFilterListBasedOnFilters() throws IOException
      Creates a FilterList based on info, config and metric filters. This filter list will be set in HBase Get to trim down results fetched from HBase back-end storage.
      Returns:
      a FilterList object.
      Throws:
      IOException - if any problem occurs while creating filter list.

    • getDataToRetrieve

      protected org.apache.hadoop.yarn.server.timelineservice.reader.TimelineDataToRetrieve getDataToRetrieve()

    • getFilters

      protected org.apache.hadoop.yarn.server.timelineservice.reader.TimelineEntityFilters getFilters()

    • createFiltersIfNull

      protected void createFiltersIfNull()
      Create a TimelineEntityFilters object with default values for filters.

    • readEntity

      public org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity readEntity(org.apache.hadoop.conf.Configuration hbaseConf, org.apache.hadoop.hbase.client.Connection conn) throws IOException
      Reads and deserializes a single timeline entity from the HBase storage.
      Parameters:
      hbaseConf - HBase Configuration.
      conn - HBase Connection.
      Returns:
      A TimelineEntity object.
      Throws:
      IOException - if there is any exception encountered while reading entity.

    • readEntities

      public Set<org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity> readEntities(org.apache.hadoop.conf.Configuration hbaseConf, org.apache.hadoop.hbase.client.Connection conn) throws IOException
      Reads and deserializes a set of timeline entities from the HBase storage. It goes through all the results available, and returns the number of entries as specified in the limit in the entity's natural sort order.
      Parameters:
      hbaseConf - HBase Configuration.
      conn - HBase Connection.
      Returns:
      a set of TimelineEntity objects.
      Throws:
      IOException - if any exception is encountered while reading entities.

    • getTable

      protected BaseTableRW<?> getTable()
      Returns the main table to be used by the entity reader.
      Returns:
      A reference to the table.

    • getResult

      protected abstract org.apache.hadoop.hbase.client.Result getResult(org.apache.hadoop.conf.Configuration hbaseConf, org.apache.hadoop.hbase.client.Connection conn, org.apache.hadoop.hbase.filter.FilterList filterList) throws IOException
      Fetches a Result instance for a single-entity read.
      Parameters:
      hbaseConf - HBase Configuration.
      conn - HBase Connection.
      filterList - filter list which will be applied to HBase Get.
      Returns:
      the Result instance or null if no such record is found.
      Throws:
      IOException - if any exception is encountered while getting result.

    • getResults

      protected abstract org.apache.hadoop.hbase.client.ResultScanner getResults(org.apache.hadoop.conf.Configuration hbaseConf, org.apache.hadoop.hbase.client.Connection conn, org.apache.hadoop.hbase.filter.FilterList filterList) throws IOException
      Fetches a ResultScanner for a multi-entity read.
      Parameters:
      hbaseConf - HBase Configuration.
      conn - HBase Connection.
      filterList - filter list which will be applied to HBase Scan.
      Returns:
      the ResultScanner instance.
      Throws:
      IOException - if any exception is encountered while getting results.

    • parseEntity

      protected abstract org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity parseEntity(org.apache.hadoop.hbase.client.Result result) throws IOException
      Parses the result retrieved from HBase backend and convert it into a TimelineEntity object.
      Parameters:
      result - Single row result of a Get/Scan.
      Returns:
      the TimelineEntity instance or null if the entity is filtered.
      Throws:
      IOException - if any exception is encountered while parsing entity.

    • readMetrics

      protected void readMetrics(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity entity, org.apache.hadoop.hbase.client.Result result, org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<?> columnPrefix) throws IOException
      Helper method for reading and deserializing TimelineMetric objects using the specified column prefix. The timeline metrics then are added to the given timeline entity.
      Parameters:
      entity - TimelineEntity object.
      result - Result object retrieved from backend.
      columnPrefix - Metric column prefix
      Throws:
      IOException - if any exception is encountered while reading metrics.

    • isSingleEntityRead

      public boolean isSingleEntityRead()
      Checks whether the reader has been created to fetch single entity or multiple entities.
      Returns:
      true, if query is for single entity, false otherwise.

    • setTable

      protected void setTable(BaseTableRW<?> baseTable)

    • hasField

      protected boolean hasField(EnumSet<org.apache.hadoop.yarn.server.timelineservice.storage.TimelineReader.Field> fieldsToRetrieve, org.apache.hadoop.yarn.server.timelineservice.storage.TimelineReader.Field requiredField)
      Check if we have a certain field amongst fields to retrieve. This method checks against TimelineReader.Field.ALL as well because that would mean field passed needs to be matched.
      Parameters:
      fieldsToRetrieve - fields to be retrieved.
      requiredField - fields to be checked in fieldsToRetrieve.
      Returns:
      true if has the required field, false otherwise.

    • createFiltersFromColumnQualifiers

      protected <T extends org.apache.hadoop.yarn.server.timelineservice.storage.common.BaseTable<T>> org.apache.hadoop.hbase.filter.FilterList createFiltersFromColumnQualifiers(org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<T> colPrefix, Set<String> columns)
      Create a filter list of qualifier filters based on passed set of columns.
      Type Parameters:
      T - Describes the type of column prefix.
      Parameters:
      colPrefix - Column Prefix.
      columns - set of column qualifiers.
      Returns:
      filter list.

    • createColQualifierPrefix

      protected <T extends org.apache.hadoop.yarn.server.timelineservice.storage.common.BaseTable<T>> byte[] createColQualifierPrefix(org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<T> colPrefix, String column)

    • readRelationship

      protected <T extends org.apache.hadoop.yarn.server.timelineservice.storage.common.BaseTable<T>> void readRelationship(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity entity, org.apache.hadoop.hbase.client.Result result, org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<T> prefix, boolean isRelatedTo) throws IOException
      Helper method for reading relationship.
      Type Parameters:
      T - Describes the type of column prefix.
      Parameters:
      entity - entity to fill.
      result - result from HBase.
      prefix - column prefix.
      isRelatedTo - if true, means relationship is to be added to isRelatedTo, otherwise its added to relatesTo.
      Throws:
      IOException - if any problem is encountered while reading result.

    • readEvents

      protected static <T extends org.apache.hadoop.yarn.server.timelineservice.storage.common.BaseTable<T>> void readEvents(org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity entity, org.apache.hadoop.hbase.client.Result result, org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnPrefix<T> prefix) throws IOException
      Read events from the entity table or the application table. The column name is of the form "eventId=timestamp=infoKey" where "infoKey" may be omitted if there is no info associated with the event.
      Type Parameters:
      T - Describes the type of column prefix.
      Parameters:
      entity - entity to fill.
      result - HBase Result.
      prefix - column prefix.
      Throws:
      IOException - if any problem is encountered while reading result.