Package org.apache.hadoop.hdfs.server.namenode

Interface JournalManager

All Superinterfaces:
AutoCloseable, Closeable, Storage.FormatConfirmable
All Known Implementing Classes:
FileJournalManager, JournalSet, QuorumJournalManager
@Private @Evolving public interface JournalManager extends Closeable, Storage.FormatConfirmable
A JournalManager is responsible for managing a single place of storing edit logs. It may correspond to multiple files, a backup node, etc. Even when the actual underlying storage is rolled, or failed and restored, each conceptual place of storage corresponds to exactly one instance of this class, which is created when the EditLog is first opened.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Interface
    Description
    static class 
    JournalManager.CorruptionException
    Indicate that a journal is cannot be used to load a certain range of edits.

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    canRollBack(StorageInfo storage, StorageInfo prevStorage, int targetLayoutVersion)
    Return true if this JM can roll back to the previous storage state, false otherwise.
    void
    close()
    Close the journal manager, freeing any resources it may hold.
    void
    discardSegments(long startTxId)
    Discard the segments whose first txid is >= the given txid.
    void
    doFinalize()
    Finalize the upgrade.
    void
    doPreUpgrade()
    Perform any steps that must succeed across all JournalManagers involved in an upgrade before proceeding onto the actual upgrade stage.
    void
    doRollback()
    Perform the rollback to the previous FS state.
    void
    doUpgrade(Storage storage)
    Perform the actual upgrade of the JM.
    void
    finalizeLogSegment(long firstTxId, long lastTxId)
    Mark the log segment that spans from firstTxId to lastTxId as finalized and complete.
    void
    format(NamespaceInfo ns, boolean force)
    Format the underlying storage, removing any previously stored data.
    long
    getJournalCTime()
     
    void
    purgeLogsOlderThan(long minTxIdToKeep)
    Remove all edit logs with transaction IDs lower than the given transaction ID.
    void
    recoverUnfinalizedSegments()
    Recover segments which have not been finalized.
    void
    selectInputStreams(Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk, boolean onlyDurableTxns)
    Get a list of edit log input streams.
    void
    setOutputBufferCapacity(int size)
    Set the amount of memory that this stream should use to buffer edits
    EditLogOutputStream
    startLogSegment(long txId, int layoutVersion)
    Begin writing to a new segment of the log stream, which starts at the given transaction ID.

    Methods inherited from interface org.apache.hadoop.hdfs.server.common.Storage.FormatConfirmable

    hasSomeData, toString

  • Method Details

    • format

      void format(NamespaceInfo ns, boolean force) throws IOException
      Format the underlying storage, removing any previously stored data.
      Throws:
      IOException

    • startLogSegment

      EditLogOutputStream startLogSegment(long txId, int layoutVersion) throws IOException
      Begin writing to a new segment of the log stream, which starts at the given transaction ID.
      Throws:
      IOException

    • finalizeLogSegment

      void finalizeLogSegment(long firstTxId, long lastTxId) throws IOException
      Mark the log segment that spans from firstTxId to lastTxId as finalized and complete.
      Throws:
      IOException

    • setOutputBufferCapacity

      void setOutputBufferCapacity(int size)
      Set the amount of memory that this stream should use to buffer edits

    • recoverUnfinalizedSegments

      void recoverUnfinalizedSegments() throws IOException
      Recover segments which have not been finalized.
      Throws:
      IOException

    • doPreUpgrade

      void doPreUpgrade() throws IOException
      Perform any steps that must succeed across all JournalManagers involved in an upgrade before proceeding onto the actual upgrade stage. If a call to any JM's doPreUpgrade method fails, then doUpgrade will not be called for any JM.
      Throws:
      IOException

    • doUpgrade

      void doUpgrade(Storage storage) throws IOException
      Perform the actual upgrade of the JM. After this is completed, the NN can begin to use the new upgraded metadata. This metadata may later be either finalized or rolled back to the previous state.
      Parameters:
      storage - info about the new upgraded versions.
      Throws:
      IOException

    • doFinalize

      void doFinalize() throws IOException
      Finalize the upgrade. JMs should purge any state that they had been keeping around during the upgrade process. After this is completed, rollback is no longer allowed.
      Throws:
      IOException

    • canRollBack

      boolean canRollBack(StorageInfo storage, StorageInfo prevStorage, int targetLayoutVersion) throws IOException
      Return true if this JM can roll back to the previous storage state, false otherwise. The NN will refuse to run the rollback operation unless at least one JM or fsimage storage directory can roll back.
      Parameters:
      storage - the storage info for the current state
      prevStorage - the storage info for the previous (unupgraded) state
      targetLayoutVersion - the layout version we intend to roll back to
      Returns:
      true if this JM can roll back, false otherwise.
      Throws:
      IOException

    • doRollback

      void doRollback() throws IOException
      Perform the rollback to the previous FS state. JMs which do not need to roll back their state should just return without error.
      Throws:
      IOException

    • discardSegments

      void discardSegments(long startTxId) throws IOException
      Discard the segments whose first txid is >= the given txid.
      Parameters:
      startTxId - The given txid should be right at the segment boundary, i.e., it should be the first txid of some segment, if segment corresponding to the txid exists.
      Throws:
      IOException

    • getJournalCTime

      long getJournalCTime() throws IOException
      Returns:
      the CTime of the journal manager.
      Throws:
      IOException

    • close

      void close() throws IOException
      Close the journal manager, freeing any resources it may hold.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • purgeLogsOlderThan

      void purgeLogsOlderThan(long minTxIdToKeep) throws IOException
      Remove all edit logs with transaction IDs lower than the given transaction ID.
      Parameters:
      minTxIdToKeep - the lowest transaction ID that should be retained
      Throws:
      IOException - in the event of error

    • selectInputStreams

      void selectInputStreams(Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk, boolean onlyDurableTxns) throws IOException
      Get a list of edit log input streams. The list will start with the stream that contains fromTxnId, and continue until the end of the journal being managed.
      Parameters:
      fromTxId - the first transaction id we want to read
      inProgressOk - whether or not in-progress streams should be returned
      onlyDurableTxns - whether or not streams should be bounded by durable TxId. A durable TxId is the committed txid in QJM or the largest txid written into file in FJM
      Throws:
      IOException - if the underlying storage has an error or is otherwise inaccessible