Package org.apache.hadoop.hdfs.server.namenode

Class FSEditLog

java.lang.Object
org.apache.hadoop.hdfs.server.namenode.FSEditLog
@Private @Evolving public class FSEditLog extends Object
FSEditLog maintains a log of the namespace modifications.

  • Field Details

  • Method Details

    • initJournalsForWrite

      public void initJournalsForWrite()

    • initSharedJournalsForRead

      public void initSharedJournalsForRead()

    • isOpenForRead

      public boolean isOpenForRead()
      Returns:
      true if the log is open in read mode.

    • beginTransaction

      protected void beginTransaction(FSEditLogOp op)

    • getLastWrittenTxId

      public long getLastWrittenTxId()
      Return the transaction ID of the last transaction written to the log.

    • getCurSegmentTxId

      @VisibleForTesting public long getCurSegmentTxId()
      Returns:
      the first transaction ID in the current log segment

    • logSync

      public void logSync()
      Sync all modifications done by this thread. The internal concurrency design of this class is as follows: - Log items are written synchronized into an in-memory buffer, and each assigned a transaction ID. - When a thread (client) would like to sync all of its edits, logSync() uses a ThreadLocal transaction ID to determine what edit number must be synced to. - The isSyncRunning volatile boolean tracks whether a sync is currently under progress. The data is double-buffered within each edit log implementation so that in-memory writing can occur in parallel with the on-disk writing. Each sync occurs in three steps: 1. synchronized, it swaps the double buffer and sets the isSyncRunning flag. 2. unsynchronized, it flushes the data to storage 3. synchronized, it resets the flag and notifies anyone waiting on the sync. The lack of synchronization on step 2 allows other threads to continue to write into the memory buffer while the sync is in progress. Because this step is unsynchronized, actions that need to avoid concurrency with sync() should be synchronized and also call waitForSyncToFinish() before assuming they are running alone.

    • logSync

      protected void logSync(long mytxid)

    • logAppendFile

      public void logAppendFile(String path, INodeFile file, boolean newBlock, boolean toLogRpcIds)

    • logOpenFile

      public void logOpenFile(String path, INodeFile newNode, boolean overwrite, boolean toLogRpcIds)
      Add open lease record to edit log. Records the block locations of the last block.

    • logCloseFile

      public void logCloseFile(String path, INodeFile newNode)
      Add close lease record to edit log.

    • logAddBlock

      public void logAddBlock(String path, INodeFile file)

    • logUpdateBlocks

      public void logUpdateBlocks(String path, INodeFile file, boolean toLogRpcIds)

    • logMkDir

      public void logMkDir(String path, INode newNode)
      Add create directory record to edit log

    • getJournalSet

      @VisibleForTesting public JournalSet getJournalSet()
      Used only by tests.

    • getEditLogManifest

      public RemoteEditLogManifest getEditLogManifest(long fromTxId) throws IOException
      Return a manifest of what finalized edit logs are available
      Throws:
      IOException

    • startLogSegment

      public void startLogSegment(long txid, boolean abortCurrentLogSegment, int layoutVersion) throws IOException
      Remote namenode just has started a log segment, start log segment locally.
      Throws:
      IOException

    • endCurrentLogSegment

      public void endCurrentLogSegment(boolean writeEndTxn)
      Finalize the current log segment. Transitions from IN_SEGMENT state to BETWEEN_LOG_SEGMENTS state.

    • purgeLogsOlderThan

      public void purgeLogsOlderThan(long minTxIdToKeep)
      Archive any log files that are older than the given txid. If the edit log is not open for write, then this call returns with no effect.

    • getSyncTxId

      public long getSyncTxId()
      Return the txid of the last synced transaction.

    • journal

      public void journal(long firstTxId, int numTxns, byte[] data)
      Write the batch of edits to edit log.

    • getSharedLogCTime

      public long getSharedLogCTime() throws IOException
      Throws:
      IOException

    • doPreUpgradeOfSharedLog

      public void doPreUpgradeOfSharedLog() throws IOException
      Throws:
      IOException

    • doUpgradeOfSharedLog

      public void doUpgradeOfSharedLog() throws IOException
      Throws:
      IOException

    • doFinalizeOfSharedLog

      public void doFinalizeOfSharedLog() throws IOException
      Throws:
      IOException

    • canRollBackSharedLog

      public boolean canRollBackSharedLog(StorageInfo prevStorage, int targetLayoutVersion) throws IOException
      Throws:
      IOException

    • doRollback

      public void doRollback() throws IOException
      Throws:
      IOException

    • discardSegments

      public void discardSegments(long markerTxid) throws IOException
      Throws:
      IOException

    • selectInputStreams

      public void selectInputStreams(Collection<EditLogInputStream> streams, long fromTxId, boolean inProgressOk, boolean onlyDurableTxns) throws IOException
      Throws:
      IOException

    • selectInputStreams

      public Collection<EditLogInputStream> selectInputStreams(long fromTxId, long toAtLeastTxId) throws IOException
      Throws:
      IOException

    • selectInputStreams

      public Collection<EditLogInputStream> selectInputStreams(long fromTxId, long toAtLeastTxId, MetaRecoveryContext recovery, boolean inProgressOK) throws IOException
      Throws:
      IOException

    • selectInputStreams

      public Collection<EditLogInputStream> selectInputStreams(long fromTxId, long toAtLeastTxId, MetaRecoveryContext recovery, boolean inProgressOk, boolean onlyDurableTxns) throws IOException
      Select a list of input streams.
      Parameters:
      fromTxId - first transaction in the selected streams
      toAtLeastTxId - the selected streams must contain this transaction
      recovery - recovery context
      inProgressOk - set to true if in-progress streams are OK
      onlyDurableTxns - set to true if streams are bounded by the durable TxId
      Throws:
      IOException

    • restart

      @VisibleForTesting public void restart()

    • getTotalSyncCount

      public long getTotalSyncCount()
      Return total number of syncs happened on this edit log.
      Returns:
      long - count