Package org.apache.hadoop.hdfs.server.namenode

Class FSImage

java.lang.Object
org.apache.hadoop.hdfs.server.namenode.FSImage
All Implemented Interfaces:
Closeable, AutoCloseable
Direct Known Subclasses:
BackupImage
@Private @Evolving public class FSImage extends Object implements Closeable
FSImage handles checkpointing and logging of the namespace edits.

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

    • SHUTDOWN_HOOK_PRIORITY

      public static final int SHUTDOWN_HOOK_PRIORITY
      Priority of the FSImageSaver shutdown hook: 10.
      See Also:

    • editLog

      protected FSEditLog editLog

    • storage

      protected NNStorage storage

    • lastAppliedTxId

      protected long lastAppliedTxId
      The last transaction ID that was either loaded from an image or loaded by loading edits files.

    • archivalManager

      protected NNStorageRetentionManager archivalManager

  • Constructor Details

    • FSImage

      public FSImage(org.apache.hadoop.conf.Configuration conf) throws IOException
      Construct an FSImage
      Parameters:
      conf - Configuration
      Throws:
      IOException - if default directories are invalid.

    • FSImage

      protected FSImage(org.apache.hadoop.conf.Configuration conf, Collection<URI> imageDirs, List<URI> editsDirs) throws IOException
      Construct the FSImage. Set the default checkpoint directories. Setup storage and initialize the edit log.
      Parameters:
      conf - Configuration
      imageDirs - Directories the image can be stored in.
      editsDirs - Directories the editlog can be stored in.
      Throws:
      IOException - if directories are invalid.

  • Method Details

    • recoverStorageDirs

      public static boolean recoverStorageDirs(HdfsServerConstants.StartupOption startOpt, NNStorage storage, Map<Storage.StorageDirectory,Storage.StorageState> dataDirStates) throws IOException
      For each storage directory, performs recovery of incomplete transitions (eg. upgrade, rollback, checkpoint) and inserts the directory's storage state into the dataDirStates map.
      Parameters:
      dataDirStates - output of storage directory states
      Returns:
      true if there is at least one valid formatted storage directory
      Throws:
      IOException

    • checkUpgrade

      public static void checkUpgrade(NNStorage storage) throws IOException
      Check if upgrade is in progress.
      Throws:
      IOException

    • hasRollbackFSImage

      public boolean hasRollbackFSImage() throws IOException
      Returns:
      true if there is rollback fsimage (for rolling upgrade) in NameNode directory.
      Throws:
      IOException

    • getEditLog

      public FSEditLog getEditLog()

    • initEditLog

      public void initEditLog(HdfsServerConstants.StartupOption startOpt) throws IOException
      Throws:
      IOException

    • loadEdits

      public long loadEdits(Iterable<EditLogInputStream> editStreams, FSNamesystem target) throws IOException
      Load the specified list of edit files into the image.
      Throws:
      IOException

    • loadEdits

      public long loadEdits(Iterable<EditLogInputStream> editStreams, FSNamesystem target, long maxTxnsToRead, HdfsServerConstants.StartupOption startOpt, MetaRecoveryContext recovery) throws IOException
      Throws:
      IOException

    • saveLegacyOIVImage

      public void saveLegacyOIVImage(FSNamesystem source, String targetDir, Canceler canceler) throws IOException
      Save FSimage in the legacy format. This is not for NN consumption, but for tools like OIV.
      Throws:
      IOException

    • updateStorageVersion

      public void updateStorageVersion() throws IOException
      Update version of all storage directories.
      Throws:
      IOException

    • saveNamespace

      public boolean saveNamespace(long timeWindow, long txGap, FSNamesystem source) throws IOException
      Parameters:
      timeWindow - a checkpoint is done if the latest checkpoint was done more than this number of seconds ago.
      txGap - a checkpoint is done also if the gap between the latest tx id and the latest checkpoint is greater than this number.
      Returns:
      true if a checkpoint has been made
      Throws:
      IOException
      See Also:

    • saveNamespace

      public void saveNamespace(FSNamesystem source) throws IOException
      Throws:
      IOException

    • saveNamespace

      public void saveNamespace(FSNamesystem source, NNStorage.NameNodeFile nnf, Canceler canceler) throws IOException
      Save the contents of the FS image to a new image file in each of the current storage directories.
      Throws:
      IOException

    • saveFSImageInAllDirs

      protected void saveFSImageInAllDirs(FSNamesystem source, long txid) throws IOException
      Throws:
      IOException
      See Also:
      • saveFSImageInAllDirs(FSNamesystem, NameNodeFile, long, Canceler)

    • addToCheckpointing

      public boolean addToCheckpointing(long txid)

    • removeFromCheckpointing

      public void removeFromCheckpointing(long txid)

    • saveDigestAndRenameCheckpointImage

      public void saveDigestAndRenameCheckpointImage(NNStorage.NameNodeFile nnf, long txid, org.apache.hadoop.io.MD5Hash digest) throws IOException
      This is called by the 2NN after having downloaded an image, and by the NN after having received a new image from the 2NN. It renames the image from fsimage_N.ckpt to fsimage_N and also saves the related .md5 file into place.
      Throws:
      IOException

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • getStorage

      public NNStorage getStorage()

    • getLayoutVersion

      public int getLayoutVersion()

    • getNamespaceID

      public int getNamespaceID()

    • getClusterID

      public String getClusterID()

    • getBlockPoolID

      public String getBlockPoolID()

    • getLastAppliedTxId

      public long getLastAppliedTxId()

    • getLastAppliedOrWrittenTxId

      public long getLastAppliedOrWrittenTxId()

    • getCorrectLastAppliedOrWrittenTxId

      public long getCorrectLastAppliedOrWrittenTxId()
      This method holds a lock of FSEditLog to get the correct value. This method must not be used for metrics.

    • updateLastAppliedTxIdFromWritten

      public void updateLastAppliedTxIdFromWritten()

    • getMostRecentCheckpointTxId

      public long getMostRecentCheckpointTxId()

    • getMostRecentNameNodeFileTxId

      public long getMostRecentNameNodeFileTxId(NNStorage.NameNodeFile nnf) throws IOException
      Given a NameNodeFile type, retrieve the latest txid for that file or HdfsServerConstants.INVALID_TXID if the file does not exist.
      Parameters:
      nnf - The NameNodeFile type to retrieve the latest txid from.
      Returns:
      the latest txid for the NameNodeFile type, or HdfsServerConstants.INVALID_TXID if there is no FSImage file of the type requested.
      Throws:
      IOException