Package org.apache.hadoop.hdfs.server.namenode

Class NNStorage

java.lang.Object
org.apache.hadoop.hdfs.server.common.StorageInfo
org.apache.hadoop.hdfs.server.common.Storage
org.apache.hadoop.hdfs.server.namenode.NNStorage
All Implemented Interfaces:
Closeable, AutoCloseable, StorageErrorReporter
@Private public class NNStorage extends Storage implements Closeable, StorageErrorReporter
NNStorage is responsible for management of the StorageDirectories used by the NameNode.

  • Field Details

    • blockpoolID

      protected String blockpoolID

    • mostRecentCheckpointTxId

      protected volatile long mostRecentCheckpointTxId
      TxId of the last transaction that was included in the most recent fsimage file. This does not include any transactions that have since been written to the edit log.

    • removedStorageDirs

      protected final List<Storage.StorageDirectory> removedStorageDirs
      List of failed (and thus removed) storages.

  • Constructor Details

    • NNStorage

      public NNStorage(org.apache.hadoop.conf.Configuration conf, Collection<URI> imageDirs, Collection<URI> editsDirs) throws IOException
      Construct the NNStorage.
      Parameters:
      conf - Namenode configuration.
      imageDirs - Directories the image can be stored in.
      editsDirs - Directories the editlog can be stored in.
      Throws:
      IOException - if any directories are inaccessible.

  • Method Details

    • isPreUpgradableLayout

      public boolean isPreUpgradableLayout(Storage.StorageDirectory sd) throws IOException
      Description copied from class: Storage
      Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.
      Specified by:
      isPreUpgradableLayout in class Storage
      Throws:
      IOException

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • getStorageDirectory

      public Storage.StorageDirectory getStorageDirectory(URI uri)
      Return the storage directory corresponding to the passed URI.
      Parameters:
      uri - URI of a storage directory
      Returns:
      The matching storage directory or null if none found

    • getMostRecentCheckpointTxId

      public long getMostRecentCheckpointTxId()
      Returns:
      the transaction ID of the last checkpoint.

    • getMostRecentCheckpointTime

      public long getMostRecentCheckpointTime()
      Returns:
      the time of the most recent checkpoint in millis since the epoch.

    • writeTransactionIdFileToStorage

      public void writeTransactionIdFileToStorage(long txid)
      Write a small file in all available storage directories that indicates that the namespace has reached some given transaction ID. This is used when the image is loaded to avoid accidental rollbacks in the case where an edit log is fully deleted but there is no checkpoint. See TestNameEditsConfigs.testNameEditsConfigsFailure()
      Parameters:
      txid - the txid that has been reached

    • writeTransactionIdFileToStorage

      public void writeTransactionIdFileToStorage(long txid, NNStorage.NameNodeDirType type)
      Write a small file in all available storage directories that indicates that the namespace has reached some given transaction ID. This is used when the image is loaded to avoid accidental rollbacks in the case where an edit log is fully deleted but there is no checkpoint. See TestNameEditsConfigs.testNameEditsConfigsFailure()
      Parameters:
      txid - the txid that has been reached
      type - the type of directory

    • getFsImageNameCheckpoint

      public File[] getFsImageNameCheckpoint(long txid)
      Return the name of the image file that is uploaded by periodic checkpointing.
      Returns:
      List of filenames to save checkpoints to.

    • getFsImageName

      public File getFsImageName(long txid, NNStorage.NameNodeFile nnf)
      Returns:
      The first image file with the given txid and image type.

    • getFsImage

      public File getFsImage(long txid, EnumSet<NNStorage.NameNodeFile> nnfs)
      Returns:
      The first image file whose txid is the same with the given txid and image type is one of the given types.

    • getFsImageName

      public File getFsImageName(long txid)

    • getHighestFsImageName

      public File getHighestFsImageName()

    • format

      public void format(NamespaceInfo nsInfo) throws IOException
      Format all available storage directories.
      Throws:
      IOException

    • format

      public void format(NamespaceInfo nsInfo, boolean isRollingUpgrade) throws IOException
      Format all available storage directories.
      Throws:
      IOException

    • newNamespaceInfo

      public static NamespaceInfo newNamespaceInfo() throws UnknownHostException
      Throws:
      UnknownHostException

    • format

      public void format() throws IOException
      Throws:
      IOException

    • setFieldsFromProperties

      protected void setFieldsFromProperties(Properties props, Storage.StorageDirectory sd) throws IOException
      Description copied from class: StorageInfo
      Get common storage fields. Should be overloaded if additional fields need to be get.
      Overrides:
      setFieldsFromProperties in class StorageInfo
      Parameters:
      props - properties
      Throws:
      IOException - on error

    • setPropertiesFromFields

      protected void setPropertiesFromFields(Properties props, Storage.StorageDirectory sd) throws IOException
      Write version file into the storage directory. The version file should always be written last. Missing or corrupted version file indicates that the checkpoint is not valid.
      Overrides:
      setPropertiesFromFields in class Storage
      Parameters:
      sd - storage directory
      props - the Properties object to write into
      Throws:
      IOException

    • getCheckpointImageFileName

      @VisibleForTesting public static String getCheckpointImageFileName(long txid)

    • getImageFileName

      @VisibleForTesting public static String getImageFileName(long txid)

    • getRollbackImageFileName

      @VisibleForTesting public static String getRollbackImageFileName(long txid)

    • getLegacyOIVImageFileName

      public static String getLegacyOIVImageFileName(long txid)

    • getInProgressEditsFileName

      @VisibleForTesting public static String getInProgressEditsFileName(long startTxId)

    • getFinalizedEditsFile

      public static File getFinalizedEditsFile(Storage.StorageDirectory sd, long startTxId, long endTxId)

    • getTemporaryEditsFile

      public static File getTemporaryEditsFile(Storage.StorageDirectory sd, long startTxId, long endTxId, long timestamp)

    • getFinalizedEditsFileName

      @VisibleForTesting public static String getFinalizedEditsFileName(long startTxId, long endTxId)

    • getTemporaryEditsFileName

      public static String getTemporaryEditsFileName(long startTxId, long endTxId, long timestamp)

    • reportErrorOnFile

      public void reportErrorOnFile(File f)
      Report that an IOE has occurred on some file which may or may not be within one of the NN image storage directories.
      Specified by:
      reportErrorOnFile in interface StorageErrorReporter
      Parameters:
      f - the file which had an error.

    • newClusterID

      public static String newClusterID()
      Generate new clusterID. clusterID is a persistent attribute of the cluster. It is generated when the cluster is created and remains the same during the life cycle of the cluster. When a new name node is formated, if this is a new cluster, a new clusterID is geneated and stored. Subsequent name node must be given the same ClusterID during its format to be in the same cluster. When a datanode register it receive the clusterID and stick with it. If at any point, name node or data node tries to join another cluster, it will be rejected.
      Returns:
      new clusterID

    • determineClusterId

      public String determineClusterId()
      Try to find current cluster id in the VERSION files. returns first cluster id found in any VERSION file null in case none found
      Returns:
      clusterId or null in case no cluster id found

    • setBlockPoolID

      public void setBlockPoolID(String bpid)
      Validate and set block pool ID.

    • getBlockPoolID

      public String getBlockPoolID()

    • getNamespaceInfo

      public NamespaceInfo getNamespaceInfo()
      Overrides:
      getNamespaceInfo in class Storage

    • getNNDirectorySize

      public String getNNDirectorySize()

    • updateNameDirSize

      public void updateNameDirSize()

    • writeAll

      public void writeAll() throws IOException
      Write all data storage files.
      Overrides:
      writeAll in class Storage
      Throws:
      IOException - When all the storage directory fails to write VERSION file