Package org.apache.hadoop.hdfs.server.common

Class Storage

java.lang.Object
org.apache.hadoop.hdfs.server.common.StorageInfo
org.apache.hadoop.hdfs.server.common.Storage
Direct Known Subclasses:
BlockPoolSliceStorage, DataStorage, NNStorage
@Private public abstract class Storage extends StorageInfo
Storage information file.

Local storage information is stored in a separate file VERSION. It contains type of the node, the storage layout version, the namespace id, and the fs state creation time.

Local storage can reside in multiple directories. Each directory should contain the same VERSION file as the others. During startup Hadoop servers (name-node and data-nodes) read their local storage information from them.

The servers hold a lock for each storage directory while they run so that other nodes were not able to startup sharing the same storage. The locks are released when the servers stop (normally or abnormally).

  • Field Details

  • Constructor Details

  • Method Details

    • getFiles

      public List<File> getFiles(Storage.StorageDirType dirType, String fileName)
      Returns:
      A list of the given File in every available storage directory, regardless of whether it might exist.

    • dirIterator

      public Iterator<Storage.StorageDirectory> dirIterator()
      Return default iterator This iterator returns all entries in storageDirs

    • dirIterator

      public Iterator<Storage.StorageDirectory> dirIterator(Storage.StorageDirType dirType)
      Return iterator based on Storage Directory Type This iterator selects entries in storageDirs of type dirType and returns them via the Iterator

    • dirIterator

      public Iterator<Storage.StorageDirectory> dirIterator(boolean includeShared)
      Return all entries in storageDirs, potentially excluding shared dirs.
      Parameters:
      includeShared - whether or not to include shared dirs.
      Returns:
      an iterator over the configured storage dirs.

    • dirIterator

      public Iterator<Storage.StorageDirectory> dirIterator(Storage.StorageDirType dirType, boolean includeShared)
      Parameters:
      dirType - all entries will be of this type of dir
      includeShared - true to include any shared directories, false otherwise
      Returns:
      an iterator over the configured storage dirs.

    • dirIterable

      public Iterable<Storage.StorageDirectory> dirIterable(Storage.StorageDirType dirType)

    • listStorageDirectories

      public String listStorageDirectories()
      generate storage list (debug line)

    • getNumStorageDirs

      public int getNumStorageDirs()

    • getStorageDirs

      public List<Storage.StorageDirectory> getStorageDirs()

    • getStorageDir

      public Storage.StorageDirectory getStorageDir(int idx)

    • getSingularStorageDir

      public Storage.StorageDirectory getSingularStorageDir()
      Returns:
      the storage directory, with the precondition that this storage has exactly one storage directory

    • addStorageDir

      protected void addStorageDir(Storage.StorageDirectory sd)

    • containsStorageDir

      protected boolean containsStorageDir(File root) throws IOException
      Returns true if the storage directory on the given directory is already loaded.
      Parameters:
      root - the root directory of a Storage.StorageDirectory
      Throws:
      IOException - if failed to get canonical path.

    • containsStorageDir

      protected boolean containsStorageDir(StorageLocation location) throws IOException
      Returns true if the storage directory on the given directory is already loaded.
      Parameters:
      location - the StorageLocation
      Throws:
      IOException - if failed to get canonical path.

    • containsStorageDir

      protected boolean containsStorageDir(StorageLocation location, String bpid) throws IOException
      Returns true if the storage directory on the given location is already loaded.
      Parameters:
      location - the StorageLocation
      bpid - the block pool id
      Returns:
      true if the location matches to any existing storage directories
      Throws:
      IOException - IOException if failed to read location or storage directory path

    • getNamespaceInfo

      public NamespaceInfo getNamespaceInfo()

    • isPreUpgradableLayout

      public abstract boolean isPreUpgradableLayout(Storage.StorageDirectory sd) throws IOException
      Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.
      Throws:
      IOException

    • checkVersionUpgradable

      public static void checkVersionUpgradable(int oldVersion) throws IOException
      Checks if the upgrade from oldVersion is supported.
      Parameters:
      oldVersion - the version of the metadata to check with the current version
      Throws:
      IOException - if upgrade is not supported

    • confirmFormat

      public static boolean confirmFormat(Iterable<? extends Storage.FormatConfirmable> items, boolean force, boolean interactive) throws IOException
      Iterate over each of the Storage.FormatConfirmable objects, potentially checking with the user whether it should be formatted. If running in interactive mode, will prompt the user for each directory to allow them to format anyway. Otherwise, returns false, unless 'force' is specified.
      Parameters:
      force - format regardless of whether dirs exist
      interactive - prompt the user when a dir exists
      Returns:
      true if formatting should proceed
      Throws:
      IOException - if some storage cannot be accessed

    • setPropertiesFromFields

      protected void setPropertiesFromFields(Properties props, Storage.StorageDirectory sd) throws IOException
      Set common storage fields into the given properties object. Should be overloaded if additional fields need to be set.
      Parameters:
      props - the Properties object to write into
      Throws:
      IOException

    • writeProperties

      public void writeProperties(Storage.StorageDirectory sd) throws IOException
      Write properties to the VERSION file in the given storage directory.
      Throws:
      IOException

    • writeProperties

      public void writeProperties(File to, Storage.StorageDirectory sd) throws IOException
      Throws:
      IOException

    • writeProperties

      public static void writeProperties(File to, Properties props) throws IOException
      Throws:
      IOException

    • rename

      public static void rename(File from, File to) throws IOException
      Throws:
      IOException

    • nativeCopyFileUnbuffered

      public static void nativeCopyFileUnbuffered(File srcFile, File destFile, boolean preserveFileDate) throws IOException
      Copies a file (usually large) to a new location using native unbuffered IO.

      This method copies the contents of the specified source file to the specified destination file using OS specific unbuffered IO. The goal is to avoid churning the file system buffer cache when copying large files. We can't use FileUtils#copyFile from apache-commons-io because it is a buffered IO based on FileChannel#transferFrom, which uses MmapByteBuffer internally. The directory holding the destination file is created if it does not exist. If the destination file exists, then this method will delete it first.

      Note: Setting preserveFileDate to true tries to preserve the file's last modified date/times using File.setLastModified(long), however it is not guaranteed that the operation will succeed. If the modification operation fails, no indication is provided.

      Parameters:
      srcFile - an existing file to copy, must not be null
      destFile - the new file, must not be null
      preserveFileDate - true if the file date of the copy should be the same as the original
      Throws:
      NullPointerException - if source or destination is null
      IOException - if source or destination is invalid
      IOException - if an IO error occurs during copying

    • deleteDir

      public static void deleteDir(File dir) throws IOException
      Recursively delete all the content of the directory first and then the directory itself from the local filesystem.
      Parameters:
      dir - The directory to delete
      Throws:
      IOException

    • writeAll

      public void writeAll() throws IOException
      Write all data storage files.
      Throws:
      IOException

    • unlockAll

      public void unlockAll() throws IOException
      Unlock all storage directories.
      Throws:
      IOException

    • getBuildVersion

      public static String getBuildVersion()

    • getRegistrationID

      public static String getRegistrationID(StorageInfo storage)

    • is203LayoutVersion

      public static boolean is203LayoutVersion(int layoutVersion)