Package org.apache.hadoop.hdfs.server.datanode

Class FileIoProvider

java.lang.Object
org.apache.hadoop.hdfs.server.datanode.FileIoProvider
@Private @Unstable public class FileIoProvider extends Object
This class abstracts out various file IO operations performed by the DataNode and invokes profiling (for collecting stats) and fault injection (for testing) event hooks before and after each file IO. Behavior can be injected into these events by enabling the profiling and/or fault injection event hooks through DFSConfigKeys.DFS_DATANODE_FILEIO_PROFILING_SAMPLING_PERCENTAGE_KEY and DFSConfigKeys.DFS_DATANODE_ENABLE_FILEIO_FAULT_INJECTION_KEY. These event hooks are disabled by default. Most functions accept an optional FsVolumeSpi parameter for instrumentation/logging. Some methods may look redundant, especially the multiple variations of move/rename/list. They exist to retain behavior compatibility for existing code.

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

  • Constructor Details

    • FileIoProvider

      public FileIoProvider(@Nullable org.apache.hadoop.conf.Configuration conf, DataNode datanode)
      Parameters:
      conf - Configuration object. May be null. When null, the event handlers are no-ops.
      datanode - datanode that owns this FileIoProvider. Used for IO error based volume checker callback

  • Method Details

    • flush

      public void flush(@Nullable FsVolumeSpi volume, Flushable f) throws IOException
      See Flushable.flush().
      Parameters:
      volume - target volume. null if unavailable.
      Throws:
      IOException

    • sync

      public void sync(@Nullable FsVolumeSpi volume, FileOutputStream fos) throws IOException
      Sync the given FileOutputStream.
      Parameters:
      volume - target volume. null if unavailable.
      Throws:
      IOException

    • dirSync

      public void dirSync(@Nullable FsVolumeSpi volume, File dir) throws IOException
      Sync the given directory changes to durable device.
      Throws:
      IOException

    • syncFileRange

      public void syncFileRange(@Nullable FsVolumeSpi volume, FileDescriptor outFd, long offset, long numBytes, int flags) throws org.apache.hadoop.io.nativeio.NativeIOException
      Call sync_file_range on the given file descriptor.
      Parameters:
      volume - target volume. null if unavailable.
      Throws:
      org.apache.hadoop.io.nativeio.NativeIOException

    • posixFadvise

      public void posixFadvise(@Nullable FsVolumeSpi volume, String identifier, FileDescriptor outFd, long offset, long length, int flags) throws org.apache.hadoop.io.nativeio.NativeIOException
      Call posix_fadvise on the given file descriptor.
      Parameters:
      volume - target volume. null if unavailable.
      Throws:
      org.apache.hadoop.io.nativeio.NativeIOException

    • delete

      public boolean delete(@Nullable FsVolumeSpi volume, File f)
      Delete a file.
      Parameters:
      volume - target volume. null if unavailable.
      f - File to delete.
      Returns:
      true if the file was successfully deleted.

    • deleteWithExistsCheck

      public boolean deleteWithExistsCheck(@Nullable FsVolumeSpi volume, File f)
      Delete a file, first checking to see if it exists.
      Parameters:
      volume - target volume. null if unavailable.
      f - File to delete
      Returns:
      true if the file was successfully deleted or if it never existed.

    • transferToSocketFully

      public void transferToSocketFully(@Nullable FsVolumeSpi volume, org.apache.hadoop.net.SocketOutputStream sockOut, FileChannel fileCh, long position, int count, org.apache.hadoop.io.LongWritable waitTime, org.apache.hadoop.io.LongWritable transferTime) throws IOException
      Transfer data from a FileChannel to a SocketOutputStream.
      Parameters:
      volume - target volume. null if unavailable.
      sockOut - SocketOutputStream to write the data.
      fileCh - FileChannel from which to read data.
      position - position within the channel where the transfer begins.
      count - number of bytes to transfer.
      waitTime - returns the nanoseconds spent waiting for the socket to become writable.
      transferTime - returns the nanoseconds spent transferring data.
      Throws:
      IOException

    • createFile

      public boolean createFile(@Nullable FsVolumeSpi volume, File f) throws IOException
      Create a file.
      Parameters:
      volume - target volume. null if unavailable.
      f - File to be created.
      Returns:
      true if the file does not exist and was successfully created. false if the file already exists.
      Throws:
      IOException

    • getFileInputStream

      public FileInputStream getFileInputStream(@Nullable FsVolumeSpi volume, File f) throws FileNotFoundException
      Create a FileInputStream using FileInputStream(File). Wraps the created input stream to intercept read calls before delegating to the wrapped stream.
      Parameters:
      volume - target volume. null if unavailable.
      f - File object.
      Returns:
      FileInputStream to the given file.
      Throws:
      FileNotFoundException

    • getFileOutputStream

      public FileOutputStream getFileOutputStream(@Nullable FsVolumeSpi volume, File f, boolean append) throws FileNotFoundException
      Create a FileOutputStream using FileOutputStream(File, boolean). Wraps the created output stream to intercept write calls before delegating to the wrapped stream.
      Parameters:
      volume - target volume. null if unavailable.
      f - File object.
      append - if true, then bytes will be written to the end of the file rather than the beginning.
      Returns:
      FileOutputStream to the given file object.
      Throws:
      FileNotFoundException

    • getFileOutputStream

      public FileOutputStream getFileOutputStream(@Nullable FsVolumeSpi volume, File f) throws FileNotFoundException
      Create a FileOutputStream using FileOutputStream(File, boolean). Wraps the created output stream to intercept write calls before delegating to the wrapped stream.
      Parameters:
      volume - target volume. null if unavailable.
      f - File object.
      Returns:
      FileOutputStream to the given file object.
      Throws:
      FileNotFoundException

    • getFileOutputStream

      public FileOutputStream getFileOutputStream(@Nullable FsVolumeSpi volume, FileDescriptor fd)
      Create a FileOutputStream using FileOutputStream(FileDescriptor). Wraps the created output stream to intercept write calls before delegating to the wrapped stream.
      Parameters:
      volume - target volume. null if unavailable.
      fd - File descriptor object.
      Returns:
      FileOutputStream to the given file object.

    • getShareDeleteFileInputStream

      public FileInputStream getShareDeleteFileInputStream(@Nullable FsVolumeSpi volume, File f, long offset) throws IOException
      Create a FileInputStream using NativeIO.getShareDeleteFileDescriptor(java.io.File, long). Wraps the created input stream to intercept input calls before delegating to the wrapped stream.
      Parameters:
      volume - target volume. null if unavailable.
      f - File object.
      offset - the offset position, measured in bytes from the beginning of the file, at which to set the file pointer.
      Returns:
      FileOutputStream to the given file object.
      Throws:
      FileNotFoundException
      IOException

    • openAndSeek

      public FileInputStream openAndSeek(@Nullable FsVolumeSpi volume, File f, long offset) throws IOException
      Create a FileInputStream using FileInputStream(File) and position it at the given offset. Wraps the created input stream to intercept read calls before delegating to the wrapped stream.
      Parameters:
      volume - target volume. null if unavailable.
      f - File object.
      offset - the offset position, measured in bytes from the beginning of the file, at which to set the file pointer.
      Throws:
      FileNotFoundException
      IOException

    • getRandomAccessFile

      public RandomAccessFile getRandomAccessFile(@Nullable FsVolumeSpi volume, File f, String mode) throws FileNotFoundException
      Create a RandomAccessFile using RandomAccessFile(File, String). Wraps the created input stream to intercept IO calls before delegating to the wrapped RandomAccessFile.
      Parameters:
      volume - target volume. null if unavailable.
      f - File object.
      mode - See RandomAccessFile for a description of the mode string.
      Returns:
      RandomAccessFile representing the given file.
      Throws:
      FileNotFoundException

    • fullyDelete

      public boolean fullyDelete(@Nullable FsVolumeSpi volume, File dir)
      Delete the given directory using FileUtil.fullyDelete(File).
      Parameters:
      volume - target volume. null if unavailable.
      dir - directory to be deleted.
      Returns:
      true on success false on failure.

    • replaceFile

      public void replaceFile(@Nullable FsVolumeSpi volume, File src, File target) throws IOException
      Move the src file to the target using FileUtil.replaceFile(File, File).
      Parameters:
      volume - target volume. null if unavailable.
      src - source path.
      target - target path.
      Throws:
      IOException

    • rename

      public void rename(@Nullable FsVolumeSpi volume, File src, File target) throws IOException
      Move the src file to the target using Storage.rename(File, File).
      Parameters:
      volume - target volume. null if unavailable.
      src - source path.
      target - target path.
      Throws:
      IOException

    • moveFile

      public void moveFile(@Nullable FsVolumeSpi volume, File src, File target) throws IOException
      Move the src file to the target using FileUtils.moveFile(File, File).
      Parameters:
      volume - target volume. null if unavailable.
      src - source path.
      target - target path.
      Throws:
      IOException

    • move

      public void move(@Nullable FsVolumeSpi volume, Path src, Path target, CopyOption... options) throws IOException
      Move the src file to the target using Files.move(Path, Path, CopyOption...).
      Parameters:
      volume - target volume. null if unavailable.
      src - source path.
      target - target path.
      options - See Files.move(java.nio.file.Path, java.nio.file.Path, java.nio.file.CopyOption...) for a description of the options.
      Throws:
      IOException

    • nativeCopyFileUnbuffered

      public void nativeCopyFileUnbuffered(@Nullable FsVolumeSpi volume, File src, File target, boolean preserveFileDate) throws IOException
      See Storage.nativeCopyFileUnbuffered(File, File, boolean).
      Parameters:
      volume - target volume. null if unavailable.
      src - an existing file to copy, must not be null
      target - the new file, must not be null
      preserveFileDate - true if the file date of the copy should be the same as the original
      Throws:
      IOException

    • mkdirs

      public boolean mkdirs(@Nullable FsVolumeSpi volume, File dir) throws IOException
      See File.mkdirs().
      Parameters:
      volume - target volume. null if unavailable.
      dir - directory to be created.
      Returns:
      true only if the directory was created. false if the directory already exists.
      Throws:
      IOException - if a directory with the given name does not exist and could not be created.

    • mkdirsWithExistsCheck

      public void mkdirsWithExistsCheck(@Nullable FsVolumeSpi volume, File dir) throws IOException
      Create the target directory using File.mkdirs() only if it doesn't exist already.
      Parameters:
      volume - target volume. null if unavailable.
      dir - directory to be created.
      Throws:
      IOException - if the directory could not created

    • listFiles

      public File[] listFiles(@Nullable FsVolumeSpi volume, File dir) throws IOException
      Get a listing of the given directory using FileUtil.listFiles(File).
      Parameters:
      volume - target volume. null if unavailable.
      dir - Directory to be listed.
      Returns:
      array of file objects representing the directory entries.
      Throws:
      IOException

    • list

      public String[] list(@Nullable FsVolumeSpi volume, File dir) throws IOException
      Get a listing of the given directory using FileUtil.listFiles(File).
      Parameters:
      volume - target volume. null if unavailable.
      dir - directory to be listed.
      Returns:
      array of strings representing the directory entries.
      Throws:
      IOException

    • listDirectory

      public List<String> listDirectory(@Nullable FsVolumeSpi volume, File dir, FilenameFilter filter) throws IOException
      Get a listing of the given directory using IOUtils.listDirectory(File, FilenameFilter).
      Parameters:
      volume - target volume. null if unavailable.
      dir - Directory to list.
      filter - FilenameFilter to filter the directory entries.
      Throws:
      IOException

    • getHardLinkCount

      public int getHardLinkCount(@Nullable FsVolumeSpi volume, File f) throws IOException
      Retrieves the number of links to the specified file.
      Parameters:
      volume - target volume. null if unavailable.
      f - file whose link count is being queried.
      Returns:
      number of hard-links to the given file, including the given path itself.
      Throws:
      IOException

    • exists

      public boolean exists(@Nullable FsVolumeSpi volume, File f)
      Check for file existence using File.exists().
      Parameters:
      volume - target volume. null if unavailable.
      f - file object.
      Returns:
      true if the file exists.

    • getProfilingEventHook

      public org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents getProfilingEventHook()