Package org.apache.hadoop.fs

Class FilterFileSystem

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.FilterFileSystem
All Implemented Interfaces:
Closeable, AutoCloseable, Configurable, BulkDeleteSource, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer
Direct Known Subclasses:
ChecksumFileSystem
@Public @Stable public class FilterFileSystem extends FileSystem
A FilterFileSystem contains some other file system, which it uses as its basic file system, possibly transforming the data along the way or providing additional functionality. The class FilterFileSystem itself simply overrides all methods of FileSystem with versions that pass all requests to the contained file system. Subclasses of FilterFileSystem may further override some of these methods and may also provide additional methods and fields.

  • Field Details

  • Constructor Details

    • FilterFileSystem

      public FilterFileSystem()

    • FilterFileSystem

      public FilterFileSystem(FileSystem fs)

  • Method Details

    • getRawFileSystem

      public FileSystem getRawFileSystem()
      Get the raw file system
      Returns:
      FileSystem being filtered

    • initialize

      public void initialize(URI name, Configuration conf) throws IOException
      Called after a new FileSystem instance is constructed.
      Overrides:
      initialize in class FileSystem
      Parameters:
      name - a uri whose authority section names the host, port, etc. for this FileSystem
      conf - the configuration
      Throws:
      IOException - on any failure to initialize this instance.

    • getUri

      public URI getUri()
      Returns a URI whose scheme and authority identify this FileSystem.
      Specified by:
      getUri in class FileSystem
      Returns:
      the URI of this filesystem.

    • getCanonicalUri

      protected URI getCanonicalUri()
      Description copied from class: FileSystem
      Return a canonicalized form of this FileSystem's URI. The default implementation simply calls FileSystem.canonicalizeUri(URI) on the filesystem's own URI, so subclasses typically only need to implement that method.
      Overrides:
      getCanonicalUri in class FileSystem
      Returns:
      the URI of this filesystem.
      See Also:

    • canonicalizeUri

      protected URI canonicalizeUri(URI uri)
      Description copied from class: FileSystem
      Canonicalize the given URI. This is implementation-dependent, and may for example consist of canonicalizing the hostname using DNS and adding the default port if not specified. The default implementation simply fills in the default port if not specified and if FileSystem.getDefaultPort() returns a default port.
      Overrides:
      canonicalizeUri in class FileSystem
      Parameters:
      uri - url.
      Returns:
      URI
      See Also:
      • NetUtils.getCanonicalUri(URI, int)

    • makeQualified

      public Path makeQualified(Path path)
      Make sure that a path specifies a FileSystem.
      Overrides:
      makeQualified in class FileSystem
      Parameters:
      path - to qualify.
      Returns:
      this path if it contains a scheme and authority and is absolute, or a new path that includes a path and authority and is fully qualified
      See Also:
      • Path.makeQualified(URI, Path)

    • checkPath

      protected void checkPath(Path path)
      Check that a Path belongs to this FileSystem.
      Overrides:
      checkPath in class FileSystem
      Parameters:
      path - to check

    • getFileBlockLocations

      public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) throws IOException
      Description copied from class: FileSystem
      Return an array containing hostnames, offset and size of portions of the given file. For nonexistent file or regions, null is returned. 
         if f == null :
     result = null
   elif f.getLen() <= start:
     result = []
   else result = [ locations(FS, b) for b in blocks(FS, p, s, s+l)]
      This call is most helpful with and distributed filesystem where the hostnames of machines that contain blocks of the given file can be determined. The default implementation returns an array containing one element: 
       BlockLocation( { "localhost:9866" },  { "localhost" }, 0, file.getLen())
      In HDFS, if file is three-replicated, the returned array contains elements like: 
       BlockLocation(offset: 0, length: BLOCK_SIZE,
   hosts: {"host1:9866", "host2:9866, host3:9866"})
 BlockLocation(offset: BLOCK_SIZE, length: BLOCK_SIZE,
   hosts: {"host2:9866", "host3:9866, host4:9866"})
      And if a file is erasure-coded, the returned BlockLocation are logical block groups. Suppose we have a RS_3_2 coded file (3 data units and 2 parity units). 1. If the file size is less than one stripe size, say 2 * CELL_SIZE, then there will be one BlockLocation returned, with 0 offset, actual file size and 4 hosts (2 data blocks and 2 parity blocks) hosting the actual blocks. 3. If the file size is less than one group size but greater than one stripe size, then there will be one BlockLocation returned, with 0 offset, actual file size with 5 hosts (3 data blocks and 2 parity blocks) hosting the actual blocks. 4. If the file size is greater than one group size, 3 * BLOCK_SIZE + 123 for example, then the result will be like: 
       BlockLocation(offset: 0, length: 3 * BLOCK_SIZE, hosts: {"host1:9866",
   "host2:9866","host3:9866","host4:9866","host5:9866"})
 BlockLocation(offset: 3 * BLOCK_SIZE, length: 123, hosts: {"host1:9866",
   "host4:9866", "host5:9866"})
      Overrides:
      getFileBlockLocations in class FileSystem
      Parameters:
      file - FilesStatus to get data from
      start - offset into the given file
      len - length for which to get locations for
      Returns:
      block location array.
      Throws:
      IOException - IO failure

    • resolvePath

      public Path resolvePath(Path p) throws IOException
      Description copied from class: FileSystem
      Return the fully-qualified path of path, resolving the path through any symlinks or mount point.
      Overrides:
      resolvePath in class FileSystem
      Parameters:
      p - path to be resolved
      Returns:
      fully qualified path
      Throws:
      FileNotFoundException - if the path is not present
      IOException - for any other error

    • open

      public FSDataInputStream open(Path f, int bufferSize) throws IOException
      Opens an FSDataInputStream at the indicated Path.
      Specified by:
      open in class FileSystem
      Parameters:
      f - the file name to open
      bufferSize - the size of the buffer to be used.
      Returns:
      input stream.
      Throws:
      IOException - IO failure

    • open

      public FSDataInputStream open(PathHandle fd, int bufferSize) throws IOException
      Description copied from class: FileSystem
      Open an FSDataInputStream matching the PathHandle instance. The implementation may encode metadata in PathHandle to address the resource directly and verify that the resource referenced satisfies constraints specified at its construciton.
      Overrides:
      open in class FileSystem
      Parameters:
      fd - PathHandle object returned by the FS authority.
      bufferSize - the size of the buffer to use
      Returns:
      input stream.
      Throws:
      InvalidPathHandleException - If PathHandle constraints are not satisfied
      IOException - IO failure

    • createPathHandle

      protected PathHandle createPathHandle(FileStatus stat, org.apache.hadoop.fs.Options.HandleOpt... opts)
      Description copied from class: FileSystem
      Hook to implement support for PathHandle operations.
      Overrides:
      createPathHandle in class FileSystem
      Parameters:
      stat - Referent in the target FileSystem
      opts - Constraints that determine the validity of the PathHandle reference.
      Returns:
      path handle.

    • append

      public FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws IOException
      Description copied from class: FileSystem
      Append to an existing file (optional operation).
      Specified by:
      append in class FileSystem
      Parameters:
      f - the existing file to be appended.
      bufferSize - the size of the buffer to be used.
      progress - for reporting progress if it is not null.
      Returns:
      output stream.
      Throws:
      IOException - IO failure

    • concat

      public void concat(Path f, Path[] psrcs) throws IOException
      Description copied from class: FileSystem
      Concat existing files together.
      Overrides:
      concat in class FileSystem
      Parameters:
      f - the path to the target destination.
      psrcs - the paths to the sources to use for the concatenation.
      Throws:
      IOException - IO failure

    • create

      public FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
      Description copied from class: FileSystem
      Create an FSDataOutputStream at the indicated Path with write-progress reporting.
      Specified by:
      create in class FileSystem
      Parameters:
      f - the file name to open
      permission - file permission
      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
      bufferSize - the size of the buffer to be used.
      replication - required block replication for the file.
      blockSize - block size
      progress - the progress reporter
      Returns:
      output stream.
      Throws:
      IOException - IO failure
      See Also:

    • create

      public FSDataOutputStream create(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt) throws IOException
      Description copied from class: FileSystem
      Create an FSDataOutputStream at the indicated Path with a custom checksum option.
      Overrides:
      create in class FileSystem
      Parameters:
      f - the file name to open
      permission - file permission
      flags - CreateFlags to use for this stream.
      bufferSize - the size of the buffer to be used.
      replication - required block replication for the file.
      blockSize - block size
      progress - the progress reporter
      checksumOpt - checksum parameter. If null, the values found in conf will be used.
      Returns:
      output stream.
      Throws:
      IOException - IO failure
      See Also:

    • listLocatedStatus

      protected org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> listLocatedStatus(Path f, PathFilter filter) throws FileNotFoundException, IOException
      Description copied from class: FileSystem
      List a directory. The returned results include its block location if it is a file The results are filtered by the given path filter
      Overrides:
      listLocatedStatus in class FileSystem
      Parameters:
      f - a path
      filter - a path filter
      Returns:
      an iterator that traverses statuses of the files/directories in the given path
      Throws:
      FileNotFoundException - if f does not exist
      IOException - if any I/O error occurred

    • createNonRecursive

      public FSDataOutputStream createNonRecursive(Path f, FsPermission permission, EnumSet<CreateFlag> flags, int bufferSize, short replication, long blockSize, Progressable progress) throws IOException
      Description copied from class: FileSystem
      Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.
      Overrides:
      createNonRecursive in class FileSystem
      Parameters:
      f - the file name to open
      permission - file permission
      flags - CreateFlags to use for this stream.
      bufferSize - the size of the buffer to be used.
      replication - required block replication for the file.
      blockSize - block size
      progress - the progress reporter
      Returns:
      output stream.
      Throws:
      IOException - IO failure
      See Also:

    • setReplication

      public boolean setReplication(Path src, short replication) throws IOException
      Set replication for an existing file.
      Overrides:
      setReplication in class FileSystem
      Parameters:
      src - file name
      replication - new replication
      Returns:
      true if successful; false if file does not exist or is a directory
      Throws:
      IOException - raised on errors performing I/O.

    • rename

      public boolean rename(Path src, Path dst) throws IOException
      Renames Path src to Path dst. Can take place on local fs or remote DFS.
      Specified by:
      rename in class FileSystem
      Parameters:
      src - path to be renamed
      dst - new path after rename
      Returns:
      true if rename is successful
      Throws:
      IOException - on failure

    • rename

      protected void rename(Path src, Path dst, Options.Rename... options) throws IOException
      Description copied from class: FileSystem
      Renames Path src to Path dst
      • Fails if src is a file and dst is a directory.
      • Fails if src is a directory and dst is a file.
      • Fails if the parent of dst does not exist or is a file.

      If OVERWRITE option is not passed as an argument, rename fails if the dst already exists.

      If OVERWRITE option is passed as an argument, rename overwrites the dst if it is a file or an empty directory. Rename fails if dst is a non-empty directory.

      Note that atomicity of rename is dependent on the file system implementation. Please refer to the file system documentation for details. This default implementation is non atomic.

      This method is deprecated since it is a temporary method added to support the transition from FileSystem to FileContext for user applications.

      Overrides:
      rename in class FileSystem
      Parameters:
      src - path to be renamed
      dst - new path after rename
      options - rename options.
      Throws:
      FileNotFoundException - src path does not exist, or the parent path of dst does not exist.
      FileAlreadyExistsException - dest path exists and is a file
      ParentNotDirectoryException - if the parent path of dest is not a directory
      IOException - on failure

    • truncate

      public boolean truncate(Path f, long newLength) throws IOException
      Description copied from class: FileSystem
      Truncate the file in the indicated path to the indicated size.
      • Fails if path is a directory.
      • Fails if path does not exist.
      • Fails if path is not closed.
      • Fails if new size is greater than current size.
      Overrides:
      truncate in class FileSystem
      Parameters:
      f - The path to the file to be truncated
      newLength - The size the file is to be truncated to
      Returns:
      true if the file has been truncated to the desired newLength and is immediately available to be reused for write operations such as append, or false if a background process of adjusting the length of the last block has been started, and clients should wait for it to complete before proceeding with further file updates.
      Throws:
      IOException - IO failure

    • delete

      public boolean delete(Path f, boolean recursive) throws IOException
      Delete a file
      Specified by:
      delete in class FileSystem
      Parameters:
      f - the path to delete.
      recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.
      Returns:
      true if delete is successful else false.
      Throws:
      IOException - IO failure

    • listStatus

      public FileStatus[] listStatus(Path f) throws IOException
      List files in a directory.
      Specified by:
      listStatus in class FileSystem
      Parameters:
      f - given path
      Returns:
      the statuses of the files/directories in the given patch
      Throws:
      FileNotFoundException - when the path does not exist
      IOException - see specific implementation

    • listCorruptFileBlocks

      public org.apache.hadoop.fs.RemoteIterator<Path> listCorruptFileBlocks(Path path) throws IOException
      Description copied from class: FileSystem
      List corrupted file blocks.
      Overrides:
      listCorruptFileBlocks in class FileSystem
      Parameters:
      path - the path.
      Returns:
      an iterator over the corrupt files under the given path (may contain duplicates if a file has more than one corrupt block)
      Throws:
      IOException - IO failure

    • listLocatedStatus

      public org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus> listLocatedStatus(Path f) throws IOException
      List files and its block locations in a directory.
      Overrides:
      listLocatedStatus in class FileSystem
      Parameters:
      f - is the path
      Returns:
      an iterator that traverses statuses of the files/directories in the given path
      Throws:
      FileNotFoundException - If f does not exist
      IOException - If an I/O error occurred

    • listStatusIterator

      public org.apache.hadoop.fs.RemoteIterator<FileStatus> listStatusIterator(Path f) throws IOException
      Return a remote iterator for listing in a directory
      Overrides:
      listStatusIterator in class FileSystem
      Parameters:
      f - target path
      Returns:
      remote iterator
      Throws:
      FileNotFoundException - if p does not exist
      IOException - if any I/O error occurred

    • getHomeDirectory

      public Path getHomeDirectory()
      Description copied from class: FileSystem
      Return the current user's home directory in this FileSystem. The default implementation returns "/user/$USER/".
      Overrides:
      getHomeDirectory in class FileSystem
      Returns:
      the path.

    • setWorkingDirectory

      public void setWorkingDirectory(Path newDir)
      Set the current working directory for the given file system. All relative paths will be resolved relative to it.
      Specified by:
      setWorkingDirectory in class FileSystem
      Parameters:
      newDir - new dir.

    • getWorkingDirectory

      public Path getWorkingDirectory()
      Get the current working directory for the given file system
      Specified by:
      getWorkingDirectory in class FileSystem
      Returns:
      the directory pathname

    • getInitialWorkingDirectory

      protected Path getInitialWorkingDirectory()
      Description copied from class: FileSystem
      Note: with the new FileContext class, getWorkingDirectory() will be removed. The working directory is implemented in FileContext. Some FileSystems like LocalFileSystem have an initial workingDir that we use as the starting workingDir. For other file systems like HDFS there is no built in notion of an initial workingDir.
      Overrides:
      getInitialWorkingDirectory in class FileSystem
      Returns:
      if there is built in notion of workingDir then it is returned; else a null is returned.

    • getStatus

      public FsStatus getStatus(Path p) throws IOException
      Description copied from class: FileSystem
      Returns a status object describing the use and capacity of the filesystem. If the filesystem has multiple partitions, the use and capacity of the partition pointed to by the specified path is reflected.
      Overrides:
      getStatus in class FileSystem
      Parameters:
      p - Path for which status should be obtained. null means the default partition.
      Returns:
      a FsStatus object
      Throws:
      IOException - see specific implementation

    • mkdirs

      public boolean mkdirs(Path f, FsPermission permission) throws IOException
      Description copied from class: FileSystem
      Make the given file and all non-existent parents into directories. Has roughly the semantics of Unix @{code mkdir -p}. Existence of the directory hierarchy is not an error.
      Specified by:
      mkdirs in class FileSystem
      Parameters:
      f - path to create
      permission - to apply to f
      Returns:
      if mkdir success true, not false.
      Throws:
      IOException - IO failure

    • mkdirs

      public boolean mkdirs(Path f) throws IOException
      Description copied from class: FileSystem
      Call FileSystem.mkdirs(Path, FsPermission) with default permission.
      Overrides:
      mkdirs in class FileSystem
      Parameters:
      f - path
      Returns:
      true if the directory was created
      Throws:
      IOException - IO failure

    • copyFromLocalFile

      public void copyFromLocalFile(boolean delSrc, Path src, Path dst) throws IOException
      The src file is on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed
      Overrides:
      copyFromLocalFile in class FileSystem
      Parameters:
      delSrc - whether to delete the src
      src - path
      dst - path
      Throws:
      IOException - IO failure.

    • copyFromLocalFile

      public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst) throws IOException
      The src files are on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed
      Overrides:
      copyFromLocalFile in class FileSystem
      Parameters:
      delSrc - whether to delete the src
      overwrite - whether to overwrite an existing file
      srcs - array of paths which are source
      dst - path
      Throws:
      IOException - IO failure

    • copyFromLocalFile

      public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path src, Path dst) throws IOException
      The src file is on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed
      Overrides:
      copyFromLocalFile in class FileSystem
      Parameters:
      delSrc - whether to delete the src
      overwrite - whether to overwrite an existing file
      src - path
      dst - path
      Throws:
      IOException - IO failure

    • copyToLocalFile

      public void copyToLocalFile(boolean delSrc, Path src, Path dst) throws IOException
      The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. delSrc indicates if the src will be removed or not.
      Overrides:
      copyToLocalFile in class FileSystem
      Parameters:
      delSrc - whether to delete the src
      src - path src file in the remote filesystem
      dst - path local destination
      Throws:
      IOException - IO failure

    • startLocalOutput

      public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
      Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.
      Overrides:
      startLocalOutput in class FileSystem
      Parameters:
      fsOutputFile - path of output file
      tmpLocalFile - path of local tmp file
      Returns:
      the path.
      Throws:
      IOException - IO failure

    • completeLocalOutput

      public void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws IOException
      Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.
      Overrides:
      completeLocalOutput in class FileSystem
      Parameters:
      fsOutputFile - path of output file
      tmpLocalFile - path to local tmp file
      Throws:
      IOException - IO failure

    • getUsed

      public long getUsed() throws IOException
      Return the total size of all files in the filesystem.
      Overrides:
      getUsed in class FileSystem
      Returns:
      the number of path used.
      Throws:
      IOException - IO failure

    • getUsed

      public long getUsed(Path path) throws IOException
      Return the total size of all files from a specified path.
      Overrides:
      getUsed in class FileSystem
      Parameters:
      path - the path.
      Returns:
      the number of path content summary.
      Throws:
      IOException - IO failure

    • getDefaultBlockSize

      public long getDefaultBlockSize()
      Description copied from class: FileSystem
      Return the number of bytes that large input files should be optimally be split into to minimize I/O time.
      Overrides:
      getDefaultBlockSize in class FileSystem
      Returns:
      default block size.

    • getDefaultReplication

      public short getDefaultReplication()
      Description copied from class: FileSystem
      Get the default replication.
      Overrides:
      getDefaultReplication in class FileSystem
      Returns:
      the replication; the default value is "1".

    • getServerDefaults

      public FsServerDefaults getServerDefaults() throws IOException
      Description copied from class: FileSystem
      Return a set of server default configuration values.
      Overrides:
      getServerDefaults in class FileSystem
      Returns:
      server default configuration values
      Throws:
      IOException - IO failure

    • getDefaultBlockSize

      public long getDefaultBlockSize(Path f)
      Description copied from class: FileSystem
      Return the number of bytes that large input files should be optimally be split into to minimize I/O time. The given path will be used to locate the actual filesystem. The full path does not have to exist.
      Overrides:
      getDefaultBlockSize in class FileSystem
      Parameters:
      f - path of file
      Returns:
      the default block size for the path's filesystem

    • getDefaultReplication

      public short getDefaultReplication(Path f)
      Description copied from class: FileSystem
      Get the default replication for a path. The given path will be used to locate the actual FileSystem to query. The full path does not have to exist.
      Overrides:
      getDefaultReplication in class FileSystem
      Parameters:
      f - of the file
      Returns:
      default replication for the path's filesystem

    • getServerDefaults

      public FsServerDefaults getServerDefaults(Path f) throws IOException
      Description copied from class: FileSystem
      Return a set of server default configuration values.
      Overrides:
      getServerDefaults in class FileSystem
      Parameters:
      f - path is used to identify an FS since an FS could have another FS that it could be delegating the call to
      Returns:
      server default configuration values
      Throws:
      IOException - IO failure

    • getFileStatus

      public FileStatus getFileStatus(Path f) throws IOException
      Get file status.
      Specified by:
      getFileStatus in class FileSystem
      Parameters:
      f - The path we want information from
      Returns:
      a FileStatus object
      Throws:
      FileNotFoundException - when the path does not exist
      IOException - see specific implementation

    • msync

      public void msync() throws IOException, UnsupportedOperationException
      Description copied from class: FileSystem
      Synchronize client metadata state.

      In some FileSystem implementations such as HDFS metadata synchronization is essential to guarantee consistency of read requests particularly in HA setting.

      Overrides:
      msync in class FileSystem
      Throws:
      IOException - If an I/O error occurred.
      UnsupportedOperationException - if the operation is unsupported.

    • access

      public void access(Path path, FsAction mode) throws AccessControlException, FileNotFoundException, IOException
      Description copied from class: FileSystem
      Checks if the user can access a path. The mode specifies which access checks to perform. If the requested permissions are granted, then the method returns normally. If access is denied, then the method throws an AccessControlException.

      The default implementation calls FileSystem.getFileStatus(Path) and checks the returned permissions against the requested permissions. Note that the FileSystem.getFileStatus(Path) call will be subject to authorization checks. Typically, this requires search (execute) permissions on each directory in the path's prefix, but this is implementation-defined. Any file system that provides a richer authorization model (such as ACLs) may override the default implementation so that it checks against that model instead.

      In general, applications should avoid using this method, due to the risk of time-of-check/time-of-use race conditions. The permissions on a file may change immediately after the access call returns. Most applications should prefer running specific file system actions as the desired user represented by a UserGroupInformation.

      Parameters:
      path - Path to check
      mode - type of access to check
      Throws:
      AccessControlException - if access is denied
      FileNotFoundException - if the path does not exist
      IOException - see specific implementation

    • createSymlink

      public void createSymlink(Path target, Path link, boolean createParent) throws AccessControlException, FileAlreadyExistsException, FileNotFoundException, ParentNotDirectoryException, UnsupportedFileSystemException, IOException
      Description copied from class: FileSystem
      See FileContext.createSymlink(Path, Path, boolean).
      Overrides:
      createSymlink in class FileSystem
      Parameters:
      target - target path.
      link - link.
      createParent - create parent.
      Throws:
      AccessControlException - if access is denied.
      FileAlreadyExistsException - when the path does not exist.
      FileNotFoundException - when the path does not exist.
      ParentNotDirectoryException - if the parent path of dest is not a directory.
      UnsupportedFileSystemException - if there was no known implementation for the scheme.
      IOException - raised on errors performing I/O.

    • getFileLinkStatus

      public FileStatus getFileLinkStatus(Path f) throws AccessControlException, FileNotFoundException, UnsupportedFileSystemException, IOException
      Description copied from class: FileSystem
      See FileContext.getFileLinkStatus(Path).
      Overrides:
      getFileLinkStatus in class FileSystem
      Parameters:
      f - the path.
      Returns:
      file status
      Throws:
      AccessControlException - if access is denied.
      FileNotFoundException - when the path does not exist.
      UnsupportedFileSystemException - if there was no known implementation for the scheme.
      IOException - raised on errors performing I/O.

    • supportsSymlinks

      public boolean supportsSymlinks()
      Description copied from class: FileSystem
      See AbstractFileSystem.supportsSymlinks().
      Overrides:
      supportsSymlinks in class FileSystem
      Returns:
      if support symlinkls true, not false.

    • getLinkTarget

      public Path getLinkTarget(Path f) throws IOException
      Description copied from class: FileSystem
      See FileContext.getLinkTarget(Path).
      Overrides:
      getLinkTarget in class FileSystem
      Parameters:
      f - the path.
      Returns:
      the path.
      Throws:
      IOException - IO failure.

    • resolveLink

      protected Path resolveLink(Path f) throws IOException
      Description copied from class: FileSystem
      See AbstractFileSystem.getLinkTarget(Path).
      Overrides:
      resolveLink in class FileSystem
      Parameters:
      f - the path.
      Returns:
      the path.
      Throws:
      IOException - IO failure.

    • getFileChecksum

      public FileChecksum getFileChecksum(Path f) throws IOException
      Description copied from class: FileSystem
      Get the checksum of a file, if the FS supports checksums.
      Overrides:
      getFileChecksum in class FileSystem
      Parameters:
      f - The file path
      Returns:
      The file checksum. The default return value is null, which indicates that no checksum algorithm is implemented in the corresponding FileSystem.
      Throws:
      IOException - IO failure

    • getFileChecksum

      public FileChecksum getFileChecksum(Path f, long length) throws IOException
      Description copied from class: FileSystem
      Get the checksum of a file, from the beginning of the file till the specific length.
      Overrides:
      getFileChecksum in class FileSystem
      Parameters:
      f - The file path
      length - The length of the file range for checksum calculation
      Returns:
      The file checksum or null if checksums are not supported.
      Throws:
      IOException - IO failure

    • setVerifyChecksum

      public void setVerifyChecksum(boolean verifyChecksum)
      Description copied from class: FileSystem
      Set the verify checksum flag. This is only applicable if the corresponding filesystem supports checksums. By default doesn't do anything.
      Overrides:
      setVerifyChecksum in class FileSystem
      Parameters:
      verifyChecksum - Verify checksum flag

    • setWriteChecksum

      public void setWriteChecksum(boolean writeChecksum)
      Description copied from class: FileSystem
      Set the write checksum flag. This is only applicable if the corresponding filesystem supports checksums. By default doesn't do anything.
      Overrides:
      setWriteChecksum in class FileSystem
      Parameters:
      writeChecksum - Write checksum flag

    • getConf

      public Configuration getConf()
      Description copied from interface: Configurable
      Return the configuration used by this object.
      Specified by:
      getConf in interface Configurable
      Overrides:
      getConf in class Configured
      Returns:
      Configuration

    • close

      public void close() throws IOException
      Description copied from class: FileSystem
      Close this FileSystem instance. Will release any held locks, delete all files queued for deletion through calls to FileSystem.deleteOnExit(Path), and remove this FS instance from the cache, if cached. After this operation, the outcome of any method call on this FileSystem instance, or any input/output stream created by it is undefined.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Overrides:
      close in class FileSystem
      Throws:
      IOException - IO failure

    • setOwner

      public void setOwner(Path p, String username, String groupname) throws IOException
      Description copied from class: FileSystem
      Set owner of a path (i.e. a file or a directory). The parameters username and groupname cannot both be null.
      Overrides:
      setOwner in class FileSystem
      Parameters:
      p - The path
      username - If it is null, the original username remains unchanged.
      groupname - If it is null, the original groupname remains unchanged.
      Throws:
      IOException - IO failure

    • setTimes

      public void setTimes(Path p, long mtime, long atime) throws IOException
      Description copied from class: FileSystem
      Set access time of a file.
      Overrides:
      setTimes in class FileSystem
      Parameters:
      p - The path
      mtime - Set the modification time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set modification time.
      atime - Set the access time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set access time.
      Throws:
      IOException - IO failure

    • setPermission

      public void setPermission(Path p, FsPermission permission) throws IOException
      Description copied from class: FileSystem
      Set permission of a path.
      Overrides:
      setPermission in class FileSystem
      Parameters:
      p - The path
      permission - permission
      Throws:
      IOException - IO failure

    • primitiveCreate

      protected FSDataOutputStream primitiveCreate(Path f, FsPermission absolutePermission, EnumSet<CreateFlag> flag, int bufferSize, short replication, long blockSize, Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt) throws IOException
      Description copied from class: FileSystem
      This create has been added to support the FileContext that processes the permission with umask before calling this method. This a temporary method added to support the transition from FileSystem to FileContext for user applications.
      Overrides:
      primitiveCreate in class FileSystem
      Parameters:
      f - path.
      absolutePermission - permission.
      flag - create flag.
      bufferSize - buffer size.
      replication - replication.
      blockSize - block size.
      progress - progress.
      checksumOpt - check sum opt.
      Returns:
      output stream.
      Throws:
      IOException - IO failure

    • primitiveMkdir

      protected boolean primitiveMkdir(Path f, FsPermission abdolutePermission) throws IOException
      Description copied from class: FileSystem
      This version of the mkdirs method assumes that the permission is absolute. It has been added to support the FileContext that processes the permission with umask before calling this method. This a temporary method added to support the transition from FileSystem to FileContext for user applications.
      Overrides:
      primitiveMkdir in class FileSystem
      Parameters:
      f - path
      abdolutePermission - permissions
      Returns:
      true if the directory was actually created.
      Throws:
      IOException - IO failure
      See Also:

    • getChildFileSystems

      public FileSystem[] getChildFileSystems()
      Description copied from class: FileSystem
      Get all the immediate child FileSystems embedded in this FileSystem. It does not recurse and get grand children. If a FileSystem has multiple child FileSystems, then it must return a unique list of those FileSystems. Default is to return null to signify no children.
      Returns:
      FileSystems that are direct children of this FileSystem, or null for "no children"

    • createSnapshot

      public Path createSnapshot(Path path, String snapshotName) throws IOException
      Description copied from class: FileSystem
      Create a snapshot.
      Overrides:
      createSnapshot in class FileSystem
      Parameters:
      path - The directory where snapshots will be taken.
      snapshotName - The name of the snapshot
      Returns:
      the snapshot path.
      Throws:
      IOException - IO failure

    • renameSnapshot

      public void renameSnapshot(Path path, String snapshotOldName, String snapshotNewName) throws IOException
      Description copied from class: FileSystem
      Rename a snapshot.
      Overrides:
      renameSnapshot in class FileSystem
      Parameters:
      path - The directory path where the snapshot was taken
      snapshotOldName - Old name of the snapshot
      snapshotNewName - New name of the snapshot
      Throws:
      IOException - IO failure

    • deleteSnapshot

      public void deleteSnapshot(Path path, String snapshotName) throws IOException
      Description copied from class: FileSystem
      Delete a snapshot of a directory.
      Overrides:
      deleteSnapshot in class FileSystem
      Parameters:
      path - The directory that the to-be-deleted snapshot belongs to
      snapshotName - The name of the snapshot
      Throws:
      IOException - IO failure

    • modifyAclEntries

      public void modifyAclEntries(Path path, List<AclEntry> aclSpec) throws IOException
      Description copied from class: FileSystem
      Modifies ACL entries of files and directories. This method can add new ACL entries or modify the permissions on existing ACL entries. All existing ACL entries that are not specified in this call are retained without changes. (Modifications are merged into the current ACL.)
      Overrides:
      modifyAclEntries in class FileSystem
      Parameters:
      path - Path to modify
      aclSpec - List<AclEntry> describing modifications
      Throws:
      IOException - if an ACL could not be modified

    • removeAclEntries

      public void removeAclEntries(Path path, List<AclEntry> aclSpec) throws IOException
      Description copied from class: FileSystem
      Removes ACL entries from files and directories. Other ACL entries are retained.
      Overrides:
      removeAclEntries in class FileSystem
      Parameters:
      path - Path to modify
      aclSpec - List describing entries to remove
      Throws:
      IOException - if an ACL could not be modified

    • removeDefaultAcl

      public void removeDefaultAcl(Path path) throws IOException
      Description copied from class: FileSystem
      Removes all default ACL entries from files and directories.
      Overrides:
      removeDefaultAcl in class FileSystem
      Parameters:
      path - Path to modify
      Throws:
      IOException - if an ACL could not be modified

    • removeAcl

      public void removeAcl(Path path) throws IOException
      Description copied from class: FileSystem
      Removes all but the base ACL entries of files and directories. The entries for user, group, and others are retained for compatibility with permission bits.
      Overrides:
      removeAcl in class FileSystem
      Parameters:
      path - Path to modify
      Throws:
      IOException - if an ACL could not be removed

    • setAcl

      public void setAcl(Path path, List<AclEntry> aclSpec) throws IOException
      Description copied from class: FileSystem
      Fully replaces ACL of files and directories, discarding all existing entries.
      Overrides:
      setAcl in class FileSystem
      Parameters:
      path - Path to modify
      aclSpec - List describing modifications, which must include entries for user, group, and others for compatibility with permission bits.
      Throws:
      IOException - if an ACL could not be modified

    • getAclStatus

      public AclStatus getAclStatus(Path path) throws IOException
      Description copied from class: FileSystem
      Gets the ACL of a file or directory.
      Overrides:
      getAclStatus in class FileSystem
      Parameters:
      path - Path to get
      Returns:
      AclStatus describing the ACL of the file or directory
      Throws:
      IOException - if an ACL could not be read

    • setXAttr

      public void setXAttr(Path path, String name, byte[] value) throws IOException
      Description copied from class: FileSystem
      Set an xattr of a file or directory. The name must be prefixed with the namespace followed by ".". For example, "user.attr".

      Refer to the HDFS extended attributes user documentation for details.

      Overrides:
      setXAttr in class FileSystem
      Parameters:
      path - Path to modify
      name - xattr name.
      value - xattr value.
      Throws:
      IOException - IO failure

    • setXAttr

      public void setXAttr(Path path, String name, byte[] value, EnumSet<XAttrSetFlag> flag) throws IOException
      Description copied from class: FileSystem
      Set an xattr of a file or directory. The name must be prefixed with the namespace followed by ".". For example, "user.attr".

      Refer to the HDFS extended attributes user documentation for details.

      Overrides:
      setXAttr in class FileSystem
      Parameters:
      path - Path to modify
      name - xattr name.
      value - xattr value.
      flag - xattr set flag
      Throws:
      IOException - IO failure

    • getXAttr

      public byte[] getXAttr(Path path, String name) throws IOException
      Description copied from class: FileSystem
      Get an xattr name and value for a file or directory. The name must be prefixed with the namespace followed by ".". For example, "user.attr".

      Refer to the HDFS extended attributes user documentation for details.

      Overrides:
      getXAttr in class FileSystem
      Parameters:
      path - Path to get extended attribute
      name - xattr name.
      Returns:
      byte[] xattr value.
      Throws:
      IOException - IO failure

    • getXAttrs

      public Map<String,byte[]> getXAttrs(Path path) throws IOException
      Description copied from class: FileSystem
      Get all of the xattr name/value pairs for a file or directory. Only those xattrs which the logged-in user has permissions to view are returned.

      Refer to the HDFS extended attributes user documentation for details.

      Overrides:
      getXAttrs in class FileSystem
      Parameters:
      path - Path to get extended attributes
      Returns:
      Map describing the XAttrs of the file or directory
      Throws:
      IOException - IO failure

    • getXAttrs

      public Map<String,byte[]> getXAttrs(Path path, List<String> names) throws IOException
      Description copied from class: FileSystem
      Get all of the xattrs name/value pairs for a file or directory. Only those xattrs which the logged-in user has permissions to view are returned.

      Refer to the HDFS extended attributes user documentation for details.

      Overrides:
      getXAttrs in class FileSystem
      Parameters:
      path - Path to get extended attributes
      names - XAttr names.
      Returns:
      Map describing the XAttrs of the file or directory
      Throws:
      IOException - IO failure

    • listXAttrs

      public List<String> listXAttrs(Path path) throws IOException
      Description copied from class: FileSystem
      Get all of the xattr names for a file or directory. Only those xattr names which the logged-in user has permissions to view are returned.

      Refer to the HDFS extended attributes user documentation for details.

      Overrides:
      listXAttrs in class FileSystem
      Parameters:
      path - Path to get extended attributes
      Returns:
      List<String> of the XAttr names of the file or directory
      Throws:
      IOException - IO failure

    • removeXAttr

      public void removeXAttr(Path path, String name) throws IOException
      Description copied from class: FileSystem
      Remove an xattr of a file or directory. The name must be prefixed with the namespace followed by ".". For example, "user.attr".

      Refer to the HDFS extended attributes user documentation for details.

      Overrides:
      removeXAttr in class FileSystem
      Parameters:
      path - Path to remove extended attribute
      name - xattr name
      Throws:
      IOException - IO failure

    • satisfyStoragePolicy

      public void satisfyStoragePolicy(Path src) throws IOException
      Description copied from class: FileSystem
      Set the source path to satisfy storage policy.
      Overrides:
      satisfyStoragePolicy in class FileSystem
      Parameters:
      src - The source path referring to either a directory or a file.
      Throws:
      IOException - If an I/O error occurred.

    • setStoragePolicy

      public void setStoragePolicy(Path src, String policyName) throws IOException
      Description copied from class: FileSystem
      Set the storage policy for a given file or directory.
      Overrides:
      setStoragePolicy in class FileSystem
      Parameters:
      src - file or directory path.
      policyName - the name of the target storage policy. The list of supported Storage policies can be retrieved via FileSystem.getAllStoragePolicies().
      Throws:
      IOException - IO failure

    • unsetStoragePolicy

      public void unsetStoragePolicy(Path src) throws IOException
      Description copied from class: FileSystem
      Unset the storage policy set for a given file or directory.
      Overrides:
      unsetStoragePolicy in class FileSystem
      Parameters:
      src - file or directory path.
      Throws:
      IOException - IO failure

    • getStoragePolicy

      public BlockStoragePolicySpi getStoragePolicy(Path src) throws IOException
      Description copied from class: FileSystem
      Query the effective storage policy ID for the given file or directory.
      Overrides:
      getStoragePolicy in class FileSystem
      Parameters:
      src - file or directory path.
      Returns:
      storage policy for give file.
      Throws:
      IOException - IO failure

    • getAllStoragePolicies

      public Collection<? extends BlockStoragePolicySpi> getAllStoragePolicies() throws IOException
      Description copied from class: FileSystem
      Retrieve all the storage policies supported by this file system.
      Overrides:
      getAllStoragePolicies in class FileSystem
      Returns:
      all storage policies supported by this filesystem.
      Throws:
      IOException - IO failure

    • getTrashRoot

      public Path getTrashRoot(Path path)
      Description copied from class: FileSystem
      Get the root directory of Trash for current user when the path specified is deleted.
      Overrides:
      getTrashRoot in class FileSystem
      Parameters:
      path - the trash root of the path to be determined.
      Returns:
      the default implementation returns /user/$USER/.Trash

    • getTrashRoots

      public Collection<FileStatus> getTrashRoots(boolean allUsers)
      Description copied from class: FileSystem
      Get all the trash roots for current user or all users.
      Overrides:
      getTrashRoots in class FileSystem
      Parameters:
      allUsers - return trash roots for all users if true.
      Returns:
      all the trash root directories. Default FileSystem returns .Trash under users' home directories if /user/$USER/.Trash exists.

    • createFile

      public FSDataOutputStreamBuilder createFile(Path path)
      Description copied from class: FileSystem
      Create a new FSDataOutputStreamBuilder for the file with path. Files are overwritten by default.
      Overrides:
      createFile in class FileSystem
      Parameters:
      path - file path
      Returns:
      a FSDataOutputStreamBuilder object to build the file HADOOP-14384. Temporarily reduce the visibility of method before the builder interface becomes stable.

    • appendFile

      public FSDataOutputStreamBuilder appendFile(Path path)
      Description copied from class: FileSystem
      Create a Builder to append a file.
      Overrides:
      appendFile in class FileSystem
      Parameters:
      path - file path.
      Returns:
      a FSDataOutputStreamBuilder to build file append request.

    • openFile

      public FutureDataInputStreamBuilder openFile(Path path) throws IOException, UnsupportedOperationException
      Description copied from class: FileSystem
      Open a file for reading through a builder API. Ultimately calls FileSystem.open(Path, int) unless a subclass executes the open command differently. The semantics of this call are therefore the same as that of FileSystem.open(Path, int) with one special point: it is in FSDataInputStreamBuilder.build() in which the open operation takes place -it is there where all preconditions to the operation are checked.
      Overrides:
      openFile in class FileSystem
      Parameters:
      path - file path
      Returns:
      a FSDataInputStreamBuilder object to build the input stream
      Throws:
      IOException - if some early checks cause IO failures.
      UnsupportedOperationException - if support is checked early.

    • openFile

      public FutureDataInputStreamBuilder openFile(PathHandle pathHandle) throws IOException, UnsupportedOperationException
      Description copied from class: FileSystem
      Open a file for reading through a builder API. Ultimately calls FileSystem.open(PathHandle, int) unless a subclass executes the open command differently. If PathHandles are unsupported, this may fail in the FSDataInputStreamBuilder.build() command, rather than in this openFile() operation.
      Overrides:
      openFile in class FileSystem
      Parameters:
      pathHandle - path handle.
      Returns:
      a FSDataInputStreamBuilder object to build the input stream
      Throws:
      IOException - if some early checks cause IO failures.
      UnsupportedOperationException - if support is checked early.

    • openFileWithOptions

      protected CompletableFuture<FSDataInputStream> openFileWithOptions(Path path, org.apache.hadoop.fs.impl.OpenFileParameters parameters) throws IOException
      Description copied from class: FileSystem
      Execute the actual open file operation. This is invoked from FSDataInputStreamBuilder.build() and from DelegateToFileSystem and is where the action of opening the file should begin. The base implementation performs a blocking call to FileSystem.open(Path, int) in this call; the actual outcome is in the returned CompletableFuture. This avoids having to create some thread pool, while still setting up the expectation that the get() call is needed to evaluate the result.
      Overrides:
      openFileWithOptions in class FileSystem
      Parameters:
      path - path to the file
      parameters - open file parameters from the builder.
      Returns:
      a future which will evaluate to the opened file.
      Throws:
      IOException - failure to resolve the link.

    • openFileWithOptions

      protected CompletableFuture<FSDataInputStream> openFileWithOptions(PathHandle pathHandle, org.apache.hadoop.fs.impl.OpenFileParameters parameters) throws IOException
      Description copied from class: FileSystem
      Execute the actual open file operation. The base implementation performs a blocking call to FileSystem.open(Path, int) in this call; the actual outcome is in the returned CompletableFuture. This avoids having to create some thread pool, while still setting up the expectation that the get() call is needed to evaluate the result.
      Overrides:
      openFileWithOptions in class FileSystem
      Parameters:
      pathHandle - path to the file
      parameters - open file parameters from the builder.
      Returns:
      a future which will evaluate to the opened file.
      Throws:
      IOException - failure to resolve the link.

    • getEnclosingRoot

      public Path getEnclosingRoot(Path path) throws IOException
      Description copied from class: FileSystem
      Return path of the enclosing root for a given path. The enclosing root path is a common ancestor that should be used for temp and staging dirs as well as within encryption zones and other restricted directories. Call makeQualified on the param path to ensure its part of the correct filesystem.
      Overrides:
      getEnclosingRoot in class FileSystem
      Parameters:
      path - file path to find the enclosing root path for
      Returns:
      a path to the enclosing root
      Throws:
      IOException - early checks like failure to resolve path cause IO failures

    • hasPathCapability

      public boolean hasPathCapability(Path path, String capability) throws IOException
      Description copied from class: FileSystem
      The base FileSystem implementation generally has no knowledge of the capabilities of actual implementations. Unless it has a way to explicitly determine the capabilities, this method returns false. Probe for a specific capability under the given path. If the function returns true, this instance is explicitly declaring that the capability is available. If the function returns false, it can mean one of:
      • The capability is not known.
      • The capability is known but it is not supported.
      • The capability is known but the filesystem does not know if it is supported under the supplied path.
      The core guarantee which a caller can rely on is: if the predicate returns true, then the specific operation/behavior can be expected to be supported. However a specific call may be rejected for permission reasons, the actual file/directory not being present, or some other failure during the attempted execution of the operation.

      Implementors: PathCapabilitiesSupport can be used to help implement this method.

      Specified by:
      hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities
      Overrides:
      hasPathCapability in class FileSystem
      Parameters:
      path - path to query the capability of.
      capability - non-null, non-empty string to query the path for support.
      Returns:
      true if the capability is supported under that part of the FS.
      Throws:
      IOException - this should not be raised, except on problems resolving paths or relaying the call.