Package org.apache.hadoop.fs.http.client

Class HttpFSFileSystem

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.http.client.HttpFSFileSystem
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.BulkDeleteSource, org.apache.hadoop.fs.DelegationTokenRenewer.Renewable, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer
Direct Known Subclasses:
HttpsFSFileSystem
@Private public class HttpFSFileSystem extends org.apache.hadoop.fs.FileSystem implements org.apache.hadoop.fs.DelegationTokenRenewer.Renewable
HttpFSServer implementation of the FileSystemAccess FileSystem.

This implementation allows a user to access HDFS over HTTP via a HttpFSServer server.

  • Field Details

  • Constructor Details

    • HttpFSFileSystem

      public HttpFSFileSystem()

  • Method Details

    • initialize

      public void initialize(URI name, org.apache.hadoop.conf.Configuration conf) throws IOException
      Called after a new FileSystem instance is constructed.
      Overrides:
      initialize in class org.apache.hadoop.fs.FileSystem
      Parameters:
      name - a uri whose authority section names the host, port, etc. for this FileSystem
      conf - the configuration
      Throws:
      IOException

    • getScheme

      public String getScheme()
      Overrides:
      getScheme in class org.apache.hadoop.fs.FileSystem

    • getUri

      public URI getUri()
      Returns a URI whose scheme and authority identify this FileSystem.
      Specified by:
      getUri in class org.apache.hadoop.fs.FileSystem
      Returns:
      the URI whose scheme and authority identify this FileSystem.

    • getDefaultPort

      protected int getDefaultPort()
      Get the default port for this file system.
      Overrides:
      getDefaultPort in class org.apache.hadoop.fs.FileSystem
      Returns:
      the default port or 0 if there isn't one

    • open

      public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f, int bufferSize) throws IOException
      Opens an FSDataInputStream at the indicated Path.

      IMPORTANT: the returned FSDataInputStream does not support the PositionReadable and Seekable methods.

      Specified by:
      open in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - the file name to open
      bufferSize - the size of the buffer to be used.
      Throws:
      IOException

    • permissionToString

      public static String permissionToString(org.apache.hadoop.fs.permission.FsPermission p)
      Converts a FsPermission to a Unix octal representation.
      Parameters:
      p - the permission.
      Returns:
      the Unix string symbolic reprentation.

    • create

      public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) throws IOException
      Opens an FSDataOutputStream at the indicated Path with write-progress reporting.

      IMPORTANT: The Progressable parameter is not used.

      Specified by:
      create in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - the file name to open.
      permission - file permission.
      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
      bufferSize - the size of the buffer to be used.
      replication - required block replication for the file.
      blockSize - block size.
      progress - progressable.
      Throws:
      IOException
      See Also:

    • append

      public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress) throws IOException
      Append to an existing file (optional operation).

      IMPORTANT: The Progressable parameter is not used.

      Specified by:
      append in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - the existing file to be appended.
      bufferSize - the size of the buffer to be used.
      progress - for reporting progress if it is not null.
      Throws:
      IOException

    • truncate

      public boolean truncate(org.apache.hadoop.fs.Path f, long newLength) throws IOException
      Truncate a file.
      Overrides:
      truncate in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - the file to be truncated.
      newLength - The size the file is to be truncated to.
      Throws:
      IOException

    • concat

      public void concat(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.Path[] psrcs) throws IOException
      Concat existing files together.
      Overrides:
      concat in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - the path to the target destination.
      psrcs - the paths to the sources to use for the concatenation.
      Throws:
      IOException

    • rename

      public boolean rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws IOException
      Renames Path src to Path dst. Can take place on local fs or remote DFS.
      Specified by:
      rename in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • delete

      @Deprecated public boolean delete(org.apache.hadoop.fs.Path f) throws IOException
      Deprecated.
      Use delete(Path, boolean) instead
      Delete a file.
      Overrides:
      delete in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • delete

      public boolean delete(org.apache.hadoop.fs.Path f, boolean recursive) throws IOException
      Delete a file.
      Specified by:
      delete in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - the path to delete.
      recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.
      Returns:
      true if delete is successful else false.
      Throws:
      IOException

    • listStatus

      public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f) throws IOException
      Get FileStatus of files/directories in the given path. If path corresponds to a file then FileStatus of that file is returned. Else if path represents a directory then FileStatus of all files/directories inside given path is returned.
      Specified by:
      listStatus in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - given path
      Returns:
      the statuses of the files/directories in the given path
      Throws:
      IOException

    • listStatusBatch

      public org.apache.hadoop.fs.FileSystem.DirectoryEntries listStatusBatch(org.apache.hadoop.fs.Path f, byte[] token) throws FileNotFoundException, IOException
      Get FileSystem.DirectoryEntries of the given path. FileSystem.DirectoryEntries contains an array of FileStatus, as well as iteration information.
      Overrides:
      listStatusBatch in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - given path
      Returns:
      FileSystem.DirectoryEntries for given path
      Throws:
      FileNotFoundException
      IOException

    • setWorkingDirectory

      public void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)
      Set the current working directory for the given file system. All relative paths will be resolved relative to it.
      Specified by:
      setWorkingDirectory in class org.apache.hadoop.fs.FileSystem
      Parameters:
      newDir - new directory.

    • getWorkingDirectory

      public org.apache.hadoop.fs.Path getWorkingDirectory()
      Get the current working directory for the given file system
      Specified by:
      getWorkingDirectory in class org.apache.hadoop.fs.FileSystem
      Returns:
      the directory pathname

    • mkdirs

      public boolean mkdirs(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission) throws IOException
      Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.
      Specified by:
      mkdirs in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getFileStatus

      public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f) throws IOException
      Return a file status object that represents the path.
      Specified by:
      getFileStatus in class org.apache.hadoop.fs.FileSystem
      Parameters:
      f - The path we want information from
      Returns:
      a FileStatus object
      Throws:
      FileNotFoundException - when the path does not exist; IOException see specific implementation
      IOException

    • getHomeDirectory

      public org.apache.hadoop.fs.Path getHomeDirectory()
      Return the current user's home directory in this filesystem. The default implementation returns "/user/$USER/".
      Overrides:
      getHomeDirectory in class org.apache.hadoop.fs.FileSystem

    • getTrashRoot

      public org.apache.hadoop.fs.Path getTrashRoot(org.apache.hadoop.fs.Path fullPath)
      Get the root directory of Trash for a path in HDFS. 1. File in encryption zone returns /ez1/.Trash/username. 2. File not in encryption zone, or encountered exception when checking the encryption zone of the path, returns /users/username/.Trash. Caller appends either Current or checkpoint timestamp for trash destination. The default implementation returns "/user/username/.Trash".
      Overrides:
      getTrashRoot in class org.apache.hadoop.fs.FileSystem
      Parameters:
      fullPath - the trash root of the path to be determined.
      Returns:
      trash root

    • setOwner

      public void setOwner(org.apache.hadoop.fs.Path p, String username, String groupname) throws IOException
      Set owner of a path (i.e. a file or a directory). The parameters username and groupname cannot both be null.
      Overrides:
      setOwner in class org.apache.hadoop.fs.FileSystem
      Parameters:
      p - The path
      username - If it is null, the original username remains unchanged.
      groupname - If it is null, the original groupname remains unchanged.
      Throws:
      IOException

    • setPermission

      public void setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission) throws IOException
      Set permission of a path.
      Overrides:
      setPermission in class org.apache.hadoop.fs.FileSystem
      Parameters:
      p - path.
      permission - permission.
      Throws:
      IOException

    • setTimes

      public void setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime) throws IOException
      Set access time of a file
      Overrides:
      setTimes in class org.apache.hadoop.fs.FileSystem
      Parameters:
      p - The path
      mtime - Set the modification time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set modification time.
      atime - Set the access time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set access time.
      Throws:
      IOException

    • setReplication

      public boolean setReplication(org.apache.hadoop.fs.Path src, short replication) throws IOException
      Set replication for an existing file.
      Overrides:
      setReplication in class org.apache.hadoop.fs.FileSystem
      Parameters:
      src - file name
      replication - new replication
      Returns:
      true if successful; false if file does not exist or is a directory
      Throws:
      IOException

    • modifyAclEntries

      public void modifyAclEntries(org.apache.hadoop.fs.Path path, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec) throws IOException
      Modify the ACL entries for a file.
      Overrides:
      modifyAclEntries in class org.apache.hadoop.fs.FileSystem
      Parameters:
      path - Path to modify
      aclSpec - describing modifications
      Throws:
      IOException

    • removeAclEntries

      public void removeAclEntries(org.apache.hadoop.fs.Path path, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec) throws IOException
      Remove the specified ACL entries from a file
      Overrides:
      removeAclEntries in class org.apache.hadoop.fs.FileSystem
      Parameters:
      path - Path to modify
      aclSpec - describing entries to remove
      Throws:
      IOException

    • removeDefaultAcl

      public void removeDefaultAcl(org.apache.hadoop.fs.Path path) throws IOException
      Removes the default ACL for the given file
      Overrides:
      removeDefaultAcl in class org.apache.hadoop.fs.FileSystem
      Parameters:
      path - Path from which to remove the default ACL.
      Throws:
      IOException

    • removeAcl

      public void removeAcl(org.apache.hadoop.fs.Path path) throws IOException
      Remove all ACLs from a file
      Overrides:
      removeAcl in class org.apache.hadoop.fs.FileSystem
      Parameters:
      path - Path from which to remove all ACLs
      Throws:
      IOException

    • setAcl

      public void setAcl(org.apache.hadoop.fs.Path path, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec) throws IOException
      Set the ACLs for the given file
      Overrides:
      setAcl in class org.apache.hadoop.fs.FileSystem
      Parameters:
      path - Path to modify
      aclSpec - describing modifications, must include entries for user, group, and others for compatibility with permission bits.
      Throws:
      IOException

    • getAclStatus

      public org.apache.hadoop.fs.permission.AclStatus getAclStatus(org.apache.hadoop.fs.Path path) throws IOException
      Get the ACL information for a given file
      Overrides:
      getAclStatus in class org.apache.hadoop.fs.FileSystem
      Parameters:
      path - Path to acquire ACL info for
      Returns:
      the ACL information in JSON format
      Throws:
      IOException

    • getContentSummary

      public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      getContentSummary in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getQuotaUsage

      public org.apache.hadoop.fs.QuotaUsage getQuotaUsage(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      getQuotaUsage in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getFileChecksum

      public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      getFileChecksum in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getDelegationToken

      public org.apache.hadoop.security.token.Token<?> getDelegationToken(String renewer) throws IOException
      Specified by:
      getDelegationToken in interface org.apache.hadoop.security.token.DelegationTokenIssuer
      Overrides:
      getDelegationToken in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • renewDelegationToken

      public long renewDelegationToken(org.apache.hadoop.security.token.Token<?> token) throws IOException
      Throws:
      IOException

    • cancelDelegationToken

      public void cancelDelegationToken(org.apache.hadoop.security.token.Token<?> token) throws IOException
      Throws:
      IOException

    • getRenewToken

      public org.apache.hadoop.security.token.Token<?> getRenewToken()
      Specified by:
      getRenewToken in interface org.apache.hadoop.fs.DelegationTokenRenewer.Renewable

    • setDelegationToken

      public <T extends org.apache.hadoop.security.token.TokenIdentifier> void setDelegationToken(org.apache.hadoop.security.token.Token<T> token)
      Specified by:
      setDelegationToken in interface org.apache.hadoop.fs.DelegationTokenRenewer.Renewable

    • setXAttr

      public void setXAttr(org.apache.hadoop.fs.Path f, String name, byte[] value, EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flag) throws IOException
      Overrides:
      setXAttr in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getXAttr

      public byte[] getXAttr(org.apache.hadoop.fs.Path f, String name) throws IOException
      Overrides:
      getXAttr in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getXAttrs

      public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      getXAttrs in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getXAttrs

      public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path f, List<String> names) throws IOException
      Overrides:
      getXAttrs in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • listXAttrs

      public List<String> listXAttrs(org.apache.hadoop.fs.Path f) throws IOException
      Overrides:
      listXAttrs in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • removeXAttr

      public void removeXAttr(org.apache.hadoop.fs.Path f, String name) throws IOException
      Overrides:
      removeXAttr in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getAllStoragePolicies

      public Collection<org.apache.hadoop.hdfs.protocol.BlockStoragePolicy> getAllStoragePolicies() throws IOException
      Overrides:
      getAllStoragePolicies in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getStoragePolicy

      public org.apache.hadoop.hdfs.protocol.BlockStoragePolicy getStoragePolicy(org.apache.hadoop.fs.Path src) throws IOException
      Overrides:
      getStoragePolicy in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • setStoragePolicy

      public void setStoragePolicy(org.apache.hadoop.fs.Path src, String policyName) throws IOException
      Overrides:
      setStoragePolicy in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • unsetStoragePolicy

      public void unsetStoragePolicy(org.apache.hadoop.fs.Path src) throws IOException
      Overrides:
      unsetStoragePolicy in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • allowSnapshot

      public void allowSnapshot(org.apache.hadoop.fs.Path path) throws IOException
      Throws:
      IOException

    • disallowSnapshot

      public void disallowSnapshot(org.apache.hadoop.fs.Path path) throws IOException
      Throws:
      IOException

    • createSnapshot

      public final org.apache.hadoop.fs.Path createSnapshot(org.apache.hadoop.fs.Path path, String snapshotName) throws IOException
      Overrides:
      createSnapshot in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • renameSnapshot

      public void renameSnapshot(org.apache.hadoop.fs.Path path, String snapshotOldName, String snapshotNewName) throws IOException
      Overrides:
      renameSnapshot in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • deleteSnapshot

      public void deleteSnapshot(org.apache.hadoop.fs.Path path, String snapshotName) throws IOException
      Overrides:
      deleteSnapshot in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getSnapshotDiffReport

      public org.apache.hadoop.hdfs.protocol.SnapshotDiffReport getSnapshotDiffReport(org.apache.hadoop.fs.Path path, String snapshotOldName, String snapshotNewName) throws IOException
      Throws:
      IOException

    • getSnapshotDiffReportListing

      public org.apache.hadoop.hdfs.protocol.SnapshotDiffReportListing getSnapshotDiffReportListing(org.apache.hadoop.fs.Path path, String snapshotOldName, String snapshotNewName, byte[] snapshotDiffStartPath, Integer snapshotDiffIndex) throws IOException
      Throws:
      IOException

    • getSnapshottableDirectoryList

      public org.apache.hadoop.hdfs.protocol.SnapshottableDirectoryStatus[] getSnapshottableDirectoryList() throws IOException
      Throws:
      IOException

    • getSnapshotListing

      public org.apache.hadoop.hdfs.protocol.SnapshotStatus[] getSnapshotListing(org.apache.hadoop.fs.Path snapshotRoot) throws IOException
      Throws:
      IOException

    • hasPathCapability

      public boolean hasPathCapability(org.apache.hadoop.fs.Path path, String capability) throws IOException
      This filesystem's capabilities must be in sync with that of DistributedFileSystem.hasPathCapability() except where the feature is not exposed (e.g. symlinks).
      Specified by:
      hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities
      Overrides:
      hasPathCapability in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getServerDefaults

      public org.apache.hadoop.fs.FsServerDefaults getServerDefaults() throws IOException
      Overrides:
      getServerDefaults in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getServerDefaults

      public org.apache.hadoop.fs.FsServerDefaults getServerDefaults(org.apache.hadoop.fs.Path p) throws IOException
      Overrides:
      getServerDefaults in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • access

      public void access(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsAction mode) throws IOException
      Overrides:
      access in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • setErasureCodingPolicy

      public void setErasureCodingPolicy(org.apache.hadoop.fs.Path path, String policyName) throws IOException
      Throws:
      IOException

    • getErasureCodingPolicy

      public org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy getErasureCodingPolicy(org.apache.hadoop.fs.Path path) throws IOException
      Throws:
      IOException

    • unsetErasureCodingPolicy

      public void unsetErasureCodingPolicy(org.apache.hadoop.fs.Path path) throws IOException
      Throws:
      IOException

    • satisfyStoragePolicy

      public void satisfyStoragePolicy(org.apache.hadoop.fs.Path path) throws IOException
      Overrides:
      satisfyStoragePolicy in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getFileBlockLocations

      public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.Path path, long start, long len) throws IOException
      Overrides:
      getFileBlockLocations in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getFileBlockLocations

      public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.FileStatus status, long offset, long length) throws IOException
      Overrides:
      getFileBlockLocations in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getFileLinkStatus

      public org.apache.hadoop.fs.FileStatus getFileLinkStatus(org.apache.hadoop.fs.Path path) throws IOException
      Overrides:
      getFileLinkStatus in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getStatus

      public org.apache.hadoop.fs.FsStatus getStatus(org.apache.hadoop.fs.Path path) throws IOException
      Overrides:
      getStatus in class org.apache.hadoop.fs.FileSystem
      Throws:
      IOException

    • getAllErasureCodingPolicies

      public Collection<org.apache.hadoop.hdfs.protocol.ErasureCodingPolicyInfo> getAllErasureCodingPolicies() throws IOException
      Throws:
      IOException

    • getAllErasureCodingCodecs

      public Map<String,String> getAllErasureCodingCodecs() throws IOException
      Throws:
      IOException

    • getTrashRoots

      public Collection<org.apache.hadoop.fs.FileStatus> getTrashRoots(boolean allUsers)
      Overrides:
      getTrashRoots in class org.apache.hadoop.fs.FileSystem