Package org.apache.hadoop.fs.viewfs

Class ViewFileSystemOverloadScheme

java.lang.Object
org.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.viewfs.ViewFileSystem
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme
All Implemented Interfaces:
Closeable, AutoCloseable, Configurable, BulkDeleteSource, PathCapabilities, DelegationTokenIssuer
@LimitedPrivate({"MapReduce","HBase","Hive"}) @Evolving public class ViewFileSystemOverloadScheme extends ViewFileSystem

This class is extended from the ViewFileSystem for the overloaded scheme file system. Mount link configurations and in-memory mount table building behaviors are inherited from ViewFileSystem. Unlike ViewFileSystem scheme (viewfs://), the users would be able to use any scheme.

To use this class, the following configurations need to be added in core-site.xml file.
1) fs.<scheme>.impl = org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme
2) fs.viewfs.overload.scheme.target.<scheme>.impl = <hadoop compatible file system implementation class name for the <scheme>>

Here <scheme> can be any scheme, but with that scheme there should be a hadoop compatible file system available. Second configuration value should be the respective scheme's file system implementation class. Example: if scheme is configured with "hdfs", then the 2nd configuration class name will be org.apache.hadoop.hdfs.DistributedFileSystem. if scheme is configured with "s3a", then the 2nd configuration class name will be org.apache.hadoop.fs.s3a.S3AFileSystem.

Use Case 1:
===========
If users want some of their existing cluster (hdfs://Cluster) data to mount with other hdfs and object store clusters(hdfs://NN1, o3fs://bucket1.volume1/, s3a://bucket1/)

fs.viewfs.mounttable.Cluster.link./user = hdfs://NN1/user
fs.viewfs.mounttable.Cluster.link./data = o3fs://bucket1.volume1/data
fs.viewfs.mounttable.Cluster.link./backup = s3a://bucket1/backup/

Op1: Create file hdfs://Cluster/user/fileA will go to hdfs://NN1/user/fileA
Op2: Create file hdfs://Cluster/data/datafile will go to o3fs://bucket1.volume1/data/datafile
Op3: Create file hdfs://Cluster/backup/data.zip will go to s3a://bucket1/backup/data.zip

Use Case 2:
===========
If users want some of their existing cluster (s3a://bucketA/) data to mount with other hdfs and object store clusters (hdfs://NN1, o3fs://bucket1.volume1/)

fs.viewfs.mounttable.bucketA.link./user = hdfs://NN1/user
fs.viewfs.mounttable.bucketA.link./data = o3fs://bucket1.volume1/data
fs.viewfs.mounttable.bucketA.link./salesDB = s3a://bucketA/salesDB/

Op1: Create file s3a://bucketA/user/fileA will go to hdfs://NN1/user/fileA
Op2: Create file s3a://bucketA/data/datafile will go to o3fs://bucket1.volume1/data/datafile
Op3: Create file s3a://bucketA/salesDB/dbfile will go to s3a://bucketA/salesDB/dbfile

Note:
(1) In ViewFileSystemOverloadScheme, by default the mount links will be represented as non-symlinks. If you want to change this behavior, please see ViewFileSystem.listStatus(Path)
(2) In ViewFileSystemOverloadScheme, only the initialized uri's hostname will be considered as the mount table name. When the passed uri has hostname:port, it will simply ignore the port number and only hostname will be considered as the mount table name.
(3) If there are no mount links configured with the initializing uri's hostname as the mount table name, then it will automatically consider the current uri as fallback( ex: fs.viewfs.mounttable.<mycluster>.linkFallback) target fs uri.

  • Constructor Details

  • Method Details

    • getScheme

      public String getScheme()
      Description copied from class: ViewFileSystem
      Return the protocol scheme for the FileSystem.
      Overrides:
      getScheme in class ViewFileSystem
      Returns:
      viewfs

    • supportAutoAddingFallbackOnNoMounts

      public boolean supportAutoAddingFallbackOnNoMounts()
      By default returns false as ViewFileSystemOverloadScheme supports auto adding fallback on no mounts.

    • setSupportAutoAddingFallbackOnNoMounts

      public void setSupportAutoAddingFallbackOnNoMounts(boolean addAutoFallbackOnNoMounts)
      Sets whether to add fallback automatically when no mount points found.
      Parameters:
      addAutoFallbackOnNoMounts - addAutoFallbackOnNoMounts.

    • initialize

      public void initialize(URI theUri, Configuration conf) throws IOException
      Description copied from class: ViewFileSystem
      Called after a new FileSystem instance is constructed.
      Overrides:
      initialize in class ViewFileSystem
      Parameters:
      theUri - a uri whose authority section names the host, port, etc. for this FileSystem
      conf - the configuration
      Throws:
      IOException - on any failure to initialize this instance.

    • fsGetter

      protected FsGetter fsGetter()
      This method is overridden because in ViewFileSystemOverloadScheme if overloaded scheme matches with mounted target fs scheme, file system should be created without going into fs.<scheme>.impl based resolution. Otherwise it will end up in an infinite loop as the target will be resolved again to ViewFileSystemOverloadScheme as fs.<scheme>.impl points to ViewFileSystemOverloadScheme. So, below method will initialize the fs.viewfs.overload.scheme.target.<scheme>.impl. Other schemes can follow fs.newInstance
      Overrides:
      fsGetter in class ViewFileSystem
      Returns:
      fs getter.

    • getRawFileSystem

      public FileSystem getRawFileSystem(Path path, Configuration conf) throws IOException
      This is an admin only API to give access to its child raw file system, if the path is link. If the given path is an internal directory(path is from mount paths tree), it will initialize the file system of given path uri directly. If path cannot be resolved to any internal directory or link, it will throw NotInMountpointException. Please note, this API will not return chrooted file system. Instead, this API will get actual raw file system instances.
      Parameters:
      path - - fs uri path
      conf - - configuration
      Returns:
      file system.
      Throws:
      IOException - raised on errors performing I/O.

    • getMountPathInfo

      public ViewFileSystemOverloadScheme.MountPathInfo<FileSystem> getMountPathInfo(Path path, Configuration conf) throws IOException
      Gets the mount path info, which contains the target file system and remaining path to pass to the target file system.
      Parameters:
      path - the path.
      conf - configuration.
      Returns:
      mount path info.
      Throws:
      IOException - raised on errors performing I/O.

    • getFallbackFileSystem

      public FileSystem getFallbackFileSystem()
      Returns:
      Gets the fallback file system configured. Usually, this will be the default cluster.

    • canonicalizeUri

      @LimitedPrivate("HDFS") public URI canonicalizeUri(URI uri)
      Description copied from class: FileSystem
      Canonicalize the given URI. This is implementation-dependent, and may for example consist of canonicalizing the hostname using DNS and adding the default port if not specified. The default implementation simply fills in the default port if not specified and if FileSystem.getDefaultPort() returns a default port.
      Overrides:
      canonicalizeUri in class FileSystem
      Parameters:
      uri - url.
      Returns:
      URI
      See Also: