Package org.apache.hadoop.io.wrappedio.impl

Class DynamicWrappedIO

java.lang.Object
org.apache.hadoop.io.wrappedio.impl.DynamicWrappedIO
public final class DynamicWrappedIO extends Object
The wrapped IO methods in org.apache.hadoop.io.wrappedio.WrappedIO, dynamically loaded.

  • Field Details

    • PARQUET_READ_POLICIES

      public static final String PARQUET_READ_POLICIES
      Read policy for parquet files: "parquet, columnar, vector, random".
      See Also:

  • Constructor Details

    • DynamicWrappedIO

      public DynamicWrappedIO()

    • DynamicWrappedIO

      public DynamicWrappedIO(String classname)

  • Method Details

    • loaded

      public boolean loaded()
      Is the wrapped IO class loaded?
      Returns:
      true if the wrappedIO class was found and loaded.

    • bulkDelete_available

      public boolean bulkDelete_available()
      Are the bulk delete methods available?
      Returns:
      true if the methods were found.

    • bulkDelete_pageSize

      public int bulkDelete_pageSize(FileSystem fileSystem, Path path) throws IOException
      Get the maximum number of objects/files to delete in a single request.
      Parameters:
      fileSystem - filesystem
      path - path to delete under.
      Returns:
      a number greater than or equal to zero.
      Throws:
      UnsupportedOperationException - bulk delete under that path is not supported.
      IllegalArgumentException - path not valid.
      IOException - problems resolving paths
      RuntimeException - invocation failure.

    • bulkDelete_delete

      public List<Map.Entry<Path,String>> bulkDelete_delete(FileSystem fs, Path base, Collection<Path> paths) throws IOException
      Delete a list of files/objects.
      • Files must be under the path provided in base.
      • The size of the list must be equal to or less than the page size.
      • Directories are not supported; the outcome of attempting to delete directories is undefined (ignored; undetected, listed as failures...).
      • The operation is not atomic.
      • The operation is treated as idempotent: network failures may trigger resubmission of the request -any new objects created under a path in the list may then be deleted.
      • There is no guarantee that any parent directories exist after this call.
      Parameters:
      fs - filesystem
      base - path to delete under.
      paths - list of paths which must be absolute and under the base path.
      Returns:
      a list of all the paths which couldn't be deleted for a reason other than "not found" and any associated error message.
      Throws:
      UnsupportedOperationException - bulk delete under that path is not supported.
      IllegalArgumentException - if a path argument is invalid.
      IOException - IO problems including networking, authentication and more.

    • fileSystem_openFile_available

      public boolean fileSystem_openFile_available()
      Is the fileSystem_openFile(FileSystem, Path, String, FileStatus, Long, Map) method available.
      Returns:
      true if the optimized open file method can be invoked.

    • fileSystem_openFile

      public FSDataInputStream fileSystem_openFile(FileSystem fs, Path path, String policy, @Nullable FileStatus status, @Nullable Long length, @Nullable Map<String,String> options) throws IOException
      OpenFile assistant, easy reflection-based access to FileSystem#openFile(Path) and blocks awaiting the operation completion.
      Parameters:
      fs - filesystem
      path - path
      policy - read policy
      status - optional file status
      length - optional file length
      options - nullable map of other options
      Returns:
      stream of the opened file
      Throws:
      IOException - if the operation was attempted and failed.

    • pathCapabilities_hasPathCapability

      public boolean pathCapabilities_hasPathCapability(Object fs, Path path, String capability)
      Does a path have a given capability? Calls PathCapabilities#hasPathCapability(Path, String), mapping IOExceptions to false.
      Parameters:
      fs - filesystem
      path - path to query the capability of.
      capability - non-null, non-empty string to query the path for support.
      Returns:
      true if the capability is supported under that part of the FS false if the method is not loaded or the path lacks the capability.
      Throws:
      IllegalArgumentException - invalid arguments

    • streamCapabilities_hasCapability

      public boolean streamCapabilities_hasCapability(Object object, String capability)
      Does an object implement StreamCapabilities and, if so, what is the result of the probe for the capability? Calls StreamCapabilities#hasCapability(String),
      Parameters:
      object - object to probe
      capability - capability string
      Returns:
      true iff the object implements StreamCapabilities and the capability is declared available.

    • byteBufferPositionedReadable_available

      public boolean byteBufferPositionedReadable_available()
      Are the ByteBufferPositionedReadable methods loaded? This does not check that a specific stream implements the API; use byteBufferPositionedReadable_readFullyAvailable(InputStream).
      Returns:
      true if the hadoop libraries have the method.

    • byteBufferPositionedReadable_readFullyAvailable

      public boolean byteBufferPositionedReadable_readFullyAvailable(InputStream in) throws IOException
      Probe to see if the input stream is an instance of ByteBufferPositionedReadable. If the stream is an FSDataInputStream, the wrapped stream is checked.
      Parameters:
      in - input stream
      Returns:
      true if the API is available, the stream implements the interface (including the innermost wrapped stream) and that it declares the stream capability.
      Throws:
      IOException - if the operation was attempted and failed.

    • byteBufferPositionedReadable_readFully

      public void byteBufferPositionedReadable_readFully(InputStream in, long position, ByteBuffer buf) throws IOException
      Delegate to ByteBufferPositionedReadable#read(long, ByteBuffer).
      Parameters:
      in - input stream
      position - position within file
      buf - the ByteBuffer to receive the results of the read operation.
      Throws:
      UnsupportedOperationException - if the input doesn't implement the interface or, if when invoked, it is raised. Note: that is the default behaviour of FSDataInputStream#readFully(long, ByteBuffer).
      IOException - if the operation was attempted and failed.

    • instance

      public static DynamicWrappedIO instance()
      Get the singleton instance.
      Returns:
      the instance

    • isAvailable

      public static boolean isAvailable()
      Is the wrapped IO class loaded?
      Returns:
      true if the instance is loaded.

    • openFile

      public static FSDataInputStream openFile(FileSystem fs, FileStatus status, String readPolicies) throws IOException
      Open a file.

      If the WrappedIO class is found, use it.

      If not, falls back to the classic fs.open(Path) call.

      Parameters:
      fs - filesystem
      status - file status
      readPolicies - read policy to use
      Returns:
      the input stream
      Throws:
      IOException - any IO failure.