Package org.apache.hadoop.fs.impl.prefetch

Class SingleFilePerBlockCache

java.lang.Object
org.apache.hadoop.fs.impl.prefetch.SingleFilePerBlockCache
All Implemented Interfaces:
Closeable, AutoCloseable, BlockCache
public class SingleFilePerBlockCache extends Object implements BlockCache
Provides functionality necessary for caching blocks of data read from FileSystem. Each cache block is stored on the local disk as a separate file.

  • Constructor Details

    • SingleFilePerBlockCache

      public SingleFilePerBlockCache(PrefetchingStatistics prefetchingStatistics, int maxBlocksCount, DurationTrackerFactory trackerFactory)
      Constructs an instance of a SingleFilePerBlockCache.
      Parameters:
      prefetchingStatistics - statistics for this stream.
      maxBlocksCount - max blocks count to be kept in cache at any time.
      trackerFactory - tracker with statistics to update

  • Method Details

    • containsBlock

      public boolean containsBlock(int blockNumber)
      Indicates whether the given block is in this cache.
      Specified by:
      containsBlock in interface BlockCache
      Parameters:
      blockNumber - the id of the given block.
      Returns:
      true if the given block is in this cache, false otherwise.

    • blocks

      public Iterable<Integer> blocks()
      Gets the blocks in this cache.
      Specified by:
      blocks in interface BlockCache
      Returns:
      the blocks in this cache.

    • size

      public int size()
      Gets the number of blocks in this cache.
      Specified by:
      size in interface BlockCache
      Returns:
      the number of blocks in this cache.

    • get

      public void get(int blockNumber, ByteBuffer buffer) throws IOException
      Gets the block having the given blockNumber.
      Specified by:
      get in interface BlockCache
      Parameters:
      blockNumber - the id of the desired block.
      buffer - contents of the desired block are copied to this buffer.
      Throws:
      IllegalArgumentException - if buffer is null.
      IOException - if there is an error reading the given block.

    • readFile

      protected int readFile(Path path, ByteBuffer buffer) throws IOException
      Throws:
      IOException

    • put

      public void put(int blockNumber, ByteBuffer buffer, Configuration conf, LocalDirAllocator localDirAllocator) throws IOException
      Puts the given block in this cache.
      Specified by:
      put in interface BlockCache
      Parameters:
      blockNumber - the block number, used as a key for blocks map.
      buffer - buffer contents of the given block to be added to this cache.
      conf - the configuration.
      localDirAllocator - the local dir allocator instance.
      Throws:
      IOException - if either local dir allocator fails to allocate file or if IO error occurs while writing the buffer content to the file.
      IllegalArgumentException - if buffer is null, or if buffer.limit() is zero or negative.

    • writeFile

      protected void writeFile(Path path, ByteBuffer buffer) throws IOException
      Throws:
      IOException

    • getCacheFilePath

      protected Path getCacheFilePath(Configuration conf, LocalDirAllocator localDirAllocator) throws IOException
      Return temporary file created based on the file path retrieved from local dir allocator.
      Parameters:
      conf - The configuration object.
      localDirAllocator - Local dir allocator instance.
      Returns:
      Path of the temporary file created.
      Throws:
      IOException - if IO error occurs while local dir allocator tries to retrieve path from local FS or file creation fails or permission set fails.

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • isCacheSpaceAvailable

      public static boolean isCacheSpaceAvailable(long fileSize, Configuration conf, LocalDirAllocator localDirAllocator)
      Determine if the cache space is available on the local FS.
      Parameters:
      fileSize - The size of the file.
      conf - The configuration.
      localDirAllocator - Local dir allocator instance.
      Returns:
      True if the given file size is less than the available free space on local FS, False otherwise.