Package org.apache.hadoop.hdfs.server.namenode

Class FSTreeTraverser

java.lang.Object
org.apache.hadoop.hdfs.server.namenode.FSTreeTraverser
@Private public abstract class FSTreeTraverser extends Object
FSTreeTraverser traverse directory recursively and process files in batches.

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

  • Constructor Details

    • FSTreeTraverser

      public FSTreeTraverser(FSDirectory dir, org.apache.hadoop.conf.Configuration conf)

  • Method Details

    • getFSDirectory

      public FSDirectory getFSDirectory()

    • traverseDir

      protected void traverseDir(INodeDirectory parent, long startId, byte[] startAfter, FSTreeTraverser.TraverseInfo traverseInfo) throws IOException, InterruptedException
      Iterate through all files directly inside parent, and recurse down directories. The listing is done in batch, and can optionally start after a position. The iteration of the inode tree is done in a depth-first fashion. But instead of holding all INodeDirectory's in memory on the fly, only the path components to the current inode is held. This is to reduce memory consumption.
      Parameters:
      parent - The inode id of parent directory
      startId - Id of the start inode.
      startAfter - Full path of a file the traverse should start after.
      traverseInfo - info which may required for processing the child's.
      Throws:
      IOException
      InterruptedException

    • traverseDirInt

      protected INode traverseDirInt(long startId, INode curr, List<byte[]> startAfters, FSTreeTraverser.TraverseInfo traverseInfo) throws IOException, InterruptedException
      Iterates the parent directory, and add direct children files to current batch. If batch size meets configured threshold, current batch will be submitted for the processing.

      Locks could be released and reacquired when a batch submission is finished.

      Parameters:
      startId - Id of the start inode.
      Returns:
      The inode which was just processed, if lock is held in the entire process. Null if lock is released.
      Throws:
      IOException
      InterruptedException

    • readLock

      protected void readLock()

    • readUnlock

      protected void readUnlock()

    • checkPauseForTesting

      protected abstract void checkPauseForTesting() throws InterruptedException
      Throws:
      InterruptedException

    • processFileInode

      protected abstract boolean processFileInode(INode inode, FSTreeTraverser.TraverseInfo traverseInfo) throws IOException, InterruptedException
      Process an Inode. Add to current batch if it's a file, no-op otherwise.
      Parameters:
      inode - the inode
      Returns:
      true if inode is added to currentBatch and should be process for next operation. false otherwise: could be inode is not a file.
      Throws:
      IOException
      InterruptedException

    • shouldSubmitCurrentBatch

      protected abstract boolean shouldSubmitCurrentBatch()
      Check whether current batch can be submitted for the processing.
      Returns:
      true if batch size meets the condition, otherwise false.

    • checkINodeReady

      protected abstract void checkINodeReady(long startId) throws IOException
      Check whether inode is ready for traverse. Throws IOE if it's not.
      Parameters:
      startId - Id of the start inode.
      Throws:
      IOException

    • submitCurrentBatch

      protected abstract void submitCurrentBatch(Long startId) throws IOException, InterruptedException
      Submit the current batch for processing.
      Parameters:
      startId - Id of the start inode.
      Throws:
      IOException
      InterruptedException

    • throttle

      protected abstract void throttle() throws InterruptedException
      Throttles the FSTreeTraverser.
      Throws:
      InterruptedException

    • canTraverseDir

      protected abstract boolean canTraverseDir(INode inode) throws IOException
      Check whether dir is traversable or not.
      Parameters:
      inode - Dir inode
      Returns:
      true if dir is traversable otherwise false.
      Throws:
      IOException