Package org.apache.hadoop.hdfs.server.namenode.sps

Class StoragePolicySatisfier

java.lang.Object
org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier
All Implemented Interfaces:
Runnable, SPSService
@Private public class StoragePolicySatisfier extends Object implements SPSService, Runnable
Setting storagePolicy on a file after the file write will only update the new storage policy type in Namespace, but physical block storage movement will not happen until user runs "Mover Tool" explicitly for such files. The StoragePolicySatisfier Daemon thread implemented for addressing the case where users may want to physically move the blocks by a dedicated daemon (can run inside Namenode or stand alone) instead of running mover tool explicitly. Just calling client API to satisfyStoragePolicy on a file/dir will automatically trigger to move its physical storage locations as expected in asynchronous manner. Here SPS will pick the file blocks which are expecting to change its storages, then it will build the mapping of source block location and expected storage type and location to move. After that this class will also prepare requests to send to Datanode for processing the physical block movements.

  • Field Details

    • LOG

      public static final org.slf4j.Logger LOG

  • Constructor Details

    • StoragePolicySatisfier

      public StoragePolicySatisfier(org.apache.hadoop.conf.Configuration conf)

  • Method Details

    • init

      public void init(Context context)
      Description copied from interface: SPSService
      Initializes the helper services.
      Specified by:
      init in interface SPSService
      Parameters:
      context - - context is an helper service to provide communication channel between NN and SPS

    • start

      public void start(org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode serviceMode)
      Start storage policy satisfier demon thread. Also start block storage movements monitor for retry the attempts if needed.
      Specified by:
      start in interface SPSService
      Parameters:
      serviceMode - sps service mode

    • stop

      public void stop(boolean forceStop)
      Description copied from interface: SPSService
      Stops the SPS service.
      Specified by:
      stop in interface SPSService
      Parameters:
      forceStop - true represents to clear all the sps path's hint, false otherwise.

    • stopGracefully

      public void stopGracefully()
      Description copied from interface: SPSService
      Stops the SPS service gracefully. Timed wait to stop storage policy satisfier daemon threads.
      Specified by:
      stopGracefully in interface SPSService

    • isRunning

      public boolean isRunning()
      Description copied from interface: SPSService
      Check whether StoragePolicySatisfier is running.
      Specified by:
      isRunning in interface SPSService
      Returns:
      true if running

    • run

      public void run()
      Specified by:
      run in interface Runnable

    • notifyStorageMovementAttemptFinishedBlk

      public void notifyStorageMovementAttemptFinishedBlk(org.apache.hadoop.hdfs.protocol.DatanodeInfo dnInfo, org.apache.hadoop.fs.StorageType storageType, org.apache.hadoop.hdfs.protocol.Block block)
      Receives storage movement attempt finished block report.
      Specified by:
      notifyStorageMovementAttemptFinishedBlk in interface SPSService
      Parameters:
      dnInfo - reported datanode
      storageType - - storage type
      block - movement attempt finished block.

    • getAttemptedItemsMonitor

      @VisibleForTesting public BlockStorageMovementAttemptedItems getAttemptedItemsMonitor()

    • clearQueues

      public void clearQueues()
      Clear the queues from to be storage movement needed lists and items tracked in storage movement monitor.

    • addFileToProcess

      public void addFileToProcess(ItemInfo trackInfo, boolean scanCompleted)
      Description copied from interface: SPSService
      Adds the Item information(file etc) to processing queue.
      Specified by:
      addFileToProcess in interface SPSService
      Parameters:
      trackInfo - file info object for which need to satisfy the policy

    • addAllFilesToProcess

      public void addAllFilesToProcess(long startPath, List<ItemInfo> itemInfoList, boolean scanCompleted)
      Description copied from interface: SPSService
      Adds all the Item information(file etc) to processing queue.
      Specified by:
      addAllFilesToProcess in interface SPSService
      Parameters:
      startPath - - directoryId/fileId, on which SPS was called.
      itemInfoList - - list of item infos
      scanCompleted - - whether the scanning of directory fully done with itemInfoList

    • processingQueueSize

      public int processingQueueSize()
      Specified by:
      processingQueueSize in interface SPSService
      Returns:
      current processing queue size.

    • getConf

      public org.apache.hadoop.conf.Configuration getConf()
      Specified by:
      getConf in interface SPSService
      Returns:
      the configuration.

    • getStorageMovementQueue

      @VisibleForTesting public BlockStorageMovementNeeded getStorageMovementQueue()

    • markScanCompletedForPath

      public void markScanCompletedForPath(long inodeId)
      Description copied from interface: SPSService
      Marks the scanning of directory if finished.
      Specified by:
      markScanCompletedForPath in interface SPSService
      Parameters:
      inodeId - - satisfier path id

    • join

      public void join() throws InterruptedException
      Join main SPS thread.
      Throws:
      InterruptedException