Package org.apache.hadoop.hdfs.server.datanode.fsdataset

Interface FsDatasetSpi<V extends FsVolumeSpi>

All Superinterfaces:
FSDatasetMBean, org.apache.hadoop.metrics2.MetricsSource
@Private public interface FsDatasetSpi<V extends FsVolumeSpi> extends FSDatasetMBean
This is a service provider interface for the underlying storage that stores replicas for a data node. The default implementation stores replicas on local drives.

  • Method Details

    • getFsVolumeReferences

      FsDatasetSpi.FsVolumeReferences getFsVolumeReferences()
      Returns a list of FsVolumes that hold reference counts. The caller must release the reference of each volume by calling FsDatasetSpi.FsVolumeReferences.close().

    • addVolume

      void addVolume(StorageLocation location, List<NamespaceInfo> nsInfos) throws IOException
      Add a new volume to the FsDataset. If the FSDataset supports block scanning, this function registers the new volume with the block scanner.
      Parameters:
      location - The storage location for the new volume.
      nsInfos - Namespace information for the new volume.
      Throws:
      IOException

    • removeVolumes

      void removeVolumes(Collection<StorageLocation> volumes, boolean clearFailure)
      Removes a collection of volumes from FsDataset. If the FSDataset supports block scanning, this function removes the volumes from the block scanner.
      Parameters:
      volumes - The paths of the volumes to be removed.
      clearFailure - set true to clear the failure information about the volumes.

    • getStorage

      org.apache.hadoop.hdfs.server.protocol.DatanodeStorage getStorage(String storageUuid)
      Returns:
      a storage with the given storage ID

    • getStorageReports

      org.apache.hadoop.hdfs.server.protocol.StorageReport[] getStorageReports(String bpid) throws IOException
      Returns:
      one or more storage reports for attached volumes.
      Throws:
      IOException

    • getVolume

      V getVolume(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
      Returns:
      the volume that contains a replica of the block.

    • getVolumeInfoMap

      Map<String,Object> getVolumeInfoMap()
      Returns:
      a volume information map (name => info).

    • getVolumeFailureSummary

      VolumeFailureSummary getVolumeFailureSummary()
      Returns info about volume failures.
      Returns:
      info about volume failures, possibly null

    • getFinalizedBlocks

      List<ReplicaInfo> getFinalizedBlocks(String bpid)
      Gets a list of references to the finalized blocks for the given block pool.

      Callers of this function should call acquireDatasetLockManager() to avoid blocks' status being changed during list iteration.

      Returns:
      a list of references to the finalized blocks for the given block pool.

    • checkAndUpdate

      void checkAndUpdate(String bpid, FsVolumeSpi.ScanInfo info) throws IOException
      Check whether the in-memory block record matches the block on the disk, and, in case that they are not matched, update the record or mark it as corrupted.
      Throws:
      IOException

    • getMetaDataInputStream

      LengthInputStream getMetaDataInputStream(org.apache.hadoop.hdfs.protocol.ExtendedBlock b) throws IOException
      Parameters:
      b - - the block
      Returns:
      a stream if the meta-data of the block exists; otherwise, return null.
      Throws:
      IOException

    • getLength

      long getLength(org.apache.hadoop.hdfs.protocol.ExtendedBlock b) throws IOException
      Returns the specified block's on-disk length (excluding metadata).
      Returns:
      the specified block's on-disk length (excluding metadta)
      Throws:
      IOException - on error

    • getReplica

      @Deprecated Replica getReplica(String bpid, long blockId)
      Deprecated.
      Get reference to the replica meta info in the replicasMap. To be called from methods that are synchronized on implementations of FsDatasetSpi
      Returns:
      replica from the replicas map

    • getReplicaString

      String getReplicaString(String bpid, long blockId)
      Returns:
      replica meta information

    • getStoredBlock

      org.apache.hadoop.hdfs.protocol.Block getStoredBlock(String bpid, long blkid) throws IOException
      Returns:
      the generation stamp stored with the block.
      Throws:
      IOException

    • getBlockInputStream

      InputStream getBlockInputStream(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long seekOffset) throws IOException
      Returns an input stream at specified offset of the specified block.
      Parameters:
      b - block
      seekOffset - offset with in the block to seek to
      Returns:
      an input stream to read the contents of the specified block, starting at the offset
      Throws:
      IOException

    • getTmpInputStreams

      ReplicaInputStreams getTmpInputStreams(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long blkoff, long ckoff) throws IOException
      Returns an input stream at specified offset of the specified block. The block is still in the tmp directory and is not finalized
      Returns:
      an input stream to read the contents of the specified block, starting at the offset
      Throws:
      IOException

    • createTemporary

      ReplicaHandler createTemporary(org.apache.hadoop.fs.StorageType storageType, String storageId, org.apache.hadoop.hdfs.protocol.ExtendedBlock b, boolean isTransfer) throws IOException
      Creates a temporary replica and returns the meta information of the replica .
      Parameters:
      b - block
      Returns:
      the meta info of the replica which is being written to
      Throws:
      IOException - if an error occurs

    • createRbw

      ReplicaHandler createRbw(org.apache.hadoop.fs.StorageType storageType, String storageId, org.apache.hadoop.hdfs.protocol.ExtendedBlock b, boolean allowLazyPersist) throws IOException
      Creates a RBW replica and returns the meta info of the replica
      Parameters:
      b - block
      Returns:
      the meta info of the replica which is being written to
      Throws:
      IOException - if an error occurs

    • createRbw

      ReplicaHandler createRbw(org.apache.hadoop.fs.StorageType storageType, String storageId, org.apache.hadoop.hdfs.protocol.ExtendedBlock b, boolean allowLazyPersist, long newGS) throws IOException
      Creates a RBW replica and returns the meta info of the replica
      Parameters:
      b - block
      Returns:
      the meta info of the replica which is being written to
      Throws:
      IOException - if an error occurs

    • recoverRbw

      ReplicaHandler recoverRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long newGS, long minBytesRcvd, long maxBytesRcvd) throws IOException
      Recovers a RBW replica and returns the meta info of the replica.
      Parameters:
      b - block
      newGS - the new generation stamp for the replica
      minBytesRcvd - the minimum number of bytes that the replica could have
      maxBytesRcvd - the maximum number of bytes that the replica could have
      Returns:
      the meta info of the replica which is being written to
      Throws:
      IOException - if an error occurs

    • convertTemporaryToRbw

      ReplicaInPipeline convertTemporaryToRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock temporary) throws IOException
      Covert a temporary replica to a RBW.
      Parameters:
      temporary - the temporary replica being converted
      Returns:
      the result RBW
      Throws:
      IOException

    • append

      ReplicaHandler append(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long newGS, long expectedBlockLen) throws IOException
      Append to a finalized replica and returns the meta info of the replica.
      Parameters:
      b - block
      newGS - the new generation stamp for the replica
      expectedBlockLen - the number of bytes the replica is expected to have
      Returns:
      the meata info of the replica which is being written to
      Throws:
      IOException

    • recoverAppend

      ReplicaHandler recoverAppend(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long newGS, long expectedBlockLen) throws IOException
      Recover a failed append to a finalized replica and returns the meta info of the replica.
      Parameters:
      b - block
      newGS - the new generation stamp for the replica
      expectedBlockLen - the number of bytes the replica is expected to have
      Returns:
      the meta info of the replica which is being written to
      Throws:
      IOException

    • recoverClose

      Replica recoverClose(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long newGS, long expectedBlockLen) throws IOException
      Recover a failed pipeline close. It bumps the replica's generation stamp and finalize it if RBW replica
      Parameters:
      b - block
      newGS - the new generation stamp for the replica
      expectedBlockLen - the number of bytes the replica is expected to have
      Returns:
      the storage uuid of the replica.
      Throws:
      IOException

    • finalizeBlock

      void finalizeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, boolean fsyncDir) throws IOException
      Finalizes the block previously opened for writing using writeToBlock. The block size is what is in the parameter b and it must match the amount of data written
      Parameters:
      b - Block to be finalized
      fsyncDir - whether to sync the directory changes to durable device.
      Throws:
      IOException
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException - if the replica can not be found when the block is been finalized. For instance, the block resides on an HDFS volume that has been removed.

    • unfinalizeBlock

      void unfinalizeBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b) throws IOException
      Unfinalizes the block previously opened for writing using writeToBlock. The temporary file associated with this block is deleted.
      Throws:
      IOException

    • getBlockReports

      Map<org.apache.hadoop.hdfs.server.protocol.DatanodeStorage,BlockListAsLongs> getBlockReports(String bpid)
      Returns one block report per volume.
      Parameters:
      bpid - Block Pool Id
      Returns:
      - a map of DatanodeStorage to block report for the volume.

    • getCacheReport

      List<Long> getCacheReport(String bpid)
      Returns the cache report - the full list of cached block IDs of a block pool.
      Parameters:
      bpid - Block Pool Id
      Returns:
      the cache report - the full list of cached block IDs.

    • contains

      boolean contains(org.apache.hadoop.hdfs.protocol.ExtendedBlock block)
      Does the dataset contain the block?

    • checkBlock

      void checkBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, long minLength, HdfsServerConstants.ReplicaState state) throws org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException, UnexpectedReplicaStateException, FileNotFoundException, EOFException, IOException
      Check if a block is valid.
      Parameters:
      b - The block to check.
      minLength - The minimum length that the block must have. May be 0.
      state - If this is null, it is ignored. If it is non-null, we will check that the replica has this state.
      Throws:
      org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException - If the replica is not found
      UnexpectedReplicaStateException - If the replica is not in the expected state.
      FileNotFoundException - If the block file is not found or there was an error locating it.
      EOFException - If the replica length is too short.
      IOException - May be thrown from the methods called.

    • isValidBlock

      boolean isValidBlock(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
      Is the block valid?
      Returns:
      - true if the specified block is valid

    • isValidRbw

      boolean isValidRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
      Is the block a valid RBW?
      Returns:
      - true if the specified block is a valid RBW

    • invalidate

      void invalidate(String bpid, org.apache.hadoop.hdfs.protocol.Block[] invalidBlks) throws IOException
      Invalidates the specified blocks.
      Parameters:
      bpid - Block pool Id
      invalidBlks - - the blocks to be invalidated
      Throws:
      IOException

    • invalidateMissingBlock

      void invalidateMissingBlock(String bpid, org.apache.hadoop.hdfs.protocol.Block block) throws IOException
      Invalidate a block which is not found on disk.
      Parameters:
      bpid - the block pool ID.
      block - The block to be invalidated.
      Throws:
      IOException

    • cache

      void cache(String bpid, long[] blockIds)
      Caches the specified block
      Parameters:
      bpid - Block pool id
      blockIds - - block ids to cache

    • uncache

      void uncache(String bpid, long[] blockIds)
      Uncaches the specified blocks
      Parameters:
      bpid - Block pool id
      blockIds - - blocks ids to uncache

    • isCached

      boolean isCached(String bpid, long blockId)
      Determine if the specified block is cached.
      Parameters:
      bpid - Block pool id
      blockId - - block id
      Returns:
      true if the block is cached

    • handleVolumeFailures

      void handleVolumeFailures(Set<FsVolumeSpi> failedVolumes)
      Check if all the data directories are healthy
      Parameters:
      failedVolumes -

    • shutdown

      void shutdown()
      Shutdown the FSDataset

    • adjustCrcChannelPosition

      void adjustCrcChannelPosition(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, ReplicaOutputStreams outs, int checksumSize) throws IOException
      Sets the file pointer of the checksum stream so that the last checksum will be overwritten
      Parameters:
      b - block
      outs - The streams for the data file and checksum file
      checksumSize - number of bytes each checksum has
      Throws:
      IOException

    • hasEnoughResource

      boolean hasEnoughResource()
      Checks how many valid storage volumes there are in the DataNode.
      Returns:
      true if more than the minimum number of valid volumes are left in the FSDataSet.

    • getReplicaVisibleLength

      long getReplicaVisibleLength(org.apache.hadoop.hdfs.protocol.ExtendedBlock block) throws IOException
      Get visible length of the specified replica.
      Throws:
      IOException

    • initReplicaRecovery

      ReplicaRecoveryInfo initReplicaRecovery(BlockRecoveryCommand.RecoveringBlock rBlock) throws IOException
      Initialize a replica recovery.
      Returns:
      actual state of the replica on this data-node or null if data-node does not have the replica.
      Throws:
      IOException

    • updateReplicaUnderRecovery

      Replica updateReplicaUnderRecovery(org.apache.hadoop.hdfs.protocol.ExtendedBlock oldBlock, long recoveryId, long newBlockId, long newLength) throws IOException
      Update replica's generation stamp and length and finalize it.
      Returns:
      the ID of storage that stores the block
      Throws:
      IOException

    • addBlockPool

      void addBlockPool(String bpid, org.apache.hadoop.conf.Configuration conf) throws IOException
      add new block pool ID
      Parameters:
      bpid - Block pool Id
      conf - Configuration
      Throws:
      IOException

    • shutdownBlockPool

      void shutdownBlockPool(String bpid)
      Shutdown and remove the block pool from underlying storage.
      Parameters:
      bpid - Block pool Id to be removed

    • deleteBlockPool

      void deleteBlockPool(String bpid, boolean force) throws IOException
      Deletes the block pool directories. If force is false, directories are deleted only if no block files exist for the block pool. If force is true entire directory for the blockpool is deleted along with its contents.
      Parameters:
      bpid - BlockPool Id to be deleted.
      force - If force is false, directories are deleted only if no block files exist for the block pool, otherwise entire directory for the blockpool is deleted along with its contents.
      Throws:
      IOException

    • getBlockLocalPathInfo

      org.apache.hadoop.hdfs.protocol.BlockLocalPathInfo getBlockLocalPathInfo(org.apache.hadoop.hdfs.protocol.ExtendedBlock b) throws IOException
      Get BlockLocalPathInfo for the given block.
      Throws:
      IOException

    • enableTrash

      void enableTrash(String bpid)
      Enable 'trash' for the given dataset. When trash is enabled, files are moved to a separate trash directory instead of being deleted immediately. This can be useful for example during rolling upgrades.

    • clearTrash

      void clearTrash(String bpid)
      Clear trash

    • trashEnabled

      boolean trashEnabled(String bpid)
      Returns:
      true when trash is enabled

    • setRollingUpgradeMarker

      void setRollingUpgradeMarker(String bpid) throws IOException
      Create a marker file indicating that a rolling upgrade is in progress.
      Throws:
      IOException

    • clearRollingUpgradeMarker

      void clearRollingUpgradeMarker(String bpid) throws IOException
      Delete the rolling upgrade marker file if it exists.
      Parameters:
      bpid -
      Throws:
      IOException

    • submitBackgroundSyncFileRangeRequest

      void submitBackgroundSyncFileRangeRequest(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, ReplicaOutputStreams outs, long offset, long nbytes, int flags)
      submit a sync_file_range request to AsyncDiskService.

    • onCompleteLazyPersist

      void onCompleteLazyPersist(String bpId, long blockId, long creationTime, File[] savedFiles, V targetVolume)
      Callback from RamDiskAsyncLazyPersistService upon async lazy persist task end

    • onFailLazyPersist

      void onFailLazyPersist(String bpId, long blockId)
      Callback from RamDiskAsyncLazyPersistService upon async lazy persist task fail

    • moveBlockAcrossStorage

      ReplicaInfo moveBlockAcrossStorage(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, org.apache.hadoop.fs.StorageType targetStorageType, String storageId) throws IOException
      Move block from one storage to another storage
      Throws:
      IOException

    • setPinning

      void setPinning(org.apache.hadoop.hdfs.protocol.ExtendedBlock block) throws IOException
      Set a block to be pinned on this datanode so that it cannot be moved by Balancer/Mover. It is a no-op when dfs.datanode.block-pinning.enabled is set to false.
      Throws:
      IOException

    • getPinning

      boolean getPinning(org.apache.hadoop.hdfs.protocol.ExtendedBlock block) throws IOException
      Check whether the block was pinned
      Throws:
      IOException

    • isDeletingBlock

      boolean isDeletingBlock(String bpid, long blockId)
      Confirm whether the block is deleting

    • moveBlockAcrossVolumes

      ReplicaInfo moveBlockAcrossVolumes(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, FsVolumeSpi destination) throws IOException
      Moves a given block from one volume to another volume. This is used by disk balancer.
      Parameters:
      block - - ExtendedBlock
      destination - - Destination volume
      Returns:
      Old replica info
      Throws:
      IOException

    • acquireDatasetLockManager

      DataNodeLockManager<? extends AutoCloseDataSetLock> acquireDatasetLockManager()
      Acquire lock Manager for the data set. This prevents other threads from modifying the volume map structure inside the datanode.
      Returns:
      The AutoClosable read lock instance.

    • deepCopyReplica

      Set<? extends Replica> deepCopyReplica(String bpid) throws IOException
      Deep copy the replica info belonging to given block pool.
      Parameters:
      bpid - Specified block pool id.
      Returns:
      A set of replica info.
      Throws:
      IOException

    • getMountVolumeMap

      MountVolumeMap getMountVolumeMap() throws IOException
      Get relationship between disk mount and FsVolume.
      Returns:
      Disk mount and FsVolume relationship.
      Throws:
      IOException

    • getVolumeList

      List<FsVolumeImpl> getVolumeList()
      Get the volume list.

    • setLastDirScannerFinishTime

      default void setLastDirScannerFinishTime(long time)
      Set the last time in milliseconds when the directory scanner successfully ran.
      Parameters:
      time - the last time in milliseconds when the directory scanner successfully ran.