Package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl

Class FsVolumeImpl

java.lang.Object

org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl

All Implemented Interfaces:

Checkable<FsVolumeSpi.VolumeCheckContext,VolumeCheckResult>, FsVolumeSpi

@Private
@VisibleForTesting
public class FsVolumeImpl
extends Object
implements FsVolumeSpi

The underlying volume used to store replica. It uses the FsDatasetImpl object for synchronization.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	FsVolumeImpl.BlockDirFilter	Filter for block file names stored on the file system volumes.

Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi

FsVolumeSpi.BlockIterator, FsVolumeSpi.ScanInfo, FsVolumeSpi.VolumeCheckContext

Field Summary

Fields

Modifier and Type	Field	Description
protected ThreadPoolExecutor	cacheExecutor	Per-volume worker pool that processes new blocks to cache.
protected long	configuredCapacity
static final org.slf4j.Logger	LOG

Method Summary

Modifier and Type	Method	Description
ReplicaInfo	activateSavedReplica(String bpid, ReplicaInfo replicaInfo, org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaTracker.RamDiskReplica replicaState)
ReplicaInPipeline	append(String bpid, ReplicaInfo replicaInfo, long newGS, long estimateBlockLen)
VolumeCheckResult	check(FsVolumeSpi.VolumeCheckContext ignored)	Query the health of this object.
void	compileReport(String bpid, Collection<FsVolumeSpi.ScanInfo> report, DirectoryScanner.ReportCompiler reportCompiler)	Compile a list of FsVolumeSpi.ScanInfo for the blocks in the block pool with id bpid.
ReplicaInPipeline	convertTemporaryToRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, ReplicaInfo temp)
File[]	copyBlockToLazyPersistLocation(String bpId, long blockId, long genStamp, ReplicaInfo replicaInfo, int smallBufferSize, org.apache.hadoop.conf.Configuration conf)
ReplicaInPipeline	createRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
ReplicaInPipeline	createTemporary(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
long	getAvailable()	Calculate the available space of the filesystem, excluding space reserved for non-HDFS and space reserved for RBW.
URI	getBaseURI()
String[]	getBlockPoolList()	Make a deep copy of the list of currently active BPIDs.
Map<String,BlockPoolSlice>	getBlockPoolSlices()
long	getCapacity()	Return either the configured capacity of the file system if configured; or the capacity of the file system excluding space reserved for non-HDFS.
File	getCurrentDir()
FsDatasetSpi<? extends FsVolumeSpi>	getDataset()	Get the FSDatasetSpi which this volume is a part of.
long	getDfsUsed()
long	getDfUsed()	This function is only used for Mock.
FileIoProvider	getFileIoProvider()
File	getFinalizedDir(String bpid)
protected File	getLazyPersistDir(String bpid)
DataNodeVolumeMetrics	getMetrics()
long	getNonDfsUsed()	Unplanned Non-DFS usage, i.e.
protected File	getRbwDir(String bpid)
int	getReferenceCount()
long	getReservedForReplicas()
String	getStorageID()
StorageLocation	getStorageLocation()
org.apache.hadoop.fs.StorageType	getStorageType()
protected File	getTmpDir(String bpid)
org.apache.hadoop.fs.DF	getUsageStats(org.apache.hadoop.conf.Configuration conf)
ReplicaInfo	hardLinkBlockToTmpLocation(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, ReplicaInfo replicaInfo)
void	incrNumBlocks(String bpid)
protected ThreadPoolExecutor	initializeCacheExecutor(File parent)
boolean	isRAMStorage()	Returns true if the volume is backed by RAM storage.
boolean	isTransientStorage()	Returns true if the volume is NOT backed by persistent storage.
FsVolumeSpi.BlockIterator	loadBlockIterator(String bpid, String name)	Load a saved block iterator.
byte[]	loadLastPartialChunkChecksum(File blockFile, File metaFile)	Load last partial chunk checksum from checksum file.
ReplicaInfo	moveBlockToTmpLocation(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, ReplicaInfo replicaInfo, int smallBufferSize, org.apache.hadoop.conf.Configuration conf)
FsVolumeSpi.BlockIterator	newBlockIterator(String bpid, String name)	Create a new block iterator.
static String	nextSorted(List<String> arr, String prev)
FsVolumeReference	obtainReference()	Obtain a reference object that had increased 1 reference count of the volume.
void	releaseLockedMemory(long bytesToRelease)	Release reserved memory for an RBW block written to transient storage i.e.
void	releaseReservedSpace(long bytesToRelease)	Release disk space previously reserved for block opened for write.
void	reserveSpaceForReplica(long bytesToReserve)	Reserve disk space for a block (RBW or Re-replicating) so a writer does not run out of space before the block is full.
void	resolveDuplicateReplicas(String bpid, ReplicaInfo memBlockInfo, ReplicaInfo diskBlockInfo, org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaMap volumeMap)
void	setCapacityForTesting(long capacity)	This function MUST NOT be used outside of tests.
String	toString()
ReplicaInPipeline	updateRURCopyOnTruncate(ReplicaInfo rur, String bpid, long newBlockId, long recoveryId, long newlength)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

LOG

public static final org.slf4j.Logger LOG

configuredCapacity

protected volatile long configuredCapacity

cacheExecutor

protected ThreadPoolExecutor cacheExecutor

Per-volume worker pool that processes new blocks to cache. The maximum number of workers per volume is bounded (configurable via dfs.datanode.fsdatasetcache.max.threads.per.volume) to limit resource contention.

Method Details

initializeCacheExecutor

protected ThreadPoolExecutor initializeCacheExecutor(File parent)

obtainReference

public FsVolumeReference obtainReference()
                                  throws ClosedChannelException

Description copied from interface: FsVolumeSpi

Obtain a reference object that had increased 1 reference count of the volume. It is caller's responsibility to close FsVolumeReference to decrease the reference count on the volume.

Specified by:

obtainReference in interface FsVolumeSpi

Throws:

ClosedChannelException

getReferenceCount

@VisibleForTesting
public int getReferenceCount()

getCurrentDir

@VisibleForTesting
public File getCurrentDir()

getRbwDir

protected File getRbwDir(String bpid)
                  throws IOException

Throws:

IOException

getLazyPersistDir

protected File getLazyPersistDir(String bpid)
                          throws IOException

Throws:

IOException

getTmpDir

protected File getTmpDir(String bpid)
                  throws IOException

Throws:

IOException

getDfsUsed

@VisibleForTesting
public long getDfsUsed()
                throws IOException

Throws:

IOException

getCapacity

@VisibleForTesting
public long getCapacity()

Return either the configured capacity of the file system if configured; or the capacity of the file system excluding space reserved for non-HDFS. When same-disk-tiering is turned on, the reported capacity will take reservedForArchive value into consideration of.

Returns:

the unreserved number of bytes left in this filesystem. May be zero.

setCapacityForTesting

@VisibleForTesting
public void setCapacityForTesting(long capacity)

This function MUST NOT be used outside of tests.

Parameters:

capacity -

getAvailable

public long getAvailable()
                  throws IOException

Calculate the available space of the filesystem, excluding space reserved for non-HDFS and space reserved for RBW.

Specified by:

getAvailable in interface FsVolumeSpi

Returns:

the available number of bytes left in this filesystem. May be zero.

Throws:

IOException

getDfUsed

@VisibleForTesting
public long getDfUsed()

This function is only used for Mock.

getNonDfsUsed

public long getNonDfsUsed()
                   throws IOException

Unplanned Non-DFS usage, i.e. Extra usage beyond reserved.

Returns:

Disk usage excluding space used by HDFS and excluding space reserved for blocks open for write.

Throws:

IOException

getReservedForReplicas

@VisibleForTesting
public long getReservedForReplicas()

getBlockPoolSlices

public Map<String,BlockPoolSlice> getBlockPoolSlices()

getBaseURI

public URI getBaseURI()

Specified by:

getBaseURI in interface FsVolumeSpi

Returns:

the base path to the volume

getUsageStats

public org.apache.hadoop.fs.DF getUsageStats(org.apache.hadoop.conf.Configuration conf)

Specified by:

getUsageStats in interface FsVolumeSpi

getStorageLocation

public StorageLocation getStorageLocation()

Specified by:

getStorageLocation in interface FsVolumeSpi

Returns:

the StorageLocation to the volume

isTransientStorage

public boolean isTransientStorage()

Description copied from interface: FsVolumeSpi

Returns true if the volume is NOT backed by persistent storage.

Specified by:

isTransientStorage in interface FsVolumeSpi

isRAMStorage

public boolean isRAMStorage()

Description copied from interface: FsVolumeSpi

Returns true if the volume is backed by RAM storage.

Specified by:

isRAMStorage in interface FsVolumeSpi

getFinalizedDir

@VisibleForTesting
public File getFinalizedDir(String bpid)
                     throws IOException

Throws:

IOException

getBlockPoolList

public String[] getBlockPoolList()

Make a deep copy of the list of currently active BPIDs.

Specified by:

getBlockPoolList in interface FsVolumeSpi

Returns:

a list of block pools.

reserveSpaceForReplica

public void reserveSpaceForReplica(long bytesToReserve)

Description copied from interface: FsVolumeSpi

Reserve disk space for a block (RBW or Re-replicating) so a writer does not run out of space before the block is full.

Specified by:

reserveSpaceForReplica in interface FsVolumeSpi

releaseReservedSpace

public void releaseReservedSpace(long bytesToRelease)

Description copied from interface: FsVolumeSpi

Release disk space previously reserved for block opened for write.

Specified by:

releaseReservedSpace in interface FsVolumeSpi

releaseLockedMemory

public void releaseLockedMemory(long bytesToRelease)

Description copied from interface: FsVolumeSpi

Release reserved memory for an RBW block written to transient storage i.e. RAM. bytesToRelease will be rounded down to the OS page size since locked memory reservation must always be a multiple of the page size.

Specified by:

releaseLockedMemory in interface FsVolumeSpi

nextSorted

@VisibleForTesting
public static String nextSorted(List<String> arr,
 String prev)

newBlockIterator

public FsVolumeSpi.BlockIterator newBlockIterator(String bpid,
 String name)

Description copied from interface: FsVolumeSpi

Create a new block iterator. It will start at the beginning of the block set.

Specified by:

newBlockIterator in interface FsVolumeSpi

Parameters:

bpid - The block pool id to iterate over.

name - The name of the block iterator to create.

Returns:

The new block iterator.

loadBlockIterator

public FsVolumeSpi.BlockIterator loadBlockIterator(String bpid,
 String name)
                                            throws IOException

Description copied from interface: FsVolumeSpi

Load a saved block iterator.

Specified by:

loadBlockIterator in interface FsVolumeSpi

Parameters:

bpid - The block pool id to iterate over.

name - The name of the block iterator to load.

Returns:

The saved block iterator.

Throws:

IOException - If there was an IO error loading the saved block iterator.

getDataset

public FsDatasetSpi<? extends FsVolumeSpi> getDataset()

Description copied from interface: FsVolumeSpi

Get the FSDatasetSpi which this volume is a part of.

Specified by:

getDataset in interface FsVolumeSpi

check

public VolumeCheckResult check(FsVolumeSpi.VolumeCheckContext ignored)
                        throws org.apache.hadoop.util.DiskChecker.DiskErrorException

Description copied from interface: Checkable

Query the health of this object. This method may hang indefinitely depending on the status of the target resource.

Specified by:

check in interface Checkable<FsVolumeSpi.VolumeCheckContext,VolumeCheckResult>

Parameters:

ignored - for the probe operation. May be null depending on the implementation.

Returns:

result of the check operation.

Throws:

org.apache.hadoop.util.DiskChecker.DiskErrorException

toString

public String toString()

Overrides:

toString in class Object

getStorageID

public String getStorageID()

Specified by:

getStorageID in interface FsVolumeSpi

Returns:

the StorageUuid of the volume

getStorageType

public org.apache.hadoop.fs.StorageType getStorageType()

Specified by:

getStorageType in interface FsVolumeSpi

Returns:

the StorageType of the volume

loadLastPartialChunkChecksum

public byte[] loadLastPartialChunkChecksum(File blockFile,
 File metaFile)
                                    throws IOException

Description copied from interface: FsVolumeSpi

Load last partial chunk checksum from checksum file. Need to be called with FsDataset lock acquired.

Specified by:

loadLastPartialChunkChecksum in interface FsVolumeSpi

Returns:

the last partial checksum

Throws:

IOException

append

public ReplicaInPipeline append(String bpid,
 ReplicaInfo replicaInfo,
 long newGS,
 long estimateBlockLen)
                         throws IOException

Throws:

IOException

createRbw

public ReplicaInPipeline createRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
                            throws IOException

Throws:

IOException

convertTemporaryToRbw

public ReplicaInPipeline convertTemporaryToRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b,
 ReplicaInfo temp)
                                        throws IOException

Throws:

IOException

createTemporary

public ReplicaInPipeline createTemporary(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)
                                  throws IOException

Throws:

IOException

updateRURCopyOnTruncate

public ReplicaInPipeline updateRURCopyOnTruncate(ReplicaInfo rur,
 String bpid,
 long newBlockId,
 long recoveryId,
 long newlength)
                                          throws IOException

Throws:

IOException

compileReport

public void compileReport(String bpid,
 Collection<FsVolumeSpi.ScanInfo> report,
 DirectoryScanner.ReportCompiler reportCompiler)
                   throws InterruptedException,
IOException

Description copied from interface: FsVolumeSpi

Compile a list of FsVolumeSpi.ScanInfo for the blocks in the block pool with id bpid.

Specified by:

compileReport in interface FsVolumeSpi

Parameters:

bpid - block pool id to scan

report - the list onto which blocks reports are placed

Throws:

InterruptedException

IOException

getFileIoProvider

public FileIoProvider getFileIoProvider()

Specified by:

getFileIoProvider in interface FsVolumeSpi

getMetrics

public DataNodeVolumeMetrics getMetrics()

Specified by:

getMetrics in interface FsVolumeSpi

moveBlockToTmpLocation

public ReplicaInfo moveBlockToTmpLocation(org.apache.hadoop.hdfs.protocol.ExtendedBlock block,
 ReplicaInfo replicaInfo,
 int smallBufferSize,
 org.apache.hadoop.conf.Configuration conf)
                                   throws IOException

Throws:

IOException

hardLinkBlockToTmpLocation

public ReplicaInfo hardLinkBlockToTmpLocation(org.apache.hadoop.hdfs.protocol.ExtendedBlock block,
 ReplicaInfo replicaInfo)
                                       throws IOException

Throws:

IOException

copyBlockToLazyPersistLocation

public File[] copyBlockToLazyPersistLocation(String bpId,
 long blockId,
 long genStamp,
 ReplicaInfo replicaInfo,
 int smallBufferSize,
 org.apache.hadoop.conf.Configuration conf)
                                      throws IOException

Throws:

IOException

incrNumBlocks

public void incrNumBlocks(String bpid)
                   throws IOException

Throws:

IOException

resolveDuplicateReplicas

public void resolveDuplicateReplicas(String bpid,
 ReplicaInfo memBlockInfo,
 ReplicaInfo diskBlockInfo,
 org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaMap volumeMap)
                              throws IOException

Throws:

IOException

activateSavedReplica

public ReplicaInfo activateSavedReplica(String bpid,
 ReplicaInfo replicaInfo,
 org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaTracker.RamDiskReplica replicaState)
                                 throws IOException

Throws:

IOException