Package org.apache.hadoop.hdfs

Class DFSClient

java.lang.Object

org.apache.hadoop.hdfs.DFSClient

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.crypto.key.KeyProviderTokenIssuer, DataEncryptionKeyFactory, RemotePeerFactory, org.apache.hadoop.security.token.DelegationTokenIssuer

@Private
public class DFSClient
extends Object
implements Closeable, RemotePeerFactory, DataEncryptionKeyFactory, org.apache.hadoop.crypto.key.KeyProviderTokenIssuer

DFSClient can connect to a Hadoop Filesystem and perform basic file tasks. It uses the ClientProtocol to communicate with a NameNode daemon, and connects directly to DataNodes to read/write block data. Hadoop DFS users should obtain an instance of DistributedFileSystem, which uses DFSClient to handle filesystem tasks.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DFSClient.DFSDataInputStream	Deprecated. use HdfsDataInputStream instead.
static class	DFSClient.Renewer

Field Summary

Fields

Modifier and Type	Field	Description
static final org.slf4j.Logger	LOG

Fields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

TOKEN_LOG

Constructor Summary

Constructors

Constructor	Description
DFSClient(InetSocketAddress address, org.apache.hadoop.conf.Configuration conf)
DFSClient(URI nameNodeUri, org.apache.hadoop.conf.Configuration conf)	Same as this(nameNodeUri, conf, null);
DFSClient(URI nameNodeUri, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem.Statistics stats)	Same as this(nameNodeUri, null, conf, stats);
DFSClient(URI nameNodeUri, ClientProtocol rpcNamenode, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem.Statistics stats)	Create a new DFSClient connected to the given nameNodeUri or rpcNamenode.
DFSClient(org.apache.hadoop.conf.Configuration conf)	Deprecated. Deprecated at 0.21

Method Summary

Modifier and Type	Method	Description
long	addCacheDirective(CacheDirectiveInfo info, EnumSet<CacheFlag> flags)
void	addCachePool(CachePoolInfo info)
AddErasureCodingPolicyResponse[]	addErasureCodingPolicies(ErasureCodingPolicy[] policies)
void	addLocatedBlocksRefresh(DFSInputStream dfsInputStream)	Adds the DFSInputStream to the LocatedBlocksRefresher, so that the underlying LocatedBlocks is periodically refreshed.
void	addNodeToDeadNodeDetector(DFSInputStream dfsInputStream, DatanodeInfo datanodeInfo)	Add given datanode in DeadNodeDetector.
void	allowSnapshot(String snapshotRoot)	Allow snapshot on a directory.
HdfsDataOutputStream	append(String src, int buffersize, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.FileSystem.Statistics statistics)	Append to an existing HDFS file.
HdfsDataOutputStream	append(String src, int buffersize, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.FileSystem.Statistics statistics, InetSocketAddress[] favoredNodes)	Append to an existing HDFS file.
BatchedDirectoryListing	batchedListPaths(String[] srcs, byte[] startAfter, boolean needLocation)	Get a batched listing for the indicated directories
void	cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)	Deprecated. Use Token.cancel instead.
void	checkAccess(String src, org.apache.hadoop.fs.permission.FsAction mode)
void	clearDataEncryptionKey()
void	close()	Close the file system, abandoning all of the leases and files being created and close connections to the namenode.
void	closeAllFilesBeingWritten(boolean abort)	Close/abort all files being written.
void	closeOutputStreams(boolean abort)	Close all open streams, abandoning all of the leases and files being created.
void	concat(String trg, String[] srcs)	Move blocks from src to trg and delete src See ClientProtocol.concat(java.lang.String, java.lang.String[]).
protected IOStreamPair	connectToDN(DatanodeInfo dn, int timeout, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken)
OutputStream	create(String src, boolean overwrite)	Call create(String, boolean, short, long, Progressable) with default replication and blockSize and null progress.
OutputStream	create(String src, boolean overwrite, short replication, long blockSize)	Call create(String, boolean, short, long, Progressable) with null progress.
OutputStream	create(String src, boolean overwrite, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)	Call create(String, boolean, short, long, Progressable, int) with default bufferSize.
OutputStream	create(String src, boolean overwrite, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, int buffersize)	Call create(String, FsPermission, EnumSet, short, long, Progressable, int, ChecksumOpt) with default permission FsPermission.getFileDefault().
OutputStream	create(String src, boolean overwrite, org.apache.hadoop.util.Progressable progress)	Call create(String, boolean, short, long, Progressable) with default replication and blockSize.
DFSOutputStream	create(String src, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, boolean createParent, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, int buffersize, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)	Create a new dfs file with the specified block replication with write-progress reporting and return an output stream for writing into the file.
DFSOutputStream	create(String src, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, boolean createParent, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, int buffersize, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, InetSocketAddress[] favoredNodes)	Same as create(String, FsPermission, EnumSet, boolean, short, long, Progressable, int, ChecksumOpt) with the addition of favoredNodes that is a hint to where the namenode should place the file blocks.
DFSOutputStream	create(String src, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, boolean createParent, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, int buffersize, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, InetSocketAddress[] favoredNodes, String ecPolicyName)	Same as create(String, FsPermission, EnumSet, boolean, short, long, Progressable, int, ChecksumOpt, InetSocketAddress[]) with the addition of ecPolicyName that is used to specify a specific erasure coding policy instead of inheriting any policy from this new file's parent directory.
DFSOutputStream	create(String src, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, boolean createParent, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, int buffersize, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, InetSocketAddress[] favoredNodes, String ecPolicyName, String storagePolicy)	Same as create(String, FsPermission, EnumSet, boolean, short, long, Progressable, int, ChecksumOpt, InetSocketAddress[], String) with the storagePolicy that is used to specify a specific storage policy instead of inheriting any policy from this new file's parent directory.
DFSOutputStream	create(String src, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, int buffersize, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)	Call create(String, FsPermission, EnumSet, boolean, short, long, Progressable, int, ChecksumOpt) with createParent set to true.
void	createEncryptionZone(String src, String keyName)
String	createSnapshot(String snapshotRoot, String snapshotName)	Create one snapshot.
void	createSymlink(String target, String link, boolean createParent)	Creates a symbolic link.
HdfsDataInputStream	createWrappedInputStream(DFSInputStream dfsis)	Wraps the stream in a CryptoInputStream if the underlying file is encrypted.
HdfsDataOutputStream	createWrappedOutputStream(DFSOutputStream dfsos, org.apache.hadoop.fs.FileSystem.Statistics statistics)	Wraps the stream in a CryptoOutputStream if the underlying file is encrypted.
HdfsDataOutputStream	createWrappedOutputStream(DFSOutputStream dfsos, org.apache.hadoop.fs.FileSystem.Statistics statistics, long startPos)	Wraps the stream in a CryptoOutputStream if the underlying file is encrypted.
DatanodeInfo[]	datanodeReport(HdfsConstants.DatanodeReportType type)
boolean	delete(String src)	Deprecated.
boolean	delete(String src, boolean recursive)	delete file or directory.
void	deleteSnapshot(String snapshotRoot, String snapshotName)	Delete a snapshot of a snapshottable directory.
void	disableErasureCodingPolicy(String ecPolicyName)
void	disallowSnapshot(String snapshotRoot)	Disallow snapshot on a directory.
void	enableErasureCodingPolicy(String ecPolicyName)
boolean	exists(String src)	Implemented using getFileInfo(src)
void	finalizeUpgrade()
org.apache.hadoop.fs.permission.AclStatus	getAclStatus(String src)
protected LocatedBlocks	getBlockLocations(String src, long length)
org.apache.hadoop.fs.BlockLocation[]	getBlockLocations(String src, long start, long length)	Get block location info about file getBlockLocations() returns a list of hostnames that store data for a specific file region.
long	getBlockSize(String f)
long	getBytesInFutureBlocks()	Returns number of bytes that reside in Blocks with future generation stamps.
String	getCanonicalServiceName()	Get a canonical token service name for this client's tokens.
ClientContext	getClientContext()
String	getClientName()
DfsClientConf	getConf()
long	getCorruptBlocksCount()	Returns count of blocks with at least one replica marked corrupt.
DatanodeStorageReport[]	getDatanodeStorageReport(HdfsConstants.DatanodeReportType type)
DeadNodeDetector	getDeadNodeDetector()	Obtain DeadNodeDetector of the current client.
ConcurrentHashMap<DatanodeInfo,DatanodeInfo>	getDeadNodes(DFSInputStream dfsInputStream)	If deadNodeDetectionEnabled is true, return the dead nodes that detected by all the DFSInputStreams in the same client.
CachingStrategy	getDefaultReadCachingStrategy()
CachingStrategy	getDefaultWriteCachingStrategy()
org.apache.hadoop.security.token.Token<?>	getDelegationToken(String renewer)
org.apache.hadoop.security.token.Token<DelegationTokenIdentifier>	getDelegationToken(org.apache.hadoop.io.Text renewer)
org.apache.hadoop.fs.FsStatus	getDiskStatus()
ECTopologyVerifierResult	getECTopologyResultForPolicies(String... policyNames)
org.apache.hadoop.fs.Path	getEnclosingRoot(String src)
DataEncryptionKey	getEncryptionKey()
Map<String,String>	getErasureCodingCodecs()
ErasureCodingPolicyInfo[]	getErasureCodingPolicies()
ErasureCodingPolicy	getErasureCodingPolicy(String src)	Get the erasure coding policy information for the specified path
EncryptionZone	getEZForPath(String src)
org.apache.hadoop.fs.MD5MD5CRC32FileChecksum	getFileChecksum(String src, long length)	Get the checksum of the whole file or a range of the file.
org.apache.hadoop.fs.FileChecksum	getFileChecksumWithCombineMode(String src, long length)	Get the checksum of the whole file or a range of the file.
HdfsFileStatus	getFileInfo(String src)	Get the file info for a specific file or directory.
HdfsFileStatus	getFileLinkInfo(String src)	Get the file info for a specific file or directory.
org.apache.hadoop.ha.HAServiceProtocol.HAServiceState	getHAServiceState()	An unblocking call to get the HA service state of NameNode.
DFSInotifyEventInputStream	getInotifyEventStream()
DFSInotifyEventInputStream	getInotifyEventStream(long lastReadTxid)
org.apache.hadoop.crypto.key.KeyProvider	getKeyProvider()
URI	getKeyProviderUri()
LeaseRenewer	getLeaseRenewer()	Return the lease renewer instance.
String	getLinkTarget(String path)	Resolve the *first* symlink, if any, in the path.
LocatedBlocksRefresher	getLocatedBlockRefresher()	Obtain LocatedBlocksRefresher of the current client.
LocatedBlocks	getLocatedBlocks(String src, long start)	Get locations of the blocks of the specified file `src` from offset `start` within the prefetch size which is related to parameter `dfs.client.read.prefetch.size`.
LocatedBlocks	getLocatedBlocks(String src, long start, long length)	This is just a wrapper around callGetBlockLocations, but non-static so that we can stub it out for tests.
HdfsLocatedFileStatus	getLocatedFileInfo(String src, boolean needBlockToken)	Get the file info for a specific file or directory.
long	getLowRedundancyBlocksCount()	Returns aggregated count of blocks with less redundancy.
long	getMissingBlocksCount()	Returns count of blocks with no good replicas left.
long	getMissingReplOneBlocksCount()	Returns count of blocks with replication factor 1 and have lost the only replica.
ClientProtocol	getNamenode()	Get the namenode associated with this DFSClient object
int	getNumOfFilesBeingWritten()
long	getPendingDeletionBlocksCount()	Returns count of blocks pending on deletion.
long	getRefreshReadBlkLocationsInterval()
SaslDataTransferClient	getSaslDataTransferClient()	Returns the SaslDataTransferClient configured for this DFSClient.
org.apache.hadoop.fs.FsServerDefaults	getServerDefaults()	Get server default values for a number of configuration params.
SnapshotDiffReport	getSnapshotDiffReport(String snapshotDir, String fromSnapshot, String toSnapshot)	Get the difference between two snapshots, or between a snapshot and the current tree of a directory.
SnapshotDiffReportListing	getSnapshotDiffReportListing(String snapshotDir, String fromSnapshot, String toSnapshot, byte[] startPath, int index)	Get the difference between two snapshots of a directory iteratively.
SnapshotStatus[]	getSnapshotListing(String snapshotRoot)	Get listing of all the snapshots for a snapshottable directory.
SnapshottableDirectoryStatus[]	getSnapshottableDirListing()	Get all the current snapshottable directories.
static long	getStateAtIndex(long[] states, int index)
BlockStoragePolicy[]	getStoragePolicies()
BlockStoragePolicy	getStoragePolicy(String path)
byte[]	getXAttr(String src, String name)
Map<String,byte[]>	getXAttrs(String src)
Map<String,byte[]>	getXAttrs(String src, List<String> names)
protected org.apache.hadoop.util.DataChecksum.Type	inferChecksumTypeByReading(LocatedBlock lb, DatanodeInfo dn)	Infer the checksum type for a replica by sending an OP_READ_BLOCK for the first byte of that replica.
boolean	isClientRunning()
boolean	isDeadNode(DFSInputStream dfsInputStream, DatanodeInfo datanodeInfo)	If deadNodeDetectionEnabled is true, judgement based on whether this datanode is included or not in DeadNodeDetector.
boolean	isFileClosed(String src)	Close status of a file
boolean	isFilesBeingWrittenEmpty()	Is file-being-written map empty?
org.apache.hadoop.fs.RemoteIterator<CacheDirectiveEntry>	listCacheDirectives(CacheDirectiveInfo filter)
org.apache.hadoop.fs.RemoteIterator<CachePoolEntry>	listCachePools()
CorruptFileBlocks	listCorruptFileBlocks(String path, String cookie)
org.apache.hadoop.fs.RemoteIterator<EncryptionZone>	listEncryptionZones()
org.apache.hadoop.fs.RemoteIterator<OpenFileEntry>	listOpenFiles()	Deprecated.
org.apache.hadoop.fs.RemoteIterator<OpenFileEntry>	listOpenFiles(String path)	Get a remote iterator to the open files list by path, managed by NameNode.
org.apache.hadoop.fs.RemoteIterator<OpenFileEntry>	listOpenFiles(EnumSet<OpenFilesIterator.OpenFilesType> openFilesTypes)	Get a remote iterator to the open files list by type, managed by NameNode.
org.apache.hadoop.fs.RemoteIterator<OpenFileEntry>	listOpenFiles(EnumSet<OpenFilesIterator.OpenFilesType> openFilesTypes, String path)	Get a remote iterator to the open files list by type and path, managed by NameNode.
DirectoryListing	listPaths(String src, byte[] startAfter)	Get a partial listing of the indicated directory No block locations need to be fetched
DirectoryListing	listPaths(String src, byte[] startAfter, boolean needLocation)	Get a partial listing of the indicated directory Recommend to use HdfsFileStatus.EMPTY_NAME as startAfter if the application wants to fetch a listing starting from the first entry in the directory
org.apache.hadoop.fs.RemoteIterator<ZoneReencryptionStatus>	listReencryptionStatus()
List<String>	listXAttrs(String src)
void	metaSave(String pathname)	Dumps DFS data structures into specified file.
boolean	mkdirs(String src)	Deprecated.
boolean	mkdirs(String src, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent)	Create a directory (or hierarchy of directories) with the given name and permission.
void	modifyAclEntries(String src, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
void	modifyCacheDirective(CacheDirectiveInfo info, EnumSet<CacheFlag> flags)
void	modifyCachePool(CachePoolInfo info)
void	msync()	A blocking call to wait for Observer NameNode state ID to reach to the current client state ID.
Peer	newConnectedPeer(InetSocketAddress addr, org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken, DatanodeID datanodeId)
DataEncryptionKey	newDataEncryptionKey()	Creates a new DataEncryptionKey.
DFSInputStream	open(String src)
DFSInputStream	open(String src, int buffersize, boolean verifyChecksum)	Create an input stream that obtains a nodelist from the namenode, and then reads from all the right places.
DFSInputStream	open(String src, int buffersize, boolean verifyChecksum, org.apache.hadoop.fs.FileSystem.Statistics stats)	Deprecated. Use open(String, int, boolean) instead.
DFSInputStream	open(HdfsPathHandle fd, int buffersize, boolean verifyChecksum)	Create an input stream from the HdfsPathHandle if the constraints encoded from DistributedFileSystem.createPathHandle(FileStatus, Options.HandleOpt...) are satisfied.
DFSOutputStream	primitiveCreate(String src, org.apache.hadoop.fs.permission.FsPermission absPermission, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, boolean createParent, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, int buffersize, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)	Same as {create(String, FsPermission, EnumSet, short, long, Progressable, int, ChecksumOpt) except that the permission is absolute (ie has already been masked with umask.
boolean	primitiveMkdir(String src, org.apache.hadoop.fs.permission.FsPermission absPermission)	Same {mkdirs(String, FsPermission, boolean) except that the permissions has already been masked against umask.
boolean	primitiveMkdir(String src, org.apache.hadoop.fs.permission.FsPermission absPermission, boolean createParent)	Same {mkdirs(String, FsPermission, boolean) except that the permissions has already been masked against umask.
void	putFileBeingWritten(String key, DFSOutputStream out)	Put a file.
void	reencryptEncryptionZone(String zone, HdfsConstants.ReencryptAction action)
void	refreshNodes()	Refresh the hosts and exclude files.
void	removeAcl(String src)
void	removeAclEntries(String src, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
void	removeCacheDirective(long id)
void	removeCachePool(String poolName)
void	removeDefaultAcl(String src)
void	removeErasureCodingPolicy(String ecPolicyName)
void	removeFileBeingWritten(String key)	Remove a file.
void	removeLocatedBlocksRefresh(DFSInputStream dfsInputStream)	Removes the DFSInputStream from the LocatedBlocksRefresher, so that the underlying LocatedBlocks is no longer periodically refreshed.
void	removeNodeFromDeadNodeDetector(DFSInputStream dfsInputStream, DatanodeInfo datanodeInfo)	Remove given datanode from DeadNodeDetector.
void	removeNodeFromDeadNodeDetector(DFSInputStream dfsInputStream, LocatedBlocks locatedBlocks)	Remove datanodes that given block placed on from DeadNodeDetector.
void	removeXAttr(String src, String name)
boolean	rename(String src, String dst)	Deprecated. Use rename(String, String, Options.Rename...) instead.
void	rename(String src, String dst, org.apache.hadoop.fs.Options.Rename... options)	Rename file or directory.
void	renameSnapshot(String snapshotDir, String snapshotOldName, String snapshotNewName)	Rename a snapshot.
long	renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)	Deprecated. Use Token.renew instead.
boolean	renewLease()	Renew leases.
void	reportBadBlocks(LocatedBlock[] blocks)	Report corrupt blocks that were discovered by the client.
void	satisfyStoragePolicy(String src)	Satisfy storage policy for an existing file/directory.
void	setAcl(String src, List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
void	setBalancerBandwidth(long bandwidth)	Requests the namenode to tell all datanodes to use a new, non-persistent bandwidth value for dfs.datanode.balance.bandwidthPerSec.
static void	setDisabledStopDeadNodeDetectorThreadForTest(boolean disabledStopDeadNodeDetectorThreadForTest)
void	setErasureCodingPolicy(String src, String ecPolicyName)
void	setKeyProvider(org.apache.hadoop.crypto.key.KeyProvider provider)
void	setOwner(String src, String username, String groupname)	Set file or directory owner.
void	setPermission(String src, org.apache.hadoop.fs.permission.FsPermission permission)	Set permissions to a file or directory.
boolean	setReplication(String src, short replication)	Set replication for an existing file.
boolean	setSafeMode(HdfsConstants.SafeModeAction action)	Enter, leave or get safe mode.
boolean	setSafeMode(HdfsConstants.SafeModeAction action, boolean isChecked)	Enter, leave or get safe mode.
void	setStoragePolicy(String src, String policyName)	Set storage policy for an existing file/directory
void	setTimes(String src, long mtime, long atime)	set the modification and access time of a file.
void	setXAttr(String src, String name, byte[] value, EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flag)
DatanodeInfo[]	slowDatanodeReport()
String	toString()
boolean	truncate(String src, long newLength)	Truncate a file to an indicated size See ClientProtocol.truncate(java.lang.String, long, java.lang.String).
void	unsetErasureCodingPolicy(String src)
void	unsetStoragePolicy(String src)	Unset storage policy set for a given file/directory.
boolean	upgradeStatus()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, getAdditionalTokenIssuers

Field Details

LOG

public static final org.slf4j.Logger LOG

Constructor Details

DFSClient

@Deprecated
public DFSClient(org.apache.hadoop.conf.Configuration conf)
          throws IOException

Deprecated.

Deprecated at 0.21

Same as this(NameNode.getNNAddress(conf), conf);

Throws:

IOException

See Also:

• DFSClient(InetSocketAddress, Configuration)

DFSClient

public DFSClient(InetSocketAddress address,
 org.apache.hadoop.conf.Configuration conf)
          throws IOException

Throws:

IOException

DFSClient

public DFSClient(URI nameNodeUri,
 org.apache.hadoop.conf.Configuration conf)
          throws IOException

Same as this(nameNodeUri, conf, null);

Throws:

IOException

See Also:

• DFSClient(URI, Configuration, FileSystem.Statistics)

DFSClient

public DFSClient(URI nameNodeUri,
 org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.fs.FileSystem.Statistics stats)
          throws IOException

Same as this(nameNodeUri, null, conf, stats);

Throws:

IOException

See Also:

• DFSClient(URI, ClientProtocol, Configuration, FileSystem.Statistics)

DFSClient

@VisibleForTesting
public DFSClient(URI nameNodeUri,
 ClientProtocol rpcNamenode,
 org.apache.hadoop.conf.Configuration conf,
 org.apache.hadoop.fs.FileSystem.Statistics stats)
          throws IOException

Create a new DFSClient connected to the given nameNodeUri or rpcNamenode. If HA is enabled and a positive value is set for HdfsClientConfigKeys.DFS_CLIENT_TEST_DROP_NAMENODE_RESPONSE_NUM_KEY in the configuration, the DFSClient will use LossyRetryInvocationHandler as its RetryInvocationHandler. Otherwise one of nameNodeUri or rpcNamenode must be null.

Throws:

IOException

Method Details

setDisabledStopDeadNodeDetectorThreadForTest

@VisibleForTesting
public static void setDisabledStopDeadNodeDetectorThreadForTest(boolean disabledStopDeadNodeDetectorThreadForTest)

getConf

public DfsClientConf getConf()

getClientName

@VisibleForTesting
public String getClientName()

getLeaseRenewer

public LeaseRenewer getLeaseRenewer()

Return the lease renewer instance. The renewer thread won't start until the first output stream is created. The same instance will be returned until all output streams are closed.

putFileBeingWritten

public void putFileBeingWritten(String key,
 DFSOutputStream out)

Put a file. Only called from LeaseRenewer, where proper locking is enforced to consistently update its local dfsclients array and client's filesBeingWritten map.

removeFileBeingWritten

public void removeFileBeingWritten(String key)

Remove a file. Only called from LeaseRenewer.

isFilesBeingWrittenEmpty

public boolean isFilesBeingWrittenEmpty()

Is file-being-written map empty?

isClientRunning

public boolean isClientRunning()

Returns:

true if the client is running

getNumOfFilesBeingWritten

@VisibleForTesting
public int getNumOfFilesBeingWritten()

renewLease

public boolean renewLease()
                   throws IOException

Renew leases.

Returns:

true if lease was renewed. May return false if this client has been closed or has no files open.

Throws:

IOException

closeAllFilesBeingWritten

public void closeAllFilesBeingWritten(boolean abort)

Close/abort all files being written.

close

public void close()
           throws IOException

Close the file system, abandoning all of the leases and files being created and close connections to the namenode.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException

closeOutputStreams

public void closeOutputStreams(boolean abort)

Close all open streams, abandoning all of the leases and files being created.

Parameters:

abort - whether streams should be gracefully closed

getBlockSize

public long getBlockSize(String f)
                  throws IOException

Throws:

IOException

See Also:

• ClientProtocol.getPreferredBlockSize(String)

getServerDefaults

public org.apache.hadoop.fs.FsServerDefaults getServerDefaults()
                                                        throws IOException

Get server default values for a number of configuration params.

Throws:

IOException

See Also:

• ClientProtocol.getServerDefaults()

getCanonicalServiceName

@LimitedPrivate("HDFS")
public String getCanonicalServiceName()

Get a canonical token service name for this client's tokens. Null should be returned if the client is not using tokens.

Specified by:

getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Returns:

the token service for the client

getDelegationToken

public org.apache.hadoop.security.token.Token<?> getDelegationToken(String renewer)
                                                             throws IOException

Specified by:

getDelegationToken in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Throws:

IOException

getDelegationToken

public org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> getDelegationToken(org.apache.hadoop.io.Text renewer)
                                                                                     throws IOException

Throws:

IOException

See Also:

• ClientProtocol.getDelegationToken(Text)

renewDelegationToken

@Deprecated
public long renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                          throws IOException

Deprecated.

Use Token.renew instead.

Renew a delegation token

Parameters:

token - the token to renew

Returns:

the new expiration time

Throws:

IOException

cancelDelegationToken

@Deprecated
public void cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                           throws IOException

Deprecated.

Use Token.cancel instead.

Cancel a delegation token

Parameters:

token - the token to cancel

Throws:

IOException

reportBadBlocks

public void reportBadBlocks(LocatedBlock[] blocks)
                     throws IOException

Report corrupt blocks that were discovered by the client.

Throws:

IOException

See Also:

• ClientProtocol.reportBadBlocks(LocatedBlock[])

getRefreshReadBlkLocationsInterval

public long getRefreshReadBlkLocationsInterval()

getLocatedBlocks

public LocatedBlocks getLocatedBlocks(String src,
 long start)
                               throws IOException

Get locations of the blocks of the specified file `src` from offset `start` within the prefetch size which is related to parameter `dfs.client.read.prefetch.size`. DataNode locations for each block are sorted by the proximity to the client. Please note that the prefetch size is not equal file length generally.

Parameters:

src - the file path.

start - starting offset.

Returns:

LocatedBlocks

Throws:

IOException

getLocatedBlocks

@VisibleForTesting
public LocatedBlocks getLocatedBlocks(String src,
 long start,
 long length)
                               throws IOException

This is just a wrapper around callGetBlockLocations, but non-static so that we can stub it out for tests.

Throws:

IOException

getBlockLocations

public org.apache.hadoop.fs.BlockLocation[] getBlockLocations(String src,
 long start,
 long length)
                                                       throws IOException

Get block location info about file getBlockLocations() returns a list of hostnames that store data for a specific file region. It returns a set of hostnames for every block within the indicated region. This function is very useful when writing code that considers data-placement when performing operations. For example, the MapReduce system tries to schedule tasks on the same machines as the data-block the task processes. Please refer to FileSystem.getFileBlockLocations(FileStatus, long, long) for more details.

Throws:

IOException

createWrappedInputStream

public HdfsDataInputStream createWrappedInputStream(DFSInputStream dfsis)
                                             throws IOException

Wraps the stream in a CryptoInputStream if the underlying file is encrypted.

Throws:

IOException

createWrappedOutputStream

public HdfsDataOutputStream createWrappedOutputStream(DFSOutputStream dfsos,
 org.apache.hadoop.fs.FileSystem.Statistics statistics)
                                               throws IOException

Wraps the stream in a CryptoOutputStream if the underlying file is encrypted.

Throws:

IOException

createWrappedOutputStream

public HdfsDataOutputStream createWrappedOutputStream(DFSOutputStream dfsos,
 org.apache.hadoop.fs.FileSystem.Statistics statistics,
 long startPos)
                                               throws IOException

Wraps the stream in a CryptoOutputStream if the underlying file is encrypted.

Throws:

IOException

open

public DFSInputStream open(String src)
                    throws IOException

Throws:

IOException

open

@Deprecated
public DFSInputStream open(String src,
 int buffersize,
 boolean verifyChecksum,
 org.apache.hadoop.fs.FileSystem.Statistics stats)
                    throws IOException

Deprecated.

Use open(String, int, boolean) instead.

Create an input stream that obtains a nodelist from the namenode, and then reads from all the right places. Creates inner subclass of InputStream that does the right out-of-band work.

Throws:

IOException

open

public DFSInputStream open(String src,
 int buffersize,
 boolean verifyChecksum)
                    throws IOException

Create an input stream that obtains a nodelist from the namenode, and then reads from all the right places. Creates inner subclass of InputStream that does the right out-of-band work.

Throws:

IOException

open

public DFSInputStream open(HdfsPathHandle fd,
 int buffersize,
 boolean verifyChecksum)
                    throws IOException

Create an input stream from the HdfsPathHandle if the constraints encoded from DistributedFileSystem.createPathHandle(FileStatus, Options.HandleOpt...) are satisfied. Note that HDFS does not ensure that these constraints remain invariant for the life of the stream. It only checks that they still held when the stream was opened.

Parameters:

fd - Handle to an entity in HDFS, with constraints

buffersize - ignored

verifyChecksum - Verify checksums before returning data to client

Returns:

Data from the referent of the HdfsPathHandle.

Throws:

IOException - On I/O error

getNamenode

public ClientProtocol getNamenode()

Get the namenode associated with this DFSClient object

Returns:

the namenode associated with this DFSClient object

create

public OutputStream create(String src,
 boolean overwrite)
                    throws IOException

Call create(String, boolean, short, long, Progressable) with default replication and blockSize and null progress.

Throws:

IOException

create

public OutputStream create(String src,
 boolean overwrite,
 org.apache.hadoop.util.Progressable progress)
                    throws IOException

Call create(String, boolean, short, long, Progressable) with default replication and blockSize.

Throws:

IOException

create

public OutputStream create(String src,
 boolean overwrite,
 short replication,
 long blockSize)
                    throws IOException

Call create(String, boolean, short, long, Progressable) with null progress.

Throws:

IOException

create

public OutputStream create(String src,
 boolean overwrite,
 short replication,
 long blockSize,
 org.apache.hadoop.util.Progressable progress)
                    throws IOException

Call create(String, boolean, short, long, Progressable, int) with default bufferSize.

Throws:

IOException

create

public OutputStream create(String src,
 boolean overwrite,
 short replication,
 long blockSize,
 org.apache.hadoop.util.Progressable progress,
 int buffersize)
                    throws IOException

Call create(String, FsPermission, EnumSet, short, long, Progressable, int, ChecksumOpt) with default permission FsPermission.getFileDefault().

Parameters:

src - File name

overwrite - overwrite an existing file if true

replication - replication factor for the file

blockSize - maximum block size

progress - interface for reporting client progress

buffersize - underlying buffersize

Returns:

output stream

Throws:

IOException

create

public DFSOutputStream create(String src,
 org.apache.hadoop.fs.permission.FsPermission permission,
 EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
 short replication,
 long blockSize,
 org.apache.hadoop.util.Progressable progress,
 int buffersize,
 org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)
                       throws IOException

Call create(String, FsPermission, EnumSet, boolean, short, long, Progressable, int, ChecksumOpt) with createParent set to true.

Throws:

IOException

create

public DFSOutputStream create(String src,
 org.apache.hadoop.fs.permission.FsPermission permission,
 EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
 boolean createParent,
 short replication,
 long blockSize,
 org.apache.hadoop.util.Progressable progress,
 int buffersize,
 org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)
                       throws IOException

Create a new dfs file with the specified block replication with write-progress reporting and return an output stream for writing into the file.

Parameters:

src - File name

permission - The permission of the directory being created. If null, use default permission FsPermission.getFileDefault()

flag - indicates create a new file or create/overwrite an existing file or append to an existing file

createParent - create missing parent directory if true

replication - block replication

blockSize - maximum block size

progress - interface for reporting client progress

buffersize - underlying buffer size

checksumOpt - checksum options

Returns:

output stream

Throws:

IOException

See Also:

• for detailed description of exceptions thrown

create

public DFSOutputStream create(String src,
 org.apache.hadoop.fs.permission.FsPermission permission,
 EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
 boolean createParent,
 short replication,
 long blockSize,
 org.apache.hadoop.util.Progressable progress,
 int buffersize,
 org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt,
 InetSocketAddress[] favoredNodes)
                       throws IOException

Same as create(String, FsPermission, EnumSet, boolean, short, long, Progressable, int, ChecksumOpt) with the addition of favoredNodes that is a hint to where the namenode should place the file blocks. The favored nodes hint is not persisted in HDFS. Hence it may be honored at the creation time only. HDFS could move the blocks during balancing or replication, to move the blocks from favored nodes. A value of null means no favored nodes for this create

Throws:

IOException

create

public DFSOutputStream create(String src,
 org.apache.hadoop.fs.permission.FsPermission permission,
 EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
 boolean createParent,
 short replication,
 long blockSize,
 org.apache.hadoop.util.Progressable progress,
 int buffersize,
 org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt,
 InetSocketAddress[] favoredNodes,
 String ecPolicyName)
                       throws IOException

Same as create(String, FsPermission, EnumSet, boolean, short, long, Progressable, int, ChecksumOpt, InetSocketAddress[]) with the addition of ecPolicyName that is used to specify a specific erasure coding policy instead of inheriting any policy from this new file's parent directory. This policy will be persisted in HDFS. A value of null means inheriting parent groups' whatever policy.

Throws:

IOException

create

public DFSOutputStream create(String src,
 org.apache.hadoop.fs.permission.FsPermission permission,
 EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
 boolean createParent,
 short replication,
 long blockSize,
 org.apache.hadoop.util.Progressable progress,
 int buffersize,
 org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt,
 InetSocketAddress[] favoredNodes,
 String ecPolicyName,
 String storagePolicy)
                       throws IOException

Same as create(String, FsPermission, EnumSet, boolean, short, long, Progressable, int, ChecksumOpt, InetSocketAddress[], String) with the storagePolicy that is used to specify a specific storage policy instead of inheriting any policy from this new file's parent directory. This policy will be persisted in HDFS. A value of null means inheriting parent groups' whatever policy.

Throws:

IOException

primitiveCreate

public DFSOutputStream primitiveCreate(String src,
 org.apache.hadoop.fs.permission.FsPermission absPermission,
 EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
 boolean createParent,
 short replication,
 long blockSize,
 org.apache.hadoop.util.Progressable progress,
 int buffersize,
 org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)
                                throws IOException

Same as {create(String, FsPermission, EnumSet, short, long, Progressable, int, ChecksumOpt) except that the permission is absolute (ie has already been masked with umask.

Throws:

IOException

createSymlink

public void createSymlink(String target,
 String link,
 boolean createParent)
                   throws IOException

Creates a symbolic link.

Throws:

IOException

See Also:

• ClientProtocol.createSymlink(String, String,FsPermission, boolean)

getLinkTarget

public String getLinkTarget(String path)
                     throws IOException

Resolve the *first* symlink, if any, in the path.

Throws:

IOException

See Also:

• ClientProtocol.getLinkTarget(String)

append

public HdfsDataOutputStream append(String src,
 int buffersize,
 EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
 org.apache.hadoop.util.Progressable progress,
 org.apache.hadoop.fs.FileSystem.Statistics statistics)
                            throws IOException

Append to an existing HDFS file.

Parameters:

src - file name

buffersize - buffer size

flag - indicates whether to append data to a new block instead of the last block

progress - for reporting write-progress; null is acceptable.

statistics - file system statistics; null is acceptable.

Returns:

an output stream for writing into the file

Throws:

IOException

See Also:

• ClientProtocol.append(String, String, EnumSetWritable)

append

public HdfsDataOutputStream append(String src,
 int buffersize,
 EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
 org.apache.hadoop.util.Progressable progress,
 org.apache.hadoop.fs.FileSystem.Statistics statistics,
 InetSocketAddress[] favoredNodes)
                            throws IOException

Append to an existing HDFS file.

Parameters:

src - file name

buffersize - buffer size

flag - indicates whether to append data to a new block instead of the last block

progress - for reporting write-progress; null is acceptable.

statistics - file system statistics; null is acceptable.

favoredNodes - FavoredNodes for new blocks

Returns:

an output stream for writing into the file

Throws:

IOException

See Also:

• ClientProtocol.append(String, String, EnumSetWritable)

setReplication

public boolean setReplication(String src,
 short replication)
                       throws IOException

Set replication for an existing file.

Parameters:

src - file name

replication - replication to set the file to

Throws:

IOException

See Also:

• ClientProtocol.setReplication(String, short)

setStoragePolicy

public void setStoragePolicy(String src,
 String policyName)
                      throws IOException

Set storage policy for an existing file/directory

Parameters:

src - file/directory name

policyName - name of the storage policy

Throws:

IOException

unsetStoragePolicy

public void unsetStoragePolicy(String src)
                        throws IOException

Unset storage policy set for a given file/directory.

Parameters:

src - file/directory name

Throws:

IOException

getStoragePolicy

public BlockStoragePolicy getStoragePolicy(String path)
                                    throws IOException

Parameters:

path - file/directory name

Returns:

Get the storage policy for specified path

Throws:

IOException

getStoragePolicies

public BlockStoragePolicy[] getStoragePolicies()
                                        throws IOException

Returns:

All the existing storage policies

Throws:

IOException

rename

@Deprecated
public boolean rename(String src,
 String dst)
               throws IOException

Deprecated.

Use rename(String, String, Options.Rename...) instead.

Rename file or directory.

Throws:

IOException

See Also:

• ClientProtocol.rename(String, String)

concat

public void concat(String trg,
 String[] srcs)
            throws IOException

Move blocks from src to trg and delete src See ClientProtocol.concat(java.lang.String, java.lang.String[]).

Throws:

IOException

rename

public void rename(String src,
 String dst,
 org.apache.hadoop.fs.Options.Rename... options)
            throws IOException

Rename file or directory.

Throws:

IOException

See Also:

• ClientProtocol.rename2(String, String, Options.Rename...)

truncate

public boolean truncate(String src,
 long newLength)
                 throws IOException

Truncate a file to an indicated size See ClientProtocol.truncate(java.lang.String, long, java.lang.String).

Throws:

IOException

delete

@Deprecated
public boolean delete(String src)
               throws IOException

Deprecated.

Delete file or directory. See ClientProtocol.delete(String, boolean).

Throws:

IOException

delete

public boolean delete(String src,
 boolean recursive)
               throws IOException

delete file or directory. delete contents of the directory if non empty and recursive set to true

Throws:

IOException

See Also:

• ClientProtocol.delete(String, boolean)

exists

public boolean exists(String src)
               throws IOException

Implemented using getFileInfo(src)

Throws:

IOException

listPaths

public DirectoryListing listPaths(String src,
 byte[] startAfter)
                           throws IOException

Get a partial listing of the indicated directory No block locations need to be fetched

Throws:

IOException

listPaths

public DirectoryListing listPaths(String src,
 byte[] startAfter,
 boolean needLocation)
                           throws IOException

Get a partial listing of the indicated directory Recommend to use HdfsFileStatus.EMPTY_NAME as startAfter if the application wants to fetch a listing starting from the first entry in the directory

Throws:

IOException

See Also:

• ClientProtocol.getListing(String, byte[], boolean)

batchedListPaths

public BatchedDirectoryListing batchedListPaths(String[] srcs,
 byte[] startAfter,
 boolean needLocation)
                                         throws IOException

Get a batched listing for the indicated directories

Throws:

IOException

See Also:

• ClientProtocol.getBatchedListing(String[], byte[], boolean)

getFileInfo

public HdfsFileStatus getFileInfo(String src)
                           throws IOException

Get the file info for a specific file or directory.

Parameters:

src - The string representation of the path to the file

Returns:

object containing information regarding the file or null if file not found

Throws:

IOException

See Also:

• for description of exceptions

getLocatedFileInfo

public HdfsLocatedFileStatus getLocatedFileInfo(String src,
 boolean needBlockToken)
                                         throws IOException

Get the file info for a specific file or directory.

Parameters:

src - The string representation of the path to the file

needBlockToken - Include block tokens in LocatedBlocks. When block tokens are included, this call is a superset of getBlockLocations(String, long).

Returns:

object containing information regarding the file or null if file not found

Throws:

IOException

See Also:

• open(HdfsPathHandle, int, boolean)
• for description of exceptions

isFileClosed

public boolean isFileClosed(String src)
                     throws IOException

Close status of a file

Returns:

true if file is already closed

Throws:

IOException

getFileLinkInfo

public HdfsFileStatus getFileLinkInfo(String src)
                               throws IOException

Get the file info for a specific file or directory. If src refers to a symlink then the FileStatus of the link is returned.

Parameters:

src - path to a file or directory. For description of exceptions thrown

Throws:

IOException

See Also:

• ClientProtocol.getFileLinkInfo(String)

clearDataEncryptionKey

@Private
public void clearDataEncryptionKey()

newDataEncryptionKey

public DataEncryptionKey newDataEncryptionKey()
                                       throws IOException

Description copied from interface: DataEncryptionKeyFactory

Creates a new DataEncryptionKey.

Specified by:

newDataEncryptionKey in interface DataEncryptionKeyFactory

Returns:

DataEncryptionKey newly created

Throws:

IOException - for any error

getEncryptionKey

@VisibleForTesting
public DataEncryptionKey getEncryptionKey()

getFileChecksumWithCombineMode

public org.apache.hadoop.fs.FileChecksum getFileChecksumWithCombineMode(String src,
 long length)
                                                                 throws IOException

Get the checksum of the whole file or a range of the file. Note that the range always starts from the beginning of the file. The file can be in replicated form, or striped mode. Depending on the dfs.checksum.combine.mode, checksums may or may not be comparable between different block layout forms.

Parameters:

src - The file path

length - the length of the range, i.e., the range is [0, length]

Returns:

The checksum

Throws:

IOException

See Also:

• DistributedFileSystem.getFileChecksum(Path)

getFileChecksum

public org.apache.hadoop.fs.MD5MD5CRC32FileChecksum getFileChecksum(String src,
 long length)
                                                             throws IOException

Get the checksum of the whole file or a range of the file. Note that the range always starts from the beginning of the file. The file can be in replicated form, or striped mode. It can be used to checksum and compare two replicated files, or two striped files, but not applicable for two files of different block layout forms.

Parameters:

src - The file path

length - the length of the range, i.e., the range is [0, length]

Returns:

The checksum

Throws:

IOException

See Also:

• DistributedFileSystem.getFileChecksum(Path)

getBlockLocations

protected LocatedBlocks getBlockLocations(String src,
 long length)
                                   throws IOException

Throws:

IOException

connectToDN

protected IOStreamPair connectToDN(DatanodeInfo dn,
 int timeout,
 org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken)
                            throws IOException

Throws:

IOException

inferChecksumTypeByReading

protected org.apache.hadoop.util.DataChecksum.Type inferChecksumTypeByReading(LocatedBlock lb,
 DatanodeInfo dn)
                                                                       throws IOException

Infer the checksum type for a replica by sending an OP_READ_BLOCK for the first byte of that replica. This is used for compatibility with older HDFS versions which did not include the checksum type in OpBlockChecksumResponseProto.

Parameters:

lb - the located block

dn - the connected datanode

Returns:

the inferred checksum type

Throws:

IOException - if an error occurs

setPermission

public void setPermission(String src,
 org.apache.hadoop.fs.permission.FsPermission permission)
                   throws IOException

Set permissions to a file or directory.

Parameters:

src - path name.

permission - permission to set to

Throws:

IOException

See Also:

• ClientProtocol.setPermission(String, FsPermission)

setOwner

public void setOwner(String src,
 String username,
 String groupname)
              throws IOException

Set file or directory owner.

Parameters:

src - path name.

username - user id.

groupname - user group.

Throws:

IOException

See Also:

• ClientProtocol.setOwner(String, String, String)

getDiskStatus

public org.apache.hadoop.fs.FsStatus getDiskStatus()
                                            throws IOException

Throws:

IOException

See Also:

• ClientProtocol.getStats()

getStateAtIndex

public static long getStateAtIndex(long[] states,
 int index)

getMissingBlocksCount

public long getMissingBlocksCount()
                           throws IOException

Returns count of blocks with no good replicas left. Normally should be zero.

Throws:

IOException

getMissingReplOneBlocksCount

public long getMissingReplOneBlocksCount()
                                  throws IOException

Returns count of blocks with replication factor 1 and have lost the only replica.

Throws:

IOException

getPendingDeletionBlocksCount

public long getPendingDeletionBlocksCount()
                                   throws IOException

Returns count of blocks pending on deletion.

Throws:

IOException

getLowRedundancyBlocksCount

public long getLowRedundancyBlocksCount()
                                 throws IOException

Returns aggregated count of blocks with less redundancy.

Throws:

IOException

getCorruptBlocksCount

public long getCorruptBlocksCount()
                           throws IOException

Returns count of blocks with at least one replica marked corrupt.

Throws:

IOException

getBytesInFutureBlocks

public long getBytesInFutureBlocks()
                            throws IOException

Returns number of bytes that reside in Blocks with future generation stamps.

Returns:

Bytes in Blocks with future generation stamps.

Throws:

IOException

listCorruptFileBlocks

public CorruptFileBlocks listCorruptFileBlocks(String path,
 String cookie)
                                        throws IOException

Returns:

a list in which each entry describes a corrupt file/block

Throws:

IOException

datanodeReport

public DatanodeInfo[] datanodeReport(HdfsConstants.DatanodeReportType type)
                              throws IOException

Throws:

IOException

getDatanodeStorageReport

public DatanodeStorageReport[] getDatanodeStorageReport(HdfsConstants.DatanodeReportType type)
                                                 throws IOException

Throws:

IOException

setSafeMode

public boolean setSafeMode(HdfsConstants.SafeModeAction action)
                    throws IOException

Enter, leave or get safe mode.

Throws:

IOException

See Also:

• ClientProtocol.setSafeMode(HdfsConstants.SafeModeAction,boolean)

setSafeMode

public boolean setSafeMode(HdfsConstants.SafeModeAction action,
 boolean isChecked)
                    throws IOException

Enter, leave or get safe mode.

Parameters:

action - One of SafeModeAction.GET, SafeModeAction.ENTER and SafeModeActiob.LEAVE

isChecked - If true, then check only active namenode's safemode status, else check first namenode's status.

Throws:

IOException

See Also:

• ClientProtocol.setSafeMode(HdfsConstants.SafeModeAction, boolean)

createSnapshot

public String createSnapshot(String snapshotRoot,
 String snapshotName)
                      throws IOException

Create one snapshot.

Parameters:

snapshotRoot - The directory where the snapshot is to be taken

snapshotName - Name of the snapshot

Returns:

the snapshot path.

Throws:

IOException

See Also:

• ClientProtocol.createSnapshot(String, String)

deleteSnapshot

public void deleteSnapshot(String snapshotRoot,
 String snapshotName)
                    throws IOException

Delete a snapshot of a snapshottable directory.

Parameters:

snapshotRoot - The snapshottable directory that the to-be-deleted snapshot belongs to

snapshotName - The name of the to-be-deleted snapshot

Throws:

IOException

See Also:

• ClientProtocol.deleteSnapshot(String, String)

renameSnapshot

public void renameSnapshot(String snapshotDir,
 String snapshotOldName,
 String snapshotNewName)
                    throws IOException

Rename a snapshot.

Parameters:

snapshotDir - The directory path where the snapshot was taken

snapshotOldName - Old name of the snapshot

snapshotNewName - New name of the snapshot

Throws:

IOException

See Also:

• ClientProtocol.renameSnapshot(String, String, String)

getSnapshottableDirListing

public SnapshottableDirectoryStatus[] getSnapshottableDirListing()
                                                          throws IOException

Get all the current snapshottable directories.

Returns:

All the current snapshottable directories

Throws:

IOException

See Also:

• ClientProtocol.getSnapshottableDirListing()

getSnapshotListing

public SnapshotStatus[] getSnapshotListing(String snapshotRoot)
                                    throws IOException

Get listing of all the snapshots for a snapshottable directory.

Returns:

Information about all the snapshots for a snapshottable directory

Throws:

IOException - If an I/O error occurred

See Also:

• ClientProtocol.getSnapshotListing(String)

allowSnapshot

public void allowSnapshot(String snapshotRoot)
                   throws IOException

Allow snapshot on a directory.

Throws:

IOException

See Also:

• ClientProtocol.allowSnapshot(String snapshotRoot)

disallowSnapshot

public void disallowSnapshot(String snapshotRoot)
                      throws IOException

Disallow snapshot on a directory.

Throws:

IOException

See Also:

• ClientProtocol.disallowSnapshot(String snapshotRoot)

getSnapshotDiffReport

public SnapshotDiffReport getSnapshotDiffReport(String snapshotDir,
 String fromSnapshot,
 String toSnapshot)
                                         throws IOException

Get the difference between two snapshots, or between a snapshot and the current tree of a directory.

Throws:

IOException

See Also:

• ClientProtocol.getSnapshotDiffReport(java.lang.String, java.lang.String, java.lang.String)

getSnapshotDiffReportListing

public SnapshotDiffReportListing getSnapshotDiffReportListing(String snapshotDir,
 String fromSnapshot,
 String toSnapshot,
 byte[] startPath,
 int index)
                                                       throws IOException

Get the difference between two snapshots of a directory iteratively.

Throws:

IOException

See Also:

• ClientProtocol.getSnapshotDiffReportListing(java.lang.String, java.lang.String, java.lang.String, byte[], int)

addCacheDirective

public long addCacheDirective(CacheDirectiveInfo info,
 EnumSet<CacheFlag> flags)
                       throws IOException

Throws:

IOException

modifyCacheDirective

public void modifyCacheDirective(CacheDirectiveInfo info,
 EnumSet<CacheFlag> flags)
                          throws IOException

Throws:

IOException

removeCacheDirective

public void removeCacheDirective(long id)
                          throws IOException

Throws:

IOException

listCacheDirectives

public org.apache.hadoop.fs.RemoteIterator<CacheDirectiveEntry> listCacheDirectives(CacheDirectiveInfo filter)
                                                                             throws IOException

Throws:

IOException

addCachePool

public void addCachePool(CachePoolInfo info)
                  throws IOException

Throws:

IOException

modifyCachePool

public void modifyCachePool(CachePoolInfo info)
                     throws IOException

Throws:

IOException

removeCachePool

public void removeCachePool(String poolName)
                     throws IOException

Throws:

IOException

listCachePools

public org.apache.hadoop.fs.RemoteIterator<CachePoolEntry> listCachePools()
                                                                   throws IOException

Throws:

IOException

refreshNodes

public void refreshNodes()
                  throws IOException

Refresh the hosts and exclude files. (Rereads them.) See ClientProtocol.refreshNodes() for more details.

Throws:

IOException

See Also:

• ClientProtocol.refreshNodes()

metaSave

public void metaSave(String pathname)
              throws IOException

Dumps DFS data structures into specified file.

Throws:

IOException

See Also:

• ClientProtocol.metaSave(String)

setBalancerBandwidth

public void setBalancerBandwidth(long bandwidth)
                          throws IOException

Requests the namenode to tell all datanodes to use a new, non-persistent bandwidth value for dfs.datanode.balance.bandwidthPerSec. See ClientProtocol.setBalancerBandwidth(long) for more details.

Throws:

IOException

See Also:

• ClientProtocol.setBalancerBandwidth(long)

finalizeUpgrade

public void finalizeUpgrade()
                     throws IOException

Throws:

IOException

See Also:

• ClientProtocol.finalizeUpgrade()

upgradeStatus

public boolean upgradeStatus()
                      throws IOException

Throws:

IOException

See Also:

• ClientProtocol.upgradeStatus()

mkdirs

@Deprecated
public boolean mkdirs(String src)
               throws IOException

Deprecated.

Throws:

IOException

mkdirs

public boolean mkdirs(String src,
 org.apache.hadoop.fs.permission.FsPermission permission,
 boolean createParent)
               throws IOException

Create a directory (or hierarchy of directories) with the given name and permission.

Parameters:

src - The path of the directory being created

permission - The permission of the directory being created. If permission == null, use FsPermission.getDirDefault().

createParent - create missing parent directory if true

Returns:

True if the operation success.

Throws:

IOException

See Also:

• ClientProtocol.mkdirs(String, FsPermission, boolean)

primitiveMkdir

public boolean primitiveMkdir(String src,
 org.apache.hadoop.fs.permission.FsPermission absPermission)
                       throws IOException

Same {mkdirs(String, FsPermission, boolean) except that the permissions has already been masked against umask.

Throws:

IOException

primitiveMkdir

public boolean primitiveMkdir(String src,
 org.apache.hadoop.fs.permission.FsPermission absPermission,
 boolean createParent)
                       throws IOException

Same {mkdirs(String, FsPermission, boolean) except that the permissions has already been masked against umask.

Throws:

IOException

setTimes

public void setTimes(String src,
 long mtime,
 long atime)
              throws IOException

set the modification and access time of a file.

Throws:

IOException

See Also:

• ClientProtocol.setTimes(String, long, long)

toString

public String toString()

Overrides:

toString in class Object

getDefaultReadCachingStrategy

public CachingStrategy getDefaultReadCachingStrategy()

getDefaultWriteCachingStrategy

public CachingStrategy getDefaultWriteCachingStrategy()

getClientContext

public ClientContext getClientContext()

modifyAclEntries

public void modifyAclEntries(String src,
 List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
                      throws IOException

Throws:

IOException

removeAclEntries

public void removeAclEntries(String src,
 List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
                      throws IOException

Throws:

IOException

removeDefaultAcl

public void removeDefaultAcl(String src)
                      throws IOException

Throws:

IOException

removeAcl

public void removeAcl(String src)
               throws IOException

Throws:

IOException

setAcl

public void setAcl(String src,
 List<org.apache.hadoop.fs.permission.AclEntry> aclSpec)
            throws IOException

Throws:

IOException

getAclStatus

public org.apache.hadoop.fs.permission.AclStatus getAclStatus(String src)
                                                       throws IOException

Throws:

IOException

createEncryptionZone

public void createEncryptionZone(String src,
 String keyName)
                          throws IOException

Throws:

IOException

getEZForPath

public EncryptionZone getEZForPath(String src)
                            throws IOException

Throws:

IOException

listEncryptionZones

public org.apache.hadoop.fs.RemoteIterator<EncryptionZone> listEncryptionZones()
                                                                        throws IOException

Throws:

IOException

reencryptEncryptionZone

public void reencryptEncryptionZone(String zone,
 HdfsConstants.ReencryptAction action)
                             throws IOException

Throws:

IOException

listReencryptionStatus

public org.apache.hadoop.fs.RemoteIterator<ZoneReencryptionStatus> listReencryptionStatus()
                                                                                   throws IOException

Throws:

IOException

setErasureCodingPolicy

public void setErasureCodingPolicy(String src,
 String ecPolicyName)
                            throws IOException

Throws:

IOException

unsetErasureCodingPolicy

public void unsetErasureCodingPolicy(String src)
                              throws IOException

Throws:

IOException

getECTopologyResultForPolicies

public ECTopologyVerifierResult getECTopologyResultForPolicies(String... policyNames)
                                                        throws IOException

Throws:

IOException

setXAttr

public void setXAttr(String src,
 String name,
 byte[] value,
 EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flag)
              throws IOException

Throws:

IOException

getXAttr

public byte[] getXAttr(String src,
 String name)
                throws IOException

Throws:

IOException

getXAttrs

public Map<String,byte[]> getXAttrs(String src)
                             throws IOException

Throws:

IOException

getXAttrs

public Map<String,byte[]> getXAttrs(String src,
 List<String> names)
                             throws IOException

Throws:

IOException

listXAttrs

public List<String> listXAttrs(String src)
                        throws IOException

Throws:

IOException

removeXAttr

public void removeXAttr(String src,
 String name)
                 throws IOException

Throws:

IOException

checkAccess

public void checkAccess(String src,
 org.apache.hadoop.fs.permission.FsAction mode)
                 throws IOException

Throws:

IOException

getErasureCodingPolicies

public ErasureCodingPolicyInfo[] getErasureCodingPolicies()
                                                   throws IOException

Throws:

IOException

getErasureCodingCodecs

public Map<String,String> getErasureCodingCodecs()
                                          throws IOException

Throws:

IOException

addErasureCodingPolicies

public AddErasureCodingPolicyResponse[] addErasureCodingPolicies(ErasureCodingPolicy[] policies)
                                                          throws IOException

Throws:

IOException

removeErasureCodingPolicy

public void removeErasureCodingPolicy(String ecPolicyName)
                               throws IOException

Throws:

IOException

enableErasureCodingPolicy

public void enableErasureCodingPolicy(String ecPolicyName)
                               throws IOException

Throws:

IOException

disableErasureCodingPolicy

public void disableErasureCodingPolicy(String ecPolicyName)
                                throws IOException

Throws:

IOException

getInotifyEventStream

public DFSInotifyEventInputStream getInotifyEventStream()
                                                 throws IOException

Throws:

IOException

getInotifyEventStream

public DFSInotifyEventInputStream getInotifyEventStream(long lastReadTxid)
                                                 throws IOException

Throws:

IOException

newConnectedPeer

public Peer newConnectedPeer(InetSocketAddress addr,
 org.apache.hadoop.security.token.Token<BlockTokenIdentifier> blockToken,
 DatanodeID datanodeId)
                      throws IOException

Specified by:

newConnectedPeer in interface RemotePeerFactory

Parameters:

addr - The address to connect to.

blockToken - Token used during optional SASL negotiation

datanodeId - ID of destination DataNode

Returns:

A new Peer connected to the address.

Throws:

IOException - If there was an error connecting or creating the remote socket, encrypted stream, etc.

getKeyProviderUri

public URI getKeyProviderUri()
                      throws IOException

Specified by:

getKeyProviderUri in interface org.apache.hadoop.crypto.key.KeyProviderTokenIssuer

Throws:

IOException

getKeyProvider

public org.apache.hadoop.crypto.key.KeyProvider getKeyProvider()
                                                        throws IOException

Specified by:

getKeyProvider in interface org.apache.hadoop.crypto.key.KeyProviderTokenIssuer

Throws:

IOException

setKeyProvider

@VisibleForTesting
public void setKeyProvider(org.apache.hadoop.crypto.key.KeyProvider provider)

getSaslDataTransferClient

public SaslDataTransferClient getSaslDataTransferClient()

Returns the SaslDataTransferClient configured for this DFSClient.

Returns:

SaslDataTransferClient configured for this DFSClient

getErasureCodingPolicy

public ErasureCodingPolicy getErasureCodingPolicy(String src)
                                           throws IOException

Get the erasure coding policy information for the specified path

Parameters:

src - path to get the information for

Returns:

Returns the policy information if file or directory on the path is erasure coded, null otherwise. Null will be returned if directory or file has REPLICATION policy.

Throws:

IOException

satisfyStoragePolicy

public void satisfyStoragePolicy(String src)
                          throws IOException

Satisfy storage policy for an existing file/directory.

Parameters:

src - file/directory name

Throws:

IOException

listOpenFiles

@Deprecated
public org.apache.hadoop.fs.RemoteIterator<OpenFileEntry> listOpenFiles()
                                                                 throws IOException

Deprecated.

Get a remote iterator to the open files list managed by NameNode.

Throws:

IOException

listOpenFiles

public org.apache.hadoop.fs.RemoteIterator<OpenFileEntry> listOpenFiles(String path)
                                                                 throws IOException

Get a remote iterator to the open files list by path, managed by NameNode.

Parameters:

path -

Throws:

IOException

listOpenFiles

public org.apache.hadoop.fs.RemoteIterator<OpenFileEntry> listOpenFiles(EnumSet<OpenFilesIterator.OpenFilesType> openFilesTypes)
                                                                 throws IOException

Get a remote iterator to the open files list by type, managed by NameNode.

Parameters:

openFilesTypes -

Throws:

IOException

listOpenFiles

public org.apache.hadoop.fs.RemoteIterator<OpenFileEntry> listOpenFiles(EnumSet<OpenFilesIterator.OpenFilesType> openFilesTypes,
 String path)
                                                                 throws IOException

Get a remote iterator to the open files list by type and path, managed by NameNode.

Parameters:

openFilesTypes -

path -

Throws:

IOException

msync

public void msync()
           throws IOException

A blocking call to wait for Observer NameNode state ID to reach to the current client state ID. Current client state ID is given by the client alignment context. An assumption is that client alignment context has the state ID set at this point. This is become ObserverReadProxyProvider sets up the initial state ID when it is being created.

Throws:

IOException

getHAServiceState

@VisibleForTesting
public org.apache.hadoop.ha.HAServiceProtocol.HAServiceState getHAServiceState()
                                                                        throws IOException

An unblocking call to get the HA service state of NameNode.

Returns:

HA state of NameNode

Throws:

IOException

getDeadNodes

public ConcurrentHashMap<DatanodeInfo,DatanodeInfo> getDeadNodes(DFSInputStream dfsInputStream)

If deadNodeDetectionEnabled is true, return the dead nodes that detected by all the DFSInputStreams in the same client. Otherwise return the dead nodes that detected by given DFSInputStream.

isDeadNode

public boolean isDeadNode(DFSInputStream dfsInputStream,
 DatanodeInfo datanodeInfo)

If deadNodeDetectionEnabled is true, judgement based on whether this datanode is included or not in DeadNodeDetector. Otherwise judgment based given DFSInputStream.

addNodeToDeadNodeDetector

public void addNodeToDeadNodeDetector(DFSInputStream dfsInputStream,
 DatanodeInfo datanodeInfo)

Add given datanode in DeadNodeDetector.

removeNodeFromDeadNodeDetector

public void removeNodeFromDeadNodeDetector(DFSInputStream dfsInputStream,
 DatanodeInfo datanodeInfo)

Remove given datanode from DeadNodeDetector.

removeNodeFromDeadNodeDetector

public void removeNodeFromDeadNodeDetector(DFSInputStream dfsInputStream,
 LocatedBlocks locatedBlocks)

Remove datanodes that given block placed on from DeadNodeDetector.

getDeadNodeDetector

public DeadNodeDetector getDeadNodeDetector()

Obtain DeadNodeDetector of the current client.

getLocatedBlockRefresher

public LocatedBlocksRefresher getLocatedBlockRefresher()

Obtain LocatedBlocksRefresher of the current client.

addLocatedBlocksRefresh

public void addLocatedBlocksRefresh(DFSInputStream dfsInputStream)

Adds the DFSInputStream to the LocatedBlocksRefresher, so that the underlying LocatedBlocks is periodically refreshed.

removeLocatedBlocksRefresh

public void removeLocatedBlocksRefresh(DFSInputStream dfsInputStream)

Removes the DFSInputStream from the LocatedBlocksRefresher, so that the underlying LocatedBlocks is no longer periodically refreshed.

Parameters:

dfsInputStream -

slowDatanodeReport

public DatanodeInfo[] slowDatanodeReport()
                                  throws IOException

Throws:

IOException

getEnclosingRoot

public org.apache.hadoop.fs.Path getEnclosingRoot(String src)
                                           throws IOException

Throws:

IOException