Package org.apache.hadoop.fs

Class ChecksumFs

java.lang.Object

org.apache.hadoop.fs.AbstractFileSystem

org.apache.hadoop.fs.FilterFs

org.apache.hadoop.fs.ChecksumFs

All Implemented Interfaces:

PathCapabilities

Direct Known Subclasses:

LocalFs

@Private
@Evolving
public abstract class ChecksumFs
extends FilterFs

Abstract Checksumed Fs. It provide a basic implementation of a Checksumed Fs, which creates a checksum file for each raw file. It generates & verifies checksums at the client side.

Field Summary

Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

statistics

Constructor Summary

Constructors

Constructor	Description
ChecksumFs(AbstractFileSystem theFs)

Method Summary

Modifier and Type	Method	Description
FSDataOutputStream	createInternal(Path f, EnumSet<CreateFlag> createFlag, FsPermission absolutePermission, int bufferSize, short replication, long blockSize, Progressable progress, Options.ChecksumOpt checksumOpt, boolean createParent)	The specification of this method matches that of AbstractFileSystem.create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.
boolean	delete(Path f, boolean recursive)	Implement the delete(Path, boolean) in checksum file system.
static double	getApproxChkSumLength(long size)
int	getBytesPerSum()	Return the bytes Per Checksum.
Path	getChecksumFile(Path file)	Return the name of the checksum file associated with a file.
long	getChecksumFileLength(Path file, long fileSize)	Return the length of the checksum file given the size of the actual file.
static long	getChecksumLength(long size, int bytesPerSum)	Calculated the length of the checksum file in bytes.
AbstractFileSystem	getRawFs()	get the raw file system.
static boolean	isChecksumFile(Path file)	Return true iff file is a checksum file name.
RemoteIterator<LocatedFileStatus>	listLocatedStatus(Path f)	The specification of this method matches that of FileContext.listLocatedStatus(Path) except that Path f must be for this file system.
FileStatus[]	listStatus(Path f)	The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.
FSDataInputStream	open(Path f, int bufferSize)	Opens an FSDataInputStream at the indicated Path.
void	renameInternal(Path src, Path dst)	Rename files/dirs.
void	renameInternal(Path src, Path dst, boolean overwrite)	The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.
boolean	reportChecksumFailure(Path f, FSDataInputStream in, long inPos, FSDataInputStream sums, long sumsPos)	Report a checksum error to the file system.
boolean	setReplication(Path src, short replication)	Set replication for an existing file.
void	setVerifyChecksum(boolean inVerifyChecksum)	Set whether to verify checksum.
boolean	truncate(Path f, long newLength)	The specification of this method matches that of FileContext.truncate(Path, long) except that Path f must be for this file system.

Methods inherited from class org.apache.hadoop.fs.FilterFs

access, checkPath, createMultipartUploader, createSnapshot, createSymlink, deleteSnapshot, getAclStatus, getAllStoragePolicies, getCanonicalServiceName, getDelegationTokens, getEnclosingRoot, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileStatus, getFsStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getMyFs, getServerDefaults, getServerDefaults, getStatistics, getStoragePolicy, getUri, getUriDefaultPort, getUriPath, getXAttr, getXAttrs, getXAttrs, hasPathCapability, isValidName, listCorruptFileBlocks, listXAttrs, makeQualified, mkdir, modifyAclEntries, msync, open, openFileWithOptions, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, renameSnapshot, resolvePath, satisfyStoragePolicy, setAcl, setOwner, setPermission, setStoragePolicy, setTimes, setXAttr, setXAttr, supportsSymlinks, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

checkScheme, clearStatistics, create, createFileSystem, equals, get, getAllStatistics, getStatistics, hashCode, listStatusIterator, methodNotSupported, printStatistics, rename

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ChecksumFs

public ChecksumFs(AbstractFileSystem theFs)
           throws IOException,
URISyntaxException

Throws:

IOException

URISyntaxException

Method Details

getApproxChkSumLength

public static double getApproxChkSumLength(long size)

setVerifyChecksum

public void setVerifyChecksum(boolean inVerifyChecksum)

Set whether to verify checksum.

Overrides:

setVerifyChecksum in class FilterFs

Parameters:

inVerifyChecksum - verify check sum flag.

getRawFs

public AbstractFileSystem getRawFs()

get the raw file system.

Returns:

abstract file system.

getChecksumFile

public Path getChecksumFile(Path file)

Return the name of the checksum file associated with a file.

Parameters:

file - the file path.

Returns:

the checksum file associated with a file.

isChecksumFile

public static boolean isChecksumFile(Path file)

Return true iff file is a checksum file name.

Parameters:

file - the file path.

Returns:

if is checksum file true,not false.

getChecksumFileLength

public long getChecksumFileLength(Path file,
 long fileSize)

Return the length of the checksum file given the size of the actual file.

Parameters:

file - the file path.

fileSize - file size.

Returns:

check sum file length.

getBytesPerSum

public int getBytesPerSum()

Return the bytes Per Checksum.

Returns:

bytes per sum.

truncate

public boolean truncate(Path f,
 long newLength)
                 throws IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.truncate(Path, long) except that Path f must be for this file system.

Overrides:

truncate in class FilterFs

Parameters:

f - the path.

newLength - new length.

Returns:

if successfully truncate success true, not false.

Throws:

AccessControlException - access control exception.

FileNotFoundException - file not found exception.

UnresolvedLinkException - unresolved link exception.

IOException - raised on errors performing I/O.

open

public FSDataInputStream open(Path f,
 int bufferSize)
                       throws IOException,
UnresolvedLinkException

Opens an FSDataInputStream at the indicated Path.

Overrides:

open in class FilterFs

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Returns:

if successfully open success true, not false.

Throws:

AccessControlException - access control exception.

FileNotFoundException - file not found exception.

UnresolvedLinkException - unresolved link exception.

IOException - raised on errors performing I/O.

getChecksumLength

public static long getChecksumLength(long size,
 int bytesPerSum)

Calculated the length of the checksum file in bytes.

Parameters:

size - the length of the data file in bytes

bytesPerSum - the number of bytes in a checksum block

Returns:

the number of bytes in the checksum file

createInternal

public FSDataOutputStream createInternal(Path f,
 EnumSet<CreateFlag> createFlag,
 FsPermission absolutePermission,
 int bufferSize,
 short replication,
 long blockSize,
 Progressable progress,
 Options.ChecksumOpt checksumOpt,
 boolean createParent)
                                  throws IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of AbstractFileSystem.create(Path, EnumSet, Options.CreateOpts...) except that the opts have been declared explicitly.

Overrides:

createInternal in class FilterFs

Parameters:

f - the path.

createFlag - create flag.

absolutePermission - absolute permission.

bufferSize - buffer size.

replication - replications.

blockSize - block size.

progress - progress.

checksumOpt - check sum opt.

createParent - create parent.

Returns:

output stream.

Throws:

AccessControlException - access control exception.

FileAlreadyExistsException - file already exists exception.

FileNotFoundException - file not found exception.

ParentNotDirectoryException - parent not directory exception.

UnsupportedFileSystemException - unsupported filesystem exception.

UnresolvedLinkException - unresolved link exception.

IOException - raised on errors performing I/O.

setReplication

public boolean setReplication(Path src,
 short replication)
                       throws IOException,
UnresolvedLinkException

Set replication for an existing file. Implement the abstract setReplication of FileSystem

Overrides:

setReplication in class FilterFs

Parameters:

src - file name

replication - new replication

Returns:

true if successful; false if file does not exist or is a directory

Throws:

IOException - if an I/O error occurs.

UnresolvedLinkException - unresolved link exception.

renameInternal

public void renameInternal(Path src,
 Path dst)
                    throws IOException,
UnresolvedLinkException

Rename files/dirs.

Overrides:

renameInternal in class FilterFs

Parameters:

src - src.

dst - dst.

Throws:

AccessControlException - access control exception.

FileAlreadyExistsException - file already exists exception.

FileNotFoundException - file not found exception.

ParentNotDirectoryException - parent not directory exception.

UnresolvedLinkException - unresolved link exception.

IOException - raised on errors performing I/O.

renameInternal

public void renameInternal(Path src,
 Path dst,
 boolean overwrite)
                    throws AccessControlException,
FileAlreadyExistsException,
FileNotFoundException,
ParentNotDirectoryException,
UnresolvedLinkException,
IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.rename(Path, Path, Options.Rename...) except that Path f must be for this file system.

Overrides:

renameInternal in class FilterFs

Parameters:

src - src.

dst - dst.

overwrite - overwrite flag.

Throws:

AccessControlException - access control exception.

FileAlreadyExistsException - file already exists exception.

FileNotFoundException - file not found exception.

ParentNotDirectoryException - parent not directory exception.

UnresolvedLinkException - unresolved link exception.

IOException - raised on errors performing I/O.

delete

public boolean delete(Path f,
 boolean recursive)
               throws IOException,
UnresolvedLinkException

Implement the delete(Path, boolean) in checksum file system.

Overrides:

delete in class FilterFs

Parameters:

f - the path.

recursive - recursive flag.

Returns:

if successfully deleted success true, not false.

Throws:

AccessControlException - access control exception.

FileNotFoundException - file not found exception.

UnresolvedLinkException - unresolved link exception.

IOException - raised on errors performing I/O.

reportChecksumFailure

public boolean reportChecksumFailure(Path f,
 FSDataInputStream in,
 long inPos,
 FSDataInputStream sums,
 long sumsPos)

Report a checksum error to the file system.

Parameters:

f - the file name containing the error

in - the stream open on the file

inPos - the position of the beginning of the bad data in the file

sums - the stream open on the checksum file

sumsPos - the position of the beginning of the bad data in the checksum file

Returns:

if retry is necessary

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException,
UnresolvedLinkException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.Util.listStatus(Path) except that Path f must be for this file system.

Overrides:

listStatus in class FilterFs

Parameters:

f - the path.

Returns:

FileStatus Iterator.

Throws:

AccessControlException - access control exception.

FileNotFoundException - file not found exception.

UnresolvedLinkException - unresolved link exception.

IOException - raised on errors performing I/O.

listLocatedStatus

public RemoteIterator<LocatedFileStatus> listLocatedStatus(Path f)
                                                    throws AccessControlException,
FileNotFoundException,
UnresolvedLinkException,
IOException

Description copied from class: AbstractFileSystem

The specification of this method matches that of FileContext.listLocatedStatus(Path) except that Path f must be for this file system. In HDFS implementation, the BlockLocation of returned LocatedFileStatus will have different formats for replicated and erasure coded file. Please refer to FileSystem.getFileBlockLocations(FileStatus, long, long) for more details.

Overrides:

listLocatedStatus in class FilterFs

Parameters:

f - the path.

Returns:

FileStatus Iterator.

Throws:

AccessControlException - access control exception.

FileNotFoundException - file not found exception.

UnresolvedLinkException - unresolved link exception.

IOException - raised on errors performing I/O.