Package org.apache.hadoop.util

Class DataChecksum

java.lang.Object
org.apache.hadoop.util.DataChecksum
All Implemented Interfaces:
Checksum
@LimitedPrivate({"HDFS","MapReduce"}) @Evolving public class DataChecksum extends Object implements Checksum
This class provides interface and utilities for processing checksums for DFS data transfers.

  • Field Details

  • Method Details

    • newCrc32

      public static Checksum newCrc32()
      Create a Crc32 Checksum object. The implementation of the Crc32 algorithm is chosen depending on the platform.
      Returns:
      Checksum.

    • newDataChecksum

      public static DataChecksum newDataChecksum(DataChecksum.Type type, int bytesPerChecksum)

    • newDataChecksum

      public static DataChecksum newDataChecksum(byte[] bytes, int offset) throws InvalidChecksumSizeException
      Creates a DataChecksum from HEADER_LEN bytes from arr[offset].
      Parameters:
      bytes - bytes.
      offset - offset.
      Returns:
      DataChecksum of the type in the array or null in case of an error.
      Throws:
      InvalidChecksumSizeException - when the stored checksum is invalid.

    • newDataChecksum

      public static DataChecksum newDataChecksum(DataInputStream in) throws IOException
      This constructs a DataChecksum by reading HEADER_LEN bytes from input stream in.
      Parameters:
      in - data input stream.
      Returns:
      DataChecksum by reading HEADER_LEN bytes from input stream.
      Throws:
      IOException - raised on errors performing I/O.

    • writeHeader

      public void writeHeader(DataOutputStream out) throws IOException
      Writes the checksum header to the output stream out.
      Parameters:
      out - output stream.
      Throws:
      IOException - raised on errors performing I/O.

    • getHeader

      public byte[] getHeader()

    • writeValue

      public int writeValue(DataOutputStream out, boolean reset) throws IOException
      Writes the current checksum to the stream. If reset is true, then resets the checksum.
      Parameters:
      out - out.
      reset - reset.
      Returns:
      number of bytes written. Will be equal to getChecksumSize();
      Throws:
      IOException - raised on errors performing I/O.

    • writeValue

      public int writeValue(byte[] buf, int offset, boolean reset) throws IOException
      Writes the current checksum to a buffer. If reset is true, then resets the checksum.
      Parameters:
      buf - buf.
      offset - offset.
      reset - reset.
      Returns:
      number of bytes written. Will be equal to getChecksumSize();
      Throws:
      IOException - raised on errors performing I/O.

    • compare

      public boolean compare(byte[] buf, int offset)
      Compares the checksum located at buf[offset] with the current checksum.
      Parameters:
      buf - buf.
      offset - offset.
      Returns:
      true if the checksum matches and false otherwise.

    • getChecksumType

      public DataChecksum.Type getChecksumType()
      Returns:
      the checksum algorithm type.

    • getChecksumSize

      public int getChecksumSize()
      the size for a checksum.
      Returns:
      the size for a checksum.

    • getChecksumSize

      public int getChecksumSize(int dataSize)
      the required checksum size given the data length.
      Parameters:
      dataSize - data size.
      Returns:
      the required checksum size given the data length.

    • getBytesPerChecksum

      public int getBytesPerChecksum()

    • getNumBytesInSum

      public int getNumBytesInSum()

    • getChecksumHeaderSize

      public static int getChecksumHeaderSize()

    • getValue

      public long getValue()
      Specified by:
      getValue in interface Checksum

    • reset

      public void reset()
      Specified by:
      reset in interface Checksum

    • update

      public void update(byte[] b, int off, int len)
      Specified by:
      update in interface Checksum

    • update

      public void update(int b)
      Specified by:
      update in interface Checksum

    • verifyChunkedSums

      public void verifyChunkedSums(ByteBuffer data, ByteBuffer checksums, String fileName, long basePos) throws ChecksumException
      Verify that the given checksums match the given data. The 'mark' of the ByteBuffer parameters may be modified by this function,. but the position is maintained.
      Parameters:
      data - the DirectByteBuffer pointing to the data to verify.
      checksums - the DirectByteBuffer pointing to a series of stored checksums
      fileName - the name of the file being read, for error-reporting
      basePos - the file position to which the start of 'data' corresponds
      Throws:
      ChecksumException - if the checksums do not match

    • calculateChunkedSums

      public void calculateChunkedSums(ByteBuffer data, ByteBuffer checksums)
      Calculate checksums for the given data. The 'mark' of the ByteBuffer parameters may be modified by this function, but the position is maintained.
      Parameters:
      data - the DirectByteBuffer pointing to the data to checksum.
      checksums - the DirectByteBuffer into which checksums will be stored. Enough space must be available in this buffer to put the checksums.

    • calculateChunkedSums

      public void calculateChunkedSums(byte[] data, int dataOffset, int dataLength, byte[] sums, int sumsOffset)
      Implementation of chunked calculation specifically on byte arrays. This is to avoid the copy when dealing with ByteBuffers that have array backing.
      Parameters:
      data - data.
      dataOffset - dataOffset.
      dataLength - dataLength.
      sums - sums.
      sumsOffset - sumsOffset.

    • equals

      public boolean equals(Object other)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object