Package org.apache.hadoop.util

Class CrcComposer

java.lang.Object
org.apache.hadoop.util.CrcComposer
@LimitedPrivate({"Common","HDFS","MapReduce","Yarn"}) @Unstable public final class CrcComposer extends Object
Encapsulates logic for composing multiple CRCs into one or more combined CRCs corresponding to concatenated underlying data ranges. Optimized for composing a large number of CRCs that correspond to underlying chunks of data all of same size.

  • Method Summary

    Modifier and Type
    Method
    Description
    byte[]
    digest()
    Returns byte representation of composed CRCs; if no stripeLength was specified, the digest should be of length equal to exactly one CRC.
    static CrcComposer
    newCrcComposer(DataChecksum.Type type, long bytesPerCrcHint)
    Returns a CrcComposer which will collapse all ingested CRCs into a single value.
    static CrcComposer
    newStripedCrcComposer(DataChecksum.Type type, long bytesPerCrcHint, long stripeLength)
    Returns a CrcComposer which will collapse CRCs for every combined underlying data size which aligns with the specified stripe boundary.
    void
    update(byte[] crcBuffer, int offset, int length, long bytesPerCrc)
    Composes length / CRC_SIZE_IN_BYTES more CRCs from crcBuffer, with each CRC expected to correspond to exactly bytesPerCrc underlying data bytes.
    void
    update(int crcB, long bytesPerCrc)
    Updates with a single additional CRC which corresponds to an underlying data size of bytesPerCrc.
    void
    update(DataInputStream checksumIn, long numChecksumsToRead, long bytesPerCrc)
    Composes numChecksumsToRead additional CRCs into the current digest out of checksumIn, with each CRC expected to correspond to exactly bytesPerCrc underlying data bytes.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • newCrcComposer

      public static CrcComposer newCrcComposer(DataChecksum.Type type, long bytesPerCrcHint)
      Returns a CrcComposer which will collapse all ingested CRCs into a single value.
      Parameters:
      type - type.
      bytesPerCrcHint - bytesPerCrcHint.
      Returns:
      a CrcComposer which will collapse all ingested CRCs into a single value.

    • newStripedCrcComposer

      public static CrcComposer newStripedCrcComposer(DataChecksum.Type type, long bytesPerCrcHint, long stripeLength)
      Returns a CrcComposer which will collapse CRCs for every combined underlying data size which aligns with the specified stripe boundary. For example, if "update" is called with 20 CRCs and bytesPerCrc == 5, and stripeLength == 10, then every two (10 / 5) consecutive CRCs will be combined with each other, yielding a list of 10 CRC "stripes" in the final digest, each corresponding to 10 underlying data bytes. Using a stripeLength greater than the total underlying data size is equivalent to using a non-striped CrcComposer.
      Parameters:
      type - type.
      bytesPerCrcHint - bytesPerCrcHint.
      stripeLength - stripeLength.
      Returns:
      a CrcComposer which will collapse CRCs for every combined. underlying data size which aligns with the specified stripe boundary.

    • update

      public void update(byte[] crcBuffer, int offset, int length, long bytesPerCrc)
      Composes length / CRC_SIZE_IN_BYTES more CRCs from crcBuffer, with each CRC expected to correspond to exactly bytesPerCrc underlying data bytes.
      Parameters:
      crcBuffer - crcBuffer.
      offset - offset.
      length - must be a multiple of the expected byte-size of a CRC.
      bytesPerCrc - bytesPerCrc.

    • update

      public void update(DataInputStream checksumIn, long numChecksumsToRead, long bytesPerCrc) throws IOException
      Composes numChecksumsToRead additional CRCs into the current digest out of checksumIn, with each CRC expected to correspond to exactly bytesPerCrc underlying data bytes.
      Parameters:
      checksumIn - checksumIn.
      numChecksumsToRead - numChecksumsToRead.
      bytesPerCrc - bytesPerCrc.
      Throws:
      IOException - raised on errors performing I/O.

    • update

      public void update(int crcB, long bytesPerCrc)
      Updates with a single additional CRC which corresponds to an underlying data size of bytesPerCrc.
      Parameters:
      crcB - crcB.
      bytesPerCrc - bytesPerCrc.

    • digest

      public byte[] digest()
      Returns byte representation of composed CRCs; if no stripeLength was specified, the digest should be of length equal to exactly one CRC. Otherwise, the number of CRCs in the returned array is equal to the total sum bytesPerCrc divided by stripeLength. If the sum of bytesPerCrc is not a multiple of stripeLength, then the last CRC in the array corresponds to totalLength % stripeLength underlying data bytes.
      Returns:
      byte representation of composed CRCs.