Package org.apache.hadoop.io.erasurecode.rawcoder

Class RawErasureEncoder

java.lang.Object
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureEncoder
Direct Known Subclasses:
DummyRawEncoder, NativeRSRawEncoder, NativeXORRawEncoder, RSLegacyRawEncoder, RSRawEncoder, XORRawEncoder
@Private public abstract class RawErasureEncoder extends Object
An abstract raw erasure encoder that's to be inherited by new encoders. Raw erasure coder is part of erasure codec framework, where erasure coder is used to encode/decode a group of blocks (BlockGroup) according to the codec specific BlockGroup layout and logic. An erasure coder extracts chunks of data from the blocks and can employ various low level raw erasure coders to perform encoding/decoding against the chunks. To distinguish from erasure coder, here raw erasure coder is used to mean the low level constructs, since it only takes care of the math calculation with a group of byte buffers. Note it mainly provides encode() calls, which should be stateless and may be made thread-safe in future.

  • Constructor Details

  • Method Details

    • encode

      public void encode(ByteBuffer[] inputs, ByteBuffer[] outputs) throws IOException
      Encode with inputs and generates outputs. Note, for both inputs and outputs, no mixing of on-heap buffers and direct buffers are allowed. If the coder option ALLOW_CHANGE_INPUTS is set true (false by default), the content of input buffers may change after the call, subject to concrete implementation. Anyway the positions of input buffers will move forward.
      Parameters:
      inputs - input buffers to read data from. The buffers' remaining will be 0 after encoding
      outputs - output buffers to put the encoded data into, ready to read after the call
      Throws:
      IOException - if the encoder is closed.

    • doEncode

      protected abstract void doEncode(org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferEncodingState encodingState) throws IOException
      Perform the real encoding work using direct ByteBuffer.
      Parameters:
      encodingState - the encoding state.
      Throws:
      IOException - raised on errors performing I/O.

    • encode

      public void encode(byte[][] inputs, byte[][] outputs) throws IOException
      Encode with inputs and generates outputs. More see above.
      Parameters:
      inputs - input buffers to read data from
      outputs - output buffers to put the encoded data into, read to read after the call
      Throws:
      IOException - raised on errors performing I/O.

    • doEncode

      protected abstract void doEncode(org.apache.hadoop.io.erasurecode.rawcoder.ByteArrayEncodingState encodingState) throws IOException
      Perform the real encoding work using bytes array, supporting offsets and lengths.
      Parameters:
      encodingState - the encoding state
      Throws:
      IOException - raised on errors performing I/O.

    • encode

      public void encode(ECChunk[] inputs, ECChunk[] outputs) throws IOException
      Encode with inputs and generates outputs. More see above.
      Parameters:
      inputs - input buffers to read data from
      outputs - output buffers to put the encoded data into, read to read after the call
      Throws:
      IOException - if the encoder is closed.

    • getNumDataUnits

      public int getNumDataUnits()

    • getNumParityUnits

      public int getNumParityUnits()

    • getNumAllUnits

      public int getNumAllUnits()

    • preferDirectBuffer

      public boolean preferDirectBuffer()
      Tell if direct buffer is preferred or not. It's for callers to decide how to allocate coding chunk buffers, using DirectByteBuffer or bytes array. It will return false by default.
      Returns:
      true if native buffer is preferred for performance consideration, otherwise false.

    • allowChangeInputs

      public boolean allowChangeInputs()
      Allow change into input buffers or not while perform encoding/decoding.
      Returns:
      true if it's allowed to change inputs, false otherwise

    • allowVerboseDump

      public boolean allowVerboseDump()
      Allow to dump verbose info during encoding/decoding.
      Returns:
      true if it's allowed to do verbose dump, false otherwise.

    • release

      public void release()
      Should be called when release this coder. Good chance to release encoding or decoding buffers