Package org.apache.hadoop.io.file.tfile

Class TFile.Writer

java.lang.Object
org.apache.hadoop.io.file.tfile.TFile.Writer
All Implemented Interfaces:
Closeable, AutoCloseable
Enclosing class:
TFile
@Evolving public static class TFile.Writer extends Object implements Closeable
TFile Writer.

  • Constructor Details

    • Writer

      public Writer(FSDataOutputStream fsdos, int minBlockSize, String compressName, String comparator, Configuration conf) throws IOException
      Constructor
      Parameters:
      fsdos - output stream for writing. Must be at position 0.
      minBlockSize - Minimum compressed block size in bytes. A compression block will not be closed until it reaches this size except for the last block.
      compressName - Name of the compression algorithm. Must be one of the strings returned by TFile.getSupportedCompressionAlgorithms().
      comparator - Leave comparator as null or empty string if TFile is not sorted. Otherwise, provide the string name for the comparison algorithm for keys. Two kinds of comparators are supported.
      • Algorithmic comparator: binary comparators that is language independent. Currently, only "memcmp" is supported.
      • Language-specific comparator: binary comparators that can only be constructed in specific language. For Java, the syntax is "jclass:", followed by the class name of the RawComparator. Currently, we only support RawComparators that can be constructed through the default constructor (with no parameters). Parameterized RawComparators such as WritableComparator or JavaSerializationComparator may not be directly used. One should write a wrapper class that inherits from such classes and use its default constructor to perform proper initialization.
      conf - The configuration object.
      Throws:
      IOException - raised on errors performing I/O.

  • Method Details

    • close

      public void close() throws IOException
      Close the Writer. Resources will be released regardless of the exceptions being thrown. Future close calls will have no effect. The underlying FSDataOutputStream is not closed.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • append

      public void append(byte[] key, byte[] value) throws IOException
      Adding a new key-value pair to the TFile. This is synonymous to append(key, 0, key.length, value, 0, value.length)
      Parameters:
      key - Buffer for key.
      value - Buffer for value.
      Throws:
      IOException - raised on errors performing I/O.

    • append

      public void append(byte[] key, int koff, int klen, byte[] value, int voff, int vlen) throws IOException
      Adding a new key-value pair to TFile.
      Parameters:
      key - buffer for key.
      koff - offset in key buffer.
      klen - length of key.
      value - buffer for value.
      voff - offset in value buffer.
      vlen - length of value.
      Throws:
      IOException - Upon IO errors.

      If an exception is thrown, the TFile will be in an inconsistent state. The only legitimate call after that would be close

    • prepareAppendKey

      public DataOutputStream prepareAppendKey(int length) throws IOException
      Obtain an output stream for writing a key into TFile. This may only be called when there is no active Key appending stream or value appending stream.
      Parameters:
      length - The expected length of the key. If length of the key is not known, set length = -1. Otherwise, the application must write exactly as many bytes as specified here before calling close on the returned output stream.
      Returns:
      The key appending output stream.
      Throws:
      IOException - raised on errors performing I/O.

    • prepareAppendValue

      public DataOutputStream prepareAppendValue(int length) throws IOException
      Obtain an output stream for writing a value into TFile. This may only be called right after a key appending operation (the key append stream must be closed).
      Parameters:
      length - The expected length of the value. If length of the value is not known, set length = -1. Otherwise, the application must write exactly as many bytes as specified here before calling close on the returned output stream. Advertising the value size up-front guarantees that the value is encoded in one chunk, and avoids intermediate chunk buffering.
      Returns:
      DataOutputStream.
      Throws:
      IOException - raised on errors performing I/O.

    • prepareMetaBlock

      public DataOutputStream prepareMetaBlock(String name, String compressName) throws IOException, MetaBlockAlreadyExists
      Obtain an output stream for creating a meta block. This function may not be called when there is a key append stream or value append stream active. No more key-value insertion is allowed after a meta data block has been added to TFile.
      Parameters:
      name - Name of the meta block.
      compressName - Name of the compression algorithm to be used. Must be one of the strings returned by TFile.getSupportedCompressionAlgorithms().
      Returns:
      A DataOutputStream that can be used to write Meta Block data. Closing the stream would signal the ending of the block.
      Throws:
      IOException - raised on errors performing I/O.
      MetaBlockAlreadyExists - the Meta Block with the same name already exists.

    • prepareMetaBlock

      public DataOutputStream prepareMetaBlock(String name) throws IOException, MetaBlockAlreadyExists
      Obtain an output stream for creating a meta block. This function may not be called when there is a key append stream or value append stream active. No more key-value insertion is allowed after a meta data block has been added to TFile. Data will be compressed using the default compressor as defined in Writer's constructor.
      Parameters:
      name - Name of the meta block.
      Returns:
      A DataOutputStream that can be used to write Meta Block data. Closing the stream would signal the ending of the block.
      Throws:
      IOException - raised on errors performing I/O.
      MetaBlockAlreadyExists - the Meta Block with the same name already exists.