Package org.apache.hadoop.util

Class LineReader

java.lang.Object
org.apache.hadoop.util.LineReader
All Implemented Interfaces:
Closeable, AutoCloseable, IOStatisticsSource
@LimitedPrivate("MapReduce") @Unstable public class LineReader extends Object implements Closeable, IOStatisticsSource
A class that provides a line reader from an input stream. Depending on the constructor used, lines will either be terminated by:
  • one of the following: '\n' (LF) , '\r' (CR), or '\r\n' (CR+LF).
  • or, a custom byte sequence delimiter
In both cases, EOF also terminates an otherwise unterminated line.

  • Constructor Summary

    Constructors
    Constructor
    Description
    LineReader(InputStream in)
    Create a line reader that reads from the given stream using the default buffer-size (64k).
    LineReader(InputStream in, byte[] recordDelimiterBytes)
    Create a line reader that reads from the given stream using the default buffer-size, and using a custom delimiter of array of bytes.
    LineReader(InputStream in, int bufferSize)
    Create a line reader that reads from the given stream using the given buffer-size.
    LineReader(InputStream in, int bufferSize, byte[] recordDelimiterBytes)
    Create a line reader that reads from the given stream using the given buffer-size, and using a custom delimiter of array of bytes.
    LineReader(InputStream in, Configuration conf)
    Create a line reader that reads from the given stream using the io.file.buffer.size specified in the given Configuration.
    LineReader(InputStream in, Configuration conf, byte[] recordDelimiterBytes)
    Create a line reader that reads from the given stream using the io.file.buffer.size specified in the given Configuration, and using a custom delimiter of array of bytes.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
    Close the underlying stream.
    protected int
    fillBuffer(InputStream in, byte[] buffer, boolean inDelimiter)
     
    protected int
    getBufferPosn()
     
    protected int
    getBufferSize()
     
    IOStatistics
    getIOStatistics()
    Return any IOStatistics provided by the source.
    int
    readLine(Text str)
    Read from the InputStream into the given Text.
    int
    readLine(Text str, int maxLineLength)
    Read from the InputStream into the given Text.
    int
    readLine(Text str, int maxLineLength, int maxBytesToConsume)
    Read one line from the InputStream into the given Text.
    protected void
    unsetNeedAdditionalRecordAfterSplit()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • LineReader

      public LineReader(InputStream in)
      Create a line reader that reads from the given stream using the default buffer-size (64k).
      Parameters:
      in - The input stream

    • LineReader

      public LineReader(InputStream in, int bufferSize)
      Create a line reader that reads from the given stream using the given buffer-size.
      Parameters:
      in - The input stream
      bufferSize - Size of the read buffer

    • LineReader

      public LineReader(InputStream in, Configuration conf) throws IOException
      Create a line reader that reads from the given stream using the io.file.buffer.size specified in the given Configuration.
      Parameters:
      in - input stream
      conf - configuration
      Throws:
      IOException - raised on errors performing I/O.

    • LineReader

      public LineReader(InputStream in, byte[] recordDelimiterBytes)
      Create a line reader that reads from the given stream using the default buffer-size, and using a custom delimiter of array of bytes.
      Parameters:
      in - The input stream
      recordDelimiterBytes - The delimiter

    • LineReader

      public LineReader(InputStream in, int bufferSize, byte[] recordDelimiterBytes)
      Create a line reader that reads from the given stream using the given buffer-size, and using a custom delimiter of array of bytes.
      Parameters:
      in - The input stream
      bufferSize - Size of the read buffer
      recordDelimiterBytes - The delimiter

    • LineReader

      public LineReader(InputStream in, Configuration conf, byte[] recordDelimiterBytes) throws IOException
      Create a line reader that reads from the given stream using the io.file.buffer.size specified in the given Configuration, and using a custom delimiter of array of bytes.
      Parameters:
      in - input stream
      conf - configuration
      recordDelimiterBytes - The delimiter
      Throws:
      IOException - raised on errors performing I/O.

  • Method Details

    • close

      public void close() throws IOException
      Close the underlying stream.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException - raised on errors performing I/O.

    • getIOStatistics

      public IOStatistics getIOStatistics()
      Return any IOStatistics provided by the source.
      Specified by:
      getIOStatistics in interface IOStatisticsSource
      Returns:
      IO stats from the input stream.

    • readLine

      public int readLine(Text str, int maxLineLength, int maxBytesToConsume) throws IOException
      Read one line from the InputStream into the given Text.
      Parameters:
      str - the object to store the given line (without newline)
      maxLineLength - the maximum number of bytes to store into str; the rest of the line is silently discarded.
      maxBytesToConsume - the maximum number of bytes to consume in this call. This is only a hint, because if the line cross this threshold, we allow it to happen. It can overshoot potentially by as much as one buffer length.
      Returns:
      the number of bytes read including the (longest) newline found.
      Throws:
      IOException - if the underlying stream throws

    • fillBuffer

      protected int fillBuffer(InputStream in, byte[] buffer, boolean inDelimiter) throws IOException
      Throws:
      IOException

    • readLine

      public int readLine(Text str, int maxLineLength) throws IOException
      Read from the InputStream into the given Text.
      Parameters:
      str - the object to store the given line
      maxLineLength - the maximum number of bytes to store into str.
      Returns:
      the number of bytes read including the newline
      Throws:
      IOException - if the underlying stream throws

    • readLine

      public int readLine(Text str) throws IOException
      Read from the InputStream into the given Text.
      Parameters:
      str - the object to store the given line
      Returns:
      the number of bytes read including the newline
      Throws:
      IOException - if the underlying stream throws

    • getBufferPosn

      protected int getBufferPosn()

    • getBufferSize

      protected int getBufferSize()

    • unsetNeedAdditionalRecordAfterSplit

      protected void unsetNeedAdditionalRecordAfterSplit()