Package opennlp.tools.tokenize.lattice

package opennlp.tools.tokenize.lattice
  • Class
    Description
    LatticeTokenizer
    Dictionary-driven segmentation for languages written without spaces: a Viterbi search over the word lattice of a MecabDictionary, minimizing the sum of word costs and connection costs.
    MecabDictionary
    An immutable, in-memory dictionary in the MeCab directory format: lexicon entries from the *.csv files, connection costs from matrix.def, character categories from char.def, and unknown-word templates from unk.def, loaded from a user-supplied dictionary directory.
    Morpheme
    One morpheme from lattice segmentation: the Span it covers in the original text, its surface form, and the feature columns its dictionary entry carries.
    UnigramSegmenter
    Frequency-driven segmentation for Chinese and similar scripts: a Viterbi search that maximizes the summed log-probability of the words in a user-supplied frequency lexicon, with unlisted characters falling back to single-character words.