Class WordpieceEncoder

java.lang.Object

opennlp.tools.tokenize.WordpieceEncoder

All Implemented Interfaces:

SubwordTokenizer

@ThreadSafe
public final class WordpieceEncoder
extends Object
implements SubwordTokenizer

A SubwordTokenizer implementing the BERT tokenization stages: basic tokenization (control removal, whitespace normalization, CJK isolation, optional lower casing with accent stripping, punctuation isolation) followed by greedy longest-match wordpiece segmentation.

Each result includes a vocabulary id and range in the original text. The range refers to the input before normalization. Classification and separator entries use empty ranges at the text boundaries, so encode(CharSequence) includes both control entries.

The wordpiece inventory was introduced by Schuster and Nakajima (2012) as the WordPieceModel: word units learned greedily from unsegmented text to maximize the language-model likelihood, so that no input is out of vocabulary. Wu et al. (2016), section 4.1, adopt it for neural machine translation, and Devlin et al. (2019), section 3, build BERT on a 30,000 entry WordPiece vocabulary with a leading classification token. Those papers describe how an inventory is trained; the greedy longest-match-first segmentation and the ## continuation marker applied here are the inference conventions of the BERT reference implementation. Wu et al. instead mark word starts with _.

Ids follow the line-number convention of BERT vocab.txt files. List constructors use the list index, while the map constructor uses the supplied ids. The classification, separator, and unknown tokens must all be present in the vocabulary, because each emitted piece must have an id. Vocabulary entries starting with ## are continuation pieces and can match only after the first piece of a word.

Lower casing applies the Unicode full case mapping, including the Final_Sigma context, so a word-final Greek capital sigma becomes U+03C2 as in the reference implementation.

A word exceeding the configured maximum number of normalized Unicode code points becomes the unknown piece. The default is 100, the value used by the Hugging Face transformers BERT tokenizer; the original google-research/bert code uses 200. Both count code points, and a constructor parameter selects another limit.

Since:

3.0.0

See Also:

• WordpieceTokenizer
• BERT tokenization reference
• Mike Schuster, Kaisuke Nakajima (2012): Japanese and Korean Voice Search. ICASSP 2012, pages 5149-5152
• Yonghui Wu et al. (2016): Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv:1609.08144
• Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (2019): BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019, pages 4171-4186

Constructor Summary

Constructors

Constructor	Description
WordpieceEncoder(List<String> vocabulary)	Instantiates an encoder for an uncased BERT model with the BERT special tokens.
WordpieceEncoder(List<String> vocabulary, boolean lowerCase)	Instantiates an encoder with the BERT special tokens.
WordpieceEncoder(List<String> vocabulary, boolean lowerCase, int maxWordCodePoints)	Instantiates an encoder with the BERT special tokens and a custom word-length limit.
WordpieceEncoder(List<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken)	Instantiates an encoder with custom special tokens.
WordpieceEncoder(List<String> vocabulary, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken, int maxWordCodePoints)	Instantiates an encoder with custom special tokens and a custom word-length limit.
WordpieceEncoder(Map<String,Integer> vocabularyIds, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken)	Instantiates an encoder from an explicit piece-to-id mapping for vocabularies with noncontiguous ids.
WordpieceEncoder(Map<String,Integer> vocabularyIds, boolean lowerCase, String classificationToken, String separatorToken, String unknownToken, int maxWordCodePoints)	Instantiates an encoder from a piece-to-id mapping with a custom word-length limit.

Method Summary

Modifier and Type	Method	Description
List<SubwordPiece>	encode(CharSequence text)	Encodes text into subword pieces.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface SubwordTokenizer

encodeToIds, encodeToPieces

Constructor Details

WordpieceEncoder

public WordpieceEncoder(List<String> vocabulary)

Instantiates an encoder for an uncased BERT model with the BERT special tokens.

Parameters:

vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.

Throws:

IllegalArgumentException - Thrown if the vocabulary is null, contains a null, empty, or duplicate entry, or is missing a BERT special token.

WordpieceEncoder

public WordpieceEncoder(List<String> vocabulary,
 boolean lowerCase)

Instantiates an encoder with the BERT special tokens.

Parameters:

vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.

lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.

Throws:

IllegalArgumentException - Thrown if the vocabulary is null, contains a null, empty, or duplicate entry, or is missing a BERT special token.

WordpieceEncoder

public WordpieceEncoder(List<String> vocabulary,
 boolean lowerCase,
 int maxWordCodePoints)

Instantiates an encoder with the BERT special tokens and a custom word-length limit.

Parameters:

vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.

lowerCase - true for uncased models, false for cased models.

maxWordCodePoints - The non-negative maximum number of normalized Unicode code points in one word.

Throws:

IllegalArgumentException - Thrown if an argument is invalid or a BERT special token is missing.

WordpieceEncoder

public WordpieceEncoder(List<String> vocabulary,
 boolean lowerCase,
 String classificationToken,
 String separatorToken,
 String unknownToken)

Instantiates an encoder with custom special tokens.

Parameters:

vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.

lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.

classificationToken - The CLS token; must not be null or empty and must be in the vocabulary.

separatorToken - The SEP token; must not be null or empty and must be in the vocabulary.

unknownToken - The UNK token; must not be null or empty and must be in the vocabulary.

Throws:

IllegalArgumentException - Thrown if any argument is null, the vocabulary contains a null, empty, or duplicate entry, or a special token is empty or missing.

WordpieceEncoder

public WordpieceEncoder(List<String> vocabulary,
 boolean lowerCase,
 String classificationToken,
 String separatorToken,
 String unknownToken,
 int maxWordCodePoints)

Instantiates an encoder with custom special tokens and a custom word-length limit.

Parameters:

vocabulary - The ordered vocabulary; the list index becomes the id. Must not be null or contain null, empty, or duplicate entries.

lowerCase - true for uncased models, false for cased models.

classificationToken - The CLS token; must be present in the vocabulary.

separatorToken - The SEP token; must be present in the vocabulary.

unknownToken - The UNK token; must be present in the vocabulary.

maxWordCodePoints - The non-negative maximum number of normalized Unicode code points in one word.

Throws:

IllegalArgumentException - Thrown if an argument is invalid or a special token is missing.

WordpieceEncoder

public WordpieceEncoder(Map<String,Integer> vocabularyIds,
 boolean lowerCase,
 String classificationToken,
 String separatorToken,
 String unknownToken)

Instantiates an encoder from an explicit piece-to-id mapping for vocabularies with noncontiguous ids.

Parameters:

vocabularyIds - The piece-to-id mapping. Must not be null or contain null or empty keys, null values, or negative ids.

lowerCase - true for uncased models (lower casing and accent stripping), false for cased models.

classificationToken - The CLS token; must not be null or empty and must be in the vocabulary.

separatorToken - The SEP token; must not be null or empty and must be in the vocabulary.

unknownToken - The UNK token; must not be null or empty and must be in the vocabulary.

Throws:

IllegalArgumentException - Thrown if any argument is null, the mapping contains a null or empty key, null value, or negative id, or a special token is empty or missing.

WordpieceEncoder

public WordpieceEncoder(Map<String,Integer> vocabularyIds,
 boolean lowerCase,
 String classificationToken,
 String separatorToken,
 String unknownToken,
 int maxWordCodePoints)

Instantiates an encoder from a piece-to-id mapping with a custom word-length limit.

Parameters:

vocabularyIds - The piece-to-id mapping. Must not be null or contain invalid entries.

lowerCase - true for uncased models, false for cased models.

classificationToken - The CLS token; must be present in the vocabulary.

separatorToken - The SEP token; must be present in the vocabulary.

unknownToken - The UNK token; must be present in the vocabulary.

maxWordCodePoints - The non-negative maximum number of normalized Unicode code points in one word.

Throws:

IllegalArgumentException - Thrown if an argument is invalid or a special token is missing.

Method Details

encode

public List<SubwordPiece> encode(CharSequence text)

Encodes text into subword pieces.

Specified by:

encode in interface SubwordTokenizer

Parameters:

text - The text to encode; must not be null.

Returns:

The pieces in model order; may be empty.