Class WordpieceTokenizer

java.lang.Object
opennlp.tools.tokenize.WordpieceTokenizer
All Implemented Interfaces:
Tokenizer

public class WordpieceTokenizer extends Object implements Tokenizer
  • Field Details

    • BERT_CLS_TOKEN

      public static final String BERT_CLS_TOKEN
      BERT classification token: [CLS].
      See Also:
    • BERT_SEP_TOKEN

      public static final String BERT_SEP_TOKEN
      BERT separator token: [SEP].
      See Also:
    • BERT_UNK_TOKEN

      public static final String BERT_UNK_TOKEN
      BERT unknown token: [UNK].
      See Also:
    • ROBERTA_CLS_TOKEN

      public static final String ROBERTA_CLS_TOKEN
      RoBERTa classification token: <s>.
      See Also:
    • ROBERTA_SEP_TOKEN

      public static final String ROBERTA_SEP_TOKEN
      RoBERTa separator token.
      See Also:
    • ROBERTA_UNK_TOKEN

      public static final String ROBERTA_UNK_TOKEN
      RoBERTa unknown token.
      See Also:
    • PUNCTUATION_PATTERN

      private static final Pattern PUNCTUATION_PATTERN
    • vocabulary

      private final Set<String> vocabulary
    • classificationToken

      private final String classificationToken
    • separatorToken

      private final String separatorToken
    • unknownToken

      private final String unknownToken
    • maxTokenLength

      private int maxTokenLength
  • Constructor Details

    • WordpieceTokenizer

      public WordpieceTokenizer(Set<String> vocabulary)
      Initializes a WordpieceTokenizer with a vocabulary and a default maxTokenLength of 50.
      Parameters:
      vocabulary - A set of tokens considered the vocabulary.
    • WordpieceTokenizer

      public WordpieceTokenizer(Set<String> vocabulary, int maxTokenLength)
      Initializes a WordpieceTokenizer with a vocabulary and a custom maxTokenLength.
      Parameters:
      vocabulary - A set of tokens considered the vocabulary.
      maxTokenLength - A non-negative number that is used as maximum token length.
    • WordpieceTokenizer

      public WordpieceTokenizer(Set<String> vocabulary, String classificationToken, String separatorToken, String unknownToken)
      Initializes a WordpieceTokenizer with a vocabulary and custom special tokens. This allows support for models like RoBERTa that use different special tokens instead of the BERT defaults.
      Parameters:
      vocabulary - The vocabulary.
      classificationToken - The CLS token.
      separatorToken - The SEP token.
      unknownToken - The UNK token.
  • Method Details

    • tokenizePos

      public Span[] tokenizePos(String text)
      Description copied from interface: Tokenizer
      Finds the boundaries of atomic parts in a string.
      Specified by:
      tokenizePos in interface Tokenizer
      Parameters:
      text - The string to be tokenized.
      Returns:
      The spans (offsets into s) for each token as the individuals array elements.
    • tokenize

      public String[] tokenize(String text)
      Description copied from interface: Tokenizer
      Splits a string into its atomic parts.
      Specified by:
      tokenize in interface Tokenizer
      Parameters:
      text - The string to be tokenized.
      Returns:
      The String[] with the individual tokens as the array elements.
    • getMaxTokenLength

      public int getMaxTokenLength()
      Returns:
      The maximum token length.