Class SimpleTokenizer

java.lang.Object
opennlp.tools.tokenize.AbstractTokenizer
opennlp.tools.tokenize.SimpleTokenizer
All Implemented Interfaces:
Tokenizer

public class SimpleTokenizer extends AbstractTokenizer
A basic Tokenizer implementation which performs tokenization using character classes.

To obtain an instance of this tokenizer use the static final INSTANCE field.

  • Field Details

  • Constructor Details

    • SimpleTokenizer

      private SimpleTokenizer()
  • Method Details

    • tokenizePos

      public Span[] tokenizePos(String s)
      Description copied from interface: Tokenizer
      Finds the boundaries of atomic parts in a string.
      Parameters:
      s - The string to be tokenized.
      Returns:
      The spans (offsets into s) for each token as the individuals array elements.
    • isLineSeparator

      private boolean isLineSeparator(char character)