Class WhitespaceTokenizer

java.lang.Object
opennlp.tools.tokenize.AbstractTokenizer
opennlp.tools.tokenize.WhitespaceTokenizer
All Implemented Interfaces:
Tokenizer

public class WhitespaceTokenizer extends AbstractTokenizer
A basic Tokenizer implementation which performs tokenization using white spaces.

To obtain an instance of this tokenizer use the static final INSTANCE field. Use newInstance(boolean) for a tokenizer with a non-default AbstractTokenizer.setKeepNewLines(boolean) setting.

  • Field Details

  • Constructor Details

    • WhitespaceTokenizer

      private WhitespaceTokenizer()
  • Method Details

    • newInstance

      public static WhitespaceTokenizer newInstance(boolean keepNewLines)
      Creates a tokenizer that is not shared with INSTANCE, for a caller that sets AbstractTokenizer.setKeepNewLines(boolean) without touching other users of the shared instance.
      Parameters:
      keepNewLines - true if line separators are returned as tokens.
      Returns:
      A new WhitespaceTokenizer with that setting.
    • tokenizePos

      public Span[] tokenizePos(String d)
      Description copied from interface: Tokenizer
      Finds the boundaries of atomic parts in a string.
      Parameters:
      d - The string to be tokenized.
      Returns:
      The spans (offsets into s) for each token as the individuals array elements.
    • isLineSeparator

      private boolean isLineSeparator(char character)