Class AbstractTokenizer

java.lang.Object
opennlp.tools.tokenize.AbstractTokenizer
All Implemented Interfaces:
Tokenizer
Direct Known Subclasses:
SimpleTokenizer, TokenizerME, WhitespaceTokenizer

abstract class AbstractTokenizer extends Object implements Tokenizer
  • Field Details

    • keepNewLines

      protected boolean keepNewLines
  • Constructor Details

    • AbstractTokenizer

      AbstractTokenizer()
  • Method Details

    • tokenize

      public String[] tokenize(String s)
      Description copied from interface: Tokenizer
      Splits a string into its atomic parts.
      Specified by:
      tokenize in interface Tokenizer
      Parameters:
      s - The string to be tokenized.
      Returns:
      The String[] with the individual tokens as the array elements.
    • setKeepNewLines

      public void setKeepNewLines(boolean keepNewLines)
      Switches whether to keep new lines or not.
      Parameters:
      keepNewLines - True if new lines are kept, false otherwise.