Class AbstractTokenizer
java.lang.Object
opennlp.tools.tokenize.AbstractTokenizer
- All Implemented Interfaces:
Tokenizer
- Direct Known Subclasses:
SimpleTokenizer, TokenizerME, WhitespaceTokenizer
-
Field Summary
Fields -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionvoidsetKeepNewLines(boolean keepNewLines) Switches whether to keep new lines or not.String[]Splits a string into its atomic parts.Methods inherited from class Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface Tokenizer
tokenizePos
-
Field Details
-
keepNewLines
protected boolean keepNewLines
-
-
Constructor Details
-
AbstractTokenizer
AbstractTokenizer()
-
-
Method Details
-
tokenize
-
setKeepNewLines
public void setKeepNewLines(boolean keepNewLines) Switches whether to keep new lines or not.- Parameters:
keepNewLines-Trueif new lines are kept,falseotherwise.
-