Class SimpleTokenizer
java.lang.Object
opennlp.tools.tokenize.AbstractTokenizer
opennlp.tools.tokenize.SimpleTokenizer
- All Implemented Interfaces:
Tokenizer
-
Nested Class Summary
Nested Classes -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final SimpleTokenizerUse this static reference to retrieve an instance of theSimpleTokenizer.Fields inherited from class AbstractTokenizer
keepNewLines -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate booleanisLineSeparator(char character) Span[]Finds the boundaries of atomic parts in a string.Methods inherited from class AbstractTokenizer
setKeepNewLines, tokenize
-
Field Details
-
INSTANCE
Use this static reference to retrieve an instance of theSimpleTokenizer.
-
-
Constructor Details
-
SimpleTokenizer
private SimpleTokenizer()
-
-
Method Details
-
tokenizePos
Description copied from interface:TokenizerFinds the boundaries of atomic parts in a string.- Parameters:
s- The string to be tokenized.- Returns:
- The
spans (offsets intofor each token as the individuals array elements.s)
-
isLineSeparator
private boolean isLineSeparator(char character)
-