Class WhitespaceTokenizer
java.lang.Object
opennlp.tools.tokenize.AbstractTokenizer
opennlp.tools.tokenize.WhitespaceTokenizer
- All Implemented Interfaces:
Tokenizer
A basic
Tokenizer implementation which performs tokenization
using white spaces.
To obtain an instance of this tokenizer use the static final
INSTANCE field. Use newInstance(boolean) for a tokenizer with a
non-default AbstractTokenizer.setKeepNewLines(boolean) setting.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final WhitespaceTokenizerUse this static reference to retrieve an instance of theWhitespaceTokenizer.Fields inherited from class AbstractTokenizer
keepNewLines -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate booleanisLineSeparator(char character) static WhitespaceTokenizernewInstance(boolean keepNewLines) Creates a tokenizer that is not shared withINSTANCE, for a caller that setsAbstractTokenizer.setKeepNewLines(boolean)without touching other users of the shared instance.Span[]Finds the boundaries of atomic parts in a string.Methods inherited from class AbstractTokenizer
setKeepNewLines, tokenize
-
Field Details
-
INSTANCE
Use this static reference to retrieve an instance of theWhitespaceTokenizer.This instance is shared by every caller in the process, so it must not be reconfigured: calling
AbstractTokenizer.setKeepNewLines(boolean)on it changes the output of all other users and is not safely published to other threads. UsenewInstance(boolean)to obtain a tokenizer with a different setting.
-
-
Constructor Details
-
WhitespaceTokenizer
private WhitespaceTokenizer()
-
-
Method Details
-
newInstance
Creates a tokenizer that is not shared withINSTANCE, for a caller that setsAbstractTokenizer.setKeepNewLines(boolean)without touching other users of the shared instance.- Parameters:
keepNewLines-trueif line separators are returned as tokens.- Returns:
- A new
WhitespaceTokenizerwith that setting.
-
tokenizePos
Description copied from interface:TokenizerFinds the boundaries of atomic parts in a string.- Parameters:
d- The string to be tokenized.- Returns:
- The
spans (offsets intofor each token as the individuals array elements.s)
-
isLineSeparator
private boolean isLineSeparator(char character)
-