Class TokenSampleStream
java.lang.Object
opennlp.tools.tokenize.lang.en.TokenSampleStream
- All Implemented Interfaces:
Iterator<TokenSample>
Class which produces an Iterator<TokenSample> from a file of space delimited token.
This class uses a number of English-specific heuristics to un-separate tokens which
are typically found together in text.
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final Patternprivate booleanprivate final BufferedReaderprivate Stringprivate static final org.slf4j.Logger -
Constructor Summary
Constructors -
Method Summary
Methods inherited from class Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface Iterator
forEachRemaining
-
Field Details
-
logger
private static final org.slf4j.Logger logger -
in
-
line
-
alphaNumeric
-
evenq
private boolean evenq
-
-
Constructor Details
-
TokenSampleStream
- Throws:
IOException
-
-
Method Details
-
hasNext
public boolean hasNext()- Specified by:
hasNextin interfaceIterator<TokenSample>
-
next
- Specified by:
nextin interfaceIterator<TokenSample>
-
remove
public void remove()- Specified by:
removein interfaceIterator<TokenSample>
-
usage
private static void usage()
-