Uses of Interface
opennlp.tools.tokenize.Tokenizer
Packages that use Tokenizer
Package
Description
Experimental package related to converting various corpora to OpenNLP Format.
Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).
Experimental package related to the
MUC corpus format.Contains classes related to finding token or words in a string.
This package contains classes for generating sequence features.
-
Uses of Tokenizer in opennlp.tools.cmdline.parser
Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer -
Uses of Tokenizer in opennlp.tools.cmdline.tokenizer
Fields in opennlp.tools.cmdline.tokenizer declared as TokenizerConstructors in opennlp.tools.cmdline.tokenizer with parameters of type Tokenizer -
Uses of Tokenizer in opennlp.tools.formats
Fields in opennlp.tools.formats declared as TokenizerConstructors in opennlp.tools.formats with parameters of type TokenizerModifierConstructorDescription(package private)TwentyNewsgroupSampleStream(Tokenizer tokenizer, Path dataDir) -
Uses of Tokenizer in opennlp.tools.formats.brat
Fields in opennlp.tools.formats.brat declared as TokenizerConstructors in opennlp.tools.formats.brat with parameters of type TokenizerModifierConstructorDescriptionBratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer) BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes) BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples) Initializes a newBratNameSampleStreamwith the specified parameters based on a straem ofdocuments.BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes) Creates a newBratNameSampleStream. -
Uses of Tokenizer in opennlp.tools.formats.muc
Fields in opennlp.tools.formats.muc declared as TokenizerModifier and TypeFieldDescriptionprivate final TokenizerMucNameContentHandler.tokenizerprivate final TokenizerMucNameSampleStream.tokenizerConstructors in opennlp.tools.formats.muc with parameters of type TokenizerModifierConstructorDescriptionMucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples) Initializes aMucNameContentHandler.protectedMucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples) Initializes aMucNameSampleStream. -
Uses of Tokenizer in opennlp.tools.tokenize
Classes in opennlp.tools.tokenize that implement TokenizerModifier and TypeClassDescription(package private) classclassA basicTokenizerimplementation which performs tokenization using character classes.classA thread-safe version ofTokenizerME.classATokenizerfor converting raw text into separated tokens.classA basicTokenizerimplementation which performs tokenization using white spaces.classATokenizerimplementation which performs tokenization using word pieces.Fields in opennlp.tools.tokenize declared as TokenizerModifier and TypeFieldDescriptionprivate final TokenizerTokenizerEvaluator.tokenizerTheTokenizerused to create the predicted tokens.private final TokenizerTokenizerStream.tokenizerConstructors in opennlp.tools.tokenize with parameters of type TokenizerModifierConstructorDescriptionTokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners) Initializes an instance to evaluate aTokenizer.TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input) Initializes ainstance. -
Uses of Tokenizer in opennlp.tools.util.featuregen
Fields in opennlp.tools.util.featuregen declared as TokenizerConstructors in opennlp.tools.util.featuregen with parameters of type TokenizerModifierConstructorDescriptionTokenPatternFeatureGenerator(Tokenizer supportTokenizer) Initializes aTokenPatternFeatureGeneratorinstance.