Class DefaultTokenContextGenerator
java.lang.Object
opennlp.tools.tokenize.DefaultTokenContextGenerator
- All Implemented Interfaces:
TokenContextGenerator
A default
TokenContextGenerator which produces events for maxent decisions
for tokenization.-
Field Summary
Fields -
Constructor Summary
ConstructorsConstructorDescriptionInitializes a plainDefaultTokenContextGeneratorinstance.DefaultTokenContextGenerator(Set<String> inducedAbbreviations) Initializes a customizedDefaultTokenContextGeneratorinstance via a set ofinducedAbbreviations. -
Method Summary
Modifier and TypeMethodDescriptionprotected voidaddCharPreds(String key, char c, List<String> preds) Helper function forcreateContext(String, int)that appends to a givenkeya fixed text sequence depending onc.createContext(String sentence, int index) String[]getContext(String sentence, int index)
-
Field Details
-
inducedAbbreviations
-
-
Constructor Details
-
DefaultTokenContextGenerator
public DefaultTokenContextGenerator()Initializes a plainDefaultTokenContextGeneratorinstance. -
DefaultTokenContextGenerator
Initializes a customizedDefaultTokenContextGeneratorinstance via a set ofinducedAbbreviations.- Parameters:
inducedAbbreviations- The induced abbreviations to be used for this instance.
-
-
Method Details
-
getContext
- Specified by:
getContextin interfaceTokenContextGenerator- Parameters:
sentence- The string that represents a sentence.index- The index to consider splitting tokens.- Returns:
- An array of features for a
sentenceat the specifiedindex.
-
createContext
Computes aListof features for the specifiedsentenceat the specifiedindex. Extensions ofDefaultTokenContextGeneratorcan override this method to create a customized behaviour.- Parameters:
sentence- The sentence to create features for.index- The positional index. Must be a non-negative number or0.- Returns:
- A
Listof features for the specifiedsentencestring at the specified index.
-
addCharPreds
Helper function forcreateContext(String, int)that appends to a givenkeya fixed text sequence depending onc. The resulting combination is added to the given listpreds.- Parameters:
key- The input string to process.c- A character used to discriminate which fixed text shall be appended.preds- The list into which the resulting combinations will be added.
-