Class TokenizerModel
java.lang.Object
opennlp.tools.util.model.BaseModel
opennlp.tools.tokenize.TokenizerModel
- All Implemented Interfaces:
Serializable, ArtifactProvider
The
TokenizerModel is the model used
by a learnable Tokenizer.- See Also:
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate static final Stringprivate static final longprivate static final StringFields inherited from class BaseModel
artifactMap, FACTORY_NAME, MANIFEST_ENTRY, toolFactory, TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY -
Constructor Summary
ConstructorsConstructorDescriptionTokenizerModel(File modelFile) Initializes aTokenizerModelinstance via a validFile.Initializes aTokenizerModelinstance via a validInputStream.TokenizerModel(URL modelURL) Initializes aTokenizerModelinstance via a validURL.TokenizerModel(Path modelPath) Initializes aTokenizerModelinstance via a validPath.TokenizerModel(MaxentModel tokenizerModel, Map<String, String> manifestInfoEntries, TokenizerFactory tokenizerFactory) Initializes aTokenizerModelinstance via aMaxentModeland related resources. -
Method Summary
Modifier and TypeMethodDescriptionbooleanprotected Class<? extends BaseToolFactory> Subclasses should override this method if their module has a defaultBaseToolFactorysubclass.inthashCode()private static booleanisModelCompatible(MaxentModel model) Checks if theTokenizerModelhas the right outcomes.booleanprotected voidValidates the parsed artifacts.Methods inherited from class BaseModel
checkArtifactMap, createArtifactSerializers, createArtifactSerializers, deserialize, getArtifact, getArtifactSerializer, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize, setManifestProperty, skipEntryForSerialization
-
Field Details
-
serialVersionUID
private static final long serialVersionUID- See Also:
-
COMPONENT_NAME
- See Also:
-
TOKENIZER_MODEL_ENTRY
- See Also:
-
-
Constructor Details
-
TokenizerModel
public TokenizerModel(MaxentModel tokenizerModel, Map<String, String> manifestInfoEntries, TokenizerFactory tokenizerFactory) Initializes aTokenizerModelinstance via aMaxentModeland related resources.- Parameters:
tokenizerModel- Themodelto be used.manifestInfoEntries- Additional information kept in the manifest.tokenizerFactory- TheTokenizerFactoryto be used internally.
-
TokenizerModel
Initializes aTokenizerModelinstance via a validInputStream.- Parameters:
in- TheInputStreamused for loading the model.- Throws:
IOException- Thrown if IO errors occurred during initialization.
-
TokenizerModel
Initializes aTokenizerModelinstance via a validFile.- Parameters:
modelFile- TheFileused for loading the model.- Throws:
IOException- Thrown if IO errors occurred during initialization.
-
TokenizerModel
Initializes aTokenizerModelinstance via a validPath.- Parameters:
modelPath- ThePathused for loading the model.- Throws:
IOException- Thrown if IO errors occurred during initialization.
-
TokenizerModel
Initializes aTokenizerModelinstance via a validURL.- Parameters:
modelURL- TheURLused for loading the model.- Throws:
IOException- Thrown if IO errors occurred during initialization.
-
-
Method Details
-
isModelCompatible
Checks if theTokenizerModelhas the right outcomes.- Parameters:
model- TheMaxentModelto be checked.- Returns:
trueif the model could be validated,falseotherwise.
-
validateArtifactMap
Description copied from class:BaseModelValidates the parsed artifacts. If something is not valid, subclasses should throw anInvalidFormatException.Note: Subclasses should generally invoke
super.validateArtifactMapat the beginning of this method.- Overrides:
validateArtifactMapin classBaseModel- Throws:
InvalidFormatException- Thrown if artifacts were found to be inconsistent.
-
getFactory
- Returns:
- Retrieves the active
TokenizerFactory.
-
getDefaultFactory
Description copied from class:BaseModelSubclasses should override this method if their module has a defaultBaseToolFactorysubclass.- Overrides:
getDefaultFactoryin classBaseModel- Returns:
- The default
BaseToolFactoryfor the component, ornullif none.
-
getMaxentModel
- Returns:
- Retrieves the model as
MaxentModelinstance.
-
getAbbreviations
- Returns:
- Retrieves the active abbreviation
Dictionary.
-
useAlphaNumericOptimization
public boolean useAlphaNumericOptimization()- Returns:
trueif alphanumeric optimization is active,falseotherwise.
-
hashCode
-
equals
-