Class BaseModel
- All Implemented Interfaces:
Serializable, ArtifactProvider
- Direct Known Subclasses:
ChunkerModel, DoccatModel, LanguageDetectorModel, LemmatizerModel, ParserModel, POSModel, SentenceModel, TokenizerModel, TokenNameFinderModel
- See Also:
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static final classIsolates all references toObjectInputFilter, which is not available on every platform (e.g. -
Field Summary
FieldsModifier and TypeFieldDescriptionprivate Map<String, ArtifactSerializer<?>> private static final Stringprivate Stringprotected static final Stringprivate booleanprivate booleanprivate static final Stringprotected static final Stringprivate static final Stringprivate static final Stringprivate static final longprivate booleanprivate static final Stringprotected BaseToolFactorystatic final Stringstatic final Stringstatic final Stringprivate static final String -
Constructor Summary
ConstructorsModifierConstructorDescriptionprivateprotectedInitializes aBaseModelinstance.protectedBaseModel(String componentName, InputStream in) Initializes aBaseModelinstance.protectedInitializes aBaseModelinstance.protectedBaseModel(String componentName, String languageCode, Map<String, String> manifestInfoEntries, BaseToolFactory factory) Initializes aBaseModelinstance.protectedInitializes aBaseModelinstance.protectedInitializes aBaseModelinstance. -
Method Summary
Modifier and TypeMethodDescriptionprotected voidChecks the artifact map.protected static Map<String, ArtifactSerializer<?>> Creates and registers defaultartifact serializes.protected voidcreateArtifactSerializers(Map<String, ArtifactSerializer<?>> serializers) Registers allArtifactSerializerfor their artifact file name extensions.private voidcreateBaseArtifactSerializers(Map<String, ArtifactSerializer<?>> serializers) static <T extends BaseModel>
Tdeserialize(Class<T> modelClass, InputStream in) Deserializes aBaseModelfrom the givenInputStreamusing the standardSerializablemechanism, with an allowlist filter applied to reject any class outside the allowlist - including the top-level object itself.private voidFinishes loading the artifacts now that it knows all serializers.<T> TgetArtifact(String key) protected ArtifactSerializer<?> getArtifactSerializer(String resourceName) protected Class<? extends BaseToolFactory> Subclasses should override this method if their module has a defaultBaseToolFactorysubclass.private StringgetEntryExtension(String entry) Extracts the "." extension from an entry name.final Stringfinal Stringfinal Versionprivate voidbooleanIndicates if this provider was loaded from a serialized form.private voidLoads theartifact serializers.private voidloadModel(InputStream in) private voidfinal voidSerializes the model to the specifiedFile.final voidserialize(OutputStream out) Serializes the model to the givenOutputStream.final voidSerializes the model to the specifiedPath.protected final voidsetManifestProperty(String key, String value) Sets a given value for a given key to themanifest.propertiesmapping.protected booleanprotected voidValidates the parsed artifacts.private void
-
Field Details
-
serialVersionUID
private static final long serialVersionUID- See Also:
-
MANIFEST_ENTRY
- See Also:
-
FACTORY_NAME
- See Also:
-
MANIFEST_VERSION_PROPERTY
- See Also:
-
COMPONENT_NAME_PROPERTY
- See Also:
-
VERSION_PROPERTY
- See Also:
-
TIMESTAMP_PROPERTY
- See Also:
-
LANGUAGE_PROPERTY
- See Also:
-
TRAINING_CUTOFF_PROPERTY
- See Also:
-
TRAINING_ITERATIONS_PROPERTY
- See Also:
-
TRAINING_EVENTHASH_PROPERTY
- See Also:
-
SERIALIZER_CLASS_NAME_PREFIX
- See Also:
-
artifactSerializers
-
artifactMap
-
toolFactory
-
componentName
-
subclassSerializersInitiated
private boolean subclassSerializersInitiated -
finishedLoadingArtifacts
private boolean finishedLoadingArtifacts -
isLoadedFromSerialized
private boolean isLoadedFromSerialized
-
-
Constructor Details
-
BaseModel
-
BaseModel
protected BaseModel(String componentName, String languageCode, Map<String, String> manifestInfoEntries, BaseToolFactory factory) Initializes aBaseModelinstance. The subclass constructor should call the methodcheckArtifactMap()to check the artifact map is in a valid state.Subclasses will have access to custom artifacts and serializers provided by the specified
factory.- Parameters:
componentName- The component name to create the model for.languageCode- The ISO language code to configure. Must not benull.manifestInfoEntries- Mapping for additional information in the manifest.factory- Thefactoryto use.
-
BaseModel
protected BaseModel(String componentName, String languageCode, Map<String, String> manifestInfoEntries) Initializes aBaseModelinstance. The subclass constructor should call the methodcheckArtifactMap()to check the artifact map is in a valid state.- Parameters:
componentName- The component name to create the model for.languageCode- The ISO language code to configure. Must not benull.manifestInfoEntries- Mapping for additional information in the manifest.
-
BaseModel
Initializes aBaseModelinstance. The subclass constructor should call the methodcheckArtifactMap()to check the artifact map is in a valid state.- Parameters:
componentName- The component name to create the model for.in- A valid, openInputStreamto read the model from.- Throws:
IOException- Thrown if IO errors occurred.
-
BaseModel
Initializes aBaseModelinstance. The subclass constructor should call the methodcheckArtifactMap()to check the artifact map is in a valid state.- Parameters:
componentName- The component name to create the model for.modelFile- A valid, accessibleFileto read the model from.- Throws:
IOException- Thrown if IO errors occurred.
-
BaseModel
Initializes aBaseModelinstance. The subclass constructor should call the methodcheckArtifactMap()to check the artifact map is in a valid state.- Parameters:
componentName- The component name to create the model for.modelPath- A valid, accessiblePathto read the model from.- Throws:
IOException- Thrown if IO errors occurred.
-
BaseModel
Initializes aBaseModelinstance. The subclass constructor should call the methodcheckArtifactMap()to check the artifact map is in a valid state.- Parameters:
componentName- The component name to create the model for.modelURL- A valid, accessibleURLto read the model from.- Throws:
IOException- Thrown if IO errors occurred.
-
-
Method Details
-
loadModel
- Throws:
IOException
-
initializeFactory
- Throws:
InvalidFormatException
-
getDefaultFactory
Subclasses should override this method if their module has a defaultBaseToolFactorysubclass.- Returns:
- The default
BaseToolFactoryfor the component, ornullif none.
-
loadArtifactSerializers
private void loadArtifactSerializers()Loads theartifact serializers. -
finishLoadingArtifacts
Finishes loading the artifacts now that it knows all serializers.- Throws:
IOException
-
getEntryExtension
Extracts the "." extension from an entry name.- Parameters:
entry- The entry name which contains the extension- Returns:
- The extension.
- Throws:
InvalidFormatException- Thrown if no extension can be extracted
-
getArtifactSerializer
- Parameters:
resourceName- The identifying name of a resource to retrieve an {link ArtifactSerializer} for.- Returns:
- Retrieves an
artifact serialize.
-
createArtifactSerializers
Creates and registers defaultartifact serializes.- Returns:
- A
Mapwith all registeredartifact serializes.
-
createArtifactSerializers
Registers allArtifactSerializerfor their artifact file name extensions. The registeredserializersare used to create and serialize resources in the model package.Override this method to register custom
serializers.Note: Subclasses should generally invoke
super.createArtifactSerializersat the beginning of this method.This method is called during construction.
- Parameters:
serializers- The key of the map is the file extension used to look up anArtifactSerializer.
-
createBaseArtifactSerializers
-
validateArtifactMap
Validates the parsed artifacts. If something is not valid, subclasses should throw anInvalidFormatException.Note: Subclasses should generally invoke
super.validateArtifactMapat the beginning of this method.- Throws:
InvalidFormatException- Thrown if artifacts were found to be inconsistent.
-
checkArtifactMap
protected void checkArtifactMap()Checks the artifact map.A subclass should call this method from a constructor which accepts the individual artifact map items, to validate that these items form a valid model.
- Throws:
IllegalArgumentException- Thrown if the artifacts are not valid.IllegalStateException- Thrown iffinishLoadingArtifacts(InputStream)was not called by a subclass.
-
getManifestProperty
- Specified by:
getManifestPropertyin interfaceArtifactProvider- Parameters:
key- A string identifying an element.- Returns:
- Retrieves the value for the given
keyfrom themanifest.properties.
-
setManifestProperty
-
getLanguage
- Specified by:
getLanguagein interfaceArtifactProvider- Returns:
- Retrieves the language code of the material which was used to train a model
or
'x-unspecified'if non was set.
-
getVersion
-
serialize
Serializes the model to the givenOutputStream.- Parameters:
out- TheOutputStreamto write the model to.- Throws:
IOException- Thrown if IO errors occurred.IllegalStateException- Thrown ifloadArtifactSerializers()was not called in a subclass constructor.
-
serialize
Serializes the model to the specifiedFile.- Parameters:
f- The write-accessibleFileto write the model to.- Throws:
IOException- Thrown if IO errors occurred.IllegalStateException- Thrown ifloadArtifactSerializers()was not called in a subclass constructor.
-
serialize
Serializes the model to the specifiedPath.- Parameters:
p- The write-accessiblePathto write the model to.- Throws:
IOException- Thrown if IO errors occurred.IllegalStateException- Thrown ifloadArtifactSerializers()was not called in a subclass constructor.
-
getArtifact
- Specified by:
getArtifactin interfaceArtifactProvider- Returns:
- Gets an artifact by an identifying key or name.
-
isLoadedFromSerialized
public boolean isLoadedFromSerialized()Description copied from interface:ArtifactProviderIndicates if this provider was loaded from a serialized form.It is useful, for example, during the validation of artifacts: Skip the time-consuming ones if those were already validated during the serialization process.
- Specified by:
isLoadedFromSerializedin interfaceArtifactProvider- Returns:
trueif this model was loaded from a serialized form,falseotherwise.
-
writeObject
- Throws:
IOException
-
deserialize
public static <T extends BaseModel> T deserialize(Class<T> modelClass, InputStream in) throws IOException, ClassNotFoundException Deserializes aBaseModelfrom the givenInputStreamusing the standardSerializablemechanism, with an allowlist filter applied to reject any class outside the allowlist - including the top-level object itself.Prefer this method (or the
BaseModel(String, InputStream|File|Path|URL)constructors) over callingnew ObjectInputStream(in).readObject()directly, since a filter installed insidereadObject(ObjectInputStream)cannot protect the top-level object - the JDK only allows anObjectInputFilterto be set on a stream before the first object is read.- Type Parameters:
T- The type of the expected model.- Parameters:
modelClass- The expectedBaseModelsubclass.in- TheInputStreamto read from. Must not benull.- Returns:
- A valid
BaseModelinstance of typeT. - Throws:
IOException- Thrown if IO errors occurred, including a rejection by the allowlist filter.ClassNotFoundException- Thrown if required classes are not found.
-
readObject
- Throws:
IOException
-
skipEntryForSerialization
-