Class SentenceDetectorME
- All Implemented Interfaces:
Probabilistic, SentenceDetector
A maximum entropy model is used to evaluate end-of-sentence characters in a string to determine if they signify the end of a sentence.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static final classAn immutable, length-bucketed index over an abbreviationDictionary. -
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final SentenceDetectorME.AbbreviationIndexThe abbreviation dictionary index that backsisAcceptableBreak(CharSequence, int, int).private final SDContextGeneratorThe feature context generator.private final MaxentModelThe maximum entropy model to use to evaluate contexts.static final StringConstant indicates no sentence split.private final EndOfSentenceScannerTheEndOfSentenceScannerto use when scanning for end of sentence offsets.The list of probabilities associated with each decision.static final StringConstant indicates a sentence split.protected final boolean -
Constructor Summary
ConstructorsConstructorDescriptionSentenceDetectorME(String language) Initializes the sentence detector by downloading a default model.SentenceDetectorME(SentenceModel model) Initializes the current instance.SentenceDetectorME(SentenceModel model, Dictionary abbDict) Instantiates aSentenceDetectorMEwith an existingSentenceModel.SentenceDetectorME(SentenceModel model, Factory factory) Deprecated. -
Method Summary
Modifier and TypeMethodDescriptiongetAbbreviations(Dictionary abbreviations) private intgetFirstNonWS(CharSequence s, int pos) private intgetFirstWS(CharSequence s, int pos) double[]Deprecated, for removal: This API element is subject to removal in a future version.Useprobs()instead.protected booleanisAcceptableBreak(CharSequence s, int fromIndex, int candidateIndex) Allows subclasses to check an overzealous (read: poorly trained) model from flagging obvious non-breaks as breaks based on some boolean determination of a break's acceptability.private static booleanisApostrophe(char c) double[]probs()Retrieves the probabilities of the last decoded sequence.String[]Detects sentences in given inputCharSequence..Span[]Detects the position of the first words of sentences in aCharSequence.static SentenceModeltrain(String languageCode, ObjectStream<SentenceSample> samples, SentenceDetectorFactory sdFactory, TrainingParameters mlParams) Starts a training of aSentenceModelwith the given parameters.
-
Field Details
-
SPLIT
-
NO_SPLIT
-
model
The maximum entropy model to use to evaluate contexts. -
cgen
The feature context generator. -
scanner
TheEndOfSentenceScannerto use when scanning for end of sentence offsets. -
sentProbs
-
abbIndex
The abbreviation dictionary index that backsisAcceptableBreak(CharSequence, int, int). It isnullif no abbreviation dictionary is available for the underlying model. -
useTokenEnd
protected final boolean useTokenEnd
-
-
Constructor Details
-
SentenceDetectorME
Initializes the sentence detector by downloading a default model.- Parameters:
language- The language of the sentence detector.- Throws:
IOException- Thrown if the model cannot be downloaded or saved.
-
SentenceDetectorME
Initializes the current instance.- Parameters:
model- theSentenceModel
-
SentenceDetectorME
Instantiates aSentenceDetectorMEwith an existingSentenceModel.- Parameters:
model- TheSentenceModelto be used.abbDict- TheDictionaryto be used. It must fit the language of themodel. Its entries are read once, here; later changes to theabbDictinstance do not affect this detector.
-
SentenceDetectorME
Deprecated.Use aSentenceDetectorFactoryto extend SentenceDetector functionality.
-
-
Method Details
-
getAbbreviations
-
sentDetect
Detects sentences in given inputCharSequence..- Specified by:
sentDetectin interfaceSentenceDetector- Parameters:
s- TheCharSequence. to be processed.- Returns:
- A string array containing individual sentences as elements.
-
getFirstWS
-
getFirstNonWS
-
sentPosDetect
Detects the position of the first words of sentences in aCharSequence.- Specified by:
sentPosDetectin interfaceSentenceDetector- Parameters:
s- TheCharSequenceto be processed.- Returns:
- An
span arraycontaining the positions of the end index of every sentence.
-
probs
public double[] probs()Retrieves the probabilities of the last decoded sequence. The sequence was determined based on the previous call tosentDetect(CharSequence).- Specified by:
probsin interfaceProbabilistic- Returns:
- An array with the same number of probabilities as tokens were sent to
sentDetect(CharSequence)when it was last called. If not applicable, an empty array is returned.
-
getSentenceProbabilities
Deprecated, for removal: This API element is subject to removal in a future version.Useprobs()instead.- Returns:
- The probability for each sentence returned for the most recent
call to
sentDetect(CharSequence). If not applicable, an empty array is returned.
-
isAcceptableBreak
Allows subclasses to check an overzealous (read: poorly trained) model from flagging obvious non-breaks as breaks based on some boolean determination of a break's acceptability.Note: The implementation always returns
trueif no abbreviation dictionary is available for the underlying model.Only a bounded region of
sis consulted. An abbreviation occurrence starting atpwith lengthLcan veto the break only when it starts at or before the candidate and reaches it, that is whencandidateIndex - L <= p <= candidateIndex. Positions outside that window are irrelevant by construction, so the decision costs time proportional to the longest dictionary entry rather than to the length ofs.A
candidateIndexgreater thans.length()cannot carry an abbreviation, so the break is accepted rather than raising anIndexOutOfBoundsException. AcandidateIndexequal tos.length()is evaluated like any other index.- Parameters:
s- theCharSequencein which the break occurred.fromIndex- the start of the segment currently being evaluated.candidateIndex- the index of the candidate sentence ending. A value belowfromIndex, negative or greater thans.length()is accepted as a break.- Returns:
trueif the break is acceptable,falseotherwise.
-
isApostrophe
private static boolean isApostrophe(char c) - Parameters:
c- The character to check.- Returns:
trueif the character represents an apostrophe,falseotherwise.
-
train
public static SentenceModel train(String languageCode, ObjectStream<SentenceSample> samples, SentenceDetectorFactory sdFactory, TrainingParameters mlParams) throws IOException Starts a training of aSentenceModelwith the given parameters.- Parameters:
languageCode- The ISO language code to train the model. Must not benull.samples- TheObjectStreamofSentenceSampleused as input for training.sdFactory- TheSentenceDetectorFactoryfor creating related objects as defined viamlParams.mlParams- TheTrainingParametersfor the context of the training process.- Returns:
- A valid, trained
SentenceModelinstance. - Throws:
IOException- Thrown if IO errors occurred.
-
SentenceDetectorFactoryto extend SentenceDetector functionality.