Class SentenceDetectorME.AbbreviationIndex
java.lang.Object
opennlp.tools.sentdetect.SentenceDetectorME.AbbreviationIndex
- Enclosing class:
SentenceDetectorME
An immutable, length-bucketed index over an abbreviation
Dictionary. It answers
SentenceDetectorME.isAcceptableBreak(CharSequence, int, int) by enumerating the few text positions
that can carry a relevant abbreviation and asking a hash set what is there, instead of
searching the whole text once per dictionary entry.-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final booleanWhether the dictionary matches case-sensitively.The dictionary entries, folded to lower case unless the dictionary is case-sensitive.private final int[]The distinct entry lengths, ascending.private final intThe longest entry length, that is how far past a candidate a relevant match can reach. -
Constructor Summary
ConstructorsModifierConstructorDescriptionprivateAbbreviationIndex(Dictionary abbDict) ReadsabbDictonce, so this instance is immutable and safe to share. -
Method Summary
Modifier and TypeMethodDescription(package private) booleanallowsBreak(CharSequence s, int fromIndex, int candidateIndex) private static intcodePointEnd(CharSequence s, int index) private static intcodePointStart(CharSequence s, int index) (package private) static SentenceDetectorME.AbbreviationIndexof(Dictionary abbDict) private static StringtoLowerCase(CharSequence s, int from, int to) Lower-cases[from, to)exactly asStringUtil.toLowerCase(CharSequence)lower-cases a whole text: per code point viaCharacter.toLowerCase(int).
-
Field Details
-
entries
-
entryLengths
private final int[] entryLengthsThe distinct entry lengths, ascending. These are the only window sizes worth probing. -
maxEntryLength
private final int maxEntryLengthThe longest entry length, that is how far past a candidate a relevant match can reach. -
caseSensitive
private final boolean caseSensitiveWhether the dictionary matches case-sensitively.
-
-
Constructor Details
-
AbbreviationIndex
ReadsabbDictonce, so this instance is immutable and safe to share.- Parameters:
abbDict- TheDictionaryto index. Must not benull.
-
-
Method Details
-
of
- Parameters:
abbDict- TheDictionaryto index, may benull.- Returns:
- An index over
abbDict, ornullifabbDictisnull.
-
allowsBreak
- Parameters:
s- The text in which the break occurred.fromIndex- The start of the segment currently being evaluated.candidateIndex- The index of the candidate sentence ending.- Returns:
trueif a break atcandidateIndexis allowed.
-
toLowerCase
Lower-cases[from, to)exactly asStringUtil.toLowerCase(CharSequence)lower-cases a whole text: per code point viaCharacter.toLowerCase(int).Folding only the window is correct only because that mapping is 1:1 in
charcount, sopos - windowStarttaken from the unfolded text still indexes the folded window.String.toLowerCase()must not be used here: full case mapping can expand a code point (for exampleİ/ U+0130) and would corrupt every offset in the window.- Parameters:
s- The text to read from.from- The first index to fold, at a code point boundary.to- The index to stop at, at a code point boundary.- Returns:
- The folded characters of
[from, to).
-
codePointStart
- Parameters:
s- The textindexrefers to.index- The index to align.- Returns:
index, moved one character left if it points at the trailing half of a surrogate pair, which is the only index a code point walk cannot start at.
-
codePointEnd
- Parameters:
s- The textindexrefers to.index- The index to align.- Returns:
index, moved one character right if it splits a surrogate pair, which is the only index a code point walk cannot stop at.
-