Class SentenceDetectorME.AbbreviationIndex

java.lang.Object
opennlp.tools.sentdetect.SentenceDetectorME.AbbreviationIndex
Enclosing class:
SentenceDetectorME

private static final class SentenceDetectorME.AbbreviationIndex extends Object
An immutable, length-bucketed index over an abbreviation Dictionary. It answers SentenceDetectorME.isAcceptableBreak(CharSequence, int, int) by enumerating the few text positions that can carry a relevant abbreviation and asking a hash set what is there, instead of searching the whole text once per dictionary entry.
  • Field Details

    • entries

      private final Set<String> entries
      The dictionary entries, folded to lower case unless the dictionary is case-sensitive. Only the first token of a multi-token entry participates, as before.
    • entryLengths

      private final int[] entryLengths
      The distinct entry lengths, ascending. These are the only window sizes worth probing.
    • maxEntryLength

      private final int maxEntryLength
      The longest entry length, that is how far past a candidate a relevant match can reach.
    • caseSensitive

      private final boolean caseSensitive
      Whether the dictionary matches case-sensitively.
  • Constructor Details

    • AbbreviationIndex

      private AbbreviationIndex(Dictionary abbDict)
      Reads abbDict once, so this instance is immutable and safe to share.
      Parameters:
      abbDict - The Dictionary to index. Must not be null.
  • Method Details

    • of

      Parameters:
      abbDict - The Dictionary to index, may be null.
      Returns:
      An index over abbDict, or null if abbDict is null.
    • allowsBreak

      boolean allowsBreak(CharSequence s, int fromIndex, int candidateIndex)
      Parameters:
      s - The text in which the break occurred.
      fromIndex - The start of the segment currently being evaluated.
      candidateIndex - The index of the candidate sentence ending.
      Returns:
      true if a break at candidateIndex is allowed.
    • toLowerCase

      private static String toLowerCase(CharSequence s, int from, int to)
      Lower-cases [from, to) exactly as StringUtil.toLowerCase(CharSequence) lower-cases a whole text: per code point via Character.toLowerCase(int).

      Folding only the window is correct only because that mapping is 1:1 in char count, so pos - windowStart taken from the unfolded text still indexes the folded window. String.toLowerCase() must not be used here: full case mapping can expand a code point (for example İ / U+0130) and would corrupt every offset in the window.

      Parameters:
      s - The text to read from.
      from - The first index to fold, at a code point boundary.
      to - The index to stop at, at a code point boundary.
      Returns:
      The folded characters of [from, to).
    • codePointStart

      private static int codePointStart(CharSequence s, int index)
      Parameters:
      s - The text index refers to.
      index - The index to align.
      Returns:
      index, moved one character left if it points at the trailing half of a surrogate pair, which is the only index a code point walk cannot start at.
    • codePointEnd

      private static int codePointEnd(CharSequence s, int index)
      Parameters:
      s - The text index refers to.
      index - The index to align.
      Returns:
      index, moved one character right if it splits a surrogate pair, which is the only index a code point walk cannot stop at.