Class DefaultTokenContextGenerator

java.lang.Object
opennlp.tools.tokenize.DefaultTokenContextGenerator
All Implemented Interfaces:
TokenContextGenerator

public class DefaultTokenContextGenerator extends Object implements TokenContextGenerator
A default TokenContextGenerator which produces events for maxent decisions for tokenization.
  • Field Details

    • inducedAbbreviations

      protected final Set<String> inducedAbbreviations
  • Constructor Details

    • DefaultTokenContextGenerator

      public DefaultTokenContextGenerator()
      Initializes a plain DefaultTokenContextGenerator instance.
    • DefaultTokenContextGenerator

      public DefaultTokenContextGenerator(Set<String> inducedAbbreviations)
      Initializes a customized DefaultTokenContextGenerator instance via a set of inducedAbbreviations.
      Parameters:
      inducedAbbreviations - The induced abbreviations to be used for this instance.
  • Method Details

    • getContext

      public String[] getContext(String sentence, int index)
      Specified by:
      getContext in interface TokenContextGenerator
      Parameters:
      sentence - The string that represents a sentence.
      index - The index to consider splitting tokens.
      Returns:
      An array of features for a sentence at the specified index.
    • createContext

      protected List<String> createContext(String sentence, int index)
      Computes a List of features for the specified sentence at the specified index. Extensions of DefaultTokenContextGenerator can override this method to create a customized behaviour.
      Parameters:
      sentence - The sentence to create features for.
      index - The positional index. Must be a non-negative number or 0.
      Returns:
      A List of features for the specified sentence string at the specified index.
    • addCharPreds

      protected void addCharPreds(String key, char c, List<String> preds)
      Helper function for createContext(String, int) that appends to a given key a fixed text sequence depending on c. The resulting combination is added to the given list preds.
      Parameters:
      key - The input string to process.
      c - A character used to discriminate which fixed text shall be appended.
      preds - The list into which the resulting combinations will be added.