Class AbstractDataIndexer

java.lang.Object
opennlp.tools.ml.model.AbstractDataIndexer
All Implemented Interfaces:
DataIndexer
Direct Known Subclasses:
OnePassDataIndexer, TwoPassDataIndexer

public abstract class AbstractDataIndexer extends Object implements DataIndexer
Abstract DataIndexer implementation for collecting event and context counts used in training.
See Also:
  • Field Details

    • logger

      private static final org.slf4j.Logger logger
    • SORT_PARAM

      public static final String SORT_PARAM
      See Also:
    • SORT_DEFAULT

      public static final boolean SORT_DEFAULT
      See Also:
    • trainingParameters

      protected TrainingParameters trainingParameters
    • reportMap

      protected Map<String,String> reportMap
    • numEvents

      private int numEvents
    • contexts

      protected int[][] contexts
      The integer contexts associated with each unique event.
    • outcomeList

      protected int[] outcomeList
      The integer outcome associated with each unique event.
    • numTimesEventsSeen

      protected int[] numTimesEventsSeen
      The number of times an event occurred in the training data.
    • predLabels

      protected String[] predLabels
      The predicate/context names.
    • outcomeLabels

      protected String[] outcomeLabels
      The names of the outcomes.
    • predCounts

      protected int[] predCounts
      The number of times each predicate occurred.
  • Constructor Details

    • AbstractDataIndexer

      public AbstractDataIndexer()
  • Method Details

    • init

      public void init(TrainingParameters indexingParameters, Map<String,String> reportMap)
      Sets parameters used during the data indexing.
      Specified by:
      init in interface DataIndexer
      Parameters:
      indexingParameters - The TrainingParameters to be used.
      reportMap - The Map used for reporting.
    • getContexts

      public int[][] getContexts()
      Specified by:
      getContexts in interface DataIndexer
      Returns:
      Retrieves a 2-dimensional array whose first dimension is the event index and array this refers to contains the contexts for that event.
    • getNumTimesEventsSeen

      public int[] getNumTimesEventsSeen()
      Specified by:
      getNumTimesEventsSeen in interface DataIndexer
      Returns:
      Retrieves an array indexed by the event index indicating the number of times a particular event was seen.
    • getOutcomeList

      public int[] getOutcomeList()
      Specified by:
      getOutcomeList in interface DataIndexer
      Returns:
      Retrieves an array indicating the outcome index for each event.
    • getPredLabels

      public String[] getPredLabels()
      Specified by:
      getPredLabels in interface DataIndexer
      Returns:
      Retrieves an array of predicate/context names indexed by context index. These indices are the value of the array returned by DataIndexer.getContexts().
    • getOutcomeLabels

      public String[] getOutcomeLabels()
      Specified by:
      getOutcomeLabels in interface DataIndexer
      Returns:
      Retrieves an array of outcome names indexed by outcome index.
    • getPredCounts

      public int[] getPredCounts()
      Specified by:
      getPredCounts in interface DataIndexer
      Returns:
      Retrieves an array of the count of each predicate in the events.
    • getNumEvents

      public int getNumEvents()
      Specified by:
      getNumEvents in interface DataIndexer
      Returns:
      Retrieves the number of total events indexed.
    • sortAndMerge

      protected int sortAndMerge(List<ComparableEvent> eventsToCompare, boolean sort) throws InsufficientTrainingDataException
      Sorts and uniques the array of comparable events and return the number of unique events. This method will alter the eventsToCompare list.

      It does an in place sort, followed by an in place edit to remove duplicates.

      Parameters:
      eventsToCompare - The List<ComparableEvent> events used as input.
      sort - Whether to use sorting, or not.
      Returns:
      The number of unique events in the specified list.
      Throws:
      InsufficientTrainingDataException - Thrown if not enough events are provided
      Since:
      maxent 1.2.6
    • index

      protected List<ComparableEvent> index(ObjectStream<Event> events, Map<String,Integer> predicateIndex) throws IOException
      Performs the data indexing.

      Note: Make sure the init(TrainingParameters, Map) method is called first.

      Parameters:
      events - A ObjectStream<Event> of events used as input.
      predicateIndex - A Map providing the data of a predicate index.
      Throws:
      IOException - Thrown if IO errors occurred during indexing.
    • update

      protected static void update(String[] ec, Map<String,Integer> counter)
      Updates the Map of predicates and counter with the specified event contexts.
      Parameters:
      ec - The contexts/features which occur in an event.
      counter - The predicate counters in form of a Map.
    • toIndexedStringArray

      protected static String[] toIndexedStringArray(Map<String,Integer> labelToIndexMap)
      Utility method for creating a String[] from a map whose keys are labels (Strings) to be stored in the array and whose values are the indices (Integers) at which the corresponding labels should be inserted.
      Parameters:
      labelToIndexMap - A Map that holds labels to index positions.
      Returns:
      The resulting String[].
    • getValues

      public float[][] getValues()
      Specified by:
      getValues in interface DataIndexer
      Returns:
      Retrieves the values associated with each event context or null if integer values are to be used.