Class TwoPassDataIndexer

java.lang.Object
opennlp.tools.ml.model.AbstractDataIndexer
opennlp.tools.ml.model.TwoPassDataIndexer
All Implemented Interfaces:
DataIndexer

public class TwoPassDataIndexer extends AbstractDataIndexer
Collecting event and context counts by making two passes over the events.

The first pass determines which contexts will be used by the model, and the second pass creates the events in memory containing only the contexts which will be used. This greatly reduces the amount of memory required for storing the events. During the first pass a temporary event file is created which is read during the second pass.

See Also:
  • Field Details

    • logger

      private static final org.slf4j.Logger logger
  • Constructor Details

    • TwoPassDataIndexer

      public TwoPassDataIndexer()
  • Method Details

    • index

      public void index(ObjectStream<Event> eventStream) throws IOException
      Performs the data indexing.

      Note: Make sure the DataIndexer.init(TrainingParameters, Map) method is called first.

      Parameters:
      eventStream - A ObjectStream<Event> of events used as input.
      Throws:
      IOException - Thrown if IO errors occurred during indexing.
    • computeEventCounts

      private int computeEventCounts(ObjectStream<Event> eventStream, DataOutputStream eventStore, Map<String,Integer> predicatesInOut, int cutoff) throws IOException
      Reads events from eventStream into a linked list. The predicates associated with each event are counted and any which occur at least cutoff times are added to the predicatesInOut map along with a unique integer index.

      Protocol: 1 - (utf string) - Event outcome 2 - (int) - Event context array length 3+ - (utf string) - Event context string 4 - (int) - Event values array length 5+ - (float) - Event value

      Parameters:
      eventStream - an EventStream value
      eventStore - a writer to which the events are written to for later processing.
      predicatesInOut - a TObjectIntHashMap value
      cutoff - an int value
      Throws:
      IOException