Class TwoPassDataIndexer
java.lang.Object
opennlp.tools.ml.model.AbstractDataIndexer
opennlp.tools.ml.model.TwoPassDataIndexer
- All Implemented Interfaces:
DataIndexer
Collecting event and context counts by making two passes over the events.
The first pass determines which contexts will be used by the model, and the second pass creates the events in memory containing only the contexts which will be used. This greatly reduces the amount of memory required for storing the events. During the first pass a temporary event file is created which is read during the second pass.
- See Also:
-
Nested Class Summary
Nested Classes -
Field Summary
FieldsFields inherited from class AbstractDataIndexer
contexts, numTimesEventsSeen, outcomeLabels, outcomeList, predCounts, predLabels, reportMap, SORT_DEFAULT, SORT_PARAM, trainingParameters -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate intcomputeEventCounts(ObjectStream<Event> eventStream, DataOutputStream eventStore, Map<String, Integer> predicatesInOut, int cutoff) Reads events from eventStream into a linked list.voidindex(ObjectStream<Event> eventStream) Performs the data indexing.Methods inherited from class AbstractDataIndexer
getContexts, getNumEvents, getNumTimesEventsSeen, getOutcomeLabels, getOutcomeList, getPredCounts, getPredLabels, getValues, index, init, sortAndMerge, toIndexedStringArray, update
-
Field Details
-
logger
private static final org.slf4j.Logger logger
-
-
Constructor Details
-
TwoPassDataIndexer
public TwoPassDataIndexer()
-
-
Method Details
-
index
Performs the data indexing.Note: Make sure the
DataIndexer.init(TrainingParameters, Map)method is called first.- Parameters:
eventStream- AObjectStream<Event>of events used as input.- Throws:
IOException- Thrown if IO errors occurred during indexing.
-
computeEventCounts
private int computeEventCounts(ObjectStream<Event> eventStream, DataOutputStream eventStore, Map<String, Integer> predicatesInOut, int cutoff) throws IOExceptionReads events from eventStream into a linked list. The predicates associated with each event are counted and any which occur at least cutoff times are added to the predicatesInOut map along with a unique integer index.Protocol: 1 - (utf string) - Event outcome 2 - (int) - Event context array length 3+ - (utf string) - Event context string 4 - (int) - Event values array length 5+ - (float) - Event value
- Parameters:
eventStream- anEventStreamvalueeventStore- a writer to which the events are written to for later processing.predicatesInOut- aTObjectIntHashMapvaluecutoff- anintvalue- Throws:
IOException
-