Class MascDocument
java.lang.Object
opennlp.tools.formats.masc.MascDocument
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate booleanprivate booleanprivate static final org.slf4j.Loggerprivate final Stringprivate Iterator<MascSentence> private final List<MascSentence> -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate voidAttaches the named entity labels to individual tokens.private voidaddPennTags(Map<String, Map<Integer, ?>> tagMaps) Attach tags and bases to MascWords in each of the sentences.private static List<MascSentence> Combines the raw text with annotations that every file should have.booleanChecks whether there is NER by GATE-5.0 ANNIE.booleanChecks whether there is Penn tagging produced by GATE-5.0 ANNIE.static MascDocumentparseDocument(String path, InputStream f_primary, InputStream f_seg, InputStream f_penn, InputStream f_s, InputStream f_ne) Initializes aMascDocumentwith all the stand-off annotations translated into the internal structure.parseNamedEntity(InputStream f_ne) Parses the named entity stand-off annotation.parsePennTags(InputStream f_penn) Parses the Penn-POS (GATE5-ANNIE) stand-off annotation.Parses the sentence annotation file, align it with the raw textparseWords(InputStream f_seg) Parses the word segmentation stand-off annotationread()private static StringreadText(InputStream stream) Reads in the corpus file text.voidreset()Resets the reading of sentences to the beginning of the document.
-
Field Details
-
logger
private static final org.slf4j.Logger logger -
sentences
-
pathToFile
-
sentenceIterator
-
hasPennTags
private boolean hasPennTags -
hasNamedEntities
private boolean hasNamedEntities
-
-
Constructor Details
-
MascDocument
-
-
Method Details
-
parseDocument
public static MascDocument parseDocument(String path, InputStream f_primary, InputStream f_seg, InputStream f_penn, InputStream f_s, InputStream f_ne) throws IOException Initializes aMascDocumentwith all the stand-off annotations translated into the internal structure.- Parameters:
path- The path where the document header is.f_primary- Thefilewith the raw corpus text.f_seg- Thefilewith segmentation into quarks.f_penn- Thefilewith tokenization and Penn POS tags produced by GATE-5.0 ANNIE application.f_s- Thefilewith sentence boundaries.f_ne- Thefilewith named entities.- Returns:
- A document containing the text and its annotations. Immutability is not guaranteed yet.
- Throws:
IOException- if the raw data cannot be read or the alignment of the raw data with annotations fails
-
readText
Reads in the corpus file text.- Parameters:
stream- A valid, openstreamfor a corpus file.- Returns:
- The text of the file.
- Throws:
IOException- Thrown if IO errors occurred.
-
parseWords
Parses the word segmentation stand-off annotation- Parameters:
f_seg- A valid, openstreamfor a file with segmentation.- Returns:
- A list of individual quarks, expressed as MascWord-s
- Throws:
IOException- Thrown if IO errors occurred.
-
parseSentences
Parses the sentence annotation file, align it with the raw text- Parameters:
f_s- A valid, openstreamfor a sentence annotation file.- Returns:
- The
List<Span>delimiting each sentence. - Throws:
IOException- if the sentence file cannot be parsed or closed
-
parsePennTags
Parses the Penn-POS (GATE5-ANNIE) stand-off annotation.- Parameters:
f_penn- A valid, openstreamfor a file with Penn POS tags.- Returns:
- A map of three sub-maps: tokenToTag, from Penn token ID (int) to Penn POS-tag, tokenToBase, from Penn token ID (int) to the base and tokenToQuarks, from Penn token ID (int) to a List of quark IDs contained in that token.
- Throws:
IOException- Thrown if IO errors occurred.
-
parseNamedEntity
Parses the named entity stand-off annotation.- Parameters:
f_ne- A valid, openstreamfor a file with named entity annotations.- Returns:
- A map with two sub-maps, entityIDtoEntityType, mapping entity ID integers to entity type Strings, and entityIDsToTokens, mapping entity ID integers to Penn token ID integers.
- Throws:
IOException- Thrown if IO errors occurred.
-
combineAnnotations
private static List<MascSentence> combineAnnotations(String text, List<Span> sentenceSpans, List<MascWord> words) throws IOException Combines the raw text with annotations that every file should have.- Parameters:
text- The raw text.sentenceSpans- The spans defining individual sentences. Overlaps are not permitted.words- The quarks of the raw text.- Returns:
- A list of sentences, each of which is a list of quarks. Some quarks may belong to more than one sentence. Quarks which do not belong to a single sentence are silently dropped.
- Throws:
IOException- If sentences and quarks cannot be aligned.
-
addNamedEntityTags
Attaches the named entity labels to individual tokens.- Parameters:
namedEntities- A map with two sub-maps, entityIDtoEntityType, mapping entity ID integers to entity type Strings, and entityIDsToTokens, mapping entity ID integers to Penn token ID integers
-
addPennTags
Attach tags and bases to MascWords in each of the sentences.- Parameters:
tagMaps- A map of three sub-maps: tokenToTag, from Penn token ID (int) to Penn POS-tag, * tokenToBase, from Penn token ID (int) to the base and tokenToQuarks, from Penn token ID * (int) to a List of quark IDs contained in that token.- Throws:
IOException
-
hasPennTags
public boolean hasPennTags()Checks whether there is Penn tagging produced by GATE-5.0 ANNIE.- Returns:
trueif this file has aligned tags/tokens,falseotherwise.
-
hasNamedEntities
public boolean hasNamedEntities()Checks whether there is NER by GATE-5.0 ANNIE.- Returns:
trueif this file has named entities,falseotherwise.
-
read
- Returns:
- Retrieves the next sentence or
nullif end of document reached.
-
reset
public void reset()Resets the reading of sentences to the beginning of the document.
-