Class MascDocument

java.lang.Object
opennlp.tools.formats.masc.MascDocument

public class MascDocument extends Object
  • Field Details

    • logger

      private static final org.slf4j.Logger logger
    • sentences

      private final List<MascSentence> sentences
    • pathToFile

      private final String pathToFile
    • sentenceIterator

      private Iterator<MascSentence> sentenceIterator
    • hasPennTags

      private boolean hasPennTags
    • hasNamedEntities

      private boolean hasNamedEntities
  • Constructor Details

  • Method Details

    • parseDocument

      public static MascDocument parseDocument(String path, InputStream f_primary, InputStream f_seg, InputStream f_penn, InputStream f_s, InputStream f_ne) throws IOException
      Initializes a MascDocument with all the stand-off annotations translated into the internal structure.
      Parameters:
      path - The path where the document header is.
      f_primary - The file with the raw corpus text.
      f_seg - The file with segmentation into quarks.
      f_penn - The file with tokenization and Penn POS tags produced by GATE-5.0 ANNIE application.
      f_s - The file with sentence boundaries.
      f_ne - The file with named entities.
      Returns:
      A document containing the text and its annotations. Immutability is not guaranteed yet.
      Throws:
      IOException - if the raw data cannot be read or the alignment of the raw data with annotations fails
    • readText

      private static String readText(InputStream stream) throws IOException
      Reads in the corpus file text.
      Parameters:
      stream - A valid, open stream for a corpus file.
      Returns:
      The text of the file.
      Throws:
      IOException - Thrown if IO errors occurred.
    • parseWords

      private static List<MascWord> parseWords(InputStream f_seg) throws IOException
      Parses the word segmentation stand-off annotation
      Parameters:
      f_seg - A valid, open stream for a file with segmentation.
      Returns:
      A list of individual quarks, expressed as MascWord-s
      Throws:
      IOException - Thrown if IO errors occurred.
    • parseSentences

      private static List<Span> parseSentences(InputStream f_s) throws IOException
      Parses the sentence annotation file, align it with the raw text
      Parameters:
      f_s - A valid, open stream for a sentence annotation file.
      Returns:
      The List<Span> delimiting each sentence.
      Throws:
      IOException - if the sentence file cannot be parsed or closed
    • parsePennTags

      private static Map<String, Map<Integer,?>> parsePennTags(InputStream f_penn) throws IOException
      Parses the Penn-POS (GATE5-ANNIE) stand-off annotation.
      Parameters:
      f_penn - A valid, open stream for a file with Penn POS tags.
      Returns:
      A map of three sub-maps: tokenToTag, from Penn token ID (int) to Penn POS-tag, tokenToBase, from Penn token ID (int) to the base and tokenToQuarks, from Penn token ID (int) to a List of quark IDs contained in that token.
      Throws:
      IOException - Thrown if IO errors occurred.
    • parseNamedEntity

      private static Map<String, Map<Integer,?>> parseNamedEntity(InputStream f_ne) throws IOException
      Parses the named entity stand-off annotation.
      Parameters:
      f_ne - A valid, open stream for a file with named entity annotations.
      Returns:
      A map with two sub-maps, entityIDtoEntityType, mapping entity ID integers to entity type Strings, and entityIDsToTokens, mapping entity ID integers to Penn token ID integers.
      Throws:
      IOException - Thrown if IO errors occurred.
    • combineAnnotations

      private static List<MascSentence> combineAnnotations(String text, List<Span> sentenceSpans, List<MascWord> words) throws IOException
      Combines the raw text with annotations that every file should have.
      Parameters:
      text - The raw text.
      sentenceSpans - The spans defining individual sentences. Overlaps are not permitted.
      words - The quarks of the raw text.
      Returns:
      A list of sentences, each of which is a list of quarks. Some quarks may belong to more than one sentence. Quarks which do not belong to a single sentence are silently dropped.
      Throws:
      IOException - If sentences and quarks cannot be aligned.
    • addNamedEntityTags

      private void addNamedEntityTags(Map<String, Map<Integer, ? extends Object>> namedEntities)
      Attaches the named entity labels to individual tokens.
      Parameters:
      namedEntities - A map with two sub-maps, entityIDtoEntityType, mapping entity ID integers to entity type Strings, and entityIDsToTokens, mapping entity ID integers to Penn token ID integers
    • addPennTags

      private void addPennTags(Map<String, Map<Integer,?>> tagMaps) throws IOException
      Attach tags and bases to MascWords in each of the sentences.
      Parameters:
      tagMaps - A map of three sub-maps: tokenToTag, from Penn token ID (int) to Penn POS-tag, * tokenToBase, from Penn token ID (int) to the base and tokenToQuarks, from Penn token ID * (int) to a List of quark IDs contained in that token.
      Throws:
      IOException
    • hasPennTags

      public boolean hasPennTags()
      Checks whether there is Penn tagging produced by GATE-5.0 ANNIE.
      Returns:
      true if this file has aligned tags/tokens, false otherwise.
    • hasNamedEntities

      public boolean hasNamedEntities()
      Checks whether there is NER by GATE-5.0 ANNIE.
      Returns:
      true if this file has named entities, false otherwise.
    • read

      public MascSentence read()
      Returns:
      Retrieves the next sentence or null if end of document reached.
    • reset

      public void reset()
      Resets the reading of sentences to the beginning of the document.