Stanford Core NLP

Home Page Download Demo FAQ

Overview

The Stanford Core NLP toolkit is a collection of tools for processing natural language text in several languages. It covers a wide range of standard NLP tasks including tokenization, sentence splitting, part of speech tagging, named entity recognition, parsing, sentiment analysis, co-reference resolution and more. See here for a more detailed list of annotators available in the toolkit.

The tools are written in Java (1.8), but third party bindings for many other languages have also been created. The most flexible way to apply the tools is through the API, which is well documented for most standard use cases on the home page and in the javadocs. Stanford also provides a command line interface to the toolkit, which is often easier to use but has some caveats.

For noncommercial the entire toolkit is released under the GPL (v3 or later). Commercial licensing is also available upon request (see licensing).

Usage

Command Line Interface

The easiest way to get started with the command line interface is to download the toolkit and follow the instructions on the CoreNLP website. A brief summary is provided below.

Download and extract the toolkit to a directory of your choosing and then navigate to that directory in a terminal. The following command illustrates basic usage of the toolkit (note the command is split across multiple lines):

java -cp "*" -Xmx2G edu.stanford.nlp.pipeline.StanfordCoreNLP  \
    -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref     \
    -file input.txt                                            \
    -outputDirectory /path/to/some/directory                   \
    -outputFormat json
  • The -cp option tells the Java VM where to look for the libraries necessary to run the program. In this case the wildcard "*" (the quotes are necessary) tells java to use all jar files in the current directory.
  • The -Xmx2G option tells the Java VM what the maximum amount of memory it can use to run the program. Depending on the options used for the toolkit and your data you will probably need to increase this value.
  • edu.stanford.nlp.pipeline.StanfordCoreNLP specifies the main Java class to run, which implements the command line interface.
  • -annotators tells the toolkit which annotators to apply to the input data. For more complex configurations it is preferable to configure the toolkit through a properties file instead (see the documentation).
  • -file input.text tells the toolkit to process a single document from the file input.txt. To process a list of files use the option -filelist instead.
  • -outputDirectory tells the system where to output the results (otherwise they are written to the current directory by default).
  • The -outputFormat option allows you to specify the output format among several possible options. Note many of these formats do not preserve the full information available in the API and can be one reason to prefer the API over the CLI. The recently introduced serialized format using Google’s protocol buffer format improves the situation but is not completely supported across languages.

A sample of the output in json format looks like this:

{
  "sentences": [
    {
      "index": 0,
      "basicDependencies": [
        {
          "dep": "ROOT",
          "governor": 0,
          "governorGloss": "ROOT",
          "dependent": 4,
          "dependentGloss": "located"
        },
        ...
      ],
      ...
      "tokens": [
        {
          "index": 1,
          "word": "Stanford",
          "originalText": "Stanford",
          "lemma": "Stanford",
          "characterOffsetBegin": 0,
          "characterOffsetEnd": 8,
          "pos": "NNP",
          "ner": "ORGANIZATION",
          "speaker": "PER0",
          "before": "",
          "after": " "
        },
        ...
      ]
    }
  ]
}

API

The best way to become familiar with the API is to read the API tutorial, annotator reference, javadocs and source code. The two core class for working with the pipeline API are the Annotator and the Annotation. The Annotator classes are what does the actual processing and you usually do not have to work with them directly. The Annotation class is typesafe map that stores the results of the annotators. You access the results of the annotators by passing the appropriate key to the get method of the map. Which keys to use for each annotator is documented on the annotators summary page.

An example program adapted from the CoreNLP API tutorial is given below to show some of the basic functionality.

import edu.stanford.nlp.pipeline.*;
import java.util.*;

public class BasicPipelineExample {

    public static void main(String[] args) {

        // creates a StanfordCoreNLP object, with POS tagging, lemmatization, NER, parsing, and coreference resolution
        // Note, it is typical to load these from a properties file instead of hard coding them in the program like this.
        Properties props = new Properties();
        props.setProperty("annotators", "tokenize, ssplit, pos, lemma, ner, parse, dcoref");
        StanfordCoreNLP pipeline = new StanfordCoreNLP(props);

        // The texts would usually be loaded from elsewhere (e.g., from disk, over a socket, etc.)
        List<String> texts = Arrays.asList("The first document you want to process.", "The second document you want to process.");

        // For each text, you create a skeleton Annotation (map) object that has the TextAnnotation.class set to the text given in the constructor.
        List<Annotation> documents = texts.stream().map(t -> new Annotation(t)).collect(Collectors.toList());

        // Run all Annotators on the documents in parallel (you can also call this with a single Annotation if that is all you need)
        // Note this changes the input document annotations by adding the key value pairs determined by the annotators specified above.
        pipeline.annotate(documents);

        // You retrieve the annotations using the appropriate key on the desired annotation
        // For example, this gets the raw text of the first document
        Annotation annotation = documents.get(0);
        String originalText = annotation.get(TextAnnotation.class);

        // This gets all the sentences in the document
        // A CoreMap is also a typesafe map that uses class objects as keys and has values with custom types (tied to the key). 
        // (Note an Annotation is a subclass of CoreMap)
        List<CoreMap> sentences = annotation.get(SentencesAnnotation.class);

        for (CoreMap sentence : sentences) {
            // Loop through the tokens of the current sentence (determined by the tokenizer annotator)
            // A CoreLabel is also a CoreMap with some additional token specific methods
            // Note that not all of the methods will return valid data depending on which annotators were specified in the properties.
            // For example, you must specify the ner annotator for the associated ner method to return a valid value.
            for (CoreLabel token : sentence.get(TokensAnnotation.class)) {
              // The text of the token
              String word = token.get(TextAnnotation.class); // Or alternatively token.word()

              // The part of speech tag of the word (using the Penn Tagset)
              String pos = token.get(PartOfSpeechAnnotation.class); // Or alternatively token.tag();

              // The NER label of the token (e.g., PERSON, ORGANIZATION, LOCATION)
              String ner = token.get(NamedEntityTagAnnotation.class); // Or alternatively token.ner();
            }

            // The (constituency) parse tree of the current sentence
            Tree tree = sentence.get(TreeAnnotation.class);

            // this is the Stanford dependency graph of the current sentence
            SemanticGraph dependencies = sentence.get(CollapsedCCProcessedDependenciesAnnotation.class);

            // This is the coreference link graph
            // Each chain stores a set of mentions that link to each other,
            // along with a method for getting the most representative mention
            // Both sentence and token offsets start at 1!
            Map<Integer, CorefChain> graph = document.get(CorefChainAnnotation.class);
        }
    }
}

Server

The final way to interact with the pipeline is through the built in server. Accessing the pipeline via the server prevents you from having to load the models on each request and the server can be put on a remote machine with adequate computing resources if available.

The server can be started as follows:

java -Xmx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 9000 -timeout 15000

There are three basic ways to interact with the server once it is running.

  • Through a web GUI at http://localhost:9000 (or whatever hostname or port you specified). This is the same GUI as the online demo, but running on your own machine. This allows you to test and inspect the output of manually input texts.
  • Through POST requests. As illustrated by the following wget command (note the command is split across multiple lines): 
    wget                                                          \
  --post-data 'The quick brown fox jumped over the lazy dog.' \
  'localhost:9000/?properties={"annotators":"tokenize,ssplit,pos","outputFormat":"json"}'
  • Through a Java client that mirrors the Java API as closely as possible. See the documentation for more details.

Things To Watch Out For

Several of the models can take a significant amount of time to load (10-30 seconds). This can be a significant problem if you do not set up your processing pipeline appropriately (i.e., in large batches). Some of the 3rd party bindings also suffer from this problem, for example the python bindings that are part of NLTK, which can make them difficult to use when jobs can’t easily be batched.

Also, depending on how you output the results from the command line interface, it is not guaranteed to be a lossless conversion from the full results of the annotators used in the underlying API. For example, the XML output does not contain all the information available about a constituency parse (e.g., the score). The serialized option or the ProtobufAnnotationSerializer with the API helps with this problem, but reading the serialized output in other languages is not fully supported. Also be aware that if you choose one of the other output options then it is generally up to you to write code to parse the output which can be a significant amount of tedious and bug prone work.

As of version 3.6 (it’s not clear if it is still a problem in 3.7) there were places in the code that called System.exit when an error state was reached making it impossible to gracefully handle the error and continue processing additional input.