Ling 573 - Natural Language Processing Systems and Applications
Spring 2017
Deliverable #4: Final Summarization Systems
Code and Results due: May 28, 2017: 23:45
Final Report due: June 5, 2017: 17:00


In this deliverable, you will complete development of your summarization system. You will

System Enhancement

This final deliverable must include substantive enhancements beyond your baseline system and further extensions over your D3 system.

Content Realization

For this deliverable, one focus will be on improving your systems through enhanced content realization. Content realization can address:

You may build on techniques presented in class, described in the reading list, and proposed in other research articles.

We will also be conducting a manual readability evaluation in addition to the ROUGE content scoring, to give improvements in this area due credit.

High-level System Behavior

We will be focusing on the TAC (Text Analytics Conference) summarization shared task. Your system will:

Document Sets

Document sets to be summarized are provided in NIST standard XML files, identifying a set of topics consisting of: NOTE: You should only evaluate your system on the the 'A' document sets. The 'B' sets were designed to evaluate so-called "update" summaries.

NOTE: The format of the document IDs and the organization of the three corpora exhibit some differences. You may include/hardcode information about this structure in your system directly, or in a configuration file as you choose.

Summary Outputs

Your system should produce one summary output file per document set (topic), structured as described below:


You will employ the standard automatic ROUGE method to evaluate the results from your base end-to-end summarization system. Code implementing the ROUGE metric is provided in /dropbox/16-17/573/code/ROUGE/ Example configuration files are given. You will need to modify the configuration file to reference your own system's summary output.


Training, Development Test, Example and Evaluation Test Files

We are focusing on the TAC summarization shared task. Since this is the final deliverable, you will evaluate not only on the 2010 devtest data you have used all term, but also on held-out test data.

Devtest Document Collections

The AQUAINT and AQUAINT-2 Corpora have been employed as the document collections for the summarization task for a number of years, and will form the basis of summarization for this deliverable. The collections can be found on patas in

Evaltest Document Collection

The held-out document sets for the final evaluation are drawn from the English Gigaword corpus, from years 2007 and 2008. This collection may be found on patas in: (Note: Given the size of this corpus, it's still fine if you use the main corpus as your background corpus.)

Core Files

The core training, development test, and evaluation files can be found in /dropbox/16-17/573/ on the CL cluster.

<file_set_type> ranges over:

Training Data

You may use any of the DUC or TAC summarization data through 2009 for training and developing your system. For previous years, there are prepared document sets and model summaries to allow you to train and tune your summarization system.

Development Test Data

You should evaluate on the TAC-2010 topic-oriented document sets and their corresponding model summaries. You should only evaluate your system on the the 'A' sets. Development test data appears in the devtest subdirectories.

Evaluation Test Data

You should also evaluate on the TAC-2011 topic-oriented document sets and their corresponding model summaries, again only on the 'A' sets. This evaluation test data appears in the evaltest subdirectories.

Evaluation Example Files

A variety of example files are provided to help you familiarize yourself with the ROUGE evaluation software.

Specific Submission Files

In addition to your source code and resources needed to support your system, your repository should include the following:

Your System Generated Summarization Files

Create two directories under the outputs directory containing the summaries based on running your final summarization system as below:

Extended Project Report

../doc/D4.pdf: This final version should include all required sections, as well as a complete system architecture description and proper bibliography including all and only the papers you have actually referenced. See this document for full details. The final version of your project report must explicitly include:


../doc/D4_presentation.{pdf|pptx|etc}: Your presentation may be prepared in any computer-projectable format, including HTML, PDF, PPT, and Word. Your presentation should take about 10 minutes to cover your main content, including: Your presentation should be deposited in your doc directory, but it is not due until the actual presentation time. You may continue working on it after the main deliverable is due.