Ling 573 - Natural Language Processing Systems and Applications
Spring 2016
Deliverable #3: Summarization Improvement; Information Ordering
Code and Results due: May 15, 2016: 23:45
Updated Project Report due: May 17, 2016: 09:00


Goals

In this deliverable, you will continue development and improvement of your summarization system. You will

Information Ordering

For this deliverable, one focus will be on improving your base summarization systems through enhanced information ordering. Information ordering can address

You may build on techniques presented in class, described in the reading list, and proposed in other research articles.

Content Selection Improvement

You should continue to revise and improve your content selection approach to enhance your summarization system. One strategy to do so in the context of TAC is through topic-focused summarization, as discussed below.

Topic-focused summarization

The TAC summarization task is a topic-focused, or "guided", summarization task. Summaries are expected to focus on the topic, specified by the title element given in test topics XML file, and address the relevant aspects for the corresponding category. Most approaches augment existing content selection strategies to further focus on the desired topics. You may build on approaches presented in lecture or readings.

Data

We will be focusing on the TAC summarization shared task. We will use one year's data as devtest for most of the term, and then use a new unseen year's data as final evaltest in the last deliverable.

Document Collection

The AQUAINT and AQUAINT-2 Corpora have been employed as the document collections for the summarization task for a number of years, and will form the basis of summarization for this deliverable. The collections can be found on patas in /corpora/LDC/LDC02T31/ (AQUAINT, 1996-2000) and /corpora/LDC/LDC08T25/ (AQUAINT-2, 2004-2006).

Training Data

You may use any of the DUC or TAC summarization data through 2009 for training and developing your system. For previous years, there are prepared document sets and model summaries to allow you to train and tune your summarization system.

All model files appear in /dropbox/14-15/573/Data/models.

All document specification files appear in /dropbox/14-15/573/Data/Documents.

Training data appear in the training subdirectories and devtest data in the devtest directory.

Development Test Data

You should evaluate on the TAC-2010 topic-oriented document sets and their corresponding model summaries. You should only evaluate your system on the the 'A' sets. Development test data appears in the devtest subdirectories.

Evaluation

You will employ the standard automatic ROUGE method to evaluate the results from your summarization ystem. Code implementing the ROUGE metric is provided in /dropbox/14-15/573/code/ROUGE/ROUGE-1.5.5.pl. Example configuration files are given.

Outputs

Create a directory D3 under the outputs directory containing the summaries based on running your updated summarization system on the test data file. You should do this as follows:

Extending the project report

This extended version should include all the sections from the original report with some content in all sections. You should especially focus on the following new material:

Please name your report D3.pdf.

Presentation

Your presentation may be prepared in any computer-projectable format, including HTML, PDF, PPT, and Word. Your presentation should take about 10 minutes to cover your main content, including: Your presentation should be deposited in your doc directory, but it is not due until the actual presentation time. You may continue working on it after the main deliverable is due.

Summary