Ling 573 - Natural Language Processing Systems and Applications
Spring 2014
Deliverable #2: Baseline Question-Answering System
Code and Results due: April 25, 2014: 23:59
Updated Project Report due: April 29, 2014: 09:00

Goals

In this deliverable, you will implement a baseline question-answering system. You will

Baseline end-to-end system

For this deliverable, you will need to implement a baseline end-to-end system. You should build on approaches presented in class and readings. You may implement any effective strategy, but you are encouraged to implement a redundancy based/ or web-based boosting strategy such as those in the AskMSR or ARANEA systems. Your system should include:

Since this is a baseline system, it is not expected that your system will be as elaborate as those presented in class You should concentrate on "connectivity first": get the system to work end-to-end first, and then work on refinements.

Retrieval

For this deliverable, you will need to implement a retrieval system. TREC QA 'strict' evaluation requires document collection support for all answers. Thus, web-based answers can be projected onto documents in the collection, and collection retrieved results can be confirmed or reranked based on web-based results. You may build on techniques presented in class, described in the reading list, and proposed in other research articles.

You system must include indexing and retrieval based on a standard IR engine, such as those described in the resource list. In addition, you system may exploit:

Data

Document Collection

The AQUAINT Corpus was employed as the document collection for the question-answering task for a number of years, and will form the basis of retrieval for this deliverable. The collection can be found on patas in /corpora/LDC/LDC02T31/.

Training Data

You may use any of the TREC question collections through 2005 for training your system. For 2003, 2004, and 2005 there are prepared gold standard documents and answer patterns to allow you to train and tune your Q/A system.

All pattern files appear in /dropbox/13-14/573/Data/patterns.

All question files appear in /dropbox/13-14/573/Data/Questions.

Training data appear in the training subdirectories.

Development Test Data

You should evaluate on the TREC-2006 questions and their corresponding documents and answer string patterns. You are only required to test on the factoid questions. Development test data appears in the devtest subdirectories.

Evaluation

You will employ the standard mean reciprocal rank (MRR) measure to evaluate the results from your baseline end-to-end question-answering ystem. These scores should be placed files called D2.results_strict and D2.results_lenientin the results directory. A simple script for calculating MRR based on the Litkowski pattern files and your outputs is provided in /dropbox/13-14/573/code/compute_mrr.py. It should be called as follows: python2.6 compute_mrr.py pattern_file D2.outputs {type} where

Outputs

Create one output file in the outputs directory, based on running your baseline question-answering system on the test data file. You should do this as follows:

Extending the project report

This extended version should include all the sections from the original report (with many still as stubs) and additionally include the following new material:

Please name your report D2.pdf.

Presentation

Your presentation may be prepared in any computer-projectable format, including HTML, PDF, PPT, and Word. Your presentation should take about 10 minutes to cover your main content, including: Your presentation should be deposited in your doc directory, but it is not due until the actual presentation time. You may continue working on it after the main deliverable is due.

Summary