Lab 2 (due 1/17 11:59 pm)

Overview

The goal of this lab is to test out an initial grammar, based on the automatically constructed grammar specification ("choices file"), and then improve on that choices file for two phenomena. Along the way, you will get your first experience with [incr tsdb()] and also do a small amount of testsuite development. You'll also begin to get familiar with the resources for your language.

This lab entails the following general steps:

1. Upload the choices file into the customization system and refine it if necessary until you are able to produce a grammar.
   • Document any fixes you had to make until you could get something working
2. Process the test corpus, using [incr tsdb()], the LKB, and the initial grammar
3. Examine the results of the test run for coverage, accuracy and ambiguity.
4. Choose two phenomena to improve in the choices file
5. Document those phenomena in a testsuite
6. Process the test corpus and your testsuite using [incr tsdb()], the LKB, and the initial grammar.
7. Examine the results of the second test run for coverage, accuracy and ambiguity, including as a diff to the original test run.
8. Write it all up :)

Back to top

Preliminaries

Canvas Discussion

In order to encourage people to get started early, this assignment requires that you post a question to the discussion on the course Canvas by Tuesday night. Examples: A question about something in these instructions that's unclear, a question about something in your grammar resources that's unclear, or a question about something in the customization system that's unclear. Or all of the above! Ask away :)

Version control

Please use version control for your work in this class. You may use any system you are comfortable with. My preferred system is still svn, so here are directions for svn, if you don't already have a preferred set up:

• Read the Treehouse wiki pages on subversion (svn).
• Request an acccount on lemur, our svn server, from linghelp-at-u (Brandon Graves)
• Decide on a project name (e.g., the name of your language) and create a directory of that name on your local computer.
• You'll put your test suite and choices files in the directory for now. (We'll put the grammars themselves under version control once we're done downloading them. Until then, it'll just be annoying, and all the information is in the choices files anyway.)
• From the directory above your project directory, invoke the following command:

  svn import project-name svn://lemur.ling.washington.edu/shared/project-name
  

  Where "project-name" should be the name of your directory (two instances).
• Move your old version of that directory out of the way:

  mv project-name project-name-old
  

• Check out a working copy from svn:

  svn checkout svn://lemur.ling.washington.edu/shared/project-name
  

• Check that it's the same as what you had:

  diff -r project-name-old project-name

• If so, delete the version that's not under svn:

  rm -r project-name-old

• As you make changes to your test suite, commit them to the svn repository:

  svn commit

• Before working on the test suite or choices file, run

  svn update

  to grab any changes your partner made.
• Added bonus: If you are working on separate machines (e.g., if you want to work in the Treehouse, say, to get my help in debugging something during my office hours), you can use your version control set up to keep the locations in sync.

Back to top

Starter grammar

The first task is to take the auto-generated choices file we have prepared and upload it into the Grammar Matrix customization questionnaire. Once you've done so, there will probably be some red asterisks indicating sections that need further answers. Please answer those sections (ask lots of questions on Canvas!) on the basis of your descriptive materials as best you can.

In the word order section, you can skip the auxiliaries if it's not already filled out, by saying "no" on that question for now. You may of course revise this answer (and any others) later.

Once you are free of red asterisks, download the grammar.

Back to top

Initial testsuite run

Create a test suite skeleton

1. Look at the contents of your grammar directory and locate the subdirectory called tsdb.
2. Make a subdirectory called corpus inside tsdb/skeletons for your test suite.
3. Edit tsdb/skeletons/Index.lisp to include a line for this directory, e.g.:

   (
   ((:path . "matrix") (:content . "matrix: A test suite created automatically from the test sentences given in the Grammar Matrix questionnaire."))
   ((:path . "corpus") (:content . "IGT provided by the linguist"))
   )
   

4. Copy the item file that will be available in the Google drive folder for your language to tsdb/skeletons/corpus/item.
   • Note: If your item file has more than 1000 lines it, it is recommended to break it into several smaller profiles. You can make one skeleton for each. You can also choose to work with just one of them if you prefer! It might be fun to test from time to time though and see how much of the data your grammar can parse.
5. Copy tsdb/skeletons/Relations to tsdb/skeletons/corpus/relations (notice the change from R to r).
6. The final directory structure should look like this:

   grammar/tsdb/skeletons/Index.lisp            (lists the testsuites)
   grammar/tsdb/skeletons/Relations             (master copy of the database schema)
   grammar/tsdb/skeletons/corpus/item             (file provided)
   grammar/tsdb/skeletons/corpus/relations        (copy of ../Relations)
   grammar/tsdb/home                            (directory to store test profiles)
   

Create and run an initial test suite instance

1. Start the lkb
2. Load your starter grammar. (The script file is in your-grammar-dir/lkb/script.)
3. Start [incr tsdb()] (within emacs, that's M-x itsdb)
4. In the [incr tsdb()] podium, select Options > Database Root and input the path to tsdb/home.
5. In the [incr tsdb()] podium, select Options > Skeleton Root and input the path to tsdb/skeletons.
6. Optional: For future use, you can set these variables ahead of time in a file called .tsdbrc in your home directory. It should contain these lines, with path names edited appropriately:

   (tsdb:tsdb  :home  "path-to-tsdb/home")
   (tsdb:tsdb  :skeleton  "path-to-tsdb/skeletons")
   

7. In the [incr tsdb()] podium, select File > Create. You should see your test suite in the menu there. Select it, and get a test suite instance. Post to Canvas if this doesn't work.
8. Make sure your grammar is loaded into the LKB.
9. Once you have a test suite instance, select it (by clicking on it), then do Process > All Items.
10. Explore the results, with functions such as Browse > Results, Analyze > Coverage, and Analyze > Overgeneration.
11. Be sure to save (i.e., not overwrite or delete) this test suite instance, as you'll be asked to turn it in.

Note If your tsdb/ directory is inside a shared folder on VirtualBox, it will not work.

Back to top

Explore the results

Collect the following information to provide in your write up:

1. How many items parsed?
2. What is the average number of parses per parsed item?
3. How many parses did the most ambiguous item receive?
4. What sources of ambiguity can you identify?
5. For 10 items (if you have at least that many parsing), do any of the parses look reasonable in the semantics? (Emily will demo in class on Wednesday.)

Back to top

Improve the choices file for two phenomena

Based on your analysis of the results of the preliminary run, choose two phenomena to improve in the choices file, from the following list. You're looking for phenomena that are affecting many sentences: either leading to lots of ambiguity or leading to lots of parse failure. I suspect that case, agreement, and morphotactics will be the most useful to attack first.

• pronouns (includes person/number/gender)
• case
• agreement
• Tense/aspect
• Morphotactics

Make sure you can parse individual sentences

Once you have created your starter grammar (or each time you create one, as you should iterate through grammar creation and testing a few times as you refine your choices), try it out on a couple of sentences interactively to see if it works:

1. Load the grammar into the LKB.
2. Using the parse dialog box (or 'C-c p' in emacs to get the parse command inserted at your prompt), enter a sentence to parse.
3. Examine the results. If it does parse, check out the semantics (pop-up menu on the little trees). If it doesn't look at the parse chart to see why not.
4. Problems with lexical rules and lexical entries often become apparent here, too: If the LKB can't find an analysis for one of your words, it will say so, and (obviously) fail to parse the sentence.

Note that the questionnaire has a section for test sentences. If you use this, then the parse dialog will be pre-filled with your test sentences.

Back to top

Create a testsuite for those phenomena

Create a small testsuite of examples, according to the general instructions for testsuites and the formatting instructions, illustrating the phenomena you worked on above. The testsuite should have both positive and negative examples, but doesn't need to be exhaustive (since we're working with test corpora this year), but you'll want both positive and negative examples for each of the phenomena you work on in this section. I expect these testsuites to have about 20 examples total by the end of this week, though you can do more if you find that useful. All examples should be simple enough that your grammar can parse them or fails to parse them because of the one thing that's wrong with them.

Create a test suite skeleton

1. Make a subdirectory called lab2 inside tsdb/skeletons for your test suite.
2. Edit tsdb/skeletons/Index.lisp to include a line for this directory, e.g.:

   (
   ((:path . "matrix") (:content . "matrix: A test suite created automatically from the test sentences given in the Grammar Matrix questionnaire."))
   ((:path . "corpus") (:content . "IGT provided by the linguist"))
   ((:path . "lab2") (:content . "Test suite collected for Lab 2."))
   )
   

3. Download the python script make_item, make sure it is executable, and run it on your test suite:

   make_item testsuite.txt

   Notes on make_item:

   • This script is going to be pretty picky about the format of your test suite. If you have questions, please post to Canvas (10 minute rule!).
   • It requires python3, which is on the current version of the Ubuntu+LKB appliance.
   • Alternatively, you can copy your testsuite and make_item over to patas and run there, or install python3 (from http://python.org/download) on your host OS (mac or windows), and run make_item outside VirtualBox.
   • If the above command is successful, testsuite.txt.item would be created in the working directory. If the testsuite contains errors, it's possible that a lot of output will appear on stderr. It maybe useful to redirect this into a file that you can use to go through and correct the errors one at a time. For example:

     ./make_item testsuite.txt item 2>errs

     The command just above attempts to create 'item' in the working directory, and stderr messages are redirected to the file 'errs'.

     make_item contains a default mapping from testsuite line types into particular fields of the [incr_tsdb()] item file. The default mapping puts 'orth' into 'i-input', the field which the is the input to the grammar. If your grammar targets a different testsuite line, override the default mapping with the -m/--map option.

     ./make_item --map orth-seg i-input testsuite.txt item

     The invocation above maps the orth-seg line into the input field.

     You can run make_item with -h/--help to see a summary of the options.

4. Copy the .item file which is output by make_item to tsdb/skeletons/lab2/item.
5. Copy tsdb/skeletons/Relations to tsdb/skeletons/lab2/relations (notice the change from R to r).

Run both the test corpus and the testsuite

Following the same procedure as the first time you ran your test corpus, do test runs over both the testsuite and the test corpus.

Again, collect the following information to provide in your write up:

1. How many items parsed?
2. What is the average number of parses per parsed item?
3. How many parses did the most ambiguous item receive?
4. What sources of ambiguity can you identify?
5. For 10 items (if you have at least that many parsing), do any of the parses look reasonable in the semantics? (Emily will demo in class on Tuesday.)

Back to top

Write up

Your write up should be a plain text file (not .doc, .rtf or .pdf) which includes the following:

1. Please answer the following question: May I share this and future write ups with Kristen for her dissertation work?
2. Description the steps you had to take to get from the automatically constructed choice file to one that would both (a) customize and (b) compile.
3. Your answers to the questions about the [incr tsdb()] run, repeated here.
   1. How many items parsed?
   2. What is the average number of parses per parsed item?
   3. How many parses did the most ambiguous item receive?
   4. What sources of ambiguity can you identify?
   5. For 10 items (if you have at least that many parsing), do any of the parses look reasonable in the semantics? (Emily will demo in class on Wednesday.)
4. Documentation of the phenomena you have added to your testsuite, illustrated with examples from the testsuite.
5. Documentation of the choices you made in the customization system, illustrated with examples from your test suite.
   • This can be interleaved with the documentation of the phenomena (so you describe each phenomenon and then the choices you used to add an analysis of it to the grammar), but the documentation of the phenomenon and choices should be logically separate. Here's an example of what this should look like.
6. Descriptions of any properties of your language illustrated in your test suite but not covered by your starter grammar and/or the customization system.
7. If you have identified ways in which the automatically created choices file is particularly off-base, please report them here. If you can include IGT from the testsuite or your descriptive materials illustrating the problem, that is even better.

Back to top

Submit your assignment

• Be sure your write up (and the text-file version of your test suite, if you made a test suite this week, but this is not required) is included in your grammar directory.
• Likewise, make sure to include your most current tsdb profile in the grammar directory (ideally inside tsdb/home/).
• If you're using svn, export the grammar so I don't get all your .svn files:

  svn export yourgrammar iso-lab2
  
  For git, please do the equivalent.

• Create a tarball:

        tar czf iso-lab2.tgz iso-lab2
      

• Upload the tarball to Canvas.

Back to top

Back to course page