Lab 2 (due 1/19 11:59 pm)

Overview

The goal of this lab is to make a start on the test suite that will be your development target, on the one hand, and to customize a first version of your grammar start on the other. I've selected phenomena to cover in this lab with an eye to starting with those that are essential to creating a working grammar. You will also be getting set up with ALL the software, including [incr tsdb()] and make_item, both of which are new this week. Please read through the instructions completely before starting and plan to work on the substasks in parallel, though they are described separately in the instructions below.

Back to top

Preliminaries

Canvas Discussion

In order to encourage people to get started early, this assignment requires that you post a question to the discussion on the course Canvas by Tuesday night. Examples: A question about something in these instructions that's unclear, a question about something in your grammar resources that's unclear, or a question about something in the customization system that's unclear. Or all of the above! Ask away :)

(When I'm grading this part of the assignment, I'm going to check that both partners for each grammar posted a question.)

When posting to Canvas, please create separate threads for separate questions and always include the name of the language we're talking about!

Version control

Please use version control for your work in this class. You may use any system you are comfortable with. Please negotiate this with your partner. If neither of you already have experience with version control systems, please post to Canvas for guidance ASAP.

Back to top

Test Suite

The first task is to create positive and negative example sentences illustrating the following phenomena, to the extent that they are relevant for your language:

• word order
• pronouns (includes person/number/gender)
• case
• the rest of the NP
• matrix wh questions
• tense/aspect

Before you start, read the general instructions for testsuites and the formatting instructions.

Back to top

Starter grammar

The second task is to create a starter grammar by filling out the required sections of the Grammar Matrix customization questionnaire. The goal here is to get as much coverage as you can over your test suite for the phenomena indicated using only the customization system (no hand-editing of tdl files yet). In particular, you'll need to address these sections:

• General information
• Word order
• Number
• Person
• Gender (if applicable)
• Case
• Direct-inverse (if appropriate)
• Lexicon

In the word order section, you can skip the auxiliaries by saying "no" on that question for now. When we get to auxiliaries, you may of course revise this answer.

Note that your testsuite for this week will include examples for wh-questions and tense/aspect, but your grammar is not expected to cover these phenomena. The plan is to have the testsuite construction run somewhat ahead of the grammar construction, so that by the time we're done with customization, we'll have testsuites mapping what the tdl editing should do.

In the lexicon section, you should define lexical types for transitive and intransitive verbs and nouns. If appropriate, you should define determiners and case-marking adpositions. Where the lexicon asks you for pred values, use English glosses instead of the stem itself. Thus no matter what language you are working on, if you have an entry for a noun which translates as dog, its pred value should be _dog_n_rel. The pred value for all pronouns should be pron_rel (the person/number/gender features will provide any further differentiation that is needed).

Please do NOT try to add any morphological rules this week! (Exception: If you have absolutely everything else working and your write up done, and you feel like doing more, then go for it.) This means that if you have any morphological complexity, you'll be creating full form lexical entries (e.g. for verbs marked for certain agreement information or nouns marked for certain case values). This means you'll want to keep your lexicon small for this week. We'll add morphological rules in Lab 3.

If you have case inflections, and you're making full form lexical entries for your nouns, then you'll need to define different noun types for different case values. When we add the lexical rules next week, your noun entries should have no case value and the rules should fill it in. (This will require replacing your multiple noun types with just one, another reason to keep the lexicon small for now.)

Make sure you can parse individual sentences

Once you have created your starter grammar (or each time you create one, as you should iterate through grammar creation and testing a few times as you refine your choices), try it out on a couple of sentences interactively to see if it works:

1. Load the grammar into the LKB.
2. Using the parse dialog box (or 'C-c p' in emacs to get the parse command inserted at your prompt), enter a sentence to parse.
   • Once you have your [incr tsdb()] profile set up, you can also try parsing sentences by doing Browse | Test items and clicing on individual items.
3. Examine the results. If it does parse, check out the semantics (pop-up menu on the little trees). If it doesn't look at the parse chart to see why not.
4. Problems with lexical rules and lexical entries often become apparent here, too: If the LKB can't find an analysis for one of your words, it will say so, and (obviously) fail to parse the sentence.

Note that the questionnaire has a section for test sentences. If you use this, then the parse dialog will be pre-filled with your test sentences.

Back to top

[incr tsdb()] profile

In this portion of the lab you will use the [incr tsdb()] grammar profiling system to test the performance of your starter grammar over your test suite, and then examine the results. (You may find in doing so that you want to refine certain aspects of your starter grammar. You can do this by uploading the file "choices" which comes with your grammar into the customization system and then tweaking from there.)

Create a test suite skeleton

1. Look at the contents of your grammar directory and locate the subdirectory called tsdb.
2. Make a subdirectory called lab2 inside tsdb/skeletons for your test suite.
3. Edit tsdb/skeletons/Index.lisp to include a line for this directory, e.g.:

   (
   ((:path . "matrix") (:content . "matrix: A test suite created automatically from the test sentences given in the Grammar Matrix questionnaire."))
   ((:path . "lab2") (:content . "Test suite collected for Lab 2."))
   )
   

4. Download the python script make_item, make sure it is executable, and run it on your test suite:

   make_item testsuite.txt

   Notes on make_item:

   • This script is going to be pretty picky about the format of your test suite. If you have questions, please post to Canvas (10 minute rule!).
   • It requires python3, which is on the current version of the Ubuntu+LKB appliance.
   • Alternatively, you can copy your testsuite and make_item over to patas and run there, or install python3 (from http://python.org/download) on your host OS (mac or windows), and run make_item outside VirtualBox.
   • If the above command is successful, testsuite.txt.item would be created in the working directory. If the testsuite contains errors, it's possible that a lot of output will appear on stderr. It maybe useful to redirect this into a file that you can use to go through and correct the errors one at a time. For example:

     ./make_item testsuite.txt item 2>errs

     The command just above attempts to create 'item' in the working directory, and stderr messages are redirected to the file 'errs'.

     make_item contains a default mapping from testsuite line types into particular fields of the [incr_tsdb()] item file. The default mapping puts 'orth' into 'i-input', the field which the is the input to the grammar. If your grammar targets a different testsuite line, override the default mapping with the -m/--map option.

     ./make_item --map orth-seg i-input testsuite.txt item

     The invocation above maps the orth-seg line into the input field.

     You can run make_item with -h/--help to see a summary of the options.

5. Copy the .item file which is output by make_item to tsdb/skeletons/lab2/item.
6. Copy tsdb/skeletons/Relations to tsdb/skeletons/lab2/relations (notice the change from R to r).
7. The final directory structure should look like this:

   grammar/data/testsuite
   grammar/data/make_item
    
   
   grammar/data/testsuite.item
   grammar/tsdb/skeletons/Index.lisp            (lists the testsuites)
   grammar/tsdb/skeletons/Relations             (master copy of the database schema)
   grammar/tsdb/skeletons/lab2/item             (copy of ../../data/testsuite.item)
   grammar/tsdb/skeletons/lab2/relations        (copy of ../Relations)
   grammar/tsdb/home                            (directory to store test profiles)
   

Create and run an initial test suite instance

1. Start the lkb
2. Load your starter grammar. (The script file is in your-grammar-dir/lkb/script.)
3. Start [incr tsdb()] (within emacs, at the lisp prompt, type (tsdb:tsdb :podium)
4. In the [incr tsdb()] podium, select Options > Database Root and input the path to tsdb/home.
5. In the [incr tsdb()] podium, select Options > Skeleton Root and input the path to tsdb/skeletons.
6. Optional: For future use, you can set these variables ahead of time in a file called .tsdbrc in your home directory. It should contain these lines, with path names edited appropriately:

   (tsdb:tsdb  :home  "path-to-tsdb/home")
   (tsdb:tsdb  :skeleton  "path-to-tsdb/skeletons")
   

7. In the [incr tsdb()] podium, select File > Create. You should see your test suite in the menu there. Select it, and get a test suite instance. Post to GoPost if this doesn't work.
8. Make sure your grammr is loaded into the LKB.
9. Once you have a test suite instance, select it (by clicking on it), then do Process > All Items.
10. Explore the results, with functions such as Browse > Results, Analyze > Coverage, and Analyze > Overgeneration.
11. Be sure to save (i.e., not overwrite or delete) this test suite instance, as you'll be asked to turn it in.

Note If your tsdb/ directory is inside a shared folder on VirtualBox, it will not work.

Back to top

Write up

NB: While the test suite and choices file creation is joint work, the write up should be done by one partner (the other will get a turn next week). The writing partner should have the non-writing partner review the write up and make suggestions. Please indicate in your write up which partner took which role this week.

Your write up should be a plain text file (not .doc, .rtf or .pdf) which includes the following:

1. Documentation of the phenomena you have added to your testsuite, illustrated with examples from the testsuite.
2. Documentation of the choices you made in the customization system, illustrated with examples from your test suite.
   • This can be interleaved with the documentation of the phenomena (so you describe each phenomenon and then the choices you used to add an analysis of it to the grammar), but the documentation of the phenomenon and choices should be logically separate. Here's an example of what this should look like.
3. Descriptions of any properties of your language illustrated in your test suite but not covered by your starter grammar and/or the customization system.
4. Documentation the coverage of your grammar over the testsuite. This should include both summary numbers, which you can get by using the Analyze | Coverage and Analyze | Overgeneration options in [incr tsdb()], and discussion of specific examples. If there are examples that thare parsed incorrectly (unanalyzed grammatical examples, analyzed ungrammatical examples, or grammatical examples assigned surprising parses), reflect on why that might be.
5. Finally, if there are any places where the customization system seems unable to cope with the properties of your language (within the phenomena addressed in this lab), describe them here.

Back to top

Submit your assignment

• Be sure your write up and the text-file version of your test suite are included in your grammar directory.
• Likewise, make sure to include your most current tsdb profile in the grammar directory (ideally inside tsdb/home/).
• If you're using svn, export the grammar so I don't get all your .svn files:

  svn export yourgrammar iso-lab2
  
  For git, please do the equivalent.

• Create a tarball:

        tar czf iso-lab2.tgz iso-lab2
      

• Check that this unapcks into a directory called iso-lab2 (where iso is replaed by the actual iso code for your language), and not lots of separate files nor something like /home/ubuntu/iso-lab2.
• Regiter your group (partnership) on Canvas and submit the tarball.

Back to top

Back to course page