nl-kr-request@CS.ROCHESTER.EDU (NL-KR Moderator Brad Miller) (09/13/88)
NL-KR Digest (9/12/88 21:11:42) Volume 5 Number 15 Today's Topics: Re: open/closed classes nl evaluation workshop Data Wanted: Re: GPSG parsers Submissions: NL-KR@CS.ROCHESTER.EDU Requests, policy: NL-KR-REQUEST@CS.ROCHESTER.EDU ---------------------------------------------------------------------- Date: Thu, 1 Sep 88 07:54 EDT From: Bruce E. Nevin <bnevin@cch.bbn.com> Subject: open/closed classes There is some recent work by Leonard Talmy on the supposed cognitive whys and wherefores of open vs closed classes. Sorry, I don't have a reference handy. The supposition that a speech recognizer has to be especially good at hearing closed-class words misses an important point: the closed-class words are unstressed and generally subject to reduction in phonemic--how shall I say--extent. This is part of a general process, apparently in all languages, of reducing the phonemic representation of words that carry less information. They are reducible to the extent that they are redundant. (Not much difficulty predicting the filler in the context 'He __ gone.' You only need enough phonemic content to distinguish the words 'has, had, was' plus of course more obvious--and less reduced-- constructions incorporating these such as their negatives, `will have gone', etc.) Historically, closed-class morphology derives from open-class words that have become more redundant and predictable, so that their reduced forms become `frozen' in their now predictable contexts. An example is the suffix -hood in `childhood', from an earlier form had meaning `state', something like `child-state'. The suffix -ly in adverbs of manner derives from the dative of a word for `form, body'. Ancestors of Proto-Indo-European not having been reconstructed, we have no confirmation that this is the origin of inflectional morphology such as the preterit in descendant languages like English, but that is certainly the most plausible assumption. In American Indian languages, Shirley Silver dubbed this process `morphemization' almost 20 years ago. So affixes (inherently closed-class morphology) appear to be derived by reduction from once free-standing words. Similarly for closed-class words. `Because' derives from `by cause'. OED cites 1305 `bi cause whi'; whi or `why' is the instrumental of the wh- pronouns typified by `what', reduced to `that' in the later `by cause that, because that'. (Compare reduction of cause to zero in `for the cause why' --> `forwhy', a common conjunction now obsolete, to which compare further `from the place where' --> `from where'.) Zeroing of `why ~ that' in `because why, because that' leaves `because' as a conjunction, a closed-class word. (See Jespersen _Modern English Grammar on Historical Principles_ V 397 and Harris _A Grammar of English on Mathematical Principles_ 195 for further details.) An example currently in progress in English is `going to' --> `gonna', a reduction that takes place before verbs but not before nouns (*`I'm gonna New York') precisely because `going to' can occur before the whole class of verbs (and consequently carries less information and is subject to reduction there) but cannot occur before every possible noun. (Note that in e.g. `I'm going to authority' an indefinite noun, one of exceptionally broad distribution, can be understood as having been elided: `I'm going to someone of/in authority'. It is not possible to reverse a reduction in this way to account for the broad distribution of `going to' before verbs.) This appears to be on the way to being a separate future tense morpheme in the closed-class set. The above example of `forwhy' illustrates that closed-class words also become obsolete and drop from the language. The class is closed with respect to distribution, and conservative but not closed with respect to change. Bruce Nevin bn@cch.bbn.com <usual_disclaimer> ------------------------------ Date: Fri, 2 Sep 88 12:19 EDT From: palmer@PRC.Unisys.COM Subject: nl evaluation workshop CALL FOR PARTICIPATION Workshop on Evaluation of Natural Language Processing Systems Dec 8-9 Wayne Hotel, Wayne, PA (Philadelphia) There has been much recent interest in the difficult problem of evaluating natural language systems. With the exception of natural language interfaces there are few work- ing systems in existence, and they tend to be concerned with very different tasks and use equally different techniques. There has been little agreement in the field about training sets and test sets, or about clearly defined subsets of problems that constitute standards for different levels of performance. Even those groups that have attempted a meas- ure of self-evaluation have often been reduced to discussing a system's performance in isolation - comparing its current performance to its previous performance rather than to another system. As this technology begins to move slowly into the marketplace, the need for useful evaluation tech- niques is becoming more and more obvious. The speech com- munity has made some recent progress toward developing new methods of evaluation, and it is time that the natural language community followed suit. This is much more easily said than done and will require a concentrated effort on the part of the field. There are certain premises that should underly any dis- cussion of evaluation of natural language processing sys- tems: (1) It should be possible to discuss system evaluation in general without having to state whether the pur- pose of the system is "question-answering" or "text processing." Evaluating a system requires the definition of an application task in terms of I/O pairs which are equally applicable to question- answering, text processing, or generation. (2) There are two basic types of evaluation: a) "black box evaluation" which measures system performance on a given task in terms of well-defined I/O pairs; and b) "glass box evaluation" which examines the internal workings of the system. For example, glass box per- formance evaluation for a system that is supposed to perform semantic and pragmatic analysis should include the examination of predicate-argument rela- tions, referents, and temporal and causal relations. Given these premises, the workshop will be structured around the following three sessions: 1) Defining "glass box evaluation" and "black box evaluation." 2) Defining criteria for "black box evaluation." _A Proposal for establishing task oriented benchmarks for NLP Systems_ (Session Chair - Beth Sundheim) 3) Defining criteria for "glass box evaluation." (Session Chair - Jerry Hobbs) Several different types of systems will be discussed, including question-answering sys- tems, text processing systems and generation systems. Researchers interested in participating are requested to submit a short (250-500 word) description of their experience and interests, and what they could contribute to the workshop. In particular, if they have been involved in any evaluation efforts that they would like to report on, they should include a short abstract (500-1000 words) as well. The number of participants at the workshop must be restricted due to limited room size. The descriptions and abstracts will be reviewed by the following committee: Mar- tha Palmer (Unisys), Mitch Marcus (University of Pennsyl- vania), Beth Sundheim (NOSC), Ed Hovy (ISI), Tim Finin (Unisys), Lynn Bates (BBN). They should arrive at the address given below no later than October 1st. Responses to all who submit abstracts or descriptions will be sent by November 1st. Martha Palmer Unisys Research & Development PO Box 517 Paoli, PA 19301 palmer@prc.unisys.com (215) 648-7228 ------------------------------ Date: Mon, 5 Sep 88 16:36 EDT From: Mark William Hopkins <markh@csd4.milw.wisc.edu> Subject: Data Wanted: I am in need of some English text, for setting up a data base. If you have any to contribute please e-mail them to me. I asked Jerry Lewis to set up a telethon for this, but he said he was busy :-) ------------------------------ Date: Mon, 12 Sep 88 08:02 EDT From: COR_HVH%HNYKUN52.BITNET@CUNYVM.CUNY.EDU Subject: GPSG parsers Some time ago I asked for information on GPSG parsers (or parser-generators) and promised to report any replies. Up to now, I have been notified of two efforts in this area. At the Technical University in Berlin a PROLOG system is being developed in a machine translation context (Eurotra). It is able to parse and generate sentences according to a small English or a medium German grammar. At Boeing work is done on a LISP GPSG parser with the eventual aim of automatic message processing. The system can parse English sentences using a fairly large grammar and dictionary. Neither system uses "pure" GPSG (in case it exists at all), the most important difference being the absence of metarules. I will ask both my contacts to do a more detailed write-up about their work and submit them to this list. Hans van Halteren COR_HVH@HNYKUN52.BITNET ------------------------------ End of NL-KR Digest *******************