nl-kr-request@cs.rpi.edu (NL-KR Moderator Chris Welty) (08/15/89)
NL-KR Digest (Tue Aug 15 11:59:34 1989) Volume 6 No. 34 Today's Topics: Another IJCAI-89 Update Papers on representation of time More Papers on representation of time Re: NLU benchmarking - request for info Controlled English Submissions: nl-kr@cs.rpi.edu Requests, policy: nl-kr-request@cs.rpi.edu Back issues are available from host archive.cs.rpi.edu [128.213.1.10] in the files nl-kr/Vxx/Nyy (ie nl-kr/V01/N01 for V1#1), mail requests will not be promptly satisfied. If you can't reach `cs.rpi.edu' you may want to use `turing.cs.rpi.edu' instead. ----------------------------------------------------------------- To: nl-kr@cs.rpi.edu >From: rjb@allegra.att.com Date: Wed, 9 Aug 89 14:35:05 EDT >From: allegra!rjb (Ron Brachman) Subject: Another IJCAI-89 Update On Friday afternoon, there will be a "summary session" presenting the highlights of the KR'89 conference (May 15-18, in Toronto). In addition to a summary of the conference, the three winners of the KR'89 Awards for Outstanding Contributions will be re-presented. Here is a draft schedule: 3:00 - 4:30 pm Friday Summary Session IV - Best Knowledge Representation Papers - 89 (Room D3-28/19 Cobo Hall) (Chaired by Raymond Reiter) 3:00-3:15 Hector J. Levesque and Ronald J. Brachman, Introduction and Summary of KR'89 3:15-3:40 Henry A. Kautz and Bart Selman, "Hard Problems for Simple Default Logics" 3:40-4:05 Lenhart K. Schubert and Chung Hee Hwang, "An Episodic Knowledge Representation for Narrative Texts" 4:05-4:30 Teodor C. Przymusinski, "Three-Valued Formalizations of Non-Monotonic Reasoning and Logic Programming" ------------------------------ To: nl-kr@cs.rpi.edu Sender: shapiro@cs.Buffalo.EDU Date: Wed, 9 Aug 89 17:11:01 EDT >From: shapiro@cs.Buffalo.EDU Subject: Papers on representation of time Here are two more papers: @INPROCEEDINGS{AlmSha83, AUTHOR = "Almeida, M. J. and Shapiro, S. C.", YEAR = 1983, TITLE = "Reasoning about the temporal structure of narrative texts", BOOKTITLE = "Proceedings of the Fifth Annual Meeting of the Cognitive Science Society", ADDRESS = "Rochester, {NY}" } @PHDTHESIS{Alm87, AUTHOR = "* Almeida, M. J.", YEAR = 1987, TITLE = "Reasoning About the Temporal Structure of Narratives", TYPE = "Technical Report", NUMBER = "87--10", SCHOOL = "Department of Computer Science, {SUNY} at Buffalo,", PAGES = 185 } ------------------------------ To: nl-kr@cs.rpi.edu Date: Wed, 9 Aug 89 15:54 EDT >From: Brad Miller <miller@CS.ROCHESTER.EDU> Subject: Re: NL-KR Digest, Volume 6 No. 33 Reply-To: miller@CS.ROCHESTER.EDU Postal-Address: 610 CS Building, Comp Sci Dept., U. Rochester, Rochester NY 14627 Phone: 716-275-1118 A number of people have asked for this, so here is waht I have, in bibtex form: I'd also add: @string{URCS = "University of Rochester, Computer Science Department"} @techreport{timelogic, Author = "Koomen, Johannes A.G.M.", Title = "The TIMELOGIC Temporal Reasoning System", Institution = URCS, Number = "231 (revised)", Month = Oct, Year = 1988} He also has a thesis forthcoming on the subject "Reasoning about Recurrence". ------------------------------ To: nl-kr@cs.rpi.edu Date: Thu, 10 Aug 89 14:09:05 -0400 >From: finin@PRC.Unisys.COM Subject: Re: NLU benchmarking - request for info > Date: Thu, 3 Aug 89 12:36:39 EDT > From: rpg@cs.brown.edu > Subject: NLU benchmarking - request for info > > Some time ago, in this digest, I recall reading about a workshop > about evaluation of natural-language understanding programs. Could > anyone provide me with more information about this workshop? Was a > report produced, or a set of position papers/abstracts? If so, I'd be > interested in seeing a copy. ... There was a Workshop on the Evaluation of Natural Language Processing Systems held in December of 1988 in Wayne, Pennsylvania. The workshop was organized by Martha Palmer (Unisys Paoli Research Center) and supported by RADC, AAAI, ACL and Unisys. Martha Palmer, Sharon Walters (RADC) and I have written a report summarizing the results of the workshop which we hope to publish soon. I will announce its availability via the NL-KR newsgroup. I've attached some additional information on the workshop below. Tim ___________________________________________________________________________ In the past few years, the computational linguistics research community has begun to wrestle with the problem of how to evaluate its progress in developing natural language processing systems. With the exception of natural language interfaces there are few working systems in existence, and they tend to focus on very different tasks and equally different techniques. There has been little agreement in the field about training sets and test sets, or about clearly defined subsets of problems that constitute standards for different levels of performance. Even those groups that have attempted a measure of self-evaluation have often been reduced to discussing a system's performance in isolation - comparing its current performance to its previous performance rather than to another system. As this technology begins to move slowly into the marketplace, the lack of useful evaluation techniques is becoming more and more painfully obvious. In order to make progress in the difficult area of natural language evaluation, a Workshop on the Evaluation of Natural Language Processing Systems was held last December at the Wayne Hotel in Wayne, Pennsylvania. There were two basic premises for this workshop. The first was that it should be possible to discuss system evaluation in general without having to state whether the purpose of the system is "question-answering" or "text processing." Evaluating a system requires the definition of an application task in terms of input/output pairs which are equally applicable to question-answering, text processing, or generation. The second premise was that there are two basic types of evaluation, black box evaluation which measures system performance on a given task in terms of well-defined input/output pairs, and glass box evaluation which examines the internal workings of the system. For example, glass box performance evaluation for a system that is supposed to perform semantic and pragmatic analysis should include the examination of predicate-argument relations, referents, and temporal and causal relations. Since there are many different stages of development that a natural language system passes through before it is in a state where black box evaluation is even possible, glass box evaluation plays an especially important role in guiding the development at early stages. With these premises in mind, the workshop was structured around the following three sessions: - Defining the notions of "glass box evaluation" and "black box evaluation" and exploring their utility. - Defining criteria for "glass box evaluation." - Defining criteria for "black box evaluation." Calls for participation were sent by electronic mail to several national and international mailing lists and posted on numerous internet newsgroups and resulted a large response. A program committee consisting of Beth Sundheim (NOSC), Ed Hovy (ISI), Tim Finin (Unisys Paoli Research Center), Lynn Bates (BBN), Martha Palmer (Unisys Paoli Research Center), Mitch Marcus (CIS, University of Pennsylvania) was put together to plan the workshop and invite participants. Those respondents interested in participating in the workshop were asked to describe their interest in the topic, describe any relevant work done in the area, and provide an abstract on evaluation topics that they would want to present. A total of fifty people were invited to participate. It was hoped that the workshop would shed light on the following questions: - What are valid measures of "black box" performance? - What linguistic theories are relevant to developing test suites? - How can we characterize efficiency? - What is a reasonable expectation for robustness? - What would constitute valid training sets and test sets? - How does all of this relate to measuring progress in the field? ------------------------------ To: nl-kr@cs.rpi.edu >From: prlb2!kulcs!siegeert@uunet.UU.NET (Geert Adriaens) Newsgroups: comp.ai.nlang-know-rep Subject: Controlled English Date: 11 Aug 89 07:30:43 GMT In view of a project at the Departement of Applied Linguistics at the University of Leuven, we are investigating the possibilities and limitations of Controlled English (CE) with particular focus on the grammatical and lexical frameworks used to achieve a form of CE. Can anyone send us a copy of the following manuals (or give me addresses where we can order them) : - the "Caterpillar Fundamental English"-manual, - the "Kodak International Service Language"-manual, - J. Smart's "Plain English Program", - B.T. White's "International Language for Servicing and Maintenance". Any information about publications on CE or addresses of people also studying CE (both in industry and at universities) will be most welcome. - - Geert Adriaens (SIEMENS-METAL Project) Maria Theresiastraat 21 siegeert@kulcs.uucp or B-3000 Leuven siegeert@blekul60.bitnet or tel: ..32 16 285091 siegeert@cs.kuleuven.ac.be ------------------------------ End of NL-KR Digest *******************