An expert advisory system for government regulations: knowledge acquisition methodology

Artificial Intelligence in Economics and Management L.F. L.P. Pau (Editor) B.V. (North-Holland), 1986 © Elsevier Science Publishers B.Y.

205

AN EXPERT ADVISORY SYSTEM FOR GOVERNMENT REGULATIONS: KNOWLEDGE ACQUISITION METHODOLOGY Tauzovich++, S. Matwin+, F. Oppacher+++, D. Skuce+, S. Szpakowicz+ B. Tauzo·vich++, + ++

+++

Dept. of Computer Science, University of Ottawa, Ottawa, Canada Cognos Inc., Ottawa, Canada School of Computer Science, Carleton University, Ottawa, Canada

A methodology is described for building knowledge base systems that handle the knowledge found in regulations, specifications and other similar documents. The approach involves manually translating the document into a simple, English-like language. which combines a production rule approach with constraints, hierarchies and scripts. Our methodology structures the process of translation transl ation and verification using usi ng several software tools. The retrieval process is based on various types of s. tool deduction and associative indexing of rules and facts. Though our example is travel regulations, the method is general enough to be useful for most technical material, such as software description, design or service specifications, or training manuals. 1.

INTRODUCTION AND OVERVIEW

The goal of the research described here is to develop a variety of advisor systems which we term 'Knowl edge Source Systems' (KSSs). Such systems are intended to initially augment and eventually replace conventional natural 1language anguage (NL) documents for storing suitable types of knowledge. Machine understanding of written NL of the kind encountered in these documents is probably one of the most challenging and practically interesting problems for AI. We believe that our approach is an important step toward this goal. The methodology we are will proposing will result in computer systems which wi 11 eventually replace repl ace the following types of documents: rules and regulations of an organization; specifications for the design of some structure or system; documentation on how to use or service some device or system; legal or medical case hi stories; textbook knowledge, e.g. in medicine, engineering, or computer science. Vi rtually all such information is stored at present in printed NL form, Virtually which is perhaps machine readable but not machine understandable. When information is expressed in unconstrained NL, retrieval and understanding are subject to all the lexical, syntactic, semantic, logical, and organizational errors and uncertainties that humans can make. To significantly reduce these serious communication errors, computers must participate in formulating, checking and retrieving knowledge in ways far beyond what today's information retrieval (library searching or document retrieval) systems or specialized documentation aids offer. To do so, we believe it is necessary to encode the information in a highlystructured form, following certain principles developed in AI research. When In addition, thus encoded, knowledge becomes more interpretable by computers. people are pl pleased eased to find that much of the frustrating vagueness usually found in even the best of NL sources has been eliminated or at least controlled.

In summary, we seek a general methodology for capturing and making easily access i b1e and understandab 1e. both to humans and to software components, the accessi understandable,

B. Tauzovich et al.

206

kinds of knowledge found in applications such as those mentioned above. KSS: There are two main components to such aa KSS: a)

is an encoding methodology whereby the unconstrained NL of the source text is replaced (manually, with computer assistance) by aa logic-based, NL-like text essence of the source source material material and eliminates the which captures the semantic essence uncertainties and ambiguities;

b)

enables the user to quickly find relerelean interactive retrieval system which enables knowl edge. vant knowledge.

In this th is paper, pa per, we attempt to describe desc ri be and systematize systemat i ze the process proces s that knowledge know I edge engineers perform, as an essential first step. Our approach involves: into declarative declarative and procedural forms (we (we deal deal only * partitioning knowledge into we have worked worked extensively extensively on on both); both); with the former in this paper, though we

*

partitioning declarative knowledge into several rul es; facts, and rules;

types:

*

two easy-to-read formats that facil itate both both structuring knowledge into two termed ELESK, the the machine and human understanding: one a restricted English, termed LESK [8] (a (a simple mapping other an English-like logical language, termed LESK exists between the two);

*

the source source NL NL into the developing aa well-defined methodology for translating the with software tools to facilitate facilitate this difficult formalized formats, together with process.

lexical knowledge,

our encoding method using as as an example Canadian In this paper we will explain our government travel regulations, which we have selected and studied extensively as typical of institutional regulations. We have attempted to keep our methodology as general as possible, so that most of the knowledge structuring devices we use woul d be useful in other applications. appl ications. The The question-answering component of our would here. It is reported in another paper [9]. system will not be discussed here. implemented in C-Prolog The software tools which assist the translation have been implemented for the VAX under UNIX. 2.

REPRESENTATION OF RULES

2.1 The Four Representational Forms

is derived from from Horn Horn clauses The basic structure of our representation for aa rule is form, i.e. similar to those in PROLOG. When written in aa PROLOG-like form, body

+

head head

form'. It It means: "any "any instantiation of the variables we call this the 'internal form'. which Since whi ch makes the body true makes the head true". Si nce the iinternal nterna 1 form is unterm LESK (Lanpleasant to read, we have defined aa simple, English-like form we term A straightforward bidirectional transguage for Exactly Stating Knowledge) [8]. A and LESK, LESK, implemented by a compiler written lation exists between internal form and (simplified) of aa rule might be in PROLOG. For example, aa (simpl ified) internal form of +

claims(Ee, Exp) & has(Ee, Ppay) claims_with(Ee, Exp, Ppay)

The corresponding LESK form is

An Expert Advisory System for Government Regulations

if then

Employee Employee Employee

207

claims Expense has Proof of Payment claims Expense with Proof_of_payment

We have defined eliminates defi ned a second version of LESK (ELESK) which el iminates the use of variables and is a restricted subset of English. This is desirable for showing encoded rul es to experts for veri fi cat i on of correctness, and al so for use as a quel"Y 1language. anguage. (Skill (Sk ill ed personnel may prefer the LESK form because conven i ent quef"y convenient it is briefer.) LESK and ELESK can be generated from the internal form, and vice will versa. We wi 11 not discuss di scuss ELESK in this paper. The fourth form is the original English text. Our system stores the internal form and the original text so that a user may always know from where a LESK rule came. 2.2

Knowledge Base Structure

There are two structural dimensions to our knowledge base organization. One, the division of rules and supplementary information into hierarchical units, will be discussed in Section 3. The other, the classification of knowledge types, is summarized in Fig. 1. We have concentrated at present on declarative knowledge; the only procedural knowledge knowl edge we di scuss here (i.e. discuss (i .e. about sequences of actions) is in elementary scripts, described in Section 3. The fact/rule dichotomy is derived from PROLOG, though we do not treat our rules as procedures in the PROLOG sense, but use them declaratively. A definition is a rule which answers the question 'what is the meaning of ••• ?'. Regular rules are those derived from source text statements which are not definitions. Facts are statements used in isolation, i.e. without syntactically attached conditions. Facts and rules use domain-specific predicates, which are derived from nouns, verbs and adjectives in the source text. A fixed set of domain-independent predicates are those useful in all subject areas, and are predefined in LESK. KNOWLEOGE REPRESENTATION KNOWLEDGE

----------- ---------~ ~\~ I STATEMENTS

DECLARATIONS

/ CONSTRAINTS

FACTS

\

PROCEDURES

RULES

REGie REG:':

SCRIPTS

~ITION

LESK PREDEF IINED NED PREDICATES

/~ SEMANTIC

LEXICAL

FIGURE 1.

3. 3.1

KNOWLEDGE REPRESENTATION HIERARCHY

METHODOLOGY OF KNOWLEDGE ACQUISITION General Organization of the Translation Process

In our system, knowledge acquisition is quite different from the normal approach

B. Tauzovich Tauzovich et et al. al. B.

208 208

in expert expert system system development. development. Rather Rather than than interviewing interviewing an an expert, expert, we we rely rely mainly mainly in on the the written written text text of of the the source source document, document, though though one one could could acqui acquire knowledge on re knowl edge verbally using using LESK LESK to to write write it it down. down. The heavily heavily interactive interactive process process of of verbally The extracting knowledge knowledge from from an an expert expert by by means means of of interviews, interviews, combined combined with with extracting disambiguation of of that that knowledge knowledge and and attempts attempts to to make make it it more more complete, complete, is is disambiguation replaced by by careful careful analysis analysis of of the the written written document document which which is is the the authoritative authoritative replaced source. Of course, course, ambiguity ambiguity and and incompleteness incompleteness of of the the knowledge knowTecige source source are are source. Of still an an issue, issue, which which can can only only be be solved solved satisfactorily satisfactorily by by consulting consulting experts. experts. still Access to to an an expert, expert, who who is is ready ready to to answer answer questions, questions, is is necessary necessary to to have have Access confidence in in the the correctness correctness of of the the translation. translation. confidence Transl ating aa document document into into LESK LESK is is broken broken into into two two principal princi pal stages: stages: Translating preliminary analysis analysis of of the the source source document; document; (b) (b) the the actual actual translation. translation. preliminary

(a) (a)

Each of of these these stages stages may may be be iterated, iterated, in in combination combi nat i on with with aa critique crit i que of of the the Each result by by an an expert, expert, giving giving the the following following overall overall scheme scheme of of the the translation translation result methodology shown shown in in Fig. Fig. 2. 2. methodology UNIT OF OF THE THE SOURCE SOURCE DOCUMENT DOCUMENT (= (= SOURCE SOURCE UNIT) UNIT) UNIT SYNTACTIC ANALYSIS ANALYSIS SYNTACTIC SYNTACTIC ENVIRONMENT ENVIRONMENT SYNTACTIC

(= LEXICON) LEXICON) (=

EXPERT CR CR IT IT IQUE IQUE EXPERT CONSTRUCTION OF OF SEMANTIC SEMANTIC ENVIRONMENT ENVIRONMENT CONSTRUCTION SEMANTIC ENVIRONMENT ENVIRONMENT SEMANTIC

I

TRANSLATION TRANSLATION

RULES && FACTS FACTS RULES

1------11

~----jl EXPERT EXPERT CRITIQUE eR ITI QUE 22 KNOWLEDGE BASE BASE UNIT UNIT KNOWLEDGE FIGURE 2. 2. FIGURE

BASIC TRANSLATION TRANSLATION SCHEME SCHEME (BTS) (BTS) BASIC

Typically, documents documents which which will will be be subject subject to to the the translation translation process process are are strucstrucTypically, tured into into source source units: units: chapters, sections, sections, subsections subsections etc. etc. The The smallest smallest tured chapters, source unit unit could could be be one one sentence. sentence. In In our our approach, approach, we we proceed proceed incrementally incrementally by by source applying the the Basic Basic Translation Translation Scheme Scheme (Fig. (Fig. 2) 2) to to source source units, units, one one at at aa time. time. applying The corresponding corresponding output output of of each each BTS BTS application appl ication is is called called aa 'Knowledge 'Knowledge Base Base The Unit' and and consists consists of of two two basic basic components: components: the the environment environment and and the the rules rules and and Unit' facts. The The syntactic syntactic analysis analysi s phase phase provides provides part part of of the the environment, envi ronment, which whIch 1S IS facts. referred to to as as the the syntactic syntactic environment environment •. • . (By (By lexicon, lexicon, we we mean mean the the syntactic syntactic referred environment.) The The translation translatIon develops develops the the essent1al essential content content of of the the unit, unit, the the environment.) rules and and facts, facts, augmented augmented by by the the semantic semantic environment, environment, which which are are other other modifying modifying rules statements that that apply apply to to the the rules rules and and facts facts in in the the unit unit (see (see section section 3.3.1). 3.3.1). statements 3.2 3.2

Syntactic Analysis AnalySiS Syntactic

The first first stage stage creates creates aa framework framework which which greatly greatly facil facil itates itates the the translation. translation. The

Advisory System for for Government Regulations An Expert Advisory

209

text The te xt of the source document is first made available as a computer file, and basic statistical information (i .e. word count, sentence count) is gathered. assigned a number by which it it can be be referred to. (UNIX can Each source unit is assigned the basic tools, e.g. the 'diction' program, to carry out some of these provide the have bui It 1t our own software to do thi s.) tasks, but we have Thus we di scovered a of facts about our document, e.g. that the verb 'reimburse' occurs much number of more often that any other verb, and that the moda 1 'may' occurs much more often (this was not at at all obvious by visual inspection of the source than 'can' (this document). The analysis program is controlled by by an an operator, operator, who need not not have have much linThe gui stic training training nor real understanding of the the subject matter of the document. guistic The program reads the source document word by word and the operator assigns one or more grammatical categories (at (at present, present, one one of: proper noun, count count noun, or proper aspects of morphological mass noun, adjective, verb, adverb) to each word. Some aspects the program program categorizes each standard verb verb as analysis are automated, e.g. the active, past participle, gerund etc. Words may If the operator is also be ignored, as not appearing in the LESK translation. If he/she can have have this word word displayed in not sure how to categorize aa given word, he/she been used used previously. the context of the enclosing sentence and review how it has been using In future versions, we intend to automate the lexical analysis further by uSing dictionaries. prepared dictionaries. task of lexicon development is largely completed, the result is reviewed Once the task an expert (the 'expe 'expert critique in Fi Fig. thiss stage, stage, the the by an rt crit i que I' process in g. 2). At thi first job of the subject matter expert is to check the word classifications. 3.3

T~anslation into LESK Translation

3.3.1. Building the Semantic Environment 3.3.1. In the second stage, translation of the regulations into LESK takes place. The proceeds according to to the source unit unit structure of of the translation normally proceeds in the document, but one might decide to restructure it completely. As defined in previous section, each knowledge base unit resulting from the translation may introduce a semantic environment, i.e. a set of definitions and certain conventions necessary to understand the meaning of rules in this unit. Units are in a computer language. Each unit can begin with its nested, analogous to blocks in own environment, which applies to to it. own syntactic environment environment is is the lexical information from stage 1. The semantic The syntactic scripts. environment introduces declarations, definitions, constraints, and script s. VARIABLES

These are are declarations declarations which which specify variables by aa Ilist of LESK ist of as statements; each statement introduces and describes a variable as belonging class ongi ng to a cl ass denoted by a count noun phrase. bel

HIERARCHY

These are declarations which specify two two types of hierarchical relationships between between classes classes of entities entities:: relationships

KINDS

unambiguously state that a class is considered declarations which unambiguously disjoint subclasses, with common noun partitioned into two or more disjoint phrases given for each.

IS-A

specify the the subset relationship relationship between classes classes declarations which specify that is not a partition into subclasses (i.e. 'kinds'). that

SYNONYMS SYNONYMOUS WORDS

declarations are divided into two two groups: These declarations are simply words (or phrases treated as as words) which may may be for the other other in this unit. instance, substituted one for For instance,

B. Tauzovich Tauzovich et et al. al. B.

210

'trip' 'journey' are are defi defined as synonyms. In each each list of 'tri p' and 'journey' ned as In synonyms, one one word word is flagged flagged as as aa canonical canonical synonym for all all the synonyms, other words words on on that that list. list. Only canonical canonical synonyms synonyms are are used used in in other Only of the source; other other synonyms play play an further translation of role in in the the question question answering answering process. important role SYNONYMOUS SYNONYMOUS STATEMENTS STATEMENTS

in this this unit, e.g. are logically equivalent statements in Employee stays stays in Accommodation Accommodation Employee uses Accommodation Employee uses Employee

I NIT IONS DEF INlTIONS

are rules rules which which serve serve to answer answer 'what 'what is is the definition definition of' of' These are They define define one one word word or or phrase phrase in in terms terms of of others others questions. They which are assumed assumed known, known, or defined defined elsewhere. which

CONSTRAINTS

These are domain-derived domain-derived predicates predicates which which must must be be true if if aa rule rule These unit is to to be be used, used, but but which one would would not want to to repeat in the unit laboriously in every every rule. rule. (For (For example, Section Section 55 of the the docudoculaboriously appl ies ies to to 'normal 'normal travel travel status', status', which is is up up to to two months months ment appl duration.) in duration.)

SCRIPT SCR IPT

statements about about actions, actions, using uSing variables and This is a list of statements Our 'basic 'basic verbs' roughly roughly correspond to to 'primitive 'primitive basic verbs. Our in Schank's Schank's Conceptual Conceptual Dependency Dependency Theory Theory (7]. As in acts' in As script organi organizes the temporal temporal and procedural procedural Schank's approach, a script zes the inferences between between basic basic verbs. The The main difference is is that that basic inferences verbs are higher higher level level concepts, concepts, derived derived directly directly from from the source source verbs scri pt pt represents the the sequence of events events whi ch document. The scri normally takes takes pI ace ace in in aa particul particul ar ar context context (e.g. (e.g. when when accommoaccommonormally dation is is mentioned mentioned in the context context of of business business travel). travel). During dation During translation, scripts scripts are are used to fill fill in gaps in in the translation translation translation, achieved so far. far. For For instance, the regulations regulations determine determine circumachieved in which which an an employee employee is is reimbursed reimbursed for for the expenses, expenses, stances in without mentioning expl icitly icitly that that before before being being reimbursed reimbursed the the without must claim claim these these expenses. expenses. The The verb verb 'claim' 'claim' will therethereemployee must fore appear in the the LESK LESK rule rule since since the the translator, translator, after after having the script, script, will will reconstruct all all the actions actions involved in consulted the the rule, either either ex expl icitly or or impl impl icitlY. icitVthe pI icitly

3.3.2 3.3.2

of Source Source Text Translation of

process, which yields rules rules and and facts facts (RFs), (RFs), must must repeatedly repeatedly The translation process, choose a part part of of the the text text which is is to to be be translated translated omitting no no essential essential knowchoose It is possible that one sentence will will be translated translated into more than than one one ledge. It RF. On the the other hand, several sentences sentences may may be be translated translated into a single single RF. RF. The person doing doing the translation (the 'translator') 'translator') determines determines first what portion The of the the source source text will will be be translated translated into into an an RF. RF. If If the the result result of of translation translation of is to be a rule, the the translator translator first first determines determines the the rule head. head. The The main main verb verb of of is text (or its its synonym) is used used as as the head head predithe translated portion of the text The body body of the the rule is is obtained by by translating translating noun noun phrases phrases and and embedded embedded cate. The sentences from from the the source source into into LESK. The trans translator first determines which The I ator fi rst determi nes whi ch common nouns nouns are are to be be mapped mapped into into existing existing variables, variables, and and what what new variables are are common introduced. phrases corresponding corresponding to to these these variables are are then to be introduced. Noun phrases translated body statements, statements, and and the the body and the head head are are assembled into a transl ated into body rul e. e. rul the translator feels feels (or learns learns from the the expert) expert) that aa rule has has In some cases, the to be be a necessary necessary and and sufficient sufficient one. Typically, one of the the conditions conditions in in the the to

An Expert J:;xpert Advisory System for Government Regulations

211

body of the rule gives the entire rule this if-and-only-if meaning, e.g. "An employee, using approved commercial accommodation, is reimbursed for the expenses provided that he presents a proof of payment". It turns out that when the proof of payment is absent, then - even though the accommodation was approved and used - no reimbursement takes place. The approach we have taken is to translate a rule like this into two rules to reflect its necessary and sufficient character: if

Employee uses Commercial-Accommodation Commercial-Accommodation is approved Expense is an expense for Commercial-Accommodation Employee has a proof_of_payment for Expense

4.

then

Employee shall be reimbursed Expense

if

Employee uses Commercial-Accommodation Commercial-Accommodation is approved Expense is an expense for Commercial-Accommodation there is no P such that P is a proof_of_payment for Expense

then

Employee shall not be reimbursed Expense

RELATED WORK

The methodology outl ined in the previous sections is meant to guide the early phase of knowl edge acqui sition, namely the initial structuring and encoding of knowledge acquisition, documents expressed in largely unconstrained natural language. Individual stages of such a process can be partly automated, either now or in the near future, after developing the necessary tools. The existing automated knowledge acquisition tools (e.g. P], [1], [5], [ll]) [11]) are concerned with later stages, wherein some formal representation has been achieved. A system which bears some similarity to our approach It accepts input in a small subset of English. New paraphrase in restricted English. The system answers under consideration, and also questions about its own

has been described in [10]. concepts can be defined by questions about the domain knowledge representation.

The system presented in [3] and [4] learns by accepting input, in restricted English, about domains that can be described in logic. Also, the system assists the user in handling generic and procedural knowledge. Our system seems to have greater deductive power but lacks the ability to interface to other software. Knowledge sources systems have some common properties with deductive data bases with natural language user interfaces (see [2]). However, rather than providing the user with facts relevant to specific domain questions, our system suppl ies "generic" information, i.e. rules appl icable to a given typical situation or a general probl em. problem. 5.

CONCLUDING REMARKS

Thi iminary design of both a KSS and an associated Thiss paper has reported the prel preliminary knowledge acquisition methodology which we believe are unique in a significant number of ways. The most important of these are the desi gn and use of LESK to bridge the gap between natural language and the computer's capabilities, and a well-defined methodology for working with a source document to transform it into suitable knowledge structures. Our approach to knowledge representation shares much with other rule- based systems, particularly those which permit logical variables (e.g. PROLOG-based

B. Tauzovich Tauzovicll et et al. al. B.

212 212

systems). The The use use of of ELESK ELESK and and LESK LESK as as convenient convenient surface surface syntax, syntax, and and the the use use of of systems). units to to structure structure the the knowl knowledge base simil similarly to aa document document we we bel believe to be be units edge base arly to ieve to valuable extensions extensions to to existing existing rule rule formats. formats. valuable A final final interestirrg interestirrg question question is is to to consider consider to to what what degree degree it it might might be be feasible feasible A to process process the the raw raw source source text text in in some some direct direct way way to to yield yield aa knowledge knowledge base. base. to Coul d aa machine machi ne read read the the source source text text and and understand understand it it sufficiently suffi c i ent 1y well well to to Could convert it it into into clear clear and and simple simple rules rules as as we we have have done done mentally? mentally? We We believe believe that that convert this process process would would involve involve all all the the complexities complexities of of machine machine understanding, understanding, and and is is this very similar similar to to the the process process of of totally totally automatic automatic machine machine translation, translation, which which is is very still aa dream dream despite despite many many years years of of research. research. What What we we have have in in mind mind for for the the near near still future, and and are are indeed indeed beginning beginning to to study study seriously, seriously, is is aa semi-sutomatic semi-sutomatic machine machine future, translation system, system, which which attempts attempts aa translation translation of of aa sentence, sentence, but but which which require require translation knowledgeable operator operator to to help help with with all all the the ambiguities ambiguities and and choices choices that that aa knowledgeable require intell igence igence and and experience. experience. Such Such aa hybrid hybrid system system should should be be practical practical requi re intell within several several years. years. within BIBLIOGRAPHY B IBLIOGRAPH Y Davis, R., R., Lenat, Lenat, D., D., "Knowledge-based "Knowledge-based Systems Systems in in Artificial Artificial Intelligence" Intelligence" 1] Davis, [[ 1] (McGraw-Hill, New New York, York, 1980). 1980). (McGraw-Hill, Gallaire, H., H., Minker, Minker, J., J., Nicholas, Nicholas, J.-M., J.-M., "Logic "Logic and and Databases: Databases: 2] Gallaire, [[2]

A A

Deductive Approach" Approach" (ACM (ACM Computing Computing Surveys, Surveys, vol. vol. 16, 16, no. no. 2, 2, 1984). 1984). Deductive

3] [[ 3]

Haas, N., N., Hendrix, Hendrix, G., G., "Learning "Learning by by Being Being Told: Told: Haas, Informat i on Management", Management", in in [6], [6], pp. pp. 405-427. 405-427. Information

4] [[4]

Hendrix, G., "KLAUS: "KLAUS: A A System System for for Managi Managing Information and Computat Computational Hend ri x, G., ng In format i on and i ona I (Technical Note Note 230, 230, SRI, SRI, Menlo Menlo Park, Park, Cal California, 1980). Resources" (Technical ifornia, 1980). Resources"

[5] [5]

Michalski, R.S., R.S., "A "A Theory Theory and and Methodology Methodology of of Inductive Inductive Learning". Learning". In In [6]. [6]. Michalski,

[ 6] [6]

Michalski, R.S., R.S., Carbonell, Carbonell, J.G., J.G., Mitchell, Mitchell, T.M., T.M., "Machine "Machine Learning" Learning" (Tioga (Tioga Michalski, Publishing Co., Co., Palo Palo Alto, Alto, California, California, 1983). 1983). Publishing

Schank, 7] Schank, [[ 7]

Acquiring Knowledge for Acqui ring Knowl edge for

N., Riesbeck, Riesbeck, C.K., C.K., (eds.), (eds.), "Inside "Inside Computer Computer N., (Lawrence Erlbaum Erlbaum Associates, Associates, Hillsdale, Hillsdale, N.J., N.J., 1981). 1981). (Lawrence

Understanding", Understanding",

[8] [8]

Skuce, D., D., "The "The LESK LESK Tutorial", Tutorial", Department Department of of Computer Computer Science, Science, University University Skuce, of Ottawa, Ottawa, TR-83-03 TR-83-03 (July (July 1983). 1983). of

[9] [9]

Skuce, D., D., Matwin, Matwin, S., S., Tauzovich, Tauzovich, B., B., Szpakowicz, Szpakowicz, S., S., Oppacher, Oppacher, F., F., "A "A Skuce, Rule-Oriented Methodology Methodology for for Constructing Constructing aa Knowledge-Base Knowledge-Base from from Natural Natural Rule-Oriented (Proc. Conference Conference on on Expert Expert Systems Systems in in Government, Government, Language Documents" Documents" (Proc. Language Oct. 1985). 1985). Washington, D.C., D.C., Oct. Washington,

[la] Thompson, Thompson, B.H. B.H. Thompson, Thompson, F.B., F.B., "Introducing "Introducing ASK, ASK, aa Simple Simple Knowledgeable Knowledgeable [10] (Proceedings of of the the Conference Conference on on Appl Appl ied ied Natural Natural Language Language System" (Proceedings System" Processing, Santa Santa Monica, Monica, California, California, February February 1983). 1983). Processing, [ll] Winston, Winston, P.H., P.H., "Learning "Learning New New Principles Principles from from Precedents Precedents and and Exercices: Exercices: [11] Details" (AI (AI Memo Memo 623, 623, MIT, MIT, Cambridge, Cambridge, Mass., Mass., 1981). 1981). Details"

the the