CAQ: A machine learning tool for engineering

CAQ: A machine learning tool for engineering B. L. Whitehall*, S. C.-Y. Lu* and R. E. Stepp*

University of Illinois at Urbana - Champaign, Urbana, 1i. 61801, USA Machine learning algorithms developed from artificial intelligence (AI) research are rapidly finding useful applications in engineering domains. They are joining a family of tools that can help engineers summarize (or synthesize) massive amounts of data to support decision making. Traditional tools in this family, such as various statistical-based techniques, deal mostly with continuous (real) valued attributes - their results are relatively accurate, but often hard to interpret. On the other hand, logic-based learning tools, such as Michalski's AQ series of programs for learning concepts from examples, deal only with discrete valued attributes - their results are easier to comprehend, but not accurate for engineering applications which rarely deal solely with discrete valued attributes. Ideally, engineers need algorithms that work with a mix of attributes including those that take continuous values and those that take discrete values. Engineering problems also demand that learning algorithms be able to handle noise and produce concepts that are comprehensible. CAQ (Continuous AQ) is an algorithm designed to meet engineering requirements. This paper explains how CAQ combines continuous and discrete values, and shows that handling continuous attributes as real numbers instead of forcing them into a discrete representation leads to more efficient concept formation. Data from a turning process simulator used in support of machine operation planning in manufacturing are used to demonstrate the new algorithm. Key Words: machine learning, inductive inference, knowledge acquisition. 1. I N T R O D U C T I O N As the use of computers in engineering becomes more widespread, larger and more complex applications are developed, often causing engineers to deal with large volumes of computer-generated data whose interpretation becomes increasingly time-consuming, error-prone, and complex. Today, engineers are overwhelmed with data and have difficulties extracting useful information from it. Raw data is plentiful, but useful information and problem-solving knowled#e remain sparse. Computerbased simulators, built upon the physics of engineering processes, can be used to produce large sets of data points describing a mapping from the input (control) space to the output (performance) space. But these data sets do little to help the engineer directly describe this mapping in a qualitative manner so that it is more comprehensible for decision making. This paper introduces a new machine learning algorithm, CAQ, that takes steps towards providing a tool for assisting engineers in extracting useful knowledge from data. Engineering models of the future will have qualitative reasoning capabilities in addition to the quantitative models presently in use. Machine learning can be used to construct the qualitative models for engineering domains. Qualitative descriptions of the mapping from control to performance space become even more important when * Department of Computer Science, and the Beckman Institute for Advanced Science and Technology. t Knowledge.Based Engineering Systems Research Laboratory, Department of Mechanical and Industrial Engineering, and the Beckman Institute for Advanced Science and Technology. Paper accepted May 1990. Discussion closes March 1991

1990 Computational Mechanics Publications

engineers are engaged in product design and process planning tasks. The qualitative descriptions will help engineers quickly comprehend the overall inter-dependencies among system parameters so that an inverse mapping (i.e., from performance space to control space) can be systematically identified to complete the design or process plan. Machine learning techniques can be used to convert the results of simulators into useful rules that facilitate these processes ~, where general heuristics a r e needed instead of specific, numerically exact calculations. The rules produced for the model must be comprehensive, proactive, and comprehensible8; goals that machine learning systems can accomplish. In other words, machine learning can help to make knowledge that is implicitly embedded in the simulation results more explicit. Machine learning can be used to construct models that are comprehensive, proactive, and comprehensible. Comprehensive models must be thorough, presenting information in whatever form needed such as mathematical formulas, I F - T H E N rules, and data tables 6. Rules produced through machine learning can be integrated* with these forms because they are semantically precise. Proactive models are flexible and work in situations other than those explicitly planned in advance or previously encountered. Learning systems take specific examples and induce more general concepts that apply to situations not yet seen. Comprehensible 8 models can easily be understood by the engineers using them. Engineers are reluctant to use models that are black * Currently by hand coded interfaces,but in the future the interaction will be more automated.

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

189

CAQ: A machine learning tool for engineering: B. L. Whitehall et al. boxes; understanding the internal workings of the models is important for an engineer. Learning systems can provide insight on how a complex set of governing equations work by producing general associative rules from the numerical results of the simulator. Inductive learning can be used to find the relationship between control space parameters and the resulting performance parameters and express the relationship in rule form. In the case of learning from examples (LFE) systems, the user specifies classes for the training events. Then, using those descriptions of the class based on the events, the system constructs rules that describe the mapping. LFE systems divide the control space into regions mapped to each class. Most LFE systems learn discriminant rules - rules whose antecedents are concise generalizations of the properties of the class. There are three main requirements for learning systems that handle real world engineering problems. First, the system must be able to handle both discrete* valued and continuous valued attributes. Neither type alone is enough. Handling both kinds of attributes allows a learning system to produce rules that are comprehensive and proactive. Second, the system must be able to handle some noise in the data. This allows the learned rules to handle unseen situations better, and thus makes the model produced more proactive. Third, the system must present the induced knowledge in such a way that the engineer can understand the rules produced and use them to gain insight into the problem (i.e., the rules must be comprehensible). These requirements are discussed in more detail in the next section. That is followed by a section describing CAQ, the new machine learning algorithm developed specifically for this type of problem. Section 4 presents some results based on data from a mechanistic simulator* for a turning process 19. This is used to show the utility of the CAQ algorithm. The paper closes with a discussion of future work. 2. REQUIREMENTS FOR MACHINE LEARNING ALGORITHMS IN ENGINEERING This section of the paper describes three important aspects of engineering problems that must be handled by machine learning systems for them to be useful tools for engineers. These requirements have served to motivate the development of CAQ. 2.1. Discrete and real values The distinguishing feature of engineering problems is that they contain both continuous (real) and discrete attributes. Neither type of attribute alone is satisfactory for expressing engineering problems. Discrete values can describe the large grain qualities of the problem such as grade of cut (rough vs. final) or the types of possible materials used (steel or wood). However, there are some attributes of problems that are naturally continuous, for example depth of cut and temperature that cannot be discretized without some prior knowledge. Some learning systems partition real valued attributes into discrete categories before learning the rules (see Section 3.3 on related work), but to get useful partitions the system * In this paper discrete valued attributes include what arc commonly called nominal, cyclic, linear, and structured attributes ~. All of these attributes have a range that is composed of a fixed set of discrete values. "1"This simulator is a deep, quantitative model based on physical laws.

190

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

would need knowledge about the domain (knowledge typically lacking in such systems). CAQ learns how to group events while the rule learning is being performed. There may be some continuous attributes whose values, when better understood, might become discrete, but for which there is not enough knowledge to perform this categorization. In other words, there exist quantitative measures in engineering that should not be treated qualitatively. Conversely, not all potentially continuous attributes need be continuous. There have been descriptive categories defined for some attributes that do not need to be treated as continuous. Forcing such items to take continuous (or even linear) values hinders the ability of learning algorithms to produce meaningful results. As an example of a discrete attribute, lubricants used in machining are bought in fixed grades even though they could be specially formulated to any desired viscosity. Also, there are values that engineers would like to handle quantitatively but are unsure how to measure or unable to measure their continuous aspects (e.g., because there is too much noise in the measurements). Thus, in order to find descriptions that can adequately describe conceptual relationships in engineering domains the learning system must handle both types of attributes effectively. CAQ was designed to deal with this specific problem. By allowing the engineer to formulate the problem in a language more closely representing the various types of values that are appropriate, the rules produced will be more useful. 2.2. Noisy data sets Noise comprises non-systematic errors in the values of attributes or class information ~4. Because most, if not all, data obtained from real engineering domains contain noise, a machine learning system that cannot tolerate noise is of limited value. There are at least three types of noise that can occur in a learning problem: uncertainty, error, and imprecision. Uncertainty occurs when the correct value of an attribute cannot be determined. Error occurs due to a misreading or incorrect coding of the information. Given another opportunity to check the value, the correct value would be observed and the error eliminated. For continuous attributes another form of noise is imprecision; the best obtainable measurement for the value is imprecise. Although noise due to uncertainty and error are problematic, there is no method of avoiding imprecision. So any learning algorithm that is working with continuous attributes must be able to handle some noise in those attributes. Engineers are quite comfortable using statistically based techniques to cope with such variations. They should expect the same from the learning algorithms they might use. CAQ can handle noise in continuous attribute values. The user of the system can specify what percentage of noise is assumed to appear in the data, if it is known. If the noise level is not supplied, the system is very conservative and assumes that less than 1% of the values provided are noise. 2.3. Rule comprehension Engineers want learning systems to produce rules that are not only correct but also comprehensible. If predictive accuracy were the sole requirement of the rules, then discrete values could be coerced into real values and

CAQ: A machine learning tool for engineering: B. L. Whitehall et ai. statistical methods applied. But these methods do not produce comprehensible results. People prefer simple, short rules, but in the engineer's situation the rules must also be consistent with given training examples. Standard numerical methods and many machine learning algorithms produce rules that are concept poor. That is, they produce a decision rule for a subset of events in the space but do not convey a useful meaning of the engineering concept underlying the role produced. A simulator can be thought of as executing a rule; but in this case the engineer must look up the numeric values in tables to determine what the rule actually means and to determine if the results can be generalized. An ideal system could produce concept rich rules constructed from attribute/value relations that are part of the engineering model for the problem. AQ ~° was selected as the basis for CAQ because AQ produces concept rich rules, compared to other learning systems 13.. The learning algorithms that produce decision trees 1'~ can handle noise but the results are generally not as concise as AQ rules. CAQ rules are understandable and clearly show the relationships among attributes. An example of a CAQ rule describing a sports car would have the form:

. .

and 1000 and [material-type = wood, aluminium'l indicates that wood or aluminium is the allowed type of material These selectors embody the construct known as internal disjunction zt. C o n j u n c t i v e C o n c e p t s are conjunctions of selectors where no two selectors of the concept refer to the same attribute. Rules are of the form conjunctive concept =~ concept class. For example, in the rule [300 < lathe-speed < IOOO][n~ttetial-type wood] . Smooth-cut,

lathe-speed and material-type are attributes, and there are two selectors in the concept.

Attributes define the dimensions and granularity of the event space. An event has a single value assigned to each attribute of the event space. For example, the r a n g e of possiblespeeds for a lathe machining station could be 0 to 5000 rpm and the types of materials handled could be wood, aluminium, and plastic. A particular example of a task on the lathe might include the facts that the lathe speed was set to 300 rpm and the material was aluminium. Selectors are predicates over attribute subranges. For example, the selector [300 < lathe-speed < 1000] indicates that the lathe speed is a value between 300

3.2. Basic AQ algorithm The AQ algorithm constructs discriminant description rules in a top-down fashion. The system is provided with two preclassified sets of examples. Events (examples) are observations from the problem domain, where each event is a list of attribute/value pairs. The set ofevents for which the algorithm is learning rules is called the positive class and the set of events not belonging to the desired concept are the negative class. AQ begins with the most general description (the empty, all inclusive conjunctive concept) and specializes the concept by adding one or more particular generalized selectors from the seed (a selected positive event) through the process known as extend against. The specialized concept covers fewer negative examples than the more general concept. When a concept is consistent (covers no negative events) it is saved in the concept set C (by disjoining it with the previous conjunctive concepts in C) and a new seed is selected. The seed event is selected from the positive events that are not covered by any of the concepts in C. The process continues until the concepts in C cover all of the positive events. Each of the conjunctive concepts in C is used to define a rule for the positive class. This mechanism ensures that all the positive events are covered by the concept set. The extend against process is used to maximally generalize the values in a selector. To extend the seed value for an attribute against the set of values of the negative set s for the attribute, the system determines the most general consistent value subrange for the attribute that includes the seed event's value on that attribute but does not intersect the negative set. In effect, each attribute of the seed is compared with the same attribute of a negative event. For nominal attributes the comparison produces the set of possible values for the attribute minus the attribute value of the negative event. In the case of linear attributes, extend against returns a range of values that include the value for the seed and immediately adjacent values not including the negative event's value. The results of extending the seed against each negative event are then combined by intersecting the values for each of the previous tests for each attribute. The system then selects these combined values to get a conjunctive description that covers the seed, but none of the negative events. (For further details see Ref. I0.) CAQ uses this basic methodology but supplements it by allowing continuous valued attributes.

* It is recognized that this is a subjective judgement and others have disputed these results tT. But the dispute has been raised on the issue of rule size, not ease of understandability.

* The negative set contains attribute values that if permitted in a selector would cause the concept to cover one or more negative events.

[ 0 < drag-coefficient < 0.4][top-sp, ccd > 120][tranu~ni~ion - m a n u a l ] ~ sports-car

This rule is quite naturally read as 'a sports car has a drag coefficient between 0 and 0.4, a top speed greater than 120 mph, and must have manual transmission'. 3. CAQ BACKGROUND The CAQ learning system is, conceptually, an extension of the AQ learning algorithm 1°, adapted so it can construct rules using both continuous valued attributes and discrete valued attributes. In addition to handling real values, the system is able to handle noise in the continuous attributes. Noisy values are handled by assuming a statistical distribution (e.g., a normal distribution) on the data values observed. The user can specify how much noise is expected in the data to aid the system in building better rules. Before describing CAQ in the next section, a brief introduction to AQ and related work is provided.

3.1. Terminology Before discussing the integration of the different types of attributes, some terminology will be introduced to help explain the AQ family of algorithms4'z°'z2"ls. 1.

2.

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

191

C A Q : A machine learniny tool f o r engineering: B. L. Whitehall et al.

There are difficulties in combining continuous and discrete attributes. Discrete values define a feature space of finite size, even though this space might be quite large. For continuous attributes this is not the case, so the search through the space must be more constrained. The extend against process does not always work well for continuous attributes. For example, if there is a large gap between the value for a positive event and the nearest negative event, should the system assume that all the space between these values signifies a positive instance? Probably not, unless one is very confident that the examples represent all the negative areas of the feature space. CAQ solves this problem by providing two options for specializing a concept, instead of just the one of AQ. CAQ uses a modified 'extend against', but first tries to specialize a concept by using 'distribution fit'. Distribution fit assumes a statistical distribution on the values of the observed events. With this assumption, the system is able to find an intermediate value locating the most likely break-off point between positive and negative events. When 'distribution fit' fails to find a way to specialize an attribute, a modified extend against procedure that works with continuous attributes is used. Consider Fig. 1 where the event space is a plane with X and Y values ranging from 0 to 100. All the negative events have both X and Y values greater than 50. In this case assuming a statistical distribution on the event will now allow any concept to be found, as the distributions between the negative event and the positive event set have too much overlap to determine a break point between the sets. However, the modified version of extend against works well by extending against the set of negative events instead of a single negative event. Both specialization techniques are described in Section 4. 3.3. Related approaches

CAQ is not the first system to integrate continuous and discrete attributes, but its approach is unique. IDS 14, PLS xr, UNIMEM 5, and STAGGER 18 all allow continuous valued attributes in addition to discrete ones. But in all four cases the values are first discretized before being used by the systems. CAQ does not analyze the real values and place them into discrete categories. Rather the real intervals for a concept are determined dynamically within the context of the current conjunctive concept being constructed. Only the events currently covered by iO0

d t.

+

~

~ m

+ +

4-

-

q. m

Y

+

50

4- 4. 4-

4+

4-

.q.

+

0

4-

+

4-

+

4-

4-

50

100

X

F/'~. I. Negatioes in one quadrant

192

4. CAQ A L G O R I T H M There are two ways that interval selector values for continuous attributes are determined by the system. The distribution fit method takes a statistical approach. The continuous extend against method is a modified version of AQ's extend against method. The CAQ algorithm processes discrete data types in essentially the same way as the standard AQ algorithm, so that process is not described here. The next subsections describe, first, the representational extensions required to allow for continuous attributes and then the two methods used for finding continuous intervals. In both methods, the intervals are determined by the events that are covered by the current inconsistent concept. A concept is inconsistent it if covers negative events in addition to positive events. CAQ does not discretize the continuous values of an attribute into 'meaningful' intervals outside of the context of an inconsistent concept. It finds intervals that are discriminatory for the events covered by the concept being specialized. 4.1. Representation

When a system handles only discrete values it can either keep the names of all the possible values or (if it is efficient) work with an encoded bit string representation of the values. For continuous values this will not work. A system could assume that the only values are the ones actually seen and encode those as discrete values. If there are too many of these values for the system to handle, the values can be aggregated into intervals (as was done in previous systems). CAQ uses a different approach. Instead, the selector values for a continuous attribute are kept as end points denoting a range of values. In order for an example's value to be covered by the selector it must fall within the specified range. CAQ allows the user to define a tolerance for each attribute to be used when evaluating a selector. The attribute f o o with a range of the positive real numbers and a tolerance of 0.005 would be defined to the system as: (foo continuous 0 +inf 0.0051

+

+ +

4-

+

+

the concept are used in determining where the interval boundary between positive examples and negative examples would be placed. An approach similar to that of CAQ is described in Ref. 1. But that system does not learn from provided examples. Instead there exists an oracle that answers queries about hypothesized points. The oracle is nice in that it does not introduce any noise into the problem by misclassifying the hypothesized points (examples). This allows the system to experiment and determine the correct concept to any desired level of accuracy. CAQ works with the given events from the feature space and makes some simplifying assumptions to facilitate the discovery of noise.

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

The system allows - inf and + inf to define negative and positive infinite end points. The tolerance is used to indicate the precision of the data. In this case, 1.000 and 1.005 are considered equal. CAQ gives up the efficiency of encoding the values as bit strings to have a more realistic description of the data, but for large numbers of events the continuous encoding of data is faster than treating data as discrete values. An

CAQ: A machine learning tool.lor engineering: B. L. Whitehall et al. for that assumed distribution, the current system, for the purpose of demonstration, assumes a normal distribution over the negative and the positive events being covered. It is important to note that the general CAQ algorithm is not limited to the normal distribution alone. Many statistical texts provide details on different distributions that could be used, e.g., Ref. 2. The user specifies a level of confidence in the data by giving 5 parameters to the system. The five parameters are listed and illustrated in Fig. 3. These parameters control the procedure for determining a break point between positive and negative values of a continuous attribute. The break point is the value that will be placed in the selector to distinguish the classes. A mark is created for each distribution and is computed as follows:

.°1 3O

TIME

2OO-

ZOO-

i00

500 Number

iO00 of

0

Continuous

0

Nominal

examples

1500 for

each

2000 class

Variables mark - ~ + (n * ol

F~. 2.

Variables

Where _ denotes either addition or subtraction, depending on whether the mean of the other class is to the right ( + ) of this class's mean or to the left ( - ) . p is the mean of the data, a is the standard deviation for the data, and n denotes the number of standard deviations allowed (given by the parameters SDeos or SDNEG). In other words, the m a r k is the place where the user strongly believes that the values are still accurate. The user is able to specify a percentage error of the data in terms of standard deviations. Figure 4 shows the three possible ways that the marks for the data can be configured. For these diagrams the negative events are on the left. In diagram A the marks are in compatible regions (e.g., M _ is a m o n g values on negative events, etc.). In this case P C T - B E T W E E N is used to determine where break point BP should be placed between the marks M _ and M +. Any negative event to the right of BP and any positive event to the left of BP are considered noise. The values in these cases are outside the confidence interval established by the parameters. In diagram B of the figure, the mark points meet and the break point is set to that position. Noise events are determined as above. In diagram C, the mark point for the other class is between the mark point and mean of the class, thus

Results of continuous and nominal processin 9

experiment was run on the concept demonstrated in Fig. 1, where the positive class is described by [0 < X < 50] or [0 < Y < 501. The test data sets had the same number of positive and negative events. The results are plotted in Fig. 2. The abscissa is labelled with the number of positive events (half the total events given) and the ordinate is the time in seconds it took for the system to run on a Texas Instruments Explorer I. Note, that when only few events (less than 100 of each class)are used the efficiency of bit encoding values outweights the complexity of the continuous processing. However, as the number of events grows, the situation changes. In this test there were only two attributes with 100 integer values encoded for the discrete representation. As more attributes and values for each attribute are used, the continuous processing gains a larger advantage over the discrete approach. Also, the continuous version always produced a description consisting of two rules for the class but when discrete values were used, up to four rules were required to describe the data. The processing of the continuous attributes does not eliminate the high order polynomial complexity of the AQ algorithm*, but it does significantly reduce the exponent.

CAQ Noise Management Parameters I. SDNEG-the recommendednumber of standard deviations from the mean of the negative events considered valud. Events outside this range may be noise. (Default--3) 2. SDt,os-the recommended number of standard deviations from the mean of the positive events considered valid. Events outside this range may be noise. (Default=3) 3. PCT-BETWEEN-the percentage from the negative "mark" to the positive "'mark" when the marks are not equal. (Default= 50 %) 4. MIN-SDNEG-the minimum number of standard deviations from the mean of the negativeevents considered acceptable. (Default= 2) 5. MIN-SDpos-tbe minimum number of standard deviations from the mean of the positive events considered acceptable. (Default= 2)

4.2. Distribution fit This section described the statistical approach used by C A Q in determining where the interval(s) for positive and negative events should be placed for a selector of a continuous valued attribute. When the system performs this algorithm an inconsistent conjunctive concept has been constructed that still covers some negative events along with some positive events. The system needs to specialize the concept by adding a selector that will 'uncover' the negative events and still leave covered as m a n y positive events as possible. To reduce the amount of data that must be processed, the system looks at only the events currently covered by the conjunctive concept. It is in this section of the algorithm that events can be identified as noise. Noise events are removed from the process of building a consistent concept. Although any type of statistical distribution can be assumed and the C A Q algorithm modified accordingly

M~

o

f °, M~N-SD,~.

°

I I I\ p_

* The true complexityof AQ has never been published, but it is believed to be high order polynomial.

Fig. 3.

m_

M.

/I

I I /If.

m.

~,,

Continuous parameters for CAQ

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

193

CAQ: A machine learning tool for engineering: B. L. Whitehall et al.

A #_

M_ B P

M+

iz+

B Iz_

M_ = M+

#_

M+ B P

Iz+

C

Fig. 4.

M_

t~+

Possible mark positions

requring a more complex computation. First the break point BP is determined using P C T - B E T W E E N . If BP is between/a - (MIN-SD * tr) and p + (MIN-SD * a) then the break point is thrown away and a selector with this attribute is not used in the conjunctive concept. If there are any events between BP and the mark M for the data set, the selector is dismissed, because the user has specified an interval that is allowed to contain valid events. Since those events do exist, had the break point been accepted at that position, the algorithm would be overriding the provided knowledge about the data. Any events farther away from/1 than M are classified as noise. If the user is very sure of the accuracy of the data, the MIN-SD for a class can be set equal to the SD for the class*.

4.3. Continuous extend against There are situations when assuming a distribution the data set will not work. Consider the data set demonstrated in Fig. 1, where there is a sharp boundary between the positive and negative values. Assuming a normal distribution, the system could never find a break point. In situations where no break point is found using the statistical methods, the system finds a break point by employing a form of extend against modified for continuous attributes. Although it might at first seem unreasonable to use extend against in situations where the statistical method failed, it works. It solves the problem of Fig. 1 and others where there is a sharp distinction in the values. If there is a better break point that could be found later (say after some other attribute first eliminates some covered negative events) then the inconsistent conjunctive

concept computed with this selector will be thrown away in favour of the better conjunctive concept. The first step in performing the modified extend against for a continuous attribute is to collect all of the covered negative events that are on the same side of the seed value as the originally selected negative value. Using the negative value closest to the seed, the system finds the most general consistent value subrange for the attribute that includes the seed event's value on that attribute but does not intersect the negative value. By doing this process the system insures that it will not have to adjust this selector when it sees other negative events on the 'same side' of the seed as the current negative event. No noise events can be determined when this process is used. The process is only productive when there are crisp boundaries for the classes. Otherwise the conjunctive concepts will not cover enough events to be accepted by the evaluation function (lexicographical evaluation functional or LEF, as in AQ). In the situations where continuous extend against works well, the values could be discretized. But it would be very difficult to determine where the class lines should be drawn without using the information in the inconsistent conjunctive concept. 5. D E M O N S T R A T I O N : U S I N G CAQ T O I N D U C E RULES F R O M A M A C H I N I N G O P E R A T I O N SIMULATOR To demonstrate the usefulness of CAQ, the system has been used with data (training examples) produced by a turning machine (lathe) simulator. The next sections describe the simulator and its application domain. After explaining the input examples, the results of CAQ are presented. This is followed by a comparison of the results of CAQ with results of AQ for the same data. A brief look at how CAQ's parameters effect its results is also presented.

5.1. The domain and the simulator Turning machines or lathes are used to remove material from cylindrical shaped parts. Modelling the process is quite complex as there are a number of input and output parameters to the process. Figure 5 shows the important parameters to the process. • • • •

Speed is the number of revolutions per unit time that the piece is being turned. Feed is the rate that the cutting tool is being moved along the longitudinal axis of the part. Depth of Cut is the distance between the uncut surface and the surface after the cut. Tool Angles describe the characteristics of the cutting tool and include back rake angle, side rake angle, end relief angle, side relief angle, end cutting edge angle, and side cutting edge angle. The characteristics of the tool are not important for the tests run with CAQ because the same tool was used for all the experiments.

r~d

S Z ~ R

Depth of Cut

* In order to compute the mark for distributions other than the assumed normal distribution, tt and o can still be used. Let M be the moment generating function for the distribution, then /z = M'(0) and a = M'(O) - [M'(O)]2.

194

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

~te~ial

Tool An91es J

Fig. 5.

L

Turning machine simulator

Surface

remo v al

F:Laish

rate

CAQ: A machine learning too~for enghleering: B. L. I~7dtehall et al.

--~ Jig. 6.

Speed

Feed

Turning machine process

Figure 6 shows the relationship of these input parameters to the part being machined. The output parameters to the process are: • • • •

Forces such as FL, FT, FR determine the work piece deflection and hence accuracy of the dimensions. Deflection of work piece. Material removal rate determines the production rate of parts. Surface finish is the quality of the parts surface and is important in precision parts.

5.2. Input specifications The input to the CAQ system for the turning machine experiment consisted of three classes. The results of running the simulator were divided into three classes representing different grades of cuts: rough, semi-finish, and finish. Each run of the simulator was placed into one of these categories. The classes were then used to let CAQ find rules for each of these classes. Because all of the simulator trials used the same cutting tool, there were three input parameters for each instance: speed, feed, and depth of cut. All were considered to be continuous attributes. Speed has a range of 149 to 650. Feed has a range of 0 to 0.025 and the range for depth of cut is 0 to 0.045. CAQ (like AQ) deals with two classes at a time. Three runs were made (one for each class) with each class's examples acting as the positive events and the events in the other two classes comprising the negative events. The noise management parameters for CAQ were set to the default values for all the runs described.

5.3. CAQ results Results produced by CAQ for the three cut classes (rough, semi-finish, and finish) are presented. The results of CAQ are of the form:

Where P is the number of positive events covered by the inconsistent concept of the partial star and N is the number of negative events covered. Negative events are covered because the description for the concept has not yet been made completely correct by extending it against all of the negative events, p% is the percentage of the positive events covered, and n% is the percentage of the negative events covered by the concept. Description is the inconsistent conjunctive concept and Class is the consequent of the rule. The CAQ system presents inconsistent conjunctive concepts because the engineer's knowledge can often at times be used to find useful results in what the system considers uninteresting. This can occur for two reasons: generality and consistency. General, concise rules are desired for expert systems because shorter rules can be matched more quickly. But, humans often want more context to a rule before 'firing' it. Engineers prefer more descriptive rules to aid in understanding the relationships between the antecedents and the consequent. Complete consistency in the constructed rules is an objective of machine learning systems. But, human engineers can use less than consistent rules as a guide to problem solving. Because of these two factors, it is beneficial to allow users to look at the inconsistent conjunctive concepts found by the system; the engineer might be able to use some of these rules in a way the system could not measure.

5.3.1. Rules for rough cutting There were 55 positive events and 177 negative events used for learning a description of rough cut. The rules identifying Rough-Cut are: 52 94%1521 [0.026< Depth <0.045] ~ Rough-Cut 13 24%121 [220.6
Most of the examples are covered by a single rule, the first one. Thus, if an engineer saw that the depth of cut was between 0.026 and 0.045, he could assume that a rough cut was occurring. The inconsistent conjunctive concepts produced while forming the above rules are: 41:1 175%:1%1 [149.0< Speed <238.8][0.0138< Feed <0.025][0.0223 < Depth <0.045]

Rough-Cut 41:9 |75%:5%| [149.0< Speed <238.8][0.0223< Depth <0.0451 -- Rough-Cut 39:3 171%:2%) [149.0< Speed <238.8][0.014< Feed <0.025] -- Rough-Cut 30:0 155%:0%1 [0.0158< Feed <0.025][0.026< Depth <0.045] ~ Rough-Cut 24:0 144%:0%) [149.0< Speed <208.2][0.0252< Depth <0.1)45] ~ Rough-Cut 13:0 (24%:0%) [220.6< Speed <238.8][0.0138< Fccd <0.02~[0.0223< Depth <0.045] Rough-Cut 8:0 115%:0%) [149.0
N % (n) O~r,cfiption - - C k ~

Where N is the number of positive events covered by Description and % is the percentage of the positive events covered. The number n is the number of events covered when this rule was constructed. N and n can be different, because many times a rule covers events that are also covered by previously bound rules. Description is a conjunctive concept that is the antecendent of a rule for Class. Inconsistent conjunctive concepts (intermediate concepts CAQ developed during its operation) are presented in the form: P:N {p%:n%] Description - - Class

An expert at using the turning machine simulator preferred the first inconsistent conjunctive concept produced to any of the completely correct rules produced for the class. This inconsistent concept was useful because it covered most of the positive events and only one of the negative events 11%). This rule was beneficial because value relationships (e.g., selectors) for all of the input attributes were described. An engineer could use this rule to understand how to set those attributes when a rough cut needed to be simulated.

5.3.2. Resui;s for semi-finish cutting The CAQ rules for the 82 semi-finish events when run

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

195

CAQ: A machine learning tool for engineering." B. L. Whitehall et al. against the 150 events of the other classes is: 41 50%(41110.0105< F~',d <0.025][0,0124< Depth < 0 . 0 2 2 ] - Semi-Finish 21 26%1211 [0.0< Feed <0.0147][0.0225< Depth <0.0274] ~ Semi-Finish 17 21%141 [277.6< Six~l <308.5] -- Semi-Finish 4 5% 131 [149.0< Speed <397.6][0,0091< Feed <0.01][0.0< D~th <0,0197] -- Semi-Finish 5 6*/° (31 [149.0< Speed <3,152][0.0091< Feed <0.0105][0.0< Depth <0.01~7] Semi-Finish 2 2°/° 121 [0.0071< Feed <0.025][0.0149< Depth <0.0156] ~ Semi-Finish 2 2% (2} [0.01 < Feed <0.0105][0.0< Depth <0.0175] -- Semi-Finish 1 1% 11t [473.5< Speed <477.5"], Semi-Finish I I% (I) [3980< Speed <650.0][0.0169< Depth <0.0171] -- Semi-Finish 2 2% 11) [294.4< Speed <397.6] -- Semi-Finish 33 40%(I) [0.0105< Feed <0.014][0.0172< Depth <0.045"] . Semi-Finish 10 12%111 [0.0225< Depth < 0 . 0 2 4 ] - Semi-Finish I 1% (11 [313.0< Speed <316.9] -- Semi-Finish

These results demonstrate one way in which a user can determine that there was noise in the events. The rules that cover few examples can be considered noise. From the number of rules covering only one or two events, it appears that either semi-finish is a hard to define concept or that there was a lot of noise in the data. The expert in this case preferred the first two rules as a description of the class. There were no interesting inconsistent conjunctive concepts for this class.

5.3.3. Results for finish cutting For the class of parts with a finish cut, there were 95 positive examples and 137 negative events. The results we re: 6366%16311396.7< Speed <650.0][0.0< Feed <0.0079] ~ Finish 15 16%115)[303.7< Speed < 394.6][0.0< Feed <0,010,1][0.0183< Depth < 0 . 0 4 5 ] - Finish 3335'/ol51 [00< Feed <0.01][0.0< Depth <0.0136] -- Finish 4 4% 141 [149.0< Speed <348.5"][0.0< Feed <0.010"7"][0.013.4< Depth <0.0165] ~ Finish 25 26%131 [400.3< Speed <474.1][0.0< Feed <0.0106] ~ Finish $ 8% 12} [371.3< Speed <650.0][0,017< Depth <0.0183] ~ Finish I I°/o (I) [0.0176< Depth <0.0178] -- Finish 1 I% 111 [0.0104< Feed <0.0106] = Finish 1 I% 111 [332.1< Speed <335.7] ~ Finish

A few of the inconsistent conjunctive concepts produced were: 41: I 175%:1%) [149,0< Speed < 238.8][0.0138 < Feed <0.025][0.0223 < Depth <0.045] Finish 92;10 (97%:7%} [303,7< Speed <650.0][0,0< Feed <0.0104] ~ Finish 92:14 197%:10%) [0.0< Feed <0.0106][0.0< Depth <0.0214] ~ Finish 84:9 188%:7%) [303.7< Speed <650.0][0,0< Feed <0.0104][0.0< Depth <0.0204] Finish 82:g 186°/.:6%1 [3,13.2< Speed <650.0][0,0< Feed <0.0107] ~ Finish

Although the first two rules the system found are very good, the expert in this case preferred the third inconsistent conjunctive concept (covering 84 positive and 9 negative events) the best. This was considered the most useful rule for helping someone decide what simulator settings to use when a finish cut was desired. Because the rule has selectors for all three input parameters, it is useful in restricting the ranges of values to be considered for this type of cut.

5.3.4. Discussion of results The results from CAQ can be used in a number of ways. The algorithm was developed to learn classification rules. Thus, someone can compare an example to the rules produced and determine in what class the example belongs. Rules that contain values for all of the parameters were found useful by the expert in determining what should go into the simulator to produce a desired result. Used in this way, CAQ provides inverse engineering capabilities in supporting design and planning tasks. It is worth noting that each run of CAQ took under 2 minutes on an engineering workstation. This allows for a quick test to determine if useful rules can be

196

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

found from a current set of data. Such tests provide the opportunity for an engineer to determine a rule before running many lengthy experiments on the simulator. If good rules cannot be found, more runs on the simulator must be made, but little time is lost.

5.4. Comparison with AQ results The same turning machine data was run on AQ, to determine if the addition of continuous attributes allowed the system to be more useful for engineers. The output for AQ is similar to that of CAQ, but here the selectors are dealing with discrete valued attributes so the output selectors are of the form [name = = low-value_highvalue] where .. indicates that any value between the low and high values is acceptable. The domains for AQ were created by defining a discrete attribute for each of the three parameters. The acceptable values for each attribute were just the values found in the examples. For the speed attribute there were 229 values, 121 values for feed, and 153 values for depth*. The output for the class of events with a rough cut finish is: I I% 1 1% 26 47% I 2% 26 47%

[Speed-163.2..214.8][Feed-0.0101..0.OI99][l~pth,0.0234..0.0,1] ~ Rough-Cut [Feed= 163.2,.24.4.0][Feed=O,0124..0.O199][Depth-O.0217..O.0229] ~ Rough-Cut [Speed - 216.3..286.0][Feed = 0.014..0.0199][l~pth ~ 0.024..0.04] ~ Rough-Cut [Speed~21g.g..2gl.I][Feed=0.OlaT..O.O199][Dcpth-O.0223..O.024] ~ Rough-Cut [Speed- 163.2..214.8][Depth ~0.0266..0.04] ~ Rough-Cut

Although the results are not bad, their coverage is inferior to those produced by CAQ, where one rule covered most of the events. AQ's results for the Semi-Finish class (which follow) were not as easily comprehensible as CAQ's results. 1 I% I I% I I% I 1% 2 2% I I% 2 2% I 1% 10 12% I 1% 12 14% I I% 15 Ig% 3 3% 7 8% 7 8% 2 2% 3 3% 3 3% I |% 2 2% 1 1% 4 4°/°

[Sp-~d-317.6..397.6][Feed-0.OO83..O.OOgg][Dcpth-O.O0~,.O.O186] -,. Semi.Finish [Spced=321.9..371.3][Feed-0.0068..O.0094][Depth-0.OI3..0.OI76] - Semi-Finish [$peed-432.7..457.9][Feed-0.0Or..O.O145][Depth-O.O16,.O.0211] ~ Semi, Finish [Speed-355.l..396.T][Feed-O.002.,0.0094][Depth-O.0203..O.0208] -- Semi-Finish [Speed-230.6..387.I][F~l=O.002,,O.OI43][Depth-O.OI65..0.0|74] -- Semi-Finish [$peed-398.O..635.1][F~1-0.0043..O.0145"J[Depth-O.O|69..0.OI7"], Semi-Finish [$p~d-23g.O..350.0][Feed=O.0062..0.OI71][Dcpth,O.01~.,O.02OI]--Semi-Finish [Speed = 316.6..332.1][Feed-O.OO68..O.OI45][Depth- 0.O123..0.0158] -- Semi.Finish [Sp¢cd-230.6..350.0][Feed =O.0104..O.OI45][Dcpth-O.OI25..O.0223] -- Semi-Finish [Speed=398,O..635.1][Feed-O.002..O.OI45][Depth=O.O201..O.0211], Semi-Finish [$peed-230,6..635.1][Feed-O.(X)2..0.OI45][Dcpth-O.0266..O.0272], Semi-Finish [SIx:ed-321.9..371.3][Fc~l-0.OO75..O.009g][l~:pth-O.O|7g..O.O194] - - Semi*Finish [Speed- 185,7..228.9][Fe0d =O.~O2..O.OI78][D~th-O.0Og..O.0271] -- Semi-Finish [Speed=321.9..350.0][Feed-O.OO75..O.O145][Depth-O.017g..O.020g]--Semi-Finish [$peed-231.5..635.I][Feed-O.OI48..0.O163][Depth-O.00&.0.0322] -- Semi-Finish [$peed-335.7..635.t][Feed-O.0094..O.0145][Depth-O.008..O.O163] ~ Semi-Finish [Speed=230.6..243.7][Fced=O.OO2,.O.O178][D~pth-O.0Og..O.0223] -- Semi-Finish [Sp~:d=317.6..397.6][Feed=O.OO77..O.0Q94][Dcpth-O.013..00176] ,semi-Finish [SFex:d-230.6..249.8][Fe~:d-O.0145..O.O|58][Dcpth=0.O(~..O.0301] -- Semi-Finish [Speed = 398.O..414.6][Feed =O.O02.,O.0145][D~pth-0.0172..0.0196] -- Semi-Finish [$pecd=233.2..635.1][Feed=O.O144..O.0145][Dcpth-0.0226..0.O336] -- seTat-Finish [Speed-473.5..626.p][Feed-0.002..O.OI45][Dcpth~O.0146..O.OI48] -- Semi.Finish [Specd,23G.6..635.1][Fcod=O.OO2..O.O145][Dcpth-O.0214..O.0217] -- Semi-Finish

CAQ produced different four rules that cover over 20% of the events in this set, including a rule that covered half of the events. All the rules produced by AQ cover less than 20% of the events. CAQ also required fewer rules to cover the complete set of events than did AQ (13 rules vs 23 rules). For the Finish cut class AQ found the rules: I% I% I% 1% I% I% 1% I%

[Speed-279.5,,635.1][Fecd-O.OO2.,O.OI4TJ[Depth-O.OI91..O.OI96] ~ Finish [Speed- 319.6..635,1][Fecd-O.OO2.,O.OI33][Depth-0.O197..0.OI98] -- Finish [Sl~"ed-279.5..635,1][Depth-0.0191..0.0194] -- Finish [Speed~279.5..635.1][Feed-0.OO2..O.0148][Depth-O.OI89] -- Finish [Speed - 290.7..326.2][Depth ,0.008..0.0135] -- Finish [Speed, 290.7..363.9][Depth-O.0147..O.015] ~ Finish [Speed,319.6..635.1][Feed-O.OO2..O.Ollg][D~pth-0.OI98..O.0203] -- Finish [Speed,290.7..384.5"J[Depth-O.OI37..O.0143] -- Finish

* The values shown in the results look like real values, but they are just names for the nominal values. The names chosen were the actual values in the given events. No other values are recognized by the system.

CAQ." A machine learnin 9 toolJor engineerin9: B. L. 14"hitehall et al. 3 2

3% 2%

4 4% 2 2% 6 6% 3 3% 2 2% 4 4% 8 8% 2 2% 16 16% 3 3% 4 4% 2 2% 25 26% I 1%

[Spe~l=2~9.$..63$.l][F¢¢d =0.002..0.0092][Dcpth=0.0177..00184] = Fimsh [Speed = 2"9.L.635.1][Dcpth = 0...01ST] . Finish [Spe~l=-~71.5..aJS.l][Depth = 0.0171..0.0174] - - Finish [ S p e e d - 227.5..635.1][F~d =0,002..0.0101][Depth = 0 . 0 2 1 7 . . 0 . 0 2 2 g ] . Finish [Sl:~'d =3tM..~..635.1][Fe~l-0.002..0.00113][Dq~th-0.OI47..0.O169] ~ Finish [Speed = 319.6..635.1][Depth - 0.0201..0.0203] - - Finish [Slx'ed = 3&.t.5..635.1][Feed-O.002..O.0078][IX'pth = 0 . 0 0 8 . . 0 . 0 1 4 9 ] , Finish [Slx'ed - 2,,'-t.4..39.tA][Depth, 0 . 0 2 0 8 . . 0 . 0 2 0 9 ] , Finish [Speed - 332.0..635.1][Depth=O.01211..O.OI35] - - Finish [Speed ---xitO.7..363.9][Depth-0.OI56..O.0164] - - Finish [Speed - 384.~;..635.1 ] [ F e e d - O . 0 0 2 . . O . 0 0 8 3 ] [ D e p t h - 0 . 0 1 4 7 . . 0 . 0 1 6 4 ] [Speed - 278.1..635-1][Depth=0.0211..0.0214] - - Finish

~

Finish

[Speed - 3 5 7 . l . . 6 3 5 . 1 ] [ l ~ p t h - 0 . 0 1 3 7 . . 0 . 0 1 4 3 ] , Finish [SIx'ed-2-t'~A..635.1][F~l-0.002..0.0091][Depth=0.0175..O.OI77] - - Finish [Speed-332.0..635.1][Dcpth-0.O08..0.0125] ~ Finish [ S I x ' ~ - 2-I-t.4..317.7][Dcpth =0.0198..0.0213] -- Finish

Again AQ did not perform as well as CAQ. For all three classes, CAQ performed better because of the extended representation and the way it can handle noise in the data.

5.5. Parameter influences on performance The CAQ system has a number of parameters that control the construction of rules. The current version of C A Q assumes a normal distribution of data values and the noise parameter is specified in the number of standard deviations from the mean. SDpos and SDNEo indicate which events are considered valid, as described earlier. One would expect that as the system allows more events to be classified as noise, that the number of disjunctions required to describe a class would shrink. As this occurs, the best rule should also cover more of the events and the system should run faster. These intuitions were confirmed in some tests. Table 1 shows the results of decreasing the confidence parameter for the data. The table is for the turning machine data used previously. The columns are labelled with the noise threshold in standard deviations from least to most noise allowed (i.e., in the first column of data, events farther than 2 standard deviations from the mean of the values are considered noise). The rows are divided into groups for each class of rough, semi-finish and finish cut. Within each group the rows are labelled with #disjunctions for the number of disjunctions describing the class, %best gives the percentage of events covered by the best disjunct, # noise is the number of events categorized as noise, and time is the number of seconds it took for the run on a Texas Instruments Explorer II workstation.

Table 1. Parametereffects on CAQ

Noise threshold in standard deviations Measure

2.0

#disjunctions %best #noise time

3 .945 0 10

#disjunctions */.best # noise time

13

#disjunctions %best # noise time

1.5 1 .964 3 3 11

.488 I 50

9 .663 0 26

1.0

Class

I 1.00 5 3

Rough

5

Semi-finish

.488 39 18

2 .905 I0 8

2 .916 22 7

In this paper, a system that can meet the requirements of a machine learning algorithm for engineering problems was presented. C A Q allows engineers to provide data in the form that is normally used to describe the problem. There does not need to be a conversion from one type to another. The C A Q system can handle limited amounts of noise, a must for any system working on real engineering problems. Also, unlike m a n y data analysis systems, CAQ presents its results in a clear and concise form that allows an engineer to gain useful knowledge about the problem. There are a number of areas of CAQ that need to be worked on in the future. One of the most interesting is trying to assign a confidence value to each rule produced. The less noise introduced by a rule or the closer it adheres to the bounds specified by the user, the higher the confidence level should be. The system should also allow the user to specify different distributions on the data. Currently it assumes a normal distribution. Another area where CAQ could be improved is to allow for conjunctive concepts using the discrete values that are not 100% correct. In other words, they could cover some negative events. This is both an advantage and at the same time a disadvantage. This might allow more useful concepts to be found, but currently the engineer can look through the inconsistent conjunctive concepts and see if there are interesting results in this area. The CAQ system makes a step towards providing engineers with a machine learning tool they can use effectively in practice. The system meets the basic requirements for such tools. Most importantly it allows discriminant rules to be constructed that contain both nominal and continuous valued attributes. The system is able to handle small amounts of noise in the input data, and produce rules that are in a format an engineer or expert system can use.

7. A C K N O W L E D G E M E N T S We would like to thank all the members of the Knowledge-Based Engineering Systems Research Laboratory for.their input when CAQ was being developed. A special thanks to Sudhakar Yerramareddy for providing the turning machine data and help in interpreting the C A Q results for the data. Funding for this research was partially supported by the Defense Advanced Research Projects Agency under grant N00014-87-K-0874, National Science Foundation under grant D M C 86-57116, a research contract from Digital Equipment Corporation, and by a gift from Texas Instruments, Inc. 8. R E F E R E N C E S I 2

.610

5 42

6. C O N C L U S I O N A N D F U T U R E W O R K

Finish

3

4 5 6

Gross, K. P. Incremental multiple concept learning using experiments, Proceedings of the 1988 International Machine Learning Workshop, Ann Arbor, MI, June 1988.65-72 Hogg. R. V. and Craig, A. T. Introduction to Mathematical Statistics. Macmillan Publishing Co., Inc., New York. 1978 HolT, W. A., Michalski, R. S. and Stepp, R. E. INDUCE 3: A Program for Learning Structural Descriptions from Examples, Technical Report UIUCDCS-F-83-904, Department of Computer Science, University of Illinois, Urbana. IL. 1983 Larson, J. and Michalsio R. S. Inductive inferenceof VL decision rules, SIGART Newsletter 63 {June 1977), 38---44 Lebowitz,M. Categorizing numeric information for generalization, Cognitive Science, 1985, 9(31, 285-308 Lu, S. C. An intelligent framework for engineering decision

Artificial Intelligence in Engineering, 1990, Vol. 5, No. 4

197

C A Q : A machine learnin 9 tool f o r engineering." B. L. Whitehall et al.

7

8

making, Society of Automotive Engineers Technical Papers. Series 870562. Detroit. MI. February 1987 Lu. S. C. and Chen, K. A machine learning approach to the automatic synthesis of mechanistic knowledge for engineering decision making, Journal of A rtificial Intelligencefor Engineering Design, Analysis, and Manufacturing. December 1987, !(2), 109-118 Lu, S. C. Machine learning approaches to knowledge synthesis and integration tasks for advanced engineering automation,

National Conference on Artificial Intelligence. Philadelphia. PA. 13

14 15

Third International Workshop on Intelligent Manufacturing Systems, Dubrovnik, Yugoslavia, August 1989 9

10

11

12

198

Lu, S. C. and Tcheng, D. K. Building layered models to support engineering decision making: a machine learning approach, to appear in Journal of Engineeringfor Industry, American Society of Mechanical Engineers Transactions, November 1989 Michalski, R. S. Synthesis of optimal and quasi-optimal variable valued logic formulas, Proceedings of the 1975 International Symposium on Multiple-Valued Logic, Bloomington, IN, 1975, 76--87 Michalski, R. S. A theory and methodology of inductive learning, in Machine"Learning: An Artificial Intelligence Approach, R. S. Michalski, J. G. Carbonell. T. M. Mitchell (ed.), Tioga Publishing Company, Palo Alto, CA, 1983, 83-134 Michalski, R. S., Mozetic, 1., Hong, J. and Lavrac, N. The multi-purpose incremental learning system AQ 15 and its testing application in three medical domains, Proceedings of the

Artificial Intelligence in Enoineering, 1990, Vol. 5, N o . 4

16 17

18

August 1986, 1041-1047 O'Rorke, P. V. A comparative study of inductive learning systems AQI 1P and ID-3 using a chess endgam¢ test problem, Report No. UIUCDCS-F-82-899, Department of Computer Science University of Illinois, Urbana, IL, 1982. (ISG 82-2) Quinlan, J. R. Induction of decision trees, Machine Learning, 1986, !(1), 81-106 Reinke R. Knowledge Acquisition and Refinement Tools.for the ADVISE Meta-Expert System, M.S. Thesis, University of Illinois at Urbana - Champaign, 1984 RendeU L. A new oasis for state-space learning systems and a successful implementation, Artificial Intelligence, 1983, 20(4), 203-226 Rendell L., Benedict, P., Cho, H. and Seshu, R. Improving the design of rule-learning systems, Report No. UIUCDCSR-87-1395. Department of Computer Science, University of Illinois Urbana, IL, 1987 Schlimmer, J. C. Learning and representation change,

Proceedings of the National Conference on Artificial Intelligence, 19

Seattle WA, July 1987, 511-515 Zhang, G. M., Yerramereddy, S., Lee, S. M. and Lu, S. C. Simulation of intermittent turning process, American Society of

Mechanical Engineers WAM Symposium on Control Issues in Manufacturing Processes, 1989