Hybrid machine learning: Myth and reality

Symbiosis of Human and Artifact Y. Anzai, K. Ogawa and H. Mori (Editors) © 1995 Elsevier Science B.V. All rights reserved.

1083

Hybrid Machine Learning: Myth and Reality Vassihs S. Moustakis a and Gavriel Salvendy b ~Department of Production and Management Engineering, Technical University of Crete, and Institute of Computer Science, FORTH, PO Box 1385, Herakhon 71110, GREECE bNEC Professor of Industrial Engineering, School of Industrial Engineering, Purdue University, 1287 Grissom Hall, West Lafayette, Indiana 47907, USA.

Abstract An important issue to consider when applying Machine Learning (ML) in a real world task is the selection of a system, algorithm or approach which should be used. In this context couphng of the right ML approach with the task at hand is not trivial. This paper reports the prehminary results of a research which targeted to couphng ML approaches with generic intelligent tasks. Prehminary analysis makes it clear that in most of tasks apphcation of a single ML approach is not satisfactory and that hybrid formations are necessary.

Introduction An important issue to consider when applying Machine Learning (ML) in a real world task is which system, algorithm or approach should I use? In this context coupling of the right ML approach with the task at hand is not trivial. As a matter of fact in some recent papers the apphcabihty of ML approaches to tasks is reviewed, i.e., [1, 3, 9]. These papers adopt a p r o b l e m - driven approach to the apphcation of ML in the workplace that complements earher research based ML classification schemes [7]. In addition, the problem - driven approach is an expected evolution and derives from the fact that ML has been demonstrated to be an effective 'team - player' in a range of real world apphcations [3, 5] To assess the effectiveness of ML approaches with respect to a range of intelligent tasks we conducted an extensive survey to elicit expert judgement from ML researchers. The survey was conducted using electronic mail and more than a hundred ML researchers provided input to it. This paper presents prehminary results from this survey and further discusses the concept of hybridness in machine learning as a prerequisite for using ML to support intelligent tasks. In section 2 we present the questionnaire upon which the survey was based. In section 3 we discuss the results and we conclude the paper in section 4 by discussing hybrid learning in relation to out results and identifying areas for further work

1084 on the subject.

ML Questionnaire The questionnaire covered twelve generic ML approaches and nine generic intelligent tasks. We summarize approaches and tasks in Table 1. Next to each approach or task we cite the acronym with which each term is referred in this paper. A complete definition of terms is excluded; however, the reader is kindly referred to the work of Michalski and Kodratoff [7], Kodratoff et al. [3] or Langley [5] for explanation of terms. Table 1: ML approaches and intelligent tasks ML Approaches Knowledge tasks Classification (CLS) Explanation Based Learning :(EBL) Prediction (PRD) Statistical Learning (SL) Optimization (OPT) Reinforcement Learning (RL) Planning (PLN) Neural Nets (NN) Scheduling (SCH) Genetic Algorithms (GA) Knowledge Acquisition (KA) Conceptual Clustering (CC) Knowledge Refinement (KR) Similarity Based Learning (SBL) Execution & Control (EC) Inductive Logic Programming (ILP) Conflict Resolution (CR) Qualitative Discovery (QD) Symbolic Empirical Learning (SEL) Analogical Learning (AL) Theory/model Revision Systems (TMR) Responders were first asked to assess their level of familiarity with regards to ML approaches and intelligent tasks and then to assess the degree of effectiveness of each ML approach with respect to each intelligent task. In both instances a seven point discrete scale was used to capture expert judgement, see Table 2. In self assessment values 2, 4, and 6 were used to capture in-between judgement. Table 2: Assessment scale description Value Self assessment with respect to a ML approach or task Not familiar at all Has some knowledge of Used it at least once/studied extensively Exceptionally well familiar

Effectiveness rating of a ML approach wi respect to each task Not applicable Little Some

Good Very good Excellent Does everything

1085

3

Initial results

Self assessment judgment values regarding mean level of familiriaty about ML approaches and intelligent tasks are summarized in Table 4. In this Table statistics correspond to 103 responses. (~u denotes mean value and a denotes standard deviation). In terms of ML approaches we note the following: • Responders were more familiar with well established ML approaches such as SBL, SEL and SL and less familiar with novel ML approaches, such as RL or approaches which are more complex, such as GA, TMR and QD. The distribution of familiarity indices agrees with the distribution of scientific papers published in the Machine Learning and other related with ML scientific or knowledge based systems journals; although formal research would be necessary to verify this hypothesis. • The rather high mean familiarity index ILP received follows from the fact that most of the responders were from Europe; most of the ILP community resides in Europe. The distribution of responses in terms of familiarity with respect to intelligent tasks confirms ML application reality, i.e., most applications correspond to classification and hence researchers are most familiar with it. On the other hand, lower mean ratings correspond to either difficult tasks, such as SCH or EC, or even to tasks which are not well defined, such as CR. Table 4: Familiarity statistics (N = 103) # o" Intelligent Task ML approach 4.8 1.7 CLS EBL SL 4.5 1.6 PRD 3.3 1.5 OPT RL NN 4.2 1.4 PLN 3.9 1.5 SCH GA CC 4.3 1.7 KA 5.5 1.7 KR SBL 4.1 1.8 EC ILP 3.1 1.7 CR QD SEL 5.1 2.0 AL 3.5 1.6 TMR 3.8 1.9

#

o"

6.2 5.5 4.3 4.1 3.5 4.9 4.4 3.0 2.9

1.1 1.4 1.6 1.7 1.6 1.6 1.9 1.9 1.6

In Table 5 we summarize mean values regarding the effectiveness of each ML approach with respect to each of the intelligent tasks. ML approaches are listed in the first column and intelligent tasks are listed in the first line of Table 5. Overviewing the values listed in Table 5 the following can be asserted: • Some ML approaches did not prove effective in none of the intelligent tasks. For instance, this is true with EBL which did not get a higher than 4.6. However, this

1086 situation should not by any means be interpreted that EBL is not useful; rather it indicates that EBL may be useful in some components of SCH or KR or even EC while the remaining task components may be readily handled by other ML approaches. • Clearly some ML approaches are superior in some tasks while judgement regarding their performance in other tasks is not positive. Such approaches include SL, NN, CC, SBL, and SEL. The thrust of ML systems that carry the tag of these approaches are very effective in handling massive amounts of data using rather simple (or flat) representation schemes (such as attribute value pairs). • None of the ML approaches received a mean effectiveness score value higher than six which indicates that none of the ML approaches can support all to any of the intelligent tasks considered. • Entries of Table 5 are based on expert judgement. However, the real value of a ML approach can only be judged with extensive experimentation. In that respect entries are indicative and by no means exclusive regarding the effectiveness of a given ML approach in a given intelligent task. There exist papers that report applications in which more than one ML approach has been used in cooperation with another ML approach to satisfy different objectives within the context of a single application. • Many responders indicated the diffculty in assessing the effectiveness of a single ML approach to a given intelligent task given that most applications incorporate a range of tasks. Table 5: ML a p p r o a c h - intelligent task effectiveness CLS PRD OPT PLN SCH KA KR EBL SL RL NN GA CC SBL ILP QD SEL AL TMR

3.4 5.4 3.3 5.3 4.3 4.5 5.3 3.7 2.6 4.9 3.2 3.3

2.9 5.0 3.4 4.8 4.3 3.5 4.7 3.4 3.3 4.5 3.7 3.0

2.8 3.9 4.1 4.6 5.4 2.2 3.1 2.3 2.3 2.9 2.6 2.6

4.6 2.7 3.6 2.6 3.2 2.5 3.2 3.3 2.6 3.4 4.3 3.4

3.9 3.0 3.7 3.4 4.2 2.4 3.2 2.9 2.6 3.3 3.4 3.0

4.4 3.5 2.5 2.5 2.7 4.5 4.8 4.1 3.8 4.8 4.0 4.6

4.5 2.9 2.7 2.4 2.7 3.4 3.5 4.6 2.9 3.8 3.6 5.7

EC 3.0 4.1 4.8 4.8 4.6 2.4 2.9 2.6 2.4 3.0 2.5 2.7

CR 3.7 2.8 3.1 2.4 2.9 2.1 3.6 2.9 2.3 3.3 3.3 3.5

In Table 6 we summarize the correlation coefficients between ML approaches given their performance with respect to the intelligent tasks listed in Table 5. Correlation results confirm that: • The taxonomy of ML research by Michalski and Kodratoff [7] in which heterogeneous ML approaches are placed next to each other based on type of input, inference strategy, and learning goal.

1087 • Similarities between the application profile of different ML approaches. For instance, NN and GA share very few things in common compared to each other in terms of conceptualization and learning formalization. However, from an application viewpoint they correlate highly with each other. The same holds between NN and SL. Highest correlation is achieved between CC and SEL (i.e., equal to 0.96) which means that both are equally relevant given the range of aforementioned intelligent tasks. • A clear dichotomy can be derived from entries of Table 6. Negative correlation distinguishes ML approaches that are not only different with each other (because the same is true between approaches that have positive correlation) but are also different in terms of ML "philosophy". For instance, the -0.69 correlation between SL and EBL reflects the wide difference between the two; SL can handle massive amounts of simple data in the abscence of domain models while EBL is most effective with small amounts of complex data in presence of domain models. Table 6: Correlation between EBL SL RL 1.00 EBL -0.69 1.00 SL -0.63 0.20 1.00 RL -0.85 0.89 0.60 NN -0.85 0.57 0.78 GA 0.20 0.52 -0.60 CC -0.02 0.64 -0.57 SBL 0.65 -0.04 -0.81 ILP 0.33 0.14 -0.63 QD 0.16 0.54 -0.65 SEL 0.76 -0.33 -0.68 AL TMR 0.74 -0.38 -0.79

4

ML approaches NN GA CC

SBL

ILP

QD

SEL

AL

TMR

1.00 0.86 0.13 0.24 -0.41 -0.22 0.11 -0.61 -0.66

1.00 0.53 0.61 0.97 0.33 0.20

1.00 0.67 0.70 0.62 0.89

1.00 0.74 0.61 0.50

1.00 0.47 0.37

1.00 0.49

1.00

1.00 -0.27 -0.16 -0.71 -0.43 -0.30 -0.72 -0.80

1.00 0.89 0.75 0.73 0.96 0.36 0.46

Hybrid Learning

Survey results confirmed our initial hypothesis that in most intelligent tasks a single ML approach (and related ML system realization) is not sufficient. In addition, several ML application studies confirm this hypothesis- see for instance [2], [4], [10], or [11] - to mention only a few of them. A recent development in ML has been multistrategy learning [6, 8]. Multistrategy ML stresses the combination of several basic strategies and paradigms to support learning. However, hybrid ML is different; the GA-WKNN algorithm in [2] combines the optimization capacity of a genetic algorithm with the classification capabihties of the weighted k nearest neighbors algorithm. Hybrid learning differs from multistrategy learning. Hybrid stresses combination while multistrategy focuses on integration. Initial survey results provide sufficient basis that hybrid ML research can support the production of systems that would be capable of

1088 enhancing the degree of effectiveness of ML in supporting intelligent tasks. Looking ahead we plan to conduct additional surveys and experimental work around selected intelligent tasks and ML approaches to: (a) index ML approaches and intelligent tasks and (b) derive effective ML hybrid formations.

References [1] C Brodley. Addressing the selective superiority problem: Automatic algorithm/model class selection. In Proceedings of the l Oth International Conference on Machine Learning, pages 17-24. Morgan Kanfmann, 1993. [2] JD Kelly Jr and L Davis., A Hybrid Genetic Algorithm for Classification. In Proceedings of IJCAI-91, pages 645-650. AAAI Press, 1991. [3] Y Kodratoff, V Moustakis, and N Graner. Can machine learning solve my problem? Applied Artificial Intelligence, 8(1):1-32, 1994. [4] Y Kodratoff, D Sleeman, M Uszynski, K Gansse, and S Craw. Building a machine learning toolbox. In B Le Pape and L Steels, editors, Enhancing the Knowledge Engineering Process- Contributions from ESPRIT. Elsevier, 1992. [5] P Langley. Areas of applications for machine learning. In Proceedings of the fifth International Symposium on Knowledge Engineering, Sevilla, Spain, 1992. [6] RS Michalski. Inferential Theory of Learning as Conceptual Basis for Multistrategy Learning. Machine Learning, 11(2/3):111-152, 1993. [7] RS. Michalski and Y. Kodratoff. Research in machine learning: Recent progress. classification of methods, and future directions. In Y. Kodratoff and RS Michalski, editors, Machine Learning: An Artificial Intelligence Approach: Vol. III. Morgan Kanfmann, 1990. [8] RS Michalski and G Tecuci. Machine Learning: A multistrategy approach, Volume IV. Morgan Kanfmann, 1993. [9] D Michie, DJ Spiegelhalter, and CC Taylor. Machine Learning, Neural and Statistical Classification. Ellis Norwood, 1994. [10] K Morik, G Potamias, V Moustakis, and G Chaxissis. Knowledgeable learning using MOBAL: a clinical case study. Applied Artificial Intelligence, 8(4):579-592, 1994. [11] VS Moustakis. CEG: A case based decision modeling architecture. European Journal of Operations Research, to appear, 1995. Acknowledgements: We wish to thank Mr. Michalis Blazadonakis, Ms. Lena Gaga and Dr. George Potamias for their help in the formulation of the questionnaire and the processing of responses.