- Email: [email protected]
A computational model for distributed knowledge systems with learning mechanisms
ExpertSystemsWithApplications,Vol. 10, No. 3/4,pp. 417--427,1996 Copyright© 1996ElsevierScience Ltd Printedin GreatBritain.All rightsreserved 0957-4174/96$15.00+0.00
Pergamon
S0957-4174(96)00020-6
A Computational Model for Distributed Knowledge Systems with Learning Mechanisms HIROKO A I B A t AND TAKAO TERANO Graduate School of SystemsManagement,The Universityof Tsukuba, 3-29-10tsuka, Bunkyo-ku,Tokyo 112, Japan
Abstract--This paper addresses the issues of machine learning in distributed knowledge systems, which will consist of distributed software agents with problem solving, communication and learning functions. To develop such systems, we must analyze the roles of problem-solving and communication capabilities among knowledge systems. To facilitate the analyses, we propose a computational model: LPC. The model consists of a set of agents with (a) a knowledge base for learned concepts, (b) a knowledge base for problem solving, (c) prolog-based inference mechanisms and (d) a set of beliefs on the reliability of the other agents. Each agent can improve its own problem-solving capabilities by deductive learning from the given problems, by memory-based learning from communications between the agents and by reinforcement learning from the reliability of communications between the other agents. An experimental system of the model has been implemented in Prolog language on a Window-based personal computer. Intensive experiments have been carried out to examine the feasibility of the machine learning mechanisms of agents for problem-solving and communication capabilities. The experimental results have shown that the multiagent system improves the performance of the whole system in problem solving, when each agent has a higher learning ability or when an agent with a very high ability for problem solving joins the organization to cooperate with the other agents in problem solving. These results suggest that the proposed model is useful in analyzing the learning mechanisms applicable to distributed knowledge systems. Copyright © 1996 Elsevier Science Ltd
I. I N T R O D U C T I O N
for intermediate results through queries on the problem it is solving, and it is desirable to learn the final results for future use. This paper focuses on such situations. We address the issues of machine learning in distributed knowledge systems, which will consist of distributed software agents with problem-solving, communication and learning functions. To develop such systems, we must analyze the roles of problem-solving and communication capabilities between individual agents or knowledge systems. In this paper, we propose a computational model: LPC. In the model, we assume that (a) a set of problems is given to one of the agents, (b) any single agent cannot solve the problems, thus, the agents must communicate with each other, (c) the agents have abilities to learn from both problem-solving results and communications and (d) the model has sufficient knowledge in total to solve the given problems. The model consists of a set of agents with (a) a knowledge base for learned concepts, (b) a knowledge base for problem solving, (c) prolog-based inference mechanisms and (d) a set of beliefs on the reliability of the other agents. Each agent can improve its own problem-solving capabilities by deductive learning
RECENTLY, A GREAT DEAL OF ARGUMENTS have been devoted to the study of very large-scale knowledgebased systems (VLKBs). There are two approaches in developing VLKBs. One approach is to develop a centralized system which can be used for various objectives, as is stated in the Cyc project (Lenat & Guha, 1990). In this approach, it is vital to facilitate the reuse of large amounts of knowledge according to the changes to the context in use. The other approach is to develop many cooperative knowledge systems in distributed environments. The concepts proposed in "Knowledgeable Community" (Nishida & Takeda, 1993) and "Knowledge Sharing Effort by DARPA" (Neches et al., 1991) are categorized in this approach. Knowledge sharing, problem solving and communication are key issues for such systems. When one knowledge system cannot solve a given problem by itself, it must ask other knowledge systems
t To whom all correspondence should be addressed at: Engineering Systems Department, Mitsubishi Research Institute Inc., 2-3-60temachi, Chiyoda-ku,Toyko 100, Japan. 417
418
from the given problems, by memory-based learning from communications between the agents and by reinforcement learning from the reliability of communications between the other agents. An experimental system of the model has been implemented in Prolog language on a Window-based personal computer. Intensive experiments have been carried out to examine the feasibility of the machine learning mechanisms of agents for problem-solving and communication capabilities. This paper is organized as follows: in Section 2 the background, motivation and related work of the research are briefly described. In Section 3 we explain the features of the proposed learning model. In Section 4 experimental results are described. In Section 5 concluding remarks are made. 2. BACKGROUND, MOTIVATION AND RELATED WORK ON THE LEARNING MODEL Very large knowledge system development efforts [e.g. Cyc project (Lenat & Guha, 1990; Lenat, 1995)] do not usually discuss the problems of learning in distributed environments. However, such discussions will become one of the main themes of distributed knowledge systems. In this section, we briefly survey the state-ofthe-art researches on distributed knowledge systems in order to clarify our research motivation. There are several researches on knowledge sharing and distributed knowledge systems. The first example is knowledge sharing effort (KSE) (Neches et al., 1991). KSE presents a vision of the future in which knowledgebased system development and operation is facilitated by the infrastructure and technology of knowledge sharing. It describes an initiative currently underway to develop these ideas and suggests steps that must be taken in the future to try to realize this vision. The second example is knowledgeable community. In Nishida and Takeda (1993), they describe the scope and goals of the knowledgeable community project and discuss how sharing of knowledge is achieved in the knowledgeable community with a framework of knowledge sharing and reuse based on a multiagent architecture. In particular, they focus on the organizational structure that facilitates mediation between those agents requesting a service and those providing the service. The visions proposed in KSE and knowledgeable community are quite plausible for future distributed knowledge systems; however, we believe that a simple and executable model for such excellent future visions must be developed. The third example is PACT (Palo Alto Collaborative Testbed) (Cutkosky et al., 1993), in which the researchers developed a concurrent engineering infrastructure that encompasses multiple sites, subsystems and disciplines. Currently, PACT is one of the few examples of
H. Aiba and T. Terano
distributed knowledge systems; however, the framework proposed in PACT is too specific for general models of knowledge sharing. Again, we believe that a model which is able to generalize the characteristics of specific distributed knowledge systems must be developed. Compared with the researches for learning agent systems [for example, Weiss (1993)], our motivation is to give some insight for developing a general framework of distributed knowledge systems. As discussed in other literature (Terano, 1993), we also think the models should be used as a common framework to analyze problems not only of the future distributed knowledge systems, but also of computer supported cooperative works (CSCWs), computer integrated manufacturing (CIM), distributed information systems (Papazoglou, 1992) and organizational behaviors of social and software agents (Carley et al., 1993; Lee et al., 1993; Riecken, 1994). In these applications, adding to planning and problem-solving problems among agents in conventional distributed artificial intelligence researches (Bond & Gasser, 1988; Fox, 1981), the problems of learning capabilities of both each agent and the whole system seem to affect the knowledge sharing, problem-solving and communication facilities. If a user requires one of the knowledge systems to solve some problems, that is, we use the WWW in Internet, the system will generate o n e feasible solution within the context the user has specified. The results should be learned for the future reuse. Such learned results are fully context dependent and may differ from each other, when another knowledge system has solved similar problems in a different context, where we might abandon the concepts of completeness of the knowledge or common knowledge. This paper examines a multiagent learning model for such distributed systems and describes some intermediate experimental results. The main objective of the research is to verify and validate the feasibility of learning mechanisms to improve the problem-solving and communication capabilities of multiagents in distributed environments (Aiba & Terano, 1995; Terano, 1994). There are several researches in the literature on multiagent learning models. The architecture of the proposed model consists of some extensions of the ideas in practical systems as, for example, presented in Sugawara & Lesser (1993) and Vittal et al. (1992). The learning mechanisms equipped in such systems are considered to be limited from the problem-solving perspective. The proposed model has provided a uniform framework for such learning mechanisms. Shaw and Whinston (1989) and Tan (1993) deal with reinforcement learning in multiagent systems. The problem is how several agents can collectively learn to coordinate their actions such that they solve a given environmental task together. We think that these ideas are effective in learning by communication in multiagent learning.
Distributed KnowledgeSystemswithLearningMechanisms Although the example presented here is a very simplified one, we think that it has a potential, as described in Bradzil and Muggleton (1991) and Sian (1991). The self-organizational properties found in Ishida et al. (1992) can also be simulated in the proposed model. Furthermore, the problem-solving schemes and communication protocols found in Smith (1980), Collin et al. (1991), Bridgeland and Huhn (1990), Mason and Johnson (1989), Goergeff (1983) and Yokoo et al. (1992) may be introduced to the proposed model. There are other researches concerning the extension of EBL (Minton, 1988; Keller, 1988). DeJong and Mooney (1986) argue that the inadequacies of the framework by Mitchell and Keller (1986) arise from the treatment of the concept of operationality, organization of knowledge into schemata and learning from observation. Yamamura and Kobayashi (1991) and Shavlik (1990) have augmented the framework of EBL on plural examples and, considering relations between generalizations and operationality, have proposed a learning method to generate operational generalizations incrementally. These researches concentrate on the extension of the EBL framework within a single knowledge system. On the other hand, the concepts of learning by problem solving is another extension of EBL researches into multiple distributed environments (Cho et al., 1994). 3. A COMPUTATIONAL MODEL FOR DISTRIBUTED KNOWLEDGE SYSTEMS
The problems given to LPC consist of a set of tuples: {(G, E)}, where G is the goal of the problem and E is a set of examples common to all the agents. E is assumed to be static during the problem solving. The information on Es is used to improve the problem-solving performance in a similar manner to that found in Multistrategy learning systems (Michalski, 1993; Michaiski & Tecuci, 1994). If E is empty, the problems become the same as those in distributed problem solving. If E contains plural examples, the agents can utilize them for inductive learning and if E contains only one example, the agents can use this for deductive learning when the agents have sufficient domain knowledge. The goal G is divided into sub-goals by each agent in the following way, if necessary. The agents of LPC solve a given problem in part by using learned results and their own communication and problem-solving knowledge. If they cannot solve the goal, they decompose it into sub-goals, and then request the other agents to solve them. The requests are done based on the memory of the other agents' information. In order to improve the problem-solving performance for a set of given similar problems (1) the agents of LPC compile problem-solving knowledge both within and between agents via deductive and/or inductive learning techniques and (2) the agents also organize themselves
419 Subtask
Tusk .
Z -. o i . . ~ml
B!
^,e.t-1 ~ "~ W Commun1"cation0 V Tusk Group
Requirements
O
So],tio,s~]
||Problem Agent-3
•
SoJvl"Og
f
Agent5
Agent-2 FIGURE 1. Architecture of the LPC model.
by changing the information on which agents they should request to solve the (sub-)problems. The agents in the proposed model are considered to be the model on heterogeneous knowledge bases, because the domain knowledge, problem solvers and communication facilities can differ from each other when we implement concrete systems. 3.1. The Agents of LPC The architecture and problem solving of the LPC are shown in Fig. 1 and the characteristics of the agent are summarized in Fig. 2. The agents of the LPC are defined by the following five tuples:
Ai=(LCi, KAi, Di, PSi, c o g i )
fSub' Pn:;bl °msI I ](Sub' 'Other)fPr°bioms) ,~ents r o m/(Sub-)F' [ S°lldi°n oe~ms °fr, Lto O Iher Agenls Li° o ~ ~en~
T ,
. . . .
i
Itl from Other/~ents
1 (Sub-IProblems
~, Row of .......... Row of Knowl~e
FIGURE 2. Architecture and problem solving of an LPC agent.
420
H. Aiba and T. Terano (1) Learned concepts LC~: the agents memorize the operational knowledge (Keller, 1988) for learned concepts of problem-solving knowledge. The agents first try to use them for new problem solving. If they are not applicable, they use their communication knowledge and communication and/or problem-solving functions. (2) Knowledge on the other agents KAi: the KA i consists of a set of tuples: (~rk,,~, R), where ,~j is the name of the other agent, Tk is the name of (sub-)problem sent to it and Rj is the credit value computed from ,~ and ~rk.That is
• An agent knows the name of the other agents but does not know what kind of knowledge they have. • An agent infers what kind of knowledge the other agents have via communication for problem solving. • An agent can solve only one (sub-)problem at a time. • An agent stores the name and the reliability values of the other agents that the agent expects to solve the problem. • Each agent has a limited size of memory to store information on the other agents. 3.3. Problem Solving in LPC
Each agent can memorize a given number of tuples at any one time. (3) Domain knowledge Di: the knowledge Dis of the agents are consistent with, but different from, each other. The contents on the Dis cannot be transferred between the agents; that is, the agents do not know who has the correct problem-solving knowledge. When the inductive learning function is executed, Di is used as the bias for the learning. When general problem-solving or deductive learning functions are executed, D i is used as non-operational domain knowledge. (4) Problem solver PSi: using PS#, the agents solve the gi yen (sub-)problems. The agent generally executes PSi twice: first using only LCi and second using both LCi and Di. PSi is further equipped with some inductive learning functions. The power of PS~s can be different for each agent. (5) Communication value COMi: each agent A i tentatively records the communication values COMi of the other agents Ak. The value of COMi has one of the following forms: (send, ~rk, "~i, "~k), (succeeded, Sk, "~i, "~k), (failed, ~rk, Ai, ,~k), where Sk means the (sub-)solutions of ~rk. The value of COMi is used to update KAy.
3.2. The Characteristics of Agents The agents have the following features: • Each agent has partial knowledge on problem solving and problem decomposing. It cannot transfer its own knowledge to another.
The agents try to solve given (sub-)problems by using learned concepts LCi, knowledge on other agents KAi or inductive learning via D;, they then try to learn via communication and/or general problem solving. The problem-solving processes of the agents are summarized in the following. Steps 2, 3 and 4 and steps 5 and 6 can be executed in different orders depending on the features of the given problems. This may produce some implementation issues. (1) Problem accepting: when Ai gets a message from another agent Ak, A i tries to accept a (sub-) problem ~rk. (2) Problem solving via learned concepts: agent A i solves ~rk via PSi, if it can solve or decompose ~rk using LCI. When the results are successful, Ai returns the results. (3) General problem solving: PSi tries to solve ~rk using both LCI and Di. When the results are successful, A i returns the results and updates LCi. If it fails, the results are moved to COMi. (4) Problem solving via knowledge on other agents: when A i fails in the above step, it will notify the name of another agent, Aj, to Ak using KAi, if Rj, of KAi has a large enough reliability value, which means that At has a greater possibility of solving l"k. When the results are successful, Ai returns the results. (5) Solving new communication paths: when the results of the problem solving are successful, Ai returns the results. When the results of the problem solving are not successful, COMi opens the communication to another agent Ai. The communication trials continue until another agent accepts it. Upon acceptance, COMi updates KAi. (6) Computing reliability on other agents: after the execution of steps 3 or 5, The credit value R~ increases when agent ,~i succeeds in solving ~rk, and decreases when ,~ fails to solve 2rk. The values are computed based on the techniques
Distributed Knowledge Systems with Learning Mechanisms of profit sharing pattern type credit assignment algorithms in genetic algorithms and reinforcement learning. 4. EXPERIMENTAL RESULTS
4.1. The Framework of the Experiments The methods of the experiments are described in this subsection. The experiments aim to simulate the process that agents with problem-solving function, communication function and learning function, cooperate and solve the problem that any single agent cannot solve by itself. The experimental system is implemented in IF-Prolog on a Window-based PC. The aims of the experiments are: (1) to investigate the relationship between organizational form and communication, (2) to evaluate the organizational effects by the improvement of each agent's ability, (3) to observe the organizational effect when a high ability agent joins the organization and to analyze the role of the high ability agent, (4) to examine if the ability of the organization in solving a problem improves when Nemawashi is executed in the organization. By Nemawashi, we mean the informal activities allowing the organisation to learn some examples of real problems in advance. In these experiments, the organization consists of 4-10 agents, which have the same amount of knowledge for problem solving. In order to analyze and evaluate the effect of communication and learning in cooperative problem solving in a distributed environment, we observe the activities of agents, and evaluate communication steps, problem solving steps and the amount of knowledge that agents have learned through problemsolving processes. The problems we deal with in this research cannot be solved by any single agent, but can be solved by all agents that make up the organization. When a problem is initially given to an agent from outside the system, each agent tries to decompose or solve a (sub-)problem through communication with the other agents. The process of problem solving consists of two steps: one is to decompose a given problem into sub-problems, the other is to solve a (sub-)problem. The problem is said to be solved when the given problems are completely solved (i.e. completely removed). We represent these kinds of problems with a list replacement problem in the experimental system, as described in the next subsection. There are two forms of communication based on the knowledge of other agents: one is memory based, the other is reliability based. The memory-based method
421 selects the agent's name by the name of the task from the knowledge on other agents that is learned through the problem-solving process. The reliability-based method selects the agent's name with high reliability values of knowledge on the other agents. The differences between this research and related works are as follows: • we represent the difference of organizational form with the limitation of communication paths to analyze organizational forms, • we analyze the effect of the change of each agent's capacity to memorize other agents' knowledge, or change of domain knowledge of each agent, • we try to estimate the effect of Nemawashi through the experiment.
4.2. List Replacement Problem In the experiments, we use the list replacement problems as a testhed. Simple illustrations of the problem are shown in Figs 3 and 4. A list of items with constants or variables accompanied with or without examples is given to the model. The goal of problem solving is to make the given list empty, by decomposing, deleting or replacing the elements via knowledge among the agents. The problem is quite simple but scalable in size. The problem aims to simulate the cooperative works of multiagents in manufacturing processes, office works, query processing in distributed systems and so on. Therefore, we think the problem is just complex enough to assess the initial feasibility of the proposed model.
4.3. Experiments on Problem Solving and Communication 4.3.1. Experiment 1. We first analyze how different the quantities of communication are among the various types of organization. We assume three types of organization: network, fiat and hierarchical, each of which differs in how each agent selects the agent for communication. They are summarized as follows: (1) network type: each agent can communicate equally and freely with each other (2) flat type: one agent plays the role of coordination and controls the knowledge of the other agents (3) hierarchical type: agents form two or three levels of hierarchy and can communicate only with agents that are fixed in advance. In this experiment, the organization consists of ten agents with an equal capability of problem solving. As we give three kinds of tasks in order (i.e. task-l, task-2, task-3, task-l, task-2, task-3 . . . . ) to the organization, we
422
14. Aiba and T. Terano
f
/ subtaSk~A
abc
a,
b,
subtaSk~B
subtask~ B
sub,ask~
subtaskc
suhtaskc ~
subtaskc
subtaSkD
subtaSk~D
~
c
J ~ b ~ a sk j
FIGURE 3. Illustrative examples of the list replacement problem.
evaluate the number of steps of communication between the agents and steps of problem solving until the organization has solved the given problem. Figure 5 shows the results of the experiments. Though experimental results do not seem to be stable in most of the trials, we can observe that the communication steps in problem solving in the network type organization tend to be stable in comparison with the other types of organization. The hierarchical organization sometimes (1) List Replacement Problem e l t h o u t Examples I n i t i a l Problem: ( [ a , cd, bcd], ()) Goal: ( D , ()) - - Problem s o l v i n g by Agent-l: Knoelodge: a --> cd Result: ([cd, cd, b c d ] , ( ) ) ~ Divided i n t o a Sub problem - - Problem s o l v i n g by Agent-2: Knowledge: cd --> • Result: ( [ o , e, be], ()) ~ Aggregate Sub problems - - Problem s o l v i n g by Agent-3:
Xnogledge:
b
--> n i l
Result: ([o, e, e l , ())
~ Partial ProblemSolved - - Problem s o l v i n g by Agent-4: Knowledge: • --> n i l Result: ( [ ] , ()) ~
A l l t h e Problem Solved
shows very inefficient communication processes because of the limitations of the communication paths in the hierarchical method. 4.3.2. Experiment 2. In the next experiment, we analyze
the influence of the learning capability of each agent on the efficiency of organizational problem-solving behavior. We use a network type of organization, which consists of ten agents with equal capability of learning (2) List Replacement Problem with P l u r a l Exalples I n i t i a l Problem: ( I s , cd, bcd], ( ( [ a , cd, bcd] -> [a, d, bcd]) ( [ a , cd, bcd] -> [a, cd, bd]) ( [ a , cd, bcd] -> [a, d, bd]))) - - Problem solving by Agent-l: Xnoeledge: a --> cd Result: ([cd, cd, b c d ] , ( ) ) ([cd cd, bcd], ( ( [ a , cd, bcd] => [a, d, bcd]) ([a, cd, bcd] -> [a, cd, bd]) ([a, cd, bcd] -> [a, d, bd]))) ~]{ Divided into a Sub problem - - Problem sol'ring by Agent-S: Knowledge: Result: c --> n i l ~ Inductive Learning ([d, d, bd], (([cd, cd, bcd] -> [a, d, bcd]) ([a, ¢d, bcd] -> [a, cd, bd]) ([a, cd, bcd] -> [a, d, bd]))) ]{]{~ P a r t i a l Problem Solved - - Problem solving by Agent-3: Xnoeledge: b --> n i l ([d, d, d], ( ( [ a , cd, bed] -> [a, d, bcd]) ( [ a , ' c d , bcd] -> [a, cd, bd]) ( [ a , cd, bcd] -> [a, d, b d ] ) ) )
FIGURE 4. Executions of the list replacement problem.
Distributed Knowledge Systems with Learning Mechanisms
423
100 90 80 70 =
60
-~
50
0 ,,-4 0
•~
40
0
30 20
o
10 t
0
l
l
l
l
l
l
l
l
l
l
l
l
[
I
I
l
I
I
I
I
I
I
t
i
t
trials *
network t y p e - - o - - f l a t
type
a
h i e r a r c h i c a l type
FIGURE 5. Problem solving in different organizations.
and problem solving. In order to compare the learning capability of agents, we first let the agents have a large capability (i.e. each agent can memorize, at most, 100 pieces of knowledge on other agents' problem solving), and we compare them to those with a small capability (i.e. each agent can memorize at most only 10 pieces of knowledge on other agents' problem solving). We give the same task continuously to the organization and observe the amount of communication in organizational problem solving. Figure 6 shows the experimental results. When agents have a high capability of learning, in other words, when agents have a large memory for the knowledge of other agents, the amount of communication decreases and they can solve the problem more efficiently.
knowledge of each agent. The process of the experiment is as follows: (1) Let the organization of three agents with a small amount of domain knowledge solve three types of problems 30 times. (2) Add an agent with a small amount of expert knowledge on decomposing problems and let the whole organization solve one problem 30 times to make a communication network. (3) Add an expert agent with a high-level knowledge to solve a complicated problem by itself. Let the organization of five agents solve one problem and observe its behavior. Figures 7 and 8 show the number of problem-solving steps and communication steps in the system. The horizontal axis shows the number of trials in the experiment, trials between 121 and 150 are the behaviors
4.3.3. Experiment 3. The third experiment aims to the investigate the influence of the difference of the 80
High__LOW
70 60 50 40 t~
30 20 10
\ l / n
/
•
•
•
•
I
I
I
I
I
I
i
I
I
I
1
I
I
J
I
I
I
I
i
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
trial FIGURE 6. Effect of leaming capability in memory-based leamlng in problem solving.
424
H. Aiba and T. Terano
15
""'""'""'""'""'""'I"
li-i-i-i-i-I-i-i-n-u-
~
l-i-i-l-l-i-i-i-i-i-i-
i-l-i-l-i-i-l-U-I
l-i-i-l-U-l-i-i-N-N-U-l-i-i-i-i-l-i-i-N-l-l-l-i-i-i-i-n-m-N
5
: ...... . . . . . . . . . . . . . . . . . . . . . . . . .
121
126
131
,
136
141
I
,
146
,
I
,
,
I
151 trials
I
I
P I
156
I
. .........
.............. 1 F I
I
I
161
I
I
i
I
I
166
I
I
I
I
I
I
171
J
I
I
I
I
I
176
- . - - Problem Solving ---D-Communication - - - Problem Solving by Expert ---o-Communication with Expert FIGURE 7. Problem $oMngpe~ormance ~a memory-basedlearnlngwhen an expertagentisavailable.
of an organization of three agents with a small amount of knowledge and an agent with a small amount of expert knowledge; and the trials between 151 and 180 are the behaviors of an organization with an expert agent with high-level knowledge. From the interval between 121 and 150 trials in Fig. 7, we can observe that the expert agent with low-level knowledge is playing an important role in the communication but its contribution to problem-solving is not much. In contrast, the expert agent with high-level knowledge makes a large contribution both in communicating and problem solving between 151 and 180 trials in Fig. 7. In a memory-based learning system, the joining of an expert does not necessarily decrease the total number of problem-solving steps. Figure 8 shows the average steps per the 30 trials of the experiment of the same problem solving and communication by the expert agent and the whole system. Figures 9 and 10 show the results of a similar experiment on the system with reliability-based learning. From Fig. 9, we can observe that participation of the expert agent with high-level knowledge improves problem-solving efficiency as a whole system of reliability-based learning. Both problem-solving steps
2025f
and communication steps decrease as the expert joined the organization. This means the expert agent contributes to improve the efficiency as a whole in problem solving. From Fig. 10, we can observe that an expert agent with high-level knowledge plays an important role. 4.3.4. Experiment 4. In human society, especially in
Japanese industries, information or indication of a wish to solve a problem are often informally spread and/or broadcast to related members before formally asking for problem solving in order to process smooth problem solving. In Japan, this is called Nemawashi, which is characterized by the prior informal communications between members of an organization before real problem solving. The fourth experiment aims to simulate Nemawashi behaviors and estimate its effects. In this experiment, we define Nemawashi as allowing agents to learn the knowledge on other agents by solving given examples. We can observe the effect of prior learning on examples that resemble the real problem. The organization consists of ten agents with the same level domain knowledge. We allow agents to solve the same example 30 times in advance of real problems in one case, and we do not let them solve any examples before the real problems in another case. We compare the
• problemsolving . . . [] .problemsolvingbyexpert .
~7~
~ 10 5 0
a/b/c/d/e/ f
cc
bb
aa
abc
FIGURE 8. The role of the expert agent in memory-based learning.
abc
Distributed Knowledge Systems with Learning Mechanisms
425
30 25
-~ 2O o 15
q-t
,$?
.~ 10
(
E
=
"m'm'm'a
m'm'm'm''
= ' m ' s - m ' "/
5
121
126
131
136
141
146
151 trials
156
161
166
171
176
Problem Solving +Communication - - - P r o b l e m Solving by Expert + C o m m u n i c a t i o n with Expert - - -
FIGURE 9. Problem solvingpe~ormancevlarella~lity-basedlearnlng whenanexpert agentlsavailable.
as a subproblem (2) a non-complicated problem that has no relationship to the example (3) a problem exactly the same as the example.
processes and results of both cases. In each case, we give three types of real problems five times in order, and observe the number of steps of problem solving and communication till the whole problem is solved. Types of real problems are as follows:
Figure 11 shows the results of the experiment. The horizontal axis shows the number of units of five trials of solving the same problem; there are twelve units of five trials in one experiment. The vertical axis shows the
(1) a complicated problem including an example
25 F 20 I
• problem solving • problem solving by expert [] communication [] communicationwith expert
~" 15 ~ lO 5 o a/b/c/d/e/
aa
bb
cc
f
abc
abc
FIGURE 10. The role of the expert agent in reliability-based leaming.
[] problemsolving [] problem solving (Nemawashi) • communication [] communication(Nemawashi)
45 r-" 40 ~ a
~ 30 .~ 25 ~ 2o
0
i i/-.-ii
1
2
3
4
5
6
7
8
9
10
units of trials FIGURE 11. Effects of
Nemawashi
activities in memory-based learning.
11
12
426
H. Aiba and T. Terano
average steps of problem solving and communicating in a unit of trials. We observe the tendency that the Nemawashi case, with prior learning, is more efficient in communication than the other case in the memory-based learning system. This result is natural because the quantity of inefficient communication decreases by prior learning example. The trial units 2, 5, 8 and 11 do not show the effect of Nemawashi. It is implied that this is because the given real problem of these units has little relationship to the example problem. 5. CONCLUDING REMARKS This paper has proposed a machine learning model from problem solving and communication for a distributed knowledge system. Our framework is appficable to the next generation information systems, which will involve distributed information agents which work in a synergistic manner (cooperatively) by exchanging information and expertise, coordinating their activities and negotiating how to solve parts of a common knowledge-intensive problem. We have carried out the simulation of problemsolving behavior based on the proposed model. The ability of problem solving improves when the agent has a large amount of memory in a memory-based learning system. By limitation of the communication paths, we can simulate various kinds of organizational structures. As we can treat different kinds of conditions on the communication paths and contents of domain knowledge of each agent, we cannot suggest which communication path is generally efficient. In selecting the agent to communicate, the cost of each communication path should be examined in a further study. Through experiment, we observe that agents form a communication network by repeating problem solving. As a new high-ability agent joins the agent system, the other agents communicate with the new agent frequently, and the efficiency of problem solving is improved. We cannot identify what kind of knowledge the agent should have in order to solve a problem efficiently in the system. At last, we have observed the effect of Nemasashi from the simulation that prior learning improves total effectiveness in problem solving. The multiagent learning model proposed here is also considered to be an extension of the knowledge acquisition tools (Boose, 1988) and machine learning techniques (Michalski, 1993) for distributed environments. The model can be extended such that the domain knowledge which each agent has (i.e. the capabilities for problem solving) will increase as time passes. Further, it is natural from our discussions that the domain knowledge is partially exchangeable between the agents. Acknowledgements--This paper is a revised version of a paper presented at both the Int. Syrup. on Fifth Generation Computer Systems
1994 (FGCS'94) Workshop on Heterogeneous Cooperative Knowledge-Bases (W3), 1994 (Terano, 1994) and the Pacific-Asian Conference on Expert Systems (PACES'95), 1995 (Aiba & Terano, 1995). The authors would like to thank the anonymous referees for their comments on earlier manuscripts. Thanks are also given to Professor Jae Kyu Lee for his recommendation. This research is supported in part by a Grant-in-Aid for Scientific Research (A06402060, Priority Areas on Emergent Systems) of the Ministry of Education, Science, Sports and Culture of Japan.
REFERENCES Aiba, H. & Terano, T. (1995). Learning mechanisms for distributed knowledge systems. In Proc. Pacific-Asian Conference on Expert Systems (PACES'95), pp. 42-49. Bond, A. H. & Gasser, L. (1988). An analysis of problems and research in DAI. Readings in distributed artificial intelligence, pp. 3-36. CA: Kaufmann. Boose, J. H. & Gaines, B. R. (Eds) (1988). Knowledge acquisition tools for expert systems. New York: Academic Press. Bradzil, P. & Muggleton, S. (1991). Learning m relate terms in a multiple agent environment. In Proc. European Working Session on Learning '91, pp. 424-439. Bridgeland, D. M. & Huhns, M. N. (1990). Distributed truth maintenance. In Proceedings 9th National Conference on Artificial Intelligence (AAAI-90), pp. 72-77, Carley, K., Park, D. & Priemla, M. (1993). Agent honesty, cooperation and benevolence in an artificial organization, AI and theories of groups and organizations: conceptual and empirical research. Working Notes: AAAI-93 Workshop on AI and Theories of Groups and Organizations, pp. 1-7. Cho, D., Kusunoki, F., Ono, S. & Terano, T. (1994). Developing a multi-agent model for distributed knowledge systems. In Proc. IEEE 2nd International Conference on Expert Systems for Development, pp. 49-54. CoUin, Z., Dechter, R. & Kaiz, S. (1991). On the feasibility of distributed constraint satisfaction. In Proc. 12th International Joint Conference on Artifwial Intelligence (IJCAI-9] ), pp. 318-324. Cutkosky, M. R., Engelmore, R. S., Fikes, R. E., Genesereth, M. R., Gruber, T. R. & Mark, W. S. (1993). PACT: an experiment in integrating concurrent software engineering. IEEE Computer, 26(1), 28-37. De Jong, G. F. & Mooney, R. J. (1986). Explanation-based generalizatinn: an alternative view. Machine Learning, 1(2), 145-176. Fox, M. (1981). An organizational view of distributed systems. IEEE Transaction on Systems, Man, and Cybernetics, 11(1), 70-80. Georgeff, M. P. (1983). Communication and interaction in multi-agent planning. In Proceedings of the 2nd National Conference on Artificial Intelligence (AAAI-83), pp. 125-129. Ishida, T., Gasser, L. & Yokoo, M. (1992). Organization self-design of distributed production systems. IEEE Transactions on Knowledge and Data Engineering, 4(2), 123-134. Keller, R. M. (1988). Defining operationality for explanation-based learning. Artificial Intelligence, 35(2), 227-241. Lee, K. C., Mansfield Jr, W. H. & Sheth, A. P. (1993). A framework for controlling cooperative agents. 1EEE Computer, 26(7), 8-16. Lenat, D. B. & Guha, R. V. (1990). Building large knowledge-based systems. Reading, MA: Addison-Wesley. Lenat, D. B. (1995). CYC: a large-scale investment in knowledge infrastructure. Communication oft he ACM, 38(11), 33-38. Mason, C. L. & Johnson, R. R. (1989). DATMS: a framework for distributed assumption based reasoning. In L. Gasser & M. N. Hubris (Eds), Distributed artificial intelligence, Vol. II, pp. 293318. CA: Kaufmann. Miehalski, R. S. (1993). Toward a unified theory of learning: multi strategy task-adaptive learning. In B. G. Buchanan & D. C. Wilkins (Eds), Readings in knowledge acquisition and learning, pp. 7-38. CA: Kanfmann.
Distributed Knowledge Systems with Learning Mechanisms Michalski, R. & Tecuci, G. (Eds) (1994). Machine learning IE: a multistrategy approach. CA: Kaufmann. Minton, S. N. (1988). Learning search control knowledge: an explanation-based approach. Boston, MA: Kluwer. Mitchell, T. M., Keller, R. M. & Kedar-Cabelli, S. (1986). Explanationbased generalization: a unifying view. Machine Learning, 1(1), 47-80. Neches, R. et al. (1991). Enabling technology for knowledge sharing. AI Magazine, 12(3), 36-56. Nishida, T. & Takeda, H. (1993). Towards the knowledgeable community. In Proc. KB&KS'93, pp. 157-166. Papazoglou, M. K., Laufmann, S. C. & Sellis, T. K. (1992). An organizational framework for cooperating intelligent information systems. International Journal of Intelligent and Cooperative Information Systems, 1(1), 169-202. Riecken, D. (Ed.) (1994). Special Issue: Intelligent agents. Communications of the ACM, 37(7), 18-147. Shavlik, J. W. (1990). Extending explanation-based learning by generalizing the structure of explanations. CA: Kaufmann (Research Notes in Artificial Intelligence Series). Shaw, M. J. & Whinston, A. B. (1989). Learning and adaptation in distributed artificial intelligence systems. In L. Gasser & M. L. Huhns (Eds), Distributed artificial intelligence H; pp. 413--430. CA: Kauffman. Sian, S. S. (1991). Adaptation based on cooperative learning in multiagent systems. In Y. Demazeau & J.-P. Miiller (Eds), Decentralized artificial intelligence, Vol. 2, pp. 257-272. Smith, R. G. (1980). The contract net protocol: high-level communication and control in a distributed problem solver. IEEE Transaction
427 on Computers, 29(12), 1104-1113. Sugawara, T & Lesser, V. (1993). On-line learning of coordination plans. In Proc. 12th International Workshop on Distributed Artificial Intelligence. Tan, M. (1993). Multi-agent reinforcement learning: independent vs cooperative agents. In Proc. lOth Machine Learning Workshop, pp. 330-337. Terano, T. (1993). Requirements of AI models applicable to organizational learning theory and two related examples. In Working Notes of AAAI-93 Workshop on AI and Theories of Groups and Organizations, pp. 92-95. Terano, T. (1994). Learning from problem solving and communication: a computational model for distributed knowledge systems. In Proc. FGCS'94, Workshop on Heterogeneous Cooperative Knowledge Bases, p. 153. Vittal, J., Silver, B., Frawley, W., lba, G., Fawcett, T., Dusseault, S. & Doleac, J. (1992). Intelligent and cooperative information systems meet machine learning. International Journal of Intelligent and Cooperative Information Systems, 1(2), 347-361. Weiss, G. (1993). Learning to coordinate actions in multi-agent systems. In 13th International Joint Conference on Artificial Intelligence (IJCAI-93), pp. 311-316. Yamamura, M. & Kobayashi, S. (1991), An augmented EBL and its application to the utility problem. In 12th International Joint Conference on Artificial Intelligence (IJCAI-91), pp. 623-629. Yokoo, M., Durfee, E. H., Ishida, T. & Kuwabara, K. (1992). Distributed constraint satisfaction for formalizing distributed problem solving. In 12th International Conference on Distributed Computing Systems (1CDC S-92), pp. 614-4521.
Pergamon
S0957-4174(96)00020-6
A Computational Model for Distributed Knowledge Systems with Learning Mechanisms HIROKO A I B A t AND TAKAO TERANO Graduate School of SystemsManagement,The Universityof Tsukuba, 3-29-10tsuka, Bunkyo-ku,Tokyo 112, Japan
Abstract--This paper addresses the issues of machine learning in distributed knowledge systems, which will consist of distributed software agents with problem solving, communication and learning functions. To develop such systems, we must analyze the roles of problem-solving and communication capabilities among knowledge systems. To facilitate the analyses, we propose a computational model: LPC. The model consists of a set of agents with (a) a knowledge base for learned concepts, (b) a knowledge base for problem solving, (c) prolog-based inference mechanisms and (d) a set of beliefs on the reliability of the other agents. Each agent can improve its own problem-solving capabilities by deductive learning from the given problems, by memory-based learning from communications between the agents and by reinforcement learning from the reliability of communications between the other agents. An experimental system of the model has been implemented in Prolog language on a Window-based personal computer. Intensive experiments have been carried out to examine the feasibility of the machine learning mechanisms of agents for problem-solving and communication capabilities. The experimental results have shown that the multiagent system improves the performance of the whole system in problem solving, when each agent has a higher learning ability or when an agent with a very high ability for problem solving joins the organization to cooperate with the other agents in problem solving. These results suggest that the proposed model is useful in analyzing the learning mechanisms applicable to distributed knowledge systems. Copyright © 1996 Elsevier Science Ltd
I. I N T R O D U C T I O N
for intermediate results through queries on the problem it is solving, and it is desirable to learn the final results for future use. This paper focuses on such situations. We address the issues of machine learning in distributed knowledge systems, which will consist of distributed software agents with problem-solving, communication and learning functions. To develop such systems, we must analyze the roles of problem-solving and communication capabilities between individual agents or knowledge systems. In this paper, we propose a computational model: LPC. In the model, we assume that (a) a set of problems is given to one of the agents, (b) any single agent cannot solve the problems, thus, the agents must communicate with each other, (c) the agents have abilities to learn from both problem-solving results and communications and (d) the model has sufficient knowledge in total to solve the given problems. The model consists of a set of agents with (a) a knowledge base for learned concepts, (b) a knowledge base for problem solving, (c) prolog-based inference mechanisms and (d) a set of beliefs on the reliability of the other agents. Each agent can improve its own problem-solving capabilities by deductive learning
RECENTLY, A GREAT DEAL OF ARGUMENTS have been devoted to the study of very large-scale knowledgebased systems (VLKBs). There are two approaches in developing VLKBs. One approach is to develop a centralized system which can be used for various objectives, as is stated in the Cyc project (Lenat & Guha, 1990). In this approach, it is vital to facilitate the reuse of large amounts of knowledge according to the changes to the context in use. The other approach is to develop many cooperative knowledge systems in distributed environments. The concepts proposed in "Knowledgeable Community" (Nishida & Takeda, 1993) and "Knowledge Sharing Effort by DARPA" (Neches et al., 1991) are categorized in this approach. Knowledge sharing, problem solving and communication are key issues for such systems. When one knowledge system cannot solve a given problem by itself, it must ask other knowledge systems
t To whom all correspondence should be addressed at: Engineering Systems Department, Mitsubishi Research Institute Inc., 2-3-60temachi, Chiyoda-ku,Toyko 100, Japan. 417
418
from the given problems, by memory-based learning from communications between the agents and by reinforcement learning from the reliability of communications between the other agents. An experimental system of the model has been implemented in Prolog language on a Window-based personal computer. Intensive experiments have been carried out to examine the feasibility of the machine learning mechanisms of agents for problem-solving and communication capabilities. This paper is organized as follows: in Section 2 the background, motivation and related work of the research are briefly described. In Section 3 we explain the features of the proposed learning model. In Section 4 experimental results are described. In Section 5 concluding remarks are made. 2. BACKGROUND, MOTIVATION AND RELATED WORK ON THE LEARNING MODEL Very large knowledge system development efforts [e.g. Cyc project (Lenat & Guha, 1990; Lenat, 1995)] do not usually discuss the problems of learning in distributed environments. However, such discussions will become one of the main themes of distributed knowledge systems. In this section, we briefly survey the state-ofthe-art researches on distributed knowledge systems in order to clarify our research motivation. There are several researches on knowledge sharing and distributed knowledge systems. The first example is knowledge sharing effort (KSE) (Neches et al., 1991). KSE presents a vision of the future in which knowledgebased system development and operation is facilitated by the infrastructure and technology of knowledge sharing. It describes an initiative currently underway to develop these ideas and suggests steps that must be taken in the future to try to realize this vision. The second example is knowledgeable community. In Nishida and Takeda (1993), they describe the scope and goals of the knowledgeable community project and discuss how sharing of knowledge is achieved in the knowledgeable community with a framework of knowledge sharing and reuse based on a multiagent architecture. In particular, they focus on the organizational structure that facilitates mediation between those agents requesting a service and those providing the service. The visions proposed in KSE and knowledgeable community are quite plausible for future distributed knowledge systems; however, we believe that a simple and executable model for such excellent future visions must be developed. The third example is PACT (Palo Alto Collaborative Testbed) (Cutkosky et al., 1993), in which the researchers developed a concurrent engineering infrastructure that encompasses multiple sites, subsystems and disciplines. Currently, PACT is one of the few examples of
H. Aiba and T. Terano
distributed knowledge systems; however, the framework proposed in PACT is too specific for general models of knowledge sharing. Again, we believe that a model which is able to generalize the characteristics of specific distributed knowledge systems must be developed. Compared with the researches for learning agent systems [for example, Weiss (1993)], our motivation is to give some insight for developing a general framework of distributed knowledge systems. As discussed in other literature (Terano, 1993), we also think the models should be used as a common framework to analyze problems not only of the future distributed knowledge systems, but also of computer supported cooperative works (CSCWs), computer integrated manufacturing (CIM), distributed information systems (Papazoglou, 1992) and organizational behaviors of social and software agents (Carley et al., 1993; Lee et al., 1993; Riecken, 1994). In these applications, adding to planning and problem-solving problems among agents in conventional distributed artificial intelligence researches (Bond & Gasser, 1988; Fox, 1981), the problems of learning capabilities of both each agent and the whole system seem to affect the knowledge sharing, problem-solving and communication facilities. If a user requires one of the knowledge systems to solve some problems, that is, we use the WWW in Internet, the system will generate o n e feasible solution within the context the user has specified. The results should be learned for the future reuse. Such learned results are fully context dependent and may differ from each other, when another knowledge system has solved similar problems in a different context, where we might abandon the concepts of completeness of the knowledge or common knowledge. This paper examines a multiagent learning model for such distributed systems and describes some intermediate experimental results. The main objective of the research is to verify and validate the feasibility of learning mechanisms to improve the problem-solving and communication capabilities of multiagents in distributed environments (Aiba & Terano, 1995; Terano, 1994). There are several researches in the literature on multiagent learning models. The architecture of the proposed model consists of some extensions of the ideas in practical systems as, for example, presented in Sugawara & Lesser (1993) and Vittal et al. (1992). The learning mechanisms equipped in such systems are considered to be limited from the problem-solving perspective. The proposed model has provided a uniform framework for such learning mechanisms. Shaw and Whinston (1989) and Tan (1993) deal with reinforcement learning in multiagent systems. The problem is how several agents can collectively learn to coordinate their actions such that they solve a given environmental task together. We think that these ideas are effective in learning by communication in multiagent learning.
Distributed KnowledgeSystemswithLearningMechanisms Although the example presented here is a very simplified one, we think that it has a potential, as described in Bradzil and Muggleton (1991) and Sian (1991). The self-organizational properties found in Ishida et al. (1992) can also be simulated in the proposed model. Furthermore, the problem-solving schemes and communication protocols found in Smith (1980), Collin et al. (1991), Bridgeland and Huhn (1990), Mason and Johnson (1989), Goergeff (1983) and Yokoo et al. (1992) may be introduced to the proposed model. There are other researches concerning the extension of EBL (Minton, 1988; Keller, 1988). DeJong and Mooney (1986) argue that the inadequacies of the framework by Mitchell and Keller (1986) arise from the treatment of the concept of operationality, organization of knowledge into schemata and learning from observation. Yamamura and Kobayashi (1991) and Shavlik (1990) have augmented the framework of EBL on plural examples and, considering relations between generalizations and operationality, have proposed a learning method to generate operational generalizations incrementally. These researches concentrate on the extension of the EBL framework within a single knowledge system. On the other hand, the concepts of learning by problem solving is another extension of EBL researches into multiple distributed environments (Cho et al., 1994). 3. A COMPUTATIONAL MODEL FOR DISTRIBUTED KNOWLEDGE SYSTEMS
The problems given to LPC consist of a set of tuples: {(G, E)}, where G is the goal of the problem and E is a set of examples common to all the agents. E is assumed to be static during the problem solving. The information on Es is used to improve the problem-solving performance in a similar manner to that found in Multistrategy learning systems (Michalski, 1993; Michaiski & Tecuci, 1994). If E is empty, the problems become the same as those in distributed problem solving. If E contains plural examples, the agents can utilize them for inductive learning and if E contains only one example, the agents can use this for deductive learning when the agents have sufficient domain knowledge. The goal G is divided into sub-goals by each agent in the following way, if necessary. The agents of LPC solve a given problem in part by using learned results and their own communication and problem-solving knowledge. If they cannot solve the goal, they decompose it into sub-goals, and then request the other agents to solve them. The requests are done based on the memory of the other agents' information. In order to improve the problem-solving performance for a set of given similar problems (1) the agents of LPC compile problem-solving knowledge both within and between agents via deductive and/or inductive learning techniques and (2) the agents also organize themselves
419 Subtask
Tusk .
Z -. o i . . ~ml
B!
^,e.t-1 ~ "~ W Commun1"cation0 V Tusk Group
Requirements
O
So],tio,s~]
||Problem Agent-3
•
SoJvl"Og
f
Agent5
Agent-2 FIGURE 1. Architecture of the LPC model.
by changing the information on which agents they should request to solve the (sub-)problems. The agents in the proposed model are considered to be the model on heterogeneous knowledge bases, because the domain knowledge, problem solvers and communication facilities can differ from each other when we implement concrete systems. 3.1. The Agents of LPC The architecture and problem solving of the LPC are shown in Fig. 1 and the characteristics of the agent are summarized in Fig. 2. The agents of the LPC are defined by the following five tuples:
Ai=(LCi, KAi, Di, PSi, c o g i )
fSub' Pn:;bl °msI I ](Sub' 'Other)fPr°bioms) ,~ents r o m/(Sub-)F' [ S°lldi°n oe~ms °fr, Lto O Iher Agenls Li° o ~ ~en~
T ,
. . . .
i
Itl from Other/~ents
1 (Sub-IProblems
~, Row of .......... Row of Knowl~e
FIGURE 2. Architecture and problem solving of an LPC agent.
420
H. Aiba and T. Terano (1) Learned concepts LC~: the agents memorize the operational knowledge (Keller, 1988) for learned concepts of problem-solving knowledge. The agents first try to use them for new problem solving. If they are not applicable, they use their communication knowledge and communication and/or problem-solving functions. (2) Knowledge on the other agents KAi: the KA i consists of a set of tuples: (~rk,,~, R), where ,~j is the name of the other agent, Tk is the name of (sub-)problem sent to it and Rj is the credit value computed from ,~ and ~rk.That is
• An agent knows the name of the other agents but does not know what kind of knowledge they have. • An agent infers what kind of knowledge the other agents have via communication for problem solving. • An agent can solve only one (sub-)problem at a time. • An agent stores the name and the reliability values of the other agents that the agent expects to solve the problem. • Each agent has a limited size of memory to store information on the other agents. 3.3. Problem Solving in LPC
Each agent can memorize a given number of tuples at any one time. (3) Domain knowledge Di: the knowledge Dis of the agents are consistent with, but different from, each other. The contents on the Dis cannot be transferred between the agents; that is, the agents do not know who has the correct problem-solving knowledge. When the inductive learning function is executed, Di is used as the bias for the learning. When general problem-solving or deductive learning functions are executed, D i is used as non-operational domain knowledge. (4) Problem solver PSi: using PS#, the agents solve the gi yen (sub-)problems. The agent generally executes PSi twice: first using only LCi and second using both LCi and Di. PSi is further equipped with some inductive learning functions. The power of PS~s can be different for each agent. (5) Communication value COMi: each agent A i tentatively records the communication values COMi of the other agents Ak. The value of COMi has one of the following forms: (send, ~rk, "~i, "~k), (succeeded, Sk, "~i, "~k), (failed, ~rk, Ai, ,~k), where Sk means the (sub-)solutions of ~rk. The value of COMi is used to update KAy.
3.2. The Characteristics of Agents The agents have the following features: • Each agent has partial knowledge on problem solving and problem decomposing. It cannot transfer its own knowledge to another.
The agents try to solve given (sub-)problems by using learned concepts LCi, knowledge on other agents KAi or inductive learning via D;, they then try to learn via communication and/or general problem solving. The problem-solving processes of the agents are summarized in the following. Steps 2, 3 and 4 and steps 5 and 6 can be executed in different orders depending on the features of the given problems. This may produce some implementation issues. (1) Problem accepting: when Ai gets a message from another agent Ak, A i tries to accept a (sub-) problem ~rk. (2) Problem solving via learned concepts: agent A i solves ~rk via PSi, if it can solve or decompose ~rk using LCI. When the results are successful, Ai returns the results. (3) General problem solving: PSi tries to solve ~rk using both LCI and Di. When the results are successful, A i returns the results and updates LCi. If it fails, the results are moved to COMi. (4) Problem solving via knowledge on other agents: when A i fails in the above step, it will notify the name of another agent, Aj, to Ak using KAi, if Rj, of KAi has a large enough reliability value, which means that At has a greater possibility of solving l"k. When the results are successful, Ai returns the results. (5) Solving new communication paths: when the results of the problem solving are successful, Ai returns the results. When the results of the problem solving are not successful, COMi opens the communication to another agent Ai. The communication trials continue until another agent accepts it. Upon acceptance, COMi updates KAi. (6) Computing reliability on other agents: after the execution of steps 3 or 5, The credit value R~ increases when agent ,~i succeeds in solving ~rk, and decreases when ,~ fails to solve 2rk. The values are computed based on the techniques
Distributed Knowledge Systems with Learning Mechanisms of profit sharing pattern type credit assignment algorithms in genetic algorithms and reinforcement learning. 4. EXPERIMENTAL RESULTS
4.1. The Framework of the Experiments The methods of the experiments are described in this subsection. The experiments aim to simulate the process that agents with problem-solving function, communication function and learning function, cooperate and solve the problem that any single agent cannot solve by itself. The experimental system is implemented in IF-Prolog on a Window-based PC. The aims of the experiments are: (1) to investigate the relationship between organizational form and communication, (2) to evaluate the organizational effects by the improvement of each agent's ability, (3) to observe the organizational effect when a high ability agent joins the organization and to analyze the role of the high ability agent, (4) to examine if the ability of the organization in solving a problem improves when Nemawashi is executed in the organization. By Nemawashi, we mean the informal activities allowing the organisation to learn some examples of real problems in advance. In these experiments, the organization consists of 4-10 agents, which have the same amount of knowledge for problem solving. In order to analyze and evaluate the effect of communication and learning in cooperative problem solving in a distributed environment, we observe the activities of agents, and evaluate communication steps, problem solving steps and the amount of knowledge that agents have learned through problemsolving processes. The problems we deal with in this research cannot be solved by any single agent, but can be solved by all agents that make up the organization. When a problem is initially given to an agent from outside the system, each agent tries to decompose or solve a (sub-)problem through communication with the other agents. The process of problem solving consists of two steps: one is to decompose a given problem into sub-problems, the other is to solve a (sub-)problem. The problem is said to be solved when the given problems are completely solved (i.e. completely removed). We represent these kinds of problems with a list replacement problem in the experimental system, as described in the next subsection. There are two forms of communication based on the knowledge of other agents: one is memory based, the other is reliability based. The memory-based method
421 selects the agent's name by the name of the task from the knowledge on other agents that is learned through the problem-solving process. The reliability-based method selects the agent's name with high reliability values of knowledge on the other agents. The differences between this research and related works are as follows: • we represent the difference of organizational form with the limitation of communication paths to analyze organizational forms, • we analyze the effect of the change of each agent's capacity to memorize other agents' knowledge, or change of domain knowledge of each agent, • we try to estimate the effect of Nemawashi through the experiment.
4.2. List Replacement Problem In the experiments, we use the list replacement problems as a testhed. Simple illustrations of the problem are shown in Figs 3 and 4. A list of items with constants or variables accompanied with or without examples is given to the model. The goal of problem solving is to make the given list empty, by decomposing, deleting or replacing the elements via knowledge among the agents. The problem is quite simple but scalable in size. The problem aims to simulate the cooperative works of multiagents in manufacturing processes, office works, query processing in distributed systems and so on. Therefore, we think the problem is just complex enough to assess the initial feasibility of the proposed model.
4.3. Experiments on Problem Solving and Communication 4.3.1. Experiment 1. We first analyze how different the quantities of communication are among the various types of organization. We assume three types of organization: network, fiat and hierarchical, each of which differs in how each agent selects the agent for communication. They are summarized as follows: (1) network type: each agent can communicate equally and freely with each other (2) flat type: one agent plays the role of coordination and controls the knowledge of the other agents (3) hierarchical type: agents form two or three levels of hierarchy and can communicate only with agents that are fixed in advance. In this experiment, the organization consists of ten agents with an equal capability of problem solving. As we give three kinds of tasks in order (i.e. task-l, task-2, task-3, task-l, task-2, task-3 . . . . ) to the organization, we
422
14. Aiba and T. Terano
f
/ subtaSk~A
abc
a,
b,
subtaSk~B
subtask~ B
sub,ask~
subtaskc
suhtaskc ~
subtaskc
subtaSkD
subtaSk~D
~
c
J ~ b ~ a sk j
FIGURE 3. Illustrative examples of the list replacement problem.
evaluate the number of steps of communication between the agents and steps of problem solving until the organization has solved the given problem. Figure 5 shows the results of the experiments. Though experimental results do not seem to be stable in most of the trials, we can observe that the communication steps in problem solving in the network type organization tend to be stable in comparison with the other types of organization. The hierarchical organization sometimes (1) List Replacement Problem e l t h o u t Examples I n i t i a l Problem: ( [ a , cd, bcd], ()) Goal: ( D , ()) - - Problem s o l v i n g by Agent-l: Knoelodge: a --> cd Result: ([cd, cd, b c d ] , ( ) ) ~ Divided i n t o a Sub problem - - Problem s o l v i n g by Agent-2: Knowledge: cd --> • Result: ( [ o , e, be], ()) ~ Aggregate Sub problems - - Problem s o l v i n g by Agent-3:
Xnogledge:
b
--> n i l
Result: ([o, e, e l , ())
~ Partial ProblemSolved - - Problem s o l v i n g by Agent-4: Knowledge: • --> n i l Result: ( [ ] , ()) ~
A l l t h e Problem Solved
shows very inefficient communication processes because of the limitations of the communication paths in the hierarchical method. 4.3.2. Experiment 2. In the next experiment, we analyze
the influence of the learning capability of each agent on the efficiency of organizational problem-solving behavior. We use a network type of organization, which consists of ten agents with equal capability of learning (2) List Replacement Problem with P l u r a l Exalples I n i t i a l Problem: ( I s , cd, bcd], ( ( [ a , cd, bcd] -> [a, d, bcd]) ( [ a , cd, bcd] -> [a, cd, bd]) ( [ a , cd, bcd] -> [a, d, bd]))) - - Problem solving by Agent-l: Xnoeledge: a --> cd Result: ([cd, cd, b c d ] , ( ) ) ([cd cd, bcd], ( ( [ a , cd, bcd] => [a, d, bcd]) ([a, cd, bcd] -> [a, cd, bd]) ([a, cd, bcd] -> [a, d, bd]))) ~]{ Divided into a Sub problem - - Problem sol'ring by Agent-S: Knowledge: Result: c --> n i l ~ Inductive Learning ([d, d, bd], (([cd, cd, bcd] -> [a, d, bcd]) ([a, ¢d, bcd] -> [a, cd, bd]) ([a, cd, bcd] -> [a, d, bd]))) ]{]{~ P a r t i a l Problem Solved - - Problem solving by Agent-3: Xnoeledge: b --> n i l ([d, d, d], ( ( [ a , cd, bed] -> [a, d, bcd]) ( [ a , ' c d , bcd] -> [a, cd, bd]) ( [ a , cd, bcd] -> [a, d, b d ] ) ) )
FIGURE 4. Executions of the list replacement problem.
Distributed Knowledge Systems with Learning Mechanisms
423
100 90 80 70 =
60
-~
50
0 ,,-4 0
•~
40
0
30 20
o
10 t
0
l
l
l
l
l
l
l
l
l
l
l
l
[
I
I
l
I
I
I
I
I
I
t
i
t
trials *
network t y p e - - o - - f l a t
type
a
h i e r a r c h i c a l type
FIGURE 5. Problem solving in different organizations.
and problem solving. In order to compare the learning capability of agents, we first let the agents have a large capability (i.e. each agent can memorize, at most, 100 pieces of knowledge on other agents' problem solving), and we compare them to those with a small capability (i.e. each agent can memorize at most only 10 pieces of knowledge on other agents' problem solving). We give the same task continuously to the organization and observe the amount of communication in organizational problem solving. Figure 6 shows the experimental results. When agents have a high capability of learning, in other words, when agents have a large memory for the knowledge of other agents, the amount of communication decreases and they can solve the problem more efficiently.
knowledge of each agent. The process of the experiment is as follows: (1) Let the organization of three agents with a small amount of domain knowledge solve three types of problems 30 times. (2) Add an agent with a small amount of expert knowledge on decomposing problems and let the whole organization solve one problem 30 times to make a communication network. (3) Add an expert agent with a high-level knowledge to solve a complicated problem by itself. Let the organization of five agents solve one problem and observe its behavior. Figures 7 and 8 show the number of problem-solving steps and communication steps in the system. The horizontal axis shows the number of trials in the experiment, trials between 121 and 150 are the behaviors
4.3.3. Experiment 3. The third experiment aims to the investigate the influence of the difference of the 80
High__LOW
70 60 50 40 t~
30 20 10
\ l / n
/
•
•
•
•
I
I
I
I
I
I
i
I
I
I
1
I
I
J
I
I
I
I
i
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
trial FIGURE 6. Effect of leaming capability in memory-based leamlng in problem solving.
424
H. Aiba and T. Terano
15
""'""'""'""'""'""'I"
li-i-i-i-i-I-i-i-n-u-
~
l-i-i-l-l-i-i-i-i-i-i-
i-l-i-l-i-i-l-U-I
l-i-i-l-U-l-i-i-N-N-U-l-i-i-i-i-l-i-i-N-l-l-l-i-i-i-i-n-m-N
5
: ...... . . . . . . . . . . . . . . . . . . . . . . . . .
121
126
131
,
136
141
I
,
146
,
I
,
,
I
151 trials
I
I
P I
156
I
. .........
.............. 1 F I
I
I
161
I
I
i
I
I
166
I
I
I
I
I
I
171
J
I
I
I
I
I
176
- . - - Problem Solving ---D-Communication - - - Problem Solving by Expert ---o-Communication with Expert FIGURE 7. Problem $oMngpe~ormance ~a memory-basedlearnlngwhen an expertagentisavailable.
of an organization of three agents with a small amount of knowledge and an agent with a small amount of expert knowledge; and the trials between 151 and 180 are the behaviors of an organization with an expert agent with high-level knowledge. From the interval between 121 and 150 trials in Fig. 7, we can observe that the expert agent with low-level knowledge is playing an important role in the communication but its contribution to problem-solving is not much. In contrast, the expert agent with high-level knowledge makes a large contribution both in communicating and problem solving between 151 and 180 trials in Fig. 7. In a memory-based learning system, the joining of an expert does not necessarily decrease the total number of problem-solving steps. Figure 8 shows the average steps per the 30 trials of the experiment of the same problem solving and communication by the expert agent and the whole system. Figures 9 and 10 show the results of a similar experiment on the system with reliability-based learning. From Fig. 9, we can observe that participation of the expert agent with high-level knowledge improves problem-solving efficiency as a whole system of reliability-based learning. Both problem-solving steps
2025f
and communication steps decrease as the expert joined the organization. This means the expert agent contributes to improve the efficiency as a whole in problem solving. From Fig. 10, we can observe that an expert agent with high-level knowledge plays an important role. 4.3.4. Experiment 4. In human society, especially in
Japanese industries, information or indication of a wish to solve a problem are often informally spread and/or broadcast to related members before formally asking for problem solving in order to process smooth problem solving. In Japan, this is called Nemawashi, which is characterized by the prior informal communications between members of an organization before real problem solving. The fourth experiment aims to simulate Nemawashi behaviors and estimate its effects. In this experiment, we define Nemawashi as allowing agents to learn the knowledge on other agents by solving given examples. We can observe the effect of prior learning on examples that resemble the real problem. The organization consists of ten agents with the same level domain knowledge. We allow agents to solve the same example 30 times in advance of real problems in one case, and we do not let them solve any examples before the real problems in another case. We compare the
• problemsolving . . . [] .problemsolvingbyexpert .
~7~
~ 10 5 0
a/b/c/d/e/ f
cc
bb
aa
abc
FIGURE 8. The role of the expert agent in memory-based learning.
abc
Distributed Knowledge Systems with Learning Mechanisms
425
30 25
-~ 2O o 15
q-t
,$?
.~ 10
(
E
=
"m'm'm'a
m'm'm'm''
= ' m ' s - m ' "/
5
121
126
131
136
141
146
151 trials
156
161
166
171
176
Problem Solving +Communication - - - P r o b l e m Solving by Expert + C o m m u n i c a t i o n with Expert - - -
FIGURE 9. Problem solvingpe~ormancevlarella~lity-basedlearnlng whenanexpert agentlsavailable.
as a subproblem (2) a non-complicated problem that has no relationship to the example (3) a problem exactly the same as the example.
processes and results of both cases. In each case, we give three types of real problems five times in order, and observe the number of steps of problem solving and communication till the whole problem is solved. Types of real problems are as follows:
Figure 11 shows the results of the experiment. The horizontal axis shows the number of units of five trials of solving the same problem; there are twelve units of five trials in one experiment. The vertical axis shows the
(1) a complicated problem including an example
25 F 20 I
• problem solving • problem solving by expert [] communication [] communicationwith expert
~" 15 ~ lO 5 o a/b/c/d/e/
aa
bb
cc
f
abc
abc
FIGURE 10. The role of the expert agent in reliability-based leaming.
[] problemsolving [] problem solving (Nemawashi) • communication [] communication(Nemawashi)
45 r-" 40 ~ a
~ 30 .~ 25 ~ 2o
0
i i/-.-ii
1
2
3
4
5
6
7
8
9
10
units of trials FIGURE 11. Effects of
Nemawashi
activities in memory-based learning.
11
12
426
H. Aiba and T. Terano
average steps of problem solving and communicating in a unit of trials. We observe the tendency that the Nemawashi case, with prior learning, is more efficient in communication than the other case in the memory-based learning system. This result is natural because the quantity of inefficient communication decreases by prior learning example. The trial units 2, 5, 8 and 11 do not show the effect of Nemawashi. It is implied that this is because the given real problem of these units has little relationship to the example problem. 5. CONCLUDING REMARKS This paper has proposed a machine learning model from problem solving and communication for a distributed knowledge system. Our framework is appficable to the next generation information systems, which will involve distributed information agents which work in a synergistic manner (cooperatively) by exchanging information and expertise, coordinating their activities and negotiating how to solve parts of a common knowledge-intensive problem. We have carried out the simulation of problemsolving behavior based on the proposed model. The ability of problem solving improves when the agent has a large amount of memory in a memory-based learning system. By limitation of the communication paths, we can simulate various kinds of organizational structures. As we can treat different kinds of conditions on the communication paths and contents of domain knowledge of each agent, we cannot suggest which communication path is generally efficient. In selecting the agent to communicate, the cost of each communication path should be examined in a further study. Through experiment, we observe that agents form a communication network by repeating problem solving. As a new high-ability agent joins the agent system, the other agents communicate with the new agent frequently, and the efficiency of problem solving is improved. We cannot identify what kind of knowledge the agent should have in order to solve a problem efficiently in the system. At last, we have observed the effect of Nemasashi from the simulation that prior learning improves total effectiveness in problem solving. The multiagent learning model proposed here is also considered to be an extension of the knowledge acquisition tools (Boose, 1988) and machine learning techniques (Michalski, 1993) for distributed environments. The model can be extended such that the domain knowledge which each agent has (i.e. the capabilities for problem solving) will increase as time passes. Further, it is natural from our discussions that the domain knowledge is partially exchangeable between the agents. Acknowledgements--This paper is a revised version of a paper presented at both the Int. Syrup. on Fifth Generation Computer Systems
1994 (FGCS'94) Workshop on Heterogeneous Cooperative Knowledge-Bases (W3), 1994 (Terano, 1994) and the Pacific-Asian Conference on Expert Systems (PACES'95), 1995 (Aiba & Terano, 1995). The authors would like to thank the anonymous referees for their comments on earlier manuscripts. Thanks are also given to Professor Jae Kyu Lee for his recommendation. This research is supported in part by a Grant-in-Aid for Scientific Research (A06402060, Priority Areas on Emergent Systems) of the Ministry of Education, Science, Sports and Culture of Japan.
REFERENCES Aiba, H. & Terano, T. (1995). Learning mechanisms for distributed knowledge systems. In Proc. Pacific-Asian Conference on Expert Systems (PACES'95), pp. 42-49. Bond, A. H. & Gasser, L. (1988). An analysis of problems and research in DAI. Readings in distributed artificial intelligence, pp. 3-36. CA: Kaufmann. Boose, J. H. & Gaines, B. R. (Eds) (1988). Knowledge acquisition tools for expert systems. New York: Academic Press. Bradzil, P. & Muggleton, S. (1991). Learning m relate terms in a multiple agent environment. In Proc. European Working Session on Learning '91, pp. 424-439. Bridgeland, D. M. & Huhns, M. N. (1990). Distributed truth maintenance. In Proceedings 9th National Conference on Artificial Intelligence (AAAI-90), pp. 72-77, Carley, K., Park, D. & Priemla, M. (1993). Agent honesty, cooperation and benevolence in an artificial organization, AI and theories of groups and organizations: conceptual and empirical research. Working Notes: AAAI-93 Workshop on AI and Theories of Groups and Organizations, pp. 1-7. Cho, D., Kusunoki, F., Ono, S. & Terano, T. (1994). Developing a multi-agent model for distributed knowledge systems. In Proc. IEEE 2nd International Conference on Expert Systems for Development, pp. 49-54. CoUin, Z., Dechter, R. & Kaiz, S. (1991). On the feasibility of distributed constraint satisfaction. In Proc. 12th International Joint Conference on Artifwial Intelligence (IJCAI-9] ), pp. 318-324. Cutkosky, M. R., Engelmore, R. S., Fikes, R. E., Genesereth, M. R., Gruber, T. R. & Mark, W. S. (1993). PACT: an experiment in integrating concurrent software engineering. IEEE Computer, 26(1), 28-37. De Jong, G. F. & Mooney, R. J. (1986). Explanation-based generalizatinn: an alternative view. Machine Learning, 1(2), 145-176. Fox, M. (1981). An organizational view of distributed systems. IEEE Transaction on Systems, Man, and Cybernetics, 11(1), 70-80. Georgeff, M. P. (1983). Communication and interaction in multi-agent planning. In Proceedings of the 2nd National Conference on Artificial Intelligence (AAAI-83), pp. 125-129. Ishida, T., Gasser, L. & Yokoo, M. (1992). Organization self-design of distributed production systems. IEEE Transactions on Knowledge and Data Engineering, 4(2), 123-134. Keller, R. M. (1988). Defining operationality for explanation-based learning. Artificial Intelligence, 35(2), 227-241. Lee, K. C., Mansfield Jr, W. H. & Sheth, A. P. (1993). A framework for controlling cooperative agents. 1EEE Computer, 26(7), 8-16. Lenat, D. B. & Guha, R. V. (1990). Building large knowledge-based systems. Reading, MA: Addison-Wesley. Lenat, D. B. (1995). CYC: a large-scale investment in knowledge infrastructure. Communication oft he ACM, 38(11), 33-38. Mason, C. L. & Johnson, R. R. (1989). DATMS: a framework for distributed assumption based reasoning. In L. Gasser & M. N. Hubris (Eds), Distributed artificial intelligence, Vol. II, pp. 293318. CA: Kaufmann. Miehalski, R. S. (1993). Toward a unified theory of learning: multi strategy task-adaptive learning. In B. G. Buchanan & D. C. Wilkins (Eds), Readings in knowledge acquisition and learning, pp. 7-38. CA: Kanfmann.
Distributed Knowledge Systems with Learning Mechanisms Michalski, R. & Tecuci, G. (Eds) (1994). Machine learning IE: a multistrategy approach. CA: Kaufmann. Minton, S. N. (1988). Learning search control knowledge: an explanation-based approach. Boston, MA: Kluwer. Mitchell, T. M., Keller, R. M. & Kedar-Cabelli, S. (1986). Explanationbased generalization: a unifying view. Machine Learning, 1(1), 47-80. Neches, R. et al. (1991). Enabling technology for knowledge sharing. AI Magazine, 12(3), 36-56. Nishida, T. & Takeda, H. (1993). Towards the knowledgeable community. In Proc. KB&KS'93, pp. 157-166. Papazoglou, M. K., Laufmann, S. C. & Sellis, T. K. (1992). An organizational framework for cooperating intelligent information systems. International Journal of Intelligent and Cooperative Information Systems, 1(1), 169-202. Riecken, D. (Ed.) (1994). Special Issue: Intelligent agents. Communications of the ACM, 37(7), 18-147. Shavlik, J. W. (1990). Extending explanation-based learning by generalizing the structure of explanations. CA: Kaufmann (Research Notes in Artificial Intelligence Series). Shaw, M. J. & Whinston, A. B. (1989). Learning and adaptation in distributed artificial intelligence systems. In L. Gasser & M. L. Huhns (Eds), Distributed artificial intelligence H; pp. 413--430. CA: Kauffman. Sian, S. S. (1991). Adaptation based on cooperative learning in multiagent systems. In Y. Demazeau & J.-P. Miiller (Eds), Decentralized artificial intelligence, Vol. 2, pp. 257-272. Smith, R. G. (1980). The contract net protocol: high-level communication and control in a distributed problem solver. IEEE Transaction
427 on Computers, 29(12), 1104-1113. Sugawara, T & Lesser, V. (1993). On-line learning of coordination plans. In Proc. 12th International Workshop on Distributed Artificial Intelligence. Tan, M. (1993). Multi-agent reinforcement learning: independent vs cooperative agents. In Proc. lOth Machine Learning Workshop, pp. 330-337. Terano, T. (1993). Requirements of AI models applicable to organizational learning theory and two related examples. In Working Notes of AAAI-93 Workshop on AI and Theories of Groups and Organizations, pp. 92-95. Terano, T. (1994). Learning from problem solving and communication: a computational model for distributed knowledge systems. In Proc. FGCS'94, Workshop on Heterogeneous Cooperative Knowledge Bases, p. 153. Vittal, J., Silver, B., Frawley, W., lba, G., Fawcett, T., Dusseault, S. & Doleac, J. (1992). Intelligent and cooperative information systems meet machine learning. International Journal of Intelligent and Cooperative Information Systems, 1(2), 347-361. Weiss, G. (1993). Learning to coordinate actions in multi-agent systems. In 13th International Joint Conference on Artificial Intelligence (IJCAI-93), pp. 311-316. Yamamura, M. & Kobayashi, S. (1991), An augmented EBL and its application to the utility problem. In 12th International Joint Conference on Artificial Intelligence (IJCAI-91), pp. 623-629. Yokoo, M., Durfee, E. H., Ishida, T. & Kuwabara, K. (1992). Distributed constraint satisfaction for formalizing distributed problem solving. In 12th International Conference on Distributed Computing Systems (1CDC S-92), pp. 614-4521.