A distributed CSP approach for collaborative planning systems

ARTICLE IN PRESS Engineering Applications of Artificial Intelligence 21 (2008) 698– 709

Contents lists available at ScienceDirect

Engineering Applications of Artificial Intelligence journal homepage: www.elsevier.com/locate/engappai

A distributed CSP approach for collaborative planning systems Oscar Sapena , Eva Onaindia, Antonio Garrido, Marlene Arangu Universidad Politecnca de Valencia, Camino de Vera s/n, 46022 - Valencia, Spain

a r t i c l e in f o

a b s t r a c t

Article history: Received 30 January 2008 Accepted 1 March 2008 Available online 22 April 2008

Distributed or multi-agent planning extends classical AI planning to domains where several agents can plan and act together. There exist many recent developments in this discipline that range over different approaches for distributed planning algorithms, distributed plan execution processes or communication protocols among agents. One of the key issues about distributed planning is that it is the most appropriate way to tackle certain kind of planning problems, specially those where a centralized solving is unfeasible. In this paper we present a new planning framework aimed at solving planning problems in inherently distributed domains where agents have a collection of private data which cannot share with other agents. However, collaboration is required since agents are unable to accomplish its own tasks alone or, at least, can accomplish its tasks better when working with others. Our proposal motivates a new planning scheme based on a distributed heuristic search and a constraint programming resolution process. & 2008 Elsevier Ltd. All rights reserved.

Keywords: Distributed planning Collaborative planning Constraint programming Multi-agent systems Distributed constraint satisfaction

1. Introduction Distributed planning is the problem of finding a course of actions that will help a set of agents collectively satisfy certain desired goals. Due to an inherent distribution of resources such as knowledge and capability among the agents, an agent in a distributed planning system is unable to accomplish its own tasks alone, or at least can accomplish its tasks better when working with others (Durfee, 2001). Distributed planning is still an open challenge, and there is an increasingly number of applications that can benefit from this research area: cooperative robotics (Wehowsky et al., 2005) (Sirin et al., 2004), composition of semantic web services (Wu et al., 2003), manufacturing systems (Hahndel et al., 1996), etc. The problem of constructing plans in a distributed environment has been approached from two different directions (desJardins et al., 1999):  One approach has begun with a focus on planning and how it can be extended into a distributed environment. This approach is usually referred to as cooperative distributed planning because it places the problem of forming a competent plan as the ultimate objective. Although in some cases the purpose of the agents is to form a central plan, more generally the purpose is that the distributed parts of the developing plan will jointly  Corresponding author. Tel.: +34 963877007; fax: +34 963877359.

E-mail addresses: [email protected] (O. Sapena), [email protected] (E. Onaindia), [email protected] (A. Garrido), [email protected] (M. Arangu). 0952-1976/$ - see front matter & 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.engappai.2008.03.005

execute in a coherent and effective manner. Thus, in cooperative distributed planning, agents typically exchange information about their plans, which they iteratively refine and revise until they fit together well.  The other approach has begun with an emphasis on the problem of controlling and coordinating the actions of multiple agents in a shared environment. This approach usually is referred to as negotiated distributed planning. From the perspective of an individual agent, the purpose of negotiating over planned activities is not to form good collective plans but to ensure that the agent’s local objectives will be met by its plan when viewed in a global context. The emphasis is therefore not on cooperatively defining and searching the space of joint plans to find the best group plan, as in the previous approach, but on having an agent provide enough information to others to convince them to accommodate its preferences. The work presented in paper follows the cooperative distributed planning approach since the main goal is to efficiently obtain a good collective plan. The literature cites many reasons for which cooperative distributed planning is an interesting approach to pursue. One of these reasons is to split the problem into smaller subproblems which are usually easier to solve. This divideand-conquer approach has been used in several distributed planning proposals (Rehak et al., 2006; Cox et al., 2005). In these approaches, multiple agents plan to achieve their individual goals independently and, then, these individual plans are merged into a global plan. The coordination/merging process is usually the most costly part since it is necessary to avoid cross-working or duplicating effort.

ARTICLE IN PRESS O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

Another reason that often comes up is that of privacy (van der Krogt, 2007). Especially in circumstances where the agents represent companies, sharing data with other parties is considered undesirable. At the same time, it is well recognized that cooperation may be mutually beneficial to all parties. In this paper we present a new planning framework aimed at solving problems of this type. Specifically, we will address problems with the following characteristics:  Distributed domains: In these domains there exists an inherent distribution of resources such as knowledge and capability among the agents. This way, agents are clearly identified and thus a problem decomposition stage is not required.  Privacy: Agents maintain a set of private data, which are the beliefs that the agent will never share with other agents. The goals of an agent are also private, although it may require help from other agents to achieve them.  Collaboration: In this framework, an agent often needs help from other agents to achieve the necessary conditions for executing an action. However, even though actions are jointly planned, they are individually executed; that is, agents do not get synchronized to carry out a same action jointly (for example, several robots pushing a single block together into a target area). This latest type of coordination is usually addressed in team-oriented planning approaches, where several agents collaborate to achieve a common global goal. The presented approach is useful to solve planning problems in inherently distributed domains where a centralized solving process is not affordable. Additionally, agents have a collection of private data which cannot share with other agents so information exchange among them can only be achieved through the public or sharable data. Our proposal motivates a new planning scheme based on a distributed heuristic search and a constraint satisfaction problem (CSP) resolution (Dechter, 2003). The overall approach is a distributed CSP resolution for solving the kind of problems that suit well into a multi-agent planning paradigm. The remainder of this paper is structured as follows. We begin by defining the problem characteristics and showing a representative problem example. Then, we present a general overview of our approach and we describe the two main stages of the planning algorithm. Finally, we show some experimental results and we present our conclusions and future work.

699

 I is a set of propositions that represents the agent’s beliefs. This knowledge is classified into private (non-sharable) and public (sharable): I ¼ hIp, Isi. The private information, Ip, is a set of propositions that are not accessible from other agents. On the contrary, interactions between agents are possible through their public repository, where other agents can consult and add propositions to Is.  A is the set of actions that can be applied in the domain. This information, which represents the agent skills, is not accesible from other agents. As a typical planning action, an action a is a triple hpreðaÞ; addðaÞ; delðaÞi, where the preconditions, preðaÞ and the effects, addðaÞ and delðaÞ, are sets of propositions. These propositions can be agent beliefs (public or private) or public beliefs from adjacent agents: 8p 2 preðaÞ [ addðaÞ [ delðaÞ=a 2 AðagÞ, p 2 IðagÞ _ p 2 Isðag0 Þ : ag0 2 AdjðagÞ.  m is the optimization function (or metric). The agent must try to achieve its goals with the minimum cost according to this function. Definition 2. A planning problem consists of finding a (partially ordered) sequence of actions that leads the system from its initial state to a goal state. Formally, it is defined as a tuple hAg; I; G; Ai, where:  Ag is a set of agents.  I is the problem initial state, which is the union of the agents’ beliefs: [ IðagÞ. I¼ 8ag2Ag

This initial state is globally consistent because information is not replicated in different agents: 8ag; ag0 2 Ag; IðagÞ \ Iðag0 Þ ¼ ;.  G is the problem goal, which is the union of the individual goals of all agents: [ GðagÞ. G¼ 8ag2Ag

 A is the problem actions, which is the union of the individual actions of all agents: [ AðagÞ. A¼ 8ag2Ag

2. Problem definition Our approach is particularly aimed at solving problems which develop in inherently distributed domains. In this type of problems, agents are clearly identified so it is not necessary to apply a problem decomposition because there exists a natural partition/distribution of the problem itself. Definition 1. An agent is an entity with planning capabilities and thus we can specify an agent as a tuple Ag ¼ hAdj, G, I, A, mi where:  Adj is the set of adjacent agents. An agent ag0 is adjacent to an agent ag, ag0 2 AdjðagÞ, if the public information of ag0 is accessible from ag. This relationship is symmetric: ag0 2 AdjðagÞ () ag 2 Adjðag0 Þ. An agent can only collaborate directly with its adjacent agents, consulting and/or modifying their shared information.  G represent the individual goals of the agent. These goals are not visible from other agents.

Definition 3. A global plan is a partially ordered set of pairs ðag; aÞ, where ag is an agent and a is an action of that agent (a 2 AðagÞ). The execution of a by ag causes a transition in the ag’s state and, possibly, in the public beliefs of adjacent agents. Thus, a global plan is valid if the execution of all actions in the plan (by their corresponding agents and respecting the ordering constraints) on the initial state leads to a final state where all problem goals hold. The cost of a global plan is computed as the sum of the cost of all actions in the plan, according to the metric function defined in their corresponding agent. Therefore, metric functions of all agents must be defined in terms of the same measurement unit and the same scale. This simplification facilitates the computation of the plan quality, but it may not be adequate for certain problems where the optimization function of one agent gets into conflict with the objective of another agent. This problem will be addressed in a future work to improve the applicability of our distributed planning model.

ARTICLE IN PRESS 700

O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

We can find many real problems that fit the distributed problem paradigm we have described in this section. In the following subsection, we show a simple example that we will use throughout the rest of the paper to illustrate our proposal. 2.1. Problem example In this section, we show a simple example from a transportation and storage domain. In this problem we can identify two types of agents: warehouses and transport companies. In each warehouse, there is a storage area and a loading area. In the storage area, packages can be stacked and unstacked onto a fixed set of pallets by means of a hoist. All the information about the storage area (packages, stacks, etc.) in a warehouse is private, so it is not visible from other agents. Packages can also be moved from/to the loading area. The loading area of a warehouse is public, so packages in that area are visible from the adjacent agents of the warehouse. In general, the goal of a warehouse is to get a certain package distribution inside its storage area, and the metric function is to minimize the number of stack, unstack and move operations carried out to achieve its goals. A transport company manages the transportation of packages between a set of warehouses located in the same geographic area. All the information about the truck fleet and the set of routes they use is private. Transport companies access the loading area of the warehouses to pick up and deliver packages. The optimization function in these companies is to minimize the total number of truck movements. There are three agents in our example: two warehouse agents, w1 and w2, and a transportation agent t1. In the initial state, agent w1 has tree packages (a, b and c) and agent w2 has two (e and d), organized as Fig. 1 shows. The transportation agent has only one truck (truck1) and links both warehouses. Warehouse agents are only adjacent to the transportation agent, so warehouses cannot exchange information between them. We use the following predicates in this example:  on ?pkg1 ?pkg2, which states that package pkg1 in on package pkg2.  at-la ?pkg, which indicates that package pkg is in the loading-area of the warehouse. The goal of agent w1 is to get the package b on top of package e, that is, on b e. The position of the remaining packages in the warehouse is not relevant for the goal. The goals of agent w2 are on d a and on a c. Agent t1 has no individual goals. The multi-agent planning approach that we propose in this paper can help agents in situations such as the one described in this example. It offers a way to co-operate while being in control of which information is shared and with whom.

Agent w1

3. Preliminary basic concepts Collaboration is required when agents cannot reach their individual goals without the help of other agents. This help is provided through a set of abstract operations which we will call services. Definition 4. A service is an abstract operation that an agent offers to another agent or to itself to fulfill a set of propositions (service goal). These propositions are always public except when the service is computed by an agent to achieve its individual goals. Internally, a service is a (partially ordered) set of actions (local plan), but how an agent provides a service is kept as private information. In order to provide a service, an agent may require the execution of one or more services from other agents. This way, the global planning problem consists of finding a partially ordered set of services that allows all agents to achieve their individual goals. The distributed planning process is started by the agents which have individual goals to accomplish. The first step for these agents is to find out which pieces of information they require from other agents to achieve their goals. We will refer to these pieces of information as requirements. Definition 5. A requirement is a public proposition or a disjunction of public propositions that an agent requires from other agents to achieve its goals or to provide a service to another agent. A service can be seen as a planning action, where the service requirements correspond to the action preconditions and the service goal corresponds to the action effects. In the proposed example, agent w2 has to stack package d on top of a and package a on top of c, but packages a and c are not in the warehouse. Therefore, the only way to achieve its goals is to get packages a and c in its loading area. Then, the requirements are at-la a and at-la c. In the proposed example, requirements are always single propositions, but in other problems it might be a disjunction of propositions. For instance, if warehouse w2 had two different loading areas (la1 and la2), then the requirements for reaching its goals would be: (at-la1 a _ at-la2 a) and (at-la1 c _ at-la2 c).

4. Resolution model for collaborative planning In the literature we can find some proposals for solving collaborative planning tasks (Cox et al., 2005; Rehak et al., 2006). However, these works address a different problem than ours because they follow a divide-and-conquer approach, that is, they split the problem into smaller subproblems to improve the solving efficiency. On the contrary, our work focuses

Agent w2

c b

e

a

d

Loading area

Storage area

Loading area

truck1 Agent t2 Fig. 1. Initial state in the distributed planning example.

Storage area

ARTICLE IN PRESS O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

on solving an inherently distributed problem in a collaborative way. In any case, the referred approaches as well as ours use the same planning approach, POP (Partial Order Planning) techniques (Penberthy and Weld, 1992). POP techniques suit very well for distributed planning because they do not explicitly require a state-based representation and so it is not necessary to compute the problem global state, which would be a quite complex process in a distributed framework. Instead of developing a new distributed POP algorithm, we have decided to translate the planning problem into a distributed constraint satisfaction problem (disCSP). The reasons for this decision are:

701

Computation of the service list

Agent priority selection

CSP initialization

CSP formulation

 There are many distributed CSP algorithms (Yokoo and Hirayama, 2000) and some available platforms, such as DisChoco.1  A planning problem can be easily formulated as a CSP and experimental results show that this approach can be very competitive (Vidal and Geffner, 2006).

Initial assignment of variables’ values

For each agent Fig. 2 shows an outline of the proposed resolution model. As it can be observed, the collaborative planning algorithm is divided into three main sequential stages: (i) Computation of the service list: Agents which have requirements to satisfy ask their adjacent agents for help. The first stage of the planning algorithm is a message exchange process to find out which services agents can provide together with the estimated costs for these services. (ii) CSP initialization: Once the set of available services is computed, the problem is formulated as a CSP and the resulting variables and constraints are distributed among the agents. At this point, it is crucial to make an accurate initial value assignment to variables as the efficiency of our approach largely depends on these initial values. This is done according to the estimated cost of the services computed in the previous stage. (iii) CSP resolution: The distributed CSP obtained from the previous stage is collaboratively solved among all agents in order to get the final solution plan. During this solving process agents perform several executability tests on the validity of the final plan by using diverse planning techniques. Executing these tests will also allow updating the service costs; this way we can work with more accurate and up-to-date values, which is important in order to find good quality plans. These three stages are described in Sections 5–7 with more detail.

5. Computation of the service list At this stage, agents send messages to their adjacent agents to request for services that satisfy their requirements. An agent that needs a proposition sends a service request message to its adjacent agents to ask for the cost of that service. Only one proposition appears in a single message. If a requirement contains disjunctive propositions, then these propositions are individually processed in separated messages. A service request message contains the following information:  aorig : The requesting agent. 1

Available at http://www.lirmm.fr/coconut/dischoco/.

Local CSP problem resolution Backtrack to previous agent No

Solution found? Yes

Distributed CSP resolution

Executability test

Feasible solution?

No Next agent

Yes

Service cost updating

Last agent?

No

Yes Solution found

Fig. 2. Distributed planning scheme.

 adst : The target agent.  p: The requested proposition.  R: The message route, which is the sequence of agents the message has passed through (the target agent is not included). This information avoids infinite message loops.  Id: The message identifier is composed of a sequence of numbers. Each time a message is propagated, a number is added at the end of the Id. This number is the same for all propositions in the same requirement (i.e., for disjunctive propositions) but different for propositions in different requirements. Let us suppose that an agent has the following requirements: (p1 _ p2) and p3. The message identifier will be, for instance, ‘‘1’’ for the request of p1 and p2 (in separated messages as they are disjunctive propositions) and ‘‘2’’ for the request of p3. An agent which receives several messages can easily check through the message identifier and the message route whether all the requested propositions are required or they represent alternative ways to accomplish a requirement (disjunctive propositions).

ARTICLE IN PRESS 702

O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

When an agent receives a service request message and it cannot provide the service, it returns an infinite cost as a reply. Otherwise, the agent must:  Analyze the necessary requirements to provide the service.  Ask its adjacent agents for help if necessary.  If the requirements can be achieved, compute a plan to estimate the cost of the service. This cost is sent back as a reply to the service request message. Additionally, if an agent receives two single different messages with non-disjunctive propositions, it automatically creates a new service for achieving both propositions together. This combination of propositions is done because achieving several proposition at a time is usually less costly than handling each subgoal independently and thus it will positively affect the plan quality. However, since it is not affordable to compute all possible combinations, we will only consider the number of 2-combinations from a set with n non-disjunctive propositions (n  ðn  1Þ=2) plus one k-combination for k42. This calculation makes about n2 =2 number of combinations to study; this relaxation provides a good trade-off between computational cost and plan quality. The message exchange process requires agents to store some information about the received messages and the provided services. For each agent, the stored information is the following:  Message database (MsgDB): In this database the agent stores the received messages (the format of these messages was described above).  Service database (SerDB): It stores the services provided by the agent. Tuples are of the form hG, aorig , NecReq, MinReq, minReqCost, Plan, costi, where:  G is the service goal, that is, the conjunction of propositions the service achieves.  aorig is the agent that requested the service.  NecReq is the set of requirements needed to provide the service goal.  MinReq is a conjunction of propositions that satisfy the service requirements (NecReq) with the minimum cost. For requirements with disjunctive propositions, the alternative with the minimum cost is selected.  minReqCost is the estimated cost to achieve MinReq.  Plan is the internal plan that allows to achieve G, assuming that the requirements hold.  cost is the estimated service cost, which corresponds to the Plan cost, computed according to the metric function of the agent.  Requirement database (ReqDB): This database stores the responses to the messages the agent has sent to request for services to other agents. This information is stored in tuples of the form hG; cost; adst i, where:  G is the service goal.  cost is the estimated cost to achieve the propositions in G.  adst is the agent that provides this service. Algorithm in Fig. 3 shows the behavior of agent adst when it receives a service request message from agent aorig . In this algorithm, we have used the following functions:  Unreachable(p): It returns true if no action allows adst to achieve proposition p, regardless of whether the action preconditions hold or not.  ComputeNecessaryRequirements(p): This function returns the necessary requirements to achieve p.

Fig. 3. Service request processing.

 RequirementsAchieved(NecReq): It returns true if NecReq can be satisfied.  ComputeMinimumRequirements(NecReq): It returns the set of propositions that satisfies NecReq with the minimum cost.  ComputePlan(S, G): It returns a plan to achieve the set of propositions G from the initial state S.  PlanCost(Plan): It returns the cost of Plan according to the defined problem metric.  Conjunctive(p1, p2): Returns true if both propositions are conjunctive. This can be easily computed through the route and the identifier of their respective messages.  ServiceCombination(G): This method computes the cost of a service that achieves jointly all propositions in G. Therefore, it is necessary to calculate the minimum requirements and a new plan to achieve jointly all propositions in G. If this service is less costly than achieving all propositions in G separately, then a message containing this information is delivered to the requesting agents.

At the end of this process, each agent will have a table with the list of services it provides along with their associated cost. Additionally, each agent must also compute a set of services to achieve its own goals. Agent w1, for instance, will compute a service for satisfying its goal on b e. Combinations with other services are also studied, in the same way as shown in Fig. 3. Tables 1–3 show the services provided by agents w1, w2 and t1, respectively, for the problem in Fig. 1. The first column assigns a number to each service. The second and the third column show the service goal: each proposition that the service achieves (third column) is preceded by the requesting agent (second column). The fourth column indicates the service cost, without taking into account the cost of the requirements. The two last columns show the necessary requirements to provide the service goal: each requirement proposition (fifth column) is followed by the agent which supplies such a proposition with the minimum cost (last column). A proposition such as at-la a or on b e is ambiguous

ARTICLE IN PRESS O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

703

Table 1 List of services provided by agent w1 #

Requesting agent

Service goal

Cost

Requirements

Supplier

1

t1

at-la a w1

6

;

2

t1

at-la c w1

2

;

3

t1

at-la a w1 ^ at-la c w1

6

;

4

w1

on b e w1

6

at-la e w1

t1

5

t1 w1

at-la a w1 ^ on b e w1

8

at-la e w1

t1

6

t1 w1

at-la c w1 ^ on b e w1

6

at-la e w1

t1

7

t1 w1

at-la a w1 ^ at-la c w1 ^ on b e w1

8

at-la e w1

t1

Table 2 List of services provided by agent w2 #

Requesting agent

Service goal

Cost

Requirements

Supplier

1

t1

at-la e w2

2

;

2

w2

on d a w2

6

at-la a w2

t1

3

t1 w2

at-la e w2 ^ on d a w2

6

at-la a w2

t1

4

w2

on a c w2

4

at-la a w2 ^ at-la c w2

t1

5

w2

on a c w2 ^ on d a w2

8

at-la a w2 ^ at-la c w2

t1

6

t1 w2

at-la e w2 ^ on a c w2

6

at-la a w2 ^ at-la c w2

t1

7

t1 w2

at-la e w2 ^ on a c w2 ^ on d a w2

8

at-la a w2 ^ at-la c w2

t1

Table 3 List of services provided by agent t1 #

Requesting agent

Service goal

Cost

Requirements

Supplier

1

w1

at-la e w1

2

at-la e w2

w2

2

w2

at-la a w2

1

at-la a w1

w1

3

w1 w2

at-la e w1 ^ at-la a w2

2

at-la e w2 ^ at-la a w1

w2 w1

4

w2

at-la c w2

1

at-la c w1

w1

5

w1 w2

at-la e w1 ^ at-la c w2

2

at-la e w2 ^ at-la c w1

w2 w1

6

w2

at-la a w2 ^ at-la c w2

1

at-la a w1 ^ at-la c w1

w1

7

w1 w2

at-la e w1 ^ at-la a w2 ^ at-la c w2

2

at-la e w2 ^ at-la a w1 ^ at-la c w1

w2 w1

ARTICLE IN PRESS 704

O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

at-la e w 1 6

W1 #4

at-la e w 2 2

on b e w 1

t1 #1

2

at-la e w 1

w2 #1

at-la e w 2

requirements service cost

AG #n service goal

at-la a w 2 6

W2 #2

on a c w 2

t1 #2

6

w1 #1

at-la a w 1

at-la a w 2

at-la a w 2 at-la c w 2 W2 #4

service number

at-la a w 1 1

on d a w 2

4

agent that provides the service

at-la c w 1 1

t1 #4

at-la c w 2

2

w1 #2

at-la c w 1

Fig. 4. Causal connections of the obtained services in the proposed example (service combinations are not included for simplicity).

unless the name of the warehouse is specified, since all warehouse agents use the same predicate names to represent their beliefs. For this reason, each proposition in the tables is augmented with its owner agent. Let us see, for example, the service #1 provided by agent t1 (Table 3). The goal of this service is to have the package e in the loading area of the warehouse w1, which is a proposition requested by agent w1. Agent t1 can satisfy this goal with only two truck movements: move truck1 from w1 to w2, load the package e, move truck1 back again to w1 and unload the package. Therefore, the estimated cost of this service provided by the transportation agent, t1, is 2; that is, agent t1 invests an effort of 2 cost units to achieve at-la e w1. However, this plan is only executable if package e is in the loading area of w2. Then, the requirement to provide this service is at-la e w2, which is provided by agent w2 (that is, w2 is the supplier of this requirement). Services #3; #5; #6 and #7 are service combinations in the tables of the three agents. For example, the service #7 of w1 (see Table 1) provides the propositions at-la a w1 and at-la c w1 to agent t1 and it also satisfies one of its own goals (on b e w1). Therefore, service #7 of w1 is the combination of services #1; #2 and #4. It can be observed that the estimated cost of service #7 (cost 8) is less than the cost of the other three executed separately (cost 6 þ 2 þ 6). This fact, which is very usual in this type of problems, is the reason for computing the service combinations. For agent t1, the need for service combinations is still more obvious since transporting two or more packages in the same trip is clearly less costly than carrying only one package in each trip. Since services can be seen as planning actions with preconditions and effects, the information contained in Tables 1–3 can also be explained through a graph-like representation. Fig. 4 shows the causal connections between the performance of the services for the proposed example (we have excluded the combination of services for the sake of simplicity). Nodes represent actions; the caption inside the nodes denotes the service number and the agent which provides the service; propositions above the node represent the service requirements (action preconditions) and propositions below the node represent the action effects, i.e. the service goal. The number at the front of each node denotes the cost of achieving the service goal, without considering

Fig. 5. Translation of the third service of agent t1 (see Table 3) to a PDDL action.

the requirement cost. The total estimated cost, for instance, to achieve the goal on b e w1 is 10, which is the sum of the cost of the three services required to reach this proposition (service #1 of w2 ! service #1 of t1 ! service #4 of w1). The top left node in the graph, for instance, represents the service #4 of w1, which achieves the goal on b e w1. To achieve this goal, w1 requires package e to be in its loading area, and this requirement is provided by the service #1 of agent t1 with and estimated cost of 2 (two truck movements). For its part, this service requires package e to be at the loading area of warehouse w2. Agent w2 provides the service #1 to satisfy this requirement (see the top right node in the graph). This service has a cost 2 since it is necessary two package movements to put package e in its loading area. 6. CSP initialization Once the set of services is available, our next task is to formulate the planning problem to solve, which consists of finding the best sequence of services that the agents must execute to achieve all their goals. The formulation as a planning problem is straightforward since each service can be easily translated into a PDDL2 action:  The requirements stand for the action preconditions as we have seen in the example of Fig. 4. 2 PDDL (Ghallab et al., 1998) is a standard language for describing planning domains.

ARTICLE IN PRESS O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

705

 The requirements may also correspond with the negative effects of the action (delete effects) in case they are deleted after the service execution, i.e. in case the requirements are not longer true after the service execution.  The service goal stands for the positive effects (add effects) of the planning actions.  The service cost can be modelled as a numeric fluent variable; effects can make use of a selection of assignment operations in order to update the values of primitive numeric expressions. These include direct assignment and relative assignments (such as increase and decrease).

For this formulation, we have defined the following integer variables:

Fig. 5 shows the translation of service #3 of agent t1 into a PDDL action. As it can be seen in Fig. 5, the two requirements are deleted after the service execution since the new location of package e is now the loading area of agent w1 and the new location of package a is now the loading area of agent w2. The service cost is 2 units, which is captured in the increase expression to update the global problem cost. The resulting planning problem is then formulated as a CSP and the resulting variables and constraints are distributed among the agents. This initialization process is described in detail in the following subsections.

If an action s is included in the plan (InplanðsÞ ¼ 1), then the following constraints must hold:

6.1. Selecting the agent priority In this type of decentralized problems it is highly recommendable to establish a priority/order between the participating agents. As we will see in Section 8, this order will substantially affect the search performance. Experimentally, we have observed that the most efficient order assignment is to set the highest priority to the agent which provides the highest number of services without requirements to be supplied by other agents. In the proposed example, w1 provides three services (#1; #2 and #3, see Table 1) to t1, and these services do not require any proposition from other agents. Agent w2 only provides one service without requirements (service #1, see Table 2), and t1 does not provide any one (see Table 3). Therefore, the highest priority is assigned to agent w1. The next agent in the priority list is the agent which provides a higher number of services to agents which already have a priority assigned, and so on. In the proposed example, t1 is the agent which provides more services to w1 (see column Supplier in Table 1) and so it will be assigned the next highest priority. Finally, w2 will be the agent with the lowest priority. As it can be observed in Fig. 4, this priority assignment follows the main causal line of the services: w1 ! t1 ! w2. This way, the number of shared variables and constraints between two sequential agents will be larger than following a different priority criteria. Therefore, when an agent communicates its partial solution to the following agent, it is possible to efficiently restrict the variable domains, which minimizes the number of required backtrackings and, as a result, the computational effort. Section 8 shows a comparison between the use of this priority assignment and a random one. It can be observed that without a good priority order, the CSP resolution becomes an unfeasible task.

 Inplan(s) 2 [0,1] represents whether the action/service s is in the final plan (value 1) or not (value 0).  Start(s) 2 [0,1] represents the start time of s.  Support(p, s) 2 S, where S is the set of actions that add p and represent a potential support for s. These variables represent the causal links between the agent services.  Time(p, s) 2 [0,1] is the time when the causal link Support(p, s) happens.

 The action that produces p for s must be in the plan: Supportðp; sÞ ¼ s0 !Inplanðs0 Þ ¼ 1.  Preconditions of s must hold before its start time: Timeðp; sÞpStartðsÞ.  The duration of an action is one time unit (we do not handle durative actions), so its effects are achieved one unit of time after the action’s start time: Supportðp; sÞ ¼ s0 !Timeðp; sÞ ¼ Startðs0 Þ þ 1.  Mutex relationship between an action s that requires p and other action s0 in the plan that deletes p: StartðsÞaStartðs0 Þ þ 1.  Threat resolution by promotion or demotion when an action s0 in the plan deletes a proposition p that is required by s: ðStartðs0 Þ þ 1oTimeðp; sÞÞ _ ðStartðsÞoStartðs0 ÞÞ. Agents handle the variables and constraints that are related to their own services. However, there are some variables and constraints that must be shared between agents. This occurs when the results of a service provided by an agent are required by the service of another agent. In this case, the agent with the highest priority will assign a value to the shared variables, whereas the other agent will check whether the shared constraints are fullfilled. The formulation of a planning problem as a CSP implies the definition of a great number of variables and constraints. For instance, the formulation of the proposed example returns 227 variables and 781 constraints (57, 86 and 84 variables and 182, 297 and 302 constraints for agents w2, t1 and w1, respectively). One of the key explanations that cause the problem complexity is the high number of computed service combinations: if two conjunctive propositions, p1 and p2, have been requested to an agent, that agent automatically computes a new service to achieve jointly p1 and p2. Let us call s the service that achieves p1 and s0 the service that achieves p1 ^ p2. Obviously, if s is already included in the plan, then s0 will not be included (and the other way around), since both services represent alternative ways to attain p1. To model this fact in the CSP formulation, we include the following additional constraints: if two services in an agent, s and s0 , produce the same proposition, then both services are mutually exclusive in the plan: ðInPlanðsÞ ¼ 1!InPlanðs0 Þ ¼ 0Þ ^ ðInPlanðs0 Þ ¼ 1!InPlanðsÞ ¼ 0Þ. These additional constraints substantially improve the processsolving performance.

6.2. Formulating the problem as a CSP 6.3. Initial assignment of variables’ values Based on the works of Vidal and Geffner (2006), Refanidis (2005) and Garrido et al. (2006), we can translate the list of obtained services (or planning actions) into a CSP formulation.

The most costly part of the CSP solving process is to find the actions that must be included in the plan, that is to make a feasible

ARTICLE IN PRESS 706

O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

assignment of the initial values for the Inplan variables. Therefore, it is important to make accurate assignments in order to achieve good search performance at low processing cost. The value assignment in our approach is carried out in two stages. For each agent ag: (i) We compute a set with the propositions requested by its adjacent agents. Agent ag itself is also considered as an adjacent agent since the services computed to satisfy its goals can be seen as self-requests. For example, let us consider agent w2 in the proposed example; the set of requested propositions is Requests of agent t1 : fat-la eg, Self-requests : fon d a; on a cg. At it can be observed in Table 2, the set of requested propositions corresponds to the goals of the services provided by the agent. (ii) We compute a set of services/actions to satisfy each requested proposition with the minimum cost. Following the example, the services for agent w2 are

7.1. Local problem resolution In the literature we can find several distributed CSP algorithms: asynchronous backtracking (ABT), asynchronous weakcommitment search (AWC), distributed breakout, etc. (Yokoo and Hirayama, 2000). However, we are not interested at present in solving the problem the most efficiently as possible, but in demonstrating that our approach is viable for solving this type of problems. For this reason, we have implemented a simple sequential forward checking algorithm. As a future work, we want to use a more sophisticated algorithm in order to decrease the number of messages that the agents need to exchange during the search. Following the priority order, each agent computes a solution for its local problem and then it communicates such a solution to the next agent. A valid local solution is a variable value assignment that satisfies all the agent constraints. However, before passing the solution, the agent must previously check that the computed solution is feasible, that is it can be executed in the current state. This verification is described in the following subsection.

Services for the requests of agentt1 : fservice #1g, 7.2. Executability test

Services for the self-requests : fservice #5g. Then the Inplan variables attached to service #1 and #5 for agent w2 (see Table 2) are initially set to 1. The Inplan variables for the rest of services are set equal to 0. We have experimentally observed that the use of this heuristic for assigning initial values outstandingly improves the search performance, above all when the selected services end up being included in the final plan. Section 8 shows an empirical analysis on the benefits of using this heuristic in the CSP solving process.

7. CSP resolution The goal of this process is to obtain a final global plan, which is a partially ordered list of services that each agent will have to carry out: Inplan variables indicate which services must be executed and Start variables establish the chronological order of their execution. Fig. 2 shows the scheme of the CSP resolution process. As it can be observed, this process is divided into three sequential stages: (i) Local problem resolution: Each agent tries to find a feasible value assignment for its CSP variables that satisfies all its CSP constraints. (ii) Executability test: An additional test is required to verify that the sequence of selected services is executable in the current state. (iii) Service cost updating: The cost of the selected services is updated so that the actual cost of the final plan can be easily computed. These stages are described in detail in the following subsections.

I(ag) ∪ requirements(s0)

S0

I1(ag) ∪ requirements(s1)

S1

This latest verification is necessary as we only can assure that the services computed in the first stage of the algorithm are executable in the initial state (and only if their requirements hold). In other words, if an agent ag provides a service s, then it means there is a plan to reach the service goal from the state S IðagÞ requirementsðsÞ, but we cannot guarantee that this plan exists if the initial state changes. Let us suppose that an agent finds a variable value assignment that satisfies all the constraints. This means that, according to the Inplan and Start variables, the agent has selected a sequence of services to execute. Let us suppose that the sequence of services is s0 !s1 !s2 !    sn . This sequence has the following properties:  Two services are never executed in parallel: Startðsi ÞaStartðsj Þ;

8iaj.

If the agent needs to achieve the goals of services si and sj at the same time, then the agent can execute a service combination instead which achieves both goals together.  The first service in the sequence, s0 , is executable as it is executed in the initial state.  It is necessary to check the executability of the rest of services (si ; i40), because the execution of a service modifies the internal state of the agent. Let us call, for example, I1 ðagÞ the resulting state after executing service s0 (see Fig. 6). Then, we have to check that there exists a plan that reaches the goals of service s1 starting from the state I1 ðagÞ [ requirementsðs1 Þ. The same test must be carried out for the rest of services (s2 ; . . . ; sn ). If there is a valid

I2(ag) ∪ requirements(s2)

Fig. 6. Executability test for the obtained sequence of services of an agent.

In+1(ag) Sn

ARTICLE IN PRESS O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

plan for each service, then the local solution is feasible and is communicated to the next agent. 7.3. Service cost updating The plans computed during the executability test are also used to update the service costs. The costs of the services computed in the first stage of the algorithm (see Section 5) are just an estimate to make the initial value assignment, but the cost of these plans represent the actual cost of the services. This information is necessary during the CSP solving process for evaluating the quality of the achieved solutions and for pruning the search space. The CSP solving process can finish when a first solution is found but, if we are interested in an optimal (or close-to-optimal) solution, the agents can continue the search for better plans using the achieved solutions as a prune. When the final plan is obtained, each agent knows which services has to execute and at what time. Table 4 shows the final plan obtained for the proposed example. This way, the plan execution becomes an easy task.

Table 4 Final plan for the proposed example Time

Agent

Service

Description

0 0 1 2 2

w1 w2 t1 w2 w1

#3 #1 #7 #5 #4

Move packages a and c to the loading area Move e to the loading area Transport a and c to w2 and e to w1 Stack a on c and d on a Stack b on e

Table 5 Obtained results for an increasing number of agents #Ag

#Services

#Vars

#Constraint

#Messages

#Time

3 4 5 6

21 28 29 33

149 198 207 233

781 805 831 954

258 69 217812 209

0.16 0.20 9.13 0.27

707

8. Results For evaluating of our approach, we have defined four problem instances with an increasing number of agents. The first problem, with three agents, corresponds to the proposed problem example in Fig. 1. The second problem adds a new warehouse, w3, with an individual goal to achieve. The third problem adds a new transport company, t2. The last problem, with six agents, additionally includes a new warehouse, w4 with another individual goal to solve. In the current implementation of the CSP solver, the search stops when the first solution plan is found. Therefore, the results presented in this section are related to the computational cost and the quality of the first solution found. However, this behavior can be easily changed to allow the search to continue until a deadline is reached or even until the optimal solution is found. The obtained results are displayed in Table 5. The first column indicates the number of agents in each problem instance. The second column shows the total number of services provided by the agents. The third and the fourth column show the total number of variables and constraints in the CSP formulation, respectively. The fifth column shows the number of messages exchanged during the CSP solving process. The last column shows the time in seconds for obtaining the first solution. The distributed CSP solver has been executed on a single computer (Pentium 4–3 GHz.), so that we have not considered message delays in the agents communication. The goal of this evaluation is not to show a broad range of benchmark results (since the whole process is not fully automated yet), but to demonstrate that our approach is valid for solving this type of problems. Although the current implementation can be improved in many ways, the small number of exchanged messages and the low computation time indicate that our approach can be successfully used for solving small and medium size distributed planning problems. There are two important parameters that substantially affect the CSP solving process performance: the agent priority and the initial value assignment for the CSP variables. Table 6 shows the effect of the agent priority order in the search performance. The first column indicates the number of agents in each problem instance and the second column shows the results when using the priority criteria proposed in Section 6.1. The results of this second column are, therefore, the same as the ones shown in Table 5. They confirm that applying a priority order consistent with the

Table 6 Effect of the agent priority assignment on the search performance #Ag

Proposed priority

Inverse priority

Random priority #1

Random priority #2

3 4 5 6

258 msg/0.16 s 69 msg/0.20 s 217 812 msg/9.13 s 209 msg/0.27 s

872 msg/0.20 s 4 764 021 msg/309.23 s 323 824 msg/15.58 s 303 msg/0.31 s

7 125 438 msg/371.52 s 415 min 415 min 415 min

415 min 4037 msg/0.47 s 17 106 msg/2.27 s 415 min

For each priority assignment, the number of exchanged messages and the time in seconds is shown (#msg/time).

Table 7 Comparison of four initial value assignment methods for the CSP variables #Ag

Proposed value initialization

All services excluded

All services included

Random initialization

3 4 5 6

22/258 msg/0.16 s 27/69 msg/0.20 s 28/217 812 msg/9.13 s 33/209 msg/0.27 s

23/165 878 msg/53.72 s 415 min 415 min 415 min

22/258 msg/0.16 s 26/10 075 msg/7.76 s 29/215 480 msg/8.93 s 31/12 015 msg/5.22 s

21/22 602/10.66 s 415 min 415 min 33/1079 msg/0.41 s

Results show the quality of the plan, the number of exchanged messages and the time in seconds (quality/#msg/time).

ARTICLE IN PRESS 708

O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

causal connections between the performance of services allows us to efficiently solve the problems. The third column shows the results when using the inverse priority order with respect to the second column, that is the inverse order of the causal connection between services. This priority assignment also provides a good rate of shared variables and constraints between two consecutive agents, but the solving process is not as efficient as in the second column. The two last columns show the results when using a random priority. In general, a random assignment is not a good choice as the CSP solving process takes more than 15 min to solve several problem instances. However, a random assignment can exceptionally obtain particular good results, as the ones obtained in the last column for the fiveagent problem instance. Overall, we can conclude that our priority criteria turns out to behave very well in most of the cases, although it may not be necessarily the optimal assignment. The initial value assignment for the CSP variables has a considerable influence on the computational cost and it also affects the quality of the first solution found. Table 7 shows a comparison of the plan quality and the required computational effort for four different initial value assignments. The first column indicates the number of agents and the second one shows the results obtained by using the proposed assignment method (see Section 6.3). The third column shows the results obtained when all services are initially excluded from the plan (Inplan variables for all services are initially set to 0). The fourth column shows the opposite case, that is, all services are initially in the plan. The last column shows the results of a random initial value assignment. As it can be observed, setting all Inplan variables initially to 0 or determining their initial value randomly is not a good choice because the computational cost is too high is many cases. Setting all Inplan variables initially to 1 (see fourth column) produces very good quality plans and the computational effort is not very high. The most efficient assignment method is the proposed one and the quality of first solution is almost as good as the obtained with the other methods. Therefore, the proposed method is very appropriate for finding rapidly a first solution and continue searching for a better quality plan.

9. Discussion and future directions According to the results obtained in our experiments, the presented approach is a good approximation for solving the type of distributed planning problems we are dealing with in this paper. However, our approach still has some improvement points that must be addressed in future works:  Regarding the issue of the global plan quality, we have assumed that the optimization functions of all agents are compatible. However, this restriction may not be adequate in certain problems where the optimization function of one agent gets into conflict with the objective of another agent. This problem could be addressed by using negotiation techniques to reach an acceptable final plan for all the participating agents.  It is not possible to guarantee the optimal solution unless all combinations of services have been computed. In this paper we have proposed to compute only a polynomial number of service combinations because this number represents a good trade-of between computational cost and plan quality. However, searching for the optimal solution may be necessary in many situations and it would be interesting to find an alternative way to get it with less computational effort. The main efforts of our future work will follow this direction. As an additional future work, we want to fully automatize the planning process, using a multi-agent platform such as, for

example, JADE (Bellifemine et al., 2007). This will allow us to test our algorithm in a wide range of benchmark domains. On the other hand, there are some improvements that can be introduced in the algorithm, such as the use of a more efficient distributed CSP solving method and the computation of new heuristics for the variable and the value selection.

10. Conclusions In this paper, we have presented a new approach for multiagent planning, based on the extraction of heuristic information and the problem formulation as a CSP. Unlike other existing proposals that follow a divide-and-conquer approach, we focus our work on inherently distributed problems in which agents are clearly identified. In this type of problems, privacy is usually a key issue: agents keep private information that they do not want to share with other agents. At the same time, some information must be shared to allow the cooperation between the agents, which is required to reach their goals. In our approach, the collaboration and the privacy between agents is achieved through the definition of services, which are abstract operations provided by the agents. This way, the planning process consists of two sequential steps:  A message exchange process where agents request the information that they need to other agents and where the set of available services is obtained. The cost of each service is used as heuristic information for the search process.  The problem is formulated as a distributed constraint satisfaction problem and solved with CSP techniques. The preliminary results show that this distributed planning approach fits very well the type of problem we have described at the beginning of this paper, particularly for solving small and medium size problems. Moreover, the decision choices on the agent priority and the value selection heuristic outperform other methods either in plan quality as well as computational cost. In summary, we can conclude that the overall scheme we propose in this paper obtains very good results for distributed planning problems where collaboration is a major issue in the problemsolving process.

Acknowledgments This work has been partially supported by the Spanish government projects MEC TIN2005-08945-C06-06 (FEDER) and Consolider-Ingenio 2010 CSD2007-00022. References Bellifemine, F.L., Caire, G., Greenwood, D., 2007. Developing multi-agent systems with JADE. Wiley Series in Agent Technology. Cox, J.S., Durfee, E.H., Bartold, T., 2005. A distributed framework for solving the multiagent plan coordination problem. In: AAMAS’05, pp. 821–827. Dechter, R., 2003. Constraint Processing. Morgan Kaufmann, Los Altos, CA. desJardins, M.E., Durfee, E.H., Ortiz, C.L., Wolverton, M.J., 1999. A survey of research in distributed, continual planning. AI Magazine 4, 13–22. Durfee, E.H., 2001. Distributed problem solving and planning. Mutli-agents Systems and Applications, pp. 118–149. Garrido, A., Onaindia, E., Arangu, M., 2006. Using constraint programming to model complex plans in an integrated approach for planning and scheduling. In: UK Planning and Scheduling SIG Workshop (PLANSIG), pp. 137–144. Ghallab, M., Howe, A., Knoblock, C., McDermott, D., Ram, A., Veloso, M., Weld, D., Wilkins D., 1998. PDDL—the planning domain definition language. AIPS-98 Planning Committee.

ARTICLE IN PRESS O. Sapena et al. / Engineering Applications of Artificial Intelligence 21 (2008) 698–709

Hahndel, S., Fuchs, F., Levi, P., 1996. Distributed negotiation-based task planning for a flexible manufacturing environment. Distributed Software Agents and Applications, Lecture Notes in Artificial Intelligence. Penberthy, J.S., Weld, D.S., 1992. UCPOP: a sound, complete, partial order planner for ADL. In: Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, pp. 103–114. Refanidis, I., 2005. Stratified heuristic POCL temporal planning based on planning graphs and constraint programming. In: Proceedings of the ICAPS-2005 Workshop on Constraint Programming for Planning and Scheduling. Rehak, M., Pechoucek, M., Volf, P., 2006. Distributed planning algorithm for coalition logistics in semi-trusted environment. In: Proceedings of the IEEE Workshop on Distributed Intelligent Systems, pp. 265–272. Sirin, E., Parsia, B., Wu, D., Hendler, J., Nau, D., 2004. HTN planning for web service composition using SHOP2. Journal of Web Semantics 1 (4), 377–396.

709

van der Krogt, R., 2007. Privacy in multiagent planning: a classical definition with illustration. In: AAMAS 2007 Workshop on Coordinating Agents Plans and Schedules. Vidal, V., Geffner, H., 2006. Branching and pruning: an optimal temporal pocl planner based on constraint programming. Artificial Intelligence 170 (3), 298–335. Wehowsky, A.F., Block, S.A., Williams, B.C., 2005. Robust distributed coordination of heterogeneous robots through temporal plan networks. In: Proceedings of the ICAPS Workshop on Multi-agent Planning and Scheduling. Wu, D., Sirin, E., Hendler, J., Nau, D., Parsia, B., 2003. Automatic web services composition using SHOP. Proceedings of ICAPS. Yokoo, M., Hirayama, K., 2000. Algorithms for distributed constraint satisfaction: a review. Autonomous Agents and Multi-Agent Systems 3, 185–207.