Bayesian-based preference prediction in bilateral multi-issue negotiation between intelligent agents

Knowledge-Based Systems 84 (2015) 108–120

Contents lists available at ScienceDirect

Knowledge-Based Systems journal homepage: www.elsevier.com/locate/knosys

Bayesian-based preference prediction in bilateral multi-issue negotiation between intelligent agents Jihang Zhang ⇑, Fenghui Ren, Minjie Zhang School of Computer Science and Software Engineering, University of Wollongong, Wollongong, NSW, Australia

a r t i c l e

i n f o

Article history: Received 9 December 2014 Received in revised form 9 February 2015 Accepted 2 April 2015 Available online 6 April 2015 Keywords: Multi-issue negotiation Multi-agent system Opponent modelling Preference prediction Bayesian learning

a b s t r a c t Agent negotiation is a form of decision making where two or more agents jointly search for a mutually agreed solution to a certain problem. In multi-issue negotiation, with information available about the agents’ preferences, a negotiation may result in a mutually beneficial agreement. In a competitive negotiation environment, however, self-interested agents may not be willing to reveal their preferences, and this can increase the difficulty of negotiating a mutually beneficial agreement. In order to solve this problem, this paper proposes a Bayesian-based approach which can help an agent to predict its opponent’s preference in bilateral multi-issue negotiation. The proposed approach employs Bayesian theory to analyse the opponent’s historical offers and to approximately predict the opponent’s preference over negotiation issues. A counter-offer proposition algorithm is also integrated into the prediction approach to help agents to propose mutually beneficial offers based on the prediction results. Experimental results indicate good performance of the proposed approach in terms of utility gain and negotiation efficiency. Ó 2015 Elsevier B.V. All rights reserved.

1. Introduction Intelligent agents are encapsulated software entities that have the ability to make decisions autonomously in dynamic environments to meet their pre-designed objectives [1–3]. In multi-agent systems, agents usually need to cooperate with each other in order to achieve certain goals in a shared environment. However, the agents may have conflicts about how to cooperate with each other to achieve these goals and this involves negotiation. Agent negotiation is a form of decision making where agents jointly explore possible solutions in order to reach an agreement [4–7]. In recent decades, agent negotiation technology has been widely developed to solve issues in different areas, such as business transactions in ecommerce [8,9] and service management in cloud computing [10,11]. With the support of agent negotiation technology, many operations which originally required human intervention can be conducted automatically and intelligently by autonomous agents, and this means that very large amounts amount of time and money can be saved. Currently, one major research challenge in this area is opponent modelling [12–15]. More precisely, during a negotiation, agents

⇑ Corresponding author at: 1/6 Cassian Street, Keiraville, NSW 2500, Australia. Tel.: +61 02 0451402321. E-mail addresses: [email protected] (J. Zhang), [email protected] (F. Ren), [email protected] (M. Zhang). http://dx.doi.org/10.1016/j.knosys.2015.04.006 0950-7051/Ó 2015 Elsevier B.V. All rights reserved.

usually need to use a number of negotiation parameters (i.e. deadline, preference, reservation utility and concession strategy) to make wise decisions so that a win–win agreement can be reached. Some cooperative negotiation strategies have assumed that these negotiation parameters are public information. In a competitive environment (non-cooperate negotiation), however, self-interested agents usually keep their negotiation parameters secret in order to avoid being exploited by their opponents [16]. Without the knowledge of opponents’ negotiation parameters, agents may have difficulty in adjusting their negotiation strategies properly to a reach win–win agreement. In order to overcome this difficulty, prediction approaches has been integrated into agents’ negotiation strategies in recent years to estimate opponents’ negotiation parameters. In multi-issue negotiation, one of the most important negotiation parameters is the negotiation preferences on negotiation issues, because the preferences can play a critical role in terms of agents utility gains and the success rate of a negotiation. Precisely speaking, in multi-issue negotiation, an agent’s preference indicates the agent’s weighting over different negotiation issues. A high weighted issue can help agents to generate more utility comparing with a low weighted issue. During a multi-issue negotiation, an offer that an agent proposed should not only maximise its own utility, but also try to minimise the damage on its opponent’s utility, so that the opponent agent will be more willing to accept the offer. In order to propose such an offer, agents need to

109

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

know their opponents’ preferences on negotiation issues. According to the opponent’s preference, an agent can trade off negotiation issues. In other words, while an agent makes some concession on its opponent highly weighted issues, it also tries to gain some payoff from the low weighted issues, so that both agents can benefit from the offer. In recent years, many different approaches have been proposed to help agents to predict their opponents’ preferences. These include: genetic algorithm-based prediction [17], statistical analysis-based prediction [18,19] and machine learning-based prediction [20]. However, all these approaches have different limitations. For example, the approaches in [18,19] require previous negotiation data to make the prediction and the approach in [20] may need a long training time before the prediction algorithm becomes effective. In this paper, a bilateral multi-issue negotiation approach is proposed in order to overcome the above prediction limitations and to improve the negotiation results. The goal of the proposed negotiation approach is to increase both agents’ utilities, which can be employed by both of them. In the proposed negotiation approach, Bayesian theory is employed to predict the opponent’s preference. The major contributions of the proposed approach are that (1) the proposed preference prediction algorithm does not require any previous negotiation data about the opponent to initialise the prediction. The prediction procedure is an online procedure and only based on the analysis of opponent’s counter-offers that are proposed in the on-going negotiation; and (2) the proposed approach has integrated a counter-offer proposition algorithm, which is capable of trading issues effectively based on the predicted preference of the opponent. Therefore, both agents can increase their utilities from the mutual beneficial offer. The rest of this paper is organised as follows. Section 2 presents the details of the proposed negotiation approach, including preference prediction and counter-offer proposition. Section 3 shows the experimental results of the proposed negotiation approach. Section 4 analyses this approach in a case study. Section 5 compares the results to some related work on multi-issue negotiation. Section 6 gives the conclusion and future work.

parameters defined in its concession strategy (see Section 2.3 for detail), then use its utility function to calculate an offer’s payoff. Based on the calculation result, the agent can decide whether to accept the offer. If the agent rejects the offer, the agent will try to predict its opponent’s preference (see Section 2.4 for detail) and propose a counter-offer based on the prediction result (see Section 2.5 for detail). The negotiation will end when the agent accepts the offer or its deadline is reached. The proposed negotiation approach also applies package deal procedure to propose offers in each negotiation round [23,24]. Package deal procedure means that agents treat all negotiation issues as a package (offer) and negotiate all issues simultaneously. By using the package deal procedure, agents can effectively trade-off negotiation issues and achieve a win–win agreement. The negotiation process in a negotiation round is depicted in Fig. 1. 2.2. The basic negotiation terms The proposed negotiation model partially employs the multi-issue negotiation model proposed by Faratin et al. [21]. 0 Let i represent one of the negotiation agent and i represent its opponent agent and j ðj 2 1; . . . ; nÞ is one of the issues that are negotiated between the two agents. Let xj ¼ ½minj ; maxj  be a value of issue j and minj ; maxj represent the lower bound and the upper bound of xj , respectively. Each agent has an evaluation function Eij : ½minj ; maxj  ! ½0; 1 that evaluates the value of issue j to a normalised value in-between 0 and 1. For example, a general and widely used evaluation function Eij for agent i on issue j can be defined as:

Eij ðxj Þ ¼

xj  minj : maxj  minj

ð1Þ

Agent i’s preference is represented by Pi ¼ fwij g, which contains a set of weighting wij ðj 2 1; . . . ; nÞ for each negotiation issue. The summation of all wij equals to 1. According to the above terms, an agent’s utility function can be defined by Eq. (2): n X

2. Negotiation approach with Bayesian-based preference predication

U i ðXti0 !i Þ ¼

This section presents the proposed negotiation approach in detail and is divided into five subsections. Section 2.1 introduces the basic negotiation model used in the proposed approach. Section 2.2 introduces the basic technical terms used in the proposed approach. Section 2.3 presents the detail of agents’ concession behaviour. Section 2.4 describes how to predict an opponent’s preference based on Bayesian theory. Section 2.5 introduces the procedure of issue trade-off and counter-offer proposition.

where Xti0 !i represents the offer proposed by opponent i to agent i at time t and xti0 !i ½j represents the value of issue j in the offer Xti0 !i . The calculation result of an agent’s utility function is also a normalised value between 0 and 1. Let timax represent the deadline for agent i to complete the

2.1. The basic negotiation model In the proposed negotiation approach, certain assumptions are made. First, both agents have target utilities and negotiation deadlines. Second, there is no dependency between negotiation issues. Third, both agents follow the concession-based negotiation strategy to decrease their target utilities [21]. The proposed negotiation approach uses Rubinstein’s alternating offer protocol as agents’ interaction rules during a negotiation [22]. More precisely, a negotiation process is divided into multiple rounds. During each negotiation round, after an agent receives an offer from its opponent, the agent will first check whether the current round has exceeded its negotiation deadline. If the deadline has not been reached yet, the agent will concede its target utility based on

wij Eij ðxti0 !i ½jÞ;

ð2Þ

j¼1 0

negotiation. Agent i also has a target utility V it ðV it ¼ ½0; 1Þ at time t, which is used to determine whether to accept an offer. As described in previous section, the negotiation protocol used in the proposed prediction approach is based on the Rubinstein’s alternating offer protocol. The formal procedure of our negotiation protocol is described as follows.  Step 1: At the beginning of a negotiation, after agent i receives 0 offer Xt1 i0 !i from its opponent i , agent i will first compare current time t with t imax . If t > t imax , the procedure will go to Step 2, while if t 6 t imax , the procedure will go to Step 3.  Step 2: Because the current negotiation time has already exceeded agent i’s negotiation deadline, agent i will terminate the negotiation and the negotiation fails.  Step 3: Because agent i still has time for further negotiation, agent i will first concede its target utility V it according to the concession strategy and then evaluate the offer Xt1 i0 !i by using its utility function U i ðXt1 i0 !i Þ. The calculation result will be used

110

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

Fig. 1. Negotiation process in a single negotiation round.

to compare with the target utility V it . If V it P U i ðXt1 i0 !i Þ, the proV it

i

ðXt1 i0 !i Þ,

t imax then 3: quits negotiation 4: else 5: 6: 7:

concedes V it

P calculates U ðXti0 !i Þ ¼ nj¼1 wij Eij ðxti0 !i ½jÞ if U i ðXti0 !i Þ P V it then accepts offer Xti0 !i

In Algorithm 1, agent i checks its deadline and decides whether to quit the negotiation (Lines 2–3). If the negotiation time does not exceed agent i’s deadline, agent i will concede its target utility and evaluate the offer to decide whether to accept it (Lines 5–8). If agent i does not accept the offer, agent i will predict its opponent 0 i ’s preference and then propose a counter-offer (Lines 10–11).

2.3. Agents’ concession behaviour During negotiation, autonomous agents usually follow certain strategies to propose offers. We assume both agents use concession-based negotiation strategies and the concession made by an agent is strongly related to negotiation time. As negotiation time increases, both agents must consider further concession on the negotiation issues [21,25,26]. Generally, agent i’s target utility is set to its maximum value at the beginning of a negotiation (usually equal to 1), when negotiation time reaches agent i’s deadline, the target utility must be decreased to the minimum value (usually equals to 0) that agent i can accept, which can be defined as:

( V it ¼

i

8: 9: else 0 10: predicts and update the estimation on opponent i ’s preference 11: proposes counter-offer Xti!i0 12: end if 13: end if

V imax

when t ¼ 0

V imin

when t ¼ t imax

ð3Þ

;

where V imax and V imin represent the maximum and minimum target utility of agent i, respectively. The agent’s concession algorithm can be defined by Eq. (4):

V it

¼

V imax



ðV imax



V imin Þ



t timax

!a ;

ð4Þ

where the value of a is used to change the concession strategy, which can be classified as: (1) when 0 < a < 1, agent i will make large concession at the beginning of the negotiation and small con-

111

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

cession when the negotiation approaches to the end; (2) when a ¼ 1, agent i will make a constant degree of concession through the whole negotiation; and (3) when a > 1, agent i will make small concession at beginning of the negotiation but increase the concession degree at latter rounds of the negotiation. The detail of the agent’s concession strategy is depicted in Fig. 2. 2.4. Bayesian-based preference prediction As described in previous sections, agents’ preferences are extremely important for issue trade-off in multi-issue negotiation. The purpose of issue trade-off is to propose an offer Xti!i0 that cannot only maximise agent i’s own utility U i ðXti!i0 Þ, but can also decrease 0

0

the loss of opponent i ’s utility U i ðXti!i0 Þ. For example, suppose that

0

1  Eij ðXti!i0 ½jÞ by agent i and opponent i ’s utility function can be estimated by agent i as: 0

U i ðXti0 !i Þ ¼

n X 0 wij ð1  Eij ðXti!i0 ½jÞÞ:

1

2

1

issues which called j and j . For agent i, issue j ’s weighting is 2

1

0

0.8 and issue j ’s weighting is 0.2, while for opponent i , issue j ’s 2

weighting is 0.1 and issue j ’s weighting is 0.9. During the negotiation, if agent i can propose an offer Xti!i0 that requests more utility 1

2

0

on j but concedes utility on j , then both agent i and opponent i can get more utilities from this offer and a mutual beneficial agreement will be achieved. 0 More formally, if agent i receives an offer Xt1 i0 !i from opponent i at round t  1 and agent i rejects this offer and proposes a counteroffer Xti!i0 at round t, the counter-offer Xti!i0 must fulfil following objectives to achieve a beneficial agreement for both agents.

(

Objective 1 : max U i ðXti!i0 Þ to V it 0

0

i t Objective 2 : min U i ðXt1 i0 !i Þ  U ðXi!i0 Þ

:

The meaning of Objective 1 is that when agent i proposes a mutually beneficial offer Xti!i0 , its utility gain from this offer (U i ðXti!i0 Þ) should be maximised to its current round’s target utility V it . The meaning of Objective 2 is that when agent i proposes a 0 mutually beneficial offer Xti!i0 , opponent i utility gain from this 0

offer (U i ðXti!i0 Þ) should be close to the utility gain from the offer 0

0

proposed by opponent i at previous round (U i ðXt1 i0 !i Þ), thus oppo0

nent i utility loss will be minimised. Apparently, in order to propose a mutual beneficial offer that 0 reach the above objectives, we need to know opponent i ’s utility 0

function U i ðXti0 !i Þ. According to Eq. (2), there are two unknown 0

0

0

parameters in U i ðXti0 !i Þ, which are opponent i ’s weighting wij on

ð5Þ

j¼1

0

in a negotiation between agent i and opponent i , there are two

0

0

each negotiation issue and opponent i ’s evaluation function Eij of each negotiation issues. In the proposed negotiation approach, we assume that most of the issues negotiated between agents are conflict issues. Conflict issues here mean that increasing the value of an issue will help agents to raise their utilities but decrease its opponent’s utility. For example, when two agents negotiate over the price for a service, the seller agent will be happy to increase the price while the buyer agent will not. Therefore, the 0 opponent i ’s evaluation function on issue j can be assumed as

0

In order to calculate the final unknown parameter wij in Eq. (5), Bayesian theory is employed. Usually, Bayesian theory is used to calculate the explicit probabilities for a hypothesis. In the Bayesian theory, there is a hypothesis space H, which contains a set of possible hypotheses and Bayesian rule is used to determine the most probable hypothesis among them [27]. The Bayesian rule can be defined as follows:

Bayesian Rule : PðhjDÞ ¼

PðDjhÞPðhÞ ; PðDÞ

ð6Þ

where h is one of the hypothesis in the hypothesis space H and D is the training dataset; PðhÞ is the prior probability of the hypothesis h; PðDÞ is the probability that the training dataset D will be observed given no knowledge about which hypothesis h holds; PðDjhÞ denotes the probability of observing dataset D given the condition that the hypothesis h holds. Finally, PðhjDÞ is the posterior probability, which represents the probability that hypothesis h holds given the observed training dataset D. It reflects the confidence that hypothesis h holds after the training dataset D has been seen. In the multi-issue negotiation field, the hypothesis space H can be used to represent all possible rankings of the negotiation issues of agent i, and the training dataset D are offers and counter-offers in the negotiation [28]. After each negotiation round, agent i must update the belief of each hypothesis h in the hypothesis space H according to the latest offer. Let Hw donate the possible ranking 0 of the negotiation issues of opponent i and hm ðhm 2 f1; . . . ; ngÞ represent one of the hypothesis (ranking of the issues) that belongs to hypothesis space Hw . The weights of negotiation issues can be normalised by Eq. (7) [28]:

wm j ¼ 2 

rm j nðn þ 1Þ

ð7Þ

;

where wm j represents the weighting of issue j in hypothesis hm and rm j donates the ranking of issue j in hypothesis hm . The ranking starts from the least important issue to the most important issue. Before Bayesian theory can be applied, a uniform distribution is assigned to the hypotheses in the hypothesis space Hw . More precisely, if there are n hypotheses in Hw , the prior probability of each hypothesis is assigned with 1n. During each round of 0

negotiation, when a new offer is received from opponent i , the Bayesian rule will be used to calculate the posterior probability of each hypothesis hm . The calculation is defined by Eq. (8):

PðXt0 jhm ÞPðhm Þ Pðhm jXti0 !i Þ ¼ Pn i !i t ; k¼1 PðXi0 !i jhk ÞPðhk Þ Fig. 2. Agent’s concession strategy.

ð8Þ

112

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

where Pðhm Þ represents the latest probability of hypothesis hm and Pðhm jXti0 !i Þ represents the posterior probability of hypothesis hm 0

given the condition that offer Xti0 !i is proposed by opponent i and received by agent i at time t. The only unknown parameter in Eq. (8) is the conditional probability PðXti0 !i jhm Þ. PðXti0 !i jhm Þ means that when the given 0

hypothesis hm is hold, the probability that opponent i offers Xti0 !i . We use Fig. 3 to demonstrate the calculation process of PðXti0 !i jhm Þ. 0

In Fig. 3, the solid line represents opponent i real concession 0 line and the square points on this solid line represent opponent i real target utility of each negotiation round. Besides, the dash lines 0 in Fig. 3 represent the estimated concession lines of opponent i 0 and the trigonal points on these dash lines are opponent i ’s estimated target utilities of each negotiation round. These estimated target utilities are calculated based on the preference hypotheses 0 0 of opponent i . For example, in Fig. 3, the opponent i estimated tar0 get utilities on dash line lh1 are calculated based on opponent i ’s 0 t preference hypothesis h1 . Term V hm represents opponent i ’s estimated target utility on time t based on hypothesis hm . In order to calculate the conditional probability PðXti0 !i jhm Þ, the 0

similarity between opponent i real concession line and agent i’s estimated concession lines needs to be analysed. A estimated con0 cession line that has the highest similarity to opponent i real concession line indicates the hypothesis used to calculate this 0 concession line is most close to opponent i ’s real preference. 0 Opponent i ’s real target utility in each negotiation round can be estimated by Eq. (4). In order to calculate the similarity between 0 opponent i real concession line and the estimated concession lines, the regression analysis is used. The non-linear correlation can be calculated by Eq. (9):

Pt0

0

0

ðV i  V i ÞðV t  V t Þ

t¼1 t hm hm ffi; chm ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

Pt0

0

0

i i t¼1 ðV t  V Þ

2P n t t¼1 ðV hm

 V thm Þ

ð9Þ

2

where chm represents the non-linear correlation for hypothesis hm ; t 0 0

0

represents current negotiation round, V it represents opponent i ’s 0

target utility at round t; V i represents the average value of oppo0

nent i ’s target utilities until round t 0 and V thm represents the average 0

value of opponent i ’s target utility, which is calculated based on preference hypothesis hm . After calculation of each Pðhm jXti0 !i Þ, agent i will use the calculation results to update the probability distribution of hypothesis space Hw . Finally, the hypothesis hm that has the maximum postermax ior probability is set as hm , which represents the most believable preference hypothesis in current negotiation round. The issues’ max weightings in hm will be used by agent i to trade off issues and 0 propose a counter-offer to opponent i . The detail of the preference prediction procedure is illustrated in Algorithm 2, which is mainly divided into four steps as follows. 0

 Step 1: Agent i uses Eq. (3) to update opponent i real target utility (Lines 1–2).  Step 2: If negotiation round is the first round, agent i will initialise the hypothesis space Hw (Lines 3–8).  Step 3: Agent i calculates the non-linear correlation of each estimated concession line by using Eq. (9) and apply the result to Bayesian rule (Eq. (8)) to calculate each hypothesis hm ’s posterior probability (Lines 9–15).  Step 4: Agent i chooses the hypothesis that has the maximum max posterior probability and set it as hm (Line 16–17).

Algorithm 2. Preference prediction 1: apply t to concession equation  a 0 0 0 0 t V it ¼ V imax  ðV imax  V imin Þ  i0 t max

0

update opponent i0 ’s target utility V it if t ¼ 0 then initialise hypothesis space Hw for all hm 2 Hw do assign probability 1n to pðhm Þ end For end if for all hm 2 Hw do 0 P j 10: apply hm and Xti0 !i to U i ðXti0 !i Þ ¼ nj¼1 r m ð1  Eij ðxti0 !i ½jÞÞ 2: 3: 4: 5: 6: 7: 8: 9:

calculate V thm

11:

Pt0 i0 i0 t ðV t V ÞðV hm V thm Þ t¼1 calculate chm ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Pt0 i0 i0 2 Pn t 2 t

12:

t¼1

13:

calculate

Pðhm jXti0 !i Þ

ðV t V Þ

t¼1

ðV hm V hm Þ

PðXti0 !i jhm ÞPðhm Þ

¼ Pn

k¼1

PðXti0 !i jhk ÞPðhk Þ

14: save result of Pðhm jXti0 !i Þ 15: end for 16: choose the maximum Pðhm jXti0 !i Þ max

17: set hm as hm

2.5. Counter-offer proposition As described above, the main purpose of preference prediction in our negotiation approach is to use the prediction result to trade offer issues and than propose beneficial offers for both agent i and 0 opponent i . According to Section 2.4, there are two objectives that agent i must try to reach during its counter-offer proposition, which are (1) maximise its owe utility and (2) minimise opponent 0 i ’s utility loss. More precisely, the proposition of the new counter-offer is based on the adjustment of the offer ðXt1 i0 !i Þ that was sent by oppo0

nent i at round t  1. In order to meet the first objective, agent i should start counter-offer utility increasing, which is to increase the new counter-offer’s utility to its next current target utility V itþ1 . Since it is assumed that the issues negotiated between agent 0

i and opponent i are conflict issues, thus every time agent i tries to 0 gain utility from an issue, opponent i will lose certain utility from 0 this issue. In order to minimise such a utility loss for opponent i (the second objective), agent i must choose an issue that has high 0 weighting for itself but low weighting for opponent i . For example, 0 assume that agent i and opponent i negotiate on three issues and their weighting of these issues are listed in Table 1. By comparing 1

0

agent i and opponent i ’s weightings, it is clearly that issue j is the most suitable issue for agent i to increase its counter-offer’s utility. 1

This is because if agent i uses issue j to increase 0.1 utility, it will 0 only cause opponent i to lose 0.067 utility. While if agent i uses 2

3

0

issue j or j to increase 0.1 utility, opponent i will lose 0.117 utility and 0.25 utility, respectively. In detail, before the counter-offer proposition starts, agent i needs to calculate the utility increasing ratio of each issue accord0 ing to the preference prediction result of opponent i , which can be calculated by Eq. (10):

gj ¼

wij 0

wij

;

ð10Þ

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

113

Fig. 3. Calculation of PðXti0 !i jhm Þ.

Table 1 Utility increase example 1.

Agent i weighting 0 Opponent i weighting

j1

j2

j3

0.6 0.4

0.3 0.35

0.1 0.25

issue j to gain more utility. Besides, the utility increasing for each issue in the offer has a boundary, which was defined by the initial ini value of issue j ðxini j Þ. The value of xj also depends on issue j’s evaluation function Eij ðxti!i0 ½jÞ. If the shape of Eij ðxti!i0 ½jÞ is monotone decreasing (see Fig. 4(a)), the value of xini j equals to issue j’s minimal acceptable value minj . Contrarily, If the shape of Eij ðxti!i0 ½jÞ is 0

where gj represents the utility increasing ratio of issue j and wij 0

max

represent opponent i weighting on issue j in hypothesis hm . After the calculation of the utility increasing ratio of each issue, agent i must choose the issue with the highest increasing ratio to start its counter-off’s utility increasing. The procedure to increase the utility of a particular issue depends on this issue’s evaluation function Eij ðxti!i0 ½jÞ. If the evaluation result of issue j is increasing with the increasing of xti!i0 ½j (see Fig. 4(b)), agent i must try to increase the value of issue j to gain more utility. On the contrary, if the evaluation result of issue j is increasing with the decreasing of xti!i0 ½j (see Fig. 4(a)), agent i must try to decrease the value of

monotone increasing (see Fig. 4(b)), the value of xini is equal to j issue j’s maximum acceptable value maxj . When a negotiation issue has reached its utility increasing boundary xini j , agent i must choose another issue according to the issue’s utility increasing ratio to purpose the counter-offer. The detail of the counter-offer proposition procedure is described in Algorithm 3, which is divided into four steps as follows.  Step 1: If the negotiation round is the first round, agent i will set all issues’ values to their initial values (Lines 1–5). Then, the procedure goes to Step 4.

Fig. 4. Utility increasing example 2.

114

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

 Step 2: If the negotiation is not the first round, agent i will ini½j (Lines 6–13). Then, tialise its counter-offer based on offer xt1 i0 !i the procedure goes to Step 3.  Step 3: After the counter-offer initialisation, agent i needs to increase the counter-offer’s utility to its next round’s target utility V itþ1 . The utility increasing starts from the issue that has the highest utility increasing ratio (Lines 16–20). During the utility increasing, agent i needs to check whether the value of the issue has reached its utility increasing boundary (Lines 22–34). The utility increasing procedure will stop when the counter-offer’s utility equals to V itþ1 . Then, the procedure goes to Step 4. 0

 Step 4: Agent i sends the counter-offer Xti!i0 to opponent i (Line 38).

Algorithm 3. Counter-offer Proposition 1: if t ¼ 0 then 2: for all xti!i0 ½j 2 Xti!i0 do 3:

set xti!i0 ½j ¼ xini j

4: end for 5: else if 0 < t 6 timax then 6: for all xti!i0 ½j 2 Xti!i0 do 7:

½j > maxj then if xt1 i0 !i

8:

set xti!i0 ½j ¼ maxj

9:

else if xt1 ½j < minj then i0 !i

10: 11: 12: 13: 14:

set xti!i0 ½j ¼ minj else ½j set xti!i0 ½j ¼ xt1 i0 !i end if end for

15:

calculate V itþ1 ¼ V imax  ðV imax  V imin Þ 

16:

for all wm j 2 hm

17:

max

calculate gj ¼



tþ1 t imax

a

do

3. Experiment In this section, experimental results are presented and the performance of our negotiation approach is analysed. The experiments focus primarily on testing the improvement in agents’ utility gain and negotiation time when employing the proposed prediction approach. The rest of this section is divided into two subsections. Section 3.1 describes the experimental settings and Section 3.2 shows the experimental results and performance analysis in three different experimental scenarios. 3.1. Experimental setting In the experiments, our negotiation approach was tested in three different scenarios, as shown in Table 2, which are: (1) both agents do not apply the preference prediction and the issue tradeoff during the negotiation, (2) only one of the negotiation agent applies the preference prediction and the issue trade-off and (3) both agents apply the preference prediction and the issue tradeoff during the negotiation. In Scenarios 1 and 2, when an agent does not apply the preference prediction and the issue trade-off approaches that were described in Algorithms 2 and 3, respectively, it will simply maximise its own utility without considering its opponent’s utility. More precisely, when a self-interested agent tries to propose offers, it will randomly choose issues to increase its utility to its target utility. For each experimental scenario, the negotiation issue’s setting and the agent’s initial parameters are same. An issue’s minimal value (minj ) is randomly selected from 0 to 500 and the maximum value (maxj ) is randomly selected from 1000 to 2000. The preference values (wj ) of all five negotiation issues are random numbers between 0 and 1. An agent’s minimum target utility (V min ) is randomly selected from 0 to 0.1 and its maximum target utility (V max ) is randomly selected from 0.9 to 1. The deadlines (tmax ) for both agents are set to 1000 and their concession strategies (a) are set to 1. The evaluation function (Eij ) used by agents is derived from Eq. (1), which can be defined as:

wij wm j

18: 19: 20:

end for choose the issue has the highest gj set this issue as k

21:

while U i ðXti!i0 Þ – V itþ1 do

Eij ðxti!i0 ½jÞ ¼

xti!i0 ½j  xres j res xini j  xj

ð11Þ

;

res where xj represents the value of issue j; xini represent agent j and xj

22:

increase xti!i0 ½k by d to make U i ðXti!i0 Þ ¼ V itþ1

i’s initial and reservation values on issue j, respectively. Like xini j , the

23:

if Eij ðxti!i0 ½kÞ is monotone increase then

also depends on the shape of Eij ðxti!i0 ½jÞ (see Section 2.5 value of xres j for detail). The detail of our experiment parameters are shown in Tables 3 and 4.

24: 25: 26: 27: 28:

d¼

ðV itþ1 U i ðXti0 !i ÞÞðmaxk mink Þ wki

þ mink  xti!i0 ½k

else d¼

ðV itþ1 U i ðXti0 !i ÞÞðmink maxk Þ wki

þ mink  xti!i0 ½k

end if if ðxti!i0 ½k þ dÞ exceed xini k then

29: set xti!i0 ½k ¼ xini k 30: choose the next highest gj issue 31: set this issue as k 32: else 33: set xti!i0 ½k ¼ xti!i0 ½k þ d 34: end if 35: update xti!i0 ½k in xti!i0 36: end while 37: end if 38: send new offer Xti!i0

3.2. Experimental results and analysis For each of the experimental scenarios, we tested an offer’s utility when the offer was accepted by agent 1 and agent 2. We also recorded the time needed by agents to accept an offer in the three scenarios. By comparing the experimental results of the three different scenarios, we can understand the overall performance of the

Table 2 Experimental scenarios. Scenario

Preference prediction

Issue trade-off

1 2 3

No agent Agent 1 Agent 1 & 2

No agent Agent 1 Agent 1 & 2

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120 Table 3 Parameters for both agent’s setting. Agent

V max

V min

tmax

a

Eij ðxti!i0 ½jÞ

Agent 1 & 2

½0:9; 1

½0; 0:1

1000

1

xt

½jxres j i!i0 xini xres j j

Table 4 Parameters for negotiation issue’s setting. Issue

maxj

minj

wj

Eij ðxti!i0 ½jÞ shape

Issue 1–6

½1000; 2000

½0; 500

[0, 1]

{monotone increase, monotone decrease}

preference prediction and issue trade-off algorithms in our negotiation approach. Furthermore, we tested our negotiation approach on different numbers of negotiation issues (from two issues to six issues), thus we could have a glimpse of how issue number could affect the performance of our negotiation approach. Since our negotiation approach has employed Bayesian theory to predict the opponent’s preference, the prediction results could be greatly affected by the opponent’s preference and the acceptance range on the negotiation issues. This could decrease the accuracy of our experimental results. In order to solve this problem, the experiment in each scenario was repeated 1000 times and the average results were recorded, thus our experimental results would be robustness and generality. Fig. 5(a) shows the average utility when an offer was accepted by agent 1 in the three experimental scenarios. Fig. 5(b) presents the average utility when agent 2 accepts the offer. Fig. 5(c) shows the total utility of agent 1 and agent 2.

115

In Fig. 5(a), we can see that when both agent 1 and agent 2 did not apply our negotiation approach, agent 1’s utilities are below 0.55 from 2-issue tests to 6-issue tests. Fig. 5(b) shows the similar result of agent 2’s utility tests. Such experimental results are not surprising, since when both self-interested agents try to maximise their own utilities, the negotiation usually will not end with a high utility agreement. In Scenario 2, after agent 1 applied our negotiation approach, we can see from Fig. 5(a) and (b) that not only agent 1’s utility had the obvious increment, but agent 2’s utility was also slightly increased. Such experimental results indicate that although only agent 1 applied our negotiation approach, the mutually beneficial offer proposed by agent 1 could still help both agents to gain more utilities from the final agreement. In Scenario 3, when both agents applied our negotiation approach, both agents’ utilities were increased significantly. Fig. 5(c) shows when the number of issues is between 3 and 6, the agents’ overall utility in Scenario 3 is higher than 1.52, while the overall utility in Scenario 2 is below 1.26. Based on the experimental results, we can conclude that our negotiation approach can help agents to predict their opponent’s preferences by analysing historical counter-offers, and to produce mutual beneficial offers based on the prediction results. In addition to agents utilities, we also recorded the negotiation time that agents needed to reach an agreement in the three experimental scenarios. From Fig. 6 we can see that agents in Scenario 3 used the least time to complete the negotiation, while agents in Scenario 1 used the most time to reach an agreement. The experimental results indicate that the preference prediction in our negotiation approach can help an agent to have a better understanding of its opponent’s preference, so as to efficiently propose a satisfying offer for its opponent before agents concede too much target utilities. Although the preference prediction will increase the computation time during each negotiation round, agents can

Fig. 5. Agents’ utility testing.

116

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

/it ¼

n X 0 jwij  wm j j;

ð12Þ

j¼1

where /it represents the preference difference between agent i’s 0

0

prediction result and opponent i ’s real preference in round t; wij 0

represents opponent i ’s real weighting of issue j and wm j represents

Fig. 6. The time needed when agent 1 and 2 reach an agreement.

still save more time by using the preference prediction to decrease the total negotiation rounds required to reach an agreement. 4. Case study In the previous section, we demonstrated that the proposed prediction approach cannot only help both negotiation agents to increase their average utility gain but also decrease their average negotiation time in 1000 experiments of the three scenarios. In order to better understand how our negotiation approach can affect agents behaviour, a case study is presented in this section. These case study has three purposes: (1) to analyse the relationship between the preference prediction and the utility of agents’ offers in each negotiation round, (2) to demonstrate that by using the result of preference prediction, agents can propose beneficial offers in each negotiation round and (3) to demonstrate that agents’ utility gain from the negotiation agreement is close to the Nash equilibrium. 4.1. Case study setting The case study was conducted in the three experimental scenarios. The parameters of negotiation issues are listed in Table 5 and the agents’ parameters are listed in Table 6. During each negotiation round, when agent 1 or agent 2 proposed an offer, this offer’s utility was calculated by both agents’ utility functions and the results were recorded. The difference between the predicted preference and the opponent agent’s real preference was also recorded during each negotiation round. The preference difference is calculated by Eq. (12):

Table 5 The settings for negotiation issues. Issue no.

MAX

MIN

Agent 1 preference

Agent 2 preference

1 2 3 4

1375.87 1730.39 1914.04 1678.17

290.79 37.39 12.71 362.75

0.177 0.208 0.471 0.141

0.517 0.160 0.132 0.190

Table 6 The settings for negotiation agents. Agent

V max

V min

t max

a

Agent 1 Agent 2

0.9 0.9

0.02 0.03

60 60

1 1

agent i’s predicted weighting of issue j in hypothesis hm . In addition to the agent offers’ utilities and the preference prediction difference in each negotiation round, a Nash equilibrium line is also calculated. Generally, the Nash equilibrium means that in a game each player knows the other players’ strategies and if each player chooses the best strategy by the consideration of other players’ strategies, the current set of strategy choices and the corresponding payoffs constitute a Nash equilibrium [29]. In a negotiation, if an offer reaches Nash equilibrium, this offer has the maximum total utility that the negotiation parties can gain considering about each other’s preferences. The purpose of recording this information is to see whether our prediction approach can help agents to reach the Nash equilibrium at the end of a negotiation. By analysing the change of both agents’ utility gains in each negotiation round before and after our prediction approach is employed, we can further discover what effects the proposed prediction approach can have on each negotiation agent. 4.2. Case study results and analysis Figs. 7(a), 8(a) and 9(a) show agent 1 and 2’s utility gains from the offer in each negotiation round in the three scenarios, respectively. The y-axis of these three figures represents an offer’s utility evaluated by agent 1, while the x-axis represents the offer’s utility evaluated by agent 2. In these charts, we can see three lines. The line with cross points was generated from agent 1’s offers and the line with circle points was generated from agent 2’s offers. The line with trigonal points was generated by Nash equilibrium offer in each negotiation round. Furthermore, the numbers marked on the utility line and Nash equilibrium line represent the negotiation round when the offer was generated. Besides, Figs. 7(b), 8(b) and 9(b) show the differences between the preference prediction results and the opponent’s real preference in each negotiation round in three scenarios, respectively. In these three figures, the lines with cross points were generated by agent 1’s prediction results and the lines with circle points were generated by agent 2’s prediction results. In Fig. 7(a), we can see that when both agents did not apply our negotiation approach, the utilities that agent 1 and agent 2 could gain from same offer were very different at the beginning of the negotiation. Using the data in the first negotiation round as an example, when agent 1 proposed an offer which had 0.9 utility, agent 2 could only gain 0.12 utility from this offer. In the second negotiation round, agent 2 proposed an offer with 0.88 utility, agent 1 could only gain 0.46 utility from this offer. Such an experimental result is not surprising, since at the beginning of the negotiation, agents’ target utilities are usually close to 1. This means agents need to try their best to gain utilities from negotiation issues, which leave only little benefit to their opponents. We can see from Fig. 7(a) that with the negotiation keeps going, both utility lines are extended in the same direction (the centre of the chart) and finally merged with each other at the 28th negotiation round. This is because both agents decreased their target utilities slightly in each negotiation round. Consequently, both agents could reach their target utilities so as to reach an agreement. Although agent 1 and agent 2 reached an agreement before their deadlines, we can see agent that 1 and agent 2 only gained 0.50 and 0.48 utilities from the final offer, respectively. Apparently, this offer is not close to the Nash equilibrium offer (the trigonal point

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

117

Fig. 7. Case study in Scenario 1.

marked as 50). This is mainly because agents did not know each other’s preferences, they randomly chose issues to increase their offers’ utilities, and this causes agent 1 and agent 2’s utility lines to have more fluctuations. For example, in the first three offers proposed by agent 1, we can see that agent 2’s utility gain from these three offers was continually increased from 0.12 to 0.31, which indicated that agent 1 might accidentally choose issues with a high utility increasing ratio when proposing these three offers (see Eq. (10) for detail). In round seven, however, agent 1 proposed an offer in which agent 2 only gained 0.15 utility, which indicates agent 1 might choose issues with a low utility increasing ratio when proposing the offer in this negotiation round. Obviously, the fluctuation of agent 1 and agent 2’s utility lines would significantly increase the merge time needed for these two lines, which caused the merge point (the agreement) to deviate from the Nash equilibrium point. In Fig. 8(a), we can see that after agent 1 applied our negotiation approach, the cross points on the utility line generated by agent 1’s offers have a greater horizontal range and less fluctuation compared with the point’s range in Fig. 7(a). Obviously, the larger horizontal range between each cross point indicates that the utility that agent 2 gained from agent 1’s offer had a big increment in each negotiation round. Such a result is due primarily to the fact that agent 1 predicted agent 2’s preference before generating a counter-offer, so agent 1 could trade off negotiation issues by using

Algorithm 3. More precisely, in Fig. 8(b), we can see that agent 1’s preference prediction difference is quite large (0.452) in the first negotiation round, which cause that agent 2 could only get 0.16 utility from agent 1’s first offer. However, in the next eight rounds, the prediction difference was continually decreased from 0.452 to 0.115. Correspondently, agent 1’s utility line has the most obvious horizontal increment during these eight negotiation rounds. After finding out the preference hypothesis that could have the smallest difference (0.115) compared to agent 2’s real preference, the horizontal increment of agent 1’s utility line slowed down and finally merged with agent 2’s utility line at round 19. Apparently, the merge point is much closer to the first Nash equilibrium point compared with the result in Scenario 1, which is mainly because agent 1’s utility line has no fluctuation after the preference prediction and issue trade-off. Through further analysis of Figs. 7(a) and 8(a), we can see that there is no significant difference between the utility lines generated by agent 2’s offers in these two figures. This result is quite normal, since in both Scenarios 1 and 2, agent 2 did not apply our negotiation approach, thus agent 2 only considered maximising its own utility. Fig. 9(a) shows the utility change after both agents applied our prediction approach. In Fig. 9(b), we can see that agent 2 used eight rounds to calculate the preference hypothesis which is most similar to agent 1’s real preference. As a result, agent 2’s utility line has a large vertical increment in the first eight rounds. Such change

118

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

Fig. 8. Agent 1 and 2’s utility in Scenario 2.

indicates that after agent 2 applied the prediction approach, agent 1 could get more utility from the offer proposed by agent 2 as well. By comparing the merging points of the utility lines from two agents in the three figures, we can also see that the merger point in Fig. 9(a) is closest to the upper right corner of the figure and this merge point is almost identical to the first Nash equilibrium offer. Apparently, if the merge point is closer to the upper right corner of the figure, agent 1 and agent 2’s utilities will be closer to 1 when the negotiation ends. This result indicates that both agents in Scenario 3 have the highest overall utility when they have reached the agreement. Based on the above case study, we can confirm that our prediction approach can help agents to regularly propose mutual beneficial offers based on the preference prediction results and to help agents to reach Nash equilibrium at the end of a negotiation.

5. Related work In this section, some related work to opponent prediction in automated negotiation is given and the difference between our approach and related work is also analysed. In [30], Zeng and Sycara proposed a sequential decision making model for multi-issue negotiation, called ’Bazaar’. This negotiation model used a Bayesian learning algorithm to predict the

opponent’s reservation value of certain negotiation issues. During the negotiation, once the agent receives information that comes from its opponent or the outside world, the agent will update its beliefs about the opponent agent’s reservation value. Our negotiation approach also employs Bayesian theory to model the opponent agent. Instead of trying to predict the opponent’s reservation value, however, our negotiation approach focuses on the preference prediction, which plays an important role in the proposition of mutually beneficial offers for multi-issue negotiations. In [20], Soo and Hung used a machine learning approach to predict opponent’s preferences. Their approach is based on Q-learning, which is a model-free reinforcement learning technology. In reinforcement learning, an agent has a set of actions. Each time an agent tries to interact with its environment by taking an action, the agent will receive a reward. Through analysing the reward, the agent will learn whether the action is good or bad. In agent negotiation, if an opponent has rejected an offer, then this offer is marked as a negative instance for the Q-learning algorithm, while a counter-offer proposed by the opponent gives a positive reward. However, their negotiation approach assumes that the opponent’s reservation price is public information, which is rarely or never the case in most automated negotiation scenarios. Our negotiation approach does not require any information of the opponent’s reservation price.

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

119

Fig. 9. Agent 1 and 2’s utility in Scenario 3.

In [19], Ros and Sierra introduced a simple statistical analysis theory to predict agents’ preferences. They considered that issues with fewer changes are more important than those with more changes during a negotiation. For example, if an agent considers delivery time as a high preference issue, the agent may try to keep it as stable as possible with small changes. A more comprehensive preference prediction approach based on statistical analysis was proposed by Coehoorn and Jennings [5], which called kernel density estimation (KDE). In order to make prediction, KDE needs an offline processing of any data available about agents’ previous negotiation for the provision of a particular service. Then according to the processing result, a probability density function over the opponent’s likely preferences for the various issues can be acquired. This function can be used by online learning to reflect new information from the ongoing negotiation. More precisely, the data that KDE uses for analyse are the offers and counter-offers in agents’ previous negotiations. In particular, KDE can be used to analyse the offers that are proposed at the beginning and end of a negotiation. The authors assumed that a relatively small change in an issue in the offer at the beginning of the negotiation might indicate that this issue is more important than other issues for the opponent. While at the end of the negotiation, a relatively large concession of an issue might indicate that this issue is important. By comparing

the difference in a negotiation issue between multiple offers, the opponent’s preference over this issue might be estimated. The advantage of their preference learning approach is the constant lookup of a prediction. However, one major problem is the requirement for offline analysis of prior negotiation data of their learning approach. For agents which encounter each other for the first time, this learning approach is not suitable. In our negotiation approach, an agent can predict its opponent’s preference based only on the offers in the on-going negotiation. In [31], Jomker and Robu proposed a model for integrative bilateral multi-issue negotiation in which all issues are negotiated simultaneously. In their negotiation model, both negotiation agents need to reveal partial preference information over some unimportant issues before a negotiation starts, but keep their preferences over important issues in secret. During the negotiation, a heuristic guessing approach is used by the agent to analyse the historical offers that have been proposed by its opponent. Based on the result of analysing the offer and the revealed preference information, an agent can predict its opponent’s complete preference information. The problem for such negotiation model, however, is that in most real world situations, opponents are not willing to reveal any information about their preferences. Our preference prediction algorithm does not require opponent agents to reveal any preference information before a negotiation.

120

J. Zhang et al. / Knowledge-Based Systems 84 (2015) 108–120

6. Conclusion and future work In this paper, an automated negotiation approach was presented to help agents to reach win–win agreements during bilateral multi-issue negotiations. The motivation for this approach was to produce mutually beneficial offers for agents through preference prediction and issue trade-off. Specifically, a set of hypothesises about the opponent’s preference is initialised before negotiation starts, and then Bayesian theory is used to analyse the counter-offer proposed by the opponent in each negotiation round and the most suitable hypothesis is chosen to help the agent to generate offers. The proposed negotiation approach was tested in different scenarios, and the experimental results have proved that our negotiation approach can help agents to reduce the time needed to reach an agreement. Agents which applied our negotiation approach could get more utilities when the negotiation ended. Our future work will focus on handling negotiations among one-to-many or many-to-many agents and improving our negotiation approach to handle situations in which an agent’s preference may change dynamically during the negotiation. Acknowledgements The authors would like to acknowledge the financial support from the Australian Research Council (ARC) Discovery Early Career Research Award (DECRA) under the grant DE140100007 for this project. The authors also would like to acknowledge the support from Dr. Madeleine Strong Cincotta, who provides help for language checking and improves the paper’s quality significantly. References [1] M. Wooldridge, Agent-based software engineering, IEE Proc.-Softw. 144 (1) (1997) 26–37. [2] M. Wooldridge, N.R. Jennings, Intelligent agents: theory and practice, Knowl. Eng. Rev. 10 (02) (1995) 115–152. [3] F. Zambonelli, A. Omicini, Challenges and research directions in agent-oriented software engineering, Auton. Agent. Multi-Agent Syst. 9 (3) (2004) 253–283. [4] T. Baarslag, K.V. Hindriks, Accepting optimally in automated negotiation with incomplete information, in: Proceedings of the 12th International Conference on Autonomous Agents and Multi-agent systems, 2013, pp. 715–722. [5] R.M. Coehoorn, N.R. Jennings, Learning on opponent’s preferences to make effective multi-issue negotiation trade-offs, in: Proceedings of the 6th International Conference on Electronic Commerce, 2004, pp. 59–68. [6] J. Gwak, K.M. Sim, Bayesian learning based negotiation agents for supporting negotiation with incomplete information, in: Proceedings of the International Multi-conference of Engineers and Computer Scientists, 2011, pp. 163–168. [7] H. Jazayeriy, M. Azmi-Murad, N. Sulaiman, N. Izura Udizir, The learning of an opponent’s approximate preferences in bilateral automated negotiation, J. Theor. Appl. Electron. Commer. Res. 6 (3) (2011) 65–84.

[8] C.-C. Huang, W.-Y. Liang, Y.-H. Lai, Y.-C. Lin, The agent-based negotiation process for B2C e-commerce, Expert Syst. Appl. 37 (1) (2010) 348–359. [9] I. Rahwan, R. Kowalczyk, H.H. Pham, Intelligent agents for automated one-tomany e-commerce negotiation, in: Australian Computer Science Communications, vol. 24, 2002, pp. 197–204. [10] S. Son, K.M. Sim, A price-and-time-slot-negotiation mechanism for cloud service reservations, IEEE Trans. Syst. Man Cybern. Part B: Cybern. 42 (3) (2012) 713–728. [11] J. Yan, R. Kowalczyk, J. Lin, M.B. Chhetri, S.K. Goh, J. Zhang, Autonomous service level agreement negotiation for service composition provision, Future Gener. Comput. Syst. 23 (6) (2007) 748–759. [12] S.-S. Leu, P.V.H. Son, P.T.H. Nhung, Hybrid bayesian fuzzy-game model for improving the negotiation effectiveness of construction material procurement, J. Comput. Civ. Eng, 2014. [13] S.-S. Leu, P.V.H. Son, P.T.H. Nhung, Optimize negotiation price in construction procurement using bayesian fuzzy game model, KSCE J. Civ. Eng. (2014) 1–7. [14] S. Chen, G. Weiss, An intelligent agent for bilateral negotiation with unknown opponents in continuous-time domains, ACM Trans. Auton. Adapt. Syst. (TAAS) 9 (3) (2014) 1–24. [15] S. Chen, G. Weiss, An approach to complex agent-based negotiations via effectively modelling unknown opponents, Expert Syst. Appl. 42 (5) (2015) 2287–2304. [16] S. Kraus, Negotiation and cooperation in multi-agent environments, Artif. Intell. 94 (1) (1997) 79–97. [17] L. Pan, X. Luo, X. Meng, C. Miao, M. He, X. Guo, A two-stage win–win multiattribute negotiation model: optimization and then concession, Comput. Intell. 29 (4) (2013) 577–626. [18] R.M. Coehoorn, N.R. Jennings, Learning on opponent’s preferences to make effective multi-issue negotiation trade-offs, in: Proceedings of the 6th International Conference on Electronic Commerce, 2004, pp. 59–68. [19] R. Ros, C. Sierra, A negotiation meta strategy combining trade-off and concession moves, Auton. Agent. Multi-Agent Syst. 12 (2) (2006) 163–181. [20] V.-W. Soo, C.-A. Hung, On-line incremental learning in bilateral multi-issue negotiation, in: Proceedings of the first International Joint Conference on Autonomous Agents and Multiagent Systems, 2002, pp. 314–315. [21] P. Faratin, C. Sierra, N.R. Jennings, Negotiation decision functions for autonomous agents, Robot. Auton. Syst. 24 (3) (1998) 159–182. [22] A. Rubinstein, Perfect equilibrium in a bargaining model, Econometrica 50 (1) (1982) 97–109. [23] S.S. Fatima, M. Wooldridge, N.R. Jennings, Multi-issue negotiation with deadlines., J. Artif. Intell. Res. 27 (2006) 381–417. [24] R. Inderst, Multi-issue bargaining with endogenous agenda, Games Econ. Behav. 30 (1) (2000) 64–82. [25] T. Baarslag, K. Hindriks, C. Jonker, Effective acceptance conditions in real-time automated negotiation, Decis. Support Syst. 60 (0) (2014) 68–77. [26] G.C. Silaghi, L.D. S ß erban, C.M. Litan, A time-constrained SLA negotiation strategy in competitive computational grids, Future Gener. Comput. Syst. 28 (8) (2012) 1303–1315. [27] K. Korb, A. Nicholson, Bayesian Artificial Intelligence, Computer and Information Science, Chapman & Hall/CRC, 2004. [28] K. Hindriks, D. Tykhonov, Opponent modelling in automated multi-issue negotiation using bayesian learning, in: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 1, 2008, pp. 331–338. [29] J.F. Nash Jr., The bargaining problem, Econom.: J. Econom. Soc. (1950) 155– 162. [30] D. Zeng, K. Sycara, Bayesian learning in negotiation, Int. J. Hum. Comput. Stud. 48 (1) (1998) 125–141. [31] C. Jonker, V. Robu, Automated multi-attribute negotiation with efficient use of incomplete preference information, in: Proceedings of the Third International Conference on Autonomous Agents and Multi-Agent Systems, vol. 3, 2004, pp. 1054–1061.