Two solution methods for dynamic game in reservoir operation

Advances in Water Resources 33 (2010) 752–761

Contents lists available at ScienceDirect

Advances in Water Resources j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / a d v wa t r e s

Two solution methods for dynamic game in reservoir operation Mehran Homayoun-far a, Arman Ganji a,⁎, Davar Khalili a, Jeff Harris b a b

College of Agriculture, Shiraz University, Iran School of Geography and Earth Sciences, McMaster University, Canada

a r t i c l e

i n f o

Article history: Received 12 March 2009 Received in revised form 31 March 2010 Accepted 8 April 2010 Available online 4 May 2010 Keywords: Water resources management Conflict resolution Ricatti equations Collocation Water allocation

a b s t r a c t During the last decade, a number of models have been developed to consider the conflict in dynamic reservoir operation. Most of these models are discrete dynamic models which are developed based on game theory. In this study, a continuous model of dynamic game and its corresponding solutions are developed for reservoir operation. Two solution methods are used to solve the model of continuous dynamic game, namely the Ricatti equations and collocation methods. The Ricatti equations method is a closed form solution, requiring less computational efforts compared with discrete models. The collocation solution method applies Newton's method or a quasi-Newton method to find the problem solution. These approaches are able to generate operating policies for dynamic reservoir operation. The Zayandeh-Rud river basin in central Iran is used as a case study and the results are compared with alternative water allocation models. The results show that the proposed solution methods are quite capable of providing appropriate reservoir operating policies, while requiring rather short computational times due to continuous formulation of state and decision variables. Reliability indices are used to compare the overall performance of the proposed models. Based on the results from this study, the collocation method leads to improved values of the reliability indices for total reservoir system and utility satisfaction of water users, compared to the Ricatti equations method. This is attributed to the flexible structure of the collocation model. When compared to alternative water allocation models, lower values of reliability indices are achieved by the collocation method. © 2010 Elsevier Ltd. All rights reserved.

1. Introduction Difficulties of classical optimization techniques [3,26,62] to analyze conflicting issues in allocating water for different objectives have encouraged a number of investigators to use multiple objective approaches such as ε-constraint method [13,45], goal programming [16,37], and compromise programming [58,65]. Other similar models which are also compatible with conflict resolution problems have been reported by a number of researchers [15,24,31,60]. Game theory as another approach has been extensively used for conflict resolution in water resources management. Carraro et al. [5] and Madani [39] reviewed the applicability of game theory to water resources management and conflict resolution through a series of noncooperative water resource games. Zara et al. [64] and Parrachino et al. [53] provided a similar review of the application of cooperative game theory to water resources and environmental issues. Conflict problems generally have a dynamic nature, whereby most of the above-mentioned references provide static solutions. Dynamic games have been used to analyze transboundary competition for groundwater [49,51,55], air pollution and global warming [2,9,41], and transboundary fishery management [8,28,33]. Application of

⁎ Corresponding author. E-mail address: [email protected] (A. Ganji). 0309-1708/$ – see front matter © 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.advwatres.2010.04.001

dynamic game models to water resources management has been quite extensive [6,15,18,20,29,32,46,48,50,51,54–56,59,61]. Negri [50], Dixon [6] and Prorencher and Burt [54] developed dynamic game theoretic models of groundwater use in a restricted access setting. Nakao [48] used dynamic game theory to the problem of transboundary competition for groundwater involving El Paso, Texas, and Ciudad Juarez, Mexico. Both cities obtain a substantial portion of their water supply from a common aquifer that lies beneath the international border. Saak and Petersom [56] developed a dynamic game of water extraction in a two-cell aquifer under incomplete information. Msangi [46] demonstrated the adaptability of dynamic game theory in solving groundwater conflict problems. Kerachian and Karamouz [29] used a stochastic conflict resolution model for water quantity management in reservoir-river basin systems. The expected value of the Nash product is considered as the objective function of the model. Xie et al. [61] developed a two-stage dynamic game for establishing the optimal allocation of water resources. Kong et al. [32] used a two-stage dynamic game to analyze the effectiveness of government organization in optimizing the allocation of water resources. More recently, Shirangi et al. [59] used the Young conflict resolution theory to solve a dynamic game problem related to water quality and quantity. Ganji et al. [18,20] developed a discrete stochastic dynamic Nash game model to analyze competition on water consumption in a reservoir downstream. A discrete Fuzzy variation of this model was also developed by Ganji et al. [21]. These last three models require a

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761

substantial computational effort to derive their solutions. This is a direct result of discrete formulation of the state variable and state transition equations. As an alternative to discrete type modeling, models with continuous state variables can be used. A well-known continuous solution methodology for dynamic games is Markov Perfect equilibrium (MPE), which can be used for the reservoir operation. A Markov perfect equilibrium (MPE) is defined as a profile of Markov strategies that yields Nash equilibrium in every proper sub-game [17,36]. Few studies have provided empirical estimates of equilibrium values, in part, because solving dynamic games using either a feedback solution or a Markov perfect solution is a challenging task. Ligon and Narain [35] describe three solution methods for obtaining MPE. They first discuss the classical approach, in which the Euler equation is used to derive MPE. The second approach solves a set of algebraic Ricatti equations in which they assume that equilibrium policies are linear and value functions are quadratic in state variable. In the third method, called a pseudo planning problem, the optima corresponding to MPE are sought. In this study, two methods (e.g. Ricatti equations and collocation methods) are presented to solve a dynamic game of reservoir system. The system's state variables and transition equation are considered continuous variables. The modeling of the game problem is mathematically solved by Ricatti equations and collocation methods. These approaches are able to generate operating policies for dynamic reservoir operation. As a comparison of these solution methods with discrete models (e.g., stochastic dynamic Nash game with perfect information (PSDNG) by Ganji et al. [18]), state variables, inflow and release are not discretized in the Ricatti equations method. The Ricatti equations method is a closed form solution based on a linearquadratic value function and requires less computational efforts when compared with discrete models. The second model based on the collocation solution method, only considers the storage state variable for discretization and applies Newton's method or a quasi-Newton method to find the solution to the problem. Table 1 summarizes the major differences among the collocation solution method, Ricatti equations method, and the alternative dynamic programming (DP) based solution methods. In order to evaluate the effectiveness of the proposed methods, they are applied to develop operating policies for the Zayandeh-Rud reservoir in the central part of Iran and the results are compared with classical DP and PSDNG models [20]. In the following sections, the theoretical concepts underlying the proposed methods are presented. The two solution methods (Ricatti equations and collocation methods) are outlined and the concept of reliability indices is then introduced. Finally, the results are presented and discussed. 2. Theoretical concepts 2.1. Conflict on water consumption: problem statement Conflict on water consumption in a river basin under a reservoir system can be stated based on the following mathematical frame-

Solution methods

Properties State variable

Decision variable

Method of solution

Computational effort

DP-based

Discretized variable Continuous variable Discretized variable

Discretized variable Continuous variable Continuous variable

Dynamic programming Closed form solution Quasi-Newton solution

High [12]

Collocation

work, as a static water allocation problem case. Assume a vector of allocated water xt = {xtj, j = 1,..., n} to n different water users and their corresponding utilities of allocated water, which are represented using a vector of Us(xtj). The task of the water manager is assumed to be maximizing the overall water users' utilities (Us), from the water allocation vector (xt) subject to water availability, which can be formulated as (the social planner's solution):
 1 n Vt ðSt Þ = max Us xt ; :::; xt s:t: x1t ;:::;xnt

n

j

∑ xt ≤Rt ðSt ; It Þ

ð1Þ

j=1

where St is reservoir storage as the state of system, It is inflow discharge at present time (t), and Vt(St) is an indirect utility function at time t, representing the maximized level of utility for any given current state of storage (St), and Rt(St, It) is the release, determined as a function of St and It values. For simplicity, Rt(St, It) is shown as Rt hereafter. Ganji et al. [18,20,21] discussed application of the utility function in water allocation problems. For the case of allocation along an infinite time horizon, the life time utility can be represented by:
 ∞ 1 n ∑ γUs xt ; :::; xt ; Rt ; t 0≤γ≤1

ð2Þ

t =1

where t is the time or stage of the system, γ is the discount factor, and US (x1t , …, xnt , Rt; t) changes during the lifetime due to reservoir storage (state variable) variations. According to this formulation, the present value of lifetime utility from water consumption should be maximized for each water user. The water users' value functions at each stage t can be split into two parts as follows: n
 1 n Vt ðSt Þ = maxx1 ;:::;xn Us xt ; :::; xt ; Rt ; t + γVt t

t

 + 1

St

+ 1

o :

ð3Þ

Considering the first component of Eq. (3) at any given time, some utility is derived from immediate consumption. The second component in the formulation is the present value of optimal lifetime consumption starting one period from the present. Eq. (3), also called the Bellman equation, holds at any given time and so the problem becomes time-independent. The evolution of state variable over time can be represented by: g ðSt ; It ; εt Þ≡St

+ 1

= St + ðIt + εt Þ−Rt

ð4Þ

where g(St, It, εt) is the transition equation function and εt is an exogenous random shock. Taking into account this statement of the problem for reservoir operation, an appropriate solution method exists in the literature, called stochastic dynamic Nash game with perfect information (PSDNG) [18,20,21]. This solution method is a discrete solution of dynamic game, requiring intensive computational efforts. In this research, two new solution methods are developed based on the Ricatti equations (continuous) and collocation methods (semi-continuous) which have not yet been presented as a solution for the allocation of water in reservoir operations. These two solution methods are discussed in the following sections. 2.2. Scalar Ricatti equations method: one player and two-player asymmetric game

Table 1 Comparing different solution methods of dynamic game.

Ricatti

753

Low Medium

The Ricatti equations are ordinary differential equations that are also used in linear-quadratic (LQ) control of hydraulic systems, e.g., adaptive control of a water supply system [43], and hydro turbine speed control [30]. Ricatti equations method is also introduced as a solution technique to dynamic programming models that are characterized by a quadratic objective function and linear transition equation (linear-quadratic problems). Miranda and Fackler [42] presented the advantages and disadvantages of linear-quadratic formulation for the solution of dynamic models, i.e., the policy

754

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761

function derived as the solution of a dynamic programming is linear in the state variables. The following discusses Ricatti equations as applied to water allocation problems. As the first step, a linearquadratic problem can be considered for a single player game as (also see Fig. 1): n
  1 Vt ðSt Þ = maxx1 Us xt ; Rt ; t + γEst Vt t

o :

+ 1 ðg ðSt ; It ; εt ÞÞ

ð5Þ

According to Ganji et al. [18,21], the value function, Vt(St), can be presented as a quadratic function of the state variable (reservoir storage). As a result, Eq. (5) can be written as: n
 h io 2 1 2 αSt + βSt + η = maxx1 Us xt ; Rt ; t + γEst αgðSt ; It ; εt Þ + βgðSt ; It ; εt Þ + η t

ð6Þ where α, β, and η are set of unknown parameters of the value function. As stated by Miranda and Fackler [42], the policy function depends only on the mean of the stochastic state variable in the linear-quadratic optimal control model. As a result of replacing the mean of the stochastic state variable, εt, the policy function of the stochastic linear-quadratic problem is the same as deterministic ones and can be solved in the same manner. To obtain the Ricatti equations, as indicated by Lockwood [36] for a Bellman equation, the first derivation of the right-hand side of Eq. (6) with respect to the decision variable (x1t ) is determined (second step, Fig. 1). Setting the first derivation equal to zero and solving the equation for x1t yields: 1 xt *

n
 o 1 2 = arg max Us xt ; Rt ; t + γEst ½αg ðSt ; It ; εt Þ + βg ðSt ; It ; εt Þ + ηÞ ð7Þ

1⁎ where x1⁎ t is the optimal allocated water to the user. Substituting xt back into the right-hand side of Eq. (6) creates three pairs of equations, called Ricatti equations. Solving the three pairs of equations simultaneously generates as many as three sets of potential solution values for parameters α, β, and η. Taking each possible solution and substituting it into Eq. (7) produces three policy functions. These policy functions describe the optimal value of x1⁎ t as a function of the reservoir storage variable. Then, it should be determined which of these alternative policy functions is the appropriate one (final step, Fig. 1). For example, an equation can be ruled out, if it indicates that along the optimal path, the amount of allocation/release is negatively related to reservoir storage. Obviously, such equations would not be economically rational, because the optimal amount of water allocation would not decrease as the reservoir storage increases. Furthermore, negative signs of intercept and slope would be considered as reasons to rule out a policy equation. Then, the selected policy function is used to simulate the release and storage during a long period of time and evaluated using the reliability indices, discussed in Section 4.2. Further discussions on the Ricatti equations method are presented in the flowchart of Fig. 1. The Ricatti equation method can also be extended for two or more players. In the case of two-player asymmetric game, two individual Bellman equations are considered as:

 2 1 2* α1 St + β1 St + η1 = maxx1 Us xt ; xt ; Rt ; t t

f

h i 2 + γEst α1 g ðSt ; It ; εt Þ + β1 g ðSt ; It ; εt Þ + η1

g ð8Þ

 2 1* 2 α2 St + β2 St + η2 = maxx2 Us xt ; xt ; Rt ; t t

f

h i 2 + γEst α2 g ðSt ; It ; εt Þ + β2 g ðSt ; It ; εt Þ + η2

g ð9Þ

2⁎ where x1⁎ t and xt are the optimal water allocations to players 1 and 2, respectively. Solution of this asymmetric game requires determination of the first derivation of the right-hand side of Eqs. (8) and (9) (called the reaction function equations) with respect to the decision variables x1t and x2t . Then, the reaction function equations are solved 2⁎ simultaneously by substituting x1⁎ t and xt on the right-hand side of Eqs. (8) and (9), leading to six sets of Ricatti equations. These six sets of possible solutions must be examined for appropriateness, considering the resulted policy functions discussed earlier in this section.

2.3. Collocation method

Fig. 1. Flowchart of Ricatti equations method.

The collocation method is a special case of the so-called weightedresidual methods, commonly used in computational physics for solving partial differential equations [14]. This method leads to very simple solutions with minimal computational effort. The collocation method is widely used in chemical engineering [7] for model reduction of distributed parameter models. This approach has also been investigated very much within the context of hydraulic systems [1,7,10,19,23]. Alam and Bhuiyan [1] used a combination of collocation and a finite-element method to solve a problem of dam-break flow. Dulhoste et al. [10] have shown how a three-point collocation model can be used to design a nonlinear controller based on a dynamic input–output linearization technique. Dulhoste et al. [11] developed a nonlinear control of open-channel water flow dynamics via a one-dimensional collocation control model for irrigation canals or dam-river systems. The linear-quadratic (LQ) approximation can be used as a numerical method to resolve the previously stated water conflicts.

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761

As an alternative, the collocation method (developed in the physical sciences) is a generally useful technique that is flexible, accurate, and numerically efficient [11]. The modeling steps (Fig. 2) are discussed in the following section. Consider Bellman Eq. (5) where the state space (reservoir storage) is a bounded interval of the dead storage and reservoir capacity (minimum and maximum, respectively), and that the actions are either discrete or continuous and subject to simple predefined bounds (minimum and maximum releases). To compute an approximate solution using collocation methods, the value function approximant should be written as a linear combination of n known basis functions (φ1, φ2,..., φm) on St (storage as the state variable) with undetermined coefficients as (see steps 2 and 3): m

Vt ðSt Þ≈∑j = 1 cj φj ðSt Þ

ð10Þ

where cj is the basis function coefficient. Then, the value function on the left hand side of Eq. (5) and the second term on the right-hand side are replaced by Eq. (10). Considering the fixed basis function coefficients, the value function is required to satisfy Eq. (5) at m

755

collocation nodes (S1t , S2t ,..., Sm t ) which leads to a system of m nonlinear equations in m unknowns as:

 m i 1 ∑j = 1 cj φj St = maxx1 Us xt ; Rt ; t t

f

h

m = 1 c j φj

+ γEst ∑j



i i g St ; It ; εt

g

i = 1; :::; m: ð11Þ

The system of equations in Eq. (11) is expressed in vector form as: Φc = vðcÞ

ð12Þ

where ΦC is m × m collocation matrix and v is the value function, and their elements are represented as follows:
 i ð13Þ Φij = φj St n
h

io  1 m i : ð14Þ vi ðcÞ = maxx1 Us xt ; Rt ; t + γEst ∑j = 1 cj φj g St ; It ; εt t

This set of equations is called the collocation equations, and can be solved with any nonlinear equation solution method, i.e., Newton's

Fig. 2. Flowchart of collocation solution method.

756

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761

method or a quasi-Newton method (step 8, Fig. 2). For this propose, collocation equations and their derivatives are evaluated (step 6), considering an initial basis function coefficients (ci) and initial vector of actions (xit). Next, the Envelope Theorem is applied to solve the optimization problem in the definition of vi(c) (step 7, Fig. 2) and finding the optimal vector of actions (xi⁎ t ) for each collocation node. Considering the updated optimal values on the right-hand side of Eq. (14), Newton's method is used to find the best basis function coefficients vector (ci) (step 8, Fig. 2). This procedure is continued up to a point when basis function coefficients remain constant for two subsequent iterations. For this propose, a predefined small value of tolerance is considered (see Tol in Fig. 2). The choice of basis-node scheme is a function of the curvature of the value function. The larger the number of basis functions and collocation nodes, the greater the computational burden, so the researcher will want to experiment with various basis-node schemes and dimensions of the problem to render it computationally efficient. The collocation method is not limited to the first- and second-degree approximations, compared to Ricatti equations, which is presented by LQ approximation. Furthermore, in the presence of non-differentiable function LQ does not work. In addition, in spite of LQ related problems, consideration of constraints on the action variables in the collocation method would be possible. This is especially important in the case of reservoir operation, which typically requires several constraints on decision variables. 3. Case study Zayandeh-Rud river basin with an area of 4,200 square kilometers is one of the major river basins in Iran. The Zayandeh-Rud riverreservoir system provides benefits for approximately three million people, primarily in the city of Isfahan (a major tourist attraction), in the form of water for domestic purposes as well as providing water for industrial and agricultural sectors, hydropower, recreation, and instream flow. Zayandeh-Rud dam with a maximum capacity of 1,460 million cubic meters, is located west of the city of Isfahan. The mean monthly flow of the river is 82.6 million cubic meters with a monthly standard deviation of 87 million cubic meters. Detailed information on Zayandeh-Rud river basin can be found in a number of research reports of the IAERI-EARC-IWMI collaborative projects (see MurrayRust et al. [47] and Sally et al. [57]), that were fully funded by the Government of the Islamic Republic of Iran. The main purpose of the project is to foster integrated approaches to managing water resources at basin, irrigation system and farm levels, thereby contributing to promoting and sustaining agriculture in the country. The project used the Zayandeh-Rud river basin as a pilot study site. Zayandeh-Rud is also one of the river basins in Iran which has been widely studied (e.g., Mousavi et al. [44], Zahraie and Hosseini [63], Madani and Marino [40]). Mousavi et al. [44] applied a fuzzy model to the Zayandeh-Rud river-reservoir system to show the robustness of their proposed model with respect to the type of discretization scheme used in calculating the transition probabilities. Zahraie and Hosseini [63] developed a genetic algorithm (GA) optimization model for reservoir operation optimization considering variations in water demands. They evaluated the efficiency of their proposed model based on the long-term operation simulation of Zayandeh-Rud Reservoir. Madani and Marino [40] developed the Zayandeh-Rud Watershed Management and Sustainability Model (ZRW-MSM) within a system dynamics framework to comprehend the dynamic and interrelated characteristics of the system. In this study, the available 27 years of monthly data on streamflow is used for reservoir optimization. In the present study, two water user groups (agriculture, and other users including industrial and domestic) are considered. These water users are representatives of different consumers. Also an additional agency, the reservoir operator, is included in this study. The utilities of water users and the resource manager (reservoir operator) are

developed based on the works by Ganji et al. [20,18]. Presently, the Zayandeh-Rud reservoir is operated by the Department of Water Supply and frequently meets problems of water supply deficit during summer and provides much surplus during winter. It has resulted in major conflicts between the Department of Water Supply and water users' agencies. 4. Results 4.1. Utility function development To develop utility functions, questionnaires were prepared and sent to experts and decision makers familiar with the water issues in the Zayandeh-Rud river basin. Twenty five questionnaires were provided considering the guidelines by Dyer et al. [12]. The questionnaire forms were used to qualify the preference points. Evaluators and experts were asked for possible ranges of their satisfied utilities in the case of meeting several levels of water demands in different months, their authority and points of disagreement. The responses to the 25 questionnaires were used to develop the utility functions. As a basis for aggregating the responses, it was assumed that the water users' judgments would be influenced primarily by the particular goal area under consideration and by his satisfaction level of his corresponding agencies. Then, utility functions are approximated by quadratic equations over the specific range of utilities. Eqs. (15), (16) and (17) show the resulted utility functions for the reservoir operator (Us(St)), agriculture (Us(x1t )), and other users (Us(x2t )) including industrial and domestic users.

2

Us ðSt Þ = −3:25St + 3:41St + 0:005 1; 400≥St ≥150 0 otherwise

ð15Þ

8

 < U x1 = −7:06e−7 x1 2 + 0:0028x1 −1:71 3; 149:1≥x1 ≥767:8 t t t t s : 0 otherwise ð16Þ 8

2 < U x2 = −9:6e−5 x2 + 0:071x2 −12:13 470≥x2 ≥269 t t t t s : ð17Þ : 0 otherwise

4.2. Reliability indices Reliability is the ability of a system or component of the system to perform its required functions under stated conditions for a specified period of time [25]. Reliability is widely used in performance analysis of water resource systems [4,19,22,27,34,52]. According to Loucks et al. [38], the reliability of any water resources system is defined as the number of times that the system works at a satisfactory state over the planning period. In the context of reliability, it represents a kind of predictive uncertainty or randomness in system performance. The concept of reliability is used in this research to compare the overall performance of the developed models for water allocation based on decisions made by the reservoir operator as well as the water users. To compare the capability and efficiency of different models, the occurrence and volumetric reliabilities of reservoir system are used. Total occurrence storage reliability is defined as:  The number of failures in design period T Rn−st ¼ 1− × 100 The length of design period ðyearsÞ

ð18Þ

where, RTn − st is the overall storage reliability of the reservoir system. Failure is defined as violating the storage limit bounds (maximum and minimum allowable storage values).

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761

Occurrence allocation reliability of the reservoir system (RTn − all) is the reliability of not violating the total required allocation of water users in the study area. It is defined the same as the occurrence storage reliability of Eq. (18). In this case, failure occurs when the allocation is below the demand. The volumetric reliability of reservoir system for water allocation (total reliability) is: T

Rv−all =

 100 nyear yearly supplied water ∑ nyear i = 1 yearly demand

ð19Þ

where, nyear is the number of years for the planning horizon. Shortfall or the overflow of reservoir storage water over the reservoir planning horizon, while not being directly measured, can be used as an index of reservoir volumetric reliability: Rv−st =

Total storage shortfall or overflow : Total available water into the reservoir system during the planning horizon

ð20Þ The value of Rv− st changes from zero to values larger than 1 and it may become 2 or even 3. The zero value indicates that failure did not occur, resulting in a system volumetric reliability of 100%. Values larger than 1 indicate that the sum of volume of failures is more than the total available water in the reservoir system during the planning horizon. Total long-term average of utility satisfaction (US) is developed as a new reservoir system reliability index in this paper. The US index represents the level of approaching the maximum achievable utility value as a criterion for possibility of agreement (equilibrium point). The possibility of agreement decreases due to the associated low mutual utility satisfaction, and social and political influences. Yearly values of US are defined as the multiplication value of water users and operator utilities. 4.3. One player game solution: Ricatti equations method Capability of the DP model to find the best solution for dynamic reservoir operation [38], and dynamic game solution [20] was previously discussed. The one player game model has a similar structure to the DP model. In this paper, the Ricatti equations method is used to solve a one player dynamic game and the results are compared with the DP model to evaluate its capabilities. As the first step, the results of the Ricatti equations method are presented for a one player game. The short term utility function (Eq. (21)) is considered as function of the state variable (storage), as previously discussed, for the Zayandeh-Rud river basin: 2

Us ðSt Þ = −3:25St + 3:41St + 0:005:

ð21Þ

Also, the transition equation function, as stated in Eq. (4), is the yearly reservoir mass balance equation. By substituting the short term utility function and system transition equation into the Bellman Eq. (5) and solving the Ricatti equations, as will be discussed later, three sets of unknown parameters (α, β, η) are determined. Next, the yearly reservoir operating rule (policy function) is derived, based on the first order derivation of the right-hand side of Eq. (6) as:
 ∂Us x1t ; Rt ; t ∂x

757

among these solutions (see Eq. (23)), considering the rationality of resulted policy functions, as discussed in Section (2.2). Rt ðst Þ = 1:76St + 0:244:

ð23Þ

In order to evaluate the reservoir operating rule, yearly reservoir storage and release were simulated for 300 years of a synthetic streamflow series, as shown by Figs. 3 and 4. The one player game model is structurally similar to the simple dynamic programming (DP) model. Therefore, yearly release and storage are also simulated based on the dynamic programming (DP) to identify the capability of Ricatti equations to generate appropriate operating rules for reservoir operation. The results of Ricatti equation applied to one player game compared quite favorably with the DP results, R-square = 92.1% (see Figs. 3 and 4). The reliability indices that resulted from DP and Ricatti equations are also shown in Table 2. As seen in this table, occurrence and volumetric reliability values of the Ricatti equations method are not significantly different from the reliability indices of the DP model. 4.4. Asymmetric two-player game solution: Ricatti equations method Considering the one player game of the previous section, a solution is proposed for an asymmetric two-player game for the Zayandeh-Rud river basin. The agricultural water user and the other water consumers are the players in this game. Their short term utilities are a function of their water demands, which are developed as discussed in Section (4.2). Also, similar to the one player game, the state transition equation is the reservoir mass balance equation. By substituting the state transition equations and short term utility functions in Eqs. (9) and (10), the first derivation of the reaction function equations, i.e., Eqs. (8) and (9), are determined with respect to the decision variables, x1t and x2t : 8  > 1 2* >  ∂U x ; x ; R ; t > t > s t t > ∂gðSt ; It ; εt Þ ∂g ðSt ; It ; εt Þ > > + γEεt 2α1 gðSt ; It ; εt Þ + β1 > 1 1 1 < ∂xt ∂xt ∂xt :  > > 1 * 2 >  ∂Us xt ; xt ; Rt ; t > > > ∂g ð S ; I ; ε Þ ∂g ð S ; I ; ε Þ t t t t t t > > + γEεt 2α2 gðSt ; It ; εt Þ + β2 : ∂x1t ∂x2t ∂x2t

ð24Þ Solution to Eq. (24) in light of the state transition equation function, yields the optimal allocated water to the water users, i.e., 2⁎ (x1⁎ t ,xt ) as well as the optimal reservoir operation rule. Substituting x1⁎ back into the right-hand side of Eqs. (8) and (9) results in six pairs t of equations, called Ricatti equations. Solving the six pairs of equations simultaneously generates as many as six sets of potential solutions for parameters {α1, α2, β1, β2, η1, η2}. Taking each possible solution and substituting it into Eq. (24) produces three policy functions. Then, the appropriate solution is selected among these

 ∂g ðSt ; It ; εt Þ ∂g ðSt ; It ; εt Þ +β = 0: + γEεt 2αg ðSt ; It ; εt Þ ∂x ∂x ð22Þ

Considering the sets of resulted parameters (α, β, η), three policy functions are derived. Finally, the appropriate solution is selected

Fig. 3. Yearly simulated release as a result of DP and Ricatti equations methods.

758

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761

Fig. 4. Yearly simulated storage as a result of DP and Ricatti equations methods. Fig. 5. The operating policy as a result from the Ricatti solution of one and two-player games.

solutions, considering the rationality of resulted policy functions as discussed in Section (2.2). Fig. 5 shows the resulted policy function as a comparison with the policy function by one player Ricatti game. Comparison of the two policy functions indicates that the line's slope and interception values are decreased for the two-player case, showing that release is raised to supply agricultural demands. Furthermore, the long-term value functions are determined by substituting the appropriate solutions {α1, α2, β1, β2, η1, η2} on the left hand side of Eqs. (8) and (9) as: 2

Vt ðSt Þ = 5:82e−7St + 9:8e−3 St −5:42

ð25Þ

2

Vagr ðSt Þ = 1:25e−6St −1:8e−3St + 4:27

ð26Þ

where Vt(St) and Vagr(St) are the long-term value functions of the reservoir operator and agricultural consumer as a function of reservoir state variable (St). These equations show that the long-term utilities of operator and agricultural users increase for higher levels of water storage in the reservoir. Considering the optimal reservoir operation rule, a simulation based on a time period of 300 years was undertaken to evaluate the optimal operating rule as well as the variation in water users' utilities. The results of the simulation are evaluated using the reliability indices, and are shown in Table 3. The results (Table 3) indicate that the current total demand can be fulfilled by an average of 69.7% of time (RTn − all, column 4) during the planning horizon and the average volumetric reliability of system (RTv − all) is about 94.6%. The second column in Table 3 shows the long-term average reliability (RT(a) n − all) and volumetric reliability (RT(a) v − all) of water users. The long-term T(a) average reliability value (RT(a) n − st) and volumetric reliability (Rv − st) of the reservoir operator are also presented in the third column. As results of the one and two-player games (Tables 2 and 3), the allocation reliability indices (RTn − all ,RTv − all) have increased. This is due to additional individual Bellman equation for a consumer (as a new player) that guarantees supplying higher levels of consumer demands. Since both players' utilities are considered to derive the reservoir operating rule, a lower level of conflict is expected compared to one player solution case and DP model. Total long-term average of

Ricatti equations DP

4.5. Asymmetric three-player game: collocation method To compute an approximate solution using collocation methods, the value function is approximated as a linear combination of 15 known basis functions for the reservoir storage state variable. The choice of basis-node scheme depends on the curvature of the value function. Various basis-node schemes and dimensions of the problem are examined to render it computationally efficient. Then, the collocation equations are solved with Newton's solution method. Once the collocation method renders a solution, the residual function is computed to evaluate the quality of the approximation. Fig. 6 shows that the approximation is accurate. This function measures the difference between the left and right sides of the Bellman equation at arbitrary states St, when the value function is replaced with its approximant (and the optimal basis coefficients cj). It is zero for all states in an exact solution, and will be zero at the collocation nodes for any solution. If the approximation is adequate, the residual function will not depart too far from zero for any arbitrary value of the state in the interval St. If large residuals are obtained, the problem should be resolved using a different basis-node scheme. The value function of the solution is presented by Fig. 7 which shows the maximum long-term utility of the water users and reservoir operator is derived at 1,175 MCM of reservoir storage. The

Table 3 The results of simulation based on the outcomes by Ricatti solution method. Reliability

Water users

T(a) RT(a) 69.7 n − all (Rn − st) T(a) RT(a) 94.6 v − all (Rv − st) A.R.D. (MCM) 959.76 A.S.D (MCM) 890.76 V.S.D (MCM) 18,584

Table 2 The reliability of DP and Ricatti models for Zayandeh-Rud reservoir system. Model

utility satisfaction (US) of water users is also determined for Ricatti equations method. The US values are respectively 55.5 and 74.10, for agricultural and other users. However, solving three and more players' games is not possible analytically due to complexity of resulted equations. As a result the collocation method is considered to solve the multi-players game problem, which is discussed next.

Reliability indices RnT − all

RTv − all

RnT − st

Rv − st

59.33 56.67

84.54 99.74

84.33 76.33

0.046 0.001

Reservoir operator Total reservoir system (total reservoir storage index) allocation indices (75.3) (0.064)⁎ – – –

RnT − all = 69.7 RTv − all = 94.6 1277 1208.6 1.46e6

*RT(a) v − st: total storage shortfall/total available water during the planning horizon. A.R.D. (U = 1): average required demand which sets utility satisfaction (US) equal to one. A.S.D.: average supplied demand. V.S.D.: variance of supplied demand.

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761

759

compared to Ricatti equations) may be the main reason for the better results. Total long-term average of utility satisfaction (US) of water users is also determined for the collocation method. The US values are (0.609, 0.844, and 0.604) for agriculture, other users and reservoir operator that show a higher level of satisfaction for the system with respect to Ricatti equations results. Fig. 8 shows the utility function of water users and reservoir operator as resulted from simulation based on the optimal reservoir operating rule. As indicated, the outcome utility for the other users (the industrial, environmental and domestic users) is higher than the utilities of agricultural water users as previously discussed by Ganji et al. [18,21]. A review of the collocation and Ricatti equations solution methods shows that the collocation has generated superior results than the Ricatti method. The collocation method addresses many of the shortcomings of the Ricatti solution method, since it employs global (rather than local) function approximation schemes, and is not limited to the first- and second-degree approximations afforded by Ricatti methods. The efficiency of the collocation solution and Ricatti equations methods can be further compared, using the information on the system reliability index of PSDNG model [20] (Table 5). The result shows that the collocation method is a powerful tool and can improve the reliability of the allocated demand for total reservoir system with respect to the Ricatti equations method. However, it also indicates that reliability values of the PSDNG model (RTn − all = 96.38, RTv − all = 97.21 and RT(a) n − st = 0.100) are greater than the corresponding index values of the collocation method (RTn − all = 69.7, RTv − all = 94.6 and RT(a) n − st = 0.753). The stochastic nature of the PSDNG model may be the main reason for the better results. It should be noted that although the proposed collocation method is a deterministic solution method, it can be extended to represent a stochastic model. The PSDNG is a stochastic conflict resolution model, which makes possible consideration of the uncertainty in input values in reservoir operation and decision maker preference. However, PSDNG model needs state and decision variables to be discretized. Furthermore, PSDNG model uses the Simulated Annealing (SA) procedure to search for the static equilibrium point in each state of n stages of the model. A fine discretization and using SA procedure increase the runtime and cause dimensionality problems [21]. Also, it should be noted that avoiding the curse of dimensionality is related to the nature and structure of model. As a result, continuous state and decision variables and the explicit solution to the set of equations require less computational efforts when compared with implicit and iterative methods, in complex multi player/reservoir problems. For example, in the case of the two-player game, only six sets of equations should be solved explicitly in the Ricatti solution method. Considering the same number of players in a discrete type of model, many attempts were made to define the best discretization scheme for reservoir storage and inflow. Therefore, Zayandeh-Rud's effective storage was divided into 20 class intervals. Also, the class

Fig. 6. The residual function as a result from collocation solution.

Fig. 7. The value function as a result from collocation solution.

policy function obtained using collocation method is also shown in Fig. 5. A simulation was done to evaluate the optimal operating rule as well as the variation in water users' utilities. Comparison of the reliabilities as resulted from collocation solution and Ricatti equations T methods illustrates that RT(a) n − st and Rn − all are increased to 82.4 and 72.3% for collocation method (see Tables 3 and 4). Furthermore, both T volumetric reliability indices (RT(a) v − st and Rv − all) are improved for collocation method. Table 4 also shows that the average level of supplied demand (A.S.D) has increased for the collocation method as well. The flexible structure of the collocation model (e.g., not limited to the first and second-degree value function approximations and capability to consider constraints on the action and state variables,

Table 4 The results of simulation based on the outcomes by collocation solution method. Reliability

T(a) RnT(a) − all (Rn − st) T(a) RT(a) (R v − all v − st) A.R.D. (MCM) A.S.D (MCM) V.S.D (MCM)

Water users Other users (domestic, industrial, environmental

Agricultural

89.33 94.95 316.92 315.93 99,179

77.33 98.19 959.76 942.35 886,180

*RT(a) v − st: total storage shortfall / total available water during the planning horizon. A.R.D. (U = 1): average required demand which sets utility satisfaction (US) equal to one. A.S.D.: average supplied demand. V.S.D.: variance of supplied demand.

Reservoir operator (total reservoir storage index)

Total reservoir system allocation indices

(82.33) (0.049)⁎ – – –

RnT − all = 72.33 RTv − all = 96.30 1277 1229.5 1,509,200

760

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761

References

Fig. 8. Yearly simulated utilities of water users and reservoir operator which resulted by the optimal reservoir operation rule.

interval of inflow was set at 6 for each month of the year. So the number of possible system state transitions to be explored was equal to 20 × 6 × 20 for each month of the year (at each stage) and for each player [20]. Also there are 12 stages within a year and 360 stages for the planning horizon, and twelve monthly transition probability matrices for monthly inflow into the reservoir. Considering the possible number of system state transitions and number of stages within the planning horizon, major computational effort is required for discrete solution methods, in contrast to the Ricatti equations (continuous) and collocation methods (semi-continuous). In this case, what should be considered when continuous and discrete types of models are compared, is reliability indices. A continuous model will be a good alternative, if it at least solves a simple problem similar to discrete models. This is the most important reason to identify computational efficiency of explicit solutions, which can be logically/mathematically concluded from the model structure and solution method. Although the strength of the collocation method is that the bounds can be placed on the values of decision variables, it requires additional computational time in comparison with the Ricatti equations method. Furthermore, it is important to choose appropriate initial values when using collocation method, to ensure solution convergence. 5. Conclusions In this research two continuous solution methods were developed for dynamic game models of reservoir operation and water allocation with conflicting objectives. As a comparison with alternative discrete solution methods, i.e., PSDNG [20], the proposed solution methods generate appropriate operating policy rules for reservoir operation using water users' preferences. Among the two proposed solution methods, the collocation method significantly improves the reliability in storage volume as compared with the alternative Ricatti solution method. Also, the methods proposed in this paper are more efficient and quicker to compute, due to continuous formulation of state variable and value function.

Table 5 The comparison of system reliability indices among Ricatti equations, collocation solution methods and PSDNG model. Reliability index

Models Ricatti equations

Collocation

PSDNG

RnT − all RTv − all RnT(a) − st

69.7 94.6 75.3 1277 1229.5

72.33 96.30 82.33 1277 1208.6

96.38 97.21 100 1277 1228.8

A.R.D. (MCM) A.S.D (MCM)

[1] Alam M, Bhuiyan MA. Collocation finite-element simulation of dam break flows. J Hydraul Eng 1995;121(2):118–28. [2] Batabyal A. Consistency and optimality in dynamic game of pollution control I: competition. Environ Resour Econ 1996;8:205–20. [3] Bellman R, Dreyfus SE. Applied Dynamic Programming. Princeton, New Jersey: Princeton University Press; 1962. [4] Cai X. Modeling Framework for Sustainable Water Resources Management. Unpublished Ph.D. dissertation, Univ. Texas at Austin, 1999. [5] Carraro C, Marchiori C, Sgobbi A. Applications of negotiation theory to water issues. World Bank Policy Research Working Paper No. 3641; 2005. [6] Dixon, L. Model of groundwater extraction with an examination of agricultural water use in Kern County, California. Ph.D. Dissertation University of California, Berkeley, 1989. [7] Dochain D, Babary JP, Tali-Maamar N. Modeling and adaptative control of nonlinear distributed parameter bioreactor via orthogonal collocation. Automatica 1992;28(5):873–83. [8] Dockner E, Feichtinger G, Mehlmann A. Noncooperative solution for differential game model of fishery. J Econ Dyn Control 1989;13:1–20. [9] Dockner JE, Van Long N. International pollution control: cooperative versus noncooperative strategies. J Environ Econ Manag 1993;24:13–29. [10] Dulhoste JF, Besancon G, Georges D. Non-linear control of water flow dynamics by input–output linearization based on a collocation method model. Eur Control Conf 2001. [11] Dulhoste JF. Contribution a` la commmande non line´aire de syste`mes d'irrigation. PhD thesis, Laboratoire d'Automatique de Grenoble, Inst. National Polytechnique de Grenoble, Grenoble, France (in French), 2001. [12] Dyer JS, Farrell W, Bradley P. Utility function for test performance. Manage Sci 1973;20(40):507–19. [13] Ejaz MS, Peralta RC. Modeling for optimal management of agricultural and domestic wastewater loading to streams. Water Resour Res 1995;31(4):1087–96. [14] Fletcher CAJ. Computational Galerkin methods. Springer Series in Computational Physics. Berlin: Springer; 1984. [15] Fontane DG, Gates TK, Moncada E. Planning reservoir operation with imprecise objectives. J Water Resour Plan Manage 1997;123(3):154–62. [16] Foued BA, Sameh M. Application of goal programming in a multi-objective reservoir operation model in Tunisia. Eur J Opera Res 2001;133(2):352–61. [17] Fundenberg D, Tirole J. Game theory. Cambridge, Massachusetts: MIT Press; 1996. [18] Ganji A, Karamouz M, Khalili D. Development of stochastic conflict resolution models for reservoir operation, II. The value of players' information availability and cooperative behavior. Adv Water Resour 2007;30:157–68. [19] Ganji A, Khalili D, Javan M. Uncertainty in the design and simulation of reservoir. Iranian J Sci Technol Trans 2001;B4(25):729–36. [20] Ganji A, Khalili D, Karamouz M. Development of stochastic conflict resolution models for reservoir operation. I. The perfect symmetric stochastic model. Adv Water Resour 2007;30:528–42. [21] Ganji A, Khalili D, Karamouz M, Ponnambalam K, Javan M. A fuzzy stochastic dynamic Nash game analysis of policies for managing water allocation in a reservoir system. Water Resour Manag 2008;22(1):51–66. [22] Ganji A. Streamflow modeling and analysis of Mollasadra and Salman Farsi reservoirs using time series models of SPIGOT. M.S. Thesis, University of Shiraz, 388pp, 1999. [23] Georges D, Dulhoste JF, Besancon G. Modelling and control of water flow dynamics via a collocation model. Math theory net sys; 2000. Perpignan, France. [24] Harboe R. Multi-objective decision making techniques for reservoir operation. Mono Ser No. 18, Am Water Resour Associ 1992;Bethesda, Md., 103–110. [25] Hashimoto T, Stedinger JR, Loucks DP. Reliability, resiliency, and vulnerability criteria for water resource system performance and evaluation. Water Resour Res 1982;18(1):14–20. [26] Karamouz M, Houck MH. Annual and monthly reservoir operating rules. Water Resour Res 1982;18(5):1337–44. [27] Karamouz M, Szidarovszky F, Zahraie B. Water resources system analysis. Boca Raton, Florida: Lewis Publishers; 2003. 33431. [28] Kennedy JOS. A computable game theoretic approach to modeling competitive fishing. Mar Res Econ 1987;4:14–24. [29] Kerachian R, Karamouz M. A stochastic conflict resolution model for water quality management in reservoir-river systems. Adv Water Resour 2007;30:866–82. [30] Kishor N, Singh SP, Raghuvanshi AS. Dynamic simulations of hydro turbine and its state estimation based LQ control. Energy Conv Manag 2006;47(19):3119–37. [31] Ko S., Fontane D.G., Labadie J.W. Multi-objective optimization of reservoir system operation. Mono. Ser. No. 18, Am Water Resour Associ 1992;Bethesda, Md., 111– 128. [32] Kong K, Xie JC, Yue XL, Chen HQ. Game analysis of water resources market. J Hydra Eng 2005;36(4):491–5. [33] Levhari D, Mirman LJ. The great fish war: an example using a dynamic Cournat– Nash solution. Bell J Econ 1980;11(1):322–34. [34] Li YP, Huang GH, Chen X. Multistage scenario-based interval-stochastic programming for planning water resources allocation. Stoch Environ Res Risk Assess 2008, doi:10.1007/s00477-008-0258-y. [35] Ligon E, Narain U. Computing the equilibria of dynamic common property games. Nat Resour Model 1997;10(4):345–69. [36] Lockwood B. Uniqueness of Markov-perfect equilibrium in infinite-time affinequadratic differential games. J Econ Dyn Control 1996;20(5):751–65. [37] Longanathan GV, Bhattacharya D. Goal programming techniques for optimal reservoir operations. J Water Resour Plan Manage 1990;111(6):820–38.

M. Homayoun-far et al. / Advances in Water Resources 33 (2010) 752–761 [38] Loucks DP, Stedinger JR, Haith DA. Water resources system planning and analysis. England Cliffs, NJ: Prentice-Hall; 1988. [39] Madani K. Game theory and water resources. J Hydro 2010;381:225–38. [40] Madani K, Mariño MA. System dynamics analysis for managing Iran's ZayandehRud River Basin. Water Resour Manage 2009;23:2163–87. [41] Martin EW, Patrick RH, Tolwinski B. A dynamic game of a trans-boundary pollutant with asymmetric players. J Environ Econ Manag 1993;24:1–12. [42] Miranda MJ, Fackler PL. Applied computational economics and finance. MIT press; 2002. [43] Mourad E, Lachhab A, Limouri M, Dahhou B, Essaid A. Adaptive control of a water supply system. Contr Eng Pract 2001;9:343–9. [44] Mousavi SJ, Karamouz M, Menhadj MB. Fuzzy-state stochastic dynamic programming for reservoir operation. J Water Resour Plan Manage 2004;130(6):460–70. [45] Mousavi SJ, Ramamurthy AS. Optimal design of multi-reservoir systems for water supply. Adv Water Resour 2000;23(6):613–24. [46] Msangi S. Managing groundwater in the presence of asymmetry: three essays. Ph. D. thesis, University of Davis, California 2004. [47] Murray-Rust H, Sally H, Salemi HR, Mamanpoush A. An overview of the hydrology of the Zayandeh Rud Basin, Esfahan Province, Iran. IAERI-IWMI Research Reports 3; 2000. [48] Nakao M. Dynamic games and competition for water resources. Ph.D. dissertation University of Rhode Island, 2003. [49] Nakas MD, Wicheins D, Montgomery I. Game theory analysis of competition for groundwater involving El Paso, Texas and Ciudad Juarez, Mexico. paper presented at “moving with speed of change” the 2002 annual meeting of the American Agricultural Economics Assosiation, Long Beach, CA, 18-31; 2002. [50] Negri DH. Strategy of the commons. Nat Resour Model 1990;4(4):521–37. [51] Negri DH. The common property aquifer as a differential game. Water Resour Res 1989;25(1):9–15. [52] Park Y, Yeghiazarian L, Stedinger JR, Montemagno CD. Numerical approach to Cryptosporidium risk assessment using reliability method. J Stoch Environ Res Risk Assess 2009;22(2):169–83. [53] Parrachino I, Dinar A, Patrone F. Cooperative game theory and its application to natural, environmental, and water resource issues: 3. Application to Water Resources. World Bank Policy Research Working Paper No. 4074, WPS4072; 2006.

761

[54] Prorencher B, Burt O. The externalities associated with the common property exploitation of groundwater. J Environ Econ Manage 1993;24:139–58. [55] Rubio SJ, Casino B. Competitive versus efficient extraction of a common property resources: the groundwater case. J Econ Dyn Control 2001;25(8):1117–37. [56] Saak AE, Peterson JM. Groundwater use under incomplete information. J Environ Econ Manage 2007;54:214–28. [57] Sally H, Murray-Rust H, Mamanpoush AR, Akbari M. Water supply and demand in four major irrigation systems in the Zayandeh Rud Basin, Iran. IAERI-IWMI Research Reports 8; 2001. [58] Shiau JT, Lee HC. Derivation of optimal hedging rules for a water-supply reservoir through compromise programming. Water Resour Manag 2005;19(2):111–32. [59] Shirangi E, Kerachian R, Shafaie Bajestan M. A simplified model for reservoir operation considering the water quality issues: application of the Young conflict resolution theory. Environ Monit Assess 2008;46(1–3):77–89. [60] Simonovic SP. Coping with changing objectives in managing an existing multipurpose reservoir. Hydro Nat Manmade Lakes, Pub. No. 206, Int. Associ. Hydro. Sci., Walingford, U.K., 181–189; 1991. [61] Xie J, Sun F, Wang Y, Kong K, Thian Y. A two-stage dynamic game model for water resources optimal allocation. Proceeding of Symposium in Guangzhou, China, June; 2006. [62] Yeh WWG. Reservoir management and operations models: a state of the art review. Water Resour Res 1985;21(12):1797–818. [63] Zahraie B, Hosseini SM. Development of reservoir operation policies considering variable agricultural water demands. Exp Sys App 2009;36:4980–7. [64] Zara S, Dinar A, Patrone F. Cooperative game theory and its application to natural, environmental, and water resource issues: 2. Application to Natural and Environmental Resources. World Bank Policy Research Working Paper 4073; 2006. [65] Zeleny M. Compromise programming. In: Cochrane J, Zeleny M, editors. Multiple criteria decision making. Columbia, USA: University of South Carolina Press; 1973.