- Email: [email protected]
Stochastic multi-commodity network design: The quality of deterministic solutions
Accepted Manuscript Stochastic multi-commodity network design: The quality of deterministic solutions Congshi Sun, Stein W. Wallace, Li Luo PII: DOI: Reference:
S0167-6377(17)30001-9 http://dx.doi.org/10.1016/j.orl.2017.03.008 OPERES 6211
To appear in:
Operations Research Letters
Received date: 2 January 2017 Accepted date: 29 March 2017 Please cite this article as: C. Sun, S.W. Wallace, L. Luo, Stochastic multi-commodity network design: The quality of deterministic solutions, Operations Research Letters (2017), http://dx.doi.org/10.1016/j.orl.2017.03.008 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Stochastic multi-commodity network design: The quality of deterministic solutions Congshi Suna , Stein W. Wallaceb,a,∗, Li Luoa b Department
a Business School of Sichuan University, Chengdu, China of Business and Management Science, NHH Norwegian School of Economics, Helleveien 30, NO-5045 Bergen, Norway
Abstract Solutions to deterministic network design problems are feasible, but often far from optimal, solutions to stochastic versions of the problem. On the other hand, deterministic models are much easier to solve. We demonstrate that deterministic solutions, despite being of rather low quality, are very useful in finding good, if not optimal, solutions to stochastic multi-commodity network design models. Keywords: Stochastic programming, network design, multi-commodity, random demand. 1. Introduction Deterministic network design problems are NP-hard [1]. Hence, solving stochastic network design problems is quite a challenge. Furthermore, since real-world problems tend to be large, simple approaches are very unlikely to 5
succeed, particularly for the stochastic case. The premise in this work is that despite the complexity of deterministic network design models, they are much easier to solve, exactly or heuristically, than the stochastic counterparts, and in fact, very good methods exist. But at the same time we know that solutions to deterministic network design models do not provide very good solutions to ∗ Corresponding
author Email address: [email protected] (Stein W. Wallace)
Preprint submitted to OR Letters
January 2, 2017
10
stochastic versions of the same models, see for example [2] for a general discussion. So we ask: If we are facing a stochastic multi-commodity network design problem, and are able to solve the deterministic (expected value) version of the problem, but not the stochastic one, can we make use of the deterministic solution even if it is rather bad in its own right?
15
Other stochastic network design problems have been investigated before. The single commodity version of our problem was investigated in [3, 4, 5], while a dynamic service network design model is discussed in [6]. In all these cases, ways were found to utilize optimal or good solutions from deterministic problems to find good solutions to stochastic ones, despite the fact that the solutions
20
themselves were not very good. We shall investigate the qualities of solutions to deterministic models by performing three comparison tests. For a general discussion, see [7]. The three tests are: • A: VSS. We calculate the classical Value of the Stochastic Solution (VSS)
25
introduced in [8]. It calculates the expected value of using a stochastic
model; or in our case, more precisely, the expected loss when using a deterministic model. • B: Skeleton. This test was first introduced in [3]. We solve the determin-
istic model, extract the discrete variables (which arcs to open) from it, fix
30
them in the stochastic model, and then solve a stochastic linear program to set arc capacities (and second-stage flows). • C: Upgrade Again we start by solving the deterministic model. We then
take the deterministic solution as a lower bound on the variables in the stochastic model, and solve it.
35
So what do these tests tell us? VSS simply tells us how good the deterministic solution is. It is well established that VSS can be made arbitrarily large by setting penalties for violating constraints (in our case not satisfying demand) very high. Also VSS can become infinitely large if the deterministic solution 2
turns out to be infeasible in one or more of the scenarios describing the stochas40
tic demands. However, we avoid these traps by being careful with the penalties and by using soft, not hard, constraints; after all, our goal is to demonstrate the usefulness of deterministic solutions, not to declare them useless. The second test helps us understand why a deterministic solution is bad (if indeed it is). If the Skeleton approach leads to a good overall solution, we know
45
that the complicating variables - the open/close decisions - are well set in the deterministic solutions and that the problem is the capacities. If this works well, we have a heuristic for the stochastic case: First solve a deterministic network design model, then a stochastic linear program, and arrive at a very good solution to the stochastic network design model.
50
The third test is there to test if the deterministic solution can be upgraded to something good. That is, if the deterministic solution is implemented, and problem owners see how bad it is (if VSS is high) can it then we upgraded to something good, or are we already lost? The upgrade problem is of the same complexity as the original problem (albeit smaller) so this is not really
55
a method. The purpose is to understand if upgrading is possible. However, if the answer is yes, we may use the deterministic solution as a starting point for a heuristic approach, as we know that the solution to the stochastic case is the deterministic solution plus ”something”. There is no reason to start the heuristic from scratch.
60
Despite out attempts to set penalties at meaningful levels, the absolute value of VSS will never be a very good measure. What we do, therefore, is to measure what portion of the VSS that can be removed when the skeleton is used, and similarly, when the deterministic solution is upgraded. Note that it is far from obvious that the skeleton is good or that a deterministic solution can be
65
upgraded, see [7] for details.
3
2. The model Let G = (N, E) be a network defined by a set N of n nodes, each of which can be a demand node, a supply node or a transshipment node for each commodity, and a set E of n directed edges, where E ⊆ {(i, j) : i, j ∈ N, i 6= j}. 70
Furthermore, K is the set of commodities, with Dk , Tk and Ak the sets of de-
mand nodes, transshipment nodes and supply nodes for commodity k ∈ K. The set of scenarios is described by S.
Our parameters are (with i, j ∈ N or the relevant subset of N , k ∈ K and
s ∈ S): Mij is the maximal capacity of an edge, Rk the unit cost of unsatisfied 75
demand, ps the probability of a scenario, cijk the unit flow cost, gij the fixed setup cost, fij the per unit variable setup cost, αik the supply and dsik the demand (negative). Decision variables: xsijk represents flow on edges, esik unsatisfied demand, vij installed capacity and yij a binary variable telling if an edge is opened or not.
80
The exact sets that each parameter and variable applies to will be apparent from the model itself. min
X
gij yij +
(i,j)∈E
X
fij vij +
xsijk −
X
X
ps
s∈S
(i,j)∈E
k∈K
subject to X
j:(i,j)∈E
X
j:(i,j)∈E
X
j:(i,j)∈E 85
xsijk −
xsijk −
j:(j,i)∈E
X
j:(j,i)∈E
X
j:(j,i)∈E
X
k∈K
X
X
cijk xsijk + Rk
(i,j)∈E
X
i∈Dk
xsjik = 0 for i ∈ Tk , k ∈ K, s ∈ S
xsjik ≤ αik for i ∈ Ak , k ∈ K, s ∈ S
xsjik = dsik + esik for i ∈ Dk , k ∈ K, s ∈ S
xsijk ≤ vij for (i, j) ∈ E, s ∈ S vij ≤ Mij yij for (i, j) ∈ E
0 ≤ esik ≤ −dsik for i ∈ Dks , k ∈ K, s ∈ S vij , xsijk ≥ 0, yij ∈ {0, 1} for (i, j) ∈ E, k ∈ K, s ∈ S 4
esik
Test cases had 16 nodes, 4 or 5 commodities, with each commodity having 2 90
supply nodes and 3 or 4 demand nodes. Demand scenarios were generated using [9]. We tested with both positively and mixed correlated demands (the case of only negative correlations does not exist, except with the correlations extremely close to zero). We needed 20 scenarios in each case to achieve in- and out-ofsample stability [10], within an accuracy of about 2%. Penalties for unsatisfied
95
demand were set such that the service rate in the stochastic model was from 90 to 99%. This way we tried to avoid unreasonable results caused by unreasonable penalties. The model was implemented in MPL and solved to optimality using CPLEX 12.6.0. Solution times varied from a few seconds to about two hours on an Intel Core i7 running at 1.80 GHz with 8.0GB of RAM.
100
3. Results The main results are based on 16 different randomly generated cases, and are summed up in Figure 1. We see that with the chosen penalties, we lose about 5.2% on average by using the deterministic solution in the stochastic environment. However, as pointed out before, this number does not mean too
105
much in its own right, whether it is large or small, as it totally depends on the chosen parameters. What is much more important is that by using the Skeleton, as much as 97% of this loss can be removed, on average. That is, by first solving a deterministic network design problem, and then a stochastic linear problem, we end up with an almost optimal solution. Furthermore, the
110
deterministic solution can be upgraded so that, on average, 94% of the VSS is recovered. This shows that the deterministic solution is almost perfectly contained in the stochastic one, and could, for example, be used as a starting point for a constructive heuristic. In other words, the deterministic solutions install too low capacities, but basically pick the right edges.
115
As always, it is interesting to see what happens when correlations vary, in particular what happens when negative correlations exist. We found that negative correlations generally led to more consolidation (of course particularly when
5
14,0% 12,0% 10,0% 8,0%
Comparison A Comparison B
6,0%
Comparison C 4,0% 2,0% 0,0% Minimum
Geometric mean
Maximum
Figure 1: Percent loss in objective function value for the three tests, for the best, average and worst cases.
the nodes with negatively correlated demands are close to each other in the network). In those cases VSS was smaller (4.7%) than with positive correlations 120
(5.7%) as consolidation led to less capacity being installed, and hence that the deterministic solutions (which are the same irrespective of correlations) were less off. Consider Figure 2 as an example. The square nodes (nodes 1, 2, 3 and 4) were the supply nodes, though only three of them ended up being used in this case. Nodes 12, 14, 15 and 16 are demand nodes, the rest transshipment
125
nodes (which in this specific example ended up not being used). Demand nodes are split into two groups, one containing nodes 12 and 14, the other nodes 15 and 16. Correlations between demands within a group are positive, while correlations between demands across groups are negative. We observe that positive correlations lead to higher installed capacities. For negative correlations, we
130
also see a loop including nodes 12, 14 and 15, used for consolidation. References [1] I. Ghamlouche, T. G. Crainic, M. Gendreau, Path relinking, cycle-based neighbourhoods and capacitated multicommodity network design, Annals 6
6
9
13 6
9
13
5 4 2
8
46 40
5
11
4 2
3 139
1 12 7
48
15
37
11
3 134
1 12
16
38
8
37 36
7
39 33 16
40
10 10
14
13 14
6
9
13
5 4 2
8
32 32
11
3 128
1 12 7
32
15
32 16
32 10 14
Figure 2: Network designs for a case of correlated demands. In the upper left graph, demands are positively correlated, while in the upper right correlations are mixed. In the bottom graph, we find the deterministic solution. The numbers on the edges indicate installed capacity.
of Operations Research 131 (1) (2004) 109–133. 135
[2] S. Wallace, Decision making under uncertainty: Is sensitivity analysis of any use?, Operations Research 48 (2000) 20–25. doi:10.1287/opre.48. 1.20.12441. [3] B. K. Thapalia, T. G. Crainic, M. Kaut, S. W. Wallace, Single source single-commodity stochastic network design, Computational Management
140
Science 9 (1) (2012) 139–160, special issue on ‘Optimal decision making under uncertainty’. doi:10.1007/s10287-010-0129-0. [4] B. K. Thapalia, T. G. Crainic, M. Kaut, S. W. Wallace, Single-commodity stochastic network design with multiple sources and sinks, INFOR: Information Systems and Operational Research 49 (3) (2011) 193–211. doi:
145
10.3138/infor.49.3.003. 7
15
[5] B. K. Thapalia, T. G. Crainic, M. Kaut, S. W. Wallace, Single-commodity network design with random edge capacities, European Journal of Operational Research 220 (2) (2012) 394–403. [6] A.-G. Lium, T. G. Crainic, S. W. Wallace, A study of demand stochasticity 150
in stochastic network design, Transportation Science 43 (2) (2009) 144–157. doi:10.1287/trsc.1090.0265. [7] F. Maggioni, S. W. Wallace, Analyzing the quality of the expected value solution in stochastic programming, Annals of Operations Research 200 (1). doi:10.1007/s10479-010-0807-x.
155
[8] J. R. Birge, The value of the stochastic solution in stochastic linear programming with fixed recourse, Mathematical Programming 24 (1982) 314– 325. [9] M. Kaut, A copula-based heuristic for scenario generation, Computational Management Science 11 (4) (2014) 503–516.
160
doi:10.1007/
s10287-013-0184-4. [10] M. Kaut, S. W. Wallace, Evaluation of scenario-generation methods for stochastic programming, Pacific Journal of Optimization 3 (2) (2007) 257– 271.
8
S0167-6377(17)30001-9 http://dx.doi.org/10.1016/j.orl.2017.03.008 OPERES 6211
To appear in:
Operations Research Letters
Received date: 2 January 2017 Accepted date: 29 March 2017 Please cite this article as: C. Sun, S.W. Wallace, L. Luo, Stochastic multi-commodity network design: The quality of deterministic solutions, Operations Research Letters (2017), http://dx.doi.org/10.1016/j.orl.2017.03.008 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Stochastic multi-commodity network design: The quality of deterministic solutions Congshi Suna , Stein W. Wallaceb,a,∗, Li Luoa b Department
a Business School of Sichuan University, Chengdu, China of Business and Management Science, NHH Norwegian School of Economics, Helleveien 30, NO-5045 Bergen, Norway
Abstract Solutions to deterministic network design problems are feasible, but often far from optimal, solutions to stochastic versions of the problem. On the other hand, deterministic models are much easier to solve. We demonstrate that deterministic solutions, despite being of rather low quality, are very useful in finding good, if not optimal, solutions to stochastic multi-commodity network design models. Keywords: Stochastic programming, network design, multi-commodity, random demand. 1. Introduction Deterministic network design problems are NP-hard [1]. Hence, solving stochastic network design problems is quite a challenge. Furthermore, since real-world problems tend to be large, simple approaches are very unlikely to 5
succeed, particularly for the stochastic case. The premise in this work is that despite the complexity of deterministic network design models, they are much easier to solve, exactly or heuristically, than the stochastic counterparts, and in fact, very good methods exist. But at the same time we know that solutions to deterministic network design models do not provide very good solutions to ∗ Corresponding
author Email address: [email protected] (Stein W. Wallace)
Preprint submitted to OR Letters
January 2, 2017
10
stochastic versions of the same models, see for example [2] for a general discussion. So we ask: If we are facing a stochastic multi-commodity network design problem, and are able to solve the deterministic (expected value) version of the problem, but not the stochastic one, can we make use of the deterministic solution even if it is rather bad in its own right?
15
Other stochastic network design problems have been investigated before. The single commodity version of our problem was investigated in [3, 4, 5], while a dynamic service network design model is discussed in [6]. In all these cases, ways were found to utilize optimal or good solutions from deterministic problems to find good solutions to stochastic ones, despite the fact that the solutions
20
themselves were not very good. We shall investigate the qualities of solutions to deterministic models by performing three comparison tests. For a general discussion, see [7]. The three tests are: • A: VSS. We calculate the classical Value of the Stochastic Solution (VSS)
25
introduced in [8]. It calculates the expected value of using a stochastic
model; or in our case, more precisely, the expected loss when using a deterministic model. • B: Skeleton. This test was first introduced in [3]. We solve the determin-
istic model, extract the discrete variables (which arcs to open) from it, fix
30
them in the stochastic model, and then solve a stochastic linear program to set arc capacities (and second-stage flows). • C: Upgrade Again we start by solving the deterministic model. We then
take the deterministic solution as a lower bound on the variables in the stochastic model, and solve it.
35
So what do these tests tell us? VSS simply tells us how good the deterministic solution is. It is well established that VSS can be made arbitrarily large by setting penalties for violating constraints (in our case not satisfying demand) very high. Also VSS can become infinitely large if the deterministic solution 2
turns out to be infeasible in one or more of the scenarios describing the stochas40
tic demands. However, we avoid these traps by being careful with the penalties and by using soft, not hard, constraints; after all, our goal is to demonstrate the usefulness of deterministic solutions, not to declare them useless. The second test helps us understand why a deterministic solution is bad (if indeed it is). If the Skeleton approach leads to a good overall solution, we know
45
that the complicating variables - the open/close decisions - are well set in the deterministic solutions and that the problem is the capacities. If this works well, we have a heuristic for the stochastic case: First solve a deterministic network design model, then a stochastic linear program, and arrive at a very good solution to the stochastic network design model.
50
The third test is there to test if the deterministic solution can be upgraded to something good. That is, if the deterministic solution is implemented, and problem owners see how bad it is (if VSS is high) can it then we upgraded to something good, or are we already lost? The upgrade problem is of the same complexity as the original problem (albeit smaller) so this is not really
55
a method. The purpose is to understand if upgrading is possible. However, if the answer is yes, we may use the deterministic solution as a starting point for a heuristic approach, as we know that the solution to the stochastic case is the deterministic solution plus ”something”. There is no reason to start the heuristic from scratch.
60
Despite out attempts to set penalties at meaningful levels, the absolute value of VSS will never be a very good measure. What we do, therefore, is to measure what portion of the VSS that can be removed when the skeleton is used, and similarly, when the deterministic solution is upgraded. Note that it is far from obvious that the skeleton is good or that a deterministic solution can be
65
upgraded, see [7] for details.
3
2. The model Let G = (N, E) be a network defined by a set N of n nodes, each of which can be a demand node, a supply node or a transshipment node for each commodity, and a set E of n directed edges, where E ⊆ {(i, j) : i, j ∈ N, i 6= j}. 70
Furthermore, K is the set of commodities, with Dk , Tk and Ak the sets of de-
mand nodes, transshipment nodes and supply nodes for commodity k ∈ K. The set of scenarios is described by S.
Our parameters are (with i, j ∈ N or the relevant subset of N , k ∈ K and
s ∈ S): Mij is the maximal capacity of an edge, Rk the unit cost of unsatisfied 75
demand, ps the probability of a scenario, cijk the unit flow cost, gij the fixed setup cost, fij the per unit variable setup cost, αik the supply and dsik the demand (negative). Decision variables: xsijk represents flow on edges, esik unsatisfied demand, vij installed capacity and yij a binary variable telling if an edge is opened or not.
80
The exact sets that each parameter and variable applies to will be apparent from the model itself. min
X
gij yij +
(i,j)∈E
X
fij vij +
xsijk −
X
X
ps
s∈S
(i,j)∈E
k∈K
subject to X
j:(i,j)∈E
X
j:(i,j)∈E
X
j:(i,j)∈E 85
xsijk −
xsijk −
j:(j,i)∈E
X
j:(j,i)∈E
X
j:(j,i)∈E
X
k∈K
X
X
cijk xsijk + Rk
(i,j)∈E
X
i∈Dk
xsjik = 0 for i ∈ Tk , k ∈ K, s ∈ S
xsjik ≤ αik for i ∈ Ak , k ∈ K, s ∈ S
xsjik = dsik + esik for i ∈ Dk , k ∈ K, s ∈ S
xsijk ≤ vij for (i, j) ∈ E, s ∈ S vij ≤ Mij yij for (i, j) ∈ E
0 ≤ esik ≤ −dsik for i ∈ Dks , k ∈ K, s ∈ S vij , xsijk ≥ 0, yij ∈ {0, 1} for (i, j) ∈ E, k ∈ K, s ∈ S 4
esik
Test cases had 16 nodes, 4 or 5 commodities, with each commodity having 2 90
supply nodes and 3 or 4 demand nodes. Demand scenarios were generated using [9]. We tested with both positively and mixed correlated demands (the case of only negative correlations does not exist, except with the correlations extremely close to zero). We needed 20 scenarios in each case to achieve in- and out-ofsample stability [10], within an accuracy of about 2%. Penalties for unsatisfied
95
demand were set such that the service rate in the stochastic model was from 90 to 99%. This way we tried to avoid unreasonable results caused by unreasonable penalties. The model was implemented in MPL and solved to optimality using CPLEX 12.6.0. Solution times varied from a few seconds to about two hours on an Intel Core i7 running at 1.80 GHz with 8.0GB of RAM.
100
3. Results The main results are based on 16 different randomly generated cases, and are summed up in Figure 1. We see that with the chosen penalties, we lose about 5.2% on average by using the deterministic solution in the stochastic environment. However, as pointed out before, this number does not mean too
105
much in its own right, whether it is large or small, as it totally depends on the chosen parameters. What is much more important is that by using the Skeleton, as much as 97% of this loss can be removed, on average. That is, by first solving a deterministic network design problem, and then a stochastic linear problem, we end up with an almost optimal solution. Furthermore, the
110
deterministic solution can be upgraded so that, on average, 94% of the VSS is recovered. This shows that the deterministic solution is almost perfectly contained in the stochastic one, and could, for example, be used as a starting point for a constructive heuristic. In other words, the deterministic solutions install too low capacities, but basically pick the right edges.
115
As always, it is interesting to see what happens when correlations vary, in particular what happens when negative correlations exist. We found that negative correlations generally led to more consolidation (of course particularly when
5
14,0% 12,0% 10,0% 8,0%
Comparison A Comparison B
6,0%
Comparison C 4,0% 2,0% 0,0% Minimum
Geometric mean
Maximum
Figure 1: Percent loss in objective function value for the three tests, for the best, average and worst cases.
the nodes with negatively correlated demands are close to each other in the network). In those cases VSS was smaller (4.7%) than with positive correlations 120
(5.7%) as consolidation led to less capacity being installed, and hence that the deterministic solutions (which are the same irrespective of correlations) were less off. Consider Figure 2 as an example. The square nodes (nodes 1, 2, 3 and 4) were the supply nodes, though only three of them ended up being used in this case. Nodes 12, 14, 15 and 16 are demand nodes, the rest transshipment
125
nodes (which in this specific example ended up not being used). Demand nodes are split into two groups, one containing nodes 12 and 14, the other nodes 15 and 16. Correlations between demands within a group are positive, while correlations between demands across groups are negative. We observe that positive correlations lead to higher installed capacities. For negative correlations, we
130
also see a loop including nodes 12, 14 and 15, used for consolidation. References [1] I. Ghamlouche, T. G. Crainic, M. Gendreau, Path relinking, cycle-based neighbourhoods and capacitated multicommodity network design, Annals 6
6
9
13 6
9
13
5 4 2
8
46 40
5
11
4 2
3 139
1 12 7
48
15
37
11
3 134
1 12
16
38
8
37 36
7
39 33 16
40
10 10
14
13 14
6
9
13
5 4 2
8
32 32
11
3 128
1 12 7
32
15
32 16
32 10 14
Figure 2: Network designs for a case of correlated demands. In the upper left graph, demands are positively correlated, while in the upper right correlations are mixed. In the bottom graph, we find the deterministic solution. The numbers on the edges indicate installed capacity.
of Operations Research 131 (1) (2004) 109–133. 135
[2] S. Wallace, Decision making under uncertainty: Is sensitivity analysis of any use?, Operations Research 48 (2000) 20–25. doi:10.1287/opre.48. 1.20.12441. [3] B. K. Thapalia, T. G. Crainic, M. Kaut, S. W. Wallace, Single source single-commodity stochastic network design, Computational Management
140
Science 9 (1) (2012) 139–160, special issue on ‘Optimal decision making under uncertainty’. doi:10.1007/s10287-010-0129-0. [4] B. K. Thapalia, T. G. Crainic, M. Kaut, S. W. Wallace, Single-commodity stochastic network design with multiple sources and sinks, INFOR: Information Systems and Operational Research 49 (3) (2011) 193–211. doi:
145
10.3138/infor.49.3.003. 7
15
[5] B. K. Thapalia, T. G. Crainic, M. Kaut, S. W. Wallace, Single-commodity network design with random edge capacities, European Journal of Operational Research 220 (2) (2012) 394–403. [6] A.-G. Lium, T. G. Crainic, S. W. Wallace, A study of demand stochasticity 150
in stochastic network design, Transportation Science 43 (2) (2009) 144–157. doi:10.1287/trsc.1090.0265. [7] F. Maggioni, S. W. Wallace, Analyzing the quality of the expected value solution in stochastic programming, Annals of Operations Research 200 (1). doi:10.1007/s10479-010-0807-x.
155
[8] J. R. Birge, The value of the stochastic solution in stochastic linear programming with fixed recourse, Mathematical Programming 24 (1982) 314– 325. [9] M. Kaut, A copula-based heuristic for scenario generation, Computational Management Science 11 (4) (2014) 503–516.
160
doi:10.1007/
s10287-013-0184-4. [10] M. Kaut, S. W. Wallace, Evaluation of scenario-generation methods for stochastic programming, Pacific Journal of Optimization 3 (2) (2007) 257– 271.
8