- Email: [email protected]
A new directional simulation method for system reliability. Part II: application of neural networks
Probabilistic Engineering Mechanics 19 (2004) 437–447 www.elsevier.com/locate/probengmech
A new directional simulation method for system reliability. Part II: application of neural networks Jinsuo Niea,*, Bruce R. Ellingwoodb b
a Department of Energy Sciences and Technology, Brookhaven National Laboratory, Bldg. 130, Upton, NY 11973-5000, USA College of Engineering Distinguished Professor, School of Civil and Environmental Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA
Received 4 June 2003; accepted 22 March 2004
Abstract A challenge in directional importance sampling is in identifying the location and the shape of the importance sampling density function when a realistic limit state for a structural system is considered in a finite element-supported reliability analysis. Deterministic point refinement schemes, previously studied in place of directional importance sampling, can be improved by prior knowledge of the limit state. This paper introduces two types of neural networks that identify the location and shape of the limit state quickly and thus facilitate directional simulation-based reliability assessment using the deterministic Fekete point sets introduced in the companion paper. A set of limit states composed of linear functions are used to test the efficiency and possible directional preference of the networks. These networks are shown in the tests and examples to reduce the simulation effort in finite element-based reliability assessment. q 2004 Elsevier Ltd. All rights reserved. Keywords: Computational mechanics; Directional importance sampling; Neural networks; Probability; Reliability; Statistics
1. Introduction The error associated with directional simulation in structural reliability assessment arises from two sources: high F-discrepancy [12] of the point set (a measure for how well the set preserves the probability distribution F) and, in situations when the limit state is defined implicitly by finite element (FE) analysis, lack-of-information regarding the limit state. The first source has been investigated extensively in a companion paper [12], in which Fekete point sets were shown to have low F-discrepancy and to reduce the error from poor representation of the underlying distribution. The second source refers to the inability to describe the limit state adequately if the number of samples is limited. This error can result from high F-discrepancy, since the samples may cluster at locations away from the limit state when they are not scattered ‘evenly’ in the probability space. It can also result from some unfavorable locations or shapes of the limit state in question. For example, when the limit state is well away from the center of the distribution, * Corresponding author. Tel.: þ 1-410-516-8443; fax: þ1-410-516. E-mail address: [email protected] (J. Nie). 0266-8920/$ - see front matter q 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.probengmech.2004.03.005
naive Monte Carlo simulation may yield an insufficient number of failed samples even if the number of samples is very large. A similar situation can happen in directional simulation. When the limit state is convex with respect to the origin and the failure probability is small (see Fig. 1), the description of the limit state may be inadequate if the number of directions that intercept the limit state is small. Attempts to reduce the error in limit state description by applying a larger set of points may be inefficient because most of the directions do not yield relevant information regarding the limit state and the supporting FE analysis is costly. Directional importance sampling is one way to focus only on the regions that contribute significantly to the failure probability [7,9]. However, the location and the shape of the directional importance density function are difficult to identify. Furthermore, the randomly generated samples in importance directional sampling do not describe the probability space as efficiently as do deterministic point sets [12]. This paper investigates point set refinement schemes based on neural networks that can focus a finer point set in specific regions of interest, do not require prior knowledge of the limit state, and can be easily automated.
438
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
Fig. 1. Concave and convex limit states.
2. Critique of current deterministic refinement schemes As an alternative to random generation in directional simulation, Yonezawa and Okuda [17] generated directions using the centerlines of finite meshes that divide the unit hypersphere by equally dividing the angular coordinates in the polar coordinate system. In this method, a two-phase refinement scheme included a search phase using rough meshes to locate the important regions, and a refinement phase involving finer meshes confined to the important regions to compute the failure probability. This approach divides the hypersphere unevenly, has a high dimensionality effect (the number of centerlines increases exponentially with the dimension), and shows no apparent improvement over directional simulation (a concern raised in the example involving four random variables in their paper [17]). The Advanced Hyperspace Division Method (AHDM) [6], formulated in standard normal probability space, also used a two-phase strategy to improve efficiency. In the first phase, a coarse AHDM point set was applied and the radii associated with these points were computed. The directions which have radii smaller than a predefined threshold define the most important regions. These selected directions were then grouped to regions and finer points were applied only in those regions. A grouping technique based on the correlation between points (directions) was adopted to define those regions (Katsuki, private communication). Dr Katsuki also noted that the grouping technique may result in error if knowledge about the limit state function is limited. This grouping technique may lead to overlapping of groups in some cases, especially in high dimensional spaces, which may result in large error. A new two-phase refinement scheme is developed using neural networks and Fekete points in this paper. In the first phase, the network is trained to identify the most significant search directions; in the second phase, the reliability is estimated. Neural networks will be used to construct the superregion, i.e. the assembly of the most significant regions which may be overlapped or even
Fig. 2. Illustration of effective region, points and limit state.
located separately, and to filter out the portions of the finer point set that are not in the superregion. The effective region is denoted the superregion on the unit hypersphere which contributes most to the failure probability; and the effective points as the subset of a point set lying within the effective region. The effective limit state is the part of the limit state corresponding to the effective region (see Fig. 2).
3. Application of neural networks Neural networks are mathematical models that approximate complex functions in a similar way that the human brain processes and remembers information [15]. They have been investigated as surrogates to complex structural systems in structural reliability analysis [4,13]. The basic structure of a neural network includes nodes and connections which link the nodes. Each connection has an associated weight property. Weights are the principal mechanism by which a neural network stores information and is trained; training is achieved by adjusting the weights so as to make the output of the network match the predefined goal (in the sense of supervised learning). A node is depicted in Fig. 3, where the output ao is called the node’s activation value, and is given by ao ¼ f ðdð~ai ÞÞ:
ð1Þ
Fig. 3. Illustration of a node.
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
439
f is the transfer function, d is the input function, a~ i is the input vector, and wij is the weight. A network must have nonlinear transfer functions in order to approximate a complex function. Common transfer functions include the step, sign and logsig functions as described below: ( 1; for x $ t stepðx; tÞ ¼ 0; otherwise ( 1; for x $ 0 ð2Þ signðxÞ ¼ 21; otherwise logsigðxÞ ¼ 1=ð1 þ e2x Þ: The input function d can be a distance function or a simple weighted-sum of the input vector. A converged neural network can then predict outputs from new inputs. In terms of statistics, the training process of a neural network is mathematically similar to nonlinear regression [15]. Two common types of neural networks, namely feedforward backpropagation networks (FFBPN) and radial basis networks (RBN), will be investigated as tools for directional point set refinement. 3.1. Feed-forward back-propagation network The effective region in deterministic directional methods is a union of small hypercaps [12], which are the approximated neighbors of the effective points identified in the first phase. Any of these hypercaps can be obtained by cutting the unit hypersphere with a hyperplane. These small overlapped hypercaps can then be grouped into larger clusters, which are larger hypercaps and can be obtained by cutting the sphere with hyperplanes as well. If the limit state is linear and the sampling points are uniformly scattered, then the effective region (hypercap) can be cut by one hyperplane, which is parallel to the limit state (see Fig. 4). This effective hypercap region can also be imagined as the radial projection of the effective limit state onto the sphere. If the effective points are assigned value 1 and other points are assigned value 0, then this binaryvalued function over the whole point set is a linearly separable function. The perceptrons are the conceptual building blocks of the FFBPN. A perceptron is a simple layered feed-forward network with no hidden layers; it uses a step transfer function, can only learn a linearly separable function but can adequately represent the effective region of a linear limit state function. Most realistic limit states are nonlinear, and thus perceptrons are not directly useful in such cases. However, the effective region of a general limit state can be approximated by hypercaps, each of which can be represented by a perceptron. The number of perceptrons required in this approximation is bounded from above by the number of effective points; however a lesser number should be used to avoid network overfitting. The desired network
Fig. 4. Effective region of linear limit state by one cutting hyperplane.
should be able to approximate the binary-valued function, which is defined in the case of a linear limit state. If the number of perceptrons in the approximation is M; then a two-layer network is constructed using the M perceptrons as the hidden layer and one perceptron as the only node in the output layer to merge the outputs of the hidden layer. Such a network design represents the union of the hypercaps, which defines the effective region. Fig. 5 shows the design of such a network, in which some necessary modifications to the perceptrons have been made to adopt the above concept into a practical FFBPN. The transfer functions have been changed to the differentiable logsig function, the shape of which is close to that of a step function (see Fig. 6), which is necessary for backpropagation learning. Bias is included in each node and the position t of a step function is set to 0. The input function d is defined as X ~ x þ b: dð~xÞ ¼ wi xi þ b ¼ w~ ð3Þ There are many training algorithms for an FFBPN. The gradient descent algorithm is a typical example of error back-propagation [10,15], but is often too slow for practical usage. Faster methods include gradient descent with momentum, variable learning rate, conjugate gradient, Levenberg-Marquardt algorithm and many others. The Levenberg-Marquardt algorithm, which is reported to be the fastest algorithm for a FFBPN of moderate size (‘up to several hundred weights’ [2]), will be used herein, since the examples in this paper are of moderate size. (Referring back to the network in Fig. 5, there are 126 weights and seven biases in a network of 6 hidden nodes for a problem of
440
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
Fig. 5. Illustration of a two layer FFBPN.
20 random variables.) Fig. 7 shows a typical convergence curve in training a FFBPN in terms of epochs, each of which contains a full cycle of updating all the weights for all the inputs. Neural network training can be made more efficient if the inputs and targets are within certain bounds. The inputs and targets are usually converted into range [2 1,1]; a converged neural network will then produce output in the range [2 1,1]. One attraction of the FFBPN in point set refinement is that the input data are within the range [2 1,1] even without such a conversion; since the points are on the unit hypersphere, the targets are purposely assigned value 0 and 1, and the range of logsig transfer function is (0,1). Caution has to be exercised in selecting M; the number of hidden nodes. It is generally easier for the network to converge if M is larger. However a larger M may not yield a more accurate estimate of failure probability, since the network may be overfitted. In that case, the converged
network cannot be generalized to other data from the same domain. Fig. 8 compares a properly fitted FFBPN to one that is overfitted on the spheres. Both networks are trained with the same input data (coarse dots in first phase) and tested with the same simulation data (fine dots in second phase). In general, the adjustable parameters in a network should be less than the number of samples in the training set of data. In addition, a simpler network usually yields better predictions when used with a new set of data [2]. For an FFBPN constructed as in Fig. 5 to have good generalization, M and the number of training samples, m; should satisfy the following inequality [2] ðn þ 2ÞM þ 1 , m;
ð4Þ
where n is the dimension of the problem. To construct an FFBPN which is as simple as possible, a special procedure is followed in this study. The network is always constructed with M ¼ 1 at first, which is repeated up
Fig. 6. Comparison of step and logsig functions.
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
441
Fig. 7. A typical convergence curve for FFBPN.
to 10 training cycles; during each cycle, the training restarts with a new randomized set of weights and biases and proceeds for 100 epochs. If the network has not converged after 10 trials, the network is reconstructed with M ˆ M þ 1; and the above procedure is repeated. If convergence has not been achieved before M gets larger than required by Eq. (4), then the number of samples is judged insufficient to represent the limit state, and additional samples are supplied. The convergence criteria in the training is set to 10212 based on trial and error. Root searching procedures are usually used to calculate the radius along each direction for a complicated limit state. A good starting guess of radii can speed up the search. Unfortunately, an FFBPN is designed mainly as a filter on directions, and provides no information on radius along any direction that has been selected in the second phase. The RBN, introduced in the next section, yields a reasonable estimate of radius as well as filtering directions. 3.2. Radial basis network If the limit state can be approximated in such a way that the radius for an arbitrary direction can be quickly determined in the first phase, the approximated limit state can serve as a filter for the region of interest and yield an initial radius estimate to initialize the root searching procedure. A response surface is one way to approximate the limit state [8,14,16]; however it is only valid around the design point. For a highly nonlinear limit state, a response surface, which usually takes a quadratic form, is not adequate to express the limit state in the whole domain. A RBN is a technique for multi-variable interpolation, and can approximate a highly nonlinear limit state in the whole domain very well if designed properly. A RBN is a twolayer network, as shown in Fig. 9. The configuration of a RBN is similar to that of a FFBPN, except that the input function to the hidden nodes is a factored distance function rather than a weighted sum. Assuming an Euclidean
Fig. 8. Comparison of a properly fitted and an overfitted FFBPN representing a linear limit state.
distance measure, the factored distance is defined by ~ bÞ ¼ bl~x 2 wl; ~ distð~x; w;
ð5Þ
~ acts as the center of the transfer where the weight vector w function, and bias b controls the shape of the transfer function. The distance can also be spherical if the RBN is constructed on the hypersphere, i.e. ~ bÞ ¼ b arccosð~x·wÞ: ~ distð~x; w;
ð6Þ
The network should be insensitive to the choice of input function if b is chosen appropriately [1]. The transfer function is one of the radial basis functions. Any univariate function which approaches 0 when its argument
442
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
Fig. 9. Schematic of RBN.
approaches 1 can be a radial basis function; a common choice is, 2
f ðxÞ ¼ e2x :
ð7Þ
The radial basis function is localized such that only inputs near the center can produce nontrivial values (see f ðxÞ on a sphere in Fig. 10). The output node has the same input function as the output node in the FFBPN, but the transfer function is linear. A RBN can be designed by one of the following two methods. In the first (exact) method, each input point is used as the center of the radial basis function of one hidden node, and the weights of the second layer are selected to make the network error equal to zero by solving a set of linear equations. However, this method yields an inefficient design because it creates as many hidden nodes as the number of inputs in the training data. In the second (iterative) method, the input points are added to the network iteratively. In each iteration, only the point which lowers the network error most is added. This process is repeated until the network error is below a predefined goal or the inputs in the training data are used up. In this study, the second method is used when the number of points in the first phase set (cardinality) is small; otherwise the first method is used. In order to make the function approximated by the RBN smooth for both methods, bias b has to be small enough to make the transfer functions of the hidden nodes overlap. However to avoid underfitting, b should not be so small that all hidden nodes respond essentially the same. In this research, b is chosen as 0:8326=Rc ; where Rc is the covering radius of the Fekete set in the first phase [10]. The parameter 0.8326 is chosen such that f ð^0:8326Þ ¼ 0:5 (in Eq. (7)). The radii cannot be used directly as the targets of the RBN, since the significant directions have smaller radii
while others have larger radii. The reciprocals of the radii are used as the values for the RBN to interpolate (see Fig. 11). Let R^ min be the minimal radius among all the directions in the first phase. The threshold for significant radii is selected as R^ min þ 5; which is larger than those suggested by Harbitz [3] or Yonezawa and Okuda [17].
4. Efficiency assessment Several limit state functions are listed in Table 1. These are used to assess the efficiency and accuracy of the two
Fig. 10. A radial basis function f ðxÞ on the sphere.
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
443
Fig. 11. Radial basis interpolation.
kinds of networks. Function 0 has been considered in previous research [5,6,11]. Functions 0 – 5 are linear and intended for testing whether the proposed method has directional preferences. Functions 6 – 7 are nonlinear functions representing serial and parallel systems respectively. Function 6 and 8 have two design points; in particular, function 8 has the two design points in opposite directions. All test limit states are selected intentionally as linear functions or are composed of linear functions, so that the calculations of the radii are inexpensive. However, conclusions drawn from these tests are still valid for realistic problems because the projection of a realistic limit state on the hypersphere can be considered as an assembly of hypercaps cut by hyperplanes. The limit state probabilities of functions 0-8 in dimensions 3 and 12 assessed using the FFBPN and RBN methods are shown in Tables 2– 5. Errors are computed against the ‘exact’ values, i.e. the theoretical values in Table 1 or estimates for functions 6 and 7 by multi-modal Importance Monte Carlo Simulation (MIMCS) using 100,000 samples. The failure probabilities by MIMCS for limit state 6 in 3D and 12D are 2:56640 £ 1023 and 2:68964 £ 1023 ; while the standard deviations in these estimates are 1:23872 £ 1025 and 1:55402 £ 1025 ; respectively. The failure probabilities by MIMCS for limit state 7 in 3D and 12D are 1:24146 £ 1024 and 2:26780 £ 1025 ; while the standard deviations of these estimates are 2:51980 £ 1026 and 3:16872 £ 1027 ; respectively. Two point sets F3-12-5 and F3-36-10 in 3D are used to test the effect of the cardinality in the first phase. (Recall that F3-12-5 means a 3D Fekete point set of 12 points, which satisfy the spherical t-design lower bound for t ¼ 5:) The applications of FFBPN and RBN are compared by the examples in Tables 2 and 3, in which limit state function 2 is analyzed using F3-12-5 and F3-240-29 as
the sets for the two phases. In the first phase, the limit state probability is calculated by the Fekete point method; the error compared to the exact solution is 10.6%. For the FFBPN (Table 2), three points out of the 12 points are identified as having significant contributions to the failure probability, and are assigned value 1. All other points are assigned value 0. A FFBPN is created with one hidden node. The weights and biases are randomly initialized. The 12 points are the input vectors for this FFBPN, and the values associated with these points define the target vector. The network is then trained until it converges. The network is fed with set F3-240-29, and 62 points are filtered out. These 62 points are then used to reanalyze this limit state probability. The error drops down to 2 0.445% with a total effort of 74 evaluations of the limit state function, rather than 240 evaluations. If directional simulation with 10,000 random directions is used for this limit state, the failure probability estimate, its standard deviation and its coefficient of variation (COV) are 1.33275 £ 1023, 4.54516 £ 1025, and 3.41%, respectively. In Monte Carlo simulation, the COV in an estimate Table 1 Test functions for point set refinement methods Func. ID
Limit state function
Exact failure probability
0 1 2 3 4 5 6 7 8
P pffiffi 2 xi þ 3 n ¼ 0 P pffiffi xi þ 3 n ¼ 0 2x1 þ 3 ¼ 0 x1 þ 3 ¼ 0 2x2 þ 3 ¼ 0 x2 þ 3 ¼ 0 Func. 0 OR Func. 2 Func. 0 AND Func. 2 Func. 2 OR Func. 3
1.34990 £ 1023 1.34990 £ 1023 1.34990 £ 1023 1.34990 £ 1023 1.34990 £ 1023 1.34990 £ 1023 MC MC 2.69980 £ 1023
444
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
Table 2 Test result of FFBPN in 3D Func.ID
0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 7 7 8 8
Hidden node #
2 1 2 1 1 1 2 1 2 1 2 1 1 2 1 1 1 1 2 2
First phase
Second phase
Point set
# Eff. points
Error (%)
Point set
# Eff. points
P^ f ð£1023 Þ
Error (%)
F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10
4 11 4 11 3 12 3 11 3 11 3 11 5 17 3 6 3 6 6 23
35.4 .591 35.4 3.24 10.6 1.83 10.6 2.08 214.4 23.10 214.4 1.38 29.2 5.14 295.5 272.4 295.5 272.4 10.6 1.95
F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-1200-67 F3-1200-67 F3-240-29 F3-240-29
65 72 64 53 62 58 54 64 66 61 58 71 91 97 37 43 192 217 134 120
1.34544 1.34950 1.34551 1.34928 1.34396 1.34837 1.35072 1.35186 1.35003 1.34894 1.34899 1.34975 2.52625 2.57574 0.12096 0.12208 0.12298 0.12384 2.63741 2.70022
20.335 20.034 20.330 20.051 20.445 20.118 20.097 0.140 0.005 0.011 20.072 20.016 21.57 0.364 22.56 21.67 20.938 20.243 22.32 0.010
reflects, by definition, a 68% confidence interval within which the true value falls, and is similar to the notion of relative error. Thus, if the relative error in the approach herein is considered to be comparable to the COV of the failure probability estimate, the proposed approach achieves a higher accuracy with substantially fewer points than directional simulation (about 99.3% of saving w.r.t. directional simulation).
For the RBN (Table 3), the reciprocals of the radii associated with the 12 points are the targets of the RBN. The RBN is constructed by the iterative method. A total of 11 points are added as the hidden nodes, which is larger than that of the FFBPN. The network is then used to filter the set F3-240-29, and 59 points out of the 240 points are selected by this RBN. The error drops down to 2 0.196% with a total effort of 71 evaluations of the limit state function.
Table 3 Test result of RBN (iterative method) in 3D Func.ID
0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 7 7 8 8
Hidden node #
10 25 10 24 11 23 11 23 11 24 10 25 11 29 8 19 8 19 7 33
First phase
Second phase
Point set
# Eff. points
Error (%)
Point set
# Eff. points
P^ f ð£1023 Þ
Error (%)
F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10
4 11 4 11 3 12 3 11 3 11 3 11 5 17 3 6 3 6 6 23
35.4 .591 35.4 23.24 10.6 1.83 10.6 2.08 214.4 23.10 214.4 1.38 29.2 5.14 295.5 272.4 295.5 272.4 10.6 1.95
F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-1200-67 F3-1200-67 F3-240-29 F3-240-29
42 62 65 61 59 67 55 64 58 62 61 62 91 98 48 37 250 176 105 132
1.34871 1.34948 1.33483 1.34990 1.34732 1.34837 1.35000 1.35176 1.34978 1.34990 1.34961 1.34975 2.57360 2.57573 0.12212 0.12213 0.12388 0.12388 2.69589 2.70023
20.094 20.036 20.007 20.005 20.196 20.118 20.002 0.132 0.014 0.012 20.027 20.016 0.281 0.364 21.63 21.63 2.212 2.211 2.150 0.011
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
445
Table 4 Test result of FFBPN in 12D Func.ID
0 1 2 3 4 5 6 7 8
Hidden node #
1 1 1 1 1 1 2 2 2
First phase
Second phase
Point set
# Eff. points
Error (%)
Point set
# Eff. points
P^ f ð£1023 Þ
Error (%)
F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5
15 26 20 22 24 24 38 7 43
242.1 227.2 29.97 24.03 215.9 55.9 225.8 298.9 27.00
F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11
1650 2328 1701 2229 1826 1972 3525 950 3872
1.32540 1.35353 1.32960 1.33852 1.32540 1.34618 2.61738 0.03622 2.66810
21.82 20.263 21.51 20.848 2.61 20.281 22.69 59.7 21.18
To summarize the results in Tables 2– 5, the FFBPN uses fewer hidden nodes than the RBN, and consequently results in more efficient networks in filtering the point set in the second phase (set II). For all test functions, the FFBPN uses only 1 or 2 hidden nodes in spite of the problem dimension. The number of hidden nodes used by the FFBPN is affected only by the nature of the limit state. On the other hand, for the RBN, the number of hidden nodes used by the iterative method depends on the number of effective points and the cardinality of the point set in the first phase (set I), while the exact method uses up all the points in set I. The effectiveness of a refinement method can be evaluated by the number of effective points filtered out in the second phase. In general, the RBN produces a smaller set of effective points in the second phase than the FFBPN, especially in high-dimensional space. For example, the FFBPN produces 2 –19 times more effective points than the RBN in 12D, which would require a corresponding increase in calls to the FE routine for a realistic structural reliability problem. The RBN requires more time in filtering set II than the FFBPN because of its larger hidden layer. However, the filtering time is usually marginal compared to the time spent in FE analysis. Both methods can dramatically decrease the number of points to be used as FE samples in structural reliability analysis. As another example for limit state 0 in 12D, the use of the RBN reduces the number of points from 13587 to 906 (156 þ 750), which is a saving of about 93.3%.
Although a set I of larger cardinality usually yields smaller error in the first phase, it has little effect on the final error, as shown in Tables 2 and 3. In this sense, set I is used only for finding the effective region of the limit state. It is usually sufficient to select a point set with t ¼ 5; in terms of spherical t-design [10 – 12]. Only when a very small number of effective points is selected in the first phase, say less than 5% of set I, may one want to try a larger set to get more information about the limit state and a better approximation of its effective region. The final error after set II has been applied is largely dependent on the cardinality of set II and the shape of the limit state. For a given set II, the application of a properly designed neural network leads to an improvement in efficiency without significant loss of accuracy. The final error is smaller for a concave failure domain than for a convex one. The higher error for convex limit states with high local curvatures, e.g. for limit state 7 in Tables 2– 5, is true for all directional methods including this one. In this case, a larger set II can be applied to get a smaller error.
5. Examples of quadratic limit state functions Quadratic functions are an important class of limit states because they frequently are used to define a response
Table 5 Test result of RBN (exact method) in 12D Func.ID
0 1 2 3 4 5 6 7 8
Hidden node #
156 156 156 156 156 156 156 156 156
First phase
Second phase
Point set
# Eff. points
Error (%)
Point set
# Eff. points
P^ f ð£1023 Þ
Error (%)
F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5
15 26 20 22 24 24 38 7 43
242.1 227.2 29.97 24.03 215.9 55.9 225.8 298.9 27.00
F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11
750 699 737 681 593 680 1542 50 1158
1.31800 1.35353 1.34313 1.32412 1.31909 1.35641 1.33275 2.59163 0.02747
22.37 20.507 21.92 22.29 0.477 21.28 23.64 221.1 22.48
446
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
surface approximation to a real limit state around the design point [8,14,16]. The effective ratio is the number of effective points over the cardinality of the whole point set. The effort ratio is defined by the number of points actually used, which is the sum of the cardinality of the first phase point set and the number of effective points in second phase, divided by the number of points required if a neural network had not been applied to refine the point selection, i.e., the cardinality of second phase point set. The limit state function Gconcave ¼ 20:5ðz21 þ z22 þ z23 2 2z1 z2 2 2z2 z3 2 2z3 z1 Þ pffiffi 2 ðz1 þ z2 þ z3 Þ= 3 þ 3 ð8Þ
Fig. 12. The spherical projection of Gconcave shown by its effective points.
Fig. 13. The spherical projection of Gconvex shown by its effective points.
represents the boundary of a concave failure domain in 3D, the effective region of which (spherical projection of the effective limit state on the sphere) is a circular stripe (see Fig. 12). By directional simulation, the exact failure probability found using 10,000 samples was 0.19798, with sampling error 1:56 £ 1023 and COV 0.79% [11]. The result from the point set/neural network analysis is shown in Table 6. The FFBPN method achieves an error of 0.13% with 212 points, and the RBN achieves 0.02% with 226 points. The proposed method yields smaller error than directional simulation, and achieves about 98% of saving in samples. The RBN has a higher effort ratio, but achieves a higher accuracy than the FFBPN. However, the improvement in efficiency using neural networks is not significant in this example because the shape of the limit state has a large spherical projection (effective region). For the limit state in Eq. (8), the RBN uses 36 hidden nodes, while the FFBPN uses only two hidden nodes. That is, the RBN yields a more complex design than does the FFBPN. It is interesting to note that although the FFBPN was conceptually constructed with imaginary hyperplanes (perceptrons) cutting the hypersphere, in this particular example the circular-stripe-shaped effective region of Gconcave cannot be cut directly from the sphere by only two planes. However, when the area complementary to the effective region on the sphere is considered, it is obvious that the effective region can still be cut from the sphere with two cutting planes, i.e. the FFBPN can identify a region by its complement on the sphere.
Table 6 Failure probabilities for quadratic limit state functions First phase
Second phase
Effort ratio (%)
Limit state
Point set
Eff. points
1ð%Þ
Network
Point set
pf
1ð%Þ
Eff. points
Gconcave Gconcave Gconvex Gconvex
F3-36-10 F3-36-10 F3-36-10 F3-36-10
27 27 4 4
1.45 1.45 13.6 13.6
FFBPN-2 RBN-36 FFBPN-2 RBN-36
F3-240-29 F3-240-29 F3-240-29 F3-240-29
0.19773 0.19801 0.01904 0.01921
20.13 0.02 21.37 20.49
176 190 33 33
87.5 94.2 28.8 28.8
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
A convex failure domain can be defined by the following limit state function: Gconvex ¼ 0:5ðz21 þ z22 þ z23 2 2z1 z2 2 2z2 z3 2 2z3 z1 Þ pffiffi 2 ðz1 þ z2 þ z3 Þ= 3 þ 3:
ð9Þ
Gconvex is depicted by its effective points in Fig. 13, which consists of two separate parts located at opposite areas on the sphere. The exact failure probability 1:93043 £ 1022 was obtained by directional simulation using 10,000 directions; the estimated sampling error is 8:04 £ 1024 and the COV is 4.16% [11]. The COV of this probability estimate is larger than that of Gconcave ; as expected. The results of the point set/neural network analysis are shown in Table 6. The errors yielded by the FFBPN and the RBN are 1.37 and 0.49%, respectively. Once again, the proposed method yields smaller error than directional simulation, and uses only 69 points for both networks (about 99.3% of saving w.r.t. directional simulation). For this convex failure domain, the application of neural networks yields an effort ratio of only 28.8%, which is more than 70% saving in effort. The FFBPN uses two hidden nodes, while the RBN uses all 36 points of first phase. The error in the FFBPN is larger than in the RBN although the same number of total points are used, which indicates that the FFBPN selects different points from those of the RBN.
6. Conclusion This paper introduced a two-phase strategy of point set refinement to reduce the number of evaluations of the limit state function in directional simulation. Two types of neural networks, the FFBPN and the RBN, have been explored and shown by examples to be effective in this regard. These methods are implemented with Fekete point sets to identify directions, which is introduced in a companion paper [12]. The combination of the Fekete point method and neural network techniques has been used successfully to assess the reliability of more complex structural systems [10]. The effective ratios of the first phase and the second phase of the directional search refinement process are roughly the same. A large effective ratio implies a relative smooth limit state function and a large effort ratio. Therefore, the benefit of the neural network refinement may be less significant when the effective ratios are high. For Gconcave (Eq. (8)), where the effective ratio is about 75%, a decrease in error from 1.45 to 0.13% or 0.02% with an effort ratio of around 90% does not appear to be costeffective, when compared with other examples such as
447
Gconvex : In this case, the effective ratio from the first-phase analysis is helpful in choosing a point set for the second phase to avoid the high computational cost. Because the failures of structural systems usually occur at high values of loads and low values of resistances, practical structural reliability problems generally have low effective ratios, and thus the use of neural networks as adjuncts to directional simulation algorithms may lead to significant saving in practical system reliability assessment [10].
References [1] Bishop CM. Neural Network for Pattern Recognition. New York: Oxford University Press; 1995. [2] Hagan MT, Demuth HB, Beale M. Neural Network Design. Boston: PWS Publishing Company; 1996. [3] Harbitz A. An efficient sampling method for probability of failure calculation. Struct Safety 1986;3:109– 15. [4] Hurtado JE, Alvarez DA. Neural-network-based reliability analysis: a comparative study. Comput Meth Appl Mech Engng 2001;191: 113–32. [5] Katsuki S, Frangopol DM. Hyper-space division method for structural reliability. J Engng Mech, ASCE 1994;120(11):2405–27. [6] Katsuki S, Frangopol DM. Advanced hyperspace division method for structural reliability. In: Shinozuka W, editor. Struct Safety Reliab. 1998. p. 631–8. [7] Kijawatworawet W, Pradlwarter HJ, Schue¨ller GI. Structural reliability estimation by adaptive importance directional sampling. In: Shiraishi N, Shinozuka M, Wen YK, editors. The Seventh International Conference on Structural Safety and Reliability, vol. 2. 1998. p. 891 –7. [8] Kim S, Na S. Response surface method using vector projected sampling points. Struct Safety 1997;19(1):3– 19. [9] Melchers RE. Radial importance sampling for structural reliability. J Engng Mech, ASCE 1990;116(1):189–203. [10] Nie J. A new directional method to assess structural system reliability in the context of performance-based design. PhD Thesis. The Johns Hopkins University, Baltimore, MD; 2003. [11] Nie J, Ellingwood BR. Directional methods for structural reliability analysis. Struct Safety 2000;22:233–49. [12] Nie J, Ellingwood BR. A new directional simulation method for system reliability. Part I: Application of deterministic point sets. Probab Engng Mech 2004; in press. [13] Papadrakakis M, Papadopoulos V, Lagaros ND. Structural reliability analysis of elastic-plastic structures using neural networks and Monte Carlo simulation. Comput Meth Appl Mech Engng 1996;136:145– 63. [14] Rajashekhar MR, Ellingwood BR. A new look at the response surface approach for reliability analysis. Struct Safety 1993;12:205 –20. [15] Russell SJ, Norvig P. Artificial Intelligence: A Modern Approach. Englewood Cliffs, NJ: Prentice Hall; 1995. [16] Yao TH, Wen YK. Response surface method for time-variant reliability analysis. J Struct Engng, ASCE 1996;122(2):193–201. [17] Yonezawa M, Okuda S. Directional vectors approximation for assessment of structural failure probability. In: Shiraishi, Shinozuka, Wen, editors. Struct Safety Reliab. Rotterdam: Balkema; 1998. p. 611–6.
A new directional simulation method for system reliability. Part II: application of neural networks Jinsuo Niea,*, Bruce R. Ellingwoodb b
a Department of Energy Sciences and Technology, Brookhaven National Laboratory, Bldg. 130, Upton, NY 11973-5000, USA College of Engineering Distinguished Professor, School of Civil and Environmental Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA
Received 4 June 2003; accepted 22 March 2004
Abstract A challenge in directional importance sampling is in identifying the location and the shape of the importance sampling density function when a realistic limit state for a structural system is considered in a finite element-supported reliability analysis. Deterministic point refinement schemes, previously studied in place of directional importance sampling, can be improved by prior knowledge of the limit state. This paper introduces two types of neural networks that identify the location and shape of the limit state quickly and thus facilitate directional simulation-based reliability assessment using the deterministic Fekete point sets introduced in the companion paper. A set of limit states composed of linear functions are used to test the efficiency and possible directional preference of the networks. These networks are shown in the tests and examples to reduce the simulation effort in finite element-based reliability assessment. q 2004 Elsevier Ltd. All rights reserved. Keywords: Computational mechanics; Directional importance sampling; Neural networks; Probability; Reliability; Statistics
1. Introduction The error associated with directional simulation in structural reliability assessment arises from two sources: high F-discrepancy [12] of the point set (a measure for how well the set preserves the probability distribution F) and, in situations when the limit state is defined implicitly by finite element (FE) analysis, lack-of-information regarding the limit state. The first source has been investigated extensively in a companion paper [12], in which Fekete point sets were shown to have low F-discrepancy and to reduce the error from poor representation of the underlying distribution. The second source refers to the inability to describe the limit state adequately if the number of samples is limited. This error can result from high F-discrepancy, since the samples may cluster at locations away from the limit state when they are not scattered ‘evenly’ in the probability space. It can also result from some unfavorable locations or shapes of the limit state in question. For example, when the limit state is well away from the center of the distribution, * Corresponding author. Tel.: þ 1-410-516-8443; fax: þ1-410-516. E-mail address: [email protected] (J. Nie). 0266-8920/$ - see front matter q 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.probengmech.2004.03.005
naive Monte Carlo simulation may yield an insufficient number of failed samples even if the number of samples is very large. A similar situation can happen in directional simulation. When the limit state is convex with respect to the origin and the failure probability is small (see Fig. 1), the description of the limit state may be inadequate if the number of directions that intercept the limit state is small. Attempts to reduce the error in limit state description by applying a larger set of points may be inefficient because most of the directions do not yield relevant information regarding the limit state and the supporting FE analysis is costly. Directional importance sampling is one way to focus only on the regions that contribute significantly to the failure probability [7,9]. However, the location and the shape of the directional importance density function are difficult to identify. Furthermore, the randomly generated samples in importance directional sampling do not describe the probability space as efficiently as do deterministic point sets [12]. This paper investigates point set refinement schemes based on neural networks that can focus a finer point set in specific regions of interest, do not require prior knowledge of the limit state, and can be easily automated.
438
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
Fig. 1. Concave and convex limit states.
2. Critique of current deterministic refinement schemes As an alternative to random generation in directional simulation, Yonezawa and Okuda [17] generated directions using the centerlines of finite meshes that divide the unit hypersphere by equally dividing the angular coordinates in the polar coordinate system. In this method, a two-phase refinement scheme included a search phase using rough meshes to locate the important regions, and a refinement phase involving finer meshes confined to the important regions to compute the failure probability. This approach divides the hypersphere unevenly, has a high dimensionality effect (the number of centerlines increases exponentially with the dimension), and shows no apparent improvement over directional simulation (a concern raised in the example involving four random variables in their paper [17]). The Advanced Hyperspace Division Method (AHDM) [6], formulated in standard normal probability space, also used a two-phase strategy to improve efficiency. In the first phase, a coarse AHDM point set was applied and the radii associated with these points were computed. The directions which have radii smaller than a predefined threshold define the most important regions. These selected directions were then grouped to regions and finer points were applied only in those regions. A grouping technique based on the correlation between points (directions) was adopted to define those regions (Katsuki, private communication). Dr Katsuki also noted that the grouping technique may result in error if knowledge about the limit state function is limited. This grouping technique may lead to overlapping of groups in some cases, especially in high dimensional spaces, which may result in large error. A new two-phase refinement scheme is developed using neural networks and Fekete points in this paper. In the first phase, the network is trained to identify the most significant search directions; in the second phase, the reliability is estimated. Neural networks will be used to construct the superregion, i.e. the assembly of the most significant regions which may be overlapped or even
Fig. 2. Illustration of effective region, points and limit state.
located separately, and to filter out the portions of the finer point set that are not in the superregion. The effective region is denoted the superregion on the unit hypersphere which contributes most to the failure probability; and the effective points as the subset of a point set lying within the effective region. The effective limit state is the part of the limit state corresponding to the effective region (see Fig. 2).
3. Application of neural networks Neural networks are mathematical models that approximate complex functions in a similar way that the human brain processes and remembers information [15]. They have been investigated as surrogates to complex structural systems in structural reliability analysis [4,13]. The basic structure of a neural network includes nodes and connections which link the nodes. Each connection has an associated weight property. Weights are the principal mechanism by which a neural network stores information and is trained; training is achieved by adjusting the weights so as to make the output of the network match the predefined goal (in the sense of supervised learning). A node is depicted in Fig. 3, where the output ao is called the node’s activation value, and is given by ao ¼ f ðdð~ai ÞÞ:
ð1Þ
Fig. 3. Illustration of a node.
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
439
f is the transfer function, d is the input function, a~ i is the input vector, and wij is the weight. A network must have nonlinear transfer functions in order to approximate a complex function. Common transfer functions include the step, sign and logsig functions as described below: ( 1; for x $ t stepðx; tÞ ¼ 0; otherwise ( 1; for x $ 0 ð2Þ signðxÞ ¼ 21; otherwise logsigðxÞ ¼ 1=ð1 þ e2x Þ: The input function d can be a distance function or a simple weighted-sum of the input vector. A converged neural network can then predict outputs from new inputs. In terms of statistics, the training process of a neural network is mathematically similar to nonlinear regression [15]. Two common types of neural networks, namely feedforward backpropagation networks (FFBPN) and radial basis networks (RBN), will be investigated as tools for directional point set refinement. 3.1. Feed-forward back-propagation network The effective region in deterministic directional methods is a union of small hypercaps [12], which are the approximated neighbors of the effective points identified in the first phase. Any of these hypercaps can be obtained by cutting the unit hypersphere with a hyperplane. These small overlapped hypercaps can then be grouped into larger clusters, which are larger hypercaps and can be obtained by cutting the sphere with hyperplanes as well. If the limit state is linear and the sampling points are uniformly scattered, then the effective region (hypercap) can be cut by one hyperplane, which is parallel to the limit state (see Fig. 4). This effective hypercap region can also be imagined as the radial projection of the effective limit state onto the sphere. If the effective points are assigned value 1 and other points are assigned value 0, then this binaryvalued function over the whole point set is a linearly separable function. The perceptrons are the conceptual building blocks of the FFBPN. A perceptron is a simple layered feed-forward network with no hidden layers; it uses a step transfer function, can only learn a linearly separable function but can adequately represent the effective region of a linear limit state function. Most realistic limit states are nonlinear, and thus perceptrons are not directly useful in such cases. However, the effective region of a general limit state can be approximated by hypercaps, each of which can be represented by a perceptron. The number of perceptrons required in this approximation is bounded from above by the number of effective points; however a lesser number should be used to avoid network overfitting. The desired network
Fig. 4. Effective region of linear limit state by one cutting hyperplane.
should be able to approximate the binary-valued function, which is defined in the case of a linear limit state. If the number of perceptrons in the approximation is M; then a two-layer network is constructed using the M perceptrons as the hidden layer and one perceptron as the only node in the output layer to merge the outputs of the hidden layer. Such a network design represents the union of the hypercaps, which defines the effective region. Fig. 5 shows the design of such a network, in which some necessary modifications to the perceptrons have been made to adopt the above concept into a practical FFBPN. The transfer functions have been changed to the differentiable logsig function, the shape of which is close to that of a step function (see Fig. 6), which is necessary for backpropagation learning. Bias is included in each node and the position t of a step function is set to 0. The input function d is defined as X ~ x þ b: dð~xÞ ¼ wi xi þ b ¼ w~ ð3Þ There are many training algorithms for an FFBPN. The gradient descent algorithm is a typical example of error back-propagation [10,15], but is often too slow for practical usage. Faster methods include gradient descent with momentum, variable learning rate, conjugate gradient, Levenberg-Marquardt algorithm and many others. The Levenberg-Marquardt algorithm, which is reported to be the fastest algorithm for a FFBPN of moderate size (‘up to several hundred weights’ [2]), will be used herein, since the examples in this paper are of moderate size. (Referring back to the network in Fig. 5, there are 126 weights and seven biases in a network of 6 hidden nodes for a problem of
440
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
Fig. 5. Illustration of a two layer FFBPN.
20 random variables.) Fig. 7 shows a typical convergence curve in training a FFBPN in terms of epochs, each of which contains a full cycle of updating all the weights for all the inputs. Neural network training can be made more efficient if the inputs and targets are within certain bounds. The inputs and targets are usually converted into range [2 1,1]; a converged neural network will then produce output in the range [2 1,1]. One attraction of the FFBPN in point set refinement is that the input data are within the range [2 1,1] even without such a conversion; since the points are on the unit hypersphere, the targets are purposely assigned value 0 and 1, and the range of logsig transfer function is (0,1). Caution has to be exercised in selecting M; the number of hidden nodes. It is generally easier for the network to converge if M is larger. However a larger M may not yield a more accurate estimate of failure probability, since the network may be overfitted. In that case, the converged
network cannot be generalized to other data from the same domain. Fig. 8 compares a properly fitted FFBPN to one that is overfitted on the spheres. Both networks are trained with the same input data (coarse dots in first phase) and tested with the same simulation data (fine dots in second phase). In general, the adjustable parameters in a network should be less than the number of samples in the training set of data. In addition, a simpler network usually yields better predictions when used with a new set of data [2]. For an FFBPN constructed as in Fig. 5 to have good generalization, M and the number of training samples, m; should satisfy the following inequality [2] ðn þ 2ÞM þ 1 , m;
ð4Þ
where n is the dimension of the problem. To construct an FFBPN which is as simple as possible, a special procedure is followed in this study. The network is always constructed with M ¼ 1 at first, which is repeated up
Fig. 6. Comparison of step and logsig functions.
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
441
Fig. 7. A typical convergence curve for FFBPN.
to 10 training cycles; during each cycle, the training restarts with a new randomized set of weights and biases and proceeds for 100 epochs. If the network has not converged after 10 trials, the network is reconstructed with M ˆ M þ 1; and the above procedure is repeated. If convergence has not been achieved before M gets larger than required by Eq. (4), then the number of samples is judged insufficient to represent the limit state, and additional samples are supplied. The convergence criteria in the training is set to 10212 based on trial and error. Root searching procedures are usually used to calculate the radius along each direction for a complicated limit state. A good starting guess of radii can speed up the search. Unfortunately, an FFBPN is designed mainly as a filter on directions, and provides no information on radius along any direction that has been selected in the second phase. The RBN, introduced in the next section, yields a reasonable estimate of radius as well as filtering directions. 3.2. Radial basis network If the limit state can be approximated in such a way that the radius for an arbitrary direction can be quickly determined in the first phase, the approximated limit state can serve as a filter for the region of interest and yield an initial radius estimate to initialize the root searching procedure. A response surface is one way to approximate the limit state [8,14,16]; however it is only valid around the design point. For a highly nonlinear limit state, a response surface, which usually takes a quadratic form, is not adequate to express the limit state in the whole domain. A RBN is a technique for multi-variable interpolation, and can approximate a highly nonlinear limit state in the whole domain very well if designed properly. A RBN is a twolayer network, as shown in Fig. 9. The configuration of a RBN is similar to that of a FFBPN, except that the input function to the hidden nodes is a factored distance function rather than a weighted sum. Assuming an Euclidean
Fig. 8. Comparison of a properly fitted and an overfitted FFBPN representing a linear limit state.
distance measure, the factored distance is defined by ~ bÞ ¼ bl~x 2 wl; ~ distð~x; w;
ð5Þ
~ acts as the center of the transfer where the weight vector w function, and bias b controls the shape of the transfer function. The distance can also be spherical if the RBN is constructed on the hypersphere, i.e. ~ bÞ ¼ b arccosð~x·wÞ: ~ distð~x; w;
ð6Þ
The network should be insensitive to the choice of input function if b is chosen appropriately [1]. The transfer function is one of the radial basis functions. Any univariate function which approaches 0 when its argument
442
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
Fig. 9. Schematic of RBN.
approaches 1 can be a radial basis function; a common choice is, 2
f ðxÞ ¼ e2x :
ð7Þ
The radial basis function is localized such that only inputs near the center can produce nontrivial values (see f ðxÞ on a sphere in Fig. 10). The output node has the same input function as the output node in the FFBPN, but the transfer function is linear. A RBN can be designed by one of the following two methods. In the first (exact) method, each input point is used as the center of the radial basis function of one hidden node, and the weights of the second layer are selected to make the network error equal to zero by solving a set of linear equations. However, this method yields an inefficient design because it creates as many hidden nodes as the number of inputs in the training data. In the second (iterative) method, the input points are added to the network iteratively. In each iteration, only the point which lowers the network error most is added. This process is repeated until the network error is below a predefined goal or the inputs in the training data are used up. In this study, the second method is used when the number of points in the first phase set (cardinality) is small; otherwise the first method is used. In order to make the function approximated by the RBN smooth for both methods, bias b has to be small enough to make the transfer functions of the hidden nodes overlap. However to avoid underfitting, b should not be so small that all hidden nodes respond essentially the same. In this research, b is chosen as 0:8326=Rc ; where Rc is the covering radius of the Fekete set in the first phase [10]. The parameter 0.8326 is chosen such that f ð^0:8326Þ ¼ 0:5 (in Eq. (7)). The radii cannot be used directly as the targets of the RBN, since the significant directions have smaller radii
while others have larger radii. The reciprocals of the radii are used as the values for the RBN to interpolate (see Fig. 11). Let R^ min be the minimal radius among all the directions in the first phase. The threshold for significant radii is selected as R^ min þ 5; which is larger than those suggested by Harbitz [3] or Yonezawa and Okuda [17].
4. Efficiency assessment Several limit state functions are listed in Table 1. These are used to assess the efficiency and accuracy of the two
Fig. 10. A radial basis function f ðxÞ on the sphere.
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
443
Fig. 11. Radial basis interpolation.
kinds of networks. Function 0 has been considered in previous research [5,6,11]. Functions 0 – 5 are linear and intended for testing whether the proposed method has directional preferences. Functions 6 – 7 are nonlinear functions representing serial and parallel systems respectively. Function 6 and 8 have two design points; in particular, function 8 has the two design points in opposite directions. All test limit states are selected intentionally as linear functions or are composed of linear functions, so that the calculations of the radii are inexpensive. However, conclusions drawn from these tests are still valid for realistic problems because the projection of a realistic limit state on the hypersphere can be considered as an assembly of hypercaps cut by hyperplanes. The limit state probabilities of functions 0-8 in dimensions 3 and 12 assessed using the FFBPN and RBN methods are shown in Tables 2– 5. Errors are computed against the ‘exact’ values, i.e. the theoretical values in Table 1 or estimates for functions 6 and 7 by multi-modal Importance Monte Carlo Simulation (MIMCS) using 100,000 samples. The failure probabilities by MIMCS for limit state 6 in 3D and 12D are 2:56640 £ 1023 and 2:68964 £ 1023 ; while the standard deviations in these estimates are 1:23872 £ 1025 and 1:55402 £ 1025 ; respectively. The failure probabilities by MIMCS for limit state 7 in 3D and 12D are 1:24146 £ 1024 and 2:26780 £ 1025 ; while the standard deviations of these estimates are 2:51980 £ 1026 and 3:16872 £ 1027 ; respectively. Two point sets F3-12-5 and F3-36-10 in 3D are used to test the effect of the cardinality in the first phase. (Recall that F3-12-5 means a 3D Fekete point set of 12 points, which satisfy the spherical t-design lower bound for t ¼ 5:) The applications of FFBPN and RBN are compared by the examples in Tables 2 and 3, in which limit state function 2 is analyzed using F3-12-5 and F3-240-29 as
the sets for the two phases. In the first phase, the limit state probability is calculated by the Fekete point method; the error compared to the exact solution is 10.6%. For the FFBPN (Table 2), three points out of the 12 points are identified as having significant contributions to the failure probability, and are assigned value 1. All other points are assigned value 0. A FFBPN is created with one hidden node. The weights and biases are randomly initialized. The 12 points are the input vectors for this FFBPN, and the values associated with these points define the target vector. The network is then trained until it converges. The network is fed with set F3-240-29, and 62 points are filtered out. These 62 points are then used to reanalyze this limit state probability. The error drops down to 2 0.445% with a total effort of 74 evaluations of the limit state function, rather than 240 evaluations. If directional simulation with 10,000 random directions is used for this limit state, the failure probability estimate, its standard deviation and its coefficient of variation (COV) are 1.33275 £ 1023, 4.54516 £ 1025, and 3.41%, respectively. In Monte Carlo simulation, the COV in an estimate Table 1 Test functions for point set refinement methods Func. ID
Limit state function
Exact failure probability
0 1 2 3 4 5 6 7 8
P pffiffi 2 xi þ 3 n ¼ 0 P pffiffi xi þ 3 n ¼ 0 2x1 þ 3 ¼ 0 x1 þ 3 ¼ 0 2x2 þ 3 ¼ 0 x2 þ 3 ¼ 0 Func. 0 OR Func. 2 Func. 0 AND Func. 2 Func. 2 OR Func. 3
1.34990 £ 1023 1.34990 £ 1023 1.34990 £ 1023 1.34990 £ 1023 1.34990 £ 1023 1.34990 £ 1023 MC MC 2.69980 £ 1023
444
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
Table 2 Test result of FFBPN in 3D Func.ID
0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 7 7 8 8
Hidden node #
2 1 2 1 1 1 2 1 2 1 2 1 1 2 1 1 1 1 2 2
First phase
Second phase
Point set
# Eff. points
Error (%)
Point set
# Eff. points
P^ f ð£1023 Þ
Error (%)
F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10
4 11 4 11 3 12 3 11 3 11 3 11 5 17 3 6 3 6 6 23
35.4 .591 35.4 3.24 10.6 1.83 10.6 2.08 214.4 23.10 214.4 1.38 29.2 5.14 295.5 272.4 295.5 272.4 10.6 1.95
F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-1200-67 F3-1200-67 F3-240-29 F3-240-29
65 72 64 53 62 58 54 64 66 61 58 71 91 97 37 43 192 217 134 120
1.34544 1.34950 1.34551 1.34928 1.34396 1.34837 1.35072 1.35186 1.35003 1.34894 1.34899 1.34975 2.52625 2.57574 0.12096 0.12208 0.12298 0.12384 2.63741 2.70022
20.335 20.034 20.330 20.051 20.445 20.118 20.097 0.140 0.005 0.011 20.072 20.016 21.57 0.364 22.56 21.67 20.938 20.243 22.32 0.010
reflects, by definition, a 68% confidence interval within which the true value falls, and is similar to the notion of relative error. Thus, if the relative error in the approach herein is considered to be comparable to the COV of the failure probability estimate, the proposed approach achieves a higher accuracy with substantially fewer points than directional simulation (about 99.3% of saving w.r.t. directional simulation).
For the RBN (Table 3), the reciprocals of the radii associated with the 12 points are the targets of the RBN. The RBN is constructed by the iterative method. A total of 11 points are added as the hidden nodes, which is larger than that of the FFBPN. The network is then used to filter the set F3-240-29, and 59 points out of the 240 points are selected by this RBN. The error drops down to 2 0.196% with a total effort of 71 evaluations of the limit state function.
Table 3 Test result of RBN (iterative method) in 3D Func.ID
0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 7 7 8 8
Hidden node #
10 25 10 24 11 23 11 23 11 24 10 25 11 29 8 19 8 19 7 33
First phase
Second phase
Point set
# Eff. points
Error (%)
Point set
# Eff. points
P^ f ð£1023 Þ
Error (%)
F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10 F3-12-5 F3-36-10
4 11 4 11 3 12 3 11 3 11 3 11 5 17 3 6 3 6 6 23
35.4 .591 35.4 23.24 10.6 1.83 10.6 2.08 214.4 23.10 214.4 1.38 29.2 5.14 295.5 272.4 295.5 272.4 10.6 1.95
F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-240-29 F3-1200-67 F3-1200-67 F3-240-29 F3-240-29
42 62 65 61 59 67 55 64 58 62 61 62 91 98 48 37 250 176 105 132
1.34871 1.34948 1.33483 1.34990 1.34732 1.34837 1.35000 1.35176 1.34978 1.34990 1.34961 1.34975 2.57360 2.57573 0.12212 0.12213 0.12388 0.12388 2.69589 2.70023
20.094 20.036 20.007 20.005 20.196 20.118 20.002 0.132 0.014 0.012 20.027 20.016 0.281 0.364 21.63 21.63 2.212 2.211 2.150 0.011
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
445
Table 4 Test result of FFBPN in 12D Func.ID
0 1 2 3 4 5 6 7 8
Hidden node #
1 1 1 1 1 1 2 2 2
First phase
Second phase
Point set
# Eff. points
Error (%)
Point set
# Eff. points
P^ f ð£1023 Þ
Error (%)
F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5
15 26 20 22 24 24 38 7 43
242.1 227.2 29.97 24.03 215.9 55.9 225.8 298.9 27.00
F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11
1650 2328 1701 2229 1826 1972 3525 950 3872
1.32540 1.35353 1.32960 1.33852 1.32540 1.34618 2.61738 0.03622 2.66810
21.82 20.263 21.51 20.848 2.61 20.281 22.69 59.7 21.18
To summarize the results in Tables 2– 5, the FFBPN uses fewer hidden nodes than the RBN, and consequently results in more efficient networks in filtering the point set in the second phase (set II). For all test functions, the FFBPN uses only 1 or 2 hidden nodes in spite of the problem dimension. The number of hidden nodes used by the FFBPN is affected only by the nature of the limit state. On the other hand, for the RBN, the number of hidden nodes used by the iterative method depends on the number of effective points and the cardinality of the point set in the first phase (set I), while the exact method uses up all the points in set I. The effectiveness of a refinement method can be evaluated by the number of effective points filtered out in the second phase. In general, the RBN produces a smaller set of effective points in the second phase than the FFBPN, especially in high-dimensional space. For example, the FFBPN produces 2 –19 times more effective points than the RBN in 12D, which would require a corresponding increase in calls to the FE routine for a realistic structural reliability problem. The RBN requires more time in filtering set II than the FFBPN because of its larger hidden layer. However, the filtering time is usually marginal compared to the time spent in FE analysis. Both methods can dramatically decrease the number of points to be used as FE samples in structural reliability analysis. As another example for limit state 0 in 12D, the use of the RBN reduces the number of points from 13587 to 906 (156 þ 750), which is a saving of about 93.3%.
Although a set I of larger cardinality usually yields smaller error in the first phase, it has little effect on the final error, as shown in Tables 2 and 3. In this sense, set I is used only for finding the effective region of the limit state. It is usually sufficient to select a point set with t ¼ 5; in terms of spherical t-design [10 – 12]. Only when a very small number of effective points is selected in the first phase, say less than 5% of set I, may one want to try a larger set to get more information about the limit state and a better approximation of its effective region. The final error after set II has been applied is largely dependent on the cardinality of set II and the shape of the limit state. For a given set II, the application of a properly designed neural network leads to an improvement in efficiency without significant loss of accuracy. The final error is smaller for a concave failure domain than for a convex one. The higher error for convex limit states with high local curvatures, e.g. for limit state 7 in Tables 2– 5, is true for all directional methods including this one. In this case, a larger set II can be applied to get a smaller error.
5. Examples of quadratic limit state functions Quadratic functions are an important class of limit states because they frequently are used to define a response
Table 5 Test result of RBN (exact method) in 12D Func.ID
0 1 2 3 4 5 6 7 8
Hidden node #
156 156 156 156 156 156 156 156 156
First phase
Second phase
Point set
# Eff. points
Error (%)
Point set
# Eff. points
P^ f ð£1023 Þ
Error (%)
F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5 F12-156-5
15 26 20 22 24 24 38 7 43
242.1 227.2 29.97 24.03 215.9 55.9 225.8 298.9 27.00
F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11 F12-13587-11
750 699 737 681 593 680 1542 50 1158
1.31800 1.35353 1.34313 1.32412 1.31909 1.35641 1.33275 2.59163 0.02747
22.37 20.507 21.92 22.29 0.477 21.28 23.64 221.1 22.48
446
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
surface approximation to a real limit state around the design point [8,14,16]. The effective ratio is the number of effective points over the cardinality of the whole point set. The effort ratio is defined by the number of points actually used, which is the sum of the cardinality of the first phase point set and the number of effective points in second phase, divided by the number of points required if a neural network had not been applied to refine the point selection, i.e., the cardinality of second phase point set. The limit state function Gconcave ¼ 20:5ðz21 þ z22 þ z23 2 2z1 z2 2 2z2 z3 2 2z3 z1 Þ pffiffi 2 ðz1 þ z2 þ z3 Þ= 3 þ 3 ð8Þ
Fig. 12. The spherical projection of Gconcave shown by its effective points.
Fig. 13. The spherical projection of Gconvex shown by its effective points.
represents the boundary of a concave failure domain in 3D, the effective region of which (spherical projection of the effective limit state on the sphere) is a circular stripe (see Fig. 12). By directional simulation, the exact failure probability found using 10,000 samples was 0.19798, with sampling error 1:56 £ 1023 and COV 0.79% [11]. The result from the point set/neural network analysis is shown in Table 6. The FFBPN method achieves an error of 0.13% with 212 points, and the RBN achieves 0.02% with 226 points. The proposed method yields smaller error than directional simulation, and achieves about 98% of saving in samples. The RBN has a higher effort ratio, but achieves a higher accuracy than the FFBPN. However, the improvement in efficiency using neural networks is not significant in this example because the shape of the limit state has a large spherical projection (effective region). For the limit state in Eq. (8), the RBN uses 36 hidden nodes, while the FFBPN uses only two hidden nodes. That is, the RBN yields a more complex design than does the FFBPN. It is interesting to note that although the FFBPN was conceptually constructed with imaginary hyperplanes (perceptrons) cutting the hypersphere, in this particular example the circular-stripe-shaped effective region of Gconcave cannot be cut directly from the sphere by only two planes. However, when the area complementary to the effective region on the sphere is considered, it is obvious that the effective region can still be cut from the sphere with two cutting planes, i.e. the FFBPN can identify a region by its complement on the sphere.
Table 6 Failure probabilities for quadratic limit state functions First phase
Second phase
Effort ratio (%)
Limit state
Point set
Eff. points
1ð%Þ
Network
Point set
pf
1ð%Þ
Eff. points
Gconcave Gconcave Gconvex Gconvex
F3-36-10 F3-36-10 F3-36-10 F3-36-10
27 27 4 4
1.45 1.45 13.6 13.6
FFBPN-2 RBN-36 FFBPN-2 RBN-36
F3-240-29 F3-240-29 F3-240-29 F3-240-29
0.19773 0.19801 0.01904 0.01921
20.13 0.02 21.37 20.49
176 190 33 33
87.5 94.2 28.8 28.8
J. Nie, B.R. Ellingwood / Probabilistic Engineering Mechanics 19 (2004) 437–447
A convex failure domain can be defined by the following limit state function: Gconvex ¼ 0:5ðz21 þ z22 þ z23 2 2z1 z2 2 2z2 z3 2 2z3 z1 Þ pffiffi 2 ðz1 þ z2 þ z3 Þ= 3 þ 3:
ð9Þ
Gconvex is depicted by its effective points in Fig. 13, which consists of two separate parts located at opposite areas on the sphere. The exact failure probability 1:93043 £ 1022 was obtained by directional simulation using 10,000 directions; the estimated sampling error is 8:04 £ 1024 and the COV is 4.16% [11]. The COV of this probability estimate is larger than that of Gconcave ; as expected. The results of the point set/neural network analysis are shown in Table 6. The errors yielded by the FFBPN and the RBN are 1.37 and 0.49%, respectively. Once again, the proposed method yields smaller error than directional simulation, and uses only 69 points for both networks (about 99.3% of saving w.r.t. directional simulation). For this convex failure domain, the application of neural networks yields an effort ratio of only 28.8%, which is more than 70% saving in effort. The FFBPN uses two hidden nodes, while the RBN uses all 36 points of first phase. The error in the FFBPN is larger than in the RBN although the same number of total points are used, which indicates that the FFBPN selects different points from those of the RBN.
6. Conclusion This paper introduced a two-phase strategy of point set refinement to reduce the number of evaluations of the limit state function in directional simulation. Two types of neural networks, the FFBPN and the RBN, have been explored and shown by examples to be effective in this regard. These methods are implemented with Fekete point sets to identify directions, which is introduced in a companion paper [12]. The combination of the Fekete point method and neural network techniques has been used successfully to assess the reliability of more complex structural systems [10]. The effective ratios of the first phase and the second phase of the directional search refinement process are roughly the same. A large effective ratio implies a relative smooth limit state function and a large effort ratio. Therefore, the benefit of the neural network refinement may be less significant when the effective ratios are high. For Gconcave (Eq. (8)), where the effective ratio is about 75%, a decrease in error from 1.45 to 0.13% or 0.02% with an effort ratio of around 90% does not appear to be costeffective, when compared with other examples such as
447
Gconvex : In this case, the effective ratio from the first-phase analysis is helpful in choosing a point set for the second phase to avoid the high computational cost. Because the failures of structural systems usually occur at high values of loads and low values of resistances, practical structural reliability problems generally have low effective ratios, and thus the use of neural networks as adjuncts to directional simulation algorithms may lead to significant saving in practical system reliability assessment [10].
References [1] Bishop CM. Neural Network for Pattern Recognition. New York: Oxford University Press; 1995. [2] Hagan MT, Demuth HB, Beale M. Neural Network Design. Boston: PWS Publishing Company; 1996. [3] Harbitz A. An efficient sampling method for probability of failure calculation. Struct Safety 1986;3:109– 15. [4] Hurtado JE, Alvarez DA. Neural-network-based reliability analysis: a comparative study. Comput Meth Appl Mech Engng 2001;191: 113–32. [5] Katsuki S, Frangopol DM. Hyper-space division method for structural reliability. J Engng Mech, ASCE 1994;120(11):2405–27. [6] Katsuki S, Frangopol DM. Advanced hyperspace division method for structural reliability. In: Shinozuka W, editor. Struct Safety Reliab. 1998. p. 631–8. [7] Kijawatworawet W, Pradlwarter HJ, Schue¨ller GI. Structural reliability estimation by adaptive importance directional sampling. In: Shiraishi N, Shinozuka M, Wen YK, editors. The Seventh International Conference on Structural Safety and Reliability, vol. 2. 1998. p. 891 –7. [8] Kim S, Na S. Response surface method using vector projected sampling points. Struct Safety 1997;19(1):3– 19. [9] Melchers RE. Radial importance sampling for structural reliability. J Engng Mech, ASCE 1990;116(1):189–203. [10] Nie J. A new directional method to assess structural system reliability in the context of performance-based design. PhD Thesis. The Johns Hopkins University, Baltimore, MD; 2003. [11] Nie J, Ellingwood BR. Directional methods for structural reliability analysis. Struct Safety 2000;22:233–49. [12] Nie J, Ellingwood BR. A new directional simulation method for system reliability. Part I: Application of deterministic point sets. Probab Engng Mech 2004; in press. [13] Papadrakakis M, Papadopoulos V, Lagaros ND. Structural reliability analysis of elastic-plastic structures using neural networks and Monte Carlo simulation. Comput Meth Appl Mech Engng 1996;136:145– 63. [14] Rajashekhar MR, Ellingwood BR. A new look at the response surface approach for reliability analysis. Struct Safety 1993;12:205 –20. [15] Russell SJ, Norvig P. Artificial Intelligence: A Modern Approach. Englewood Cliffs, NJ: Prentice Hall; 1995. [16] Yao TH, Wen YK. Response surface method for time-variant reliability analysis. J Struct Engng, ASCE 1996;122(2):193–201. [17] Yonezawa M, Okuda S. Directional vectors approximation for assessment of structural failure probability. In: Shiraishi, Shinozuka, Wen, editors. Struct Safety Reliab. Rotterdam: Balkema; 1998. p. 611–6.