Structural reliability analysis based on ensemble learning of surrogate models

Structural Safety 83 (2020) 101905

Contents lists available at ScienceDirect

Structural Safety journal homepage: www.elsevier.com/locate/strusafe

Structural reliability analysis based on ensemble learning of surrogate models

T

Kai Cheng, Zhenzhou Lu

⁎

School of Aeronautics, Northwestern Polytechnical University, Xi’an 710072, PR China

ARTICLE INFO

ABSTRACT

Keywords: Ensemble learning Surrogate model Reliability analysis Active learning

Assessing the failure probability of complex structure is a difficult task in presence of various uncertainties. In this paper, a new adaptive approach is developed for reliability analysis by ensemble learning of multiple competitive surrogate models, including Kriging, polynomial chaos expansion and support vector regression. Ensemble of surrogates provides a more robust approximation of true performance function through a weighted average strategy, and it helps to identify regions with possible high prediction error. Starting from an initial experimental design, the ensemble model is iteratively updated by adding new sample points to regions with large prediction error as well as near the limit state through an active learning algorithm. The proposed method is validated with several benchmark examples, and the results show that the ensemble of multiple surrogate models is very efficient for estimating failure probability (> 10−4) of complex system with less computational costs than the traditional single surrogate model.

1. Introduction Structural reliability analysis is of great importance in engineering, it aims at computing the probability of failure of a system with respect to some performance criterion in the presence of various uncertainties. For a given structural system with n-dimensional input parameter x = (x1, x2, …, xn)T , the performance function (also known as limit state function) g (x ) divides the input variable space into two domains, i.e, the safety domain ( g (x ) > 0 ) and failure domain ( g (x ) 0 ). Thus the failure probability Pf reads:

Pf =

Ig (x )

0 (x ) f X

(x ) dx ,

{

(1)

g (x ) 0 = 1 is the indicator function of the failure 0 g (x ) > 0 domain and fX (x ) is the joint probability density function (PDF) of x . The task of reliability analysis is to perform the integration in Eq. (1). Generally, the reliability analysis method in literature can be categorized into three types: approximate analytical methods [1,2], numerical integration methods [3–7] and numerical simulation methods [8–16]. The approximate analytical methods expand the performance function g (x ) at mean point or design point by Taylor expansion, and ignore the higher order terms to estimate the failure probability, such as First order reliability method (FORM) [1] and Second order reliability analysis method (SORM) [2]. However, their accuracy can hardly be where Ig (x )

⁎

0 (x )

guaranteed especially for highly non-linear problems. For numerical integration methods, the first few moments of the performance function are computed by the point estimation method [3–5] or sparse grid integration method [6], and then failure probability is estimated by these moments. However, the computation cost of these methods increases sharply (exponentially) with the input variable dimensionality. The numerical simulation methods include Monte Carlo simulation (MCS), Importance sampling (IS) [8,9,17], Subset simulation (SS) [11–13,18] and recent work called thermodynamic integration and parallel tempering (TIPT) method [19]. These methods are relatively robust to the type and dimension of the problem, but they cannot satisfy the computational efficiency requirements for time-consuming model. To reduce the computational cost for reliability analysis, surrogate-assisted methods have received much attention in the past few decades. These methods aim at constructing a surrogate model (also known as metamodel) with an explicit expression based on a set of observed points to approximate the true performance function, and thus one can perform reliability analysis efficiently based on the cheap-to-evaluate surrogate model. For decades, several types of surrogate models are available in literatures for reliability analysis including polynomial chaos expansion (PCE) [20], Kriging (Gaussian Process, GP) [21–24], support vector machine (SVM) and support vector regression (SVR) [12,25,26], high dimensional model representation (HDMR) [27] and so on. Recently, surrogate models combined with active learning strategies

Corresponding author. E-mail address: [email protected] (Z. Lu).

https://doi.org/10.1016/j.strusafe.2019.101905 Received 8 March 2019; Received in revised form 31 October 2019; Accepted 3 November 2019 0167-4730/ © 2019 Elsevier Ltd. All rights reserved.

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

have been well developed for reliability analysis [21,23,24,28–34] of complex system, especially for Kriging model. These methods start from an initial design of experiment (DoE), and enrich it sequentially by adding new sample points based on the predefined learning function. The learning function is usually developed based on the statistical information of surrogate model from different perspectives. The expected feasibility function (EFF) proposed by Bichon et al. [35] and U function developed by Echard et al. [28,29] both select points near the limit state surface of Kriging model with large prediction uncertainty. The expected risk function (ERF) in Ref [36] and H function in Ref [21] search for points with large prediction error and information entropy in the vicinity of limit state surface of Kriging model respectively. Sun et al. [23] developed the least improvement function (LIF) based on Kriging, which quantifies the improvement of the accuracy of estimated failure probability when new points are added to the DoE. Marelli et al. [20] presented a learning function that focuses on the probability of misclassification of PCE model based on bootstrap resampling strategy. In Ref [37], adaptive SVR model is presented to estimate the failure probability, where the learning function is defined by the distance criteria. It has been proven that these well-developed learning functions are very efficient to improve the accuracy and efficiency for reliability analysis of complex models. In this paper, instead of fitting the performance function with single surrogate model, we explore the possibility of ensemble of multiple competitive surrogate models to approximate the performance function for reliability analysis. Each surrogate model is required to predict the model response, and the final prediction is obtained by a weighted average of multiple surrogate models. In the meanwhile, the local prediction error is estimated by the variance of the multiple surrogate models, thus an active learning algorithm is developed to select some dangerous sample points sequentially in the regions with large prediction error to improve the prediction accuracy as much as possible. The layout of this paper is as follows. Section 2 presents an overview of PCE, Kriging and SVR surrogate models. In section 3, the proposed active learning strategy for reliability analysis is presented. Four examples are employed in Section 4 to demonstrate the efficiency and accuracy of the proposed method. Finally, some conclusions are drawn in Section 5.

coefficients are sparse (i.e. having only several dominant coefficients). Given the training sample {X , Y } , where X = {x1, …, xN }T are the input data, Y = {Y1, …, YN }T are the corresponding model responses and N is the size of sample, the dominant PCE coefficients can be recovered by solving the following optimization problem

= arg min

(3)

Kriging model (also known as GP) was first proposed in the field of geostatistics by Krige [52] and Matheron [53]. It tends to find the best linear unbiased predictor while minimizes the mean square error of the prediction. The universal Kriging is composed of a polynomial term used for global trend prediction and a Gaussian process term used for local deviation regression, which can be expressed as (4)

gK (x ) = pT (x ) + Z (x ),

where p (x ) = [p1 (x ), …, pM (x )]T is the polynomial basis function, = [ 1, …, M ]T represents the corresponding regression coefficient vector, and Z (x ) is a Gaussian process with zero mean and covariance defined as

Cov (Z (x i ), Z (xj )) =

2R (x

i,

(5)

x j , ),

where is the variance of Z (x ) , and R (xi , xj , ) is the correlation coefficient between Z (xi ) and Z (xj ) with parameters = [ 1, …, n]T . The correlation function controls the smoothness of the Kriging model, and here the Gaussian correlation function is used in the present work, which is defined as 2

n

R (x i , x j , ) =

exp[

(k ) k (x i

x j(k ) ) 2].

(6)

k=1

) of Kriging Generally, the unknown hyper-parameters = ( , model can be tuned by maximum likelihood estimation technique. After the optimal parameters = ( , 2, ) are obtained, the posterior distribution of gK (x ) is a Gaussian process gK (x )Ñ(µ (x ), 2 (x )) with mean 2,

~ gK (x ) = µ (x ) = pT (x ) + r T (x ) R 1 (Y

(7)

F ),

and variance

2.1. Polynomial chaos expansion

2 (x )

=

The classic PCE was first proposed by Wiener [38,39] in the 30 s. The key concept of PCE is to expand the model response onto basis made of multivariate polynomials that are orthogonal with respect to the joint distribution of the input variables. In this setting, characterizing the response probability density function (PDF) is equivalent to evaluate the PC coefficients, i.e. the coordinates of the random response in this basis. The classic PCE of order p for n-dimensional random variable x can be expressed as [40]:

(x ), | | =

Y

2.2. Kriging/Gaussian process

In this section, three kinds of surrogate models, namely, PCE, Kriging and SVR are briefly reviewed.

0 | | p

subject to

where is a tolerance para1 is the l1 norm of PCE coefficients, N × P is the measure meter of the truncation error and : = j (xi )ij matrix. In this paper, the least angle regression (LAR) technique is used to develop sparse PCE, which is available from the UQLab toolbox [51].

2. Review of surrogate models

g~P (x ) =

1

n i=1

i,

2 [1

(r (x ) R

r T (x ) R 1r (x ) + (r (x ) R 1F 1F

p (x ))T (F TR 1F )

1

(8)

p (x ))],

where r (x ) = [R (x , x1), …, R (x , xN represents the correlation vector N × N is the correbetween x and N observed points, R: =R (xi , xj )ij N ×M lation matrix and F : =f j (xi )ij . The posterior mean in Eq. (7) is known as the Kriging predictor ~ gK (x ) . The posterior variance formula of Eq. (8) corresponds to the mean squared error (MSE) of this predictor and it is also known as the Kriging variance.

)]T

(2)

2.3. Support vector regression

where = { 1, …, n}( i 0) is the multidimensional index notation is the unknown deterministic coefficients vector, and (x ) is vector, the multivariate polynomial vector. The total number of the expansion terms in the summation of Eq. (2) is P = (p + n)!/p ! n! + 1. To calculate the PCE coefficients, the traditional projection method and regression method [41,42] suffer from the so-called curse of dimensionality. To overcome this issue, many attempts have been made to develop sparse PCE model in the field of uncertainty quantification (UQ) [43–50], the common idea holding in these methods is that the PCE

Support-vector regression (SVR) was developed on statistical learning theory by Vapnik [54,55]. Generally, a linear SVR model is formulated as: ~ gS ( x ) = · x + b , (9) n where is the coefficient vector and b is a constant. The goal gS (x ) that can estimate the output response of SVR is to find a function ~ value whose deviation is less than ε from the real targets of the training

2

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu M w (x ) ~ gi (x ) = w (x )Tg~ (x ), i=1 i M w (x ) = 1Tw (x ) = 1, i=1 i

MWAS (x ) =

(15)

where MWAS (x ) is the predictor of the weighted average surrogate (WAS) model, M is the number of the different surrogates, wi (x ) is the gi (x ) is the weight associated to the i-th surrogate model at point x , ~ predicted response of i-th surrogate model, and 1 is a n × 1 vector of 1. In addition, ensemble of surrogates can be used to identify regions where we expect large prediction error. Here we utilize the prediction variance VWAS (x ) of multiple surrogate models to assess the prediction error, which reads as M

VWAS (x ) =

wi (x )(g~i (x )

MWAS (x ))2 .

(16)

i=1

The prediction variance here measures the dispersion degree of multiple surrogates, thus the regions with large variance tends to have large prediction error. The variance information helps us to developed active learning algorithm for reliability analysis, which is presented in Section 3.3.

Fig. 1. Geometric interpretation for SVR.

data (ε-tube) and is as flat as possible. Flatness in Eq. (9) means that one seeks the biggest tube width as 2 shown in Fig. 1, which is equivalent to minimize . Therefore, the optimal regression function is determined by solving the following optimization problem: 1 2

min s. t .

yi

·xi

3.2. Weights selection for ensemble model Generally, the weights of surrogate should reflect our confidence in the surrogate model as well as filter out the over-fitting effects. In the current work, the weights are selected based on global leave-one-out cross-validation error (denoted as eLOO), which leads to w (x ) = w, x [59] . In this regard, a surrogate model with small eLOO will have large weight in the ensemble model. In Ref [56], Goel et al. proposed a heuristic scheme for calculation of the weights, which is formulated as

2

b

,

i, i

(10)

(1, 2, …N )

By introducing the slack variables i and i (i = 1, …, N ) and according to the Karush-Kuhn-Tucker condition, the original optimization problem can be transformed into the following dual form:

min

1 2

N i=1

N j=1

(

i

N i = 1 yi ( i

s. t .

n i=1

(

i

i

i i

)(

j ) x i · xj

j

N i=1

+

(

i

+

i

)

)

) = 0, 0

i,

i

C,

i, i

(1, 2, …, N )

wi =

(11)

(a i

i

) x i · x + b.

(12)

For nonlinear problems, the input variables are mapped into a high (x ) , dimensional linear feature space by the nonlinear transform x then the prediction function in this feature space can be derived as follows:

g~S (x ) =

N

y = (6x

N

(a i

i

) (xi )· (x ) + b =

i=1

(a i

i

) k (x i , x ) + b ,

i=1

where

xi

x 2 / 2 ),

(17)

2) 2sin(12x

4), x ~U (0, 1).

(18)

It is observed from Fig. 2 that the predicted response of the WAS model in Eq. (15) yields the better approximation than the single surrogate model of the PCE, Kriging and SVR. Also, the prediction variance VWAS (x ) shown in Fig. 2(c) accurately identifies the regions with large prediction errors.

(13)

where k (xi , x ) = (xi )· (x ) is the so-called kernel function. Among the various kernel functions in literature [12], the Gaussian kernel is used in the present work, which is formulated as

k (xi , x ) = exp(

, wi = (Ei + Eavg )

Ei / M is the where Ei is the eLOO of i-th surrogate model, Eavg = and are two parameters which control the immean of Ei , and portance of average and importance of individual surrogate respectively. Small values of and large values of impart high weights to the optimal surrogate, and thus the surrogate with the small eLOO will play a leading role in the ensemble model. Conversely, large values of and small values of impart relatively average weights to each surrogate. In this study, we set = 0.05 and = 1 [56]. Fig. 2 depicts the schematic diagram by ensemble of the three kinds of surrogate models introduced in Section 2 of a 1-dimensional function

N i=1

wi

M i=1

where i and i are Lagrange multipliers. After solving the dual optimization problem above, can be obtained explicitly as N ~ = i=1 ( i i ) x i , and the final SVR predictor gS (x ) is expressed as

~ gS ( x ) =

wi M i=1

(14)

3.3. Active learning ensemble of surrogates for reliability analysis

is the hyper-parameter of kernel function.

In this section, we present a new approach for reliability analysis by ensemble of multiple different surrogate models.

3. Reliability analysis by ensemble of surrogate models

3.3.1. Probability of failure estimation Based on the predictor of the WAS model, the estimation of failure probability reads:

3.1. Ensemble of surrogate models Ensemble of surrogates has been investigated in Refs [56–58] for optimization purpose, where the best predictor is obtained by a weighted average of multiple different surrogate models. Ensemble of surrogates intends to take advantage of multiple different surrogates in the hope of canceling errors in prediction through proper weighting, which is formulated as

Pf =

IMWAS (x )

0 (x ) f X

(x ) dx .

(19)

In practices, MCS and some variance reduction techniques (IS, SS, LS and so on) are often applied to compute the Pf approximately. In the present work, the MCS is employed here to estimate Pf , which reads: 3

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

Fig. 2. Schematic diagram of ensemble of surrogate models.

1 Ns

Pf

Ns

IMWAS (x )

0 (x i ),

(20)

i=1

2.

where xi (i = 1, …, Ns ) are Ns random samples generated from fX (x ) . 3.3.2. Learning function The learning function aims at ranking the candidate samples, and it helps us to select the best sample from the candidate sample pool to enrich the current DoE. For reliability analysis problem, more samples should be placed in the regions which are near the limit state surface of the current predictor as well as have large prediction error. To this end, we adopt the following learning function in the present work:

UWAS (x ) =

|MWAS (x )| VWAS (x )

.

3. 4. 5. 6. 7.

(21)

From Fig. 3, one can conclude that each surrogate model is required to predict the model response of all the samples in S. Therefore, to estimate small failure probability (〈1 0 −4), the presented active learning algorithm is time-consuming due to the large candidate sample pool (> 107). To address this problem, one can embed the ensemble model with more efficient numerical simulation methods such as IS [22] or SS [60], and this is out the scope of the current paper.

It is observed that UWAS (x ) becomes smaller when × is nearer the predicted limit state surface (MWAS (x ) 0 ) or has larger prediction error VWAS (x ) . Thus one can choose the best training sample point from the candidate sample pool by minimizing UWAS (x ) , and adds it into the current DoE to refine the surrogate model as much as possible. It is worth noting that the learning function in Eq. (21) is consistent with the popular U function [28,29] in form, which focuses on the probability of misclassification made by the Kriging model.

4. Numerical example

3.3.3. Convergence criterion In the active learning process, the convergence criterion is usually defined by the accuracy of the surrogate model or the stability of the failure probability. In this paper, we assume that the ensemble model is accurate enough when the failure probability estimated by the multiple surrogates are consistent with each other. Therefore, we propose a relatively conservative convergence criterion, which reads

P+ f

Pf Pf

0.01,

In this section, four test examples are investigated to demonstrate the performance of the proposed method. In the developed algorithm, the sparse PCE and Kriging are constructed based on the UQLab toolbox [51], and the SVR is constructed by the Libsvm [61] toolbox. For Kriging model, the polynomial order in the first part of the Kriging model is set to be 1, and all the hyper-parameters are tuned by maximum likelihood estimation method in each iteration. For SVR model the hyper-parameters are determined by grid search technique in each iteration by minimizing eLOO . For sparse PCE model, the maximum polynomial order is set to be 10 in each example, and the optimal PCE order is optimized by minimizing eLOO . All other parameters of Kriging, PCE and SVR model are chosen to be their default values in the corresponding toolbox. To demonstrate the superiority of the proposed method, we compare the proposed method with Kriging model combined with U and EFF learning functions [28], adaptive SVR model [37] and bootstrap resampling based active learning spare PCE model (Bootstrap PCE) [20] with 100 replications. The convergence criterions for the U and EFF learning functions are set as U (x ) > 2 and EFF (x ) < 10 3 respectively [28]. The convergence criterion of adaptive SVR algorithm we use is related to the stability of failure vs. safe predictions (See [37] for more details), and the convergence thresholds is set to SC = 10 5 . The convergence criteria of active learning PCE model is related to the stability of the failure probability estimated by the PCE model (See [20] for more details), and the convergence thresholds is set to P = 0.05.

(22)

= Pf = where and here and P fK are the failure of probability computed by the single surrogate model of SVR, PCE and Kriging respectively. P+ f

max(P fS ,

P fP ,

P fK ) ,

min(P fS ,

P fP ,

P fK )

P fS ,

X = {x1, …, xN }T from S as the initial training sample set, and compute the corresponding model response Y = {Y1, …, YN }T . Train the Kriging model ~ gK (x ) , PCE model ~ gP (x ) and SVR model ~ gS (x ) using{X , Y } simultaneously, and estimate eLOO of each surrogate model. Compute the weights wi (i = 1, 2, 3) by Eq. (17) of each surrogate model. Obtain MWAS (x ) by Eq. (15) and VWAS (x ) by Eq. (16), then compute UWAS (x ) for all the samples in S. Compute the failure probability Pf by Eq. (19). Search a new point xNew from S by minimizing UWAS (x ) , and compute the true model response g (xNew ) . Then, let X = X xNew , Y = Y g (xNew ) . Repeat steps 2–6 until the convergence criterion is satisfied.

P fP

3.3.4. Active learning algorithm In this subsection, the active learning algorithm for reliability analysis based on ensemble of surrogates is presented. The algorithm starts from a small initial DoE and subsequently refines it to optimize the surrogate performance for structural reliability. The aim of the active learning algorithm is to obtain an accurate estimation of Pf with the least true model evaluations. The step by step implement of the proposed method is shown below, and the flowchart of the proposed method is presented in Fig. 3. 1. Generate Ns samples S = {x1, x2, …, x NS }T using Latin Hypercube Sampling (LHS) technique as the sample pool, then select N samples

f

4

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

Fig. 3. The flowchart of the active learning algorithm for reliability analysis.

4.1. Series system with four branches The first example consists of a series system with four branches, which has already been investigated in [23,28,62]. Its performance function is defined as

g (x ) = min

3 + 0.1(x1

x2) 2

3 + 0.1(x1

x2) 2 + (x1 + x2)/ 2

(x1 + x2)/ 2

(x1

x2 ) + 6/ 2

(x2

x1) + 6/ 2

where x1 and x2 are independent standard normal distributed random variables. In this example, 106 random samples generated with LHS are used as candidate sample pool. The initial DoE is obtained by selecting 20 sample points from the sample pool randomly. Then, the Kriging, PCE and SVR are constructed simultaneously with the same initial DoE. By ensemble of the three kinds of surrogate models, new samples are

Fig. 4. Convergent limit state of the proposed method.

5

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

Fig. 5. Convergence curve of the failure probability and weights.

added sequentially by the proposed learning function. Convergence is achieved after 59 iterations, thus the total sample size of the presented method is 79 for this example. Fig. 4 shows the final limit state of the ensemble model and the corresponding DoE. As expected, the developed learning function tends to enrich the DoE close to the limit state in the iteration process, and the final limit state of the ensemble model is very close to the true one. In addition, a graphical representation of the convergence curves of failure probability and the weights of the three kinds of surrogate models in the iteration process are shown in Fig. 5. It is observed that the failure probability obtained by the three kinds of surrogate models with the initial DoE are quite different. With the enrichment of the sample pointss, these surrogate models provide the consistent results. In addition, one can see that the weights of each surrogate model are varying dynamically in the iterations process. PCE and SVR provide more contribution in the ensemble model in the early stage, whereas SVR and Kriging provide more contribution in the ensemble model in the later period of the active learning process. The final results of the obtained failure probability are summarized in Table 1. For comparison purpose, the reference value obtained by MCS as well as the estimation based on AK-MCS combined with U and EFF learning function, adaptive SVR model and active learning PCE model are also given. One can conclude from Table 1 that the presented method provides comparative results with much fewer model evaluations compared to other single surrogate models. For this example, the Bootstrap PCE method [20] is prone to over-fitting, thus it needs more samples to yield a convergent result. However, our ensemble model provides more robust estimation of failure probability as well as improves the computational efficiency.

Fig. 6. Diagram of two-bar supporting structure. Table 2 Distribution parameters of random variables of two-bar structure. Input variable

Distribution

Mean value

Standard deviation

S/MPa W/kN h/mm S/mm douter/mm dinner/mm /°

Normal Normal Normal Normal Normal Normal Normal

200 47.75 100 100 30 18 60

20 5 3 3 0.9 0.54 3

g (x ) = S 4.2. Two-bar supporting structure

Table 1 Reliability analysis results of the series system.

MCS AK-MCS + U AK-MCS + EFF Adaptive SVR Bootstrap PCE Proposed method

Model evaluations 6

10 126 124 105 188 73

h2 + (s/2)2 2 (douter

2 dinner )

sin h

+2

cos s

,

where douter and dinner represent the outer and inner radius of the two bars, h, s, S , W and are the height, span, yield limit, load and angle respectively. All the random variables are normal distributed and mutually independent, and their distribution parameters are listed in Table 2. In this example, 107 random samples generated with LHS are used as candidate sample pool, and the initial DoE is obtained by selecting 20 sample points from the sample pool randomly. Then, the proposed active learning algorithm is performed, and 31 new samples are added sequentially by the proposed learning function, thus the final sample size is 51 for this example. The convergence curves of failure probability and the weights of the three kinds of surrogate models in the iteration process are shown in Fig. 7. It is observed that the failure probabilities obtained by the three kinds of surrogate models converge to the reference value progressively with the enrichment of new sample points. Also, the weights tell us that

A two-bar supporting structure is shown in Fig. 6. According to the rule of maximum stress is less than the ultimate strength of the two-bar supporting structure, the following limit state function is established:

Method

2W

Pf 4.42 × 10−3 4.42 × 10−3 4.41 × 10−3 4.41 × 10−3 4.50 × 10−3 4.37 × 10−3

6

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

Fig. 7. Convergence curve of the failure probability and weights.

example are consistent, and Ai(i = 1,…6) and Ei(i = 1,…6) denote the cross-sectional areas and Young's modulus of the corresponding units as represented in Fig. 9. Four nodal forces with values PAy = PAz = PBy = PBz = 104 are imposed at the A and B nodes. Additionally, two nodal forces PDx and PCx with random values are applied to the nodes C and D. All the random variables are independent, and their distribution parameters are listed in Table 4. The ANSYS 15.0 is used to calculate the maximum displacement of this structure, and failure occurs when the displacement is > 43 cm. Thus the corresponding limit state function reads

Table 3 Reliability analysis results of two-bar structure. Method

Model evaluations

MCS AK-MCS + U AK-MCS + EFF Adaptive SVR Bootstrap PCE Proposed method

7

10 136 137 116 66 51

Pf 6.62 × 10−4 6.62 × 10−4 6.62 × 10−4 7.11 × 10−4 6.56 × 10−4 6.63 × 10−4

g (x ) = 0.43

PCE and Kriging model provide more contribution in the ensemble model in this example. In Table 3, the final results of the failure probability are summarized, where the reference value obtained by MCS with 107 samples. One can see from Table 3 that AK-MCS + U and AK-MCS + EFF yield the most accurate results, and the presented method provides comparative results with much fewer model evaluations. In the meanwhile, the accuracy of adaptive SVR is poor for this example, which is due to that the convergence criteria used in [37] sometimes can't guarantee the actual convergence. In addition, the sparse PCE model also provides desired results for this example, but the proposed method outperforms it both in accuracy and efficiency.

max .

For this structure, 3 × 105 random samples generated with LHS are used as candidate sample pool, and the active learning algorithm is also initialized with 20 samples. In the active learning process, 30 new samples are added adaptively when the convergence criterion is reached. The convergence curves of failure probability and the weights of the ensemble model are shown in Fig. 9. It is found that the reliability analysis results of the three kinds of surrogate model disagree with each other to a great degree at the beginning. With the enrichment of samples by active learning algorithm, the estimated failure probabilities of different surrogate models converge to the reference value progressively. In addition, due to the smoothness of the performance function of twenty-five bar structure, we see that PCE model and Kriging model plays the dominant role in the ensemble model. The final results of failure probability are summarized in Table 5, and the reference value obtained by MCS is also provided. From Table 5, one can conclude that the bootstrap PCE and the presented method provide desired results with much fewer model evaluations compared to other methods. However, the model evaluations of the presented method is larger than that of bootstrap PCE method to some degree, which is because that the convergence criteria we used is relatively conservative. Indeed, we see from Fig. 9 that the failure probability obtained by PCE and Kriging converges to the reference value very fast, but SVR model delays the convergence rate of the ensemble model. In Table 6, we list the computational times of different methods to illustrate the overall efficiency. It is observed that although bootstrap PCE (100 sparse PCE models) and the proposed ensemble model (3 different surrogate models) require to construct multiple surrogate models, the computational costs of the two methods are still less than that of Kriging and SVR models. Due to the fast convergence rate, the bootstrap PCE and presented ensemble model obtained accurate results with much fewer iterations than that of Kriging and SVR models. Also, with the increase of the training samples size, the training time of Kriging and SVR models increases gradually in the active learning process, which also leads to high computational cost. This example

4.3. Twenty-five bar structure A twenty-five bar truss structure is shown in Fig. 8. All units in this

Fig. 8. 3D structural sketch of twenty-five bar structure. 7

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

Fig. 9. Convergence curve of the failure probability and weights. Table 4 Distribution parameters of the inputs for twenty-five bar structure. Inputs

Distribution

E1-5 E6 PCx PDx Ai (i = 1, …, 6)

Mean 7

Normal Normal Normal Normal Normal

Standard deviation 2 × 105 1.5 × 106 50 50 0.115

10 107 500 500 2.3

Table 5 Reliability analysis results of twenty-five bar structure. Method

Model evaluations

Pf

MCS AK-MCS + U AK-MCS + EFF Adaptive SVR Bootstrap PCE Proposed method

3 × 105 161 217 205 45 50

0.00821 0.00821 0.00821 0.00845 0.00827 0.00827

Fig. 10. Heat conduction problem: Domain and boundary conditions.

domain within D. The thermal conductivity k (z ) is a lognormal Gaussian random field described byk (z ) = exp(0.5g (z )) , where g (z ) is a standard Gaussian random field with square-exponential autocorrelation function:

Table 6 Computational time comparisons of twenty-five bar structure. Methods

Total computational time

MCS AK-MCS + U AK-MCS + EFF Adaptive SVR Bootstrap PCE Proposed method

432000 s 2035.85 s 3559.53 s 7649.50 s 236.12 s 159.32 s

(z , z ) = exp(

z

2 / 2),

(24)

where controls the correlation of thermal conductivity at two points. The response quantity of interest (QoI) of this model is the average temperature in the whole domain ofB = ( 0.2, 0.3) m× ( 0.2, 0.3) m

Y=

demonstrate the accuracy and efficiency of our method for engineering applications.

1 |B|

z B

T (z ) dz .

(25)

The square domain D = ( 0.5, 0.5) m× ( 0.5, 0.5) m is uniformly dispersed as M = 11 × 11 points, and we can obtain the correlation matrix of the random process as Cg (i, j ) = (z i , z j ) for i , j = 1, …, 121. By retaining the first M1 terms in the expansion optimal linear estimation (EOLE) series, g (z ) can be approximated as [63]:

4.4. Heat conduction problem Consider a stationary heat conduction problem [63] in the two-dimensional square domain D = ( 0.5, 0.5) m× ( 0.5, 0.5) m shown in Fig. 10. The temperature field T (z ), z D is described by the partial differential equation (PDE)

(k (z ) T (z )) = IA (z ) Q,

z

~ g (z ) =

M1 i=1

(23)

i

li

T i Cg (z ),

(26)

where i (i = 1, …, M1) are independent standard normal variables, (li , i ) is the eigenvalues and eigenvectors of correlation matrix Cg , and Cg (z ) is the vector of correlation coefficients with element (z , zj ) for j = 1, …, M respectively. The number of terms retained in the EOLE series is determined by the following rule:

with boundary conditions T = 0 on the top boundary and T·n = 0 on the left, right and bottom boundaries, where n denotes the vector normal to the boundary. In Eq. (36), Q = 2000W/m3 and 1z A IA (z ) = , where A = (0.2, 0.3) m× (0.2, 0.3)m is a square 0z A

{

8

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

Fig. 11. Finite element mesh and a realizations of temperature field. M1

M

li / i=1

li i=1

0.99.

Table 7 Reliability analysis results for case 1 of heat conduction problem.

(27)

Method

For this heat conduction problem, the underlying deterministic problem is solved with the finite element method in Matlab environment. The square domain D is discretized into 3738 triangular elements as shown in Fig. 11(a). The temperature field T (z ) for a realization of the thermal conductivity random field is depicted in Fig. 11(b). In this example, the temperature threshold is 1.6 °C , thus the limit state function is defined as g (x ) = Y 1.6. Case 1: we firstly set = 0.5, which leads to Ml = 13 according to Eq. (27). In this case, 105 random samples generated with LHS are used as candidate sample pool. In the active learning algorithm, 20 samples are used as the initial DoE, and the convergent result is obtained when 134 new samples are added iteratively. The convergence curves of failure probability and the weights of the ensemble model are shown in Fig. 12. It is found that the failure probabilities obtained by the three kinds of surrogate models converge to the reference value nearly simultaneously. The variation trend of the weights tell us that the three kinds of surrogate models are competitive in this example, and each surrogate is of importance to the ensemble model. The final results of failure probability are summarized in Table 7, and one can conclude that the presented method provides desired results with much fewer model evaluations compared to other methods. To compare the efficiency of each method comprehensively, the total computational time of various methods are listed in Table 8. One can

MCS AK-MCS + U AK-MCS + EFF Adaptive SVR Bootstrap PCE Proposed method

Model evaluations 5

10 340 422 226 572 154

Pf 0.01043 0.01044 0.01043 0.01110 0.01045 0.01048

Table 8 Computational time comparisons for case 1 of heat conduction problem. Method

Total computational time

MCS AK-MCS + U AK-MCS + EFF Adaptive SVR Bootstrap PCE Proposed method

787000 s 5311.99 s 8313.16 s 8599.08 s 7021.13 s 3210.10 s

conclude from Table 8 that the presented method is superior to other methods in efficiency. Due to the fast convergence rate, the proposed method requires less iterations than other methods. This example

Fig. 12. Convergence curve of the failure probability and weights. 9

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

Fig. 13. Convergence curve of the failure probability.

Fig. 14. Convergence curve of weights in the ensemble model.

demonstrates the efficiency and accuracy of the presented method for complex engineering applications. Case 2: here we set = 0.35, which leads to Ml = 22 according to Eq. (27). In this case, 2 × 105 random samples generated with LHS are used as candidate sample pool. To ensure the least square problem in Kriging model is solvable, 30 samples are used as the initial DoE. For this highdimensional problem, the convergent result is obtained when 361 new samples are added iteratively. The convergence curves of failure probability and the weights of the ensemble model are shown in Figs. 13 and 14. One can see that the failure probabilities obtained by the three kinds of surrogate models converge to the reference value nearly simultaneously. The variation trend of the weights in Fig. 14 indicates that the PCE and Kriging make more contribution to the ensemble model. The final results of failure probability are listed in Table 9, and one

Table 10 Computational time comparisons for case 2 of heat conduction problem.

MCS AK-MCS + U AK-MCS + EFF Adaptive SVR Bootstrap PCE Proposed method

Model evaluations 5

2 × 10 751 > 1000 531 557 391

Total computational time

MCS AK-MCS + U AK-MCS + EFF Adaptive SVR Bootstrap PCE Proposed method

4.85 × 106s 6.65 × 104s > 1.11 × 105s 1.16 × 105s 3.08 × 104s 3.68 × 104s

can conclude that the presented method provides desired results with much fewer true model evaluations compared to other single surrogate model methods. In this case, the convergence criteria of EFF learning function is too conservative. Therefore, AK-MCS + EFF method provides the most accurate result with the cost of > 1000 true model evaluations. To compare the efficiency of each method comprehensively, the total computational time of various methods are summarized in Table 10. For this high-dimensional problem, we see that Bootstrap PCE and the presented methods are superior to other methods in efficiency. Compared to Bootstrap PCE method, the presented ensemble model saves > 150 true model evaluations, but the tedious training process of the ensemble model reduces the efficiency of the presented method. However, for very expensive computational models in engineering, the time for training the ensemble model is much lower than single true

Table 9 Reliability analysis results for case 2 of heat conduction problem. Method

Method

Pf 0.00535 0.00536 0.00535 0.00551 0.00539 0.00531

10

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu

model evaluations. Therefore, the proposed ensemble model is an effective method to improve the computational efficiency of reliability analysis for complex engineering problems.

[17] Peherstorfer B, Cui T, Marzouk Y, Willcox K. Multifidelity importance sampling. Comput Methods Appl Mech Eng 2016;300:490–509. [18] Papadopoulos V, Giovanis DG, Lagaros ND, Papadrakakis M. Accelerated subset simulation with neural networks for reliability analysis. Comput Methods Appl Mech Eng 2012;223–224:70–80. [19] Xiao S, Reuschen S, Köse G, Oladyshkin S, Nowak W. Estimation of small failure probabilities based on thermodynamic integration and parallel tempering. Mech Syst Sig Process 2019;133:106248. [20] Marelli S, Sudret B. An active-learning algorithm that combines sparse polynomial chaos expansions and bootstrap for structural reliability analysis. Struct Saf 2018;75:67–74. [21] Lv Z, Lu Z, Wang P. A new learning function for Kriging and its applications to solve reliability problems in engineering. Comput Math Appl 2015;70:1182–97. [22] Dubourg V, Sudret B, Deheeger F. Metamodel-based importance sampling for structural reliability analysis. Probab Eng Mech 2013;33:47–57. [23] Sun Z, Wang J, Li R, Tong C. LIF: A new Kriging based learning function and its application to structural reliability analysis. Reliab Eng Syst Saf 2017;157:152–65. [24] Zhang L, Lu Z, Wang P. Efficient structural reliability analysis method based on advanced Kriging model. Appl Math Model 2015;39:781–93. [25] Kang F, Li JS, Li JJ. System reliability analysis of slopes using least squares support vector machines with particle swarm optimization. Neurocomputing 2016;209:46–56. [26] Bourinet JM. Rare-event probability estimation with adaptive support vector regression surrogates. Reliab Eng Syst Saf 2016;150:210–21. [27] Chowdhury R, Rao BN. Hybrid High Dimensional Model Representation for reliability analysis. Comput Methods Appl Mech Eng 2009;198:753–65. [28] Echard B, Gayton N, Lemaire M. AK-MCS: An active learning reliability method combining Kriging and Monte Carlo Simulation. Struct Saf 2011;33:145–54. [29] Echard B, Gayton N, Lemaire M, Relun N. A combined Importance Sampling and Kriging reliability method for small failure probabilities with time-demanding numerical models. Reliab Eng Syst Saf 2013;111:232–40. [30] Gaspar B, Teixeira AP, Guedes Soares C. Adaptive surrogate model with active refinement combining Kriging and a trust region method. Reliab Eng Syst Saf 2017;165:277–91. [31] Li X, Qiu H, Chen Z, Gao L, Shao X. A local Kriging approximation method using MPP for reliability-based design optimization. Comput Struct 2016;162:102–15. [32] Peijuan Z, Ming WC, Zhouhong Z, Liqi W. A new active learning method based on the learning function U of the AK-MCS reliability analysis method. Eng Struct 2017;148:185–94. [33] Xiao N-C, Zuo MJ, Guo W. Efficient reliability analysis based on adaptive sequential sampling design and cross-validation. Appl Math Model 2018;58:404–20. [34] Yang X, Liu Y, Zhang Y, Yue Z. Probability and convex set hybrid reliability analysis based on active learning Kriging model. Appl Math Model 2015;39:3954–71. [35] Bichon BJ, Eldred MS, Swiler LP, Mahadevan S, McFarland JM. Efficient Global Reliability Analysis for Nonlinear Implicit Performance Functions. AIAA Journal 2008;46:2459–68. [36] Yang X, Liu Y, Gao Y, Zhang Y, Gao Z. An active learning kriging model for hybrid reliability analysis with both random and interval variables. Struct Multidiscip Optim 2015;51:1–14. [37] Moustapha M, Sudret B. Surrogate-assisted reliability-based design optimization: a survey and a new general. framework. 2019. [38] The Wiener N, Chaos Homogeneous. American Journal of Mathematics 1938;60:897–936. [39] Xiu D, Karniadakis GE. The Wiener–Askey Polynomial Chaos for Stochastic Differential Equations. SIAM J SCI COMPUT2002. p. 619–44. [40] Salehi S, Raisee M, Cervantes MJ, Nourbakhsh A. An efficient multifidelity ℓ1minimization method for sparse polynomial chaos. Comput Methods Appl Mech Eng 2018;334:183–207. [41] Sudret B. Global sensitivity analysis using polynomial chaos expansions. Reliab Eng Syst Saf 2008;93:964–79. [42] Crestaux T, Le Maı ̂tre O, Martinez J-M. Polynomial chaos expansion for sensitivity analysis. Reliab Eng Syst Saf 2009;94:1161–72. [43] Cheng K, Lu Z. Adaptive sparse polynomial chaos expansions for global sensitivity analysis based on support vector regression. Comput Struct 2018;194:86–96. [44] Yang X, Karniadakis GE. Reweighted ℓ1 minimization method for stochastic elliptic differential equations. J Comput Phys 2013;248:87–108. [45] Abraham S, Raisee M, Ghorbaniasl G, Contino F, Lacor C. A robust and efficient stepwise regression method for building sparse polynomial chaos expansions. J Comput Phys 2017;332:461–74. [46] Cheng K, Lu Z. Sparse polynomial chaos expansion based on D-MORPH regression. Appl Math Comput 2018;323:17–30. [47] Diaz P, Doostan A, Hampton J. Sparse polynomial chaos expansions via compressed sensing and D-optimal design. Comput Methods Appl Mech Eng 2018;336:640–66. [48] Mathelin L, Gallivan KA. A Compressed Sensing Approach for Partial Differential Equations with Random Input Data. Comm Comput Phys 2015;12:919–54. [49] Blatman G, Sudret B. Adaptive sparse polynomial chaos expansion based on least angle regression. J Comput Phys 2011;230:2345–67. [50] Blatman G, Sudret B. An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis. Probab Eng Mech 2010;25:183–97. [51] Marelli S, Sudret B. UQLab: a framework for Uncertainty Quantification in. MATLAB. ETH-Zürich 2015. [52] Krige DG. A Statistical Approach to Some Basic Mine Valuation Problems on the Witwatersrand. OR 1953;4:18. [53] Matheron G. Principles of geostatistics. Econ Geol 1963;58:1246–66. [54] Vapnik Vladimir N. The nature of statistical learning theory. IEEE Trans Neural

5. Conclusions This paper presents an active learning framework for structural reliability analysis by ensemble learning of multiple surrogate models. By fitting a performance function with multiple different surrogate models, one can generally get a more robust predictor to estimate the failure probability. In addition, the presented ensemble strategy provides the prediction variance to measure the local prediction disagreement degree of multiple surrogate models, and thus allows one to place more samples to regions with large prediction discordance to refine the ensemble model as much as possible. The performance of the proposed method has been validated by several benchmark examples, and the results show that our method provides comparative results compared to other well-developed active learning algorithm with fewer true model evaluations. In some cases, some “bad” surrogate models may degenerate the performance of the ensemble model. In this regard, we need to develop more efficient weight strategy and robust convergence criteria to filter out the “bad” surrogate models from the ensemble model. To estimate small failure probability (〈1 0 −4), the presented ensemble model is computational demanding due to the large candidate sample pool (> 107). Thus more advanced active learning algorithm by combining ensemble model with more efficient numerical simulation methods such as IS or SS needs to be developed in the future. Acknowledgments The authors would like to express the gratitude to two reviewers for helpful comments and constructive suggestions. This work was supported by the National Natural Science Foundation of China (Grant No. NSFC 51775439), National Science and Technology Major Project (Grant No. 2017-IV-0009-0046), and “Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University” with project code of CX201933. References [1] Hasofer AM, Lind NC. An Exact and Invariant First Order Reliability Format. J Eng Mech 1974. [2] Zhao Y-G, Ono T. A general procedure for first/second-order reliabilitymethod (FORM/SORM). Struct Saf 1999;21:95–112. [3] Lu Z, Song J, Song S, Yue Z, Wang J. Reliability sensitivity by method of moments. Appl Math Model 2010;34:2860–71. [4] Zhao YG, Ono T. On the problems of the Fourth moment method. Struct Saf 2004;26:343–7. [5] Zhao YG, Ono T. Moment methods for structural reliability. Struct Saf 2001;23:47–75. [6] Xiong F, Greene S, Chen W, Xiong Y, Yang S. A new sparse grid based method for uncertainty propagation. Struct Multidiscip Optim 2010;41:335–49. [7] Xu J, Dang C. A new bivariate dimension reduction method for efficient structural reliability analysis. Mech Syst Sig Process 2019;115:281–300. [8] Melchers RE. Importance sampling in structural system. Struct Saf 1989;6:3–10. [9] Wei P, Lu Z, Hao W, Feng J, Wang B. Efficient sampling methods for global reliability sensitivity analysis. Comput Phys Commun 2012;183:1728–43. [10] Au SK, Beck JL. Important sampling in high dimensions. Struct Saf 2003;25:139–63. [11] Au SK, Beck JL. Estimation of small failure probabilities in high dimensions by subset simulation. Probab Eng Mech 2001;16:263–77. [12] Bourinet JM, Deheeger F, Lemaire M. Assessing small failure probabilities by combined subset simulation and Support Vector Machines. Struct Saf 2011;33:343–53. [13] Song S, Lu Z, Qiao H. Subset simulation for structural reliability sensitivity analysis. Reliab Eng Syst Saf 2009;94:658–65. [14] Lu Z, Song S, Yue Z, Wang J. Reliability sensitivity method by line sampling. Struct Saf 2008;30:517–32. [15] Pradlwarter HJ, Schuëller GI, Koutsourelakis PS, Charmpis DC. Application of line sampling simulation method to reliability benchmark problems. Struct Saf 2007;29:208–21. [16] Ditlevsen O. General multi-dimensional probability integration by directional simulation. Comput Struct 1990;36:355–68.

11

Structural Safety 83 (2020) 101905

K. Cheng and Z. Lu Networks 1997;38:409-. [55] Vapnik VN. Statistical Learning Theory. Encyclopedia of the Sciences of. Learning 2008;41:3185-. [56] Goel T, Haftka RT, Wei S, Queipo NV. Ensemble of surrogates. Struct Multidiscip Optim 2007;33:199–216. [57] Viana FAC, Haftka RT, Watson LT. Efficient global optimization algorithm assisted by multiple surrogate techniques. J Global Optim 2013;56:669–89. [58] Zhao D, Xue D. A multi-surrogate approximation method for metamodeling. Eng Comput 2011;27:139–53. [59] Viana FAC. Multiple surrogates for prediction and optimization. Dissertations & Theses - Gradworks 2011.

[60] Huang X, Chen J, Zhu H. Assessing small failure probabilities by AK–SS: an active learning method combining Kriging and Subset Simulation. Struct Saf 2016;59:86–95. [61] Chang C, Lin C. LIBSVM: A library for support vector machines. ACM Trans Intell Syst Technol 2011;2:1–27. [62] Jian W, Zhili S, Qiang Y, Rui L. Two accuracy measures of the Kriging model for structural reliability analysis. Reliab Eng Syst Saf 2017;167:494–505. [63] Konakli K, Sudret B. Polynomial meta-models with canonical low-rank approximations: numerical insights and comparison to sparse polynomial chaos expansions. J Comput Phys 2016;321:1144–69.

12