Sensitivity analysis in conjunction with evidence theory representations of epistemic uncertainty

ARTICLE IN PRESS

Reliability Engineering and System Safety 91 (2006) 1414–1434 www.elsevier.com/locate/ress

Sensitivity analysis in conjunction with evidence theory representations of epistemic uncertainty J.C. Heltona,, J.D. Johnsonb, W.L. Oberkampfc, C.J. Sallaberryc a

Department of Mathematics and Statistics, Arizona State University, Tempe, AZ 85287-1804, USA b ProStat, Mesa, AZ 85204-5326, USA c Sandia National Laboratories, Albuquerque, NM 87185-0828, USA Available online 19 January 2006

Abstract Three applications of sampling-based sensitivity analysis in conjunction with evidence theory representations for epistemic uncertainty in model inputs are described: (i) an initial exploratory analysis to assess model behavior and provide insights for additional analysis; (ii) a stepwise analysis showing the incremental effects of uncertain variables on complementary cumulative belief functions and complementary cumulative plausibility functions; and (iii) a summary analysis showing a spectrum of variance-based sensitivity analysis results that derive from probability spaces that are consistent with the evidence space under consideration. r 2005 Elsevier Ltd. All rights reserved. Keywords: Epistemic uncertainty; Evidence theory; Sensitivity analysis; Uncertainty analysis; Variance decomposition

1. Introduction Uncertainty analysis and sensitivity analysis should be important components of any analysis of a complex system, with (i) uncertainty analysis providing a representation of the uncertainty present in the estimates of analysis outcomes and (ii) sensitivity analysis identifying the contributions of individual analysis inputs to the uncertainty in analysis outcomes [1]. Probability theory provides the mathematical structure traditionally used in the representation of epistemic (i.e., state of knowledge) uncertainty, with the uncertainty in analysis outcomes represented with probability distributions and typically summarized as cumulative distribution functions (CDFs) or complementary cumulative distribution functions (CCDFs) [2–4]. A variety of sensitivity analysis procedures have been developed for use in conjunction with probabilistic representations of uncertainty, including differential analysis [5,6], the Fourier amplitude sensitivity test (FAST) Corresponding author. Department 1533, MS 0779, Sandia National Laboratories, Albuquerque, NM 87185-0779, USA. Tel.: +1 505 284 4808; fax: +1 505 844 2348. E-mail address: [email protected] (J.C. Helton).

0951-8320/$ - see front matter r 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.ress.2005.11.055

and related variance decomposition procedures [7–11], regression-based techniques [12,13], and searches for nonrandom patterns [14]. Additional background information on uncertainty and sensitivity analysis is available in several review presentations [1,14–19]. Although probabilistic representations of uncertainty have been successfully employed in many analyses, such representations have been criticized for inducing an appearance of more refined knowledge with respect to the existing uncertainty than is really present [20,21]. Much of this criticism derives from the use of uniform distributions to characterize uncertainty in the presence of little or no knowledge with respect to where the appropriate value to use for a parameter is located within a set of possible values [22, pp. 52–62]. As a result, a number of alternative mathematical structures for the representation of epistemic uncertainty have been proposed, including evidence theory, possibility theory, and fuzzy set theory [23–26]. Evidence theory provides a promising alternative to probability theory that allows for a fuller representation of the implications of uncertainty than is the case in a probabilistic representation of uncertainty [27–33]. In particular, evidence theory involves two representations of the uncertainty associated with a set of possible analysis

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

inputs or results: (i) a belief, which provides a measure of the extent to which the available information implies that the true value is contained in the set under consideration, and (ii) a plausibility, which provides a measure of the extent to which the available information implies that the true value might be contained in the set under consideration. One interpretation of the belief and plausibility associated with a set is that (i) the belief is the smallest possible probability for the set that is consistent with all available information and (ii) the plausibility is the largest possible probability for the set that is consistent with all available information. An alternative interpretation is that evidence theory is an internally consistent mathematical structure for the representation of uncertainty without any explicit conceptual link to probability theory. The mathematical operations associated with evidence theory are the same for both interpretations. Just as probability theory uses CDFs and CCDFs to summarize uncertainty, evidence theory uses cumulative belief functions (CBFs), cumulative plausibility functions (CPFs), complementary cumulative belief functions (CCBFs), and complementary cumulative plausibility functions (CCPFs) to summarize uncertainty. Although evidence theory is beginning to be used in the representation of uncertainty in applied analyses, the authors are unaware of any attempts to develop sensitivity analysis procedures for use in conjunction with evidence theory. Due to the importance of sensitivity analysis in any decision-aiding analysis, the potential usefulness of evidence theory will be enhanced if meaningful and practicable sensitivity analysis procedures are available for use in analyses that employ evidence theory in the representation of uncertainty. As a result, the focus of this presentation is on the development of sensitivity analysis procedures for use in conjunction with evidence theory representations of uncertainty. After a brief overview of evidence theory (Section 2), the following topics are considered: (i) exploratory sensitivity analysis (Section 3), (ii) use of sensitivity analysis results in the stepwise construction of CCBFs and CCPFs (Section 4), (iii) summary sensitivity analysis of evidence theory representations of uncertainty (Section 5), (iv) example results (Section 6), (v) formal justification of procedure used in stepwise construction of CCBFs and CCPFs (Section 7), and (vi) concluding summary (Section 8). 2. Evidence theory Evidence theory is based on the specification of a triple ðS; S; mÞ, where (i) S is the set that contains everything that could occur in the particular universe under consideration, (ii) S is a countable collection of subsets of S, and (iii) m is a function defined on subsets of S such that mðEÞ40 if E 2 S; mðEÞ ¼ 0 if E
S and SeS, and SE2S mðEÞ ¼ 1. For a subset E of S; mðEÞ characterizes the amount of ‘‘likelihood’’ that can be assigned to E but to no proper subset of E. In the terminology of evidence theory,

1415

(i) S is the sample space or universal set, (ii) S is the set of focal elements for S and m, and (iii) mðEÞ is the basic probability assignment (BPA) associated with a subset E of S. The elements of S are often vectors x ¼ ½x1 ; x2 ; . . . ; xn , where each element xi of x is a variable with its own evidence space ðSi ; Si ; mi Þ. When the xi’s are assumed to be independent, (i) mðEÞ ¼ Pi mi ðEi Þ if E ¼ E1  E2      En and Ei 2 Si for i ¼ 1; 2; . . . ; n, and (ii) mðEÞ ¼ 0 otherwise. An evidence space ðS; S; mÞ plays the same role in evidence theory that a probability space ðP; P; pÞ plays in probability theory, where P is the sample space, P is a suitably restricted set of subsets of P (i.e., a s-algebra), and p is the function (i.e., probability measure) that assigns probabilities to elements of P [34, Section IV.3]. The belief, BelðEÞ, and plausibility, PlðEÞ, for a subset E of S are defined by X X BelðEÞ ¼ mðUÞ and PlðEÞ ¼ mðUÞ. (2.1) U
E

U\Eaf

In concept, BelðEÞ is the amount of ‘‘likelihood’’ that must be assigned to E, and PlðEÞ is the maximum amount of ‘‘likelihood’’ that could possibly be assigned to E. When the elements of S are real valued, a CCBF and a CCPF provide a convenient summary of an evidence space ðS; S; mÞ and correspond to plots of the points CCBF ¼ f½v; BelðSv Þ; v 2 Sg CCPF ¼ f½v; PlðSv Þ; v 2 Sg,

and ð2:2Þ

where Sv ¼ fx : x 2 S and x4vg. An important situation in the application of evidence theory is the consideration of a variable y ¼ f ðxÞ, where f is a function defined for elements x of the sample space X associated with an evidence space ðX; X; mX Þ and x is represented as a vector because this is the case in most real analyses. The properties of f and ðX; X; mX Þ induce an evidence space ðY; X; mY Þ on y, which provides a characterization of the uncertainty associated with y. In turn, this uncertainty can be summarized with a CCBF and a CCPF defined by CCBF ¼ f½v; BelX ff 1 ðYv Þg; v 2 Yg CCBF ¼ f½v; PlX ff

1

ðYv Þg; v 2 Yg,

and ð2:3Þ

where BelX and PlX denote belief and plausibility defined with respect to ðX; X; mX Þ and Yv ¼ fy : y 2 Y and y4vg. The generation and analysis of CCBFs and CCPFs of the preceding form are fundamental parts of the use of evidence theory to characterize the uncertainty in model predictions. Additional discussion of evidence theory and its relationship to probability theory employing the same notation used in this presentation is available elsewhere [24,35]. Evidence theory derives from initiating work by Dempster [27,36,37] and Shafer [28], and as a result, Dempster–Shafer theory is often used as an alternative designation for evidence theory. Evidence theory has been widely studied and a number of summary references are available (e.g., [29–33]).

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1416

3. Exploratory sensitivity analysis An initial exploratory sensitivity analysis plays an important role in helping to guide any study that involves uncertain inputs. This is particularly true in uncertainty analyses based on evidence theory as the uncertainties are likely to be large and an appropriate understanding of these uncertainties and their implications can provide insights that facilitate the computational estimation of beliefs and plausibilities. Given that large uncertainties in many variables are likely to be present, a sampling-based approach to sensitivity analysis with Latin hypercube sampling [38,39] is a broadly applicable procedure for an exploratory analysis in conjunction with an evidence theory representation for uncertainty. Use of this approach requires the specification of distributions for the uncertain variables for sampling purposes. This specification should provide for an adequate exploration of the range of each uncertain variable and be consistent, in some sense, with the evidence theory specification of the uncertainty associated with individual analysis inputs. A distribution that meets the preceding criteria can be obtained by sampling each focal element associated with a variable in consistency with its BPA and then sampling uniformly within that focal element. With the assumption that each focal element for a variable xi with an evidence space ðXi ; Xi ; mi Þ is an interval, this corresponds to defining a sampling distribution with a density function di given by d i ðvÞ ¼

CðX Xi Þ

dij ðvÞmi ðEij Þ=ðbij  aij Þ,

(3.1)

j¼1

where (i) v 2 Xi , (ii) CðXi Þ is the cardinality of Xi , (iii) Eij ¼ ½aij ; bij ; j ¼ 1; 2; . . . ; CðXi Þ, are the focal elements associated with xi (i.e., the elements of Xi ), and (iv) dij ðvÞ ¼ 1 if v 2 Eij and 0 otherwise. Appropriate modifications can be made to the preceding definition to handle focal elements with a finite number of elements and focal elements that are unions of disjoint intervals. Given that a relationship of the form y ¼ f ðxÞ; x ¼ ½x1 ; x2 ; . . . ; xn , is under consideration, sampling according to the distributions indicated in Eq. (3.1) generates a mapping yj ¼ f ðxj Þ from uncertain analysis inputs to uncertain analysis results, where xj ; j ¼ 1; 2; . . . ; nS, are the sampled values for x. As previously indicated, Latin hypercube sampling is a likely candidate for the sampling procedure because of its efficient stratification properties. Once this mapping is generated, it can be explored with various sensitivity analysis procedures to develop an understanding of the relationship between y and the individual elements of x. A variety of techniques are available for use in samplingbased sensitivity analyses [13,16]. However, given that the analysis problem is based on evidence theory, sensitivity analysis procedures that do not place excessive reliance on the sampling distributions indicated in Eq. (3.1) are

desirable. Of course, no approach can fully divorce itself from these distributions because they ultimately give rise to the raw material of the sensitivity analysis (i.e., the mapping ½xj ; yj ; j ¼ 1; 2; . . . ; nS); however, this is an unavoidable situation when the sample space associated with x is infinite as no approach can consider all values of x and so a subset of the values for x must be selected in some manner. The examination of scatterplots is a natural initial procedure. Then, rank-based procedures (e.g., rank regression, partial rank correlation, squared rank differences) are natural techniques to employ because they reduce the effects of both nonlinearities and the original sampling distributions [13,40,41]. If carried out successfully, an initial exploratory sensitivity analysis should provide important insights with respect to the relationship between y and the elements of x. Often, only a few of the elements of x will have significant effects on y. This is information that can be productively used in the estimation of the evidence theory structure associated with y. 4. Stepwise construction of CCBFs and CCPFs For most models, the determination of beliefs and plausibilities for model predictions in general, and CCBFs and CCPFs in particular, is a demanding numerical challenge due to the need to determine the inverse of the model (i.e., function) involved. Sampling-based (i.e., Monte Carlo) procedures provide one way to carry out such determinations. With this approach, a sample xj ; j ¼ 1; 2; . . . ; nS, is generated from X (e.g., with distributions for the elements of x of the form indicated in Eq. (3.1)), and y is evaluated for each xk to create the mapping ½xj ; yj ; j ¼ 1; 2; . . . nS, from X to Y. Then, the CCBF and CCPF for y can be estimated by CCBF ffi f½y; 1  PlX ðfxj : yj pygÞ; y 2 Yg

(4.1)

and CCPF ffi f½y; PlX ðfxj : yj 4ygÞ; y 2 Yg,

(4.2)

respectively. The approximation to CCBF for y in Eq. (4.1) is based on the equality BelðEÞ þ PlðEc Þ ¼ 1 and the fact that the subset criterion in the definition of belief (see Eq. (2.1)) does not allow for the direct estimate of belief with a finite sample when sets with infinite numbers of elements are under consideration. In general, the same approach can be used to estimate the belief BelY ðEÞ and plausibility PlY ðEÞ for any subset E of Y. The problem with the preceding approach is that it can be prohibitively expensive computationally when the cardinality CðXÞ of X is high, which is usually the case in real analyses. Specifically, CðXÞ ¼ Pi CðXi Þ, where CðXi Þ is the cardinality of Xi . For example, if n ¼ 8 and CðXi Þ ¼ 10, then CðXÞ ¼ 108 ; and as a result, a very large sample would be required to converge the approximations to the CCBF and CCPF in Eqs. (4.1) and (4.2).

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

The results of the exploratory sensitivity analysis described in Section 3 provide a basis for a potential path forward in developing the CCBF and CCPF approximations in Eqs. (4.1) and (4.2). The uncertainty in most analysis outcomes is significantly affected by the uncertainty in only a small number of analysis inputs (e.g., [4,13,14]). Of course, this does not have to be the case but it does seem usually to be the case. In this situation, the approximations in Eqs. (4.1) and (4.2) can be determined by only considering the uncertainty (i.e., the evidence spaces ðXi ; Xi ; mi Þ) associated with the xi that significantly affect y. The remaining xi (i.e., those that do not have a significant effect on y) can be assigned degenerate evidence spaces (i.e., spaces ðXi ; Xi ; mi Þ for which mi ðXi Þ ¼ 1) for use in evaluating the approximations in Eqs. (4.1) and (4.2). Increasing the resolution in the evidence spaces assigned to individual xi (i.e., by subdividing elements of Xi and then apportioning the BPA for an original element of Xi over the subsets into which it is subdivided) tends to decrease, and can never increase, the uncertainty associated with evidence space for y. Specifically, beliefs tend to increase (and can never decrease) and plausibilities tend to decrease (and can never increase); or put another way, beliefs and plausibilities for subsets of Y move closer together as the resolution in the characterization of the uncertainties associated with the xi is increased. The preceding observations provide a basis for the use of sensitivity analysis results to guide a stepwise procedure for the construction of the CCBF and CCPF approximations in Eqs. (4.1) and (4.2). At Step 1, the approximations in Eqs. (4.1) and (4.2) are determined with the most important variable affecting the uncertainty in y assigned its original evidence space and all other variables assigned evidence spaces in which their original sample spaces are assigned a BPA of 1. At Step 2, the approximations in Eqs. (4.1) and (4.2) are determined with the two most important variables affecting the uncertainty in y assigned their original evidence spaces and all other variables assigned evidence spaces in which their original sample spaces are assigned a BPA of 1. Analogous steps follow for additional important variables determined in the sensitivity analysis until substantive changes in the CCBF and CCPF approximations in Eqs. (4.1) and (4.2) no longer occur, at which point the approximation procedure stops. This approach can produce substantial computational savings over what would be incurred if the approximations in Eqs. (4.1) and (4.2) were evaluated with the original evidence spaces assigned to all the xi. The construction procedure just outlined can also be viewed as a sensitivity analysis in the context of evidence theory. The changes in the location of the CCBF and CCPF as additional variables are added in the preceding procedure provides an indication of the importance of individual variables with respect to the uncertainty in y characterized by ðY; Y; mY Þ. At an intuitive level, this

1417

approach is analogous to the use of stepwise regression analysis in traditional sensitivity analyses. 5. Summary sensitivity analysis Together, a CCBF and CCPF for y provide bounds on all possible CCDFs for y that could derive from different distributions for the xi that are consistent with their specified evidence spaces ðXi ; Xi ; mi Þ. In particular, if ðPi ; Pi ; pi Þ is a probability space for xi that is consistent with the evidence space ðXi ; Xi ; mi Þ for i ¼ 1; 2; . . . ; n, then these probability spaces give rise to corresponding probability spaces ðPX ; PX ; pX Þ and ðPY ; PY ; pY Þ for x and y with the CCDF associated with ðPY ; PY ; pY Þ falling somewhere between the CCBF and CCPF for y. Traditional sensitivity analysis methods can be used to investigate the relationships between the uncertainty in the xi characterized by the probability spaces ðPi ; Pi ; pi Þ and the uncertainty in y characterized by the probability space ðPY ; PY ; pY Þ. A possible approach is a variance decomposition for y that partitions the variance for y into the contributions to this variance from the individual xi [8–10]. However, unlike a traditional sensitivity analysis in which the probability spaces ðPi ; Pi ; pi Þ are uniquely specified, there are many possibilities for the spaces ðPi ; Pi ; pi Þ in an evidence theory context and thus many possible variance decompositions for y. In variance-based sensitivity analysis, the variance V(y) of y is expressed as V ðyÞ ¼

n X

Vi þ

i¼1

n X n X

V ij þ    þ V 12n ,

(5.1)

i¼1 j¼iþ1

where Vi is the contribution of xi to V(y), Vij is the contribution of the interaction of xi and xj to V(y), and so on up to V12yn which is the contribution of the interaction of x1 ; x2 ; . . . ; xn to V(y). Possible sensitivity measures are provided by si ¼ V i =V ðyÞ and siT ¼

Vi þ

X

V ij þ    þ V 12n

!, V ðyÞ,

ð5:2Þ

jai

where si the fraction of V(y) contributed by xi alone and siT is the fraction of V(y) contributed by xi and interactions of xi with other variables. The term Vi is defined by iterated integrals involving the probability spaces for the individual variables. For example, when n ¼ 3, 2 Z
Z Z V1 ¼ f ðx1 ; x2 ; x3 Þd 3 ðx3 Þd 2 ðx2 Þ dx3 dx2 P1

P2

P3

 d 1 ðx1 Þ dx1  E 2 ðyÞ,

ð5:3Þ

where di denotes the density function associated with ðPi ; Pi ; pi Þ and E(y) denotes the expected value of y; similar defining integrals hold for V2 and V3, and related, but more complicated, integrals define V12, V13, V23 and V123. Analogous relationships hold for n43. By suitably

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1418

orchestrating an analysis, Vi and si for i ¼ 1; 2; . . . ; n can be estimated with two random or Latin hypercube samples; further, si and siT for i ¼ 1; 2; . . . ; n can be estimated with a total of n+2 suitably defined samples. Three questions arise with respect to the implementation of a variance-based sensitivity analysis in the context of evidence theory: (i) How to select an appropriate spectrum of distributions for each xi from the infinite number of distributions that are consistent with ðXi ; Xi ; mi Þ?, (ii) How to implement the analysis in a computationally practicable manner for multiple distributions (i.e., multiple probability spaces ðPi ; Pi ; pi Þ) for each xi? (iii) How to display the results of the sensitivity analyses for multiple distributions of the xi and hence multiple distributions for x and y? The first question arises because there is no inherent structure associated with the infinite number of distributions for xi that are consistent with ðXi ; Xi ; mi Þ. The situation is analogous to that encountered in an interval analysis for a real-valued quantity except that the uncertain quantity is now a probability space rather than a number. As there is no way to consider all probability spaces consistent with ðXi ; Xi ; pi Þ and also no specific structure to guide the selection of individual probability spaces, some type of ad hoc procedure is needed to select representative probability spaces that are consistent with ðXi ; Xi ; pi Þ. Further, the number of selected distributions for each xi must be relatively small; otherwise, the total number of combinations of selected distributions for all n variables will be too large to be computationally practicable. An exploratory approach that should provide valuable information for many situations is to select three distributions for each xi, with (i) one distribution emphasizing the smaller values associated with each focal element, (ii) one distribution uniform over the range of each focal element, and (iii) one distribution emphasizing the larger values associated with each focal element. The distributions indicated in (i) and (iii) could be left and right triangular or left and right quadratic. Left and right triangular distributions are actually quite similar to uniform distributions and thus may not be good choices. For focal element Eij ¼ ½aij ; bij  associated with xi, the corresponding density functions dlij, duij and drij for left quadratic, uniform, and right quadratic distributions, respectively, over Eij are 2

d lij ðvÞ ¼

3ðbij  vÞ ; ðbij  aij Þ3

d rij ðvÞ ¼

3ðv  aij Þ2 , ðbij  aij Þ3

d uij ðvÞ ¼

1 ðbij  aij Þ

and ð5:4Þ

if v 2 Eij and d lij ðvÞ ¼ d uij ðvÞ ¼ d rij ðvÞ ¼ 0 otherwise. In turn, the left quadratic, uniform and right quadratic distribution functions dli, dui and dri for xi are given by d ci ðvÞ ¼

CðX Xi Þ

mi ðEij Þd cij ðvÞ,

j¼1

for v 2 Xi and c ¼ l; u; r.

(5.5)

The second question arises because computational cost can easily become unreasonable unless the analysis is carefully planned. As a first step, only those variables that actually affect y need to be considered. The preliminary sensitivity analysis described in Section 3 should, in most analyses, identify the four or five variables that have significant effects on y. It is only those variables that require consideration of their original evidence spaces as indicated in Eq. (5.5); the remaining variables can be assigned a uniform or some other convenient distribution. For example, if four xi affect y and the three distributions defined in Eq. (5.5) are considered for each of these xi, then 34 ¼ 81 different probability spaces result for x and hence for y. As a second step, the analysis can be designed to use the same samples in the evaluation of si and siT for all probability spaces defined for x (e.g., the 81 spaces indicated above). For example, if Latin hypercube sampling is used, it is necessary to actually evaluate f for samples from only one of the probability spaces for x; after these evaluations for f are performed, results for the other probability spaces for x under consideration (e.g., the remaining 80 probability spaces in the example above) can be obtained by reweighting the results obtained for the individual sample elements on the basis of the changed distributions for the xi’s [39,42]. A similar reweighting procedure is also available for random sampling [43]. The third question arises because of the difficulty of displaying the results of multiple sensitivity analyses for y in a reasonably compact and understandable format. Presenting the sensitivity analyses individually is unlikely to be adequate because of the large number of analyses involved and the resultant difficulty of observing trends in variable importance across analyses. A promising presentation format to employ for this representation is a cobweb plot, which provides a representation for a multidimensional distribution in a two-dimensional plot [44]. For example, if nPS probability spaces ðPXj ; PXj ; pXj Þ for x are under consideration and four uncertain variables have been identified for analysis, the results of the sensitivity analyses for y might be of the form sj ¼ ½ej ; vj ; s1j ; s2j ; s3j ; s4j ;

j ¼ 1; 2; . . . ; nPS,

(5.6)

where ej and vj are the expected value and variance for y that derive from the probability space ðPXj ; PXj ; pXj Þ for x and sij, i ¼ 1; 2; 3; 4, are the fractional contributions to vj as defined in the first equality in Eq. (5.2) for the four uncertain variables under consideration. With a cobweb plot, the nPS vectors in Eq. (5.6) can be presented in a single plot frame. Specifically, the individual elements of sj are designated by locations on the horizontal axis and their values correspond to locations on the vertical axis. In general, it may be necessary to use multiple axis scales for the vertical axis or to plot quantiles for the elements of sj rather than their actual values. Each sj results in a single point in each of the vertical columns associated with its elements. The identity of sj is maintained by a line that connects the values of its elements. As desired, the

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1419

cobweb plot allows the presentation of all sensitivity analysis results in a single plot frame and also facilitates the recognition of interactions between variables. In summary, the approach presented in this section to the performance and presentation of a sensitivity analysis for a function defined on an evidence space has three components: (i) definition of representative probability spaces for the analysis input x that are consistent with the evidence space for x; (ii) use of efficient sampling-based numerical procedures to decompose the variance of the analysis outcome y for each probability space for x; and (iii) use of cobweb plots to summarize the results of the sensitivity analyses for y carried out for the individual probability spaces for x. Thus, rather than having a single set of sensitivity analysis results for y, a spectrum of sensitivity analysis results for y is obtained that is consistent with the evidence space that characterizes the uncertainty in x. 6. Illustration of sensitivity analysis procedures Fig. 1. Example WL/SL system with one WL and one SL [45, Fig. 1].

6.1. Example analysis problem The procedures for sensitivity analysis in conjunction with evidence theory representations for epistemic uncertainty are now illustrated with a hypothetical competing risk problem. This problem involves a system with two weak links (WLs) and two strong links (SLs) in an accident involving a fire that has the potential to place the system in a condition that could allow an unintended, and undesirable, operation of the system [45]. The role of the SLs is to permit operation of the system only under intended conditions. The role of the WLs is to fail under accident conditions and thereby render the system incapable of operation. Although the system under study involves two WLs and two SLs, a system with only one WL and one SL is used to illustrate the ideas involved (Fig. 1). Under nonoperational conditions, the WL is closed (e.g., permits the passage of an electrical signal) and the SL is open (e.g., does not permit the passage of an electrical signal) (Fig. 1a). For the entire system to operate, the SL must close and thereby allow the passage of an activating signal to the system (Fig. 1b). In the event of an accident, it is highly undesirable for the SL to close and place the system in a configuration in which it can be accidentally activated (Fig. 1b). To prevent the potential for such an accidental activation, the WL is designed to fail before the SL under accident conditions (Fig. 1c) and thus render impossible the passage of an activating signal should the SL fail at a subsequent time (Fig. 1d). As an aside, the phrase ‘‘WL failure,’’ although widely used, is an oxymoron as such failure actually constitutes ‘‘WL success’’ in that the system has been deactivated by the intended (i.e., designed) operation of the WL. In the system under study, failure of both SLs before the failure of either WL is considered to be the undesirable event as this places the system in a configuration in which

an activating signal could result in operation of the system. The likelihood that such a configuration occurs conditional on a specific type of accident is referred to as probability of loss of assured safety (PLOAS). As previously indicated, the problem under consideration involves a fire accident with heating of the WLs and SLs. In essence, there is a race (i.e., a competing risk [46–49]) to determine whether the SLs or the WLs fail first as they increase in temperature as the accident progresses. The indicated probability (i.e., PLOAS) derives from the assumption that the exact temperatures at which the individual links will fail is not known precisely. Rather, there is assumed to be a random (i.e., aleatory) uncertainty resulting from manufacturing variability that determines the exact temperatures at which the individual links fail. The formal representation for the PLOAS is based on the following system properties for j ¼ 1; 2 and k ¼ 1; 2: TMPW Lj ðtÞ ¼ temperature ð CÞ of WL j at time t ðminÞ, (6.1) TMPSLk ðtÞ ¼ temperature ð CÞ of SL k at time t ðminÞ, (6.2) fW Lj ðTÞ ¼ density function ð C1 Þ for failure temperature of WL j

ð6:3Þ

and fSLk ðTÞ ¼ density function ð C1 Þ for failure temperature of SL k.

ð6:4Þ

Further, time is assumed to range from tMIN to tMAX with TMNSLk ¼ TMPSLk ðtMINÞ; and TMXSLk ¼ TMPSLk ðtMAX Þ,

ð6:5Þ

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1420

and TMPSLj(t) and TMPSLk(t) are assumed to be increasing functions of time. Given the properties in Eqs. (6.1)–(6.5) and the assumption that a link fails immediately upon reaching its failure temperature, the numeric value pF for PLOAS is given by 2 Z TMXSLk X pF ¼ fSLk ðT SL Þ TMNSLk

k¼1





8
: l¼1 lak ( 2 Y

9 = I½1; TMPSLl ½TMPSL1 k ðT SL Þ; fSLl  ; )

I½TMPWLj ½TMPSL1 k ðT SL Þ; 1; fWLj  dT SL ,

j¼1

ð6:6Þ where Z

b

f ðTÞ dT

Iða; b; f Þ ¼ a

for notational convenience. A derivation of the preceding result for an arbitrary number of WLs and an arbitrary number of SLs is presented in conjunction with Eq. (4.9) of Ref. [45]. Further, the preceding expression for pF can be evaluated with quadrature and also Monte Carlo procedures [45, Sections 4.2–4.4]. An alternate and equivalent representation for pF based on integration on time rather than on temperature is also possible [45, Eq. (4.8)]. In a typical real problem, the temperature curves TMPWLj(t) and TMPSLk(t) would be determined by the numerical solution of a system of nonlinear partial differential equations. However, to simplify the present example, these curves are assumed to be defined by TMPW Lj ðtÞ ¼ c1 þ ½c2 þ c3j expðc4j tÞ sinðc5j tÞ tanhðc6j tÞ, (6.7) TMPSLk ðtÞ ¼ c1 þ c2 tanh½c62 ð1 þ c7k Þt,

(6.8)

where the assumed functional forms mimic results observed in actual analyses. The nature of the constants (i.e., the c’s)

in Eqs. (6.7) and (6.8) is indicated in Table 1. Further, the density functions fWLj(TWL) and fSLk(TSL) are defined by pffiffiffiffiffiffi fW Lj ðT WL Þ ¼ ð1=c9 2pÞ exp½ðT WL  c8 Þ2 =2c29 , (6.9) pffiffiffiffiffiffi fSLk ðT SL Þ ¼ ð1=c11 2pÞ exp½ðT SL  c10 Þ2 =2c211 .

(6.10)

Again, the nature of the constants (i.e., the c’s) in Eqs. (6.9) and (6.10) is indicated in Table 1. Further, the two WL failure temperature distributions are assumed to be independent (i.e., although the two WLs have the same distributional form for failure temperature, the failure temperatures for the two WLs are independent). A similar assumption is made for the SL failure temperature distributions. The 16 variables used to characterize the system defined by Eqs. (6.6)–(6.10) are treated as being uncertain (Table 1). Each variable has an uncertainty range [a, b] as indicated in Table 1. As this example is for illustration of ideas, it is assumed for simplicity that the uncertainty in each variable’s possible values is specified in the same manner by four independent experts (Table 2, Fig. 1). This is unlikely to be the case in a real analysis but providing different uncertainty specifications for each variable would complicate the presentation of this example while adding little to its illustrative value. The information indicated in Table 2 is encoded into a probability distribution for each variable in Table 1 for use in a probabilistic representation of the uncertainty in pF and into BPAs for use in an evidence theory representation of the uncertainty in pF (Fig. 2). 6.2. Construction of probability theory and evidence theory uncertainty representations The construction of a probability distribution from the information in Table 2 is considered first. Expert 1 only specifies an interval. The usual probabilistic encoding of this type of sparse information is to assume a uniform distribution over the specified interval [22, pp. 52–62]. That

Table 1 Uncertain variables and associated uncertainty ranges considered in example uncertainty analyses c1—Temperature (1C) of WLs and SLs before start of fire. Range: [30, 40 1C]. c2—Temperature increase (1C) above c1 at steady state. Range: [800, 1000 1C]. c31—Peak amplitude of temperature transient for WL 1. Range: [2600, 100 1C]. c32—Peak amplitude of temperature transient for WL 2. Range: [2600, 100 1C]. c41—Thermal heating time constant (min1) for WL 1. Range: [0.2, 0.4 min1]. c42—Thermal heating time constant (min1) for WL 2. Range: [0.2, 0.4 min1]. c51—Frequency response (min1) of temperature transient for WL 1. Range: [0.1, 0.2 min1]. c52—Frequency response (min1) of temperature transient for WL 2. Range: [0.1, 0.2 min1]. c61—Time constant (min1) determining the rate at which WL 1 reaches steady state temperature. Range: [0.02, 0.04 min1]. c62—Time constant (min1) determining the rate at which WL 2 reaches steady state temperature. Range: [0.02, 0.04 min1]. c71—Factor (dimensionless) used to account for more rapid heating in SL 1 than in the associated WL (i.e., WL 2). Range: [0.5, 0.8]. c72—Factor (dimensionless) used to account for more rapid heating in SL 2 than in the associated WL (i.e., WL 2). Range: [0.5, 0.8]. c8—Expected value (1C) of normal distribution for WL failure temperatures. Range: [255, 285 1C]. c9—Standard deviation (1C) of normal distribution for WL failure temperatures. Range: [4, 12 1C]. c10—Expected value (1C) of normal distribution for SL failure temperature. Range: [590, 610 1C]. c11—Standard deviation (1C) of normal distribution for SL failure temperature. Range: [15, 22 1C].

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1421

Table 2 Illustrative specification of uncertainty information used in example uncertainty analyses with probability theory and evidence theory for variables in Table 1 Expert 1: States appropriate value for variable is in the interval I11 ¼ ½a; b but provides no information on uncertainty structure within [a,b] Expert 2: Divides [a,b] into five nonoverlapping intervals of equal length (i.e., I21 ¼ ½a þ ðb  aÞði  1Þ=5; a þ ðb  aÞi=5Þ for i ¼ 1; 2; 3; 4 and I25 ¼ ½a þ ðb  aÞði  1Þ=5; a þ ðb  aÞi=5 for i ¼ 5) and states that the appropriate value for the variable is equally likely to be in each of these intervals Expert 3: Divides [a,b] into following five nonoverlapping intervals: I31 ¼ ½a; a þ ðb  aÞ=10, I32 ¼ ½a þ ðb  aÞ=10; a þ ðb  aÞ=10Þ, I33 ¼ ½a þ 4ðb  aÞ=10; a þ 6ðb  aÞ=10Þ, I34 ¼ ½a þ 6ðb  aÞ=10; a þ 9ðb  aÞ=10Þ, I35 ¼ ½a þ 9ðb  aÞ=10; b. States that the probability (i.e., likelihood) that the appropriate value for the variable is contained in each of these intervals is 0.05, 0.2, 0.5, 0.2 and 0.05, respectively Expert 4: Divides [a,b] into following five nested intervals: I41 ¼ ½a þ 4ðb  aÞ=10; a þ 6ðb  aÞ=10Þ, I42 ¼ ½a þ 3ðb  aÞ=10; a þ 7ðb  aÞ=10Þ, I43 ¼ ½a þ 2ðb  aÞ=10; a þ 8ðb  aÞ=10Þ, I44 ¼ ½a þ ðb  aÞ=10; a þ 9ðb  aÞ=10Þ, I45 ¼ ½a; b. States that amount of probability (i.e., likelihood) that can be assigned to the proposition that a given interval contains the appropriate value to use for the variable is 0.2 See Fig. 2 for a graphical representation of the indicated uncertainty specifications.

a BPA assignment for the variable. This specification can be converted to a probability distribution in a manner consistent in spirit with the handling of the information supplied by Experts 1, 2 and 3 by assuming a uniform distribution over each of the specified intervals and then weighting each of these distributions by the BPA assigned to the corresponding interval. The outcome of this process for the information supplied by Expert 4 is the density function d 4 ðvÞ ¼

4 X

d4j ðvÞ prob4 ðI4j Þ=LðI4j Þ,

(6.13)

j¼1

Fig. 2. Graphical illustration of uncertainty information in Table 2 with variable range [a,b] normalized to [0,1] for notational convenience.

is, Expert 1 is assumed to have specified a probability distribution with a density function given by  1=ðb  aÞ if apvpb; d 1 ðvÞ ¼ (6.11) 0 otherwise:

where d4j(v) ¼ 1 if v 2 I4j and 0 otherwise, prob4 ðI4j Þ denotes the amount of probability that can be assigned to I4j but not to any particular subset of I4j , and LðI4j Þ is the same as in Eq. (6.12). The distributions obtained from the individual experts can now be combined to obtain a single distribution that characterizes the uncertainty in the variable under consideration. Such ‘‘aggregation’’ of information from multiple sources is a much studied topic [50–54]. Here, the widely used approach of assigning equal weight (i.e., credibility) to each expert to produce a single distribution is used. Specifically, the resultant density function from this approach is given by dðvÞ ¼

nE X

d i ðvÞ=nE;

(6.14)

i¼1

Experts 2 and 3 have in essence specified quantiles on CDFs. Again, in consistency with common procedure, uniform distributions are assumed for the variable between these quantiles. As a result, the density functions associated with Experts 2 and 3 have the form  probi ðIij Þ=LðIij Þ if v 2 Iij ; d i ðvÞ ¼ (6.12) 0 otherwise for i ¼ 2; 3, where probi ðIij Þ denotes the probability for interval Iij specified by Expert i and LðIij Þ denotes the length of interval Iij . Expert 4 specifies what is, in essence,

where nE ¼ 4 is the number of experts. The described approach results in each variable in Table 1 having an uncertainty distribution of the form shown in Fig. 3, with only the values of a and b and associated scaling between a and b changing from variable to variable. The construction of BPAs from the information in Table 2 is now considered. The experts specify intervals and associated probabilities. These probabilities can be interpreted as BPAs for the corresponding intervals (i.e., I11 for Expert 1, and Iij , j ¼ 1; 2; . . . ; 5, for Expert i, i ¼ 2; 3; 4). Specifically, the BPA mi associated with

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1422

Table 3 Basic probability assignments (BPAs) for a variable on the interval [a,b] derived from the information in Table 2 mðUÞ

¼ 3=10 if U ¼ I1 ¼ ½a; b ¼ 1=20 if U ¼ I2 ¼ ½a; a þ ðb  aÞ=5Þ ¼ 1=20 if U ¼ I3 ¼ ½a þ ðb  aÞ=5; a þ 2ðb  aÞ=5Þ ¼ 9=40 if U ¼ I4 ¼ ½a þ 2ðb  aÞ=5; a þ 3ðb  aÞ=5Þ ¼ 1=20 if U ¼ I5 ¼ ½a þ 3ðb  aÞ=5; a þ 4ðb  aÞ=5Þ ¼ 1=20 if U ¼ I6 ¼ ½a; a þ 4ðb  aÞ=5; b ¼ 1=80 if U ¼ I7 ¼ ½a; a þ ðb  aÞ=10Þ ¼ 1=20 if U ¼ I8 ¼ ½a þ ðb  aÞ=10; a þ 4ðb  aÞ=10Þ ¼ 1=20 if U ¼ I9 ¼ ½a þ 6ðb  aÞ=10; a þ 9ðb  aÞ=10Þ ¼ 1=80 if U ¼ I10 ¼ ½a þ 9ðb  aÞ=10; b ¼ 1=20 if U ¼ I11 ¼ ½a þ 3ðb  aÞ=10; a þ 7ðb  aÞ=10Þ ¼ 1=20 if U ¼ I12 ¼ ½a þ 2ðb  aÞ=10; a þ 8ðb  aÞ=10Þ ¼ 1=20 if U ¼ I13 ¼ ½a þ ðb  aÞ=10; a þ 9ðb  aÞ=10Þ ¼ 0 otherwise 1:0

Fig. 3. Distribution of a variable over the interval [a,b] derived from the information in Table 2.

Expert i is given by  probi ðUÞ if U 2 Mi ; mi ðUÞ ¼ 0 otherwise

(6.15)

for an arbitrary set U of points from [a,b], where M1 ¼ fI11 g and Mi ¼ fIij ; j ¼ 1; 2; . . . ; 5g for i ¼ 2; 3; 4. Analogously to the weighting process implemented in Eq. (6.14) for density functions, the BPAs from the individual experts can be equally weighted to produce a final BPA m. In particular, this final BPA is given by mðUÞ ¼

nE X

mi ðUÞ=nE,

(6.16)

i¼1

where nE ¼ 4 is the number of experts and U is an arbitrary subset of points from [a,b]. The preceding procedure produces 13 sets with nonzero BPAs (Table 3). The indicated approach results in each variable in Table 1 having a BPA structure of the form indicated in Fig. 4. 6.3. Exploratory sensitivity analysis As discussed in Section 3, an exploratory sensitivity analysis plays an important role in helping to guide any study that involves uncertain inputs. Sensitivity analysis with the uncertainty representations developed in Section 6.2 are now illustrated. Specifically, a random sample of size 200 is generated from the 16 uncertain variables listed in Table 1 in consistency with the distributions indicated in Eq. (6.14). The sample elements are vectors of the form   xj ¼ x1j ; x2j ; . . . ; x16j (6.17) for j ¼ 1; 2; . . . ; 200, where each component xij, i ¼ 1; 2; . . . ; 16, of xj corresponds to one of the 16 variables in Table 1. Each sample element results in four time– temperature curves (i.e., one curve for each of the two WLs

Fig. 4. Graphical illustration of the 13 sets in Table 3 assigned nonzero BPAs with variable range [a,b] normalized to [0,1] for notational convenience.

and two SLs) as defined in Eqs. (6.7)–(6.8) and a corresponding failure probability pF as defined in Eq. (6.6) (Fig. 5). The 200 failure probabilities that result from the 200 sample elements indicated in Eq. (6.17) can be displayed as a CCDF (Fig. 6), which provides a representation of the epistemic uncertainty associated with the probability pF that both SLs fail before either WL fails. A natural adjunct to the uncertainty analysis results presented in Fig. 6 is a sensitivity analysis to determine the importance of the individual variables in Table 1 in determining the uncertainty in pF. In particular, the sampling-based uncertainty analysis in use has generated

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

Table 4 Stepwise regression analysis with log-transformed and rank-transformed data for probability pF that both SLs fail before either WL fails

1000 SL2

Log-transformed data

Rank-transformed data

SL1

800 Temperature (°C)

1423

a

WL2

600 WL1

400

200 pF = 9.8x10-6

b

c

2d

Step

Variable

SRC

R

1 2 3 4 5 6 7 8 9

c11 c71 c8 c72 c9 c10 c62 c42 c2

0.77 0.31 0.24 0.18 0.18 0.15 0.11 0.08 0.07

0.59 0.69 0.75 0.78 0.81 0.84 0.85 0.85 0.86

Variableb

SRRCe

R2d

c11 c71 c8 c10 c9 c72 c2 c42 c1

0.84 0.34 0.26 0.16 0.12 0.12 0.10 0.09 0.06

0.67 0.79 0.86 0.89 0.90 0.91 0.92 0.93 0.93

a

0

0

20

40 60 Time (min)

80

100

Fig. 5. Example WL/SL temperature curves and associated failure probability for sample element x144.

100

Prob(>pF)

10-1

10-2

10-3 10-10

10-9

10-8

10-7

10-6

10-5

10-4

pF, Failure Probability Fig. 6. Use of CCDF generated with a random sample of size 200 to display the epistemic uncertainty in the probability pF that both SLs fail before either WL fails.

a mapping ½xj ; pF j ;

j ¼ 1; 2; . . . ; 200,

(6.18)

from the uncertain analysis inputs contained in xj to the corresponding failure probability pFj. A variety of sensitivity analysis procedures exist that can be used in the exploration of this mapping [1,14,16–19,39]. As an example, stepwise regression procedures with log-transformed and rank-transformed data [13,40] are used to explore this mapping (Table 4). In the analysis with logtransformed data, the dependent variable is log(pF) rather than pF, and the independent variables retain their original (i.e., raw or untransformed) values.

Steps in stepwise regression analysis with significance levels of a ¼ 0:02 and 0.05 required of a variable for entry into and retention in a regression model, respectively. b Variables listed in order of selection in regression analysis. c Standardized regression coefficients (SRCs) in final regression model with log-transformed values for pF. d Cumulative R2 value with entry of each variable into regression model. e Standardized rank regression coefficients (SRRCs) in final regression model with rank-transformed values for all variables.

The regression results with log-transformed and ranktransformed data are similar. In particular, both regression approaches identify c11 (standard deviation of normal distribution for SL failure temperature), c71 (factor used to account for more rapid heating in SL 1 than in the associated WL), and c8 (expected value of normal distribution for WL failure temperature) as the most important variables with respect to the observed uncertainty in pF. The positive regression coefficients are consistent with patterns observed in the associated scatterplots (Fig. 7) and the known effects of these variables. In particular, (i) increasing c11 increases the number of low SL failure temperatures and thus increases pF, (ii) increasing c71 increases the temperature of SL 1 relative to WL 1 and thus increases pF, and (iii) increasing c8 increases the number of high WL failure temperatures and thus increases pF. As is often the case due to the linearizing effects of the rank transformation, the analyses with rank-transformed data result in somewhat higher R2 values than is the case for the analyses with raw data. For perspective, the analysis was also performed for three samples of size 200, three samples of size 1000 and three samples of size 10,000 (Fig. 8). Except for getting better resolution with respect to the likelihood of obtaining very large values for pF (i.e., close to 103), the results for the different sample sizes are quite similar. Thus, as is usually the case, the sample size needed to determine where most of the uncertainty is located is not particularly large. Of course, this changes if the goal of the analysis is to identify large but unlikely analysis outcomes. For example, if the goal of the analysis was to determine an epistemic (i.e., degree of belief or subjective) probability that the value for pF exceeds 106, then the same conclusion would

ARTICLE IN PRESS 1424

J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

be drawn from any of the nine CCDFs presented in Fig. 8. Specifically, and conditional on the assumptions of the hypothetical example under consideration, this probability is approximately 0.1. 6.4. Stepwise construction of CCBFs and CCPFs The representation of the epistemic uncertainty associated with pF in the context of evidence theory is now considered. This representation is based on the BPAs for the 16 elements of x defined in Eq. (6.16) and illustrated in Table 3 and Fig. 4. In concept, the CCPF and CCBF for pF can be constructed from the evidence space (X; X; mX ) defined in Section 6.2 with the Monte-Carlo procedure defined in conjunction with Eqs. (4.1) and (4.2). Unfortunately, the large number of sets contained in X (i.e., 1316ffi6.7  1017) makes direct use of this procedure with the evidence space (X; X; mX ) computationally impracticable. Specifically, it is not computationally possible to carry out an analysis with a sufficiently large sample to assure adequate coverage of all sets contained in X. Thus, some approach other than a direct application of the procedures in Eqs. (4.1) and (4.2) with (X; X; mX ) must be sought to obtain the CCPF and CCBF for pF. In particular, the stepwise construction procedure for CCBFs and CCPFs described in Section 4 provides a way to estimate the CCBF and CCPF for the probability pF. The approach described in Section 4 to the construction of CCBFs and CCPFs when the cardinality of X is large can be represented as an algorithm of the form shown below, where x ¼ ½x1 ; x2 ; . . . ; xn , (X; X; mX ) is an evidence space constructed from evidence spaces (Xi ; Xi ; mi ) for the individual elements xi of x, and f is a function that produces the analysis outcome y ¼ f ðxÞ under study: Step 0: Perform a sensitivity analysis to determine the most important variables x~ 1 ; x~ 2 ; . . . ; x~ n with respect to the uncertainty in y, where x~ 1 is the most important variable, x~ 2 is the next most important variable, and so on. Step 1: Estimate a CCBF CCBF1 (see Eq. (4.1)) and a CCBF CCPF1 (see Eq. (4.2)) for y on the basis of the evidence space (X; S1 ; mS1 ) obtained from the original evidence space for x~ 1 and degenerate evidence spaces for x~ 2 ; x~ 3 ; . . . ; x~ n in which the sample spaces are assigned BPAs of 1. Step 2: Estimate a CCBF CCBF2 and a CCPF CCPF2 for y on the basis of the evidence space (X; S2 ; mS2 ) obtained from the original evidence spaces for x~ 1 and x~ 2 and degenerate evidence spaces for x~ 3 ; x~ 4 ; . . . ; x~ n in which the. sample spaces are assigned BPAs of 1. .. Step s: Estimate a CCBF CCBFs and a CCPF CCPFs for y on the basis of the evidence space (X; Ss ; mSs ) obtained from the original evidence spaces for x~ 1 ; x~ 2 ; . . . ; x~ s and degenerate evidence spaces for x~ sþ1 ; x~ sþ2 ; . . . ; x~ n in which the sample spaces are assigned BPAs .. of 1. .

Termination: End process when no significant difference exists between CCBFs1 and CCPFs1 obtained at Step s1 and CCBFs and CCPFs obtained at Step s. The preceding approach is used in the construction of CCBFs and CCPFs for the failure probability pF. Step 0 corresponds to the sensitivity analysis discussed in Section 6.3 and summarized in Table 4, which identified (in the rank regression) c11 , c71 , c8 , c10 and c9 as the dominant variables with respect to the uncertainty in pF (i.e., c11 x~ 1 , c71 x~ 2 , c8 x~ 3 , c10 x~ 4 , c9 x~ 5 ). Steps 1–5 are then carried out with the successive inclusion of: (i) c11 to produce CCBF1 and CCPF1, (ii) c11 and c71 to produce CCBF2 and CCPF2, (iii) c11, c71 and c8 to produce CCBF3 and CCPF3, (iv) c11, c71, c8 and c10 to produce CCBF4 and CCPF4, and (v) c11, c71, c8, c10 and c9 to produce CCBF5 and CCPF5 (Fig. 9). As illustrated in Fig. 9, the CCBFs and CCPFs move closer together as the resolution increases in the evidence spaces for the elements of x. The CCPFs, which are shown in greater detail in Fig. 10, are probably close to being converged with the five variables under consideration. The addition of several more variables (i.e., steps in the construction algorithm) may be needed to converge the CCBFs. The CCDF in Figs. 9 and 10 is the CCDF associated with sampling distribution used in the estimation of the CCBFs and CCPFs (i.e., the distribution defined by the density functions in Eq. (6.14)). As such, it is one of the many CCDFs for pF that are consistent with the evidence space for x under consideration (i.e., the evidence space defined by the BPAs for the individual variables defined in Eq. (6.16)). The evidence spaces for the individual elements of x are defined with 13 focal elements. Thus, as the number of steps in the construction algorithm for CCBFs and CCPFs increases, the number of focal elements in the evidence space (X; Ss ,mSs ) for x rapidly increases. For example, consideration of 5, 6 and 7 variables results in evidence spaces (X; Ss ,mSs ), s ¼ 5, 6, 7, for x with approximately 3.71  105, 4.83  106 and 6.27  107 focal elements, respectively. Adequate sampling of this many focal elements is not possible except for models that are trivially inexpensive to evaluate. A possible numerical solution in this situation is to replace the original evidence spaces for the elements of x with evidence spaces that have a similar structure but a smaller number of focal elements. For example, such a reduction can be carried out by (i) evaluating the CBF and CPF for a variable, (ii) determining intervals [ai,bi] of variable values associated with the CBF and CPF values of i/n for i ¼ 0; 1; . . . ; n, and then (iii) assigning a BPA of 1/(n+1) to each interval [ai,bi] for i ¼ 0; 1; . . . ; n. 6.5. Summary sensitivity analysis The summary sensitivity analysis discussed in Section 5 is now illustrated. For this illustration, five different

ARTICLE IN PRESS 0

-5

-5

-5

-5

-10

-10

-10

-10

-15

-15

-20

-20

-20

-25

-25

-25

-25

-20

0 C1

20

-30

40

800

850

0

900 C2

950

-30

1000

-30

-2500 -2000 -1500 -1000 -500 0 C31

0

0 -5

-10

-10

-10

-10

-15

-15

-20

-20

-25

-25

-30

0.20

0.25

0.30 C41

0.35

-30

0.40

0.20

0.25

0.30 C42

0.35

-15

-20

-25

-25

0

-5

-5

-5

-10

-10

-10

-10

-15

Log(pF)

-5

Log(pF)

0

Log(pF)

0

-15

0

-30

0.10 0.12 0.14 0.16 0.18 0.20 C51

-15

-20

-20

-20

-25

-25

-25

-25

0

-30

0.10 0.11 0.12 0.13 0.14 0.15 C61(/10)

0

-30

0.20 0.21 0.22 0.23 0.24 0.25 C62(/10)

0.30

0.35

0.40 C71

0.45

-30

0.50

0 -5

-10

-10

-10

Log(pF)

-5

-10

Log(pF)

-5

-15

-15

-20

-20

-20

-25

-25

-25

-25

285

295

305 C8

315

-30

4

6

8 C9

10

12

-30

560

565

570 C10

1.0

1.5

2.0

-15

-20

-30

0.5

C72

0

-5

-15

0.10 0.12 0.14 0.16 0.18 0.20 C52

-15

-20

-30

-2500 -2000 -1500 -1000 -500 0 C32

-15

-20

-30

0.40

Log(pF)

-5

Log(pF)

-5 Log(pF)

-5

Log(pF)

Log(pF)

-15

-20

0

Log(pF)

-15

Log(pF)

0

Log(pF)

0

-30

Log(pF)

1425

0

Log(pF)

Log(pF)

J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

575

580

-30

15

20

25 C11

30

35

Fig. 7. Scatterplots for 16 variables in Table 1 sampled in uncertainty analysis of the probability pF that both SLs fail before either WL.

assignments of distributions to the focal elements associated with individual variables are considered: (i) uniform, (ii) left quadratic, (iii) right quadratic, (iv) left triangular, and (v) right triangular (see Eq. (5.5) for the density functions associated with uniform, left quadratic and right quadratic assignments). Each assignment of distributions to the focal elements results in a different CCDF for pF (Fig. 11). As should be the case, these individual CCDFs fall between the CCBF and CCPF associated with the original evidence space (i.e., the space defined in conjunction with Tables 1–3 and Figs. 2–4). The results in Fig. 11 were generated with random samples of size 1  106 from the uncertain

variables and the associated distributions. Further, the CCBF and CCPF were generated with a sample of size 1  106 from the density functions that resulted from the assignment of uniform distributions to the focal elements and the consideration of nondegenerate evidence spaces only for the variables identified as important in Section 6.3 (i.e., c11, c71, c8, c10, c9). For each of the uncertain variables (i.e., the variables in Table 1) and each of the five distribution possibilities, variance decompositions si and siT of the form indicated in Eq. (5.2) were calculated for log (pF) as indicated in Table 5 with individual samples of size 1  106 and software written by the authors. Given that 16 independent

ARTICLE IN PRESS 1426

J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

Fig. 8. Estimation of epistemic uncertainty in pF with three replicated random samples of size 200 (a), 1000 (b), and 10,000 (c).

variables are under consideration, the evaluation of si and siT for these variables required a total of (2+16)(1  106) evaluation of pF. This large number of evaluations was possible because the integral that defines pF is computationally quick to evaluate. The results of the variance decompositions for the five dominant variables (i.e., c11, c71, c8, c10, c9; see Table 1 for definitions) are shown in the cobweb plot [44] in Fig. 12. The variance decompositions are similar for the five distribution choices, with the decomposition dominated by c11 for every choice. Thus, in this example, the sensitivity (i.e., variance decomposition) results are not particularly affected by the five distribution choices under consideration.

Although the uncertainty decompositions for log(pF) are similar for the five distribution choices, the uncertainty in log(pF) is not at all similar. In particular, the scaled results in the last three columns of Fig. 12 (i.e., results normalized to the largest values of mean, standard deviation (SD) and coefficient of variation (CV) for the three distribution choices illustrated in the figure, with max {log(pF)} in the numerator for the mean and with max {SD} and max {|CV|} in the denominator for standard deviation and coefficient of variation, respectively) indicate that the distributions for pF are very different. For example, the expected value for log(pF) is 7.72 for the right quadratic distributions and only 13.84 for the left quadratic distributions.

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

Fig. 9. Successive CCBFs (i.e., CCBFs, s ¼ 1; 2; . . . ; 5, with s ¼ 1; 2 off scale) and CCPFs (i.e., CCPFs, s ¼ 1; 2; . . . ; 5) estimated in application of algorithm described in Section 6.5 to develop an evidence theory representation of the epistemic uncertainty in the probability pF that both SLs fail before either WL fails.

Fig. 10. Successive CCPFs (i.e., CCPFs, s ¼ 1; 2; 3; 4; 5) appearing in Fig. 9 plotted with shortened intervals on the axes.

6.6. Additional example of summary sensitivity analysis This example considers the function f ðU; V Þ ¼ U þ V þ UV þ U 2 þ V 2 þ UVgðV Þ cos½3pðU  1:25Þ,

ð6:19Þ

1427

Fig. 11. Example CCDFs generated with different distribution assignments to the focal elements associated with the individual variables in Table 1: uniform (U), left quadratic (LQ), right quadratic (RQ), left triangular (LT), and right triangular (RT).

The variables U and V are defined on [1.0,1.5] and [0.0, 1.0], respectively (Fig. 13). Further, U and V are assumed to have associated evidence spaces with (i) focal elements of [1.0,1.25), [1.25,1.375) and [1.25,1.5] for U, (ii) focal elements of [0.0,0.5), [0.25,0.75) and [0.5,1.0] for V, and (iii) each focal element assigned a BPA of 1=3. As in the preceding section (Section 6.5), different distributions can be assigned to the points associated with individual focal elements (e.g., uniform, left quadratic, right quadratic; see Eq. (5.4)) without violating the properties of the specified evidence spaces. In turn, these different distribution choices lead to different point densities in the joint range for U and V (Fig. 14) and thus to different variance decompositions (Fig. 15). In the generation of Figs. 14 and 15, the same distribution type was assigned to each focal element of U and V; thus, for example, the specification of a left quadratic distribution for U indicates that all focal elements for U are assigned a left quadratic distribution. The calculation of the variance decompositions in Fig. 15 was performed with the SIMLAB program [55] with samples of size nS ¼ 104 . In this example, the dominant influence of V on the uncertainty in y is quite insensitive to the distributions selected for consideration (Fig. 15). However, this could change for extreme choices of distributions (e.g., discrete distributions with a small number of points having nonzero probabilities).

with

7. Justification of assumptions for estimation of CCBFs and CCPFs

gðV Þ ¼ minfmaxf1=ðV  11=43Þ þ 1=ðV  22=43Þ þ 1=ðV  33=43Þ; 10g10g.

Two key assumptions underlie the algorithm presented in Section 4 and illustrated in Section 6.4 for the estimation

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1428

Table 5 Numerical procedure to determine variance decompositions s1, s2, s3, s1T, s2T and s3T for f(x1,x2, x3) defined in Eq. (5.2); procedure is the same but slightly more complicated notationally for a function of n variables 1. Generate random or LHS from x1, x2, x3: xj ¼ ½x1j ; x2j ; x3j ; j ¼ 1; 2; . . . ; nS: 2. Estimate mean and standard deviation for f: , , nS nS P P 2 2 2 ^ ^ V ðf Þ ¼ ½f ðxj Þ  E ðf Þ nS ¼ f ðxj Þ nS  E^ ðf Þ: j¼1

j¼1

Requires nS function evaluations. 3. Generate second random or LHS by randomly permuting, without replacement, the individual variable values from the sample generated in Step 1: rj ¼ ½r1j ; r2j ; r3j ; j ¼ 1; 2; . . . ; nS 4. Construct three reorderings of the sample from Step 3: aj ¼ ½a1j ; a2j ; a3j ; j ¼ 1; 2; . . . ; nS; such that a1j ¼ x1j bj ¼ ½b1j ; b2j ; b3j ; j ¼ 1; 2; . . . ; nS; such that b2j ¼ x2j cj ¼ ½c1j ; c2j ; c3j ; j ¼ 1; 2; . . . ; nS; such that c3j ¼ x3j 5. Estimate s1, s2, s3: " #, nS P 2 s1 ffi f ðxj Þf ðaj Þ=nS  E^ ðf Þ V^ ð f Þ " s2 ffi " s3 ffi

j¼1 nS P

. 2 f ðxj Þf ðbj Þ nS  E^ ðf Þ

j¼1 nS P

. 2 f ðxj Þf ðcj Þ nS  E^ ðf Þ

#, V^ ðf Þ

#, V^ ðf Þ:

j¼1

Requires nS additional function evaluations. 6. Generate three additional samples: dj ¼ ½d 1j ; x2j ; x3j ; j ¼ 1; 2; . . . ; nS; with d1j random or LHS from x1 and x2j and x3j same as in xj ej ¼ ½x1j ; e2j ; x3j ; j ¼ 1; 2; . . . ; nS; with e2j random or LHS from x2 and x1j and x3j same as in xj f j ¼ ½x1j ; x2j ; f 3j ; j ¼ 1; 2; . . . ; nS; with f3j random or LHS from x3 and x1j and x2j same as in xj. 7. Estimate s1T, s2T, s3T: " #!, . nS P 2 s1T ffi V^ ðf Þ  f ðxj Þf ðdj Þ nS  E^ ðf Þ V^ ðf Þ " s2T ffi

V^ ð f Þ  "

s3T ffi

V^ ðf Þ 

j¼1 nS P

. 2 f ðxj Þf ðej Þ nS  E^ ðf Þ

j¼1 nS P

. 2 f ðxj Þf ðf j Þ nS  E^ ðf Þ

#!, V^ ðf Þ

#!, V^ ðf Þ:

j¼1 2 Requires 3 nS additional function evaluations. The quantity E^ ðf Þ in the approximation to V^ ðf Þ cancels. For s1T this results in , , nS nS P P s1T ffi ½f 2 ðxj Þ  f ðxj Þf ðdj Þ ½nS V^ ðf Þ ¼ f ðxj Þ½f ðxj Þ  f ðdj Þ ½nSV^ ðf Þ: j¼1

j¼1

Similar forms results for s2T and s3T. 8. A total of 5 nS function evaluations are required to estimate s1, s2, s3, s1T, s2T and s3T for a function f(x1, x2, x3) of 3 variables. Procedure is the same for a function of n variables with a total of (2+n) nS function evaluations required to estimate s1 ; s2 ; . . . ; sn ; s1T ; s2T ; . . . ; snT:

of the CCBF and CCPF that results from the mapping of an evidence space (X; X,mX) into an evidence space (Y; Y, mY) by a function f defined on X. The first assumption is that, if a variable does not affect an analysis outcome of interest, then the calculated beliefs and plausibilities for this outcome are unaffected by the evidence theory structure assigned to the variable. The second assumption is that the uncertainty bounds represented by CCBFs and CCPFs narrow as the resolution in the evidence spaces assigned to input variables increases. These assumptions are now formalized and justified through a sequence of definitions and theorems. In particular, the first assumption is justified by Theorem 7.3, and the second assumption is justified by Theorem 7.2. Intuitively, an evidence space (V; V; mV ) refines an evidence space (U; U; mU ) provided: (i) U and V are the same, (ii) the elements of V are obtained by subdividing (i.e., refining) the elements of U, (iii) the BPA for each

element E of U is partitioned over the sets into which E is subdivided, and (iv) the BPA for each element of V is the sum of the values assigned to this set in the partitioning of the elements of U. The preceding summation is necessary when an element of V appears in the partitioning of two or more elements of U. The following definition provides a formal statement of the intuitive idea that (V; V; mV ) refines (U; U; mU ). Definition 7.1. An evidence space (V; V; mV ) is a refinement of an evidence space (U; U; mU ) provided: (i) U ¼ V, (ii) the sets U and V are related by U ¼ fUi : i ¼ 1; 2; . . . ; mg,

Ui ¼

nðiÞ [ j¼1

Vij ;

i ¼ 1; 2; . . . ; m;

(7.1)

(7.2)

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434 max=-7.72

max=5.39

1429 max=0.41

1.00

S_C11 = contribution of C11 to variance of log (pF)

0.80

ST_C11 = contribution of C11 and interaction of C11 with other variables to variance of log(pF) similarly for other variables

Value of index

Scaled mean= mean value (scaled between 0 and 1)

Uniform left quad. right quad.

0.60 Scaled stdev= standard deviation (scaled between 0 and 1) Scaled CV = coefficient of variation (scaled between 0 and 1)

0.40

0.20

0.00 S_C11 ST_C11 S_C71 ST_C71 S_C8

ST_C8 S_C10 ST_C10 S_C9 Type of index

ST_C9 Scaled Scaled Scaled Mean SD CV

Fig. 12. Variance for decompositions for uniform, left quadratic and right quadratic distributions for the five dominant variables (i.e., c11, c71, c8, c10, c9; see Table 1 for definitions) with respect to log (pF); the decompositions for left and right triangular distributions are essentially the same as those for left and right quadratic distributions and are not shown to reduce clutter in the figure.

n X

f ik ¼ 1;

i ¼ 1; 2; . . . ; m,

(7.5)

k¼1

and 8 m < P f m ðU Þ i ik U mV ðEÞ ¼ i¼1 : 0

if E ¼ Vk 2 V

(7.6)

otherwise

for each subset E of U. Theorem 7.1. Suppose the evidence space (V; V; mV ) refines the evidence space (U; U; mU ) and E is a subset of U. Then, BelU ðEÞpBelV ðEÞpPlV ðEÞpPlU ðEÞ,

Fig. 13. Function f(U,V) defined in Eq. (6.19) (Fig. 11, Ref. [24]).

(7.7)

where the subscripts U and V designate beliefs and plausibilities defined with respect to (U; U; mU ) and (V; V; mV ), respectively.

V ¼ fVij : j ¼ 1; 2; . . . ; nðiÞ; i ¼ 1; 2; . . . ; mg ¼ fVk : k ¼ 1; 2; . . . ; ng,

ð7:3Þ

Proof. The inequality

and (iii) there exists a nonnegative m  n matrix F ¼ ½f ik  such that

BelV ðEÞpPlV ðEÞ

f ik ¼ 0

is a fundamental property of belief and plausibility and follows immediately from their definitions (see Eq. (2.1)).

if Vk is not a subset of Ui ,

(7.4)

(7.8)

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1430

Fig. 14. Illustration of point densities for function f(U,V) in Eq. (6.19) with the focal elements for U and V assigned the following distributions (see Eq. (5.4)): (i) uniform (left frame); (ii) left quadratic (middle frame); and (iii) right quadratic (right frame). max=5.47

max=5.48

max=1.079

1.00

quad_ll quad_rr

0.80

quad_rl

Value of index

quad_lr tri_ll

0.60

tri_rr

0.40

S_U= contribution of U to variance of f(U,V)

tri_rl

ST_U = contribution of U and interaction of U with V to variance of f(U,V)

tri_lr tri_cc

similarly for variable V

uniform

Scaled mean =mean value (scaled between 0 and 1)

0.20

normal wide

Scaled stdev = standard deviation (scaled between 0 and 1)

normal narrow

Scaled CV = coefficient ofvariation (scaled between 0 and 1)

0.00 S_U

ST_U

S_V

ST_V

Scaled mean

Scaled stdev

Scaled CV

Type of index Fig. 15. Variance decompositions for the function f(U,V) in Eq. (6.19) for 12 different distribution assignments to the focal elements associated with U and V, where on the key to the right: (i) quad, tri, uniform and normal designate the assignment of quadratic, triangular, uniform and normal distributions to the individual focal elements; (ii) lr designates the assignment of left distributions to the focal elements for U and right distributions to the focal elements for V and ll, rr, rl and cc correspond to analogous designations with cc indicating modes at the centers of focal elements; and (iii) wide and narrow in conjunction with normal distributions designating normal distributions centered at the midpoint m ¼ ða þ bÞ=2 of an interval [a,b] corresponding to a focal element with standard deviations sW and sN given by sW ¼ ðb  aÞ=4 and sN ¼ ðb  aÞ=8, respectively, and the resultant density functions normalized so that their integrals over [a,b] equal one.

,

Thus, the middle inequality in Eq. (7.7) is valid. Further, X BelU ðEÞ ¼ mU ðUi Þ; with I ¼ fi : Ui
Eg

with cik ¼ f ik mU ðUi Þ

l¼1

i2I

¼

n XX i2I k¼1

cik mV ðVk Þ,

m X

¼

n X k¼1

mV ðVk Þ

X i2I

cik

f lk mU ðVl Þ

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

X

¼

mV ðVk Þ

X

cik ;

i2I

k2K

with K ¼ fk : Vk
Eg; p

X

mV ðVk Þ;

since

X

X

cik ¼ 0 for keK

i2I

cik p1

¼ BelV ðEÞ.

ð7:9Þ

Thus, the left inequality in Eq. (7.7) is established. Finally, X PlV ðEÞ ¼ mV ðVk Þ; with K ¼ fk : Vk \ Eafg k2K m X X

f ik mU ðUi Þ

k2K i¼1 m X

X

i¼1

k2K

mU ðUi Þ

¼

X

¼

mU ðUi Þ

i2I

X

f ik f ik ;

k2K

with I ¼ fi : Ui \ Eafg; p

X

mU ðUi Þ;

since

i2I

X

X

f ik ¼ 0 for ieI

k2K

f ik p1

k2K

¼ PlU ðEÞ.

ð7:10Þ

Thus, the right inequality in Eq. (7.7) is valid, which completes the proof of Theorem 7.1. & Theorem 7.2. Suppose the evidence space (V; V; mV ) refines the evidence space (U; U; mU ), f is a function defined on U with range Y (i.e., f maps U onto Y), and (Y; YUY ; mUY ) and (Y; YVY ; mVY ) are the evidence spaces that derive from (U; U; mU ) and (V; V; mV ) and the mapping from U to Y defined by f, and E is a subset of Y. Then, BelUY ðEÞpBelVY ðEÞpPlVY ðEÞpPlUY ðEÞ,

(7.11)

where the subscripts UY and VY designate beliefs and plausibilities defined with respect to (Y; YUY ; mUY ) and (Y; YVY ; mVY ), respectively. Proof. This result follows immediately from Theorem 7.1 in the following manner: BelUY ðEÞ ¼ BelU ½f 1 ðEÞ pBelV ½f pPlV ½f

~ i ,m ~ i ) is an a set satisfying faI
f1; 2; . . . ; ng, and (Xi ,X alternative evidence space for xi for i 2 I. Then, the evidence space (S; S; mS ) for x constructed from ~ i ,m ~ i ) for i 2 I is called a (Xi ; Xi ; mi ) for i 2 Ic and (Xi ; X substitution evidence space for x.

i2I

k2K

¼

1431

1

1

ðEÞ

ðEÞ

pPlU ½f 1 ðEÞ ¼ PlUY ðEÞ,

ð7:12Þ

Definition 7.3. Suppose xi 2 Xi for i ¼ 1; 2; . . . ; n, x ¼ ½x1 ; x2 ; . . . ; xn , f is a function defined on X ¼ X1  X2      Xn , and I is a set satisfying faI
f1; 2; . . . ; ng. Then, the following two statements are equivalent: (i) the variables xi, i 2 I, are nonaffecting with respect to f, and (ii) if x 2 X, x~ 2 X, and xi ¼ x~ i for ~ i 2 Ic , then f ðxÞ ¼ f ðxÞ. Theorem 7.3. Suppose xi are variables with evidence spaces (Xi ; Xi ; mi ) for i ¼ 1; 2; . . . ; n, (X; X; mX ) is the corresponding evidence space for x ¼ ½x1 ; x2 ; . . . ; xn , I is a set ~ i ,m ~ i ) is an alternative satisfying faI
f1; 2; . . . ; ng, (Xi ,X evidence space for xi for i 2 I, (S; S; mS ) is the substitution evidence space for x constructed from (Xi ; Xi ; mi ) for i 2 Ic ~ i ,m ~ i ) for i 2 I, f is a function defined on X, and (Xi ,X (Y; YXY ; mXY ) and (Y; YSY ; mSY ) are the evidence spaces that derive from (X; X; mX ) and (X; S; mS ) and the mapping from X onto Y defined by f, the xi for i 2 I are nonaffecting with respect to f, and E is a subset of Y. Then, BelXY ðEÞ ¼ BelSY ðEÞpPlSY ðEÞ ¼ PlXY ðEÞ,

(7.13)

where the subscripts XY and SY designate beliefs and plausibilities defined with respect to (Y; YXY ; mXY ) and (Y; YSY ; mSY ), respectively.

Proof. For notational convenience, assume that the elements of x are reordered so that Ic ¼ f1; 2; . . . ; mg

and

I ¼ fm þ 1; m þ 1; . . . ; ng.

(7.14)

Thus, the first m elements of x are assumed to potentially affect the value of y ¼ f(x) and the remaining elements are assumed to be nonaffecting. Further, the following notation is introduced for use later in the proof: nðiÞ ¼ cardinality of Xi for i ¼ 1; 2; . . . ; n,

(7.15)

Xði; jÞ ¼ element j of Xi for j ¼ 1; 2; . . . ; nðiÞ;

(7.16)

~ i for i ¼ m þ 1; m þ 2; . . . ; n, ~ ¼ cardinality of X nðiÞ

(7.17)

~ jÞ ¼ element j of Xi for j ¼ 1; 2; . . . ; nðiÞ, ~ Xði;

(7.18)

IA ¼ fa : a ¼ ½j 1 ; j 2 ; . . . ; j m ; 1pj i pnðiÞ for i ¼ 1; 2; . . . ; mg,

ð7:19Þ

where the three inequalities follow from Theorem 7.1. Thus, Theorem 7.2 is established. &

IB ¼ fb : b ¼ ½j mþ1 ; j mþ2 ; . . . ; j n ; 1pj i pnðiÞ

Definition 7.2. Suppose xi are variables with evidence spaces (Xi ; Xi ; mi ) for i ¼ 1; 2; . . . ; n, (X; X; mX ) is the corresponding evidence space for x ¼ ½x1 ; x2 ; . . . ; xn , I is

~ IC ¼ fc : c ¼ ½j mþ1 ; j mþ2 ; . . . ; j n ; 1pj i pnðiÞ

for i ¼ m þ 1; m þ 2; . . . ; ng,

for i ¼ m þ 1; m þ 2; . . . ; ng,

ð7:20Þ

ð7:21Þ

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1432

IX ¼ fn : n ¼ ½a; b; a 2 IA; b 2 IBg,

(7.22)

IES ¼

[

fn : n ¼ ½a; c; c 2 ICg

(7.36)

a2IAS

IS ¼ fn : n ¼ ½a; c; a 2 IA; c 2 ICg,

(7.23)

Aa ¼ Xð1; j 1 Þ  Xð2; j 2 Þ      Xðm; j m Þ for a ¼ ½j 1 ; j 2 ; . . . ; j m  2 IA,

ð7:24Þ

Bb ¼ Xðm þ 1; j mþ1 Þ  Xðm þ 2; j mþ2 Þ      Xðm; j n Þ for b ¼ ½j mþ1 ; j mþ2 ; . . . ; j n  2 IB,

ð7:25Þ

~ ~ ~ þ 1; j mþ1 Þ  Xðm þ 2; j mþ2 Þ      Xðm; jnÞ Cc ¼ Xðm for c ¼ ½j mþ1 ; j mþ2 ; . . . ; j n  2 IC, ð7:26Þ mA ðAa Þ ¼

m Y

mk ½Xðk; j k Þ

mB ðBb Þ ¼

n Y

ð7:27Þ

mC ðCc Þ ¼

n Y

mk ½Xðk; j k Þ

for ½a; b 2 IX, and mS ðAa  Cc Þ ¼ mA ðAa ÞmC ðCc Þ

(7.38)

for ½a; c 2 IS. Further, the equalities X mC ðBb Þ ¼ 1

(7.39)

and X

mC ðCc Þ ¼ 1

ð7:28Þ

~ ~ k ½Xðk; j k Þ m

½a;b2IEX

ð7:29Þ

The preceding notation facilitates a distinction between the elements of x that affect and do not affect the value of f(x). The equality BelXY ðEÞ ¼ BelSY ðEÞ is considered first. The following sets are defined involving f 1 ðEÞ: IEX ¼ fn : n ¼ ½a; b 2 IX; U ¼ Aa  Bb
f 1 ðEÞg, (7.30) IAX ¼ fa : n ¼ ½a; b 2 IEXg,

(7.31)

IES ¼ fn : n ¼ ½a; c 2 IS; V ¼ Aa  Cc
f 1 ðEÞg, (7.32) IAS ¼ fa : n ¼ ½a; c 2 IESg.

(7.33)

Further, the equality IAX ¼ IAS

¼

(7.34)

X

X

mX ðAa  Bb Þ

a2IAX b2IB

¼

X

mA ðAa Þ

a2IAX

¼

X

X

mB ðBb Þ

b2IB

mA ðAa Þ,

ð7:41Þ

a2IAX

where the second, third and fourth equalities follow from Eqs. (7.35), (7.37) and (7.39), respectively. Similarly, X BelSY ðEÞ ¼ mS ðAa  Cc Þ ½a;c2IES

¼

X

X

mS ðAa  Cc Þ

a2IAS c2IC

¼

X

mA ðAa Þ

a2IAS

¼

follows from the assumption that the variables associated with I (i.e., xmþ1 ; xmþ2 ; . . . ; xn ) are nonaffecting with respect to f. Specifically, if a 2 IAX and n ¼ ½a; b is an element of IEX, then f ðAa  Bb Þ
E; however, because the variables associated with I are nonaffecting, f ðAa  Cc Þ ¼ f ðAa  Bb Þ for any c 2 IC. Hence, f ðAa  Cc Þ
E, a 2 IAS, and so IAX
IAS. Similarly, IAS
IAX and so the equality in Eq. (7.34) is valid. The equalities [ IEX ¼ fn : n ¼ ½a; b; b 2 IBg, (7.35) a2IAX

(7.40)

follow from the assumption that the elements of x associated with I are nonaffecting with respect to f. The belief BelXY ðEÞ is given by X BelXY ðEÞ ¼ mX ðAa  Bb Þ

k¼mþ1

for c ¼ ½j mþ1 ; j mþ2 ; . . . ; j n  2 IC.

(7.37)

c2IC

k¼mþ1

for b ¼ ½j mþ1 ; j mþ2 ; . . . ; j n  2 IB,

mX ðAa  Bb Þ ¼ mA ðAa ÞmB ðBb Þ

b2IB

k¼1

for a ¼ ½j 1 ; j 2 ; . . . ; j m  2 IA,

also derive from the assumption that the variables associated with I are nonaffecting. From the underlying assumption of independence among the elements of x,

X

X

mC ðCc Þ

c2IC

mA ðAa Þ,

ð7:42Þ

a2IAS

where the second, third and fourth equalities follow from Eqs. (7.36), (7.38) and (7.40), respectively. The equality BelXY ðEÞ ¼ BelSY ðEÞ now follows from Eqs. (7.41) and (7.42) and the equality of IAX and IAS indicated in Eq. (7.34). The equality PlXY ðEÞ ¼ PlSY ðEÞ follows by an argument analogous to the one used to establish the equality BelXY ðEÞ ¼ BelSY ðEÞ. The only difference is the consideration of sets that intersect E rather than sets that are contained in E. Further, the inequality BelSY ðEÞpPlSY ðEÞ is a basic property of belief and plausibility. Thus, Theorem 7.3 is established. &

ARTICLE IN PRESS J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

1433

8. Summary Three applications of sampling-based sensitivity analysis in conjunction with evidence theory representations for epistemic uncertainty in model inputs have been described: (i) an initial exploratory analysis to assess model behavior and provide insights for additional analysis, (ii) a stepwise analysis showing the incremental effects of uncertain variables on CCBFs and CCPFs, and (iii) a summary analysis showing a spectrum of variance-based sensitivity analysis results that derive from probability spaces that are consistence with the evidence space under consideration. It is hoped that the ideas associated with these approaches will provide a start towards the development of effective sensitivity analysis procedures for use in conjunction with evidence theory representations for epistemic uncertainty. Acknowledgments Work performed for Sandia National Laboratories (SNL), which is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Security Administration under contract DE-AC04-94AL-85000. Review at SNL provided by Laura Swiler and Floyd Spencer. Editorial support provided by F. Puffer, K. Best and J. Ripple of Tech Reps, a division of Ktech Corporation. References [1] Saltelli A, Chan K, Scott EM. Sensitivity analysis. New York: Wiley; 2000. [2] Parry GW, Winter PW. Characterization and evaluation of uncertainty in probabilistic risk analysis. Nucl Saf 1981;22(1):28–42. [3] Apostolakis G. The concept of probability in safety assessments of technological systems. Science 1990;250(4986):1359–64. [4] Helton JC. Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. J Statist Comput Simul 1997;57(1–4):3–76. [5] Tura´nyi T. Sensitivity analysis of complex kinetic systems. Tools and applications. J Math Chem 1990;5(3):203–48. [6] Cacuci DG. Sensitivity and uncertainty analysis, vol. 1: theory. Boca Raton, FL: CRC Press; 2003. [7] Cukier RI, Levine HB, Shuler KE. Nonlinear sensitivity analysis of multiparameter model systems. J Comput Phys 1978;26(1):1–42. [8] Sobol’ IM. Sensitivity analysis for nonlinear mathematical models. Math Modeling Comput Exper 1993;1(4):407–14. [9] Rabitz H, Alis OF, Shorter J, Shim K. Efficient input–output model representations. Comput Phys Commun 1999;117(1–2):11–20. [10] Saltelli A, Tarantola S, Chan KP-S. A quantitative model-independent method for global sensitivity analysis of model output. Technometrics 1999;41(1):39–56. [11] McKay MD, Morrison JD, Upton SC. Evaluating prediction uncertainty in simulation models. Comput Phys Commun 1999; 117(1–2):44–51. [12] Iman RL. Uncertainty and sensitivity analysis for computer modeling applications. In: Cruse TA, editor. Reliability technology—1992, the winter annual meeting of the American Society of Mechanical Engineers, Anaheim, CA, vol. 28. New York: American Society of Mechanical Engineers, Aerospace Division; November 8–13, 1992. p. 153–68.

[13] Helton JC, Davis FJ. Sampling-based methods. In: Saltelli A, Chan K, Scott EM, editors. Sensitivity analysis. New York: Wiley; 2000. p. 101–53. [14] Kleijnen JPC, Helton JC. Statistical analyses of scatterplots to identify important factors in large-scale simulations, 1: review and comparison of techniques. Reliab Eng Syst Saf 1999;65(2): 147–85. [15] Frey HC, Patil SR. Identification and review of sensitivity analysis methods. Risk Anal 2002;22(3):553–78. [16] Helton JC, Davis FJ. Illustration of sampling-based methods for uncertainty and sensitivity analysis. Risk Anal 2002;22(3): 591–622. [17] Blower SM, Dowlatabadi H. Sensitivity and uncertainty analysis of complex models of disease transmission: an HIV model, as an example. Int Statist Rev 1994;62(2):229–43. [18] Hamby DM. A review of techniques for parameter sensitivity analysis of environmental models. Environ Monit Assess 1994;32(2): 135–54. [19] Helton JC. Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal. Reliab Eng Syst Saf 1993;42(2–3):327–67. [20] Klir GJ. Is there more to uncertainty than some probability theorists might have us believe? Int J Gen Syst 1989;15:347–78. [21] Klir GJ. On the alleged superiority of probability representation of uncertainty. IEEE Trans Fuzzy Syst 1994;2:27–31. [22] Howson C, Urbach P. Scientific reasoning: the Bayesian approach. 2nd ed. Chicago: Open Court; 1993. [23] Klir GJ, Wierman MJ. Uncertainty-based information. New York: Physica-Verlag; 1999. [24] Helton JC, Johnson JD, Oberkampf WL. An exploration of alternative approaches to the representation of uncertainty in model predictions. Reliab Eng Syst Saf 2004;85(1–3):39–71. [25] Kangas AS, Kangas J. Probability, possibility and evidence: approaches to consider risk and uncertainty in forestry decision analysis. For Policy Econ 2004;6:169–88. [26] Klir GJ. Generalized information theory: aims, results, and open problems. Reliab Eng Syst Saf 2004;85(1–3):21–38. [27] Dempster AP. Upper and lower probabilities induced by a multivalued mapping. Ann Math Statist 1967;38:325–39. [28] Shafer G. A mathematical theory of evidence. Princeton, NJ: Princeton University Press; 1976. [29] Wasserman L. Belief functions and likelihood. Technical report 420, Department of Statistics, Carnegie-Mellon University, Pittsburgh, PA, 1988. [30] Wasserman LA. Belief functions and statistical inference. Can J Statist 1990;18(3):183–96. [31] Guan JW, Bell DA. Evidence theory and its applications, vols. 1, 2. New York: North-Holland; 1991. [32] Halpern JY, Fagin R. Two views of belief: belief as generalized probability and belief as evidence. Artif Intell 1992;54:275–317. [33] Yager RR, Kacprzyk J, Fedrizzi M, editors. Advances in the Dempster-Shafer theory of evidence. New York: Wiley; 1994. [34] Feller W. 2nd ed. An introduction to probability theory and its applications, vol. 2. New York: Wiley; 1971. [35] Helton JC, Oberkampf WL, Johnson JD. Competing failure risk analysis using evidence theory. Risk Anal 2005;25(4):973–95. [36] Dempster AP. Upper and lower probability inferences based on a sample from a finite univariate population. Biometrika 1967;54(2–3): 515–28. [37] Dempster AP. A generalization of Bayesian inference. J Roy Statist Soc Ser B 1968;30:205–47. [38] McKay MD, Beckman RJ, Conover WJ. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 1979;21(2): 239–45. [39] Helton JC, Davis FJ. Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems. Reliab Eng Syst Saf 2003;81(1):23–69.

ARTICLE IN PRESS 1434

J.C. Helton et al. / Reliability Engineering and System Safety 91 (2006) 1414–1434

[40] Iman RL, Conover WJ. The use of the rank transform in regression. Technometrics 1979;21(4):499–509. [41] Hora SC, Helton JC. A distribution-free test for the relationship between model input and output when using Latin hypercube sampling. Reliab Eng Syst Saf 2003;79(3):333–9. [42] Iman RL, Conover WJ. Small sample sensitivity analysis techniques for computer models, with an application to risk assessment. Commun Statist Theory Methods 1980;A9(17):1749–842. [43] Beckman RJ, McKay MD. Monte-Carlo estimation under different distributions using the same simulation. Technometrics 1987;29(2): 153–60. [44] Cooke RM, van Noortwijk JM. Graphical methods. In: Saltelli A, Chan K, Scott EM, editors. Sensitivity analysis. New York: Wiley; 2000. p. 245–64. [45] Helton JC, Johnson JD, Oberkampf WL. Probability of loss of assured safety in temperature dependent systems with multiple weak and strong links. Reliab Eng Syst Saf 2006;91(3):320–48. [46] Crowder MJ. Classical competing risks. Boca Raton, FL: Chapman & Hall/CRC; 2001. [47] David HA, Moeschberger ML. The theory of competing risks. London: Griffin; 1978.

[48] Seal HL. Studies in the history of probability and statistics, XXXV. Multiple decrements or competing risks. Biometrika 1977;64:429–39. [49] Gail M. Review and critique of some models used in competing risk analysis. Biometrics 1975;31(1):209–22. [50] Ayyub BM. Elicitation of expert opinions for uncertainty and risks. Boca Raton, FL: CRC Press; 2001. [51] Ferson S, Kreinovich V, Ginzburg L, Myers DS, Sentz K. Constructing probability boxes and Dempster-Shafer structures. SAND2003-4015. Albuquerque, NM: Sandia National Laboratories; 2003. [52] Dubois D, Prade H. On the combination of evidence in various mathematical frameworks. In: Flamm J, Luisi T, editors. Reliability data collection and analysis. Boston, MA: Kluwer Academic Publishers; 1992. p. 213–41. [53] Sentz K, Ferson S. Combination of evidence in Dempster-Shafer theory. SAND2002-0835. Albuquerque, NM: Sandia National Laboratories; 2002. [54] Hora SC, Iman RL. Expert opinion in risk analysis: the NUREG1150 methodology. Nucl Sci Eng 1989;102(4):323–31. [55] Saltelli A, Tarantola S, Campolongo F, Ratto M. Sensitivity analysis in practice. New York: Wiley; 2004.