- Email: [email protected]
On the separation of aleatory and epistemic uncertainties in probabilistic assessments
Nuclear Engineering and Design 303 (2016) 68–74
Contents lists available at ScienceDirect
Nuclear Engineering and Design journal homepage: www.elsevier.com/locate/nucengdes
Technical Note
On the separation of aleatory and epistemic uncertainties in probabilistic assessments Mikko I. Jyrkama ∗ , Mahesh D. Pandey Department of Civil and Environmental Engineering, University of Waterloo, 200 University Avenue West, Waterloo, Ontario N2L 3G1, Canada
a r t i c l e
i n f o
Article history: Received 27 November 2015 Received in revised form 7 April 2016 Accepted 9 April 2016
a b s t r a c t The objective of this paper is to demonstrate how the process of uncertainty designation (aleatory vs. epistemic) and subsequent separation in a two-staged nested Monte Carlo simulation approach may lead to misinterpretation of the study results, particularly with respect to the confidence (i.e., lower and upper bounds) in the estimated probabilities. Using simple examples, a critical distinction is made between first- and second-order random variable problems, which has a direct bearing on the calculated outcomes. The results show how using the nested simulation approach in a first-order problem renders the uncertainty in the probability estimate to be conditional on the separated variables, and therefore leads to the estimation of sensitivity bounds, rather than confidence bounds on the probability. The probability itself is essentially a fixed number, and only subject to sampling error from the simulation. In contrast, the two-staged simulation approach is naturally applicable in the context of second-order problems, and allows the impact of both aleatory and epistemic uncertainties to be displayed effectively and separately on the final results, including the estimation of the true confidence bounds. © 2016 Elsevier B.V. All rights reserved.
1. Introduction The separation of aleatory and epistemic uncertainties has been a source of interest in many studies (e.g., Hoffman and Hammonds, 1994; RESS, 1996, 2004; Wu and Tsang, 2004; Der Kiureghian and Ditlevsen, 2007; Rao et al., 2007; Aven and Steen, 2010; U.S. NRC, 2012; Sankararaman and Mahadevan, 2013; Duan et al., 2015) with the primary goal of characterizing and displaying their contribution in the final outcomes. The motivation for the separation has generally been regulatory or safety related concerns (e.g., Helton and Breeding, 1993; Helton et al., 1999). Depending on the authors, aleatory uncertainty has often been referred to as simply variability, or random (or stochastic) heterogeneity in a population that cannot be reduced, while epistemic uncertainty has been used to refer to any kind of lack of information (i.e., ignorance) with respect to model parameters, variables, structure or form, which may be reduced by further measurement or study. To quantify the contribution of each uncertainty on the final results, the uncertainties are typically separated in a two-staged
∗ Corresponding author. Tel.: +1 519 888 4567x38222; fax: +1 519 888 4349. E-mail addresses: [email protected] (M.I. Jyrkama), [email protected] (M.D. Pandey). http://dx.doi.org/10.1016/j.nucengdes.2016.04.013 0029-5493/© 2016 Elsevier B.V. All rights reserved.
nested Monte Carlo simulation approach, where the epistemic parameters are sampled in the outer loop, while the aleatory variables are simulated as part of the inner loop (Wu and Tsang, 2004; U.S. NRC, 2012; Sankararaman and Mahadevan, 2013; Wang and Duan, 2013; Duan et al., 2015). The overall uncertainty in the estimated results (e.g., probability of failure, leak, rupture, etc.) should then be governed solely by the respective number of simulations within each loop. While there are many ways of representing epistemic uncertainty, including fuzzy sets and possibility theory (see RESS, 2004, for examples and detailed discussion), both uncertainties are described using a probabilistic representation in this paper. This approach is not only consistent with the other simulation studies listed above, but also convenient for problems (including the example problem presented here) where the uncertainty designation of a particular variable may be changed between the two types as required. The main objective of this study is to demonstrate, using simple examples, how the process of uncertainty separation may lead to misinterpretation of the results, particularly with respect to the confidence (i.e., lower and upper bounds) in the estimated probabilities. Much of the confusion is related not only to how the uncertainty in the variables and parameters is defined, but also the way in which the uncertainties are propagated to the final results.
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
69
The present discussion is especially relevant for probabilistic assessments, such as the xLPR probabilistic fracture mechanics project (Rudland, 2011; U.S. NRC, 2012), where the estimation of extremely low probabilities naturally raises the question of confidence associated with the results. The issue of confidence is directly related to the establishment of suitable acceptance criteria, which is one of the key challenges currently preventing the widespread application of probabilistic assessments, for example, as a standalone approach for structural integrity assessment (Duan et al., 2015). The paper is organized as follows. The simple example problem is presented in Section 2, including the key distinction between first- and second-order random variable problems. The central point of the paper is explored in Section 3, which demonstrates the consequences of applying the two-staged simulation approach in a first-order problem. Section 4 illustrates the natural application of the two-staged approach in a second-order problem, with the summary and conclusions presented in Section 5.
2.2. Simple model
2. Problem statement
Consider the following simple model for the time to leak for a pipe, for example, from stress corrosion cracking, as
All probabilistic assessments involve a computational model (e.g., complex mathematical equations, finite element code, fault tree, etc.) that defines the relationships and interactions between key parameters and variables (e.g., material properties and crack development characteristics in a fracture mechanics code used to model the timing of pipe rupture). Depending on the problem, the parameters and variables can be characterized as either deterministic (i.e., having known fixed values) or as uncertain (i.e., subject to aleatory or epistemic uncertainty). Regardless of the uncertainty designation (aleatory vs. epistemic) all uncertain variables (also referred to as random variables in probabilistic literature) are described using probability distributions that define the range and nature of variability associated with each variable. The uncertainty in the model output will then directly reflect the prescribed randomness or uncertainty in the model input parameters.
2.1. Model uncertainty Clearly, the results are also impacted by any uncertainties in the model itself (i.e., the ability and applicability of the mathematical equations to represent “reality”). This is a fundamental problem with all computational models, and is exceedingly difficult to quantify. The typical approach is to “validate” and “verify” the model against well-defined cases and other models (e.g., Wang and Duan, 2013), however, this does not completely eliminate or quantify the uncertainty in the model results. Probabilistic assessments also contain a second source of model uncertainty, attributed to the probabilistic models used to describe the uncertain input variables. The chosen or prescribed distributions (e.g., normal, log-normal, etc.) have a direct impact on the model results, however, the true or exact distribution types and their parameters can never be known with certainty. This issue is typically addressed through sensitivity analysis, which may involve not only the uncertain variables, but also an assortment of deterministic parameters. Because the main focus of this study is on the uncertain input variables (labelled as aleatory or epistemic) and how their relative contributions are propagated and expressed in the final results (i.e., model output), the issue of model uncertainty, while fundamentally important, will generally be ignored (i.e., all models, both computational and probabilistic, are assumed essentially to be “correct”).
f(a,b)
f(c,d)
Hyper Parameters (a, b, c, d)
Parameter (α) Parameter (β)
f(α,β) Distribuon Parameters
2nd Order 1st Order f(μ,σ)
(α, β, μ, σ)
Time to Iniaon (TI)
Model Parameters
TL = T I + W R
Growth Rate (R)
Fig. 1. First- vs. second-order random variable probabilistic model definition.
TL = TI +
W R
(1)
where TL is the time to leak, TI is the time to crack initiation, W is the wall thickness of the pipe, and R is the crack growth rate. The term W/R represents the time it takes for an initiated crack to grow through the pipe wall, resulting in a leak. The key question from a regulatory or fitness-for-service perspective is to determine When is the pipe going to leak (and ultimately rupture)? Assuming all the variables are known precisely (i.e., deterministic model), the time to leak can be assessed directly without any uncertainty (except for model uncertainty, of course). In reality, many of the parameters contributing to the leakage and failure of a pipe are unknown, and hence described as uncertain or random variables. Because of the complexity of the problem, the solution is also generally obtained through numerical simulations with their own challenges and limitations. This then raises the secondary, and perhaps more important question confronted in this paper What is the uncertainty or confidence in the estimated time (or probability) of leak? 2.3. Input variable uncertainty Describing any of the model parameters in Eq. (1) as uncertain variables implies that the model output, i.e. time to leak, also becomes an uncertain or random variable. While the computational model may be highly complex mathematically (which is not the case here), probabilistically this is referred to as a basic or first-order random variable model. As shown in Fig. 1, this means the addition of one or more (constant) probability distribution parameters (e.g., ˛, ˇ) to the problem by each of the uncertain input variables. In some cases, the new distribution parameters themselves may be subject to uncertainty, and hence described as random variables. As shown in Fig. 1, this adds a second layer (i.e., second-order) uncertainty to the problem, and introduces additional parameters (referred to as hyperparameters in the Bayesian context) to be estimated. Given the prescribed distributions and their parameters, the main challenge then is to determine the probability distribution of the model output (i.e., time to leak), which would not only reflect the combined influence of uncertainty of all the input variables, but also quantify the relative contribution of aleatory and epistemic uncertainties.
70
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
1 Cumulave Fracon
Mean Predicon 0.8
95 % Confidence Interval
0.6 0.4 0.2 0 0
100
200
300
400
500
600
700
800
Time to Leak (months) Fig. 2. Cumulative distribution of time to leak, TL , estimated using 1000 Monte Carlo simulation trials.
3. Basic (first-order) random variable model Let us assume the pipe wall thickness, W, in the simple model is constant and equal to 40 mm. Because of the uncertain nature of the cracking process, assume the time to initiation, TI , follows the Weibull distribution with shape parameter equal to 3 and scale parameter equal to 480 months (i.e., 40 years), and the growth rate, R, follows the Normal distribution with a mean of 5 mm/month and standard deviation of 1 mm/month. This means that the time to initiation is highly variable over time (i.e., the average time to initiation is approximately 429 months with a standard deviation of 156 months), while the crack growth is relatively fast (i.e., reaching through-wall in approximately 8 months on average in this case). For the time being, assume the random input variables are simply uncertain, with no specific designation as either epistemic or aleatory. 3.1. Uncertainty in time to leak Even for this relatively simple problem, no analytical expression can be derived for the distribution of time to leak. Fig. 2 shows the distribution of time to leak estimated from 1000 Monte Carlo simulation trials. The simulation was conducted by sampling each input distribution independently (i.e., time to initiation, TI , and growth rate, R), evaluating the model deterministically, and aggregating the results. The model output shown in Fig. 2 reflects both the range of variability in the time to leak (i.e., between zero and 800 months), as well as the sampling error from the simulation process (i.e., as reflected by the confidence interval). 3.1.1. Prediction interval The inherent or aleatory uncertainty in the time to leak can be characterized numerically using the concept of a prediction interval. For example, the 95% prediction interval for time to leak can be obtained directly from Fig. 2, with the lower limit (i.e., 0.025 percentile value) equal to approximately 150 months, and the upper limit (0.975 percentile value) estimated as 750 months. This means that for a given 40 mm thick pipe, the time to leak is estimated to be between 150 and 750 months, with 95% probability. It is also possible to estimate other quantities, such as the average or expected time to leak, which is equal to 437 months. 3.2. Uncertainty in probability of leak Given the uncertainty in time to leak (as reflected in Fig. 2), it is impossible to answer the first question of “when is the pipe going to leak?” (i.e., time to leak is a random variable). Therefore, the focus is on estimating the probability of leak at a particular time, i.e., P(TL ≤ t). In many probabilistic assessments, the output of the
computational model is not a probability in and of itself. Rather, it is some other key variable, such as time, or physical quantity such as stress, strength, temperature, pressure, wall thickness, etc., which must be compared to some critical (often code or regulatory specified) value (this is sometimes referred to as a performance function, or limit-state equation). The actual probability then arises from the fact that the key variable is not a fixed (deterministic) value, but an uncertain quantity with a distribution. In the present problem, the distribution of time to leak shown in Fig. 2, is equivalent to the distribution of probability of leak over time (i.e., Fig. 2 represents the proportion of the distribution (i.e., simulated samples) with a TL less than or equal to t, which is equivalent to probability of leak at time t). For example, the probability of leak at 720 months (i.e., 60 years) is equal to approximately 0.96, meaning that there is a high probability that a leak will take place before 60 years. The critical comment here is that, for any problem involving a first-order function of random variables with known distributions and their parameters (again, the function can be highly complex mathematically), only the estimated output variable, e.g., the time to leak, is by definition (i.e., inherently) random or uncertain, not the actual probability itself. The probability associated with the output variable is essentially a fixed number (while it varies over time in this problem, it is still constant at any specific time), and is only subject to uncertainty arising from the estimation (e.g., when using simulation methods). Again, this assumes that all random variables and their distributions are known precisely, and is only applicable in the context of the model (i.e., model uncertainty is ignored).
3.2.1. Confidence interval The sampling error (often referred to as epistemic uncertainty) due to simulation is straightforward to quantify using standard methods (Hammersley and Handscomb, 1975). The confidence interval in the estimated probability is obtained using the Beta distribution (or the equivalent Normal approximation) and is shown for the example problem by the dashed grey lines in Fig. 2. As shown in Fig. 2, the 95% confidence interval is fairly narrow, even for only 1000 simulations. Again, the issue of uncertainty or confidence (i.e., upper or lower bounds) regarding the probability of leak arises purely from the simulation process, not the basic random variable model itself.
3.3. Uncertainty separation The concept of uncertainty separation has been of interest in many studies, ranging from its proponents (e.g., Hoffman and Hammonds, 1994; Rao et al., 2007; Sankararaman and Mahadevan, 2013) to those questioning the entire premise (e.g., Der Kiureghian and Ditlevsen, 2007). In principle, identifying the relative contribution of lack of knowledge (i.e., epistemic) uncertainty and inherent (i.e., aleatory) variability on the final results appears highly attractive, especially from a risk management perspective. Because epistemic uncertainty can generally be reduced with the additional expenditure of resources, (risk-informed) decisions can readily be made with respect to the cost/benefit of such activities. However, the main challenge with the separation of uncertainty is agreeing on which uncertain input variables should be described as aleatory as opposed to epistemic. To facilitate this issue, many models, including the xLPR project (Rudland, 2011; EPRI, 2011; U.S. NRC, 2012), allow many of the model parameters and variables to be designated either as epistemic or aleatory (or also constant). While beneficial to the end user, the arbitrary separation of variables may lead to unexpected results in a first-order random variable problem, specifically with respect to the confidence intervals (i.e., upper
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
71
Fig. 4. Probability of leak over time including the conditional mean, median and 5% and 95% bounds, assuming the time to initiation, TI , is sampled in the outer (epistemic) loop with 100 trials, while the growth rate, R, is simulated in the inner (aleatory) loop with 100 trials. (For interpretation of the references to colour in text near the reference citation, the reader is referred to the web version of this article.)
and lower bounds) of the estimated probabilities, as shown in the following. 3.3.1. Two-staged nested Monte Carlo simulation The relative contribution of each type of uncertainty on the final results is typically obtained through a two-staged nested Monte Carlo simulation approach. As shown in Fig. 3, the standard two-staged nested simulation approach involves first sampling the epistemic variables in an outer loop, followed by simulation of the aleatory variables in an inner loop for a given set of random parameters (Wu and Tsang, 2004; Sankararaman and Mahadevan, 2013; Wang and Duan, 2013; Duan et al., 2015). The results from each loop are then aggregated and analyzed separately in order to display the relative contributions in the final results. 3.3.2. Separating variables To illustrate the impact of uncertainty separation in a first-order problem, let us assume (arbitrarily) that the time to initiation, TI , in our simple model is subject to epistemic uncertainty, while the growth rate, R, is assumed to be inherently random (i.e., an aleatory variable). According to the process outlined in Fig. 3, time to initiation is now sampled as part of the outer epistemic loop, while the growth rate is part of the inner aleatory loop. This means that for each random (but constant) single value of time to initiation, ti , from the outer loop, the time to leak is estimated in the inner loop as TL = TI +
W R
(2)
which is now a function of only a single random variable (i.e., the growth rate, R). Given the random sampling of R in the inner (aleatory) loop, Fig. 4 shows the estimated distribution of probability of leak over time, including the lower and upper bounds (with q corresponding to the percentile level), using 100 inner and outer simulation trials. As shown in Fig. 4, the mean estimate (red line) is very close to the
1 q = 0.95 Mean esmate q = 0.5 ~ Median q = 0.05 Exact value
0.8 Probability of Leak
Fig. 3. Two-staged nested Monte Carlo simulation approach involving separated aleatory and epistemic random variables.
exact result, and should be comparable to the result in Fig. 2. Clearly, however, the bounds between Figs. 2 and 4 are very different. Each grey line in Fig. 4, sometimes referred to as a “hair” (Duan et al., 2015), represents a single realization (out of 100) of the time to leak distribution for each outer (epistemic) loop in the simulation. The lines are nearly vertical, because the variability in the inner loop (i.e., in the growth rate, R) is much less than the variability in the time to initiation. That is, for any given value of time to initiation, ti , the crack growth is very rapid, reaching through-wall in approximately 8 months on average (as specified by the Normal distribution), while the value of time to initiation itself varies greatly between zero and 800 months (according to the prescribed Weibull distribution). Therefore, as opposed to confidence bounds on the probability of leak, the lower and upper bounds in Fig. 4 essentially represent the large uncertainty associated with the time to initiation, as described by the Weibull distribution. More specifically, the bounds are conditional on the random time to initiation as P(TL ≤ t | TI ), and therefore should be referred to as sensitivity bounds, rather than confidence bounds. For further illustration and comparison, Fig. 5 shows the results for the opposite case, where the growth rate, R, is now assumed to be epistemic and hence, sampled as part of the outer loop, while the time to initiation, TI , is assumed to be inherently random and
0.6 0.4 0.2 0 0
100
200
300 400 500 Time (months)
600
700
800
Fig. 5. Probability of leak over time including the conditional mean, median and 5% and 95% sensitivity bounds, assuming the growth rate, R, is sampled in the outer (epistemic) loop with 100 trials, while the time to initiation, TI , is simulated in the inner (aleatory) loop with 100 trials. (For interpretation of the references to colour in text near the reference citation, the reader is referred to the web version of this article.)
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
part of the inner aleatory loop. As before, both loops utilized 100 simulation trials. Similar to Fig. 4, the family of curves (i.e., grey lines or “hairs”) in Fig. 5 represents the probability of leak over time for each outer loop (i.e., having a constant growth rate, R, in this case). Rather than time to initiation, the variation in the curves (i.e., sensitivity bounds) now reflects the variability in the growth rate, R. As shown by Figs. 4 and 5, it is evident that the probability of leak (or the distribution of time to leak) is more sensitive to the uncertainty in the time to initiation, TI , as compared to the uncertainty in the crack growth rate, R. Regarding the overall uncertainty, the key point again is that the uncertainty in the probability of leak itself is not dependent on the uncertainty in the input variables. It only depends on the number of simulation trials, as illustrated by the mean estimates (red lines) in Figs. 4 and 5. While the sensitivity bounds would largely be unaffected, increasing the number of trials would naturally result in a better estimate for the (mean) value of probability of leak. The mean estimates should be the same in both cases (except for the sampling error) regardless of the order of separation or looping structure, because the conditional expectations are the same. Using the rule of total probability, the conditional expectation can be expressed as E[P(TL ≤ t|TI = ti )] = P(TL ≤ t) = E[P(TL ≤ t|R = ri )]
(3)
which is simply equal to the unconditional distribution of time to leak. In summary, the sensitivity bounds describe how the model output is impacted by the uncertainty or variation in the outer (epistemic) variables. This is useful for sensitivity analysis, because it not only identifies the most critical variables, but also quantifies their impact on the estimated results. For example, reducing the uncertainty in the distribution of time to initiation, TI , as opposed to the crack growth rate, R, would have a much larger impact on the uncertainty in the distribution of time to leak, and hence the estimated value of probability of leak (i.e., the answer to the first question). However, the uncertainty in the probability of leak would not be affected, because it would only depend on the simulation sample size (i.e., the answer to the second question). In the end, the critical question regarding the confidence on the estimated probability (i.e., the true lower and upper bounds) cannot be answered in this context. Applying the two-staged nested simulation approach in a first-order problem results in conditional sensitivity bounds, rather than the confidence bounds on the probability. In a first-order random variable problem, the true bounds can only be obtained without the separation, and applying the standard methods of estimation as discussed in Section 3.2.1. 4. Second-order random variable model The two-staged nested Monte Carlo simulation approach outlined in Fig. 3 is most applicable in the context of second-order random variable problems. As shown in Fig. 1, this type of problem arises when the parameters of the basic random variables are considered to be uncertain or random themselves. Consider again the simple model where the time to initiation, TI , was assumed to follow the Weibull distribution with shape parameter equal to 3 and scale parameter equal to 480 months (i.e., 40 years). This distribution (i.e., probabilistic model) describes the inherent variability or uncertainty associated with the time to initiation. In this context, the range of variability is defined precisely by the chosen distribution parameters of 3 and 480. The obvious question then is, how good are the chosen parameters in representing the true underlying uncertainty, or in other words, what is the uncertainty associated with the chosen parameters?
0.004 Probability Density
72
0.003 0.002 0.001 0 0
200 400 600 Time to Inaon (months)
800
Fig. 6. Family of distributions for the Weibull model of time to initiation, TI , assuming the shape parameter follows the Normal distribution with mean and standard deviation equal to 3 and 0.5, respectively (each grey line represents a single distribution of 100 realizations of time to initiation with a random shape parameter, while the red line represents the mean or average of the 100 realizations).
4.1. Second-order uncertainty Let us assume that rather than a constant value equal to 3, the shape parameter in the Weibull model of time to initiation is uncertain (e.g., because of lack of information, confidence, etc.) and follows a distribution. For simplicity, assume the shape parameter is described by the Normal distribution with a mean and standard deviation (i.e., hyperparameters) equal to 3 and 0.5, respectively, while the scale parameter remains constant and equal to 480 months. As opposed to a single distribution, this second layer of uncertainty then results in a “family” of distributions for the time to initiation, as illustrated in Fig. 6. The obvious outcome of the second-order uncertainty is that the actual probability of leak will now be uncertain or random as well. In addition to the uncertainty from estimation (e.g., sampling error from simulation), the uncertainty or confidence in the probability of leak will now further be influenced by the degree of uncertainty prescribed to the distribution parameters. The resulting impact of the additional level of uncertainty can readily be quantified using the two-staged nested simulation approach. 4.2. Two-staged simulation The premise for the nested Monte Carlo simulation approach is directly related to the second-order random variable problem definition, as described in Fig. 1. The first-order distributions describe the inherent variability in the computational model parameters, and hence are referred to as aleatory variables. Because the parameters of the distributions are commonly estimated from statistical data analysis, the second-order parameter uncertainty is typically referred to as epistemic (i.e., they depend on the sample size, and hence may be reduced with additional data). Therefore, the uncertain (epistemic) distribution parameters are first sampled in the outer loop, followed by simulation of the uncertain (aleatory) variables in the inner loop. Fig. 7 shows the simulated results for the simple problem assuming the Weibull shape parameter is also a random variable. Similar to before, the two-staged nested simulation approach leads to the generation of a family of distributions for the time to leak, TL , as shown in Fig. 7. Each grey line in Fig. 7 represents a single realization of the time to leak distribution, reflecting the characterized (epistemic) uncertainty in the underlying distribution parameters (i.e., only the shape parameter of the Weibull time to initiation distribution in this case). Because of the simulation approach, the uncertainty or confidence in the estimated probability of leak will depend
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
1
1 q = 0.95 Mean esmate q = 0.5 ~ Median q = 0.05
0.6
q = 0.95 Mean esmate q = 0.5 ~ Median q = 0.05
0.8 Probability of Leak
0.8 Probability of Leak
73
0.4 0.2
0.6 0.4 0.2
(b)
(a) 0
0 0
100
200
300 400 500 Time (months)
600
700
800
0
100
200
300 400 500 Time (months)
600
700
800
Fig. 7. Estimated probability of leak over time, including the mean, median and 5% and 95% confidence bounds, assuming the random Weibull shape parameter for the time to initiation, and given 100 outer (epistemic) trials and (a) 50, and (b) 104 inner (aleatory) simulations.
additionally on the number of simulations. Increasing the number of simulations in the inner and outer loops reduces the overall uncertainty, as shown in Fig. 7b. Naturally, some uncertainty in the probability of leak will always remain, regardless of the number of simulations, due to the (epistemic) uncertainty in the distribution parameters. The main benefit and reason for adopting the two-staged simulation approach in the second-order problem is the relatively simple estimation and transparent presentation of the confidence bounds for the probability estimate. As opposed to the sensitivity bounds in the first-order problem, the estimated bounds are the true confidence bounds in this case, and account for the sampling error from both the inner and outer loops explicitly, without the need for any complex analytical computations. In fact, no simple formulae exist for computing the confidence intervals in a general second-order random variable problem. Naturally, this means the estimated confidence bounds are “estimates” themselves, and hence depend on the number of simulation trials as shown in Fig. 7. In summary, the two-staged nested simulation approach allows the impact of both aleatory and epistemic uncertainties to be displayed effectively and separately on the final results in this context. The aleatory uncertainty or variability is reflected in the shape of the time to leak distribution (i.e., the “s” shape of the curves in Fig. 7), while the epistemic uncertainty (both from parameter uncertainty and sampling error) is characterized by the estimated true confidence bounds. 5. Summary and conclusions The computational models used in probabilistic assessments involve many uncertain or random variables. These variables may be characterized either as aleatory (intrinsically random) or epistemic (uncertain due to lack of knowledge). From a regulatory or fitness-for-service perspective, it is important to estimate not only the probability associated with the performance function (e.g., leak, rupture, failure, etc.), but also the uncertainty or confidence in the final results. Using simple examples, this paper explored the process of uncertainty separation using the two-staged nested Monte Carlo simulation approach in the context of first- and second-order random variable problems. The results of the study demonstrated how using the nested simulation approach in a first-order problem renders the uncertainty in the probability estimate to be conditional on the separated variables, and therefore leads to the estimation of sensitivity bounds, rather than the actual confidence bounds on the probability itself. While the value of the probability estimate does
depend on the uncertainty in the input parameters, it is essentially a fixed number and only subject to error from estimation (assuming all random variables and their distributions are known precisely, and model error is ignored). A second-order random variable problem implies that the parameters of the uncertain input variables are considered themselves to be uncertain. As a result, the probability estimate will also become uncertain or random, reflecting not only the inherent or aleatory uncertainty of the input variables, but also the epistemic uncertainty associated with the distribution parameters, as well as the sampling error from estimation. As opposed to the first-order problem, the two-staged simulation approach is naturally applicable in this context, by allowing the impact of both uncertainties to be displayed effectively and separately on the final results, including the estimation of the true confidence bounds on the estimated probability. Acknowledgements This work is part of the Industrial Research Chair program at the University of Waterloo funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) in partnership with the University Network of Excellence in Nuclear Engineering (UNENE). References Aven, T., Steen, R., 2010. The concept of ignorance in a risk assessment and risk management context. Reliab. Eng. Syst. Saf. 95, 1117–1122. Der Kiureghian, A., Ditlevsen, O., 2007. Aleatory or epistemic? Does it matter? In: Special Workshop on Risk Acceptance and Risk Communication, March 26–27. Stanford University. Duan, X., Wang, M., Kozluk, M., 2015. Acceptance criterion for probabilistic structural integrity assessment: prediction of the failure pressure of steam generator tubing with fretting flaws. Nucl. Eng. Des. 281, 154–162. Electric Power Research Institute (EPRI), 2011. Materials Reliability Program: Models and Inputs Developed for Use in the xLPR Pilot Study (MRP-302). EPRI Report 1022528, Palo Alto, CA. Hammersley, J.M., Handscomb, D.C., 1975. Monte Carlo Methods. Methuen, London. Helton, J.C., Breeding, R.J., 1993. Calculation of reactor accident safety goals. Reliab. Eng. Syst. Saf. 39, 129–158. Helton, J.C., Anderson, D.R., Jow, H.N., Marietta, M.G., Basabilvazo, G., 1999. Performance assessment in support of the 1996 compliance certification application for the waste isolation pilot plant. Risk Anal. 19 (5), 959–986. Hoffman, F.O., Hammonds, J.S., 1994. Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability. Risk Anal. 14, 707–712. Rao, K.D., Kushwaha, H.S., Verma, A.K., Srividya, A., 2007. Quantification of epistemic and aleatory uncertainties in level-1 probabilistic safety assessment studies. Reliab. Eng. Syst. Saf. 92, 947–956. RESS, 1996. Special issue on the “Treatment of Aleatory and Epistemic Uncertainty”. Reliab. Eng. Syst. Saf. 54 (2–3).
74
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
RESS, 2004. Special issue on the “Alternative Representations of Epistemic Uncertainty”. Reliab. Eng. Syst. Saf. 85 (1–3). Rudland, D., 2011. xLPR Version 1.0 Report, Technical Basis and Pilot Study Problem Results. NRC Letter Report, xLPR Computational Group, ML110660292. Sankararaman, S., Mahadevan, S., 2013. Separating the contributions of variability and parameter uncertainty in probability distributions. Reliab. Eng. Syst. Saf. 112, 187–199.
U.S. NRC, 2012. xLPR Pilot Study Report. NUREG-2110. U.S. NRC, Washington, DC. Wang, M., Duan, X., 2013. Benchmarking PRAISE-CANDU 1.0 with xLPR 1.0. In: Proceedings of the ASME 2013 Pressure Vessels & Piping Division Conference, July 14–16, 2013, Paris, France, PVP2013-98010. Wu, F.-C., Tsang, Y.-P., 2004. Second-order Monte Carlo uncertainty/variability analysis using correlated model parameters: application to salmonid embryo survival risk assessment. Ecol. Model. 177, 393–414.
Contents lists available at ScienceDirect
Nuclear Engineering and Design journal homepage: www.elsevier.com/locate/nucengdes
Technical Note
On the separation of aleatory and epistemic uncertainties in probabilistic assessments Mikko I. Jyrkama ∗ , Mahesh D. Pandey Department of Civil and Environmental Engineering, University of Waterloo, 200 University Avenue West, Waterloo, Ontario N2L 3G1, Canada
a r t i c l e
i n f o
Article history: Received 27 November 2015 Received in revised form 7 April 2016 Accepted 9 April 2016
a b s t r a c t The objective of this paper is to demonstrate how the process of uncertainty designation (aleatory vs. epistemic) and subsequent separation in a two-staged nested Monte Carlo simulation approach may lead to misinterpretation of the study results, particularly with respect to the confidence (i.e., lower and upper bounds) in the estimated probabilities. Using simple examples, a critical distinction is made between first- and second-order random variable problems, which has a direct bearing on the calculated outcomes. The results show how using the nested simulation approach in a first-order problem renders the uncertainty in the probability estimate to be conditional on the separated variables, and therefore leads to the estimation of sensitivity bounds, rather than confidence bounds on the probability. The probability itself is essentially a fixed number, and only subject to sampling error from the simulation. In contrast, the two-staged simulation approach is naturally applicable in the context of second-order problems, and allows the impact of both aleatory and epistemic uncertainties to be displayed effectively and separately on the final results, including the estimation of the true confidence bounds. © 2016 Elsevier B.V. All rights reserved.
1. Introduction The separation of aleatory and epistemic uncertainties has been a source of interest in many studies (e.g., Hoffman and Hammonds, 1994; RESS, 1996, 2004; Wu and Tsang, 2004; Der Kiureghian and Ditlevsen, 2007; Rao et al., 2007; Aven and Steen, 2010; U.S. NRC, 2012; Sankararaman and Mahadevan, 2013; Duan et al., 2015) with the primary goal of characterizing and displaying their contribution in the final outcomes. The motivation for the separation has generally been regulatory or safety related concerns (e.g., Helton and Breeding, 1993; Helton et al., 1999). Depending on the authors, aleatory uncertainty has often been referred to as simply variability, or random (or stochastic) heterogeneity in a population that cannot be reduced, while epistemic uncertainty has been used to refer to any kind of lack of information (i.e., ignorance) with respect to model parameters, variables, structure or form, which may be reduced by further measurement or study. To quantify the contribution of each uncertainty on the final results, the uncertainties are typically separated in a two-staged
∗ Corresponding author. Tel.: +1 519 888 4567x38222; fax: +1 519 888 4349. E-mail addresses: [email protected] (M.I. Jyrkama), [email protected] (M.D. Pandey). http://dx.doi.org/10.1016/j.nucengdes.2016.04.013 0029-5493/© 2016 Elsevier B.V. All rights reserved.
nested Monte Carlo simulation approach, where the epistemic parameters are sampled in the outer loop, while the aleatory variables are simulated as part of the inner loop (Wu and Tsang, 2004; U.S. NRC, 2012; Sankararaman and Mahadevan, 2013; Wang and Duan, 2013; Duan et al., 2015). The overall uncertainty in the estimated results (e.g., probability of failure, leak, rupture, etc.) should then be governed solely by the respective number of simulations within each loop. While there are many ways of representing epistemic uncertainty, including fuzzy sets and possibility theory (see RESS, 2004, for examples and detailed discussion), both uncertainties are described using a probabilistic representation in this paper. This approach is not only consistent with the other simulation studies listed above, but also convenient for problems (including the example problem presented here) where the uncertainty designation of a particular variable may be changed between the two types as required. The main objective of this study is to demonstrate, using simple examples, how the process of uncertainty separation may lead to misinterpretation of the results, particularly with respect to the confidence (i.e., lower and upper bounds) in the estimated probabilities. Much of the confusion is related not only to how the uncertainty in the variables and parameters is defined, but also the way in which the uncertainties are propagated to the final results.
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
69
The present discussion is especially relevant for probabilistic assessments, such as the xLPR probabilistic fracture mechanics project (Rudland, 2011; U.S. NRC, 2012), where the estimation of extremely low probabilities naturally raises the question of confidence associated with the results. The issue of confidence is directly related to the establishment of suitable acceptance criteria, which is one of the key challenges currently preventing the widespread application of probabilistic assessments, for example, as a standalone approach for structural integrity assessment (Duan et al., 2015). The paper is organized as follows. The simple example problem is presented in Section 2, including the key distinction between first- and second-order random variable problems. The central point of the paper is explored in Section 3, which demonstrates the consequences of applying the two-staged simulation approach in a first-order problem. Section 4 illustrates the natural application of the two-staged approach in a second-order problem, with the summary and conclusions presented in Section 5.
2.2. Simple model
2. Problem statement
Consider the following simple model for the time to leak for a pipe, for example, from stress corrosion cracking, as
All probabilistic assessments involve a computational model (e.g., complex mathematical equations, finite element code, fault tree, etc.) that defines the relationships and interactions between key parameters and variables (e.g., material properties and crack development characteristics in a fracture mechanics code used to model the timing of pipe rupture). Depending on the problem, the parameters and variables can be characterized as either deterministic (i.e., having known fixed values) or as uncertain (i.e., subject to aleatory or epistemic uncertainty). Regardless of the uncertainty designation (aleatory vs. epistemic) all uncertain variables (also referred to as random variables in probabilistic literature) are described using probability distributions that define the range and nature of variability associated with each variable. The uncertainty in the model output will then directly reflect the prescribed randomness or uncertainty in the model input parameters.
2.1. Model uncertainty Clearly, the results are also impacted by any uncertainties in the model itself (i.e., the ability and applicability of the mathematical equations to represent “reality”). This is a fundamental problem with all computational models, and is exceedingly difficult to quantify. The typical approach is to “validate” and “verify” the model against well-defined cases and other models (e.g., Wang and Duan, 2013), however, this does not completely eliminate or quantify the uncertainty in the model results. Probabilistic assessments also contain a second source of model uncertainty, attributed to the probabilistic models used to describe the uncertain input variables. The chosen or prescribed distributions (e.g., normal, log-normal, etc.) have a direct impact on the model results, however, the true or exact distribution types and their parameters can never be known with certainty. This issue is typically addressed through sensitivity analysis, which may involve not only the uncertain variables, but also an assortment of deterministic parameters. Because the main focus of this study is on the uncertain input variables (labelled as aleatory or epistemic) and how their relative contributions are propagated and expressed in the final results (i.e., model output), the issue of model uncertainty, while fundamentally important, will generally be ignored (i.e., all models, both computational and probabilistic, are assumed essentially to be “correct”).
f(a,b)
f(c,d)
Hyper Parameters (a, b, c, d)
Parameter (α) Parameter (β)
f(α,β) Distribuon Parameters
2nd Order 1st Order f(μ,σ)
(α, β, μ, σ)
Time to Iniaon (TI)
Model Parameters
TL = T I + W R
Growth Rate (R)
Fig. 1. First- vs. second-order random variable probabilistic model definition.
TL = TI +
W R
(1)
where TL is the time to leak, TI is the time to crack initiation, W is the wall thickness of the pipe, and R is the crack growth rate. The term W/R represents the time it takes for an initiated crack to grow through the pipe wall, resulting in a leak. The key question from a regulatory or fitness-for-service perspective is to determine When is the pipe going to leak (and ultimately rupture)? Assuming all the variables are known precisely (i.e., deterministic model), the time to leak can be assessed directly without any uncertainty (except for model uncertainty, of course). In reality, many of the parameters contributing to the leakage and failure of a pipe are unknown, and hence described as uncertain or random variables. Because of the complexity of the problem, the solution is also generally obtained through numerical simulations with their own challenges and limitations. This then raises the secondary, and perhaps more important question confronted in this paper What is the uncertainty or confidence in the estimated time (or probability) of leak? 2.3. Input variable uncertainty Describing any of the model parameters in Eq. (1) as uncertain variables implies that the model output, i.e. time to leak, also becomes an uncertain or random variable. While the computational model may be highly complex mathematically (which is not the case here), probabilistically this is referred to as a basic or first-order random variable model. As shown in Fig. 1, this means the addition of one or more (constant) probability distribution parameters (e.g., ˛, ˇ) to the problem by each of the uncertain input variables. In some cases, the new distribution parameters themselves may be subject to uncertainty, and hence described as random variables. As shown in Fig. 1, this adds a second layer (i.e., second-order) uncertainty to the problem, and introduces additional parameters (referred to as hyperparameters in the Bayesian context) to be estimated. Given the prescribed distributions and their parameters, the main challenge then is to determine the probability distribution of the model output (i.e., time to leak), which would not only reflect the combined influence of uncertainty of all the input variables, but also quantify the relative contribution of aleatory and epistemic uncertainties.
70
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
1 Cumulave Fracon
Mean Predicon 0.8
95 % Confidence Interval
0.6 0.4 0.2 0 0
100
200
300
400
500
600
700
800
Time to Leak (months) Fig. 2. Cumulative distribution of time to leak, TL , estimated using 1000 Monte Carlo simulation trials.
3. Basic (first-order) random variable model Let us assume the pipe wall thickness, W, in the simple model is constant and equal to 40 mm. Because of the uncertain nature of the cracking process, assume the time to initiation, TI , follows the Weibull distribution with shape parameter equal to 3 and scale parameter equal to 480 months (i.e., 40 years), and the growth rate, R, follows the Normal distribution with a mean of 5 mm/month and standard deviation of 1 mm/month. This means that the time to initiation is highly variable over time (i.e., the average time to initiation is approximately 429 months with a standard deviation of 156 months), while the crack growth is relatively fast (i.e., reaching through-wall in approximately 8 months on average in this case). For the time being, assume the random input variables are simply uncertain, with no specific designation as either epistemic or aleatory. 3.1. Uncertainty in time to leak Even for this relatively simple problem, no analytical expression can be derived for the distribution of time to leak. Fig. 2 shows the distribution of time to leak estimated from 1000 Monte Carlo simulation trials. The simulation was conducted by sampling each input distribution independently (i.e., time to initiation, TI , and growth rate, R), evaluating the model deterministically, and aggregating the results. The model output shown in Fig. 2 reflects both the range of variability in the time to leak (i.e., between zero and 800 months), as well as the sampling error from the simulation process (i.e., as reflected by the confidence interval). 3.1.1. Prediction interval The inherent or aleatory uncertainty in the time to leak can be characterized numerically using the concept of a prediction interval. For example, the 95% prediction interval for time to leak can be obtained directly from Fig. 2, with the lower limit (i.e., 0.025 percentile value) equal to approximately 150 months, and the upper limit (0.975 percentile value) estimated as 750 months. This means that for a given 40 mm thick pipe, the time to leak is estimated to be between 150 and 750 months, with 95% probability. It is also possible to estimate other quantities, such as the average or expected time to leak, which is equal to 437 months. 3.2. Uncertainty in probability of leak Given the uncertainty in time to leak (as reflected in Fig. 2), it is impossible to answer the first question of “when is the pipe going to leak?” (i.e., time to leak is a random variable). Therefore, the focus is on estimating the probability of leak at a particular time, i.e., P(TL ≤ t). In many probabilistic assessments, the output of the
computational model is not a probability in and of itself. Rather, it is some other key variable, such as time, or physical quantity such as stress, strength, temperature, pressure, wall thickness, etc., which must be compared to some critical (often code or regulatory specified) value (this is sometimes referred to as a performance function, or limit-state equation). The actual probability then arises from the fact that the key variable is not a fixed (deterministic) value, but an uncertain quantity with a distribution. In the present problem, the distribution of time to leak shown in Fig. 2, is equivalent to the distribution of probability of leak over time (i.e., Fig. 2 represents the proportion of the distribution (i.e., simulated samples) with a TL less than or equal to t, which is equivalent to probability of leak at time t). For example, the probability of leak at 720 months (i.e., 60 years) is equal to approximately 0.96, meaning that there is a high probability that a leak will take place before 60 years. The critical comment here is that, for any problem involving a first-order function of random variables with known distributions and their parameters (again, the function can be highly complex mathematically), only the estimated output variable, e.g., the time to leak, is by definition (i.e., inherently) random or uncertain, not the actual probability itself. The probability associated with the output variable is essentially a fixed number (while it varies over time in this problem, it is still constant at any specific time), and is only subject to uncertainty arising from the estimation (e.g., when using simulation methods). Again, this assumes that all random variables and their distributions are known precisely, and is only applicable in the context of the model (i.e., model uncertainty is ignored).
3.2.1. Confidence interval The sampling error (often referred to as epistemic uncertainty) due to simulation is straightforward to quantify using standard methods (Hammersley and Handscomb, 1975). The confidence interval in the estimated probability is obtained using the Beta distribution (or the equivalent Normal approximation) and is shown for the example problem by the dashed grey lines in Fig. 2. As shown in Fig. 2, the 95% confidence interval is fairly narrow, even for only 1000 simulations. Again, the issue of uncertainty or confidence (i.e., upper or lower bounds) regarding the probability of leak arises purely from the simulation process, not the basic random variable model itself.
3.3. Uncertainty separation The concept of uncertainty separation has been of interest in many studies, ranging from its proponents (e.g., Hoffman and Hammonds, 1994; Rao et al., 2007; Sankararaman and Mahadevan, 2013) to those questioning the entire premise (e.g., Der Kiureghian and Ditlevsen, 2007). In principle, identifying the relative contribution of lack of knowledge (i.e., epistemic) uncertainty and inherent (i.e., aleatory) variability on the final results appears highly attractive, especially from a risk management perspective. Because epistemic uncertainty can generally be reduced with the additional expenditure of resources, (risk-informed) decisions can readily be made with respect to the cost/benefit of such activities. However, the main challenge with the separation of uncertainty is agreeing on which uncertain input variables should be described as aleatory as opposed to epistemic. To facilitate this issue, many models, including the xLPR project (Rudland, 2011; EPRI, 2011; U.S. NRC, 2012), allow many of the model parameters and variables to be designated either as epistemic or aleatory (or also constant). While beneficial to the end user, the arbitrary separation of variables may lead to unexpected results in a first-order random variable problem, specifically with respect to the confidence intervals (i.e., upper
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
71
Fig. 4. Probability of leak over time including the conditional mean, median and 5% and 95% bounds, assuming the time to initiation, TI , is sampled in the outer (epistemic) loop with 100 trials, while the growth rate, R, is simulated in the inner (aleatory) loop with 100 trials. (For interpretation of the references to colour in text near the reference citation, the reader is referred to the web version of this article.)
and lower bounds) of the estimated probabilities, as shown in the following. 3.3.1. Two-staged nested Monte Carlo simulation The relative contribution of each type of uncertainty on the final results is typically obtained through a two-staged nested Monte Carlo simulation approach. As shown in Fig. 3, the standard two-staged nested simulation approach involves first sampling the epistemic variables in an outer loop, followed by simulation of the aleatory variables in an inner loop for a given set of random parameters (Wu and Tsang, 2004; Sankararaman and Mahadevan, 2013; Wang and Duan, 2013; Duan et al., 2015). The results from each loop are then aggregated and analyzed separately in order to display the relative contributions in the final results. 3.3.2. Separating variables To illustrate the impact of uncertainty separation in a first-order problem, let us assume (arbitrarily) that the time to initiation, TI , in our simple model is subject to epistemic uncertainty, while the growth rate, R, is assumed to be inherently random (i.e., an aleatory variable). According to the process outlined in Fig. 3, time to initiation is now sampled as part of the outer epistemic loop, while the growth rate is part of the inner aleatory loop. This means that for each random (but constant) single value of time to initiation, ti , from the outer loop, the time to leak is estimated in the inner loop as TL = TI +
W R
(2)
which is now a function of only a single random variable (i.e., the growth rate, R). Given the random sampling of R in the inner (aleatory) loop, Fig. 4 shows the estimated distribution of probability of leak over time, including the lower and upper bounds (with q corresponding to the percentile level), using 100 inner and outer simulation trials. As shown in Fig. 4, the mean estimate (red line) is very close to the
1 q = 0.95 Mean esmate q = 0.5 ~ Median q = 0.05 Exact value
0.8 Probability of Leak
Fig. 3. Two-staged nested Monte Carlo simulation approach involving separated aleatory and epistemic random variables.
exact result, and should be comparable to the result in Fig. 2. Clearly, however, the bounds between Figs. 2 and 4 are very different. Each grey line in Fig. 4, sometimes referred to as a “hair” (Duan et al., 2015), represents a single realization (out of 100) of the time to leak distribution for each outer (epistemic) loop in the simulation. The lines are nearly vertical, because the variability in the inner loop (i.e., in the growth rate, R) is much less than the variability in the time to initiation. That is, for any given value of time to initiation, ti , the crack growth is very rapid, reaching through-wall in approximately 8 months on average (as specified by the Normal distribution), while the value of time to initiation itself varies greatly between zero and 800 months (according to the prescribed Weibull distribution). Therefore, as opposed to confidence bounds on the probability of leak, the lower and upper bounds in Fig. 4 essentially represent the large uncertainty associated with the time to initiation, as described by the Weibull distribution. More specifically, the bounds are conditional on the random time to initiation as P(TL ≤ t | TI ), and therefore should be referred to as sensitivity bounds, rather than confidence bounds. For further illustration and comparison, Fig. 5 shows the results for the opposite case, where the growth rate, R, is now assumed to be epistemic and hence, sampled as part of the outer loop, while the time to initiation, TI , is assumed to be inherently random and
0.6 0.4 0.2 0 0
100
200
300 400 500 Time (months)
600
700
800
Fig. 5. Probability of leak over time including the conditional mean, median and 5% and 95% sensitivity bounds, assuming the growth rate, R, is sampled in the outer (epistemic) loop with 100 trials, while the time to initiation, TI , is simulated in the inner (aleatory) loop with 100 trials. (For interpretation of the references to colour in text near the reference citation, the reader is referred to the web version of this article.)
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
part of the inner aleatory loop. As before, both loops utilized 100 simulation trials. Similar to Fig. 4, the family of curves (i.e., grey lines or “hairs”) in Fig. 5 represents the probability of leak over time for each outer loop (i.e., having a constant growth rate, R, in this case). Rather than time to initiation, the variation in the curves (i.e., sensitivity bounds) now reflects the variability in the growth rate, R. As shown by Figs. 4 and 5, it is evident that the probability of leak (or the distribution of time to leak) is more sensitive to the uncertainty in the time to initiation, TI , as compared to the uncertainty in the crack growth rate, R. Regarding the overall uncertainty, the key point again is that the uncertainty in the probability of leak itself is not dependent on the uncertainty in the input variables. It only depends on the number of simulation trials, as illustrated by the mean estimates (red lines) in Figs. 4 and 5. While the sensitivity bounds would largely be unaffected, increasing the number of trials would naturally result in a better estimate for the (mean) value of probability of leak. The mean estimates should be the same in both cases (except for the sampling error) regardless of the order of separation or looping structure, because the conditional expectations are the same. Using the rule of total probability, the conditional expectation can be expressed as E[P(TL ≤ t|TI = ti )] = P(TL ≤ t) = E[P(TL ≤ t|R = ri )]
(3)
which is simply equal to the unconditional distribution of time to leak. In summary, the sensitivity bounds describe how the model output is impacted by the uncertainty or variation in the outer (epistemic) variables. This is useful for sensitivity analysis, because it not only identifies the most critical variables, but also quantifies their impact on the estimated results. For example, reducing the uncertainty in the distribution of time to initiation, TI , as opposed to the crack growth rate, R, would have a much larger impact on the uncertainty in the distribution of time to leak, and hence the estimated value of probability of leak (i.e., the answer to the first question). However, the uncertainty in the probability of leak would not be affected, because it would only depend on the simulation sample size (i.e., the answer to the second question). In the end, the critical question regarding the confidence on the estimated probability (i.e., the true lower and upper bounds) cannot be answered in this context. Applying the two-staged nested simulation approach in a first-order problem results in conditional sensitivity bounds, rather than the confidence bounds on the probability. In a first-order random variable problem, the true bounds can only be obtained without the separation, and applying the standard methods of estimation as discussed in Section 3.2.1. 4. Second-order random variable model The two-staged nested Monte Carlo simulation approach outlined in Fig. 3 is most applicable in the context of second-order random variable problems. As shown in Fig. 1, this type of problem arises when the parameters of the basic random variables are considered to be uncertain or random themselves. Consider again the simple model where the time to initiation, TI , was assumed to follow the Weibull distribution with shape parameter equal to 3 and scale parameter equal to 480 months (i.e., 40 years). This distribution (i.e., probabilistic model) describes the inherent variability or uncertainty associated with the time to initiation. In this context, the range of variability is defined precisely by the chosen distribution parameters of 3 and 480. The obvious question then is, how good are the chosen parameters in representing the true underlying uncertainty, or in other words, what is the uncertainty associated with the chosen parameters?
0.004 Probability Density
72
0.003 0.002 0.001 0 0
200 400 600 Time to Inaon (months)
800
Fig. 6. Family of distributions for the Weibull model of time to initiation, TI , assuming the shape parameter follows the Normal distribution with mean and standard deviation equal to 3 and 0.5, respectively (each grey line represents a single distribution of 100 realizations of time to initiation with a random shape parameter, while the red line represents the mean or average of the 100 realizations).
4.1. Second-order uncertainty Let us assume that rather than a constant value equal to 3, the shape parameter in the Weibull model of time to initiation is uncertain (e.g., because of lack of information, confidence, etc.) and follows a distribution. For simplicity, assume the shape parameter is described by the Normal distribution with a mean and standard deviation (i.e., hyperparameters) equal to 3 and 0.5, respectively, while the scale parameter remains constant and equal to 480 months. As opposed to a single distribution, this second layer of uncertainty then results in a “family” of distributions for the time to initiation, as illustrated in Fig. 6. The obvious outcome of the second-order uncertainty is that the actual probability of leak will now be uncertain or random as well. In addition to the uncertainty from estimation (e.g., sampling error from simulation), the uncertainty or confidence in the probability of leak will now further be influenced by the degree of uncertainty prescribed to the distribution parameters. The resulting impact of the additional level of uncertainty can readily be quantified using the two-staged nested simulation approach. 4.2. Two-staged simulation The premise for the nested Monte Carlo simulation approach is directly related to the second-order random variable problem definition, as described in Fig. 1. The first-order distributions describe the inherent variability in the computational model parameters, and hence are referred to as aleatory variables. Because the parameters of the distributions are commonly estimated from statistical data analysis, the second-order parameter uncertainty is typically referred to as epistemic (i.e., they depend on the sample size, and hence may be reduced with additional data). Therefore, the uncertain (epistemic) distribution parameters are first sampled in the outer loop, followed by simulation of the uncertain (aleatory) variables in the inner loop. Fig. 7 shows the simulated results for the simple problem assuming the Weibull shape parameter is also a random variable. Similar to before, the two-staged nested simulation approach leads to the generation of a family of distributions for the time to leak, TL , as shown in Fig. 7. Each grey line in Fig. 7 represents a single realization of the time to leak distribution, reflecting the characterized (epistemic) uncertainty in the underlying distribution parameters (i.e., only the shape parameter of the Weibull time to initiation distribution in this case). Because of the simulation approach, the uncertainty or confidence in the estimated probability of leak will depend
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
1
1 q = 0.95 Mean esmate q = 0.5 ~ Median q = 0.05
0.6
q = 0.95 Mean esmate q = 0.5 ~ Median q = 0.05
0.8 Probability of Leak
0.8 Probability of Leak
73
0.4 0.2
0.6 0.4 0.2
(b)
(a) 0
0 0
100
200
300 400 500 Time (months)
600
700
800
0
100
200
300 400 500 Time (months)
600
700
800
Fig. 7. Estimated probability of leak over time, including the mean, median and 5% and 95% confidence bounds, assuming the random Weibull shape parameter for the time to initiation, and given 100 outer (epistemic) trials and (a) 50, and (b) 104 inner (aleatory) simulations.
additionally on the number of simulations. Increasing the number of simulations in the inner and outer loops reduces the overall uncertainty, as shown in Fig. 7b. Naturally, some uncertainty in the probability of leak will always remain, regardless of the number of simulations, due to the (epistemic) uncertainty in the distribution parameters. The main benefit and reason for adopting the two-staged simulation approach in the second-order problem is the relatively simple estimation and transparent presentation of the confidence bounds for the probability estimate. As opposed to the sensitivity bounds in the first-order problem, the estimated bounds are the true confidence bounds in this case, and account for the sampling error from both the inner and outer loops explicitly, without the need for any complex analytical computations. In fact, no simple formulae exist for computing the confidence intervals in a general second-order random variable problem. Naturally, this means the estimated confidence bounds are “estimates” themselves, and hence depend on the number of simulation trials as shown in Fig. 7. In summary, the two-staged nested simulation approach allows the impact of both aleatory and epistemic uncertainties to be displayed effectively and separately on the final results in this context. The aleatory uncertainty or variability is reflected in the shape of the time to leak distribution (i.e., the “s” shape of the curves in Fig. 7), while the epistemic uncertainty (both from parameter uncertainty and sampling error) is characterized by the estimated true confidence bounds. 5. Summary and conclusions The computational models used in probabilistic assessments involve many uncertain or random variables. These variables may be characterized either as aleatory (intrinsically random) or epistemic (uncertain due to lack of knowledge). From a regulatory or fitness-for-service perspective, it is important to estimate not only the probability associated with the performance function (e.g., leak, rupture, failure, etc.), but also the uncertainty or confidence in the final results. Using simple examples, this paper explored the process of uncertainty separation using the two-staged nested Monte Carlo simulation approach in the context of first- and second-order random variable problems. The results of the study demonstrated how using the nested simulation approach in a first-order problem renders the uncertainty in the probability estimate to be conditional on the separated variables, and therefore leads to the estimation of sensitivity bounds, rather than the actual confidence bounds on the probability itself. While the value of the probability estimate does
depend on the uncertainty in the input parameters, it is essentially a fixed number and only subject to error from estimation (assuming all random variables and their distributions are known precisely, and model error is ignored). A second-order random variable problem implies that the parameters of the uncertain input variables are considered themselves to be uncertain. As a result, the probability estimate will also become uncertain or random, reflecting not only the inherent or aleatory uncertainty of the input variables, but also the epistemic uncertainty associated with the distribution parameters, as well as the sampling error from estimation. As opposed to the first-order problem, the two-staged simulation approach is naturally applicable in this context, by allowing the impact of both uncertainties to be displayed effectively and separately on the final results, including the estimation of the true confidence bounds on the estimated probability. Acknowledgements This work is part of the Industrial Research Chair program at the University of Waterloo funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) in partnership with the University Network of Excellence in Nuclear Engineering (UNENE). References Aven, T., Steen, R., 2010. The concept of ignorance in a risk assessment and risk management context. Reliab. Eng. Syst. Saf. 95, 1117–1122. Der Kiureghian, A., Ditlevsen, O., 2007. Aleatory or epistemic? Does it matter? In: Special Workshop on Risk Acceptance and Risk Communication, March 26–27. Stanford University. Duan, X., Wang, M., Kozluk, M., 2015. Acceptance criterion for probabilistic structural integrity assessment: prediction of the failure pressure of steam generator tubing with fretting flaws. Nucl. Eng. Des. 281, 154–162. Electric Power Research Institute (EPRI), 2011. Materials Reliability Program: Models and Inputs Developed for Use in the xLPR Pilot Study (MRP-302). EPRI Report 1022528, Palo Alto, CA. Hammersley, J.M., Handscomb, D.C., 1975. Monte Carlo Methods. Methuen, London. Helton, J.C., Breeding, R.J., 1993. Calculation of reactor accident safety goals. Reliab. Eng. Syst. Saf. 39, 129–158. Helton, J.C., Anderson, D.R., Jow, H.N., Marietta, M.G., Basabilvazo, G., 1999. Performance assessment in support of the 1996 compliance certification application for the waste isolation pilot plant. Risk Anal. 19 (5), 959–986. Hoffman, F.O., Hammonds, J.S., 1994. Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability. Risk Anal. 14, 707–712. Rao, K.D., Kushwaha, H.S., Verma, A.K., Srividya, A., 2007. Quantification of epistemic and aleatory uncertainties in level-1 probabilistic safety assessment studies. Reliab. Eng. Syst. Saf. 92, 947–956. RESS, 1996. Special issue on the “Treatment of Aleatory and Epistemic Uncertainty”. Reliab. Eng. Syst. Saf. 54 (2–3).
74
M.I. Jyrkama, M.D. Pandey / Nuclear Engineering and Design 303 (2016) 68–74
RESS, 2004. Special issue on the “Alternative Representations of Epistemic Uncertainty”. Reliab. Eng. Syst. Saf. 85 (1–3). Rudland, D., 2011. xLPR Version 1.0 Report, Technical Basis and Pilot Study Problem Results. NRC Letter Report, xLPR Computational Group, ML110660292. Sankararaman, S., Mahadevan, S., 2013. Separating the contributions of variability and parameter uncertainty in probability distributions. Reliab. Eng. Syst. Saf. 112, 187–199.
U.S. NRC, 2012. xLPR Pilot Study Report. NUREG-2110. U.S. NRC, Washington, DC. Wang, M., Duan, X., 2013. Benchmarking PRAISE-CANDU 1.0 with xLPR 1.0. In: Proceedings of the ASME 2013 Pressure Vessels & Piping Division Conference, July 14–16, 2013, Paris, France, PVP2013-98010. Wu, F.-C., Tsang, Y.-P., 2004. Second-order Monte Carlo uncertainty/variability analysis using correlated model parameters: application to salmonid embryo survival risk assessment. Ecol. Model. 177, 393–414.