- Email: [email protected]
Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management
Reliability Engineering and System Safety 54 (1996) 217 - 223 ELSEVIER
PII:
S0951-8320(96)00077-4
~) 1996 Elsevier Science Limited Printed in Northern Ireland. All rights reserved 0951-8320/96/$15.00
Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management Stephen C. Hora University of Hawaii at Hilo, Hilo, Hawaii 96720-4091, USA
The quantification of a risk assessment model often requires the elicitation of expert judgments about quantities that cannot be precisely measured. The aims of the model being quantified provide important guidance as to the types of questions that should be asked of the experts. The uncertainties underlying a quantity may be classified as aleatory or epistemic according to the goals of the risk process. This paper discusses the nature of such a classification and how it affects the probability elicitation process and implementation of the resulting judgments. Examples from various areas of risk assessment are used to show the practical implications of how uncertainties are treated. An extended example from hazardous waste disposal is given. © 1996 Elsevier Science Limited. I INTRODUCTION
communication must exist between the experts and the parties seeking their knowledge. Achieving such effective communication is not easy. Often, parties are apparently agreeing while holding different but unarticulated views about the issues in question. 4 As an example, consider a hydrologist responding to questions about the permeability of a geologic formation in which a radioactive waste repository is to be embedded. 5 The hydrologist normally thinks in terms of measurements taken on cores. In contrast, the risk modeller may have in mind the average permeability because the lumped parameters model employed by the risk modeller uses permeabilities that are volume averages over an entire formation. Now, when asked for an uncertainty distribution on permeability, the hydrologist may recall many samples cores that have been drawn from the formation. The inherent variability from core to core provides the experience on which to base the uncertainty of permeability-permeability in the small. Conversely, the modeller really requires uncertainty about an average or integrated permeability-permeability in the large. While the means of the uncertainty distributions for permeability in the small and permeability in the large may be identical, the spread or uncertainty of the distributions may be quite different. Aleatory uncertainty in permeability arises from local spatial variation due to heterogeneity within the formation. The epistemic uncertainty arises from uncertainty about the spatially integrated permeability. Confounding the situation is the
The quantification of risk models for policy and decision making often requires the elicitation of probability distributions from experts. I-3 The distinction between sources of uncertainty often comes into play in the elicitation of these probabilities. Uncertainties are sometimes distinguished as being either aleatory or epistemic. Aleatory uncertainty arises because of natural, unpredictable variation in the performance of the system. The knowledge of experts cannot be expected to reduce aleatory uncertainty although their knowledge may be useful in quantifying the uncertainty. Thus, this type of uncertainty is sometimes referred to as irreducible uncertainty. Conversely, epistemic uncertainty is due to a lack of knowledge about the behaviour of the system that is conceptu~lly resolvable. The epistemic uncertainty can, in principle, be eliminated with sufficient study and, therefore, expert judgments may be useful in its reduction. The position taken in this paper is that a sharp or natural distinction between these two types of uncertainty does not usually exist. The distinction arises because of the specific model to be quantified and the purposes to which the model is to be put. Too often, the questions asked of experts do not clarify how these uncertainties are to be treated. Central to this issue is a clear understanding of the use that will be made of the judgments. To ensure that the expert is responding in a manner suitable to the use, effective 217
218
S. C. Hora
uncertainty about the nature of the spatial variation, that is, how to model the spatial variability. This confounding can lead to epistemic uncertainty about the aleatory uncertainty. Noting that the expression of aleatory uncertainty is conditional on characteristics that are deemed to have epistemic uncertainty leads to a factorization of uncertainties using conditional probabilities that are aleatory in origin and marginal probabilities that are epistemic in origin. Kaplan & Garrick 6 refer to frequency and probability and the notion of a probability of a frequency. Their approach also provides a conditioning of aleatory uncertainty on epistemic uncertainty. The distinctions among types of uncertainties can also be thought of in terms of the sources of uncertainty. The uncertainty labelled aleatory in the preceding example corresponds to the variation within a formation. In contrast, the uncertainty labelled epistemic can be thought of as variation across formations. The distinction between uncertainties, then, is a matter of choice of scale and is, therefore, mutable. The categorization of uncertainties as reducible and irreducible, however, does not seem to fit so well here indicating that the pairs (aleatory, epistemic), (reducible, irreducible) are not truly interchangeable.
property. There may exist epistemic uncertainty because the long-run frequency of point up is not apt to be known. Assuming the subject is rational, me/ruing among other things that the subject's cognitive processes follow the axioms of subjective probability, 8 the probability distribution provided by the subject should be of the form
prob(x) =
The normative specialist, the person leading and directing a probability elicitation, 7 plays a key role in distinguishing between types of uncertainty. The normative specialist must grasp both the nature of the models and the scientific bases that the experts employ in forming the judgments. This knowledge is needed to distinguish between the aleatory and epistemic sources of uncertainty. Without such an understanding, mistakes in probability elicitation are inevitable. The result is often a mixture of aleatory and epistemic uncertainties leaving the analyst with the difficult task of later unravelling the assessments, or even worse, having distributions inappropriate for the purpose they are put to. The aleatory and epistemic components of a subjective probability distribution can be illustrated mathematically by a straightforward example. Consider the experiment of asking a subject to provide a probability distribution for the number of times a tack lands point-up in two tosses. Assuming the tosses are Bernoulli trials (independent with a constant probability of point up), a binomial distribution could be used to codify the uncertainty in the outcome of the exercise. However, the binomial distribution only reflects the aleatory uncertainty in the process. Unlike a coin, where one usually is safe in assuming equally likely heads and tails, the tack lacks a symmetry
fbi~om,,,(xlp,n)dF(p )
where the probability measure F(p) measures the subject's epistemic uncertainty in p, the frequency of point up. Thus, when asked for the probability distribution of x, our subject should respond with prob(x), something quite different to the binomial distribution.
3 DEPENDENCIES
When both aleatory and epistemic uncertainties are present, dependencies or correlations may be introduced into a process. Returning to the tack example, one can easily compute the probability of a tack landing point up as
prob(1) = 2 PROBABILITY ELICITATION
f'
f~,,omio,(llp,1)dF(p )= E(p)
where E(p) is the expected value or mean of the uncertainty distribution for p. By assumption, two tosses of the tack are independent events conditional on a given value of p. However, epistemic uncertainty in the value p introduces dependency between the outcomes of the tosses. The probability of point-up in both tosses is correctly calculated as
prob (2) = ~'fbi,o,~,a,(2lp, 2)dF(p ) = [E(p)] z + var(p) where var(p) is the variance of the uncertainty distribution for p. Clearly, the outcomes of tossing the tack are no longer independent. The probability of a point up on the second toss given a point up on the first toss is E(p)+ [var(p)/E(p)] indicating a positive dependence--a dependence induced solely by the epistemic uncertainty in p. Dependence introduced by epistemic uncertainty is sometimes ignored or not recognized in the modelling of complex systems. Components of the same type, for example, may be assumed to fail independently in presence of only aleatory uncertainty. However, if epistemic uncertainty exists about the common failure probability for the components, the resulting failures will no longer be independent a la the tack example. Consider a system with several redundant components so that the system fails only if all of these
Aleatory and epistemic uncertainty in probability elicitation components fail. Multipte computers on the space shuttle and dual braking systems are well known examples of redundant systems. Suppose that the failure probability for each of rn components is known to be p. The system failure probability is then pro. If, however, the common component failure probability is uncertain and the uncertainty is described by a distribution F(p) for p, then the failure probability for the system is
P(failure ) = -lo~prudE(p). The difference in the calculated values for system failure can be dramatic. If m = 4 and p is known to be 1/2, the system failure probability is 1/16. In contrast, if the common component failure probability is uncertain and is assigned a uniform distribution on [0,1] so that the mean is still 1/2, the resulting system failure probability rises to 1/5, more than three times larger than when only the aleatory uncertainty is considered. 4 DISTRIBUTIONS FOR MODEL PARAMETERS Often, the objective of asing expert judgments is to obtain information about the parameters of a process. The p in the forgoing system failure expression is one such parameter. If, however, one has asked questions about the output of the process rather than the parameters of the process, the resulting distribution will embody both epistemic (here parametric) uncertainty and the aleatory uncertainty inherent in the process. One might ask, then, questions directly related to the parameters of the process and thus avoid the introduction of the confounding aleatory uncertainty. Sometimes this approach becomes problematic because the experts cannot reasonably be expected to respond to questions about parameters because parameters are most often not directly observable. For example, competing models may exist or there may be uncertainty about the correctness of the posited model. Various models will have different parameters. The expert, ~hen, could be put in the most uncomfortable position of having to respond to questions about something which is not believed. This potential conflict between the modellers assumptions and the beliefs of the expert can be a major source of concern. This potential for this difficulty arose in a recent effort to quantify consequence models for nuclear reactor accidents.9'~° The model to be quantified is a Gaussian plume model given by
C(x,y,z)-~
1
.expt
l[(y~2+
z 2
where C(x,y,z) is the concentration in plume at a
219
downwind distance x from a point source and y and z are displacements from the centre of the plume in the horizontal and vertical directions, respectively. Both try and o'~ are functions of the downwind distance x where try = ax b and try = cx d. Here the parameters of interest are a, b, c and d. These parameters are uncertain and the goal of the elicitation is t o quantify uncertainty distributions for these parameters under a number of different classes of atmospheric conditions, i.e., stability classes. An expert who does not subscribe to the Gaussian model cannot be expected to provide meaningful judgments about its parameters. Questions about these parameters are no more meaningful to such an expert than is a question about the sex of God meaningful to an atheist. Because alternative models are available and an expert may subscribe to some other model, or to no model at all for that matter, a decision was made to ask for probability distributions only for observable or potentially observable quantities. This is a well established principle in probability elicitation. ~ Thus, questions about actual concentrations were asked from which distributions were obtained by solving an inverse problem. Roughly, the inverse problem is finding input distributions for the parameters of a model that cause the output of the model to follow some specified distribution. Usually the problem is under-determined so that some additional criteria such as maximum entropy is needed to select from the potential distributions. ~2 A byproduct of this process is a confounding of aleatory uncertainties with epistemic uncertainties. Here, the aleatory uncertainty expresses the inherent randomness and lack of uniformity of plumes. In contrast, the epistemic uncertainty portrays the lack of knowledge about the average of many such plumes generated during a given class of weather conditions. The target of the study, however, was to portray the uncertainty about such an average plume rather than address the randomness from plume to plume within a given a class of plumes.
5 SEPARATING ALEATORY A N D EPISTEMIC UNCERTAINTIES
Which sources of uncertainty, variables, or probabilities are labelled epistemic and which are labelled aleatory depends upon the mission of the study. This is an important consideration in probability elicitation because onceconfounded, it may be very difficult to develop probability distributions that correctly reflect the various uncertainties. One cannot make the distinction between aleatory and epistemic uncertainties purely through physical properties or the experts' judgments. The same quantity in one study may be treated as having aleatory uncertainty while in
220
S. C. Hora
another study the uncertainty maybe treated as epistemic. For example, a study of reactor safety may be designed to produce estimates of the long term risks averaged over a variety of weather conditions as discussed in the preceding section. In such a study the representation of uncertainty about the behaviour of the system averaged over a very long period results in averaging over the aleatory uncertainty about short term weather fluctuations. Another study, in contrast, may be designed for emergency response and individual weather conditions may be meaningful if not critical. In the second study weather conditions may be dominate factors in determining risk and thus uncertainty in weather is the largest contributor to uncertainty in risk. Here uncertainty about the weather at the time of an emergency is resolvable and should be considered epistemic. Careful definition of the endpoint of the study is necessary to make these distinctions and to communicate them to the participants. The design of the probability elicitation process and the questions put to experts should acknowledge the need to separate types of uncertainty. Questions should be constructed so that these uncertainties retain distinct representations. Often, this is accomplished by using conditional probability distributions for the quantities having aleatory uncertainty and marginal probability distributions for quantities having epistemic uncertainty. The same strategy can be used for quantities embodying both types of uncertainty. For example, the measured concentration of a radionuclide at a given point that is traversed by a plume of radioactive materials as described earlier may embody both aleatory and epistemic uncertainty. The aleatory uncertainty reflects the random deviations in wind direction and heterogeneity in concentrations, perhaps as measured as a departure from the ideal Gaussian plume. The epistemic uncertainty reflects the variation due to lack of knowledge about how plumes will behave on average given the chosen set of weather conditions. To ensure that the elicitation process retrieves the desired information, the questions asked need to be carefully posed. In particular, qucstions about quantities that are integrated or averaged over time or space should be carefully constructed to decompose the elicitation quantities so that the two types of uncertainty are appropriately represented. In some cases, it is possible to model aleatory and epistemic uncertainty in terms of an additive or multiplicative model with independent components. For example, the uncertainty in rainfall during a given period, say a year, may be a function of the annual average rainfall #, and a deviation from the annual average, say E. The rainfall, X, can then be expressed as X =/x + e. Probability distributions on X and e or X and # can then be used to derive the distribution of
the third quantity through deconvolution by LaPlace or Fourier transforms.
6 EXAMPLE FROM H A Z A R D O U S WASTE MANAGEMENT Disposal of highly toxic waste often entails placing the waste in containers for shipment. A potential hazard in the handling of the waste is dropping the container from a height sufficient to cause failure. The height required to induce a failure depends upon a number of factors including the weight and construction of the container, the surface on which the container falls and the orientation of the container at contact. In this case study, failure probabilities associated with dropping a large container onto a probe or stub will be considered. A major difficulty encountered is that no drop tests on stubs have been conducted for this specific type of container although drop tests have been conducted on other types of containers. During a probability elicitation associated with this problem, one of the experts presented a failure model for the threshold failure height for an object dropped on a cylindrical probe. The model was later revised to be H = ksyemaxtR~/(mg) where k is a constant, Sy is the yield strength of casing material, e,,ax is the maximum strain at failure, t is the casing thickness, R is the lessor of the plastic wave penetration depth and one half the circumference of the container, m is the mass of the object, and g is local acceleration due to gravity. The model returns the value H which is the threshold height above which failure occurs and below which failure does not occur. This model assumes the formation of cosine shaped dimple and will, therefore, be called the dimpling model. The dimpling model is inherently deterministic and does not provide for any uncertainty regarding failure given the values of several of these parameters. The parameters themselves, however, may be uncertain inducing uncertainty in the threshold height. There are two types of uncertainty in the parameters that must be dealt with. First, values of some parameters may not be exactly known because the description of the container may not provide this information and therefore estimates must be used such as representative values extracted from secondary sources. Maximum strain at failure, for example, is relatively constant across similar containers so oncc this value were known it could be applied to all containers of a given type. Uncertainty about maximum strain at failure will be characterized as epistemic. Second, values of thc parameters may differ from container to container of the same type or even at various locations with in a container. For example, the model assumes that the probe contacts the container perpendicular to cylinder. The actual orientation of a container may
Aleatory and episternic uncertainty in probability elicitation vary and cause the effective mass, m, to vary significantly from drop lo drop. This uncertainty is characterized as aleatory uncertainty. In addition to uncer:ainty in parameters, there exists uncertainty about how well the model captures reality. The dimpling model can not be expected to predict unerringly, even if the parameters could be known with certainty. Moreover, the model might exhibit consistent biases. There is, therefore, modelling uncertainty that is epistemic. Each of these uncertainties plays an important role in using both the mathematical model of failure and analogous data in forming judgments about the failure of dropped containers. How each type of uncertainty enters the process will now be discussed. One parameter with aleatory uncertainty is m, the effective mass of the object. This uncertainty arises first because the actual weight of the container is uncertain due to the po,;sibility of partial filling and, second, to the orientation at impact. Denote by x the vector of parameters having aleatory uncertainty. The uncertainty in the elements of x is captured by a joint probability distribution. But here, there may be uncertainty about the probability distribution for x. That is, one may be uncertain about how x varies. Even the appropriate family of distributions may be uncertain. The uncertainty about the probability distribution for x is captured through epistemic uncertainty about parameters indexing the distribution of x. The parameters having epistemic uncertainty are denoted by the vector y and the conditional probability density of x given y is denoted by f ( x l y ). The model of threshold failure height will bc denoted by H(x,y). Let l(w)= 1 if w >-0 l ( w ) = 0 otherwise. Let d; be the drop height of a container and let 6; = 1 if the container fails and ~, = (I otherwise. Suppose that the elements of y were known with certainty but those of x were not. The joint probability of a sequence of drops giving the data (d~, &) i = 1..... n is given by
p(6, ..... 6,,[d, ..... d,,,y) II','_ t~,,,~ 112.~, - 1)(d, - H(x,y))]f(xJy)dr. If the data are for the appropriate type of container. then the above probability expression can be used as a likelihood in conjunction with the experts initial distribution for y to obtain a posterior distribution on y via Bayes" theorem. The problem is more complex when the data arc not taken on the target type of containers. However, if some of the parameters in the dimpling model are generic with respect to the type of container, then failure data from other types of containers should be useful. To model the use of analogous data from the drops of other types af containers, the vector of
221
parameters y will be partitioned into w and z where w represents generic parameters applicable to all containers and z represents parameters specific to the various container types. The joint probability density for y ' = ( w , z ) is written in factored form as fAzlw)g(w) where j indicates t h e / t h type of container. Now suppose that there are nj observations on the jth type of container and m types of containers. The combined data set is then D = {d,r 6;j, i = 1, nj, j = 1,m}. Let j = 1 signify the container that is of central interest here. Because no drop data on probes exists for this type of container, n~ = O. It is assumed that nj > 0 for ] > 1. The posterior density of w is then
f(wlD )=cg(w)I1;''_,f,,
{II'" ,=,I[(28, i
1)
Ilzt
× (d,, -
H(x,w,z~))]~(xlw,z,)d.r}~(z~lw)dzj
where c is a positive constant which makes this function a density. This posterior distribution and the specific distributions applicable to target containers but not other containers can then be used to calculate the probability of a container failing or the probability of several containers failing in multiple drops. The most general case, that of several drops from different heights yields the equation p(81 ..... 8,,[d, ..... x
d,)=J,,,,,..J,,,,.
[I1,_ ,.,, f , , m l [ ( 2 & - l ) ( d , -
H(x,y))]f(x[z)dx ] x f ( z [w)f(wlD) dz dw.
To reinforce the concepts, a simple numerical example will be given. Suppose that data on three types of containers are available to compute the posterior distribution for the generic parameters. The constant but uncertain quantity w = k / g in the dimpling model is handled as a generic parameter while y = s , e .... tR 2 are assumed to be known constants specific to each type of container. The remaining term, x = l/m embodies the aleatory uncertainty. The data, both drop height and drop result, are given in Table 1. Table 1. Hypothetical drop test data
Type
Drop height
Number of drops
Number of failures
I
20 FI" 30 FI" 7.5 FT 15FT 4.5 FI"
10 10 16 4 10
0
II FT
I0
11 111
1 10 4 0 6
S. C. Hora
222
container specific uncertainty.
Table 2. H y p o t h e t i c a l iognormal distributions (m = E [ I n ( Y ) ] , s = var(YP) °'~
Type
Y
and
aleatory
X
m = 11.3 m=14.7 m = 11.8 rn = 11.9
I
II III Target
parameters,
s=l s=l s=l s=1
m = -5.3
s=l
m = -9.4 m = -6.9 m = -7.8
s=l s=l s=1
7 CONCLUSIONS
Table 2 gives the prior distributions that would, in practice, be obtained by expert elicitation or other means. The prior distribution of W is assumed to be uniform over the interval [0.01,0.I0]. The parameter m = 11.9 for the target containers corresponds to an expected value of the term syem,×tR z of about 153125 while m = - 7 . 8 corresponds to an expected weight of 2500 lb. The density of W is derived from the data for the Type I, II, and III containers and is shown in Fig. 1. From this distribution and the container specific distributions for Y and X, a graph of the marginal probability of failure vs failure height can be derived and is shown in Fig. 2. This graph incorporates all three types of uncertainty-epistemic uncertainty about generic parameters, epistemic uncertainty about
Thc examples discussed in this paper are united by a common thread. This thread is the notion of conditional probability. In these examples the aleatory, stochastic, or unresolvable uncertainty is often expressed through a probability function conditioned on variables that represent the epistemic, knowledge, or resolvable uncertainty. It is also illustrated, however, that this factorization is not unique and, in fact, depends heavily on the models and intent of the investigators. Without careful consideration of the sources of uncertainty and their relation to model's data requirements, mistakes in probability elicitation are likely as are the inappropriate use of probability judgments in quantifying the model. The concepts of aleatory and epistemic uncertainty are useful in clarifying data requirements and facilitating communication among experts, modellers, and probability assessors.
REFERENCES
1. Bonano, E. J., Hora, S. C., Keeney, R. L. & von Winterfeldt, D., Elicitation and use of expert judgment in performance assessment for high-level radioactive waste repositories. SAND89-1821, NUREG/CR-5411, Sandia National Laboratories, Albuquerque, NM, 1989. 2. Morgan, M. & Henrion, M., Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis, Cambridge University Press, Cambridge, UK,
/ '
'x
,"
~..
.."
'\,
i
1991.
\
I
0.5
1.5
2
2.5
3
3.5
( T i m e s 1E-1 )
Fig. 1. Density of the generic constant W.
1.2
/
• 0.8 /
,,.- 0.6 0 ,,D 2 0.4 G.
/'
/
,I
1/1 /
/
0.2 0
/
/
//
0
/
2
4
6
8
10
Height in Feet
Fig. 2. Failure probability vs drop height.
12
3. Cooke, R. M., Experts in uncertainty: expert opinion and subjective probability m science, Oxford University Press, Oxford, UK, 1991. 4. Hora, S. C., The acquisition of expert judgment: examples from risk assessment. J. Energy Engng, (1992) 136-148. 5. WIPP performance assessment division, Preliminary comparison with 40 CFR part 191, subpart B for the waste isolation pilot plant vol. 1. SAND91-0893~4, Sandia National Laboratories, Albuquerque, NM, 1992. 6. Kaplan, S. & Garrick, B. J., On the quantitative definition of risk. Risk analysis, 1 (1981) 11-27. 7. Meyer, M. A. & Booker, J. M., Eliciting and analyzing expert judgment: A practical guide, Academic press, New York, NY, 1991. 8. Savage, L.J., The foundations of statistics, Dover, New York, 1972. 9. Harper, F. T., Hora, S. C., Young, M. L., Miller, L. A., Lui, C. H., M~:Kay, M. D., Helton, J. C., Goossens, L. H. J., Cooke, R. M., Pasler-Sauer, J., Kraan, B. & Jones, J. A., Probabilistic accident consequence uncertainty analysis: dispersion and deposition uncertainty assessment. Vol 1, 2, & 3, NUREG/CR-6244, SAND94-1453, Sandia National Laboratories, NM, 1995. 10. Cooke, R. M., Expert judgment study on atmospheric
Aleatory and epistemic uncertainty in probability elicitation dispersion and deposition. Reports of the faculties of technical mathematics and infomatics No. 91-81, Delft University of Technology, Delft, Netherlands, 1991. 11. Winkler, R. Probabilistic prediction: Some experimental results. J. Am. Stat. Ass., 66 (1971) 675-685.
223
12. Cooke, R. M., Goossens, L. H. J. & Kraan, B. C. P., Methods for CEC/USNRC Accident consequence uncertainty analysis of dispersion and deposition. EUR 15856 EN, European Commission, Brussels, 1994.
PII:
S0951-8320(96)00077-4
~) 1996 Elsevier Science Limited Printed in Northern Ireland. All rights reserved 0951-8320/96/$15.00
Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management Stephen C. Hora University of Hawaii at Hilo, Hilo, Hawaii 96720-4091, USA
The quantification of a risk assessment model often requires the elicitation of expert judgments about quantities that cannot be precisely measured. The aims of the model being quantified provide important guidance as to the types of questions that should be asked of the experts. The uncertainties underlying a quantity may be classified as aleatory or epistemic according to the goals of the risk process. This paper discusses the nature of such a classification and how it affects the probability elicitation process and implementation of the resulting judgments. Examples from various areas of risk assessment are used to show the practical implications of how uncertainties are treated. An extended example from hazardous waste disposal is given. © 1996 Elsevier Science Limited. I INTRODUCTION
communication must exist between the experts and the parties seeking their knowledge. Achieving such effective communication is not easy. Often, parties are apparently agreeing while holding different but unarticulated views about the issues in question. 4 As an example, consider a hydrologist responding to questions about the permeability of a geologic formation in which a radioactive waste repository is to be embedded. 5 The hydrologist normally thinks in terms of measurements taken on cores. In contrast, the risk modeller may have in mind the average permeability because the lumped parameters model employed by the risk modeller uses permeabilities that are volume averages over an entire formation. Now, when asked for an uncertainty distribution on permeability, the hydrologist may recall many samples cores that have been drawn from the formation. The inherent variability from core to core provides the experience on which to base the uncertainty of permeability-permeability in the small. Conversely, the modeller really requires uncertainty about an average or integrated permeability-permeability in the large. While the means of the uncertainty distributions for permeability in the small and permeability in the large may be identical, the spread or uncertainty of the distributions may be quite different. Aleatory uncertainty in permeability arises from local spatial variation due to heterogeneity within the formation. The epistemic uncertainty arises from uncertainty about the spatially integrated permeability. Confounding the situation is the
The quantification of risk models for policy and decision making often requires the elicitation of probability distributions from experts. I-3 The distinction between sources of uncertainty often comes into play in the elicitation of these probabilities. Uncertainties are sometimes distinguished as being either aleatory or epistemic. Aleatory uncertainty arises because of natural, unpredictable variation in the performance of the system. The knowledge of experts cannot be expected to reduce aleatory uncertainty although their knowledge may be useful in quantifying the uncertainty. Thus, this type of uncertainty is sometimes referred to as irreducible uncertainty. Conversely, epistemic uncertainty is due to a lack of knowledge about the behaviour of the system that is conceptu~lly resolvable. The epistemic uncertainty can, in principle, be eliminated with sufficient study and, therefore, expert judgments may be useful in its reduction. The position taken in this paper is that a sharp or natural distinction between these two types of uncertainty does not usually exist. The distinction arises because of the specific model to be quantified and the purposes to which the model is to be put. Too often, the questions asked of experts do not clarify how these uncertainties are to be treated. Central to this issue is a clear understanding of the use that will be made of the judgments. To ensure that the expert is responding in a manner suitable to the use, effective 217
218
S. C. Hora
uncertainty about the nature of the spatial variation, that is, how to model the spatial variability. This confounding can lead to epistemic uncertainty about the aleatory uncertainty. Noting that the expression of aleatory uncertainty is conditional on characteristics that are deemed to have epistemic uncertainty leads to a factorization of uncertainties using conditional probabilities that are aleatory in origin and marginal probabilities that are epistemic in origin. Kaplan & Garrick 6 refer to frequency and probability and the notion of a probability of a frequency. Their approach also provides a conditioning of aleatory uncertainty on epistemic uncertainty. The distinctions among types of uncertainties can also be thought of in terms of the sources of uncertainty. The uncertainty labelled aleatory in the preceding example corresponds to the variation within a formation. In contrast, the uncertainty labelled epistemic can be thought of as variation across formations. The distinction between uncertainties, then, is a matter of choice of scale and is, therefore, mutable. The categorization of uncertainties as reducible and irreducible, however, does not seem to fit so well here indicating that the pairs (aleatory, epistemic), (reducible, irreducible) are not truly interchangeable.
property. There may exist epistemic uncertainty because the long-run frequency of point up is not apt to be known. Assuming the subject is rational, me/ruing among other things that the subject's cognitive processes follow the axioms of subjective probability, 8 the probability distribution provided by the subject should be of the form
prob(x) =
The normative specialist, the person leading and directing a probability elicitation, 7 plays a key role in distinguishing between types of uncertainty. The normative specialist must grasp both the nature of the models and the scientific bases that the experts employ in forming the judgments. This knowledge is needed to distinguish between the aleatory and epistemic sources of uncertainty. Without such an understanding, mistakes in probability elicitation are inevitable. The result is often a mixture of aleatory and epistemic uncertainties leaving the analyst with the difficult task of later unravelling the assessments, or even worse, having distributions inappropriate for the purpose they are put to. The aleatory and epistemic components of a subjective probability distribution can be illustrated mathematically by a straightforward example. Consider the experiment of asking a subject to provide a probability distribution for the number of times a tack lands point-up in two tosses. Assuming the tosses are Bernoulli trials (independent with a constant probability of point up), a binomial distribution could be used to codify the uncertainty in the outcome of the exercise. However, the binomial distribution only reflects the aleatory uncertainty in the process. Unlike a coin, where one usually is safe in assuming equally likely heads and tails, the tack lacks a symmetry
fbi~om,,,(xlp,n)dF(p )
where the probability measure F(p) measures the subject's epistemic uncertainty in p, the frequency of point up. Thus, when asked for the probability distribution of x, our subject should respond with prob(x), something quite different to the binomial distribution.
3 DEPENDENCIES
When both aleatory and epistemic uncertainties are present, dependencies or correlations may be introduced into a process. Returning to the tack example, one can easily compute the probability of a tack landing point up as
prob(1) = 2 PROBABILITY ELICITATION
f'
f~,,omio,(llp,1)dF(p )= E(p)
where E(p) is the expected value or mean of the uncertainty distribution for p. By assumption, two tosses of the tack are independent events conditional on a given value of p. However, epistemic uncertainty in the value p introduces dependency between the outcomes of the tosses. The probability of point-up in both tosses is correctly calculated as
prob (2) = ~'fbi,o,~,a,(2lp, 2)dF(p ) = [E(p)] z + var(p) where var(p) is the variance of the uncertainty distribution for p. Clearly, the outcomes of tossing the tack are no longer independent. The probability of a point up on the second toss given a point up on the first toss is E(p)+ [var(p)/E(p)] indicating a positive dependence--a dependence induced solely by the epistemic uncertainty in p. Dependence introduced by epistemic uncertainty is sometimes ignored or not recognized in the modelling of complex systems. Components of the same type, for example, may be assumed to fail independently in presence of only aleatory uncertainty. However, if epistemic uncertainty exists about the common failure probability for the components, the resulting failures will no longer be independent a la the tack example. Consider a system with several redundant components so that the system fails only if all of these
Aleatory and epistemic uncertainty in probability elicitation components fail. Multipte computers on the space shuttle and dual braking systems are well known examples of redundant systems. Suppose that the failure probability for each of rn components is known to be p. The system failure probability is then pro. If, however, the common component failure probability is uncertain and the uncertainty is described by a distribution F(p) for p, then the failure probability for the system is
P(failure ) = -lo~prudE(p). The difference in the calculated values for system failure can be dramatic. If m = 4 and p is known to be 1/2, the system failure probability is 1/16. In contrast, if the common component failure probability is uncertain and is assigned a uniform distribution on [0,1] so that the mean is still 1/2, the resulting system failure probability rises to 1/5, more than three times larger than when only the aleatory uncertainty is considered. 4 DISTRIBUTIONS FOR MODEL PARAMETERS Often, the objective of asing expert judgments is to obtain information about the parameters of a process. The p in the forgoing system failure expression is one such parameter. If, however, one has asked questions about the output of the process rather than the parameters of the process, the resulting distribution will embody both epistemic (here parametric) uncertainty and the aleatory uncertainty inherent in the process. One might ask, then, questions directly related to the parameters of the process and thus avoid the introduction of the confounding aleatory uncertainty. Sometimes this approach becomes problematic because the experts cannot reasonably be expected to respond to questions about parameters because parameters are most often not directly observable. For example, competing models may exist or there may be uncertainty about the correctness of the posited model. Various models will have different parameters. The expert, ~hen, could be put in the most uncomfortable position of having to respond to questions about something which is not believed. This potential conflict between the modellers assumptions and the beliefs of the expert can be a major source of concern. This potential for this difficulty arose in a recent effort to quantify consequence models for nuclear reactor accidents.9'~° The model to be quantified is a Gaussian plume model given by
C(x,y,z)-~
1
.expt
l[(y~2+
z 2
where C(x,y,z) is the concentration in plume at a
219
downwind distance x from a point source and y and z are displacements from the centre of the plume in the horizontal and vertical directions, respectively. Both try and o'~ are functions of the downwind distance x where try = ax b and try = cx d. Here the parameters of interest are a, b, c and d. These parameters are uncertain and the goal of the elicitation is t o quantify uncertainty distributions for these parameters under a number of different classes of atmospheric conditions, i.e., stability classes. An expert who does not subscribe to the Gaussian model cannot be expected to provide meaningful judgments about its parameters. Questions about these parameters are no more meaningful to such an expert than is a question about the sex of God meaningful to an atheist. Because alternative models are available and an expert may subscribe to some other model, or to no model at all for that matter, a decision was made to ask for probability distributions only for observable or potentially observable quantities. This is a well established principle in probability elicitation. ~ Thus, questions about actual concentrations were asked from which distributions were obtained by solving an inverse problem. Roughly, the inverse problem is finding input distributions for the parameters of a model that cause the output of the model to follow some specified distribution. Usually the problem is under-determined so that some additional criteria such as maximum entropy is needed to select from the potential distributions. ~2 A byproduct of this process is a confounding of aleatory uncertainties with epistemic uncertainties. Here, the aleatory uncertainty expresses the inherent randomness and lack of uniformity of plumes. In contrast, the epistemic uncertainty portrays the lack of knowledge about the average of many such plumes generated during a given class of weather conditions. The target of the study, however, was to portray the uncertainty about such an average plume rather than address the randomness from plume to plume within a given a class of plumes.
5 SEPARATING ALEATORY A N D EPISTEMIC UNCERTAINTIES
Which sources of uncertainty, variables, or probabilities are labelled epistemic and which are labelled aleatory depends upon the mission of the study. This is an important consideration in probability elicitation because onceconfounded, it may be very difficult to develop probability distributions that correctly reflect the various uncertainties. One cannot make the distinction between aleatory and epistemic uncertainties purely through physical properties or the experts' judgments. The same quantity in one study may be treated as having aleatory uncertainty while in
220
S. C. Hora
another study the uncertainty maybe treated as epistemic. For example, a study of reactor safety may be designed to produce estimates of the long term risks averaged over a variety of weather conditions as discussed in the preceding section. In such a study the representation of uncertainty about the behaviour of the system averaged over a very long period results in averaging over the aleatory uncertainty about short term weather fluctuations. Another study, in contrast, may be designed for emergency response and individual weather conditions may be meaningful if not critical. In the second study weather conditions may be dominate factors in determining risk and thus uncertainty in weather is the largest contributor to uncertainty in risk. Here uncertainty about the weather at the time of an emergency is resolvable and should be considered epistemic. Careful definition of the endpoint of the study is necessary to make these distinctions and to communicate them to the participants. The design of the probability elicitation process and the questions put to experts should acknowledge the need to separate types of uncertainty. Questions should be constructed so that these uncertainties retain distinct representations. Often, this is accomplished by using conditional probability distributions for the quantities having aleatory uncertainty and marginal probability distributions for quantities having epistemic uncertainty. The same strategy can be used for quantities embodying both types of uncertainty. For example, the measured concentration of a radionuclide at a given point that is traversed by a plume of radioactive materials as described earlier may embody both aleatory and epistemic uncertainty. The aleatory uncertainty reflects the random deviations in wind direction and heterogeneity in concentrations, perhaps as measured as a departure from the ideal Gaussian plume. The epistemic uncertainty reflects the variation due to lack of knowledge about how plumes will behave on average given the chosen set of weather conditions. To ensure that the elicitation process retrieves the desired information, the questions asked need to be carefully posed. In particular, qucstions about quantities that are integrated or averaged over time or space should be carefully constructed to decompose the elicitation quantities so that the two types of uncertainty are appropriately represented. In some cases, it is possible to model aleatory and epistemic uncertainty in terms of an additive or multiplicative model with independent components. For example, the uncertainty in rainfall during a given period, say a year, may be a function of the annual average rainfall #, and a deviation from the annual average, say E. The rainfall, X, can then be expressed as X =/x + e. Probability distributions on X and e or X and # can then be used to derive the distribution of
the third quantity through deconvolution by LaPlace or Fourier transforms.
6 EXAMPLE FROM H A Z A R D O U S WASTE MANAGEMENT Disposal of highly toxic waste often entails placing the waste in containers for shipment. A potential hazard in the handling of the waste is dropping the container from a height sufficient to cause failure. The height required to induce a failure depends upon a number of factors including the weight and construction of the container, the surface on which the container falls and the orientation of the container at contact. In this case study, failure probabilities associated with dropping a large container onto a probe or stub will be considered. A major difficulty encountered is that no drop tests on stubs have been conducted for this specific type of container although drop tests have been conducted on other types of containers. During a probability elicitation associated with this problem, one of the experts presented a failure model for the threshold failure height for an object dropped on a cylindrical probe. The model was later revised to be H = ksyemaxtR~/(mg) where k is a constant, Sy is the yield strength of casing material, e,,ax is the maximum strain at failure, t is the casing thickness, R is the lessor of the plastic wave penetration depth and one half the circumference of the container, m is the mass of the object, and g is local acceleration due to gravity. The model returns the value H which is the threshold height above which failure occurs and below which failure does not occur. This model assumes the formation of cosine shaped dimple and will, therefore, be called the dimpling model. The dimpling model is inherently deterministic and does not provide for any uncertainty regarding failure given the values of several of these parameters. The parameters themselves, however, may be uncertain inducing uncertainty in the threshold height. There are two types of uncertainty in the parameters that must be dealt with. First, values of some parameters may not be exactly known because the description of the container may not provide this information and therefore estimates must be used such as representative values extracted from secondary sources. Maximum strain at failure, for example, is relatively constant across similar containers so oncc this value were known it could be applied to all containers of a given type. Uncertainty about maximum strain at failure will be characterized as epistemic. Second, values of thc parameters may differ from container to container of the same type or even at various locations with in a container. For example, the model assumes that the probe contacts the container perpendicular to cylinder. The actual orientation of a container may
Aleatory and episternic uncertainty in probability elicitation vary and cause the effective mass, m, to vary significantly from drop lo drop. This uncertainty is characterized as aleatory uncertainty. In addition to uncer:ainty in parameters, there exists uncertainty about how well the model captures reality. The dimpling model can not be expected to predict unerringly, even if the parameters could be known with certainty. Moreover, the model might exhibit consistent biases. There is, therefore, modelling uncertainty that is epistemic. Each of these uncertainties plays an important role in using both the mathematical model of failure and analogous data in forming judgments about the failure of dropped containers. How each type of uncertainty enters the process will now be discussed. One parameter with aleatory uncertainty is m, the effective mass of the object. This uncertainty arises first because the actual weight of the container is uncertain due to the po,;sibility of partial filling and, second, to the orientation at impact. Denote by x the vector of parameters having aleatory uncertainty. The uncertainty in the elements of x is captured by a joint probability distribution. But here, there may be uncertainty about the probability distribution for x. That is, one may be uncertain about how x varies. Even the appropriate family of distributions may be uncertain. The uncertainty about the probability distribution for x is captured through epistemic uncertainty about parameters indexing the distribution of x. The parameters having epistemic uncertainty are denoted by the vector y and the conditional probability density of x given y is denoted by f ( x l y ). The model of threshold failure height will bc denoted by H(x,y). Let l(w)= 1 if w >-0 l ( w ) = 0 otherwise. Let d; be the drop height of a container and let 6; = 1 if the container fails and ~, = (I otherwise. Suppose that the elements of y were known with certainty but those of x were not. The joint probability of a sequence of drops giving the data (d~, &) i = 1..... n is given by
p(6, ..... 6,,[d, ..... d,,,y) II','_ t~,,,~ 112.~, - 1)(d, - H(x,y))]f(xJy)dr. If the data are for the appropriate type of container. then the above probability expression can be used as a likelihood in conjunction with the experts initial distribution for y to obtain a posterior distribution on y via Bayes" theorem. The problem is more complex when the data arc not taken on the target type of containers. However, if some of the parameters in the dimpling model are generic with respect to the type of container, then failure data from other types of containers should be useful. To model the use of analogous data from the drops of other types af containers, the vector of
221
parameters y will be partitioned into w and z where w represents generic parameters applicable to all containers and z represents parameters specific to the various container types. The joint probability density for y ' = ( w , z ) is written in factored form as fAzlw)g(w) where j indicates t h e / t h type of container. Now suppose that there are nj observations on the jth type of container and m types of containers. The combined data set is then D = {d,r 6;j, i = 1, nj, j = 1,m}. Let j = 1 signify the container that is of central interest here. Because no drop data on probes exists for this type of container, n~ = O. It is assumed that nj > 0 for ] > 1. The posterior density of w is then
f(wlD )=cg(w)I1;''_,f,,
{II'" ,=,I[(28, i
1)
Ilzt
× (d,, -
H(x,w,z~))]~(xlw,z,)d.r}~(z~lw)dzj
where c is a positive constant which makes this function a density. This posterior distribution and the specific distributions applicable to target containers but not other containers can then be used to calculate the probability of a container failing or the probability of several containers failing in multiple drops. The most general case, that of several drops from different heights yields the equation p(81 ..... 8,,[d, ..... x
d,)=J,,,,,..J,,,,.
[I1,_ ,.,, f , , m l [ ( 2 & - l ) ( d , -
H(x,y))]f(x[z)dx ] x f ( z [w)f(wlD) dz dw.
To reinforce the concepts, a simple numerical example will be given. Suppose that data on three types of containers are available to compute the posterior distribution for the generic parameters. The constant but uncertain quantity w = k / g in the dimpling model is handled as a generic parameter while y = s , e .... tR 2 are assumed to be known constants specific to each type of container. The remaining term, x = l/m embodies the aleatory uncertainty. The data, both drop height and drop result, are given in Table 1. Table 1. Hypothetical drop test data
Type
Drop height
Number of drops
Number of failures
I
20 FI" 30 FI" 7.5 FT 15FT 4.5 FI"
10 10 16 4 10
0
II FT
I0
11 111
1 10 4 0 6
S. C. Hora
222
container specific uncertainty.
Table 2. H y p o t h e t i c a l iognormal distributions (m = E [ I n ( Y ) ] , s = var(YP) °'~
Type
Y
and
aleatory
X
m = 11.3 m=14.7 m = 11.8 rn = 11.9
I
II III Target
parameters,
s=l s=l s=l s=1
m = -5.3
s=l
m = -9.4 m = -6.9 m = -7.8
s=l s=l s=1
7 CONCLUSIONS
Table 2 gives the prior distributions that would, in practice, be obtained by expert elicitation or other means. The prior distribution of W is assumed to be uniform over the interval [0.01,0.I0]. The parameter m = 11.9 for the target containers corresponds to an expected value of the term syem,×tR z of about 153125 while m = - 7 . 8 corresponds to an expected weight of 2500 lb. The density of W is derived from the data for the Type I, II, and III containers and is shown in Fig. 1. From this distribution and the container specific distributions for Y and X, a graph of the marginal probability of failure vs failure height can be derived and is shown in Fig. 2. This graph incorporates all three types of uncertainty-epistemic uncertainty about generic parameters, epistemic uncertainty about
Thc examples discussed in this paper are united by a common thread. This thread is the notion of conditional probability. In these examples the aleatory, stochastic, or unresolvable uncertainty is often expressed through a probability function conditioned on variables that represent the epistemic, knowledge, or resolvable uncertainty. It is also illustrated, however, that this factorization is not unique and, in fact, depends heavily on the models and intent of the investigators. Without careful consideration of the sources of uncertainty and their relation to model's data requirements, mistakes in probability elicitation are likely as are the inappropriate use of probability judgments in quantifying the model. The concepts of aleatory and epistemic uncertainty are useful in clarifying data requirements and facilitating communication among experts, modellers, and probability assessors.
REFERENCES
1. Bonano, E. J., Hora, S. C., Keeney, R. L. & von Winterfeldt, D., Elicitation and use of expert judgment in performance assessment for high-level radioactive waste repositories. SAND89-1821, NUREG/CR-5411, Sandia National Laboratories, Albuquerque, NM, 1989. 2. Morgan, M. & Henrion, M., Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis, Cambridge University Press, Cambridge, UK,
/ '
'x
,"
~..
.."
'\,
i
1991.
\
I
0.5
1.5
2
2.5
3
3.5
( T i m e s 1E-1 )
Fig. 1. Density of the generic constant W.
1.2
/
• 0.8 /
,,.- 0.6 0 ,,D 2 0.4 G.
/'
/
,I
1/1 /
/
0.2 0
/
/
//
0
/
2
4
6
8
10
Height in Feet
Fig. 2. Failure probability vs drop height.
12
3. Cooke, R. M., Experts in uncertainty: expert opinion and subjective probability m science, Oxford University Press, Oxford, UK, 1991. 4. Hora, S. C., The acquisition of expert judgment: examples from risk assessment. J. Energy Engng, (1992) 136-148. 5. WIPP performance assessment division, Preliminary comparison with 40 CFR part 191, subpart B for the waste isolation pilot plant vol. 1. SAND91-0893~4, Sandia National Laboratories, Albuquerque, NM, 1992. 6. Kaplan, S. & Garrick, B. J., On the quantitative definition of risk. Risk analysis, 1 (1981) 11-27. 7. Meyer, M. A. & Booker, J. M., Eliciting and analyzing expert judgment: A practical guide, Academic press, New York, NY, 1991. 8. Savage, L.J., The foundations of statistics, Dover, New York, 1972. 9. Harper, F. T., Hora, S. C., Young, M. L., Miller, L. A., Lui, C. H., M~:Kay, M. D., Helton, J. C., Goossens, L. H. J., Cooke, R. M., Pasler-Sauer, J., Kraan, B. & Jones, J. A., Probabilistic accident consequence uncertainty analysis: dispersion and deposition uncertainty assessment. Vol 1, 2, & 3, NUREG/CR-6244, SAND94-1453, Sandia National Laboratories, NM, 1995. 10. Cooke, R. M., Expert judgment study on atmospheric
Aleatory and epistemic uncertainty in probability elicitation dispersion and deposition. Reports of the faculties of technical mathematics and infomatics No. 91-81, Delft University of Technology, Delft, Netherlands, 1991. 11. Winkler, R. Probabilistic prediction: Some experimental results. J. Am. Stat. Ass., 66 (1971) 675-685.
223
12. Cooke, R. M., Goossens, L. H. J. & Kraan, B. C. P., Methods for CEC/USNRC Accident consequence uncertainty analysis of dispersion and deposition. EUR 15856 EN, European Commission, Brussels, 1994.