- Email: [email protected]
Estimating the Type II error of detecting changes in foodborne illnesses via public health surveillance
Microbial Risk Analysis 7 (2017) 1–7
Contents lists available at ScienceDirect
Microbial Risk Analysis journal homepage: www.elsevier.com/locate/mran
Estimating the Type II error of detecting changes in foodborne illnesses via public health surveillance
MARK
Eric D. Ebel*, Michael S. Williams*, Wayne D. Schlosser Risk Assessment Division Office of Public Health Science Food Safety Inspection Service, USDA 2150 Centre Avenue, Building D Fort Collins, Colorado 80526, U.S.
A R T I C L E I N F O
A B S T R A C T
Keywords: Passive surveillance Power Underreporting fraction Attribution
Many countries operate public health surveillance systems to monitor foodborne disease occurrence. The illness counts for pathogens of interest are monitored across time to assess if changes have occurred in case rates or the overall illness burden. Common to all of these systems is that only a fraction of all foodborne illnesses are reported and the relationship between observed and the total number of illnesses is uncertain. Food-safety policies intend to affect both the total and observed number of illnesses. Ideally, the surveillance system would be sufficiently sensitive to detect these changes, but the statistical power of these systems is generally not well understood. This study proposes two approaches for estimating the power of a foodborne illness surveillance system. These methods are then applied to assess the power of detecting specified changes in the total number of illnesses in an existing surveillance system. The findings suggest that the power of a national foodborne disease surveillance system to detect modest annual reductions in Salmonella illnesses may be limited. For example, a naïve presumptive model that assumes observed illnesses are reduced directly by 10% predicts that the power to discern this difference, from one year to the next, is 0.21. A Bayesian model, that accounts for uncertainty in projecting changes in total illnesses to the number of observed illnesses, predicts that the power to detect a true 10% reduction is 0.04 (i.e., 4% confidence of detecting this magnitude of reduction or 96% chance of a Type II error) one year after the change has taken effect. Although the power of a national surveillance system increases as the magnitude of reductions increases – or as the number of years the reduction is maintained increases – the Bayesian model demonstrates that power is less than 0.5 for reductions up to 50% for one year and is less than 0.53 for reductions of 30% that are maintained for four years. The limited power to detect intended changes in total annual illnesses of national public health surveillance systems may highlight the need for regulators to also monitor food contamination evidence to gauge progress towards achieving intended policy effects.
1. Introduction More than 75 countries operate surveillance systems to monitor foodborne disease occurrence across time (Herikstad et al., 2002; Scallan and Mahon, 2012; Yeni et al., 2017). Large changes in confirmed illness counts for a specific pathogen indicate the effectiveness of government and industry efforts to improve food safety (Collard et al., 2008). Distinguishing smaller changes in overall food safety may be problematic if the public health surveillance system's power to detect and report changes in foodborne disease occurrence is limited. A conundrum for the food-safety community is that the power of a surveillance system to detect reductions in the national occurrence of foodborne illness, associated with a particular product-pathogen pair, can only be lessened as control programs succeed in reducing observable illnesses (Williams and Ebel, 2012). This observation is
*
based on the realization that it becomes more difficult to distinguish changes in observed illnesses as the number of observed illnesses becomes smaller. For example, assuming observed illnesses are Poisson distributed (with the mean equal to the variance), then it is expected that the signal from a 30% reduction in observed illnesses is readily distinguished when the baseline average illness count is 1000 per year (approximate 95% confidence limits of 938 – 1062) but that signal is lost in the random variability if the baseline average illness count is 10 per year (approximate 95% confidence limits of 4 – 16). In this example, a 30% reduction of 1000 observed illnesses should result in an average of 700 observed illnesses; this is clearly outside the range of random Poisson variability for the baseline. In contrast, a 30% reduction of 10 observed illnesses results in an average of 7 observed illnesses; that is within the variability of the baseline. Although the power to discriminate a change in illnesses depends on
Corresponding authors. E-mail address: [email protected] (M.S. Williams).
http://dx.doi.org/10.1016/j.mran.2017.10.001 Received 28 August 2017; Received in revised form 23 October 2017; Accepted 30 October 2017 Available online 31 October 2017 2352-3522/ Published by Elsevier B.V.
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
For the purposes of illustration, this paper calculates the power of the FoodNet surveillance system to detect a true change in total illnesses by only considering the effect of under-diagnosing the illnesses that occur in the population. Therefore, our analysis ignores other factors that might influence the relationship between total illnesses from a foodborne pathogen and the number of illnesses observed in FoodNet for that pathogen. Our illustration considers Salmonella illnesses and the effects of hypothetical public policies intended to reduce the national occurrence of domestic foodborne illnesses associated with this pathogen. Because Salmonella is one of the most frequently reported bacterial illnesses in FoodNet, the power to detect its differences should be larger than the power for less-commonly reported pathogens. We use two methods for estimating the power of the surveillance system. The first presumptive method assumes that a reduction in total illnesses results in a direct reduction in observed illnesses. The other approach uses Bayesian reasoning (and two alternative prior distributions) to model the effect of a change in total illnesses on the observed illnesses. Additionally, the Bayesian approach supports consideration of the power for determining a reduction given multiple years of observation.
the baseline number of illnesses against which to compare the change, it also depends on the process of projecting an actual change in total illnesses in the population to a change in illnesses observed in a sampled population, like the Foodborne Diseases Active Surveillance Network (FoodNet) surveillance system (Scallan and Mahon, 2012). This projection, from a true but unobservable total annual case count to an observed annual case count, is a source of uncertainty not considered by the previous Poisson distribution argument. Public health agencies need to track progress in their programs. The U.S. Department of Agriculture's Food Safety and Inspection Service uses FoodNet reports to monitor human illnesses on an annual basis.1 Nevertheless, despite evidence of reductions in pathogen occurrence on some raw products it regulates, the success of FSIS programs are difficult to fully support if the public health surveillance evidence suggests no change, or possibly an increase, in occurrence of human illness (Powell, 2016). Given recent work by the Department of Health and Human Services’ Centers for Disease Control and Prevention (CDC), it is possible to estimate total illnesses associated with a pathogen from the annual reported FoodNet illness counts (Ebel et al., 2012; Scallan et al., 2011). This direction of inference, from a limited number of FoodNet cases, reported by 10 sites in the United States, to the entire U.S. population is accomplished by accounting for the catchment area of the FoodNet sites and pathogen-specific multipliers that describe the number of unreported, domestically-acquired foodborne cases for each observed illness reported to FoodNet. Observed illnesses from a pathogen are a subset of its true total number of illnesses. Consequently, total illnesses can be estimated from the subset of reported illnesses, although this estimation may include uncertainty. Uncertainties inherent in a national surveillance system also will influence the observed illnesses. For example, uncertain effects of changes to a surveillance system (e.g., changes in testing methodology and health care systems (CDC, 2017)) and growth in population size may affect the number of observed illnesses. Also, disease occurrence may cluster spatially or temporally in ways that bias a surveillance system that was designed under the assumption of randomly distributed illnesses in space and time. This can be confounded by the different levels of consumer education on safe food handling practices across different geographic regions. In the context of hypothesis testing, power describes the confidence of correctly rejecting the null hypothesis – that total illnesses are unchanged – given that an alternative hypothesis (that total illnesses are reduced) is true. The illnesses observed for a given year is reflected by the surveillance system. Absent a policy that intends to reduce total illnesses, it is reasonable to hypothesize that the observed illnesses will be similar from one year to the next. If a policy is enacted to reduce total illnesses by some expected fraction, 2 then regulators will monitor the observed illnesses in the following years to assess the policy's performance. If observed illnesses are unchanged or increase following the policy's enactment, then regulators may wonder if such a result indicates policy under-performance, or insufficient power to detect the expected magnitude of reduction (i.e., Type II error). Similarly, if observed illnesses decline, regulators may wonder if the effect is indicative of a true change in illnesses or a random fluctuation (i.e., Type I error). At least part of the answer to such policy analysis questions involves understanding the power of the surveillance system to detect changes.
2. Methods The number of illnesses reported by a surveillance system for a pathogen (Iobserved) will differ from the total number of illnesses for the pathogen (Itotal) (Williams et al., 2011). The FoodNet surveillance system identifies human cases associated with a pathogen, but it is subject to underreporting bias (Scallan et al., 2011). Two factors that relate the total number of illnesses for a pathogen to the number of illnesses observed by FoodNet are the number of unreported illnesses(ρ) for each observed FoodNet illness (Ebel et al., 2012) and the fraction of the total population that exists in the catchment area of the surveillance system (κ) (Scallan and Mahon, 2012). The inverse of ρ is the probability that an illness in the population is diagnosed and reported. These two factors modify the total number of foodborne illnesses to estimate the number of observed illnesses whose etiology is the pathogen, so that
Iobserved =
κ Itotal. ρ
2.1. Presumptive approach If we ignore this relationship and simply consider Iobserved as a random variable, then we can calculate the power of detecting some intended proportional reduction using a Z statistic (Weiss, 2015). We refer to this approach as “Presumptive” because it assumes a direct correspondence between an expected reduction in total illnesses and the reduction in observed illnesses. In this case,
crit + DE × δ × ϕ × E [Iobserved] ⎞ ⎛ power = p ⎜Z < * V [Iobserved ] + V [Iobserved] ⎟⎠ ⎝ where Z ∼ Normal(0, 1), E [Iobserved] and V[Iobserved] are the mean and * variance of Iobserved, Iobserved is the observed illnesses reduced by some proportional design effect (DE) that is modified by δ (the fraction of illnesses domestically-acquired) and ϕ (the fraction of domestic illnesses that are foodborne3); and crit is a critical value determined as − 1.96 2V [Iobserved] . E [Iobserved] and V[Iobserved] are estimated from FoodNet evidence. In this development, the design effect should reflect the expected reduction from a policy on observed illnesses. Because the effect of a
1
https://www.cdc.gov/foodnet/reports/index.html. Total illnesses represent those that are foodborne and domestically-acquired, as well as those that stem from non-food sources and/or international exposures. Furthermore, domestically-acquired foodborne illnesses are attributed to particular foods that are regulated by different governmental agencies. Usually, a food safety policy is enacted to reduce domestically-acquired foodborne illnesses associated with a specific product-pathogen pairing (e.g., chicken-Salmonella). Consequently, the reduction in total illnesses for the pathogen will be less than the expected reduction in illnesses attributed to the specific product. 2
3 Scallan et al. (2011) estimate that 89% of Salmonella illnesses are domestically-acquired and 94% of these are foodborne.
2
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
60
8,000 50
Surveillance populaon (millions)
Annual Salmonella cases reported by FoodNet
9,000
7,000 6,000
40
5,000
30 4,000 3,000
20
FoodNet cases
Surveillance populaon
2,000
10 1,000 0
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
0
Fig. 1. Annual Salmonella cases reported to FoodNet, 1996 – 2015 and the size of the surveillance population are shown.
illnesses (Williams et al., 2011). Therefore, the annual total number of salmonellosis cases is modeled as
typical domestic policy will be limited to reducing domestic foodborne illnesses, and not influence illnesses that result from non-food or nondomestic exposures, the resultant observed illnesses should account for this discrepancy. Therefore, the design effect in the power calculation must be reduced by the fraction of illnesses expected to be domestic and foodborne. For example, if a policy is expected to reduce domestic foodborne illnesses by 20%, and we assume that 89% of illnesses are domestic and 94% of these are foodborne, then the reduction in observed illnesses is actually 16.7% (20% × 89% × 94%). Because the Presumptive approach assumes a direct reduction of observed illnesses – and ignores the uncertainty of projecting a reduction in total illnesses to observed illnesses – it is expected that its power will overestimate the true power and be greater than the Bayesian approach described next. Nevertheless, the Presumptive results for power are presented as an upper bound on the actual power and for comparison with those of the Bayesian approach.
Itotal ∼ Poisson (Nservings P (ill exp) P (exp)) = Poisson (λtotal ) ρ
where λtotal is estimated from κ Iobserved . An adjustment to this model is needed when both illness counts include non-foodborne and/or non-domestically acquired disease. The latter is particularly important if food safety policies can only be expected to influence domestic foodborne exposures. Although food safety analysts can estimate total annual illnesses conditioned on the observed illnesses for a given year (i.e., P (Itotal|Iobserved)), interest lies in examining the reverse conditional probability so that we might assess the effects of hypothetical changes in total illnesses on the observed illnesses when modeling potential policy options. Using Bayes Theorem, we can represent the solution as
P (Iobserved Itotal ) = 2.2. Bayesian approach
P (Itotal Iobserved ) × P (Iobserved ) P (Itotal )
(1)
This relationship suggests that P(Iobserved|Itotal)∝P(Itotal|Iobserved) × P (Iobserved). To evaluate the likelihood function on the right-hand side of this ρ proportionality, note that Itotal ∼ Poisson κ Iobserved . If ρ ∼ gamma(α, θ)4
This approach intends to estimate posterior distributions for observed illnesses conditioned on total annual illness counts for baseline and reduction scenarios. The power calculation is similar to that described for the Presumptive approach. For a foodborne pathogen like Salmonella, the total annual illness count is a parameter that results from the aggregate outcomes of consuming millions of servings of different foods. Generally, we can represent this relationship as
(
)
(
)
I
θ . while κ and Iobserved are scalar, then λtotal ∼ gamma α, observed κ A Poisson distributed random variable with a gamma mixture distribution generates a negative binomial distribution (i.e.,
(
Itotal Iobserved ∼ NegativeBinomial r = α, β =
Itotal = Nservings P (ill exp) P (exp)
2012) so the likelihood function is
where Nservings is the number of food servings consumed annually, P (exp) is the frequency of exposure to contaminated servings and P(ill|exp) is the probability that a contaminated serving results in human illness. This probability summarizes the variable effects of factors such as partitioning and mixing of food ingredients, storage, handling, cooking, and pathogenicity of different strains of the pathogen. Given a large number of servings, probability theory suggests that the Poisson process reasonably models the number of annual sporadic
Itotal + α − 1⎞ ⎛ P (Itotal Iobserved ) = ⎜⎛ ⎟ ⎜ Itotal ⎝ ⎠⎝ 1 +
Iobserved θ κ
))
Itotal Iobserved θ ⎞ ⎛ κ Iobserved ⎟ ⎜1 θ κ ⎝ ⎠
(Klugman et al.,
α
⎞ I ⎟ . + observed θ κ ⎠ 1
(2)
4
3
f (ρ) =
ρ ρ α − ⎛ ⎞ e θ ⎝θ⎠ . ρΓ(α )
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
Fig. 2. Distributions of annual observed Salmonella illnesses are shown for the baseline and alternative scenarios where, using the Presumptive approach, observed illnesses are reduced by design effects of 10% to 50%.
Fig. 3. Posterior distributions of annual observed Salmonella illnesses, using the Bayesian approach, are shown for a baseline scenario and alternative scenarios where the assumed total domestic foodborne illnesses is reduced by design effects of 10% to 50%. The prior distribution for each curve is assumed to be uniform across the range of possible observed illnesses.
Fig. 4. Posterior distributions of annual observed Salmonella illnesses, using the Bayesian approach, are shown for a baseline scenario and alternative scenarios where the assumed total domestic foodborne illnesses is reduced by design effects 10% to 50%. The prior distribution for each curve is shown as the shorter and wider solid line in this graph.
4
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
0.2
0.4
Power
0.6
0.8
1.0
Fig. 5. The power to detect proportional reductions in annual observed, or total domestic foodborne, human illnesses by monitoring observed illnesses is shown. If a design effect is applied directly to observed illnesses (the Presumptive approach), then the power tends to rapidly increase with the size of the proportional reduction. In contrast, if the design effect is applicable to total domestic foodborne illnesses (the Bayesian approach), then power increases more slowly. The power for the Bayesian approach with an informed prior is generally least of the three examples.
0.0
Presumptive approach Bayes uniform prior Bayes informed prior
0.0
0.2 0.4 0.6 0.8 Proportional reduction in annual foodborne, domestic illnesses
1.0
Assuming independence between the baseline and reduction distributions, power is calculated using the Z-statistic with the means and variances determined from the moments of the baseline and reduction posterior distributions. The Bayesian approach is amenable to assessing the power to discern a difference if it persists for multiple years. For example, we can estimate power for a reduction that persists for two consecutive years. We estimate the second year posterior distributions for the baseline and reduction scenarios by using prior distributions that are the posterior distributions from the first year. In this manner, we are updating the second year posterior distributions to account for two years of evidence that total illnesses have either remained similar or have been reduced. Similarly, we continue this process for three and four years. The power calculations are based on the moments of the updated posterior distributions.
An expression for the prior probability in Eq. (1) (i.e., P(Iobserved)) is problematic because we are interested in examining how the posterior probability (i.e., P(Iobserved|Itotal)) changes for alternative values of Itotal. A prior distribution that reflects past observed illness values might unnecessarily constrain the posterior if the magnitude of change in total illnesses is large. Therefore, our analysis examines a uniform prior for P (Iobserved) across a plausible range of observed illnesses values (i.e., 1 to 15,000 in our example). We further examine the effects of an informed prior distribution based on analysis of the historic FoodNet data. This alternative prior is developed as a negative binomial distribution intended to capture variability in observed illnesses that arbitrarily ranges from 33% below (2.5th percentile of the distribution) to 33% above (97.5th percentile) its mean value. This range was chosen to illustrate a noticeable effect on the resulting power relative to the uniform prior assumption. To evaluate the effect of an assumed change in total illnesses on the observed illnesses, we first solve Eq. (1) for the baseline number of total illnesses. Next, we consider alternative design effects (DE), ranging from 0 to 1, that progressively reduce the total number of illnesses * , is calculated below the baseline. This reduced number of illnesses, Itotal as
2.3. FoodNet Salmonella data Annual FoodNet data are summarized from the program's inception in 1996 through 2015. Each year's observed Salmonella illnesses (Iobserved, t) are adjusted, for the size of the surveillance population, to generate an ultimate year comparison using the following calculation;
* = Itotal × (1 − DE × δ × ϕ) Itotal
P Iobserved − adjusted, t = Iobserved, t ⎛ U ⎞ ⎝ Pt ⎠
We solve Eq. (1) again for the reduction scenario (i.e., * ) ) and graphically compare how the posterior distribution P (Iobserved Itotal for observed illnesses changes from a baseline number of total illnesses to a reduced number of total illnesses. Solving Eq. (1) requires the application of Eq. (2) to the range of Iobserved values using α = 33.27 and θ = 0.88 estimated previously for Salmonella under-diagnosis (Ebel et al., 2012).
⎜
⎟
where PU is the size of the surveillance population in the ultimate year of adjustment and Pt is the size for the year considered. For the Presumptive approach, the mean and standard deviation of the observed illnesses random variable is estimated directly from these adjusted observed illnesses. For the Bayesian approach, the average of the 5
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
(μ = 6930 × (1 − DE ), σ = 0.06 × 6930 × (1 − DE )) . The resultant distributions are shown in Fig. 2. There is substantial overlap between the baseline distribution and the 10% reduction scenario distribution, but there is little overlap with the baseline distribution for the larger reductions. For the Bayesian approach that assumes a uniform prior, the resulting posterior distribution for the baseline observed illnesses overlaps substantially with the posterior for the scenario representing a 10% reduction in domestic foodborne illnesses (Fig. 3). The degree of overlap with the baseline posterior decreases as the magnitude of illness reduction increases, such that there is less overlap between the baseline and 50% reduction distributions. Although a uniform prior distribution avoids introducing bias into the estimated posterior distribution in Eq. (1), it ignores historic evidence about how observed illnesses vary across time. A somewhat informed prior was developed to reflect uncertainty about future changes to observed illnesses while maintaining a central tendency similar to the value estimated from the FoodNet data for Salmonella. In this case, a NegativeBinomial (r = 35.28, β = 196.45) was used (Fig. 4). Use of an informed prior in Eq. (1) generates slightly different posterior distributions for the baseline and reductions scenarios (Fig. 4). As expected, there is a greater degree of overlap between the baseline distribution and the three reduction distributions than seen with the uniform prior example. Use of an informed prior reduces both the magnitude of the apparent reduction in observed illnesses and the variance of each distribution. Power increases as the magnitude of the assumed proportional reduction in illnesses increases (Fig. 5). For the Presumptive approach, the assumed proportional reduction is with respect to observed illnesses. The power curve in this case suggests that power increases rapidly with the proportional reduction; such that the power is 0.15, 0.87 and 1.0 for 10%, 30% and 50% proportional reductions, respectively. For the Bayesian approach, the proportional reduction is with respect to total domestic foodborne illnesses. The power curves for the uniform and informed prior scenarios suggest that power increases more slowly with the proportional reduction in illnesses (Fig. 5). Using a uniform prior, the power curve is 0.04, 0.13 and 0.34 for 10%, 30% and 50% proportional reductions, respectively. Using an informed prior, the power is 0.04, 0.11 and 0.26 for 10%, 30% and 50% proportional reductions, respectively. If a reduction were to persist across a two-year period, the power to discern a difference increases substantially relative to just one year of reduction (Fig. 6). The Bayesian approach (using the uniform prior as an example) affords a direct updating of the evidence of a consistent reduction in total domestic foodborne illnesses. Power continues to improve as the reduction persists across a three to four year period. For a 10% proportional reduction in total illnesses, the power for the one through four year curves is 0.04, 0.06, 0.08 and 0.09. For a 30% proportional reduction in total illnesses, the power for the one through four year curves is 0.13, 0.27, 0.40 and 0.53. For a 50% proportional reduction in total illnesses, the power for the one through four year curves is 0.34, 0.69, 0.88 and 0.96.
Fig. 6. Power curves based on one to four years of proportional reductions of domestic foodborne illnesses are shown. These curves are generated using a Bayesian approach with the one-year curve assuming a uniform prior for observed illnesses. The 2–4 year curves assume the prior for observed illnesses is the preceding year's posterior distribution.
adjusted observed illnesses is used to determine a baseline number of total illnesses for use in the model;
ρ Itotal = Iobserved − adjusted , κ where ρ = 29.3 is the mean under-diagnosis multiplier for Salmonella (Ebel et al., 2012) and κ = 0.15 is the catchment area for FoodNet since 2004 (Scallan and Mahon, 2012). 3. Results FoodNet data provide evidence of a correspondence between observed Salmonella illnesses and the size of the catchment population until about 2009 (Fig. 1). In 2010, Salmonella cases increased substantially then decreased somewhat through 2013 before increasing slightly through 2015. For illustrative purposes, we estimate the average observed illnesses – adjusted for the surveillance population size – for the period 1996 – 2009.5 We chose this period because the relationship between illness counts and surveillance population was reasonably stable. The average observed illnesses, in terms of 2009s surveillance population, was 6930 (standard deviation =395). The coefficient of variation of 6% suggests the distribution of adjusted observed illnesses is over-dispersed relative to a Poisson distribution, which would have a standard deviation of 83. For the Bayesian approach, this average observed illnesses implies that our baseline total illnesses are 1,353,660. For the Presumptive approach, we assume Iobserved ∼ Normal (μ = 6930, σ = 0.06 × 6930) (i.e., we use the rounded coefficient of variation estimated from the data to model the standard deviation of the Normal distribution). For the reduced * observed illnesses scenarios, we assume Iobserved ∼ Normal 5
4. Discussion FoodNet rivals or exceeds any national surveillance system for foodborne disease. This surveillance system provides important data concerning the occurrence of foodborne disease, trends across time, emergence of new pathogenic strains and severity of foodborne illnesses. It has detected substantial changes in illness rates across time since its inception. For example, E. coli O157: H7 illnesses (hospitalizations, deaths) per capita decreased substantially between 1996 and 2008. Similarly, FoodNet has detected a substantial decrease in Listeria monocytogenes illnesses between 1996 and 2002 (Powell, 2016). Our analysis, however, addresses the question “how confident should we be in detecting changes in the counts of illnesses for a
https://www.cdc.gov/foodnet/reports/data/xls/table2a.xlsx.
6
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
Scallan et al. (2011), estimation of under-diagnosis is based on “variations in medical care seeking, specimen submission, laboratory testing, and test sensitivity” generated from surveys. It is possible that under-diagnosis might vary across cases and/or regions and/or seasons and/or years for various reasons, including cultural differences. If this is true, then the observed illnesses for a baseline and reduction scenario are truly independent random samples of their respective total illnesses. On the other hand, it is possible that under-diagnosis is relatively invariant to case factors such that a true, but uncertain, value is appropriate to apply in our calculations. If this were known to be true, then we would expect correlated posterior distributions and greater power than estimated here. Our findings may be most applicable to occasions when annual patterns of observed illnesses from a surveillance system suggest little change, or possible increases, despite intentional efforts to reduce the risk of illness from particular food sources. As we suggested previously (Williams and Ebel, 2012), the limited power of FoodNet to detect modest changes should spur food safety regulators to monitor data that directly measures the effect of policies on exposure of consumers to pathogens. For example, FSIS collects data to monitor the contamination status of many meat, poultry and egg products on an ongoing basis. Such data, though not directly measuring the public health effects of a policy, potentially provide regulators with more powerful information to assess progress towards reducing foodborne illnesses than relying on evidence of change from public health surveillance.
pathogen from one year to the next?” This question should be asked by stakeholders (e.g., consumers, food producers, government analysts) who monitor annual FoodNet data to assess whether the effectiveness of some food safety policy is apparent. The answer is not trivial or easily determined. The Presumptive approach examines power for direct reductions in observed illnesses. Nevertheless, that approach ignores the fact that food safety risk management normally projects the expected effects of a policy on total domestic foodborne illnesses. In reality, observed illnesses are a subsample of total illnesses and an accurate assessment of power from monitoring observed illnesses cannot ignore the influence of under-diagnosis that relates total to observed illnesses. The Bayesian approach examines power by accounting for how a change in total domestic foodborne illnesses projects to a change in observed illnesses. Although the power for the Presumptive approach is substantially greater than the Bayesian approach, the Bayesian approach is a better description of the variability and uncertainty associated with monitoring the public health effect of a policy. Both approaches suggest limited power to discriminate a modest change in observed illnesses that results from a policy. For example, a projected 10% reduction in total illnesses has a power of 0.04 using the Bayesian approach and 0.15 using the Presumptive approach. Therefore, it is not appropriate to assume that such a reduction will be detected after just one year of monitoring. Instead, multiple years of monitoring are necessary before a modest reduction could be concluded from the surveillance evidence. Food safety policy that is directed at specific product-pathogen pairs may generate modest expectations for reducing domestic foodborne illness numbers. In 2011, FSIS announced new Salmonella performance standards for chicken and turkey carcasses intended to reduce annual Salmonella illnesses associated with consumption of those products by 12% and 1%, respectively (FSIS, 2011). More recently, FSIS announced new Salmonella performance standards for chicken parts and ground chicken and turkey (FSIS, 2015). These standards were expected to reduce annual human Salmonella illnesses associated with these products by 25%. Nevertheless, poultry is estimated to be responsible for only 19% of all Salmonella illnesses (Painter et al., 2013). Therefore, if fully effective, both of these policies are expected to reduce total Salmonella illnesses by only about 5%. Given our results, it would take more than a decade to determine that implementation of these policies was effective if the foodborne illness environment remained unchanged other than the influence of these policies. Our analysis may overstate power because it assumes a static environment where the only factor influencing total illnesses is the food safety policy under consideration. In reality, total illnesses from a pathogen – and the foods contributing to those illnesses – are dynamic. For example, an intended reduction of illnesses associated with one commodity can occur simultaneously with an independent upsurge in illnesses associated with another less controlled commodity, or the emergence of illnesses associated with a new commodity heretofore unrecognized as a food source for the pathogen. If ignored, annual increases in the surveillance population can generate increased observed illnesses despite modest reductions from a particular commodity. Therefore, our limited understanding of the dynamics of foodborne illness in the population, beyond those captured by the under-diagnosis probability, might mask the effectiveness of a policy. For example, changes in demographics based on race and cultural food practices may influence frequency of foodborne illnesses, especially in geographic areas where certain cultural groups reside. The Bayesian approach to power calculation assumes independence between the baseline posterior distribution and the posterior distributions for reduced total illnesses. Although this assumption might be tenuous, determining the degree of dependency between these distributions is not possible given the limited information available. At this time, under-diagnosis is a poorly understood phenomenon. From
Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.mran.2017.10.001. References CDC, 2017. Incidence and trends of infection with pathogens transmitted commonly through food and the effect of increasing use of culture-independent diagnostic tests on surveillance—foodborne diseases active surveillance network, 10 U.S. Sites, 2013–2016. MMWR Morbid. Mortal. Week. Rep. Collard, J., Bertrand, S., Dierick, K., Godard, C., Wildemauwe, C., Vermeersch, K., Duculot, J., Van Immerseel, F., Pasmans, F., Imberechts, H., 2008. Drastic decrease of Salmonella Enteritidis isolated from humans in Belgium in 2005, shift in phage types and influence on foodborne outbreaks. Epidemiol. Infect. 136, 771–781. Ebel, E.D., Williams, M.S., Schlosser, W.D., 2012. Parametric distributions of underdiagnosis parameters used to estimate annual burden of Illness for five foodborne pathogens. J. Food Protect. 75, 775–778. FSIS, 2011. New Performance Standards for Salmonella and Campylobacter in Young Chicken and Turkey Slaughter Establishments: Response to Comments and Implementation Schedule. U. S. Department of Agriculture, Washington, D.C. FSIS, 2015. Public Health Effects of Raw Chicken Parts and Comminuted Chicken and Poultry Performance Standards. United States Department of Agriculture, Washington, D.C. Herikstad, H., Motarjemi, Y., Tauxe, R.V., 2002. Salmonella surveillance: a global survey of public health serotyping. Epidemiol. Infect. 129, 1–8. Klugman, S.A., Panjer, H.H., Willmot, G.E., 2012. Loss Models: From Data to Decisions, fourth ed. John Wiley and Sons, New York. Painter, J.A., Hoekstra, R.M., Ayers, T., Tauxe, R.V., Braden, C.R., Angulo, F.J., Griffin, P.M., 2013. Attribution of foodborne illnesses, hospitalizations, and deaths to food commodities, United States, 1998–2008. Emerg. Infect. Dis. 19, 407–415. Powell, M.E., 2016. Trends in reported foodborne illness in the United States: 1996–2013. Risk Anal. 36, 1589–1598. Scallan, E., Hoekstra, R.M., Angulo, F.J., Tauxe, R.V., Widdowson, M.-A., Roy, S.L., Jones, J.L., Griffin, P.M., 2011. Foodborne illness acquired in the United States—major pathogens. Emerg. Infect. Dis. 17, 7–15. Scallan, E., Mahon, B.E., 2012. Foodborne diseases active surveillance network (FoodNet) in 2012: a foundation for food safety in the United States. Clinic. Infect. Dis. 54, S381–S384. Weiss, N.A., 2015. Introductory Statistics. Pearson. Williams, M.S., Ebel, E.D., 2012. Estimating changes in public health following implementation of hazard analysis and critical control point in the United States broiler slaughter industry. Foodborne Pathogens Disease 12, 59–67. Williams, M.S., Ebel, E.D., Vose, D., 2011. Framework for microbial food-safety risk assessments amenable to Bayesian modeling. Risk Anal. 31, 548–565. Yeni, F., Acar, S., Soyer, Y., Alpas, H., 2017. How can we improve foodborne disease surveillance systems: a comparison through EU and US systems. Food Rev. Int. 33, 406–423.
7
Contents lists available at ScienceDirect
Microbial Risk Analysis journal homepage: www.elsevier.com/locate/mran
Estimating the Type II error of detecting changes in foodborne illnesses via public health surveillance
MARK
Eric D. Ebel*, Michael S. Williams*, Wayne D. Schlosser Risk Assessment Division Office of Public Health Science Food Safety Inspection Service, USDA 2150 Centre Avenue, Building D Fort Collins, Colorado 80526, U.S.
A R T I C L E I N F O
A B S T R A C T
Keywords: Passive surveillance Power Underreporting fraction Attribution
Many countries operate public health surveillance systems to monitor foodborne disease occurrence. The illness counts for pathogens of interest are monitored across time to assess if changes have occurred in case rates or the overall illness burden. Common to all of these systems is that only a fraction of all foodborne illnesses are reported and the relationship between observed and the total number of illnesses is uncertain. Food-safety policies intend to affect both the total and observed number of illnesses. Ideally, the surveillance system would be sufficiently sensitive to detect these changes, but the statistical power of these systems is generally not well understood. This study proposes two approaches for estimating the power of a foodborne illness surveillance system. These methods are then applied to assess the power of detecting specified changes in the total number of illnesses in an existing surveillance system. The findings suggest that the power of a national foodborne disease surveillance system to detect modest annual reductions in Salmonella illnesses may be limited. For example, a naïve presumptive model that assumes observed illnesses are reduced directly by 10% predicts that the power to discern this difference, from one year to the next, is 0.21. A Bayesian model, that accounts for uncertainty in projecting changes in total illnesses to the number of observed illnesses, predicts that the power to detect a true 10% reduction is 0.04 (i.e., 4% confidence of detecting this magnitude of reduction or 96% chance of a Type II error) one year after the change has taken effect. Although the power of a national surveillance system increases as the magnitude of reductions increases – or as the number of years the reduction is maintained increases – the Bayesian model demonstrates that power is less than 0.5 for reductions up to 50% for one year and is less than 0.53 for reductions of 30% that are maintained for four years. The limited power to detect intended changes in total annual illnesses of national public health surveillance systems may highlight the need for regulators to also monitor food contamination evidence to gauge progress towards achieving intended policy effects.
1. Introduction More than 75 countries operate surveillance systems to monitor foodborne disease occurrence across time (Herikstad et al., 2002; Scallan and Mahon, 2012; Yeni et al., 2017). Large changes in confirmed illness counts for a specific pathogen indicate the effectiveness of government and industry efforts to improve food safety (Collard et al., 2008). Distinguishing smaller changes in overall food safety may be problematic if the public health surveillance system's power to detect and report changes in foodborne disease occurrence is limited. A conundrum for the food-safety community is that the power of a surveillance system to detect reductions in the national occurrence of foodborne illness, associated with a particular product-pathogen pair, can only be lessened as control programs succeed in reducing observable illnesses (Williams and Ebel, 2012). This observation is
*
based on the realization that it becomes more difficult to distinguish changes in observed illnesses as the number of observed illnesses becomes smaller. For example, assuming observed illnesses are Poisson distributed (with the mean equal to the variance), then it is expected that the signal from a 30% reduction in observed illnesses is readily distinguished when the baseline average illness count is 1000 per year (approximate 95% confidence limits of 938 – 1062) but that signal is lost in the random variability if the baseline average illness count is 10 per year (approximate 95% confidence limits of 4 – 16). In this example, a 30% reduction of 1000 observed illnesses should result in an average of 700 observed illnesses; this is clearly outside the range of random Poisson variability for the baseline. In contrast, a 30% reduction of 10 observed illnesses results in an average of 7 observed illnesses; that is within the variability of the baseline. Although the power to discriminate a change in illnesses depends on
Corresponding authors. E-mail address: [email protected] (M.S. Williams).
http://dx.doi.org/10.1016/j.mran.2017.10.001 Received 28 August 2017; Received in revised form 23 October 2017; Accepted 30 October 2017 Available online 31 October 2017 2352-3522/ Published by Elsevier B.V.
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
For the purposes of illustration, this paper calculates the power of the FoodNet surveillance system to detect a true change in total illnesses by only considering the effect of under-diagnosing the illnesses that occur in the population. Therefore, our analysis ignores other factors that might influence the relationship between total illnesses from a foodborne pathogen and the number of illnesses observed in FoodNet for that pathogen. Our illustration considers Salmonella illnesses and the effects of hypothetical public policies intended to reduce the national occurrence of domestic foodborne illnesses associated with this pathogen. Because Salmonella is one of the most frequently reported bacterial illnesses in FoodNet, the power to detect its differences should be larger than the power for less-commonly reported pathogens. We use two methods for estimating the power of the surveillance system. The first presumptive method assumes that a reduction in total illnesses results in a direct reduction in observed illnesses. The other approach uses Bayesian reasoning (and two alternative prior distributions) to model the effect of a change in total illnesses on the observed illnesses. Additionally, the Bayesian approach supports consideration of the power for determining a reduction given multiple years of observation.
the baseline number of illnesses against which to compare the change, it also depends on the process of projecting an actual change in total illnesses in the population to a change in illnesses observed in a sampled population, like the Foodborne Diseases Active Surveillance Network (FoodNet) surveillance system (Scallan and Mahon, 2012). This projection, from a true but unobservable total annual case count to an observed annual case count, is a source of uncertainty not considered by the previous Poisson distribution argument. Public health agencies need to track progress in their programs. The U.S. Department of Agriculture's Food Safety and Inspection Service uses FoodNet reports to monitor human illnesses on an annual basis.1 Nevertheless, despite evidence of reductions in pathogen occurrence on some raw products it regulates, the success of FSIS programs are difficult to fully support if the public health surveillance evidence suggests no change, or possibly an increase, in occurrence of human illness (Powell, 2016). Given recent work by the Department of Health and Human Services’ Centers for Disease Control and Prevention (CDC), it is possible to estimate total illnesses associated with a pathogen from the annual reported FoodNet illness counts (Ebel et al., 2012; Scallan et al., 2011). This direction of inference, from a limited number of FoodNet cases, reported by 10 sites in the United States, to the entire U.S. population is accomplished by accounting for the catchment area of the FoodNet sites and pathogen-specific multipliers that describe the number of unreported, domestically-acquired foodborne cases for each observed illness reported to FoodNet. Observed illnesses from a pathogen are a subset of its true total number of illnesses. Consequently, total illnesses can be estimated from the subset of reported illnesses, although this estimation may include uncertainty. Uncertainties inherent in a national surveillance system also will influence the observed illnesses. For example, uncertain effects of changes to a surveillance system (e.g., changes in testing methodology and health care systems (CDC, 2017)) and growth in population size may affect the number of observed illnesses. Also, disease occurrence may cluster spatially or temporally in ways that bias a surveillance system that was designed under the assumption of randomly distributed illnesses in space and time. This can be confounded by the different levels of consumer education on safe food handling practices across different geographic regions. In the context of hypothesis testing, power describes the confidence of correctly rejecting the null hypothesis – that total illnesses are unchanged – given that an alternative hypothesis (that total illnesses are reduced) is true. The illnesses observed for a given year is reflected by the surveillance system. Absent a policy that intends to reduce total illnesses, it is reasonable to hypothesize that the observed illnesses will be similar from one year to the next. If a policy is enacted to reduce total illnesses by some expected fraction, 2 then regulators will monitor the observed illnesses in the following years to assess the policy's performance. If observed illnesses are unchanged or increase following the policy's enactment, then regulators may wonder if such a result indicates policy under-performance, or insufficient power to detect the expected magnitude of reduction (i.e., Type II error). Similarly, if observed illnesses decline, regulators may wonder if the effect is indicative of a true change in illnesses or a random fluctuation (i.e., Type I error). At least part of the answer to such policy analysis questions involves understanding the power of the surveillance system to detect changes.
2. Methods The number of illnesses reported by a surveillance system for a pathogen (Iobserved) will differ from the total number of illnesses for the pathogen (Itotal) (Williams et al., 2011). The FoodNet surveillance system identifies human cases associated with a pathogen, but it is subject to underreporting bias (Scallan et al., 2011). Two factors that relate the total number of illnesses for a pathogen to the number of illnesses observed by FoodNet are the number of unreported illnesses(ρ) for each observed FoodNet illness (Ebel et al., 2012) and the fraction of the total population that exists in the catchment area of the surveillance system (κ) (Scallan and Mahon, 2012). The inverse of ρ is the probability that an illness in the population is diagnosed and reported. These two factors modify the total number of foodborne illnesses to estimate the number of observed illnesses whose etiology is the pathogen, so that
Iobserved =
κ Itotal. ρ
2.1. Presumptive approach If we ignore this relationship and simply consider Iobserved as a random variable, then we can calculate the power of detecting some intended proportional reduction using a Z statistic (Weiss, 2015). We refer to this approach as “Presumptive” because it assumes a direct correspondence between an expected reduction in total illnesses and the reduction in observed illnesses. In this case,
crit + DE × δ × ϕ × E [Iobserved] ⎞ ⎛ power = p ⎜Z < * V [Iobserved ] + V [Iobserved] ⎟⎠ ⎝ where Z ∼ Normal(0, 1), E [Iobserved] and V[Iobserved] are the mean and * variance of Iobserved, Iobserved is the observed illnesses reduced by some proportional design effect (DE) that is modified by δ (the fraction of illnesses domestically-acquired) and ϕ (the fraction of domestic illnesses that are foodborne3); and crit is a critical value determined as − 1.96 2V [Iobserved] . E [Iobserved] and V[Iobserved] are estimated from FoodNet evidence. In this development, the design effect should reflect the expected reduction from a policy on observed illnesses. Because the effect of a
1
https://www.cdc.gov/foodnet/reports/index.html. Total illnesses represent those that are foodborne and domestically-acquired, as well as those that stem from non-food sources and/or international exposures. Furthermore, domestically-acquired foodborne illnesses are attributed to particular foods that are regulated by different governmental agencies. Usually, a food safety policy is enacted to reduce domestically-acquired foodborne illnesses associated with a specific product-pathogen pairing (e.g., chicken-Salmonella). Consequently, the reduction in total illnesses for the pathogen will be less than the expected reduction in illnesses attributed to the specific product. 2
3 Scallan et al. (2011) estimate that 89% of Salmonella illnesses are domestically-acquired and 94% of these are foodborne.
2
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
60
8,000 50
Surveillance populaon (millions)
Annual Salmonella cases reported by FoodNet
9,000
7,000 6,000
40
5,000
30 4,000 3,000
20
FoodNet cases
Surveillance populaon
2,000
10 1,000 0
2015
2014
2013
2012
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
0
Fig. 1. Annual Salmonella cases reported to FoodNet, 1996 – 2015 and the size of the surveillance population are shown.
illnesses (Williams et al., 2011). Therefore, the annual total number of salmonellosis cases is modeled as
typical domestic policy will be limited to reducing domestic foodborne illnesses, and not influence illnesses that result from non-food or nondomestic exposures, the resultant observed illnesses should account for this discrepancy. Therefore, the design effect in the power calculation must be reduced by the fraction of illnesses expected to be domestic and foodborne. For example, if a policy is expected to reduce domestic foodborne illnesses by 20%, and we assume that 89% of illnesses are domestic and 94% of these are foodborne, then the reduction in observed illnesses is actually 16.7% (20% × 89% × 94%). Because the Presumptive approach assumes a direct reduction of observed illnesses – and ignores the uncertainty of projecting a reduction in total illnesses to observed illnesses – it is expected that its power will overestimate the true power and be greater than the Bayesian approach described next. Nevertheless, the Presumptive results for power are presented as an upper bound on the actual power and for comparison with those of the Bayesian approach.
Itotal ∼ Poisson (Nservings P (ill exp) P (exp)) = Poisson (λtotal ) ρ
where λtotal is estimated from κ Iobserved . An adjustment to this model is needed when both illness counts include non-foodborne and/or non-domestically acquired disease. The latter is particularly important if food safety policies can only be expected to influence domestic foodborne exposures. Although food safety analysts can estimate total annual illnesses conditioned on the observed illnesses for a given year (i.e., P (Itotal|Iobserved)), interest lies in examining the reverse conditional probability so that we might assess the effects of hypothetical changes in total illnesses on the observed illnesses when modeling potential policy options. Using Bayes Theorem, we can represent the solution as
P (Iobserved Itotal ) = 2.2. Bayesian approach
P (Itotal Iobserved ) × P (Iobserved ) P (Itotal )
(1)
This relationship suggests that P(Iobserved|Itotal)∝P(Itotal|Iobserved) × P (Iobserved). To evaluate the likelihood function on the right-hand side of this ρ proportionality, note that Itotal ∼ Poisson κ Iobserved . If ρ ∼ gamma(α, θ)4
This approach intends to estimate posterior distributions for observed illnesses conditioned on total annual illness counts for baseline and reduction scenarios. The power calculation is similar to that described for the Presumptive approach. For a foodborne pathogen like Salmonella, the total annual illness count is a parameter that results from the aggregate outcomes of consuming millions of servings of different foods. Generally, we can represent this relationship as
(
)
(
)
I
θ . while κ and Iobserved are scalar, then λtotal ∼ gamma α, observed κ A Poisson distributed random variable with a gamma mixture distribution generates a negative binomial distribution (i.e.,
(
Itotal Iobserved ∼ NegativeBinomial r = α, β =
Itotal = Nservings P (ill exp) P (exp)
2012) so the likelihood function is
where Nservings is the number of food servings consumed annually, P (exp) is the frequency of exposure to contaminated servings and P(ill|exp) is the probability that a contaminated serving results in human illness. This probability summarizes the variable effects of factors such as partitioning and mixing of food ingredients, storage, handling, cooking, and pathogenicity of different strains of the pathogen. Given a large number of servings, probability theory suggests that the Poisson process reasonably models the number of annual sporadic
Itotal + α − 1⎞ ⎛ P (Itotal Iobserved ) = ⎜⎛ ⎟ ⎜ Itotal ⎝ ⎠⎝ 1 +
Iobserved θ κ
))
Itotal Iobserved θ ⎞ ⎛ κ Iobserved ⎟ ⎜1 θ κ ⎝ ⎠
(Klugman et al.,
α
⎞ I ⎟ . + observed θ κ ⎠ 1
(2)
4
3
f (ρ) =
ρ ρ α − ⎛ ⎞ e θ ⎝θ⎠ . ρΓ(α )
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
Fig. 2. Distributions of annual observed Salmonella illnesses are shown for the baseline and alternative scenarios where, using the Presumptive approach, observed illnesses are reduced by design effects of 10% to 50%.
Fig. 3. Posterior distributions of annual observed Salmonella illnesses, using the Bayesian approach, are shown for a baseline scenario and alternative scenarios where the assumed total domestic foodborne illnesses is reduced by design effects of 10% to 50%. The prior distribution for each curve is assumed to be uniform across the range of possible observed illnesses.
Fig. 4. Posterior distributions of annual observed Salmonella illnesses, using the Bayesian approach, are shown for a baseline scenario and alternative scenarios where the assumed total domestic foodborne illnesses is reduced by design effects 10% to 50%. The prior distribution for each curve is shown as the shorter and wider solid line in this graph.
4
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
0.2
0.4
Power
0.6
0.8
1.0
Fig. 5. The power to detect proportional reductions in annual observed, or total domestic foodborne, human illnesses by monitoring observed illnesses is shown. If a design effect is applied directly to observed illnesses (the Presumptive approach), then the power tends to rapidly increase with the size of the proportional reduction. In contrast, if the design effect is applicable to total domestic foodborne illnesses (the Bayesian approach), then power increases more slowly. The power for the Bayesian approach with an informed prior is generally least of the three examples.
0.0
Presumptive approach Bayes uniform prior Bayes informed prior
0.0
0.2 0.4 0.6 0.8 Proportional reduction in annual foodborne, domestic illnesses
1.0
Assuming independence between the baseline and reduction distributions, power is calculated using the Z-statistic with the means and variances determined from the moments of the baseline and reduction posterior distributions. The Bayesian approach is amenable to assessing the power to discern a difference if it persists for multiple years. For example, we can estimate power for a reduction that persists for two consecutive years. We estimate the second year posterior distributions for the baseline and reduction scenarios by using prior distributions that are the posterior distributions from the first year. In this manner, we are updating the second year posterior distributions to account for two years of evidence that total illnesses have either remained similar or have been reduced. Similarly, we continue this process for three and four years. The power calculations are based on the moments of the updated posterior distributions.
An expression for the prior probability in Eq. (1) (i.e., P(Iobserved)) is problematic because we are interested in examining how the posterior probability (i.e., P(Iobserved|Itotal)) changes for alternative values of Itotal. A prior distribution that reflects past observed illness values might unnecessarily constrain the posterior if the magnitude of change in total illnesses is large. Therefore, our analysis examines a uniform prior for P (Iobserved) across a plausible range of observed illnesses values (i.e., 1 to 15,000 in our example). We further examine the effects of an informed prior distribution based on analysis of the historic FoodNet data. This alternative prior is developed as a negative binomial distribution intended to capture variability in observed illnesses that arbitrarily ranges from 33% below (2.5th percentile of the distribution) to 33% above (97.5th percentile) its mean value. This range was chosen to illustrate a noticeable effect on the resulting power relative to the uniform prior assumption. To evaluate the effect of an assumed change in total illnesses on the observed illnesses, we first solve Eq. (1) for the baseline number of total illnesses. Next, we consider alternative design effects (DE), ranging from 0 to 1, that progressively reduce the total number of illnesses * , is calculated below the baseline. This reduced number of illnesses, Itotal as
2.3. FoodNet Salmonella data Annual FoodNet data are summarized from the program's inception in 1996 through 2015. Each year's observed Salmonella illnesses (Iobserved, t) are adjusted, for the size of the surveillance population, to generate an ultimate year comparison using the following calculation;
* = Itotal × (1 − DE × δ × ϕ) Itotal
P Iobserved − adjusted, t = Iobserved, t ⎛ U ⎞ ⎝ Pt ⎠
We solve Eq. (1) again for the reduction scenario (i.e., * ) ) and graphically compare how the posterior distribution P (Iobserved Itotal for observed illnesses changes from a baseline number of total illnesses to a reduced number of total illnesses. Solving Eq. (1) requires the application of Eq. (2) to the range of Iobserved values using α = 33.27 and θ = 0.88 estimated previously for Salmonella under-diagnosis (Ebel et al., 2012).
⎜
⎟
where PU is the size of the surveillance population in the ultimate year of adjustment and Pt is the size for the year considered. For the Presumptive approach, the mean and standard deviation of the observed illnesses random variable is estimated directly from these adjusted observed illnesses. For the Bayesian approach, the average of the 5
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
(μ = 6930 × (1 − DE ), σ = 0.06 × 6930 × (1 − DE )) . The resultant distributions are shown in Fig. 2. There is substantial overlap between the baseline distribution and the 10% reduction scenario distribution, but there is little overlap with the baseline distribution for the larger reductions. For the Bayesian approach that assumes a uniform prior, the resulting posterior distribution for the baseline observed illnesses overlaps substantially with the posterior for the scenario representing a 10% reduction in domestic foodborne illnesses (Fig. 3). The degree of overlap with the baseline posterior decreases as the magnitude of illness reduction increases, such that there is less overlap between the baseline and 50% reduction distributions. Although a uniform prior distribution avoids introducing bias into the estimated posterior distribution in Eq. (1), it ignores historic evidence about how observed illnesses vary across time. A somewhat informed prior was developed to reflect uncertainty about future changes to observed illnesses while maintaining a central tendency similar to the value estimated from the FoodNet data for Salmonella. In this case, a NegativeBinomial (r = 35.28, β = 196.45) was used (Fig. 4). Use of an informed prior in Eq. (1) generates slightly different posterior distributions for the baseline and reductions scenarios (Fig. 4). As expected, there is a greater degree of overlap between the baseline distribution and the three reduction distributions than seen with the uniform prior example. Use of an informed prior reduces both the magnitude of the apparent reduction in observed illnesses and the variance of each distribution. Power increases as the magnitude of the assumed proportional reduction in illnesses increases (Fig. 5). For the Presumptive approach, the assumed proportional reduction is with respect to observed illnesses. The power curve in this case suggests that power increases rapidly with the proportional reduction; such that the power is 0.15, 0.87 and 1.0 for 10%, 30% and 50% proportional reductions, respectively. For the Bayesian approach, the proportional reduction is with respect to total domestic foodborne illnesses. The power curves for the uniform and informed prior scenarios suggest that power increases more slowly with the proportional reduction in illnesses (Fig. 5). Using a uniform prior, the power curve is 0.04, 0.13 and 0.34 for 10%, 30% and 50% proportional reductions, respectively. Using an informed prior, the power is 0.04, 0.11 and 0.26 for 10%, 30% and 50% proportional reductions, respectively. If a reduction were to persist across a two-year period, the power to discern a difference increases substantially relative to just one year of reduction (Fig. 6). The Bayesian approach (using the uniform prior as an example) affords a direct updating of the evidence of a consistent reduction in total domestic foodborne illnesses. Power continues to improve as the reduction persists across a three to four year period. For a 10% proportional reduction in total illnesses, the power for the one through four year curves is 0.04, 0.06, 0.08 and 0.09. For a 30% proportional reduction in total illnesses, the power for the one through four year curves is 0.13, 0.27, 0.40 and 0.53. For a 50% proportional reduction in total illnesses, the power for the one through four year curves is 0.34, 0.69, 0.88 and 0.96.
Fig. 6. Power curves based on one to four years of proportional reductions of domestic foodborne illnesses are shown. These curves are generated using a Bayesian approach with the one-year curve assuming a uniform prior for observed illnesses. The 2–4 year curves assume the prior for observed illnesses is the preceding year's posterior distribution.
adjusted observed illnesses is used to determine a baseline number of total illnesses for use in the model;
ρ Itotal = Iobserved − adjusted , κ where ρ = 29.3 is the mean under-diagnosis multiplier for Salmonella (Ebel et al., 2012) and κ = 0.15 is the catchment area for FoodNet since 2004 (Scallan and Mahon, 2012). 3. Results FoodNet data provide evidence of a correspondence between observed Salmonella illnesses and the size of the catchment population until about 2009 (Fig. 1). In 2010, Salmonella cases increased substantially then decreased somewhat through 2013 before increasing slightly through 2015. For illustrative purposes, we estimate the average observed illnesses – adjusted for the surveillance population size – for the period 1996 – 2009.5 We chose this period because the relationship between illness counts and surveillance population was reasonably stable. The average observed illnesses, in terms of 2009s surveillance population, was 6930 (standard deviation =395). The coefficient of variation of 6% suggests the distribution of adjusted observed illnesses is over-dispersed relative to a Poisson distribution, which would have a standard deviation of 83. For the Bayesian approach, this average observed illnesses implies that our baseline total illnesses are 1,353,660. For the Presumptive approach, we assume Iobserved ∼ Normal (μ = 6930, σ = 0.06 × 6930) (i.e., we use the rounded coefficient of variation estimated from the data to model the standard deviation of the Normal distribution). For the reduced * observed illnesses scenarios, we assume Iobserved ∼ Normal 5
4. Discussion FoodNet rivals or exceeds any national surveillance system for foodborne disease. This surveillance system provides important data concerning the occurrence of foodborne disease, trends across time, emergence of new pathogenic strains and severity of foodborne illnesses. It has detected substantial changes in illness rates across time since its inception. For example, E. coli O157: H7 illnesses (hospitalizations, deaths) per capita decreased substantially between 1996 and 2008. Similarly, FoodNet has detected a substantial decrease in Listeria monocytogenes illnesses between 1996 and 2002 (Powell, 2016). Our analysis, however, addresses the question “how confident should we be in detecting changes in the counts of illnesses for a
https://www.cdc.gov/foodnet/reports/data/xls/table2a.xlsx.
6
Microbial Risk Analysis 7 (2017) 1–7
E.D. Ebel et al.
Scallan et al. (2011), estimation of under-diagnosis is based on “variations in medical care seeking, specimen submission, laboratory testing, and test sensitivity” generated from surveys. It is possible that under-diagnosis might vary across cases and/or regions and/or seasons and/or years for various reasons, including cultural differences. If this is true, then the observed illnesses for a baseline and reduction scenario are truly independent random samples of their respective total illnesses. On the other hand, it is possible that under-diagnosis is relatively invariant to case factors such that a true, but uncertain, value is appropriate to apply in our calculations. If this were known to be true, then we would expect correlated posterior distributions and greater power than estimated here. Our findings may be most applicable to occasions when annual patterns of observed illnesses from a surveillance system suggest little change, or possible increases, despite intentional efforts to reduce the risk of illness from particular food sources. As we suggested previously (Williams and Ebel, 2012), the limited power of FoodNet to detect modest changes should spur food safety regulators to monitor data that directly measures the effect of policies on exposure of consumers to pathogens. For example, FSIS collects data to monitor the contamination status of many meat, poultry and egg products on an ongoing basis. Such data, though not directly measuring the public health effects of a policy, potentially provide regulators with more powerful information to assess progress towards reducing foodborne illnesses than relying on evidence of change from public health surveillance.
pathogen from one year to the next?” This question should be asked by stakeholders (e.g., consumers, food producers, government analysts) who monitor annual FoodNet data to assess whether the effectiveness of some food safety policy is apparent. The answer is not trivial or easily determined. The Presumptive approach examines power for direct reductions in observed illnesses. Nevertheless, that approach ignores the fact that food safety risk management normally projects the expected effects of a policy on total domestic foodborne illnesses. In reality, observed illnesses are a subsample of total illnesses and an accurate assessment of power from monitoring observed illnesses cannot ignore the influence of under-diagnosis that relates total to observed illnesses. The Bayesian approach examines power by accounting for how a change in total domestic foodborne illnesses projects to a change in observed illnesses. Although the power for the Presumptive approach is substantially greater than the Bayesian approach, the Bayesian approach is a better description of the variability and uncertainty associated with monitoring the public health effect of a policy. Both approaches suggest limited power to discriminate a modest change in observed illnesses that results from a policy. For example, a projected 10% reduction in total illnesses has a power of 0.04 using the Bayesian approach and 0.15 using the Presumptive approach. Therefore, it is not appropriate to assume that such a reduction will be detected after just one year of monitoring. Instead, multiple years of monitoring are necessary before a modest reduction could be concluded from the surveillance evidence. Food safety policy that is directed at specific product-pathogen pairs may generate modest expectations for reducing domestic foodborne illness numbers. In 2011, FSIS announced new Salmonella performance standards for chicken and turkey carcasses intended to reduce annual Salmonella illnesses associated with consumption of those products by 12% and 1%, respectively (FSIS, 2011). More recently, FSIS announced new Salmonella performance standards for chicken parts and ground chicken and turkey (FSIS, 2015). These standards were expected to reduce annual human Salmonella illnesses associated with these products by 25%. Nevertheless, poultry is estimated to be responsible for only 19% of all Salmonella illnesses (Painter et al., 2013). Therefore, if fully effective, both of these policies are expected to reduce total Salmonella illnesses by only about 5%. Given our results, it would take more than a decade to determine that implementation of these policies was effective if the foodborne illness environment remained unchanged other than the influence of these policies. Our analysis may overstate power because it assumes a static environment where the only factor influencing total illnesses is the food safety policy under consideration. In reality, total illnesses from a pathogen – and the foods contributing to those illnesses – are dynamic. For example, an intended reduction of illnesses associated with one commodity can occur simultaneously with an independent upsurge in illnesses associated with another less controlled commodity, or the emergence of illnesses associated with a new commodity heretofore unrecognized as a food source for the pathogen. If ignored, annual increases in the surveillance population can generate increased observed illnesses despite modest reductions from a particular commodity. Therefore, our limited understanding of the dynamics of foodborne illness in the population, beyond those captured by the under-diagnosis probability, might mask the effectiveness of a policy. For example, changes in demographics based on race and cultural food practices may influence frequency of foodborne illnesses, especially in geographic areas where certain cultural groups reside. The Bayesian approach to power calculation assumes independence between the baseline posterior distribution and the posterior distributions for reduced total illnesses. Although this assumption might be tenuous, determining the degree of dependency between these distributions is not possible given the limited information available. At this time, under-diagnosis is a poorly understood phenomenon. From
Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.mran.2017.10.001. References CDC, 2017. Incidence and trends of infection with pathogens transmitted commonly through food and the effect of increasing use of culture-independent diagnostic tests on surveillance—foodborne diseases active surveillance network, 10 U.S. Sites, 2013–2016. MMWR Morbid. Mortal. Week. Rep. Collard, J., Bertrand, S., Dierick, K., Godard, C., Wildemauwe, C., Vermeersch, K., Duculot, J., Van Immerseel, F., Pasmans, F., Imberechts, H., 2008. Drastic decrease of Salmonella Enteritidis isolated from humans in Belgium in 2005, shift in phage types and influence on foodborne outbreaks. Epidemiol. Infect. 136, 771–781. Ebel, E.D., Williams, M.S., Schlosser, W.D., 2012. Parametric distributions of underdiagnosis parameters used to estimate annual burden of Illness for five foodborne pathogens. J. Food Protect. 75, 775–778. FSIS, 2011. New Performance Standards for Salmonella and Campylobacter in Young Chicken and Turkey Slaughter Establishments: Response to Comments and Implementation Schedule. U. S. Department of Agriculture, Washington, D.C. FSIS, 2015. Public Health Effects of Raw Chicken Parts and Comminuted Chicken and Poultry Performance Standards. United States Department of Agriculture, Washington, D.C. Herikstad, H., Motarjemi, Y., Tauxe, R.V., 2002. Salmonella surveillance: a global survey of public health serotyping. Epidemiol. Infect. 129, 1–8. Klugman, S.A., Panjer, H.H., Willmot, G.E., 2012. Loss Models: From Data to Decisions, fourth ed. John Wiley and Sons, New York. Painter, J.A., Hoekstra, R.M., Ayers, T., Tauxe, R.V., Braden, C.R., Angulo, F.J., Griffin, P.M., 2013. Attribution of foodborne illnesses, hospitalizations, and deaths to food commodities, United States, 1998–2008. Emerg. Infect. Dis. 19, 407–415. Powell, M.E., 2016. Trends in reported foodborne illness in the United States: 1996–2013. Risk Anal. 36, 1589–1598. Scallan, E., Hoekstra, R.M., Angulo, F.J., Tauxe, R.V., Widdowson, M.-A., Roy, S.L., Jones, J.L., Griffin, P.M., 2011. Foodborne illness acquired in the United States—major pathogens. Emerg. Infect. Dis. 17, 7–15. Scallan, E., Mahon, B.E., 2012. Foodborne diseases active surveillance network (FoodNet) in 2012: a foundation for food safety in the United States. Clinic. Infect. Dis. 54, S381–S384. Weiss, N.A., 2015. Introductory Statistics. Pearson. Williams, M.S., Ebel, E.D., 2012. Estimating changes in public health following implementation of hazard analysis and critical control point in the United States broiler slaughter industry. Foodborne Pathogens Disease 12, 59–67. Williams, M.S., Ebel, E.D., Vose, D., 2011. Framework for microbial food-safety risk assessments amenable to Bayesian modeling. Risk Anal. 31, 548–565. Yeni, F., Acar, S., Soyer, Y., Alpas, H., 2017. How can we improve foodborne disease surveillance systems: a comparison through EU and US systems. Food Rev. Int. 33, 406–423.
7