- Email: [email protected]
Reviewing model application to support animal health decision making
Preventive Veterinary Medicine 99 (2011) 60–67
Contents lists available at ScienceDirect
Preventive Veterinary Medicine journal homepage: www.elsevier.com/locate/prevetmed
Reviewing model application to support animal health decision making Alexander Singer a,∗ , Mo Salman b , Hans-Hermann Thulke a a
Department of Ecological Modelling, Helmholtz Centre for Environmental Research-UFZ, Permoserstr. 15, Leipzig 04318, Germany Animal Population Health Institute, College of Veterinary Medicine and Biomedical Sciences, Campus Stop 1644, Colorado State University, Fort Collins, CO 80523-1644, USA b
a r t i c l e Keywords: Policy support Animal health Modelling Review Risk assessment Model purpose
i n f o
a b s t r a c t Animal health is of societal importance as it affects human welfare, and anthropogenic interests shape decision making to assure animal health. Scientific advice to support decision making is manifold. Modelling, as one piece of the scientific toolbox, is appreciated for its ability to describe and structure data, to give insight in complex processes and to predict future outcome. In this paper we study the application of scientific modelling to support practical animal health decisions. We reviewed the 35 animal health related scientific opinions adopted by the Animal Health and Animal Welfare Panel of the European Food Safety Authority (EFSA). Thirteen of these documents were based on the application of models. The review took two viewpoints, the decision maker’s need and the modeller’s approach. In the reviewed material three types of modelling questions were addressed by four specific model types. The correspondence between tasks and models underpinned the importance of the modelling question in triggering the modelling approach. End point quantifications were the dominating request from decision makers, implying that prediction of risk is a major need. However, due to knowledge gaps corresponding modelling studies often shed away from providing exact numbers. Instead, comparative scenario analyses were performed, furthering the understanding of the decision problem and effects of alternative management options. In conclusion, the most adequate scientific support for decision making – including available modelling capacity – might be expected if the required advice is clearly stated. © 2011 Elsevier B.V. All rights reserved.
1. Introduction Models used in applied science have evolved with technical advances of computer power (Grimm and Railsback, 2005). Today’s computer capabilities together with advances in modelling theory (Grimm et al., 2005) and model standardisation (Grimm et al., 2006; Grimm and Railsback, 2006; Schmolke et al., 2010) allow tackling of complex questions on disease control and specific veterinary management issues (Levin and Durrett, 1996; Eisenberg et al., 2002; Harvey et al., 2007).
∗ Corresponding author. Tel.: +49 341 2351718; fax: +49 341 2351473. E-mail address: [email protected] (A. Singer). 0167-5877/$ – see front matter © 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.prevetmed.2011.01.004
Indeed, today’s epidemiological models are routine scientific tools both in wildlife disease epidemiology (Smith and Harris, 1991; Tischendorf et al., 1998; Augustine, 1998; Gross and Miller, 2001; Shirley et al., 2003; Eisinger et al., 2005; Lloyd-Smith et al., 2005; Kramer-Schadt et al., 2009) and veterinary epidemiology (Noordegraaf et al., 2000; Mangen et al., 2001; Keeling et al., 2003; Murray, 2006; Kudahl et al., 2007; Backer et al., 2009; Szmaragd et al., 2010; Garner et al., 2010). Without a doubt modelling has also played a role in providing policy support and enhance decision making (Taylor, 2003). We were interested in whether modern computer capabilities influenced the use of models in routine decision making. Which methodical solutions are applied? Do these reflect recent advances in modelling science? What
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
Download AHAW opinions and related material from EFSA website
Record: - opinion ID - publication issue - Terms of reference (ToR) verbose data set
61
Count ToR: - total - by aim summary data set
Quantitative model ? Yes
Identify all ToR
Iteratively develop categories of model tasks verbose data set
Analyse summary data set quantitatively
Identify all model steps
For each model
Set of categories of tasks & models exhaustive?
Collate tasks and models into these categories summary data set
Iteratively develop categories of model tools verbose data set
Yes
For each model step Summarise - question - purpose - characteristics - analysis - documentation - interpretation verbose data set
Analysis of endpoint-oriented models Report on quantitative analysis; and qualitative details this paper
Select documents on end point questions
Re-review models related to endpoint questions for available data, typical features, methods of analysis. verbose data set
Categorise models according to refined model characteristics
Report findings on models related to endpoint questions this paper
Fig. 1. Schematic work flow depicting the reviewing procedure on the documents published by EFSA’s Animal Health and Animal Welfare Panel. Outcome related procedural steps are highlighted: bold letters – raw data assembled or fact sheet produced; grey shaded boxes – categories defined and refined to structure aggregated information. The annexed box comprises procedural steps related to the re-review of those documents that contained a model related to an end-point question. Review: First, documents applying quantitative tools were identified. Next, for each quantitative tool detailed data were recorded by tabulation (verbose data set) before further aggregation (summary data set). A classification framework was developed to structure the data on model tasks and model tools. The final assignment of existing models to the categories was summarized quantitatively and qualitative characteristics were described in text-form. Refined data on the largest category of model tasks (end point-oriented questions) was gathered through revisiting the associated documents (boxed procedure at the end).
kinds of constraints follow from the focus on decision making? Documents containing information on the needs of the decision making bodies, the modelling approach used to provide support, and the provided outcome are seldom published in a standardised, open-access manner. Scientific literature focuses on the originality of the presented modelling, or certain innovative insight; but these papers less frequently link their motivation to the relevant practical decision problem. We recognised the scientific output by the European Food Safety Authority (EFSA) as useful source to investigate these issues due to its standardised structure and relevant focus. The EFSA scientific opinions are documents which, by intention, combine urgent needs from the decision making perspective with respective scientific expertise compiled by special advisory groups. Hence, EFSA scientific opinions jointly provide access to both perspectives, those of the requestor and the modeller. The availability of both viewpoints gives the chance for an integrated assessment of modelling as tool to support decision making. The aim of this paper is to evaluate the application of modelling in animal health with specific emphasis on the decision making process. Output on animal health problems adopted by the EFSA’s Panel of Animal Health and
Animal Welfare (AHAW), representative of animal health issues, was subjected to a systematic review. 2. Materials and methods Fig. 1 displays the work-flow of the review of scientific output published by EFSA’s Panel of Animal Health and Animal Welfare (AHAW). Initially, all documents were scanned for the application of quantitative tools (Fig. 1). The detailed reviewing was restricted to those documents, which were at least partly based on the application of quantitative tools. The review addressed stakeholder requests (‘Terms of Reference’ – ToR) and modelling, i.e. related sections of the documents were examined. Other parts, e.g. topical literature reviews, were neglected. Terms of Reference and models were analysed separately. 2.1. Material All scientific opinions related to animal health issues adopted by the EFSA AHAW Panel from its initiation in 2004 until May 2010 were reviewed. There were 35 opinions that fit this criterion and they were downloaded from the EFSA web site together with accessible appendices.
62
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
EFSA opinion output follows a standardised document structure which allows effective re-collection of information on (a) what kind of decision problem was stated, and (b) whether and how the scientific working groups used modelling techniques to answer to the requests. The structural detail of EFSA opinion documents that relates to the review procedure has been described by Singer (2010). Documents are referred to by the standard EFSA nomenclature detailing the year when the requestor handed in the request accompanied with an annually running number (e.g. EFSA-Q-2007-427). All documents were checked for the use of formalised quantitative tools (i.e. models with a mathematical approach aiming at quantitative output). The resulting sub-set of documents is listed in Supplementary material. These documents were subjected to a thorough review. 2.2. Review of stakeholder requests and models In EFSA opinions, stake holder requests are listed by the ‘Terms of Reference’ (ToR). ToR are usually formulated by the requestor and specify the task of a mandate. We used ToR as proxy to capture stakeholder requests. All ToR were recorded and evaluated as to whether a modelling task was specified or quantitative analysis explicitly demanded. The ToR were often split into separate modelling questions if the expert group applied more than one model to address more complex ToR. For the purpose of this review, all ToR were represented by their respective (set of) modelling questions derived from the opinion document (see Supplementary material for correspondence between ToR and modelling questions). Models found in the reviewed documents were characterised without reference to the modelling question. Model steps, if leading to an intermediary quantitative result, were treated as self-contained parts of the model formulation. Each model or model step was recorded and its features summarised. Explicitly identified data were: • model purpose (according to model description); • provision of a conceptual work plan, and whether it was displayed as a diagram; • details of model implementation; and • method of model computation. The last two features were collected along the list of modelling concepts suggested by literature (Hurd and Kaneene, 1993; Grimm et al., 2006; Appendix A in EFSA, 2009). The data comprised, for example, representation of randomness or the mathematical–statistical technique of model formulation (see Singer, 2010 for details). Data were extracted exclusively from the model description provided by the EFSA opinion documents and annexed material. If the model documentation insufficiently described particular model characteristics, an attempt was made to retrieve the detail from related content in the document. If this could not be achieved, the feature was recorded as unclear. It was decided not to follow up references to literature assuming that a document for decision support should be self-contained.
2.3. Analysis of review information The data extracted from the reviewed documents was structured by categories. These categories were developed to represent the specific modelling questions and model characteristics found in the EFSA output. The categorised data were described quantitatively to identify relevant structures. Based on the categories, patterns of correspondence between certain model tasks and model characteristics were searched. Differences and similarities were identified. 3. Results Thirteen out of the 35 animal health related EFSA mandates (37%) were answered by at least one quantitative tool. In total 23 models were applied. Below are the presentations of the categories for modelling tasks, purpose and models developed to represent the data about these 23 models. 3.1. Categorisation of modelling tasks—the request The modelling tasks were grouped into 3 categories representing the scope of the problem (Table 1). The majority of modelling tasks (56%) asked for the quantification of a certain end-point quantity. In contrast, less than 10% of model tasks addressed disease transmission dynamics. 3.2. Purpose of modelling Model purpose was recently considered as one of a few general characteristics of scientific models (Starfield et al., 1990; Philips et al., 2004; Grimm and Railsback, 2005). The model purpose (i.e. the objective of modelling) could be identified for all but one (EFSA-Q-2006-179) of the reviewed models (Table 1). The model purpose was often prominently placed in the subheadings or introductory text. In other cases, the purpose could be derived from the type of model analysis performed (e.g. comparative analysis vs. predictive calculation). The model purpose was considered as proxy to the modeller’s perception of the requested task, likely promoting the choice of a specific modelling approach. Three types of purposes were distinguished: description of data, process understanding and prediction (Hall and DeAngelis, 1985; EFSA, 2009). Several models were applied under the premise of more than one purpose. Seventy percent of models (16) aimed at understanding the modelled system. Prediction was an objective for 43% of models (10). However, in half of these analyses predictions were subsequently used for scenario comparison aimed at understanding. Four models aimed at statistical data description; for one of these ‘understanding’ was identified as a supplemental purpose (EFSA-Q-2004161). 3.3. Categorisation of quantitative models The models were classified using four types discriminating their structural differences (Hurd and Kaneene, 1993; Appendix A in EFSA, 2009). We identified decision tree
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
63
Table 1 Categories of modelling tasks, their frequency, expected and realised purposes. Modelling task
Description of modelling task
Scope
Freq.
Diagnostic test characteristics
Assessing diagnostic test statistics, or testing protocols, or surveillance systems (e.g. sensitivity, specificity or negative predictive value)
(Statistical) comparison of diagnostic testing
Transmission
Assessing disease spread and efficacy of disease control methods
Disease spread dynamics
End point quantification
Quantifying the outcome of situations or scenarios (e.g. risk assessment of disease introduction or food contamination)
Potential outcome of a predefined scenario
Intrinsic purpose of question
Addressed purposes of models
8
Descript.
Descript.: 3 Underst.: 4 Predict.: 1
2
Underst.
Descript: 0 Underst.: 2 Predict.: 0
Predict.
Descript.: 1 Underst.: 11 Predict.: 9
13
Abbreviations for purposes: Descript.: description; Underst.: understanding; and Predict.: prediction.
models (13), meta analyses (3), compound probability calculations (4), and mechanistic models (3). 3.3.1. Decision tree models Decision trees or risk pathways are an aid to structure a modelling problem. The translation of a decision tree into a model tool is straight forward. Decision tree models represent pathways through a directed hierarchical network (tree) starting from a single starting point (root level). In the reviewed opinions, decision probabilities were either derived from literature or calculated by independent submodels. Sub-models were used to represent stochasticity or heterogeneity. Usually, sub-models were analysed by simulation. Different studies applied different methods to aggregate the probabilities along the decision tree models. 3.3.2. Meta-analysis Meta-analysis is a statistical approach to combine quantitative data from different sources, taking into account their reliability. In the reviewed opinions, meta-analyses were performed to assess and compare the reliability of diagnostic tests. For this purpose, data from literature reviews and expert opinions were combined to improve estimates of sensitivity and specificity of respective diagnostic tests. 3.3.3. Compound probability calculation This model type is comprised of approaches for calculating standard characteristics of diagnostic tests or test regimes, such as sensitivity, specificity, or the negative predictive value. The sparse documentation of the reviewed models did not allow for exact identification of their specialities. 3.3.4. Mechanistic models Mechanistic or transmission models describe a system by its mechanisms, which are assumed to drive the system’s dynamics, either implicitly by the mathematical structure of the model (top–down) or explicitly by a detailed set of logical rules (bottom–up). Output can vary from analytic solutions of the model to fully simulated distributions of argument-wise output values. In the EFSA opinions, mechanistic models were applied to represent spatio-temporal dynamics of infection in populations. The
three mechanistic models found in this review were of SIR type (Susceptible-infected-recovered – Anderson et al., 1981) that considered subsequent infection stages of host units. However, the concepts of the models strongly varied. We found two equation-based models (with largely different numbers of equations) and one spatially explicit individual-based model. In contrast to other model types found in the reviewed material, only mechanistic models could be formulated rich enough to map complex dynamics like feed-back. 3.3.5. Model purpose, model task and corresponding model type Fig. 2 displays the frequency of model purpose (grey shading) stratified by model tasks (rows) and model types (columns). A clear relationship was found between modelling tasks and approaches, starting at statistical description of diagnostic test characteristics by Meta analysis, and ending at mechanistic models uniquely responding to questions concerning disease transmission. The relation suggested that modelling tasks trigger the type of applied models. Meta-analyses are uniquely applied for one model task – diagnostic test characteristics – with the aim of data description. Compound probabilities of diagnostic tests were exclusively applied for questions on diagnostic test statistics, in particular to analyse definitions of freedom of disease. Interestingly, these models rather aimed at understanding than at prediction (of the suitability of test strategies). The single decision tree model that targeted a question on diagnostic test characteristics (EFSA-Q-2007200) was applied to analyse efficacy of a monitoring and surveillance system (MOSS) in wildlife. End point quantifications were the most frequent modelling task. They were almost exclusively answered by decision tree models. In one exception (EFSA-Q-2007-427) a complex mechanistic model was applied. The model had been developed independently of the opinion (Thulke et al., 2007) and was adapted to answer requests of the mandate. The purposes of end-point oriented models varied strongly. Particularly for end-point quantifications, Table 1 revealed the shift of model purpose from the requested quantification (Table 1 Intrinsic) towards understanding (Table 1 Addressed). It might be hypothesised that
64
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
Fig. 2. Model purpose stratified by model task and model type (for all 22 models that did allow identification of purpose). Models addressing more than one purpose category got multiple counts.
such a shift is reasonable if sufficient precision of quantitative model results cannot be achieved given the available knowledge base. Therefore, all end-point related models were revisited and their modelling approach, parameterisation method, model analysis method, and documentation (EFSA, 2009) were reviewed in greater detail. 3.3.6. Models related to an end-point question Nine of the 13 studies oriented to prediction actually provided quantitative predictions (Table 1). But more frequently the model purpose was shifted to seek understanding (i.e., 11 gave qualitative predictions). In detail, we found only two strictly predictive models (both EFSA-Q2004-100, that performed scenario analyses but did not explicitly compare between scenarios), seven that additionally aimed at understanding, and three models that solely provided understanding. Fig. 3 displays the relation of predictive model purpose (rows) given the knowledge base (columns) and the representation of uncertainty in model output analysis (grey shades) in 11 end-point oriented modelling studies. Strikingly, only one of the models could be fully based on literature data. To do so, this study (EFSA-Q-2008-665) considered a highly controllable environment (semen col-
lection centres) and parameterised the model from the results of a meta-analysis. This allowed quantification of uncertainties of risk estimates for different control scenarios. Other studies had to deal with sometimes severe knowledge gaps. Most commonly, parameter variation (sensitivity analysis) was applied in order to represent the impact of knowledge gaps on model output. Other studies considered probability distributions of uncertain parameter values as well as biological variability. The impact on the results was shown by confidence intervals around the model output. Two studies presented model results as point estimates without explicitly specifying uncertainties in model outcome. However, these studies relied on other approaches to allow uncertainty assessment. EFSA-Q-2006-014 provided the model itself as Supplementary material, such that users could test different assumptions. EFSA-Q-2007-427 validated a complex mechanistic model against data (applying the technique of pattern oriented modelling (Grimm et al., 2005)) before running comparative analyses. In contrary to the working assumption, most studies aimed at fulfilling the requested end-point prediction, independent of uncertain model parameterisation due to missing knowledge. The methods applied to present the resulting output uncertainty differed but were indepen-
Fig. 3. End point oriented models stratified by their purpose (being predictive or not; rows), the knowledge base (columns), and the way to demonstrate outcome uncertainty (grey shading). A strong knowledge base means that all model parameters could be derived from literature. Presentation of outcome uncertainty refers to the main means by which propagated parameter uncertainty and model variability are disclosed in the document: ‘Point estimate’ comprises presentation of single statistics (e.g. a mean value); ‘Range estimates’ represent indication of result uncertainty (e.g. as confidence interval), ‘Parameter variation’ comprises structured parameter variation to illustrate effects of parameter uncertainty. Two of the 13 end point oriented models had to be excluded from this analysis: opinion EFSA-Q-2006-112 assessed an exemplary situation because data for model parameterisation was not available. The third model in EFSA-Q-2008-665 was almost identical to the second model in that opinion and therefore was omitted to avoid double counting.
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
Diagnostic test characteristics
Literature summary Test accuracy
Meta-Analysis
Test system efficacy (MOSS)
Compound Probability
65
Introduction of disease End point quantification
Risk of secondary cases
Decision Tree
Infective agents in food Transmission
Mechanistic Model
Fig. 4. Schematic representation of the relation between modelling questions and model categories. The graph indicates that modelling tasks (blocks in left column) can be split into topically specific questions (middle columns) and how they were tackled by different modelling approaches (right column). The strong correspondence between modelling tasks and approaches becomes obvious in the vertical separation into task-approach-complexes. The pattern is only broken by two singular examples (Note: solid lines denote that several examples could be found for the link. Dotted lines denote singular cases.).
dent of whether the predictive approach was chosen or not. 4. Discussion To assess the applicability of scientific modelling to support policy and decision making, we reviewed the 13 animal-health related scientific opinions adopted by European Food Safety Authority’s Panel on Animal Health and Animal Welfare that were based on quantitative modelling. EFSA scientific opinions document a scientific assessment of questions relevant to EU animal health policy. The preparation of opinions is commissioned by EFSA to a working group of scientific experts. Hence, the documents may be considered as revealing an immediate link between policy needs and scientific assessment. The presented structured review (Fig. 1) revealed modelling as a common method to substantiate EFSA’s scientific opinions. Although rarely explicitly demanded, quantitative modelling was applied in more than a third of the studies included in the review. We limited the review to one particular system: the policy needs mandated to EFSA and the approaches EFSA working groups used to reply to the need. This system generates standardised documents reflecting policy demands and the respective scientific response. Although scientific policy support to animal health might be organised differently nationally or beyond the EU, we considered the EFSA documents to constitute an excellent source of examples for the process and outcome of applied scientific policy support. Unfortunately, EFSA opinions do not allow insight into the integration of scientific advice into policy. Thus, the final step in communication (EFSA, 2009) could not be assessed in this review. Perception and classification of the reviewed material naturally was subjective. The aim for objective insight was sought by assessing the modelling for EFSA opinions against established modelling theory such as standard models, parameterisation methods and analysis tech-
niques (Starfield et al., 1990; Philips et al., 2004; Saltelli et al., 2000; Grimm and Railsback, 2005; Bolker, 2008; EFSA, 2009). The objective of this review was to reveal how requests by animal health policy stakeholders translate into modelling methods used for response. A striking correspondence of modelling tasks and applied models was found (Fig. 4). To answer questions on the accuracy of diagnostic tests, data gathered in literature reviews were modelled by meta-analysis technique aggregating scientific evidence and expert knowledge. The particular approach found in the respective documents was developed in one EFSA opinion and applied in subsequent documents, apparently making it a quasi-standard. Tasks addressing the diagnostic efficacy of the monitoring and surveillance systems (MOSS) were tackled by calculating joined or combined probability; e.g., to detect a disease by a certain strategy (sensitivity, specificity, freedom of disease). These methods were presumably considered standard, because model description was brief in all documents. Slight differences could, therefore, not be retrieved. One MOSS question was addressed by a decision tree model. In contrast to the other examples, the model focused on wildlife species which are not readily accessible for diagnostics. Hence, sampling bias and uncertainty in sample access had to be considered. Such complexity would hamper the analytic tractability of compound probability calculation (but see e.g., Martin et al., 2007). End-point quantifications were the most common request that was treated by modelling. The common task was risk calculation. The prevailing model type was the decision tree model, likely because these allowed a formal structure to the hazard pathways. A different model category was chosen in only one case (mechanistic model). This assessment accounted for the dynamic allocation of control zones during the spread of a disease. Such spatio-temporal stochastic processes could not efficiently be represented by decision tree models. The application of the more complex model type was supported by the existence of a pre-developed model framework (EFSA-Q-2007-427).
66
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
Astonishingly, there were only few requests explicitly addressing disease spread and contingency planning where the response used modelling. The few questions were all tackled by mechanistic models, probably because only this model category is capable of tracking temporally changing dynamics. The complexity of these few models varied strongly depending on the level of heterogeneity mapped from the biological system. The correspondence between the modelling task and the applied model category underpins that minimal capabilities have to be provided by the modelling to tackle certain tasks (Singer, 2010). But simultaneously that the formulation of the request influenced the quantitative tools applied while preparing an answer. There is the need of transparent communication of the purpose of the request to initiate the actually intended scientific assessment (EFSA, 2009). Clear specification of the requestor’s intention – e.g. understanding or prediction – will make it more likely that the outcome of the assessment is adequate. For example, a request for quantitative prediction where there is uncertainty or gaps in knowledge will promote decision tree models. Often the main advantage of these models comes from structuring knowledge and knowledge gaps around the pathway that is supposed to lead to an end-point event. However, the models were applied to calculate risk estimates for specific exposure scenarios. In this case necessary assumptions had to be made to get rid of knowledge gaps or uncertainties (including all side effects of being precisely predictive under uncertainty). Structured awareness of the weak and strong steps in a decision tree will be beneficial to making informed decisions, and sometimes may even satisfy the intended outcome from the requestor’s perspective. The more knowledge gaps that exist regarding a topic the less likely the strict request of an end-point estimate will lead to an exhaustively informative decision tree. This will instead lead to a formal model calculation during which the unknowns need to be overcome. Similarly, the current discussion in eco-toxicology is the need for ecology-based rather than end-point focused modelling when the consequences of things like pesticide treatment must be assessed to support regulatory decision making (Forbes et al., 2009; Thorbek et al., 2010). Interestingly, the purpose of the responding models often diverted from the purpose implied by the question. The purpose shift was likely caused by knowledge gaps. We found that the vast majority of end-point oriented studies could not be fully parameterised from known data (e.g. literature data). Several opinions provided a comparative analysis (e.g. explored uncertainty and sensitivity by presenting a list of scenarios). Hence, the pragmatic result was to focus on understanding the modelled biological system rather than predicting its ultimate outcome. Some EFSA opinions even demonstrated that limitations in available knowledge limited the predictive power of applied models. 5. Conclusion Our review showed how requests triggered modelling outcome, e.g. how seeking prediction of endpoint values promoted a dominance of structurally similar decision tree models. However, a limited knowledge base, revealed by
the majority of reviewed assessments, impeded precise endpoint estimates and, strictly speaking, the requested task could not be answered. However, the implicit effort to structure knowledge furthered the understanding of the decision problem and the management options, thus providing background for informed decision making. Scientific support for decision making is sought for complex situations where there is a range of options of action. These circumstances implicate incomplete knowledge and, as highlighted by our review, the predictive power of models often is limited. However, decisions have to be made no matter whether there will be complete scientific justification. Hence, providing background for informed decisions is already an important driver for purposeful modelling; the result could be the shift from intuition to evidence based decision making. Conflicts of interest statement None declared. Acknowledgements We thank the two anonymous reviewers for valuable comments and suggestions to improve the manuscript. Parts of data collection were funded by EFSA. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.prevetmed.2011.01.004. References Anderson, R.M., Jackson, H.C., May, R.M., Smith, A.M., 1981. Population dynamics of fox rabies in Europe. Nature 289, 765–771. Augustine, D.J., 1998. Modelling Chlamydia–koala interactions: coexistence, population dynamics and conservation implications. J. Appl. Ecol. 35, 261–272. Backer, J.A., Hagenaars, T.J., van Roermund, H.J.W., de Jong, M.C.M., 2009. Modelling the effectiveness and risks of vaccination strategies to control classical swine fever epidemics. J. Royal Soc. Interface 6, 849–861. Bolker, B.M., 2008. Ecological Models and Data in R. Princeton University Press, Princeton. EFSA, 2009. Panel on animal health & welfare (AHAW); guidance on good practice in conducting scientific assessments in animal health using modelling. EFSA J. 7 (12:1419), 38. Eisenberg, J.N.S., Brookhart, M.A., Rice, G., Brown, M., Colford, J.M., 2002. Disease transmission models for public health decision making: analysis of epidemic and endemic conditions caused by waterborne pathogens. Environ. Health Perspect. 110, 783–790. Eisinger, D., Thulke, H.-H., Müller, T., Selhorst, T., 2005. Emergency vaccination of rabies under limited resources – combating or containing? BMC Infect. Dis. 5, 10. Forbes, V.E., Hommen, U., Thorbek, P., Heimbach, F., van den Brink, P., Wogram, J., Thulke, H.-H., Grimm, V., 2009. Ecological models in support of regulatory risk assessments of pesticides: developing a strategy for the future. Integr. Environ. Assess. Manage. 5, 167–172. Garner, M.G., et al., 2010. Evaluating the effectiveness of early vaccination in the control and eradication of equine influenza—a modelling approach. Pre. Vet. Med. 99, 15–27. Grimm, V., Berger, U., Bastiansen, F., Eliassen, S., Ginot, V., Giske, J., GossCustard, J., Grand, T., Heinz, S., Huse, G., Huth, A., Jepsen, J.U., Jørgensen, C., Mooij, W.M., Müller, B., Pe’er, G., Piou, C., Railsback, S.F., Robbins, A.M., Robbins, M.M., Rossmanith, E., Rüger, N., Strand, E., Souissi, S., Stillman, R.A., Vabø, R., Visser, U., DeAngelis, D.L., 2006. A standard protocol for describing individual-based and agent-based models. Ecol. Modell. 198, 115–126.
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67 Grimm, V., Railsback, S.F., 2005. Individual-Based Modeling and Ecology. Princeton University Press, Princeton. Grimm, V., Railsback, S.F., 2006. Agent-based models in ecology: patterns and alternative theories of adaptive behaviour. In: Billari, F., Fent, T., Prskawetz, A., Scheffran, J. (Eds.), Agent-based Computational Modelling—Applications in Demography, Social, Economic and Environmental Sciences. Physica, Heidelberg, pp. 139–152. Grimm, V., Revilla, E., Berger, U., Jeltsch, F., Mooij, W.M., Railsback, S.F., Thulke, H.-H., Weiner, J., Wiegand, T., DeAngelis, D.L., 2005. Patternoriented modeling of agent-based complex systems: lessons from ecology. Science 310, 987–991. Gross, J.E., Miller, M.W., 2001. Chronic wasting disease in mule deer: disease dynamics and control. J. Wildl. Manage. 65, 205–215. Hall, C.A.S., DeAngelis, D.L., 1985. Eco-forum: models in ecology: paradigms found or paradigms lost? B. Ecol. Soc. Am. 66, 339–346. Harvey, N., Reeves, A., Schoenbaum, M.A., Zagmutt-Vergara, F.J., Dubé, C., Hill, A.E., Corso, B.A., McNab, W.B., Cartwright, C.I., Salman, M.D., 2007. The north american animal disease spread model: a simulation model to assist decision making in evaluating animal disease incursions. Prev. Vet. Med. 82, 176–197. Hurd, H.S., Kaneene, J.B., 1993. The application of simulation models and systems analysis in epidemiology: a review. Prev. Vet. Med. 15, 81–99. Keeling, M.J., Woolhouse, M.E.J., May, R.M., Davies, G., Grenfell, B.T., 2003. Modelling vaccination strategies against foot-and-mouth disease. Nature 421, 136–142. Kramer-Schadt, S., Fernández, N., Grimm, V., Thulke, H.-H., 2009. Individual variation in infectiousness explains long-term disease persistence in wildlife populations. Oikos 118, 199–208. Kudahl, A.B., Østergaard, S., Sørensen, J.T., Nielsen, S.S., 2007. A stochastic model simulating paratuberculosis in a dairy herd. Prev. Vet. Med. 78, 97–117. Levin, S.A., Durrett, R., 1996. From individuals to epidemics. Phil. Trans. R. Soc. London B 351, 1615–1621. Lloyd-Smith, J.O., Cross, P.C., Briggs, C.J., Daugherty, M., Getz, W.M., Latto, J., Sanchez, M.S., Smith, A.B., Swei, A., 2005. Should we expect population thresholds for wildlife disease? Trends Ecol. Evol. 20, 511–519. Mangen, M.-J.J., Jalvingh, A.W., Nielen, M., Mourits, M.C.M., Klinkenberg, D., Dijkhuizen, A.A., 2001. Spatial and stochastic simulation to compare two emergency-vaccination strategies with a marker vaccine in the 1997/1998 Dutch classical swine fever epidemic. Prev. Vet. Med. 48, 177–200. Martin, P.A.J., Cameron, A.R., Greiner, M., 2007. Demonstrating freedom from disease using multiple complex data sources: 1. A new methodology based on scenario trees. Prev. Vet. Med. 79, 71–97. Murray, A.G., 2006. A model of the emergence of infectious pancreatic necrosis virus in Scottish salmon farms 1996–2003. Ecol. Modell. 199, 64–72.
67
Noordegraaf, A.V., Jalvingh, A.W., de Jong, M.C.M., Franken, P., Dijkhuizen, A.A., 2000. Evaluating control strategies for outbreaks in BHV1-free areas using stochastic and spatial simulation. Prev. Vet. Med. 44, 21–42. Philips, Z., Ginnelly, L., Sculpher, L., Claxton, K., Golder, S., Roiemsma, R., Woolacott, N., Glanville, J., 2004. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol. Assess. 8 (36). Saltelli, A., Chan, K., Scott, E.M., 2000. Sensitivity Analysis. Wiley, Chichesters. Schmolke, A., Thorbek, P., DeAngelis, D.L., Grimm, V., 2010. Ecological modelling supporting environmental decision making: a strategy for the future. Trends Ecol. Evol. 25, 479–486. Shirley, M.D.F., Rushton, S.P., Smith, G.C., South, A.B., Lurz, P.W.W., 2003. Investigating the spatial dynamics of bovine tuberculosis in badger populations: evaluating an individual-based simulation model. Ecol. Modell. 167, 139–157. Singer, A., 2010. External report reviewing the previous opinions of the Panel on Animal Health and Welfare concerning the application of quantitative tools, in the sequence of the current self mandate on “Good Practice in Conducting Scientific Assessments in Animal Health Using Modelling” (Question No EFSA-Q-2009-408)., http://www.efsa.europa.eu/en/scdocs/scdoc/43e.htm. Smith, G.C., Harris, S., 1991. Rabies in urban foxes (Vulpes vulpes) in Britain: the use of a spatial stochastic simulation model to examine the pattern of spread and evaluate the efficacy of different control regimes. Phil. Trans. R. Soc. London B 334, 459–479. Starfield, A.M., Smith, K.A., Bleloch, A.L., 1990. How to Model It. McGrawHill, Inc., New York, St. Louis, San Francisco. Szmaragd, C., Wilson, A.J., Carpenter, S., Mellor, P.S., Gubbins, S., 2010. The spread of Bluetongue virus serotype 8 in Great Britain and its control by vaccination. PLoS one 5, e9353. Taylor, N., 2003. 2003 Review of the use of models in informing disease control policy development and adjustment. 1286. Department for Environmental, Food, and Rural Affairs, UK., http:// epicentre.massey.ac.nz/resources/acvsc grp/docs/Taylor 2003.pdf. Thorbek, P., Forbes, V., Heimbach, F., Hommen, U., Thulke, H.-H., van den Brink, P.J., Wogram, J., Grimm, V., 2010. Ecological Models for Regulatory Risk Assessments of Pesticides. Society of Environmental and Chemistry (SETAC) and CRC Press, Pensacola and Boca Raton, FL. Thulke, H.-H., Eisinger, D., Beer, M., 2007. CSF emergency control – from the control ideal towards the ideal control , Presentation at AVIDmeeting, Riems. Tischendorf, L., Thulke, H.-H., Staubach, C., Müller, M.S., Jeltsch, F., Goretzki, J., Selhorst, T., Müller, T., Schlüter, H., Wissel, C., 1998. Chance and risk of controlling rabies in large-scale and long-term immunized fox populations. Proc. R. Soc. London B 265, 839–846.
Contents lists available at ScienceDirect
Preventive Veterinary Medicine journal homepage: www.elsevier.com/locate/prevetmed
Reviewing model application to support animal health decision making Alexander Singer a,∗ , Mo Salman b , Hans-Hermann Thulke a a
Department of Ecological Modelling, Helmholtz Centre for Environmental Research-UFZ, Permoserstr. 15, Leipzig 04318, Germany Animal Population Health Institute, College of Veterinary Medicine and Biomedical Sciences, Campus Stop 1644, Colorado State University, Fort Collins, CO 80523-1644, USA b
a r t i c l e Keywords: Policy support Animal health Modelling Review Risk assessment Model purpose
i n f o
a b s t r a c t Animal health is of societal importance as it affects human welfare, and anthropogenic interests shape decision making to assure animal health. Scientific advice to support decision making is manifold. Modelling, as one piece of the scientific toolbox, is appreciated for its ability to describe and structure data, to give insight in complex processes and to predict future outcome. In this paper we study the application of scientific modelling to support practical animal health decisions. We reviewed the 35 animal health related scientific opinions adopted by the Animal Health and Animal Welfare Panel of the European Food Safety Authority (EFSA). Thirteen of these documents were based on the application of models. The review took two viewpoints, the decision maker’s need and the modeller’s approach. In the reviewed material three types of modelling questions were addressed by four specific model types. The correspondence between tasks and models underpinned the importance of the modelling question in triggering the modelling approach. End point quantifications were the dominating request from decision makers, implying that prediction of risk is a major need. However, due to knowledge gaps corresponding modelling studies often shed away from providing exact numbers. Instead, comparative scenario analyses were performed, furthering the understanding of the decision problem and effects of alternative management options. In conclusion, the most adequate scientific support for decision making – including available modelling capacity – might be expected if the required advice is clearly stated. © 2011 Elsevier B.V. All rights reserved.
1. Introduction Models used in applied science have evolved with technical advances of computer power (Grimm and Railsback, 2005). Today’s computer capabilities together with advances in modelling theory (Grimm et al., 2005) and model standardisation (Grimm et al., 2006; Grimm and Railsback, 2006; Schmolke et al., 2010) allow tackling of complex questions on disease control and specific veterinary management issues (Levin and Durrett, 1996; Eisenberg et al., 2002; Harvey et al., 2007).
∗ Corresponding author. Tel.: +49 341 2351718; fax: +49 341 2351473. E-mail address: [email protected] (A. Singer). 0167-5877/$ – see front matter © 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.prevetmed.2011.01.004
Indeed, today’s epidemiological models are routine scientific tools both in wildlife disease epidemiology (Smith and Harris, 1991; Tischendorf et al., 1998; Augustine, 1998; Gross and Miller, 2001; Shirley et al., 2003; Eisinger et al., 2005; Lloyd-Smith et al., 2005; Kramer-Schadt et al., 2009) and veterinary epidemiology (Noordegraaf et al., 2000; Mangen et al., 2001; Keeling et al., 2003; Murray, 2006; Kudahl et al., 2007; Backer et al., 2009; Szmaragd et al., 2010; Garner et al., 2010). Without a doubt modelling has also played a role in providing policy support and enhance decision making (Taylor, 2003). We were interested in whether modern computer capabilities influenced the use of models in routine decision making. Which methodical solutions are applied? Do these reflect recent advances in modelling science? What
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
Download AHAW opinions and related material from EFSA website
Record: - opinion ID - publication issue - Terms of reference (ToR) verbose data set
61
Count ToR: - total - by aim summary data set
Quantitative model ? Yes
Identify all ToR
Iteratively develop categories of model tasks verbose data set
Analyse summary data set quantitatively
Identify all model steps
For each model
Set of categories of tasks & models exhaustive?
Collate tasks and models into these categories summary data set
Iteratively develop categories of model tools verbose data set
Yes
For each model step Summarise - question - purpose - characteristics - analysis - documentation - interpretation verbose data set
Analysis of endpoint-oriented models Report on quantitative analysis; and qualitative details this paper
Select documents on end point questions
Re-review models related to endpoint questions for available data, typical features, methods of analysis. verbose data set
Categorise models according to refined model characteristics
Report findings on models related to endpoint questions this paper
Fig. 1. Schematic work flow depicting the reviewing procedure on the documents published by EFSA’s Animal Health and Animal Welfare Panel. Outcome related procedural steps are highlighted: bold letters – raw data assembled or fact sheet produced; grey shaded boxes – categories defined and refined to structure aggregated information. The annexed box comprises procedural steps related to the re-review of those documents that contained a model related to an end-point question. Review: First, documents applying quantitative tools were identified. Next, for each quantitative tool detailed data were recorded by tabulation (verbose data set) before further aggregation (summary data set). A classification framework was developed to structure the data on model tasks and model tools. The final assignment of existing models to the categories was summarized quantitatively and qualitative characteristics were described in text-form. Refined data on the largest category of model tasks (end point-oriented questions) was gathered through revisiting the associated documents (boxed procedure at the end).
kinds of constraints follow from the focus on decision making? Documents containing information on the needs of the decision making bodies, the modelling approach used to provide support, and the provided outcome are seldom published in a standardised, open-access manner. Scientific literature focuses on the originality of the presented modelling, or certain innovative insight; but these papers less frequently link their motivation to the relevant practical decision problem. We recognised the scientific output by the European Food Safety Authority (EFSA) as useful source to investigate these issues due to its standardised structure and relevant focus. The EFSA scientific opinions are documents which, by intention, combine urgent needs from the decision making perspective with respective scientific expertise compiled by special advisory groups. Hence, EFSA scientific opinions jointly provide access to both perspectives, those of the requestor and the modeller. The availability of both viewpoints gives the chance for an integrated assessment of modelling as tool to support decision making. The aim of this paper is to evaluate the application of modelling in animal health with specific emphasis on the decision making process. Output on animal health problems adopted by the EFSA’s Panel of Animal Health and
Animal Welfare (AHAW), representative of animal health issues, was subjected to a systematic review. 2. Materials and methods Fig. 1 displays the work-flow of the review of scientific output published by EFSA’s Panel of Animal Health and Animal Welfare (AHAW). Initially, all documents were scanned for the application of quantitative tools (Fig. 1). The detailed reviewing was restricted to those documents, which were at least partly based on the application of quantitative tools. The review addressed stakeholder requests (‘Terms of Reference’ – ToR) and modelling, i.e. related sections of the documents were examined. Other parts, e.g. topical literature reviews, were neglected. Terms of Reference and models were analysed separately. 2.1. Material All scientific opinions related to animal health issues adopted by the EFSA AHAW Panel from its initiation in 2004 until May 2010 were reviewed. There were 35 opinions that fit this criterion and they were downloaded from the EFSA web site together with accessible appendices.
62
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
EFSA opinion output follows a standardised document structure which allows effective re-collection of information on (a) what kind of decision problem was stated, and (b) whether and how the scientific working groups used modelling techniques to answer to the requests. The structural detail of EFSA opinion documents that relates to the review procedure has been described by Singer (2010). Documents are referred to by the standard EFSA nomenclature detailing the year when the requestor handed in the request accompanied with an annually running number (e.g. EFSA-Q-2007-427). All documents were checked for the use of formalised quantitative tools (i.e. models with a mathematical approach aiming at quantitative output). The resulting sub-set of documents is listed in Supplementary material. These documents were subjected to a thorough review. 2.2. Review of stakeholder requests and models In EFSA opinions, stake holder requests are listed by the ‘Terms of Reference’ (ToR). ToR are usually formulated by the requestor and specify the task of a mandate. We used ToR as proxy to capture stakeholder requests. All ToR were recorded and evaluated as to whether a modelling task was specified or quantitative analysis explicitly demanded. The ToR were often split into separate modelling questions if the expert group applied more than one model to address more complex ToR. For the purpose of this review, all ToR were represented by their respective (set of) modelling questions derived from the opinion document (see Supplementary material for correspondence between ToR and modelling questions). Models found in the reviewed documents were characterised without reference to the modelling question. Model steps, if leading to an intermediary quantitative result, were treated as self-contained parts of the model formulation. Each model or model step was recorded and its features summarised. Explicitly identified data were: • model purpose (according to model description); • provision of a conceptual work plan, and whether it was displayed as a diagram; • details of model implementation; and • method of model computation. The last two features were collected along the list of modelling concepts suggested by literature (Hurd and Kaneene, 1993; Grimm et al., 2006; Appendix A in EFSA, 2009). The data comprised, for example, representation of randomness or the mathematical–statistical technique of model formulation (see Singer, 2010 for details). Data were extracted exclusively from the model description provided by the EFSA opinion documents and annexed material. If the model documentation insufficiently described particular model characteristics, an attempt was made to retrieve the detail from related content in the document. If this could not be achieved, the feature was recorded as unclear. It was decided not to follow up references to literature assuming that a document for decision support should be self-contained.
2.3. Analysis of review information The data extracted from the reviewed documents was structured by categories. These categories were developed to represent the specific modelling questions and model characteristics found in the EFSA output. The categorised data were described quantitatively to identify relevant structures. Based on the categories, patterns of correspondence between certain model tasks and model characteristics were searched. Differences and similarities were identified. 3. Results Thirteen out of the 35 animal health related EFSA mandates (37%) were answered by at least one quantitative tool. In total 23 models were applied. Below are the presentations of the categories for modelling tasks, purpose and models developed to represent the data about these 23 models. 3.1. Categorisation of modelling tasks—the request The modelling tasks were grouped into 3 categories representing the scope of the problem (Table 1). The majority of modelling tasks (56%) asked for the quantification of a certain end-point quantity. In contrast, less than 10% of model tasks addressed disease transmission dynamics. 3.2. Purpose of modelling Model purpose was recently considered as one of a few general characteristics of scientific models (Starfield et al., 1990; Philips et al., 2004; Grimm and Railsback, 2005). The model purpose (i.e. the objective of modelling) could be identified for all but one (EFSA-Q-2006-179) of the reviewed models (Table 1). The model purpose was often prominently placed in the subheadings or introductory text. In other cases, the purpose could be derived from the type of model analysis performed (e.g. comparative analysis vs. predictive calculation). The model purpose was considered as proxy to the modeller’s perception of the requested task, likely promoting the choice of a specific modelling approach. Three types of purposes were distinguished: description of data, process understanding and prediction (Hall and DeAngelis, 1985; EFSA, 2009). Several models were applied under the premise of more than one purpose. Seventy percent of models (16) aimed at understanding the modelled system. Prediction was an objective for 43% of models (10). However, in half of these analyses predictions were subsequently used for scenario comparison aimed at understanding. Four models aimed at statistical data description; for one of these ‘understanding’ was identified as a supplemental purpose (EFSA-Q-2004161). 3.3. Categorisation of quantitative models The models were classified using four types discriminating their structural differences (Hurd and Kaneene, 1993; Appendix A in EFSA, 2009). We identified decision tree
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
63
Table 1 Categories of modelling tasks, their frequency, expected and realised purposes. Modelling task
Description of modelling task
Scope
Freq.
Diagnostic test characteristics
Assessing diagnostic test statistics, or testing protocols, or surveillance systems (e.g. sensitivity, specificity or negative predictive value)
(Statistical) comparison of diagnostic testing
Transmission
Assessing disease spread and efficacy of disease control methods
Disease spread dynamics
End point quantification
Quantifying the outcome of situations or scenarios (e.g. risk assessment of disease introduction or food contamination)
Potential outcome of a predefined scenario
Intrinsic purpose of question
Addressed purposes of models
8
Descript.
Descript.: 3 Underst.: 4 Predict.: 1
2
Underst.
Descript: 0 Underst.: 2 Predict.: 0
Predict.
Descript.: 1 Underst.: 11 Predict.: 9
13
Abbreviations for purposes: Descript.: description; Underst.: understanding; and Predict.: prediction.
models (13), meta analyses (3), compound probability calculations (4), and mechanistic models (3). 3.3.1. Decision tree models Decision trees or risk pathways are an aid to structure a modelling problem. The translation of a decision tree into a model tool is straight forward. Decision tree models represent pathways through a directed hierarchical network (tree) starting from a single starting point (root level). In the reviewed opinions, decision probabilities were either derived from literature or calculated by independent submodels. Sub-models were used to represent stochasticity or heterogeneity. Usually, sub-models were analysed by simulation. Different studies applied different methods to aggregate the probabilities along the decision tree models. 3.3.2. Meta-analysis Meta-analysis is a statistical approach to combine quantitative data from different sources, taking into account their reliability. In the reviewed opinions, meta-analyses were performed to assess and compare the reliability of diagnostic tests. For this purpose, data from literature reviews and expert opinions were combined to improve estimates of sensitivity and specificity of respective diagnostic tests. 3.3.3. Compound probability calculation This model type is comprised of approaches for calculating standard characteristics of diagnostic tests or test regimes, such as sensitivity, specificity, or the negative predictive value. The sparse documentation of the reviewed models did not allow for exact identification of their specialities. 3.3.4. Mechanistic models Mechanistic or transmission models describe a system by its mechanisms, which are assumed to drive the system’s dynamics, either implicitly by the mathematical structure of the model (top–down) or explicitly by a detailed set of logical rules (bottom–up). Output can vary from analytic solutions of the model to fully simulated distributions of argument-wise output values. In the EFSA opinions, mechanistic models were applied to represent spatio-temporal dynamics of infection in populations. The
three mechanistic models found in this review were of SIR type (Susceptible-infected-recovered – Anderson et al., 1981) that considered subsequent infection stages of host units. However, the concepts of the models strongly varied. We found two equation-based models (with largely different numbers of equations) and one spatially explicit individual-based model. In contrast to other model types found in the reviewed material, only mechanistic models could be formulated rich enough to map complex dynamics like feed-back. 3.3.5. Model purpose, model task and corresponding model type Fig. 2 displays the frequency of model purpose (grey shading) stratified by model tasks (rows) and model types (columns). A clear relationship was found between modelling tasks and approaches, starting at statistical description of diagnostic test characteristics by Meta analysis, and ending at mechanistic models uniquely responding to questions concerning disease transmission. The relation suggested that modelling tasks trigger the type of applied models. Meta-analyses are uniquely applied for one model task – diagnostic test characteristics – with the aim of data description. Compound probabilities of diagnostic tests were exclusively applied for questions on diagnostic test statistics, in particular to analyse definitions of freedom of disease. Interestingly, these models rather aimed at understanding than at prediction (of the suitability of test strategies). The single decision tree model that targeted a question on diagnostic test characteristics (EFSA-Q-2007200) was applied to analyse efficacy of a monitoring and surveillance system (MOSS) in wildlife. End point quantifications were the most frequent modelling task. They were almost exclusively answered by decision tree models. In one exception (EFSA-Q-2007-427) a complex mechanistic model was applied. The model had been developed independently of the opinion (Thulke et al., 2007) and was adapted to answer requests of the mandate. The purposes of end-point oriented models varied strongly. Particularly for end-point quantifications, Table 1 revealed the shift of model purpose from the requested quantification (Table 1 Intrinsic) towards understanding (Table 1 Addressed). It might be hypothesised that
64
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
Fig. 2. Model purpose stratified by model task and model type (for all 22 models that did allow identification of purpose). Models addressing more than one purpose category got multiple counts.
such a shift is reasonable if sufficient precision of quantitative model results cannot be achieved given the available knowledge base. Therefore, all end-point related models were revisited and their modelling approach, parameterisation method, model analysis method, and documentation (EFSA, 2009) were reviewed in greater detail. 3.3.6. Models related to an end-point question Nine of the 13 studies oriented to prediction actually provided quantitative predictions (Table 1). But more frequently the model purpose was shifted to seek understanding (i.e., 11 gave qualitative predictions). In detail, we found only two strictly predictive models (both EFSA-Q2004-100, that performed scenario analyses but did not explicitly compare between scenarios), seven that additionally aimed at understanding, and three models that solely provided understanding. Fig. 3 displays the relation of predictive model purpose (rows) given the knowledge base (columns) and the representation of uncertainty in model output analysis (grey shades) in 11 end-point oriented modelling studies. Strikingly, only one of the models could be fully based on literature data. To do so, this study (EFSA-Q-2008-665) considered a highly controllable environment (semen col-
lection centres) and parameterised the model from the results of a meta-analysis. This allowed quantification of uncertainties of risk estimates for different control scenarios. Other studies had to deal with sometimes severe knowledge gaps. Most commonly, parameter variation (sensitivity analysis) was applied in order to represent the impact of knowledge gaps on model output. Other studies considered probability distributions of uncertain parameter values as well as biological variability. The impact on the results was shown by confidence intervals around the model output. Two studies presented model results as point estimates without explicitly specifying uncertainties in model outcome. However, these studies relied on other approaches to allow uncertainty assessment. EFSA-Q-2006-014 provided the model itself as Supplementary material, such that users could test different assumptions. EFSA-Q-2007-427 validated a complex mechanistic model against data (applying the technique of pattern oriented modelling (Grimm et al., 2005)) before running comparative analyses. In contrary to the working assumption, most studies aimed at fulfilling the requested end-point prediction, independent of uncertain model parameterisation due to missing knowledge. The methods applied to present the resulting output uncertainty differed but were indepen-
Fig. 3. End point oriented models stratified by their purpose (being predictive or not; rows), the knowledge base (columns), and the way to demonstrate outcome uncertainty (grey shading). A strong knowledge base means that all model parameters could be derived from literature. Presentation of outcome uncertainty refers to the main means by which propagated parameter uncertainty and model variability are disclosed in the document: ‘Point estimate’ comprises presentation of single statistics (e.g. a mean value); ‘Range estimates’ represent indication of result uncertainty (e.g. as confidence interval), ‘Parameter variation’ comprises structured parameter variation to illustrate effects of parameter uncertainty. Two of the 13 end point oriented models had to be excluded from this analysis: opinion EFSA-Q-2006-112 assessed an exemplary situation because data for model parameterisation was not available. The third model in EFSA-Q-2008-665 was almost identical to the second model in that opinion and therefore was omitted to avoid double counting.
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
Diagnostic test characteristics
Literature summary Test accuracy
Meta-Analysis
Test system efficacy (MOSS)
Compound Probability
65
Introduction of disease End point quantification
Risk of secondary cases
Decision Tree
Infective agents in food Transmission
Mechanistic Model
Fig. 4. Schematic representation of the relation between modelling questions and model categories. The graph indicates that modelling tasks (blocks in left column) can be split into topically specific questions (middle columns) and how they were tackled by different modelling approaches (right column). The strong correspondence between modelling tasks and approaches becomes obvious in the vertical separation into task-approach-complexes. The pattern is only broken by two singular examples (Note: solid lines denote that several examples could be found for the link. Dotted lines denote singular cases.).
dent of whether the predictive approach was chosen or not. 4. Discussion To assess the applicability of scientific modelling to support policy and decision making, we reviewed the 13 animal-health related scientific opinions adopted by European Food Safety Authority’s Panel on Animal Health and Animal Welfare that were based on quantitative modelling. EFSA scientific opinions document a scientific assessment of questions relevant to EU animal health policy. The preparation of opinions is commissioned by EFSA to a working group of scientific experts. Hence, the documents may be considered as revealing an immediate link between policy needs and scientific assessment. The presented structured review (Fig. 1) revealed modelling as a common method to substantiate EFSA’s scientific opinions. Although rarely explicitly demanded, quantitative modelling was applied in more than a third of the studies included in the review. We limited the review to one particular system: the policy needs mandated to EFSA and the approaches EFSA working groups used to reply to the need. This system generates standardised documents reflecting policy demands and the respective scientific response. Although scientific policy support to animal health might be organised differently nationally or beyond the EU, we considered the EFSA documents to constitute an excellent source of examples for the process and outcome of applied scientific policy support. Unfortunately, EFSA opinions do not allow insight into the integration of scientific advice into policy. Thus, the final step in communication (EFSA, 2009) could not be assessed in this review. Perception and classification of the reviewed material naturally was subjective. The aim for objective insight was sought by assessing the modelling for EFSA opinions against established modelling theory such as standard models, parameterisation methods and analysis tech-
niques (Starfield et al., 1990; Philips et al., 2004; Saltelli et al., 2000; Grimm and Railsback, 2005; Bolker, 2008; EFSA, 2009). The objective of this review was to reveal how requests by animal health policy stakeholders translate into modelling methods used for response. A striking correspondence of modelling tasks and applied models was found (Fig. 4). To answer questions on the accuracy of diagnostic tests, data gathered in literature reviews were modelled by meta-analysis technique aggregating scientific evidence and expert knowledge. The particular approach found in the respective documents was developed in one EFSA opinion and applied in subsequent documents, apparently making it a quasi-standard. Tasks addressing the diagnostic efficacy of the monitoring and surveillance systems (MOSS) were tackled by calculating joined or combined probability; e.g., to detect a disease by a certain strategy (sensitivity, specificity, freedom of disease). These methods were presumably considered standard, because model description was brief in all documents. Slight differences could, therefore, not be retrieved. One MOSS question was addressed by a decision tree model. In contrast to the other examples, the model focused on wildlife species which are not readily accessible for diagnostics. Hence, sampling bias and uncertainty in sample access had to be considered. Such complexity would hamper the analytic tractability of compound probability calculation (but see e.g., Martin et al., 2007). End-point quantifications were the most common request that was treated by modelling. The common task was risk calculation. The prevailing model type was the decision tree model, likely because these allowed a formal structure to the hazard pathways. A different model category was chosen in only one case (mechanistic model). This assessment accounted for the dynamic allocation of control zones during the spread of a disease. Such spatio-temporal stochastic processes could not efficiently be represented by decision tree models. The application of the more complex model type was supported by the existence of a pre-developed model framework (EFSA-Q-2007-427).
66
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67
Astonishingly, there were only few requests explicitly addressing disease spread and contingency planning where the response used modelling. The few questions were all tackled by mechanistic models, probably because only this model category is capable of tracking temporally changing dynamics. The complexity of these few models varied strongly depending on the level of heterogeneity mapped from the biological system. The correspondence between the modelling task and the applied model category underpins that minimal capabilities have to be provided by the modelling to tackle certain tasks (Singer, 2010). But simultaneously that the formulation of the request influenced the quantitative tools applied while preparing an answer. There is the need of transparent communication of the purpose of the request to initiate the actually intended scientific assessment (EFSA, 2009). Clear specification of the requestor’s intention – e.g. understanding or prediction – will make it more likely that the outcome of the assessment is adequate. For example, a request for quantitative prediction where there is uncertainty or gaps in knowledge will promote decision tree models. Often the main advantage of these models comes from structuring knowledge and knowledge gaps around the pathway that is supposed to lead to an end-point event. However, the models were applied to calculate risk estimates for specific exposure scenarios. In this case necessary assumptions had to be made to get rid of knowledge gaps or uncertainties (including all side effects of being precisely predictive under uncertainty). Structured awareness of the weak and strong steps in a decision tree will be beneficial to making informed decisions, and sometimes may even satisfy the intended outcome from the requestor’s perspective. The more knowledge gaps that exist regarding a topic the less likely the strict request of an end-point estimate will lead to an exhaustively informative decision tree. This will instead lead to a formal model calculation during which the unknowns need to be overcome. Similarly, the current discussion in eco-toxicology is the need for ecology-based rather than end-point focused modelling when the consequences of things like pesticide treatment must be assessed to support regulatory decision making (Forbes et al., 2009; Thorbek et al., 2010). Interestingly, the purpose of the responding models often diverted from the purpose implied by the question. The purpose shift was likely caused by knowledge gaps. We found that the vast majority of end-point oriented studies could not be fully parameterised from known data (e.g. literature data). Several opinions provided a comparative analysis (e.g. explored uncertainty and sensitivity by presenting a list of scenarios). Hence, the pragmatic result was to focus on understanding the modelled biological system rather than predicting its ultimate outcome. Some EFSA opinions even demonstrated that limitations in available knowledge limited the predictive power of applied models. 5. Conclusion Our review showed how requests triggered modelling outcome, e.g. how seeking prediction of endpoint values promoted a dominance of structurally similar decision tree models. However, a limited knowledge base, revealed by
the majority of reviewed assessments, impeded precise endpoint estimates and, strictly speaking, the requested task could not be answered. However, the implicit effort to structure knowledge furthered the understanding of the decision problem and the management options, thus providing background for informed decision making. Scientific support for decision making is sought for complex situations where there is a range of options of action. These circumstances implicate incomplete knowledge and, as highlighted by our review, the predictive power of models often is limited. However, decisions have to be made no matter whether there will be complete scientific justification. Hence, providing background for informed decisions is already an important driver for purposeful modelling; the result could be the shift from intuition to evidence based decision making. Conflicts of interest statement None declared. Acknowledgements We thank the two anonymous reviewers for valuable comments and suggestions to improve the manuscript. Parts of data collection were funded by EFSA. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.prevetmed.2011.01.004. References Anderson, R.M., Jackson, H.C., May, R.M., Smith, A.M., 1981. Population dynamics of fox rabies in Europe. Nature 289, 765–771. Augustine, D.J., 1998. Modelling Chlamydia–koala interactions: coexistence, population dynamics and conservation implications. J. Appl. Ecol. 35, 261–272. Backer, J.A., Hagenaars, T.J., van Roermund, H.J.W., de Jong, M.C.M., 2009. Modelling the effectiveness and risks of vaccination strategies to control classical swine fever epidemics. J. Royal Soc. Interface 6, 849–861. Bolker, B.M., 2008. Ecological Models and Data in R. Princeton University Press, Princeton. EFSA, 2009. Panel on animal health & welfare (AHAW); guidance on good practice in conducting scientific assessments in animal health using modelling. EFSA J. 7 (12:1419), 38. Eisenberg, J.N.S., Brookhart, M.A., Rice, G., Brown, M., Colford, J.M., 2002. Disease transmission models for public health decision making: analysis of epidemic and endemic conditions caused by waterborne pathogens. Environ. Health Perspect. 110, 783–790. Eisinger, D., Thulke, H.-H., Müller, T., Selhorst, T., 2005. Emergency vaccination of rabies under limited resources – combating or containing? BMC Infect. Dis. 5, 10. Forbes, V.E., Hommen, U., Thorbek, P., Heimbach, F., van den Brink, P., Wogram, J., Thulke, H.-H., Grimm, V., 2009. Ecological models in support of regulatory risk assessments of pesticides: developing a strategy for the future. Integr. Environ. Assess. Manage. 5, 167–172. Garner, M.G., et al., 2010. Evaluating the effectiveness of early vaccination in the control and eradication of equine influenza—a modelling approach. Pre. Vet. Med. 99, 15–27. Grimm, V., Berger, U., Bastiansen, F., Eliassen, S., Ginot, V., Giske, J., GossCustard, J., Grand, T., Heinz, S., Huse, G., Huth, A., Jepsen, J.U., Jørgensen, C., Mooij, W.M., Müller, B., Pe’er, G., Piou, C., Railsback, S.F., Robbins, A.M., Robbins, M.M., Rossmanith, E., Rüger, N., Strand, E., Souissi, S., Stillman, R.A., Vabø, R., Visser, U., DeAngelis, D.L., 2006. A standard protocol for describing individual-based and agent-based models. Ecol. Modell. 198, 115–126.
A. Singer et al. / Preventive Veterinary Medicine 99 (2011) 60–67 Grimm, V., Railsback, S.F., 2005. Individual-Based Modeling and Ecology. Princeton University Press, Princeton. Grimm, V., Railsback, S.F., 2006. Agent-based models in ecology: patterns and alternative theories of adaptive behaviour. In: Billari, F., Fent, T., Prskawetz, A., Scheffran, J. (Eds.), Agent-based Computational Modelling—Applications in Demography, Social, Economic and Environmental Sciences. Physica, Heidelberg, pp. 139–152. Grimm, V., Revilla, E., Berger, U., Jeltsch, F., Mooij, W.M., Railsback, S.F., Thulke, H.-H., Weiner, J., Wiegand, T., DeAngelis, D.L., 2005. Patternoriented modeling of agent-based complex systems: lessons from ecology. Science 310, 987–991. Gross, J.E., Miller, M.W., 2001. Chronic wasting disease in mule deer: disease dynamics and control. J. Wildl. Manage. 65, 205–215. Hall, C.A.S., DeAngelis, D.L., 1985. Eco-forum: models in ecology: paradigms found or paradigms lost? B. Ecol. Soc. Am. 66, 339–346. Harvey, N., Reeves, A., Schoenbaum, M.A., Zagmutt-Vergara, F.J., Dubé, C., Hill, A.E., Corso, B.A., McNab, W.B., Cartwright, C.I., Salman, M.D., 2007. The north american animal disease spread model: a simulation model to assist decision making in evaluating animal disease incursions. Prev. Vet. Med. 82, 176–197. Hurd, H.S., Kaneene, J.B., 1993. The application of simulation models and systems analysis in epidemiology: a review. Prev. Vet. Med. 15, 81–99. Keeling, M.J., Woolhouse, M.E.J., May, R.M., Davies, G., Grenfell, B.T., 2003. Modelling vaccination strategies against foot-and-mouth disease. Nature 421, 136–142. Kramer-Schadt, S., Fernández, N., Grimm, V., Thulke, H.-H., 2009. Individual variation in infectiousness explains long-term disease persistence in wildlife populations. Oikos 118, 199–208. Kudahl, A.B., Østergaard, S., Sørensen, J.T., Nielsen, S.S., 2007. A stochastic model simulating paratuberculosis in a dairy herd. Prev. Vet. Med. 78, 97–117. Levin, S.A., Durrett, R., 1996. From individuals to epidemics. Phil. Trans. R. Soc. London B 351, 1615–1621. Lloyd-Smith, J.O., Cross, P.C., Briggs, C.J., Daugherty, M., Getz, W.M., Latto, J., Sanchez, M.S., Smith, A.B., Swei, A., 2005. Should we expect population thresholds for wildlife disease? Trends Ecol. Evol. 20, 511–519. Mangen, M.-J.J., Jalvingh, A.W., Nielen, M., Mourits, M.C.M., Klinkenberg, D., Dijkhuizen, A.A., 2001. Spatial and stochastic simulation to compare two emergency-vaccination strategies with a marker vaccine in the 1997/1998 Dutch classical swine fever epidemic. Prev. Vet. Med. 48, 177–200. Martin, P.A.J., Cameron, A.R., Greiner, M., 2007. Demonstrating freedom from disease using multiple complex data sources: 1. A new methodology based on scenario trees. Prev. Vet. Med. 79, 71–97. Murray, A.G., 2006. A model of the emergence of infectious pancreatic necrosis virus in Scottish salmon farms 1996–2003. Ecol. Modell. 199, 64–72.
67
Noordegraaf, A.V., Jalvingh, A.W., de Jong, M.C.M., Franken, P., Dijkhuizen, A.A., 2000. Evaluating control strategies for outbreaks in BHV1-free areas using stochastic and spatial simulation. Prev. Vet. Med. 44, 21–42. Philips, Z., Ginnelly, L., Sculpher, L., Claxton, K., Golder, S., Roiemsma, R., Woolacott, N., Glanville, J., 2004. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol. Assess. 8 (36). Saltelli, A., Chan, K., Scott, E.M., 2000. Sensitivity Analysis. Wiley, Chichesters. Schmolke, A., Thorbek, P., DeAngelis, D.L., Grimm, V., 2010. Ecological modelling supporting environmental decision making: a strategy for the future. Trends Ecol. Evol. 25, 479–486. Shirley, M.D.F., Rushton, S.P., Smith, G.C., South, A.B., Lurz, P.W.W., 2003. Investigating the spatial dynamics of bovine tuberculosis in badger populations: evaluating an individual-based simulation model. Ecol. Modell. 167, 139–157. Singer, A., 2010. External report reviewing the previous opinions of the Panel on Animal Health and Welfare concerning the application of quantitative tools, in the sequence of the current self mandate on “Good Practice in Conducting Scientific Assessments in Animal Health Using Modelling” (Question No EFSA-Q-2009-408)., http://www.efsa.europa.eu/en/scdocs/scdoc/43e.htm. Smith, G.C., Harris, S., 1991. Rabies in urban foxes (Vulpes vulpes) in Britain: the use of a spatial stochastic simulation model to examine the pattern of spread and evaluate the efficacy of different control regimes. Phil. Trans. R. Soc. London B 334, 459–479. Starfield, A.M., Smith, K.A., Bleloch, A.L., 1990. How to Model It. McGrawHill, Inc., New York, St. Louis, San Francisco. Szmaragd, C., Wilson, A.J., Carpenter, S., Mellor, P.S., Gubbins, S., 2010. The spread of Bluetongue virus serotype 8 in Great Britain and its control by vaccination. PLoS one 5, e9353. Taylor, N., 2003. 2003 Review of the use of models in informing disease control policy development and adjustment. 1286. Department for Environmental, Food, and Rural Affairs, UK., http:// epicentre.massey.ac.nz/resources/acvsc grp/docs/Taylor 2003.pdf. Thorbek, P., Forbes, V., Heimbach, F., Hommen, U., Thulke, H.-H., van den Brink, P.J., Wogram, J., Grimm, V., 2010. Ecological Models for Regulatory Risk Assessments of Pesticides. Society of Environmental and Chemistry (SETAC) and CRC Press, Pensacola and Boca Raton, FL. Thulke, H.-H., Eisinger, D., Beer, M., 2007. CSF emergency control – from the control ideal towards the ideal control , Presentation at AVIDmeeting, Riems. Tischendorf, L., Thulke, H.-H., Staubach, C., Müller, M.S., Jeltsch, F., Goretzki, J., Selhorst, T., Müller, T., Schlüter, H., Wissel, C., 1998. Chance and risk of controlling rabies in large-scale and long-term immunized fox populations. Proc. R. Soc. London B 265, 839–846.