Decision analysis: dealing with uncertainty in diagnostic testing

Preventive Veterinary Medicine 45 (2000) 139±162

Decision analysis: dealing with uncertainty in diagnostic testing R.D. Smitha,*, B.D. Slenningb a

Division of Epidemiology and Preventive Medicine, College of Veterinary Medicine, University of Illinois, Urbana, IL 61802, USA b Population Medicine Program, Department of Food Animal and Equine Medicine, College of Veterinary Medicine, North Carolina State University, Raleigh, NC 27606, USA

Abstract Decision analysis is a process for systematically analyzing complex choices by considering all pertinent information. In this paper, we discuss how uncertainty associated with diagnostic testing can be included in a decision analysis using pay-off tables and decision trees (decision-flow diagrams). Variables associated with diagnostic test interpretation (such as pre-test and post-test probability of disease; test sensitivity, specificity and predictive values; fixed cut-offs versus continuous measurement scales; test dependence associated with the use of multiple tests) are considered. Several decision criteria and output measures are discussed (including MAXIMIN and MAXIMAX criteria, opportunity costs, expected monetary values, expected utility, sensitivity and risk-profile analysis, and threshold analysis). The application of decision analysis to diagnostic testing for Johne's disease and traumatic reticuloperitonitis of cattle, and for canine heartworm disease are used to illustrate both population- and patient-oriented applications and criteria for ranking the desirability of different outcomes. # 2000 Elsevier Science B.V. All rights reserved. Keywords: Decision analysis; Pay-off table; Decision tree; Sensitivity analysis; Modeling; Veterinary; Multiple testing

1. Introduction For most choices in medicine, uncertainty with regard to diagnosis, prognosis, and cost of medical decisions occurs. The best option might not be readily apparent because of the interaction of a number of variables such as disease likelihood, test performance, and the *

Corresponding author. Tel.: ‡1-217-333-2449; fax: ‡1-217-244-7421. E-mail address: [email protected] (R.D. Smith) 0167-5877/00/$ ± see front matter # 2000 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 5 8 7 7 ( 0 0 ) 0 0 1 2 1 - 5

140

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

likelihood and desirability of possible outcomes. Decision analysis is a process for systematically analyzing complex choices by considering all pertinent information. Diagnostic testing is an important part of the medical decision-making process. General aspects of the application of decision analysis in veterinary and human medicine have been described (Fetrow et al., 1985; Kassirer et al., 1987; Pauker and Kassirer, 1987; Erb, 1988; Dargatz and Salman, 1990a,b; Clemen, 1995; Smith, 1995a, pp. 231±233; Lilford et al., 1998), as has the use of decision analysis to evaluate diagnostic tests (Smith, 1993, 1995a, pp. 239±242, 1995b). In this report, we use pay-off tables and decision trees to illustrate how uncertainties associated with diagnostic testing can be incorporated into a decision analysis. Alternate methods for evaluating outcomes will also be discussed. Throughout this report, we discuss testing in the context of disease detection, but the principles are equally applicable to detection of any condition (such as estrus, pregnancy, or contaminants in bulk-tank milk). Unless otherwise specified, terms such as ``diagnostic tests'' and ``diagnostic testing'' are used in the generic sense, whether they are being applied in a screening (low pre-test probability of disease) or diagnostic (high pre-test probability of disease) context. Definitions and common usage of terms related to diagnostic testing can be found in standard veterinary epidemiology texts and references (Martin et al., 1987; Smith, 1995a, Thrusfield, 1997; Vaillancourt et al., 1999). 2. Sources of uncertainty in diagnostic testing Several attributes of diagnostic tests and testing can be incorporated into a decision analysis. These are listed below along with a brief discussion of how they can be accommodated. More detailed examples will be provided in later sections. 2.1. Test properties and performance Test attributes such as sensitivity and specificity are generally considered to be absolute properties of a test, although exceptions occur as described in Section 2.3 below. They are usually treated as independent variables in decision-analysis models. Attributes such as positive and negative predictive values describe test performance under a specific set of conditions and are treated as dependent variables in decision-analysis models. Test properties and performance can be influenced by several factors related to the way tests are standardized and performed. Examples are cut-off values (for differentiating normal from abnormal), systematic and random errors inherent in the test procedure, and biologic heterogeneity of the population(s) being tested. Decision analyses should be performed over the range of possible values for the above parameters. 2.2. The clinical context The clinical context in which diagnostic testing is performed can influence several components of the decision-analysis problem (including the pre-test and post-test probability of disease, and credibility of test results). Pre-test probability (or likelihood)

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

141

of disease may be low when a test is being used to screen for inapparent infections in clinically normal individuals. Pre-test probability of disease is higher when the patient history and clinical signs are consistent with the disease in question. Pre-test probabilities may also be high for individuals that reside in geographic regions, or are members of populations, in which the incidence or prevalence of the disease in question is high. The pre-test probability of disease may decline dramatically over the course of a disease eradication program. Pre-test probability, test sensitivity and test specificity determine the predictive value of test results (e.g., the likelihood that a positive or negative test result is correct). Pre-test probability usually has a greater influence upon predictive values because it typically varies over a much-broader range than either test sensitivity or specificity. Bayes theorem of conditional probabilities deals with the general case of estimating the probability than an event will occur under a given condition. When applied to diagnostic testing, it is often used to estimate the positive predictive value of a test result (also referred to as the post-test probability of disease given a positive test result); Pr(D‡|T‡). Bayes' theorem can be expressed algebraically as Pr…D ‡ jT‡† ˆ ‰Pr…T ‡ jD‡† PŠ=‰PrT‡Š where Pr(D‡|T‡) is the post-test probability of disease; Pr(T‡|D‡) is the sensitivity of the diagnostic test; P is the prevalence, or pre-test probability of disease; PrT‡ is the likelihood of a positive test, or (sensitivity‡(1ÿspecificity)) (Kramer, 1988, p. 213; Smith, 1995a, pp. 44±45). Bayes' theorem is important because is describes (in mathematical terms) the relationship among pre-test and post-test probability of disease, test properties and results. Although the precise form of the algebraic expression might not be recognizable in a decision analysis, the relationships embodied in Bayes' theorem should always be represented whenever a decision analysis includes a diagnostic-testing component. 2.3. Changes of test properties with prevalence Test sensitivity may be associated with prevalence of infection. For example, animals living in regions that are hyperendemic for a disease may regularly be re-exposed to infection Ð resulting in elevated antibody titers from anamnestic responses. Antibody based tests in such a population may exhibit higher sensitivity than would be observed among infected animals in less-endemic regions (where antibody titers may decrease below the detection level of the test). Similarly, antigen-detection tests may be more sensitive among animals residing in areas highly endemic for a particular parasite Ð resulting in heavier parasite burdens (and antigen load) than their infected counterparts in less-endemic regions. The effect of variable test sensitivity can be evaluated in a decision analysis through sensitivity analysis over the likely range of variation. An example is given in Section 5.1.5. 2.4. Quantitative test results For test results that fall on a continuum (such as antibody titers, blood chemistries, and radiographic measurements), it may be more useful to perform the decision analysis using

142

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

test values as the input variable than a series of discrete sensitivity/specificity combinations. This can be accomplished by converting test sensitivity and specificity at each test value to the corresponding likelihood ratio, and then using the likelihood ratio to convert pre-test odds of disease to post-test odds in the decision-analysis model (Kramer, 1988, pp. 143±145; Smith, 1995a, pp. 55±58). A lookup table or equation (derived from curve-fitting) can be used in the model to convert test values to likelihood ratios (Smith, 1995a, p. 44). The conversion between probability of disease and odds of disease is straightforward (because the odds of disease is simply the ratio of the probability of disease to its complement). An example of using likelihood ratios in a decision-analysis model can be found in Section 5.2. 2.5. Issues in multiple testing Tests may be used in combination to improve the sensitivity (parallel testing; herd retest) or specificity (serial testing) of the overall test strategy (Smith, 1995a, pp. 58±64). The actual improvement in test performance may be less than expected due to dependence between test sensitivities or specificities. The degree of test dependence is expressed mathematically as the conditional covariance, and its use to revise sensitivities and specificities of test combinations is discussed in the paper by Gardner et al. (2000). An example of including conditional covariance of tests in decision analysis may be found in Section 5.3. 3. Elements of a decision-analysis problem As with any formal system, decision-analysis techniques can be broken down into a series of steps or elements. Each step builds upon the previous activities. Not all clinical problems will require all steps Ð but the general outline will serve in most instances.  Identify and bound the decision problem. Define alternative actions, possible future clinical states, and other (non-monetary) considerations in the context of the decision.  Structure the decision problem. Construct a decision diagram representing the logic and timing of the problem (e.g., the initial decision point, choices or options available, likelihood of probabilistic events, and relative desirability of the different outcomes.  Select decision criteria. Select the most appropriate means of choosing the preferred course of action (revenues, costs, utilities, efficiencies, risk), whether the decision is to choose a minimum or maximum, and method of analysis (non-probabilistic methods, pay-off tables, decision trees).  Choose a preferred course of action. Synthesis of structure and available information (quantification and sensitivity analysis) allowing for a conclusion to be drawn and for an interpretation of the outcome. Each of the above steps is illustrated in the examples that follow.

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

143

4. Decision structure one: pay-off tables Pay-off tables are often chosen in situations where there are few contingent events to account for (or which can be accounted for), for which there are both at least two possible states of nature and at least two plausible management options to take. Pay-off tables take the form of a matrix in which (by convention) the rows depict management options and the columns depict states of nature. Each individual cell of the matrix represents one combination of ``state of nature'' and ``management option''. The cell entries are the costs incurred (or revenues enjoyed) if that combination were to occur. 4.1. Example: screening for Johnes disease in a purebred beef herd The diagnostic challenge before us is how best to determine the status of Mycobacterium paratuberculosis infection (Johne's disease) in a purebred beef-cattle herd consisting of 173 adult animals (152 (88%) females and 21 (12%) males). There are three possible herd prevalence levels: (1) no infected animals (none), (2) 10% prevalence (low), or (3) 25% prevalence (high). Four tests are available: (1) the agar-gel immunodiffusion test (AGID), (2) the complement-fixation test (CF), (3) standard fecal-culture techniques (CULT), and (4) the enzyme-linked immunosorbent assay (ELISA). The sensitivity, specificity, and cost of performing each test differ. Baseline values and assumptions used in the problem are summarized in Table 1. The financial impact of replacing test-positive animals is great Ð particularly for falsepositive bulls. An additional cost of having infected animals in the herd (false negatives; FN) is the assumption that an infected animal will infect three other animals prior to detection and removal. 4.2. Representation of diagnostic uncertainty in the pay-off table To evaluate the testing programs for each potential herd prevalence, we first need to calculate the expected test outcomes (true positives or negatives and false positives or negatives) and their associated costs. One way to do this is to create a standard test outcome versus disease state two-by-two table for each diagnostic-test±prevalence combination Ð yielding 12 such tables. From those tables, the financial impacts can be calculated. Table 2 depicts test performance and overall net returns for each test±prevalence scenario using baseline values and assumptions from Table 1. It can be seen that the higher the prevalence of disease, the less net income the herd realizes (irrespective of which test is used). This makes intuitive sense. When disease is present, lower test sensitivity results in larger incurred costs than does lower test specificity. For example, at 10% prevalence incurred costs increase up to $28 602 because of decreasing test sensitivity (increasing false negatives) versus $9698 because of decreasing test specificity (increasing false positives). This is in part because the range in sensitivity (25±55%) is greater than that for specificity (95±100%). But which test is the best? According to the model, the CF testing option displays the lowest overall testing costs Ð but also shows the poorest overall performance across all three prevalence scenarios. The relative advantage among different tests varies with

144

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Table 1 Test characteristics, herd and market qualities, and assumptions used in the pay-off table-based Johne's diseasetesting decisiona

(1) Test characteristics (A) Agar-gel immunodiffusion (B) Complement fixation (C) Bacteriologic culture of feces (D) Enzyme-linked immunosorbent assay

(2) Herd and market qualities (A) Herd size (B) No. cows sold/year (C) No. bulls sold/year (D) Percent (E) Market value of Johne's-positive animal (F) Market value of Johne's-negative animal (G) Market value of false-positive animal (H) Market value of false-negative animal

(3) Assumptions (A) 1 FN transmits disease to XX animals before being found (B) Johne's exists in the herd at one of three possible levels (C) Disease is spread proportionately to either sex

Abbreviation

Sensitivity

Specificity

Costs

AGID CF CULT ELISA

0.35 0.25 0.45 0.55

1.00 0.95 1.00 0.99

$6.00 $4.00 $14.25 $4.75

Female

Male

Average animal

152 48 ± 88% $270

21 ± 12 12% $350

± ± ± ± $280

$1400

$5000

$1837

($1130)

($4650)

($1557)

($4200)

($15000)

($5511)

Value

Units

Label

3

Animals infected

0%

Herd prevalence

None

10% 25%

Herd prevalence Herd prevalence

Low High

a Test sensitivity and specificity values are estimates for stage I/II disease (subclinical); from Wren (1998). Test costs averaged from Wren (1998); Johne's Testing Center Website , 8/12/99; Iowa State University, Veterinary Diagnostic Laboratory Website; http//www.vdl.iastate.edu/ VDL/MainMenu/Fee_Table/feeALL.htm>, 8/12/99. ``Average animal'' is average value weighted for both proportion of herd by sex and differential value by sex. Values in parentheses are losses.

disease prevalence. The choice of ``best test'' is neither trivial nor intuitive Ð but is amenable to pay-off-table analysis. 4.3. Analysis of the pay-off table using different decision criteria The first step in performing a pay-off-table analysis is to establish the initial cost± revenue matrix. For our case the matrix will have three states of nature (the levels of

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Table 2 Calculation of two-by-two tables and financial outcomes for each of the 12 possible combinations of test±prevalence options for the pay-off table-based Johne's disease-testing decision

145

146

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Table 2 (Continued )

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

147

Table 3 Cost±revenue matrix for the pay-off table-based Johne's disease-testing decisiona Test

State of nature for prevalence in herd

AGID GF CULT ELISA a

None

Low

High

$316.8 $287.7 $315.3 $311.1

$224.7 $188.6 $233.3 $239.7

$86.6 $39.9 $110.2 $132.5

Developed from the revenue outcomes of Table 2; values are in thousands of dollars.

possible Johne's prevalence; 0, 10, and 25%), and four testing strategies (AGID, CF, CULT, ELISA). The entries into the matrix will, in our case, be the total system revenues for each of the testing strategy/prevalence scenarios developed in Table 2. The resulting cost/revenue matrix for this case is shown in Table 3. The second step in performing a pay-off-table analysis is to select a decision criterion. The most-commonly used standard criteria categories (Dijkhuizen et al., 1997) are demonstrated in the set of four examples below. 4.3.1. MAXIMIN and MAXIMAX criteria The MAXIMIN criterion takes a pessimistic or conservative risk attitude by assuming that the worst will happen. The selection process would be to choose across the testing strategies, selecting for each the cell displaying the worst gain or return. Then the analyst selects the strategy showing the least damaging of these worst-case scenario outcomes. In our example, the worst-case scenario for each testing strategy happens in the highprevalence state of nature (Table 3). The strategy with the best outcome in this state of nature is ELISA, with a revenue outcome of $132 500. ELISA, then, shows the maximum minimum gain; it is the MAXIMIN selection. For a MAXIMAX criterion-based choice, the table is set up identically Ð but this time strategies are rated based on their best-case scenarios. In this table all tests show the best outcomes under the zero prevalence state of nature. The strategy that does the best is the AGID strategy at an income of nearly $317 000. Hence, an optimist would follow the MAXIMAX criterion and would choose the AGID testing strategy, whereas the pessimistic MAXIMIN decision-maker would choose the ELISA testing strategy. The MAXIMIN and MAXIMAX criteria as applied to the Johne's testing strategies are summarized in Table 4. A disadvantage of MAXIMIN and MAXIMAX criteria is that they are simply ranking strategies; they are insensitive to relative differences. Thus, while the MAXIMAX choice above was for AGID at approximately $317 000, the CULT strategy was only $1500 less Ð a difference of less than 1%. Considering only absolute rankings (as do the MAXIMIN and MAXIMAX criteria) might not be the best decision criteria. 4.3.2. MINIMAX regret criterion The MINIMAX regret criterion takes a more-sophisticated, comparative, view of the decision process. This criterion has as its focus opportunity costs of alternative,

148

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Table 4 Demonstration of the MAXIMIN and MAXIMAX decision criteria for the pay-off table-based Johne's diseasetesting decisiona Test

Minimum return

Maximum return

MAXIMIN

MAXIMAX

AGID CF CULT ELISA

$86.6 $39.9 $110.2 $132.5

$316.8 $287.7 $315.3 $311.1

± ± ± $132.5

$316.8 ± ± ±

a These criteria rank outcomes for worst-case and best-case scenarios, respectively (based on data in Table 3; values are in thousands of dollars).

suboptimal strategies. It opts for the strategy that minimizes the opportunity cost (regret) of not choosing an optimal strategy. Thus, if the MAXIMIN criterion is for pessimists, then the MINIMAX regret criterion is for analysts who not only think the worst-case will occur, but also wish to consider the consequences of making the wrong choice. Opportunity costs are calculated by first selecting the best outcome by state of nature Ð not by strategy as was done in the previous criteria. For instance, for the ``low'' prevalence state of nature, the best outcome is attained by choosing the ELISA strategy. It represents the best we can do, given disease exists in the herd at 10% prevalence. Were, we have mistakenly chosen to test with AGID, for example, we would not ``lose'' $224 700, we would have only lost the difference between AGID and ELISA. In other terms, we would have achieved a net revenue of $15 000 less ($239.7ÿ$224.7ˆ$15) Ð making this the opportunity cost for choosing to test with AGID in a low prevalence situation. The decision-maker establishes the best choice for each state of nature, and calculates the opportunity costs for each of the suboptimal choices relative to the best choice. Because this generates costs or losses (no matter whether the initial matrix were in the form of revenues, costs, or a mixture thereof) the analyst then selects the strategy with the minimum opportunity cost. In our case, the ELISA testing strategy displays the lesser of the maximal opportunity costs, and would therefore be the choice when using the MINIMAX regret criterion. The process is demonstrated in Table 5. None of the preceding criteria (MAXIMIN, MAXIMAX, MINIMAX regret) took into account the differing odds of occurrence for each state of nature. That is, they do not Table 5 Demonstration of the MINIMAX regret decision criterion for the pay-off table-based Johne's disease-testing decisiona Test

AGID CF CULT ELISA a

State of nature for prevalence in head None

Low

High

± $(29.0) $(1.4) $(5.7)

$(15.0) $(51.1) $(6.4) ±

$(45.9) $(92.6) $(22.3) ±

Maximum regret

MINIMAX regret

$(45.9) $(92.6) $(22.3) $(5.7)

± ± ± $(5.7)

This criterion considers the opportunity cost potential of selecting a suboptimal strategy for each state of nature (e.g., the costs associated with making the wrong choice; values are in thousands of dollars).

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

149

consider the possibility that one state of nature is more likely to occur than another. In fact, by not explicitly considering the odds of occurrence for the states of nature, the above analyses made the implicit assumption that they were equally likely. However, if the high prevalence only occurred in 1% of herds, or if the likelihood of a low prevalence were 75%, it would not make sense to ignore the relative a priori probabilities of the disease states. This is where the next criterion comes into play. 4.3.3. Likelihood or expected monetary value criterion In this criterion, the decision-maker is willing to estimate the odds of the individual states occurring. Each cell's value is multiplied by its state's probability of occurrence to derive an expected monetary value (EMV) for that cell (for example, if the ``none'' state of nature were estimated to be 90% likely, the entry into the AGID cell for ``none'' would be 0.9$316.8ˆ$285.1). All cells are calculated this way. Then, the EMVs for each strategy are added and the strategy with the best total EMV (i.e., lowest cost or highest revenue) is chosen. The advantage that EMVs bring to the decision process is that they allow a summary valuation of a strategy to be weighted both for the magnitude of revenue or cost as well as for the likelihood of the given event to occur. Therefore, unlikely yet high-magnitude outcomes will carry an impact on the decision proportional to an outcome that is highly likely, yet of low magnitude. It can be thought of as bringing in the concept of risk to an analysis. This is a powerful addition to the decision process. Assume that (by literature review, referral laboratory database, or by local knowledge) our best estimate for the probability for each of the three states of nature was: none, 40%; low, 50%; high, 10%. After making the separate EMV calculations as shown in Table 6, the best total EMV occurs with the ELISA, at an EMV of $257 600. Note that the EMV of CULT is $253 800 which is only 1.5% less than the EMV for ELISA. Also, the EMV of AGID is also relatively close (approximately 3.8% less than the EMV for ELISA). However, as with the MAXIMIN and MAXIMAX criteria, a straightforward likelihood or EMV criterion only ranks outcomes; it ignores relative differences. We can, though, offer the same addition to a likelihood or EMV criterion that we offered for the MINIMAX regret criteria: We can add in the opportunity cost perspective by generating regret tables, as in the next pay-off table. Table 6 Demonstration of the likelihood decision criterion for the pay-off table-based Johne's disease-testing decisiona Test

Likelihood AGID CF CULT ELISA a

State of nature for prevalence in herd None

Low

High

0.40 $126.7 $115.1 $126.1 $124.4

0.50 $112.4 $94.3 $116.7 $119.9

0.10 $8.7 $4.0 $11.0 $13.3

EMV

MAX EMV

$247.8 $213.4 $253.8 $257.6

± ± ± $257.6

This decision criterion considers the differing likelihoods of occurrence for each state of nature by generating EMVs. Values are in thousands of dollars.

150

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Table 7 Demonstration of the likelihood regret decision criterion for the pay-off table-based Johne's disease-testing decisiona Test

AGID CF CULT ELISA

State of nature for prevalence in herd None

Low

High

± $(11.6) $(0.6) $(2.3)

$(7.5) $(25.6) $(3.2) ±

$(4.6) $(9.3) $(2.3) ±

Maximum likelihood regret

MINIMAX likelihood regret

$(7.5) $(25.6) $(3.2) $(2.3)

± ± ± $(2.3)

a This criterion combines the EMVs of the likelihood process with the consideration of opportunity costs from the regret criterion. Values are in thousands of dollars.

4.3.4. Likelihood regret or EMV regret criterion In this, we will consider the last pay-off table criterion. We have the most sophisticated perspective on a given choice. It allows us to account for three issues at once: (1) magnitude of effect, (2) likelihood of occurrence, and (3) opportunity costs of suboptimal choices. A likelihood regret analysis starts with the likelihood table as generated in Table 6. Then, we follow the same process as outlined under the simple MINIMAX regret criterion section, above. The strategy with the lowest regret EMV is selected as the best strategy. This is demonstrated in Table 7 where the ELISA testing strategy again emerges as superior to the others. Decision-makers should be aware that when outcomes are this close, the decision process is not perfect; we are close to being indifferent between the CULT strategy and the ELISA strategy. When this occurs, it is best to perform a sensitivity analysis on the ``soft'' numbers to be certain that the decision is stable. Sensitivity analysis is discussed in Section 5. 4.4. Problems with pay-off tables There are other ways of approaching a probability based decision using pay-off tables; break-even point analysis is one. Another is to apply disease frequency ranges and define those ranges where one strategy is the best. Unfortunately, with these methods the decisionmaker is still left making potentially ad hoc estimates of the most likely state of nature. Pay-off tables, by virtue of the way they are constructed, tend to hide assumptions and relationships. Further, they do not assist the decision-maker in planning the logistics (e.g., What do I do first? What actions do I need to have finished before I can begin this step of the process?). It is better to leave pay-off tables at this point and move to the other major method of decision analysis, for it accounts for internal probabilities and the above concerns to a greater degree than pay-off tables. The second tool used in decision analysis is the decision tree. 5. Decision structure two: decision trees A decision tree (decision-flow diagram) is a graphical depiction of the logical and temporal sequence of events involved in a decision-making process. Decision trees

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

151

consist of nodes and branches, with smaller branches representing consequences of the larger branches from which they originate. By convention, decision trees are read from left to right. The first branch point (``decision node'') represents the decision being analyzed. Chance nodes represent events that are at least partially determined by chance (such as the likelihood that disease is present or that a test result is correct). A terminal node represents a final outcome with no further significant options or consequences. Values for both chance and terminal nodes may be treated as variables to evaluate their effect upon the optimal decision. Once a decision tree is constructed, it can be analyzed by techniques for fold-back of the tree, sensitivity analysis and risk-profile analysis. Decision trees can be constructed and analyzed with spreadsheets or specialized software. All of the following examples were created and analyzed with D-Maker (Digital Medicine, 39 South Main St., Hanover, NH 03755). 5.1. Example: clinical utility of canine heartworm testing At least 14 other diseases share the common clinical findings of cough, exercise intolerance, and abnormal heart and lung sounds characteristic of canine heartworm (Dirofilaria immitis) infections (CONSULTANT veterinary diagnostic support system, College of Veterinary Medicine, Cornell University, Ithaca, NY 14853 ). In the following example, we use decision tree analysis to evaluate the clinical utility of an ELISA for circulating D. immitis antigen in the differential diagnosis of heartworm infection in dogs. For a more complete analysis that includes additional diagnostic perspectives (see Smith, 1995b). 5.1.1. Structure of the heartworm decision tree The heartworm decision tree (Fig. 1) begins with three possible decisions by the clinician:  Never treat for heartworm (rule outs). Treat the patient for one of the other diseases on the differential list. This option might be chosen by a clinician who considers the likelihood (pre-test probability) of heartworm disease to be too low to warrant heartworm diagnostic testing. Some of these patients will actually have heartworm infections (heartworm) while others will not (other disease) Ð depending on the pretest probability, or prevalence, of heartworm infection in the patient population (P). Probability of improvement for dogs correctly treated for non-heartworm disease is fixed at 0.67 throughout the decision tree, equivalent to that for dogs correctly treated for heartworm disease. Inappropriate treatment of heartworm-infected dogs (heartworm) will result in spontaneous improvement of only 5% of patients (the same as that for non-heartworm disease dogs receiving heartworm therapy elsewhere in the tree). Throughout the decision tree, patient improvement (recovery) is assigned a utility of 1.0, whereas failure to improve (no recovery) or death from an adverse reaction to heartworm therapy (complications) are assigned a utility of 0.  Always treat for heartworm (Htwm Rx). This option might be chosen by a clinician who considers the likelihood (pre-test probability) of heartworm disease to be so high that heartworm testing is unnecessary. The possible outcome scenarios are similar to

152

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Fig. 1. Decision tree for clinical management of a dog suspected of having occult heartworm disease. The decision tree is shown with baseline probability and utility values. Probabilities (because of constraints imposed by the decision-analysis software, the following names for variables are used in Figs. 1 and 5: pDˆpre-test probability of disease; pTposˆprobability of a positive test result; pDTposˆprobability of disease given a positive test; pDTnegˆprobability of disease given a negative test) appear beneath each branch and expected utilities beneath each node of the tree.

those for the ``rule outs'' branch with the exception that some dogs will die from complications of heartworm therapy. The probability of complications is higher for heartworm-infected (0.20) than uninfected (0.05) due to the combined effect of drug toxicity and parasite death in the former.  Let heartworm-test results guide case management (Htwm test). Animals testing positive will be treated for heartworm infection, while test-negative animals will be treated for the most-likely disease other than heartworm on the differential list. The

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

153

Table 8 Baseline probabilities and utilities used in the heartworm decision tree Event

Probability

Utility

Pre-test probability of heartworm infection

0.067

±

Positive test result In heartworm infection (sensitivity) In non-heartworm disease (false-positive proportion)

0.960 0.000

± ±

Negative test result In heartworm infection (false-negative proportion) In non-heartworm disease (specificity)

0.04 1.000

± ±

Complications from heartworm therapy In heartworm-infected dogs In non-heartworm disease

0.20 0.05

0 0

Recovery without severe sequelae Specific therapy for heartworm infection In untreated heartworm infection Specific therapy of non-heartworm disease Untreated non-heartworm disease

0.67 0.05 0.67 0.05

1 1 1 1

Failure to recover or death of the patient

±

0

remaining tree structure for these two test outcomes is identical to those for the ``Htwm Rx'' and ``rule outs'' decision branches, respectively; the exception is that the probability that the patient is afflicted with heartworm or non-heartworm disease (the post-test probability of disease) will be influenced by the test result. The relationship among pre-test probability of disease, test results, and post-test probability of disease are discussed in the next section. Baseline probabilities and utilities for the heartworm decision tree are listed in Table 8. A baseline pre-test probability of heartworm disease (P) of 1/15 (0.067) has been chosen to reflect the fact that heartworm is but one of at least 15 diseases that can cause a similar clinical presentation in dogs. ELISA sensitivity (Se) and specificity (Sp) have been set at 0.96 and 1.00 Ð reflecting test properties in clinically ill dogs where worm burdens (and circulating antigen) are likely to be highest (Courtney and Zeng, 1993; Courtney et al., 1993). 5.1.2. Representation of diagnostic uncertainty in the heartworm decision tree The diagnostic test node (Htwm test) gives rise to two possible outcomes: ``positive test'' and ``negative test''. The probability of a positive test (PrT‡) is given by the Bayesian equation: PrT‡ ˆ …Se P† ‡ …1 ÿ Sp† …1 ÿ P† The probability of a negative test result is its complementary probability: (1ÿPrT‡). Each test result can be either correct (true positives and true negatives) or incorrect (false positives and false negatives). The probability that a test result will fall into one of

154

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

these categories is determined by the pre-test probability of heartworm infection (P), and by the sensitivity (Se) and specificity (Sp) of the test. The respective Bayesian equations (which are also associated with the diagnostic-test node of the decision tree) are:    

probability probability probability probability

of of of of

a a a a

true positive ‰Pr…D ‡ jT‡†Š ˆ …Se P†=…PrT‡†, false negative ‰Pr…D ‡ jTÿ†Š ˆ ……1 ÿ Se† P†=…1 ÿ PrT‡†, false positive ˆ 1 ÿ Pr…D ‡ jT‡†, true negative ˆ 1 ÿ Pr…D ‡ jTÿ†.

The expressions Pr(D‡|T‡) and (1ÿPr(D‡|Tÿ)) correspond to the positive and negative predictive values of the test, respectively (as discussed in Section 2.2). 5.1.3. Fold-back of the heartworm decision tree; what is the best strategy under baseline conditions? In a fold-back, the expected utility for each decision is calculated by adding the values obtained when the utility of each possible outcome of that decision (terminal node) is multiplied by the probability that the outcome will occur. Every fold-back starts from some node in the tree, which is referred to as the root node for the fold-back. In most cases, the root node for a fold-back is a decision node. The expected utility expresses the average utility of each management option when that option is chosen for a large number of animals. The management option with the highest expected utility is usually the option of choice. Fold-back of the heartworm decision tree using baseline values (Fig. 1) reveals that reliance on the diagnostic test (``Htwm test''; expected utilityˆ0.660) provides the highest expected utility, followed closely by empirical treatment for other diseases on the differential list (``rule outs''; expected utilityˆ0.628). Empirical treatment for heartworm disease (``Htwm Rx''; expected utilityˆ0.080) would not be a justifiable alternative. The expected utility does not express how likely each result is. One might be more concerned about reducing the likelihood of a particular adverse outcome (such as death of the patient) than obtaining the highest expected utility. The overall probabilities of terminal branches (Kramer, 1988, pp. 227±228) express the probability of occurrence of each possible outcome of a particular decision in a decision tree. Starting at the root node, probabilities for each outcome are multiplied consecutively along to each terminal node. The resulting probabilities can be compared to finding the decision associated with the lowest risk of an unfavorable outcome. Probabilities for an identical outcome (such as death) from a particular decision path can be summed Ð yielding a risk-profile analysis for each possible outcome. It is important to note that neither overall probabilities nor risk-profile analysis require that the actual values of utilities be assigned. This is useful in cases when the desirability of different outcomes can be ranked Ð but when it is difficult to assign actual values to them. The high expected utility for treatment for other diseases without heartworm testing Ð despite the high sensitivity and specificity of the heartworm test Ð demonstrates the important role that the prior probability of disease plays in determining the predictive values and utility of diagnostic tests. Under baseline conditions, the heartworm ELISA yielded a positive predictive value [Pr(D‡|T‡)] of 1.00 Ð but because of the low prior probability of heartworm disease, only 6.4% of dogs would test positive.

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

155

Consequently, the best overall strategy would tend to favor pursuing other diseases on the differential list. This analysis is somewhat misleading in that none of the other 14 diseases on the differential list has a prior probability greater than that of heartworm, 0.067. Thus, the clinician electing to treat a disease other than heartworm without testing would still have to choose which one without any increase in the probability of being correct. Furthermore, any additional pertinent history about the patient (such as being an outdoor dog that resides in a heartworm-endemic region but is not on preventive medication) would increase the prior probability of heartworm disease (P) and the expected utility of performing the heartworm ELISA. The effect of pre-test probability of disease on the desirability of heartworm testing can be explored using one-way sensitivity analysis. 5.1.4. One-way sensitivity and threshold analysis; how does pre-test probability affect interpretation of test results? Sensitivity analysis (which evaluates the degree of confidence one can have in a particular decision) is simply a series of fold-backs over a range of values for one or more variables. One-way sensitivity analysis is used to calculate the changes in expected utility that occur when the value for only one variable is varied. One-way sensitivity analysis of the heartworm decision tree (Fig. 2) indicates that reliance on ELISA results (Htwm test) is the preferred strategy from a pre-test probability of 0 (testing threshold) to a pre-test probability of 0.97 (treatment threshold). This means that testing is the preferred strategy unless the pre-test probability of heartworm disease (based on history and other clinical and laboratory findings) exceeds 0.97. Because none of these findings is pathognomonic for heartworm disease, testing would always be the best strategy. 5.1.5. Two-way sensitivity analysis Ð is testing still the best strategy at lower test sensitivities? Two-way sensitivity analysis (in which two values are varied simultaneously) results in a series of thresholds (``break-even points'') at which the expected utility for each

Fig. 2. One-way sensitivity analysis of the heartworm decision tree to determine the effect of pre-test probability of heartworm disease on the interpretation of test results. When prevalenceˆ0, expected utilities are: Htwm Rx, 0.048; Htwm test, 0.0670; rule outs, 0.670.

156

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Fig. 3. Two-way sensitivity analysis of the heartworm decision tree to determine the effect of test sensitivity (as a reflection of worm burden) and pre-test probability of heartworm disease on the interpretation of test results.

decision is equal. The resulting curves are referred to as ``indifference'' or ``break-even'' curves (Madison et al., 1984; Fetrow et al., 1985; Dargatz and Salman, 1990b). Threshold values indicate whether a change in a given variable would change the optimal decision (i.e., would result in a different management option being the option of choice) Ð but do not indicate how much would be gained or lost by choosing a given management option. The sensitivity of heartworm ELISAs which detect circulating antigen may decline to approximately 0.50 when only one or two worms are present (Courtney and Zeng, 1993; Courtney et al., 1993). The effect of the interaction of test sensitivity and prior probability of disease upon the preferred strategy in the heartworm decision tree is depicted in Fig. 3. The pre-test probability of heartworm disease would have to exceed 0.70 before empirical treatment for heartworm disease becomes a justifiable alternative to diagnostic testing. Most clinically affected dogs carry higher worm burdens and test sensitivities in these dogs are 0.95. Under these conditions, the pre-test probability of heartworm disease would have to exceed 0.95 before empiric therapy would be the preferred strategy. These results support reliance upon heartworm-antigen testing for case management Ð even when worm burdens are likely to be low. 5.2. Example: are total plasma protein concentrations useful in the differential diagnosis of bovine traumatic reticuloperitonitis? Total plasma protein (TPP) concentrations are elevated in cattle with traumatic reticuloperitonitis (TRP) (Dubensky and White, 1983). Furthermore, the likelihood that the patient is suffering from TRP increases with the concentration of TPP. The question, then, is whether TPP concentrations have any clinical utility when applied to the individual patient. A decision analysis of this question differs from the heartworm example in that there is no fixed positive±negative cut-off. Rather, test properties change with TPP concentration. The key to performing a decision analysis in this case is to convert TPP concentrations in TRP-affected and -unaffected cattle to their corresponding sensitivity and specificity

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

157

values. This can be done by converting the pre-test probability of TRP (P) to odds, and then use the likelihood ratio for a positive test (LR‡) at each TPP concentration to convert pre-test odds to post-test odds (and thereby probability) of disease as described in Section 2.4. A regression equation was fitted to the relationship between TPP concentration and LR for TRP. This relationship was then incorporated into the diagnostic-testing nodef decision analysis of the benefit of using TPP concentration to guide case management, versus empiric treatment for TRP or for other diseases on the differential list. The pre-test probability of TRP in the population under study was 0.373. Theiagnostic-test node has been simplified by assigning true positives a utility of 1 and false positives a value of 0. The corresponding equations associated with the testing node are:    

likelihood ratio for a positive test (LR‡)ˆe(0.046 TPPÿ2.927), pre-test odds of TRP (PreTPPOdds)ˆP/(1ÿP), post-test odds of TRP (PostTPPOdds)ˆPreTPPOdds LR‡, post-test probability of disease Pr(D‡|T‡)ˆPostTPP Odds/(PostTPOdds‡1).

This diagnostic-test node differs from corresponding nodes in other decision trees (Figs. 1 and 6) in that test-positive and test-negative branches are not necessary. Any value for TPP is a positive test result. Note that the post-test probability of disease Pr(D‡|T‡) is the ``positive'' predictive value of a positive test (e.g., of a measured concentration of TPP). A one-way sensitivity analysis of the relationship between TPP concentration and positive predictive value of TRP is depicted in Fig. 4. The positive predictive value of the test increased from 0.39 to 0.83 over the range of TPP concentrations (65±110 g/l) measured by the investigators. The positive predictive value is low Ð even at the highest TPP concentration. However, test performance is only a part of the clinical utility of a diagnostic test. A complete decision analysis (incorporating revenues, costs, risks and prognoses) confirmed that TPP was of little value in the management of suspected TRP cases (Smith, 1993).

Fig. 4. One-way sensitivity analysis of the diagnostic-test node of a decision tree depicting the positive value of TPP concentration for the diagnosis of TRP in cattle.

158

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

5.3. Example: does multiple testing improve Johnes disease detection? This example illustrates how multiple test strategies (Section 2.5) with conditionally dependent tests can be evaluated using decision analysis. In this case, cattle are to be tested prior to purchase and only test-negative animals are allowed to enter the herd. The objective is to identify the test strategy that yields the highest possible negative predictive value. The decision tree depicted in Fig. 5 is designed to compare test performance when an ELISA (Allied Laboratories; Seˆ0.59, Spˆ0.95) and complement-fixation test (CF; Seˆ0.39, Spˆ0.99) are used alone or in parallel or serial testing schemes for the detection of infected cattle. Prevalence (pre-test probability) of infection among cattle being considered for purchase is assumed to be 0.02. Data on test properties and covariance between tests are from the paper by Gardner et al. (2000). To facilitate the analysis, true-negative and true-positive test results are assigned a utility of 1, and falsenegative and false-positive test results are given a utility of 0. Thus, the expected utilities for the negative- and positive-test nodes are equal to the negative and positive predictive values of respective tests. The expected utility for each test strategy is equal to its accuracy. Equations used in the diagnostic-test nodes to calculate the probability of a positive test result (PrT‡), probability of a true positive Pr(D‡|T‡) and probability of a false negative Pr(D‡|Tÿ) are the same as used in the heartworm tree. When used together, the overall sensitivity and specificity of the test strategy (parallel or serial) must be calculated Ð taking into account the effect of test covariance. The corresponding equations for sensitivity and specificity of multiple-test strategies follow (Gardner et al., 2000):  Parallel-testing strategy  Sensitivity (Se)ˆ1ÿ(1ÿSe1) (1ÿSe2)ÿCovSe,  Specificity (Sp)ˆ(Sp1 Sp2)‡CovSp.  Serial-testing strategy  Sensitivity (Se)ˆSe1 Se2‡CovSe,  Specificity (Sp)ˆ1ÿ(1ÿSp1) (1ÿSp2)ÿCovSp, where CovSe and CovSp are the sensitivity and specificity covariances between tests, respectively. Fold-back of the tree reveals that a parallel-testing strategy (0.992) and ELISA testing alone (0.991) yield the highest negative predictive values. Negative predictive values for all test strategies decline as prevalence increases Ð but the parallel-testing strategy still yields the highest negative predictive value. If both false positives and false negatives are to be averted, then text accuracy should be maximized. At low prevalences, ELISA and parallel testing have lower test accuracies than do either CF or serial-testing strategies (Fig. 6). Test accuracies decrease as prevalence increases Ð and at a threshold prevalence of approximately 0.20, ELISA and parallel testing become the most accurate testing strategies. The relative importance of differences in test performance will depend on the objectives of the testing program and the costs of false-positive and false-negative test results. These issues can be explored by assigning appropriate values to the utilities of terminal nodes.

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

159

Fig. 5. Decision tree designed to evaluate single and multiple test strategies for detection of Johne's disease (Mycobacterium paratuberculosis infection) in cattle.

160

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Fig. 6. One-way sensitivity analysis of the accuracy of single and multiple test strategies for detection of Johne's disease (Mycobacterium paratuberculosis infection in cattle over a range of pre-test probabilities of disease.

6. Discussion and conclusions Decision analysis is a useful method for evaluating the effect of uncertainty in medical decision-making. The interpretation of diagnostic-test results is part of this uncertainty and should be included in any decision analysis problem that includes diagnostic testing. In this paper, we compared two basic approaches to decision analysis, and provided examples of how some of the most-common issues in diagnostic testing can be analyzed. The assignment of utility values to the possible outcomes of a decision analysis can be difficult Ð particularly when quality of life issues are at stake. In these cases, comparison of the overall probability of different outcomes may be useful. We used examples that express utilities in terms of economics or prognoses for recovery. Pay-off tables provide a straightforward way of structuring and evaluating a decision analysis problem. However, their rigid structure can make analysis of more-complex decisions difficult. The logic and complexity of the heartworm decision tree would have been difficult to incorporate into a pay-off table. Decision-tree analysis of diagnostic test strategies is most-commonly used in veterinary medicine to evaluate decisions at the population level for topics such as disease-control programs in food animals (Carpenter et al., 1987; Eloit et al., 1990; Collins and Morgan, 1991; Gardner et al., 1996), optimal cut-offs for diagnostic tests (Ridge and Vizard, 1993), herd reproductive programs (Mohammed et al., 1990; Oltenacu et al., 1990; Galligan et al., 1991; Slenning, 1994) and milk-residue testing programs (Slenning and Gardner, 1997). In recent years, decision analysis has begun to be used to evaluate diagnostic decision-making at the individual-patient level (Ducharme et al., 1989; Smith, 1995b; Knox et al., 1998). Throughout this paper, we have treated test sensitivity, specificity, and disease prevalence as fixed, known values. Even during sensitivity analysis, the effect of variability was evaluated deterministically (e.g., no matter how many times the model was run we would have obtained the same results). In reality, values for these parameters may vary unpredictably with stage and severity of disease, parasite load, environmental or management factors. In these cases, it may be desirable to make a number of stochastic

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

161

runs of the model and allow values for these parameters to vary randomly over the range of possible values. The result would be a profile of the range and likelihood of possible outcomes Ð which may give the decision-maker a better idea of risks associated with each decision. This might be important when comparing two kinds of diagnostic tests for the same disease (e.g., fecal culture and an antibody based test for Johne's disease) where the range of sensitivity and specificity could be quite different for the two different test types. An emerging application of decision analysis in human medicine is as a means to translate evidence-based medical data into clinical guidelines for categories of patients or scenarios. Policy can be based on those management strategies that result in the highest expected utility and the lowest likelihood of unfavorable outcomes (Lilford et al., 1998). This approach Ð which is already being applied to risk assessment for food safety and foreign-pathogen introduction Ð has great potential in other areas of veterinary medicine as well.

References Carpenter, T.E., Berry, S.L., Glenn, J.S., 1987. Economics of Brucella ovis control in sheep: computerized decision-tree analysis. J. Am. Vet. Med. Assoc. 190, 983±987. Clemen, R.T., 1995. Making Hard Decisions: An Introduction to Decision Analysis. Wadsworth, Belmont, CA, 752 pp. Collins, M.T., Morgan, I.R., 1991. Economic decision analysis model of a paratuberculosis test and cull program. J. Am. Vet. Med. Assoc. 199, 1724±1729. Courtney, C.H., Zeng, Q.-Y., 1993. Sensitivity and specificity of two heartworm antigen tests. Can. Pract. 18, 20±22. Courtney, C.H., Zeng, Q.-Y., MacKinnon, B.R., 1993. Comparison of two antigen tests and the modified Knott's test for the detection of canine heartworm at different worm burdens. Can. Pract. 18, 5±7. Dargatz, D.A., Salman, M.D., 1990a. Decision tree analysis and its value to herd health. Comp. Cont. Educ. Pract. Vet. 12, 433±439. Dargatz, D.A., Salman, M.D., 1990b. The use of break-even curves in decision making. Comp. Cont. Educ. Pract. Vet. 12, 1347±1351. Dijkhuizen, A.A., Huirne, R.B.M., Hardaker, J.B., 1997. Scope and concepts of risky decision making. In: Dijkhuizen, A.A., Morris, R.S. (Eds.), Animal Health Economics. University of Sydney, pp. 135±148. Dubensky, R.A., White, M.E., 1983. The sensitivity, specificity and predictive value of total plasma protein in the diagnosis of traumatic reticuloperitonitis. Can. J. Comp. Med. 47, 241±244. Ducharme, N.G., Pascoe, P.J., Lumsden, J.H., Ducharme, G.R., 1989. A computer-derived protocol to aid in selecting medical versus surgical treatment of horses with abdominal pain. Equine Vet. J. 21, 447±450. Eloit, M., Lamy, F., Glevarec, M., Galaup, F., Turmel, R., Vigouroux, A., Benet, J.J., Toma, B., 1990. Detection of bovine leukemia virus antibodies in bulk tank milk using an ELISA test: improvement of the predictive value of results by repeated testing. Biologicals 18, 19±23. Erb, H.N., 1988. The benefit±cost analysis of disease control programs. Vet. Clin. North. Am. Food. Anim. Pract. 4, 169±181. Fetrow, J., Madison, J.B., Galligan, D., 1985. Economic decisions in veterinary practice: a method for field use. J. Am. Vet. Med. Assoc. 186, 792±797. Galligan, D.T., Ramberg, C., Curtis, C., Ferguson, J., Fetrow, J., 1991. Application of portfolio theory in decision tree analysis. J. Dairy Sci. 74, 2138±2144. Gardner, I.A., Carpenter, T.E., Leontides, L., Parsons, T.D., 1996. Financial evaluation of vaccination and testing alternatives for control of parvovirus-induced reproductive failure in swine. J. Am. Vet. Med. Assoc. 208, 863±869.

162

R.D. Smith, B.D. Slenning / Preventive Veterinary Medicine 45 (2000) 139±162

Gardner, I.A., Stryhn, H., Lind, P., Collins, M.T., 2000. Conditional dependence between tests affects the diagnosis and surveillance of animal diseases. Prev. Vet. Med. 45, 107±122. Kassirer, J., Moskowitz, P.A.J., Lau, J., Pauker, S.G., 1987. Decision analysis: a progress report. Ann. Internat. Med. 106, 275±291. Knox, K.M., Reid, S.W., Irwin, T., Murray, M., Gettinby, G., 1998. Objective interpretation of bovine clinical biochemistry data: application of Bayes law to a database model. Prev. Vet. Med. 33, 147±158. Kramer, M.S., 1988. Clinical Epidemiology and Biostatistics. Springer, New York, 286 pp. Lilford, R.J., Pauker, S.G., Braunholtz, D.A., Chard, J., 1998. Decision analysis and the implementation of research findings. Br. Med. J. 317, 405±409. Madison, J.B., Fetrow, J., Galligan, D., 1984. Economic decisions in food animal practice: to treat or not to treat. J. Am. Vet. Med. Assoc. 185, 520±521. Martin, S.W., Meek, A.H., Willeberg, P., 1987. Veterinary Epidemiology: Principles and Methods. Iowa State University Press, Ames, 356 pp. Mohammed, H.O., Loefler, S., Shearer, J., 1990. Financial comparison of three testing strategies for detection of estrus in cattle. J. Am. Vet. Med. Assoc. 196, 865±869. Oltenacu, P.A., Ferguson, J.D., Lednor, A.J., 1990. Economic evaluation of pregnancy diagnosis in dairy cattle: a decision analysis approach. J. Dairy. Sci. 73, 2826±2831. Pauker, S.G., Kassirer, J.P., 1987. Decision analysis. N. Engl. J. Med. 316, 250±258. Ridge, S.E., Vizard, A.L., 1993. Determination of the optimal cutoff value for a serological assay: an example using the Johne's absorbed EIA. J. Clinic. Microbiol. 31, 1256±1261. Slenning, B.D., 1994. Financial analysis of a clinical trial comparing simple estrus detection with estrus detection after prostaglandin-based appointment breeding in a commercial dairy herd in California, USA. Prev. Vet. Med. 18, 239±257. Slenning, B.D., Gardner, I.A., 1997. Economic evaluation of risks to producers who use milk residue testing programs. J. Am. Vet. Med. Assoc. 211, 419±427. Smith, R.D., 1993. Decision analysis in the evaluation of diagnostic tests. J. Am. Vet. Med. Assoc. 203, 1184± 1192. Smith, R.D., 1995a. Veterinary Clinical Epidemiology. A Problem-Oriented Approach. CRC Press, Boca Raton, 270 pp. Smith, R.D., 1995b. Decision-analysis of heartworm diagnostic-tests and management options. In: Soll, M.D., Knight, D.H. (Eds.), Proceedings of the Heartworm Symposium'95. American Heartworm Society, Batavia, pp. 147±156. Thrusfield, M., 1997. Veterinary Epidemiology. Iowa State University Press, Ames, 498 pp. Vaillancourt, J.-P., Toma, B., et al., 1999. Dictionary of Veterinary Epidemiology. Iowa State University Press, Ames, 300 pp. Wren, G., 1998. Testing for Johne's. Bovine Veterinarian: A supplement to Dairy Herd Management and Drovers for Bovine Practitioners. July±August, 4±8, Vance Publishing Corp., Lenexa, KS, USA.