Use and misuse of cost-effectiveness terminology in the gastroenterology literature: a systematic review

THE AMERICAN JOURNAL OF GASTROENTEROLOGY © 2002 by Am. Coll. of Gastroenterology Published by Elsevier Science Inc.

Vol. 97, No. 1, 2002 ISSN 0002-9270/02/$22.00 PII S0002-9270(01)03995-8

Use and Misuse of Cost-Effectiveness Terminology in the Gastroenterology Literature: A Systematic Review John K. Marshall, M.D., M.Sc., F.R.C.P.C., Ruth Cawdron, M.A., Deborah L. R. Yamamura, M.D., F.R.C.P.C., Subhas Ganguli, M.D., F.R.C.P.C., Rameeta Lad, M.D., M.Sc., F.R.C.P.C., and Bernie J. O’Brien, Ph.D. Division of Gastroenterology, Department of Medicine, and Departments of Laboratory Medicine and Clinical Epidemiology & Biostatistics, McMaster University; and Centre for Evaluation of Medicines, St. Joseph’s Hospital, Hamilton, Ontario, Canada

OBJECTIVES: The increased popularity of economic analyses for evaluating medical interventions has given rise to concern about the rigor with which economic constructs and terminology are used. True cost-effectiveness analysis considers both the costs and outcomes of alternative interventions. A systematic review of the gastroenterology literature was undertaken to evaluate how appropriately cost-effectiveness is assessed. METHODS: A structured MEDLINE search identified all studies published in major gastroenterology journals between 1980 and 1998 that claimed in their abstracts to have assessed the cost-effectiveness of an intervention. Blinded copies of eligible studies were assessed by two independent reviewers who used standard criteria to evaluate the use of economic terminology and key economic constructs. Discrepancies were resolved by consensus. Studies met a “broad criterion” for appropriateness by evaluating both costs and effects and a “strict criterion” by demonstrating dominance of one strategy or considering both incremental costs and incremental effects. RESULTS: Of 110 eligible studies, 77 (70.0%) met the broad criterion and 62 (56.4%) met the strict criterion for appropriateness. This did not seem to vary with either journal impact factor or publication year. Only eight of 18 studies reporting an incremental cost-effectiveness ratio compared its value to an external standard. Few studies explicitly stated their analytic perspective, and a minority of those with time horizons longer than 1 yr had discounted future costs or effects. CONCLUSIONS: Although most studies seem to use costeffectiveness terminology well, there remains room to improve the rigor with which economic terminology and constructs are applied. (Am J Gastroenterol 2002;97:172–179. © 2002 by Am. Coll. of Gastroenterology)

INTRODUCTION As constrained health care resources have encouraged providers to consider and justify both the clinical and resource implications of their interventions, formal economic evaluations of health care strategies and technologies have become increasingly prevalent in the medical literature (1). However, this rapid growth in the volume of health economic literature has fostered an increased concern about the rigor with which economic constructs and terminology are used (2– 6). In many cases, an assertion that an intervention is “cost-effective” is not supported by the evidence presented; for example, a review of economic analyses published before 1987 found only a minority to have reported appropriate summary measures (7). A more recent systematic review of cost-benefit analyses published in the general medical literature found that only 32% had applied terminology correctly and reported bona fide cost-benefit analyses (8). Cost-effectiveness analysis (CEA) is the most common form of economic analysis for the evaluation of health care services (1). CEA evaluates two or more alternative strategies with respect to both their costs and their health outcomes, with the latter measured in terms of physical or natural units (e.g., years of life gained or cases detected), and expresses the relationship between estimated costs and outcomes (9). In some cases, one strategy provides better health outcomes than an alternative at lower cost. This outcome is described as “dominance” and, as a “win-win” scenario for patients and providers, argues strongly for adoption of the dominant strategy. Often, CEA finds the more effective strategy to also be the more costly. Here, an incremental cost-effectiveness ratio (ICER) can be calculated as the ratio of incremental costs to incremental effects (the increment being the difference between the treatment and control groups). The ICER presents an estimate of the additional costs that would be incurred to obtain each additional health outcome if the more effective program were

AJG – January, 2002

to replace the less effective alternative. This ratio informs the decisions of health care providers and policy makers who must decide whether the additional cost is worthwhile. In Canada, the United States, and elsewhere, guidelines have been produced to assist researchers in conducting and reporting CEA (10 –13), and journals have begun to standardize their approach to reviewing economic submissions (14). A recent review by Provenzale and Lipscomb (4) discussed misuses of the term cost-effectiveness with examples from the field of gastroenterology and reinforced the need to for researchers to consider both incremental costs and incremental effects. However, no systematic review of cost-effectiveness terminology use in the gastroenterology literature has since been published. By definition, an assertion that a strategy or technology is “cost-effective” should follow consideration of both its costs and its incremental benefits compared to a relevant alternative. An intervention can be cost-effective if it improves outcomes while reducing costs, or if its gain in health outcomes is considered to be worth its additional cost (2). Inappropriate decisions about cost-effectiveness can be made by considering costs without attention to differences in outcome, by considering health benefits without direct consideration of costs, and by using average cost-effectiveness ratios (4). The latter examine the relationships between a strategy’s total costs and its total effects, and ignores those of other strategies. Because no alternative, including taking no action, is without costs and effects, this ratio is meaningless and can convey misleading information (3). Because of the increased attention given to appropriate design and reporting of CEA over the last decade, we undertook a systematic review of the gastroenterology literature to characterize and quantify the appropriate and inappropriate use of cost-effectiveness terminology.

MATERIALS AND METHODS An online search of the MEDLINE database was conducted to identify all citations published between 1966 and 1998 (inclusive) that used the phrases cost-effective or cost-effectiveness in their titles or abstracts. The search was limited to 19 core gastroenterology journals: Alimentary Pharmacology & Therapeutics, The American Journal of Gastroenterology, The Canadian Journal of Gastroenterology, Digestion, Digestive Diseases, Digestive Diseases and Science, Endoscopy, The European Journal of Gastroenterology & Hepatology, Gastroenterology, Gastrointestinal Endoscopy, Gut, Hepato-Gastroenterology, Hepatology, The Italian Journal of Gastroenterology, The Journal of Clinical Gastroenterology, The Journal of Gastroenterology, The Journal of Gastroenterology & Hepatology, The Journal of Pediatric Gastroenterology and Nutrition, and The Scandinavian Journal of Gastroenterology. The abstract of each citation was reviewed independently by two investigators (J.K.M., R.L.) for inclusion in the review. Articles were considered eligible if the author(s) claimed or implied in the

Cost-Effectiveness Terminology

173

title or abstract that they had evaluated the cost-effectiveness of a strategy or technology. Only full and original publications in the English language were accepted. Abstracts, case reports, editorials, and letters to the editor were excluded. Discrepancies between observers were resolved by consensus. Full offprints were retrieved for each eligible abstract citation (see Appendix). Blinded copies were prepared by obscuring all information related to 1) the authors’ names and affiliations, 2) the journal title, 3) the center where the research was conducted, 4) the year of publication, and 5) the source(s) of any financial support. A separate pair of investigators familiar with economic analysis (D.L.R.Y., S.G.), one of whom was unfamiliar with the gastroenterology literature (D.L.R.Y.), then reviewed each eligible article in detail for data extraction and scoring using standard data collection forms. After review of the first 20 articles, the data extractors and the principal investigator convened to compare results, confirm consistency of coding, and resolve discrepancies. All subsequent disagreement between the two reviewers was resolved by consensus. For each reference, the role of the economic analysis was classified as 1) a primary study outcome, 2) a planned secondary study outcome with methods and results described, or 3) a post hoc consideration raised as an issue for discussion. The primary research design was identified as one of the following: 1) a case series, 2) a prospective cohort study, 3) a case-control study, 4) a prospective controlled trial, or 5) a decision analysis or Markov model. The definition of cost-effectiveness used or implied by the authors of each citation was classified according to the system summarized in Table 1. Among the studies that reported an ICER, it was recorded whether the investigators compared its value to some external criterion representing societal willingness to pay. A predefined “broad criterion” for the appropriate use of cost-effectiveness terminology was satisfied where both costs and outcomes had been considered in some manner (i.e., dominant, incremental cost-effective, average costeffective, or other cost-effective). A “strict criterion” for appropriateness was satisfied only if the authors demonstrated dominance of one strategy or a an acceptable tradeoff of incremental costs and incremental effects (i.e., dominant or incremental cost-effective). Binary logistic regression was used to test relationships among the proportions of studies which satisfied these criteria, publication year, and journal impact factor (15). Where a measure of health outcome was acknowledged in the economic analysis, it was classified as one of 1) natural units (e.g., life years), 2) preference-weighted measures (e.g., quality-adjusted life-years), or 3) dollar value equivalents. The use of natural units was considered congruent with true CEA, and the specific unit of effect was recorded. The use of preference-weighted or dollar value measures was considered to imply cost-utility analysis (CUA) or cost-benefit analysis (CBA), respectively. Because CEA,

174

Marshall et al.

AJG – Vol. 97, No. 1, 2002

Table 1. Summary of Definitions Used to Define Strategies as “Cost-Effective” Term Dominant Incremental cost-effective Average cost-effective

Other cost-effective Cost minimizing Effective

Definition Strategy incurred lower costs and provided better outcomes than alternative Strategy has an acceptable ICER (ratio of incremental costs to incremental effects) Strategy has lower average costeffectiveness ratio (ACER; ratio of total costs to total effects) Both costs and outcomes considered but no dominance, ICER or ACER defined Strategy incurred lower costs than alternative Strategy provided better outcomes than alternative

CUA, and CBA assess both costs and outcomes in some manner, all were considered eligible for the strict and broad appropriateness criteria. For each citation, the components considered in estimating the costs of alternative strategies were classified as direct medical, direct nonmedical, or indirect. Although overlap between categories is common, direct costs typically reflect the consumption of goods, services, and other resources attributable to an intervention or its consequences (16). These can be further divided into medical components (e.g., supplies and personnel) and nonmedical components (e.g., child care and transportation). Indirect costs refer to changes in productivity related to illness or death. Each reference was also reviewed for an explicit statement of the primary perspective of its economic analysis. When provided, the viewpoint was categorized as that of society, a third party payer, a government provider, or the patient. The time horizon over which costs and effects were considered was noted. Where the horizon of the analysis exceeded 1 yr, the authors’ use of discounting to estimate the present value of costs and/or effects was sought, and the specific discounting rate was recorded. An ordinal scale developed by Zarnke et al. (8) for a similar review of cost-benefit studies was used to score the comprehensiveness of each article’s assessment of effectiveness, costing, and economic analysis (Table 2). Although this scale has not been validated elsewhere, it assesses cost-effectiveness design attributes similar to those advocated in recent guidelines (9, 10, 16). The rating of comprehensiveness of assessment of effectiveness considered the design of studies supporting the effectiveness claim, the use of primary and secondary data, and the transparency and extent of literature searching (score ⫽ 0 – 4). The rating of comprehensiveness of costing incorporated the measurement of program costs, costs associated with adverse effects (including false positive and false negative test results), and indirect costs, plus the use of dis-

Costs Considered

Effects Considered

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

No

No

Yes

counting (score ⫽ 0 –3). The comprehensiveness of the economic analysis synthesized the above features, plus the analytic perspective and use of sensitivity analysis to explore uncertainty (score ⫽ 0 –3). A total comprehensiveness score (range ⫽ 0 –10) was calculated as the sum of the three individual domain scores and averaged between the two reviewers. Linear regression was then used to test relationships among comprehensiveness scores, publication year, and 1999 journal impact factor (15).

RESULTS The MEDLINE database search identified 231 citations that had used the terms cost-effective or cost-effectiveness in the title and/or abstract text. Of these, 110 (47.6%) were judged eligible for inclusion. Agreement between the two reviewers in judging eligibility was high (␬ ⫽ 0.79). Discrepancies were resolved by consensus for 24 citations (10.4%). The

Table 2. Ordinal Scoring System Adapted From Zarnke et al. (8) Criterion 1. Comprehensiveness of demonstration of effectiveness A) Very B) Fairly C) Not very D) Cursory E) None 2. Comprehensiveness of costing A) Very B) Fairly C) Not very D) None 3. Comprehensiveness of economic analysis A) Very B) Fairly C) Not very D) Cursory Total possible score

Score 4 3 2 1 0 3 2 1 0 3 2 1 0 10

AJG – January, 2002

Cost-Effectiveness Terminology

175

Table 3. Characteristics of Studies Included in Review N (%) Primary study design Case series Prospective cohort study Prospective controlled trial Decision model Systematic literature review Year of publication 1981–1985 1986–1990 1991–1995 1996–1998 Role of the CEA Primary outcome Planned secondary outcome Post hoc analysis

19 (15.5%) 28 (25.5%) 27 (24.5%) 35 (31.8%) 1 (0.9%) 6 (5.5%) 19 (17.3%) 37 (33.6%) 48 (43.6%) 49 (44.5%) 21 (19.1%) 40 (36.4%)

distribution of eligible citations by research design, year of publication, and role of CEA is shown in Table 3. Of the eligible studies, 74 (67.3%) reported original clinical research, 35 (31.8%) reported decision analytic models, and one (0.9%) reported a systematic literature review. The CEA was determined to be a primary study outcome in 49 citations (44.5%), a planned secondary outcome in 21 (19.1%), and a post hoc consideration in the remaining 40 (36.4%). The definitions of cost-effectiveness that were used or implied by the authors are shown in Table 4. Among the 110 eligible citations that claimed to have evaluated cost-effectiveness, 11 (10.0%) were actually cost-minimization analyses that considered only the costs of alternative strategies and 22 (20.0%) were effectiveness analyses that analyzed only differences in effectiveness. The remaining 77 citations (70.0%) had given some consideration to both costs and effects in judging cost-effectiveness and satisfied the predetermined broad criterion for appropriateness. Five studies (4.5%) assessed cost-effectiveness using average, but not incremental, cost-effectiveness ratios. Sixty-two studies (56.4%) had considered both incremental costs and incremental benefits and reported an ICER (18 studies) or a dominance relationship (44 studies), to meet the predetermined strict criterion for appropriateness. However, only eight of the 18 studies that presented an ICER (44.4%) compared that ratio to some external criterion as a metric of societal willingness to pay. An additional seven studies (38.9%) suggested that the incremental effect justified the

Figure 1. Appropriateness of cost-effectiveness terminology use by publication year.

increase in cost but made no reference to an external standard. Of the 49 studies in which the CEA was a primary study outcome, the use of terminology was better: 42 (85.7%) met the broad criterion for appropriateness and 35 (71.4%) met the strict criterion. In binary logistic regression analyses, neither publication year nor journal impact factor appeared to predict appropriateness of use of cost-effectiveness terminology by broad or strict criteria (Figs. 1 and 2). These results were similar when the regression analysis was limited to studies in which the CEA was a primary outcome. Among the 99 studies that had considered the effectiveness of an intervention, 92 (92.9%) measured these in natural units (i.e., CEA), six (6.1%) used preference-weighted units (i.e., CUA), and one (1.0%) used both preferenceweighted values and dollar value equivalents (i.e., both CUA and CBA). Among the 88 studies that had evaluated explicitly the costs of alternative strategies, 74 (84.1%) had considered direct medical components, of which five (5.7%) had also included indirect costs and three (3.4%) had also included direct nonmedical costs. The remaining 14 studies (12.7%) did not state the components considered in their estimates of cost. Only 21 citations (19.1%) made explicit reference to the

Table 4. Definitions of Cost-Effectiveness Definition

N (%)

Appropriateness Criterion

Dominant Incremental cost-effective Average cost-effective Other cost-effective Cost minimizing Effective

44 (40.0%) 18 (16.4%) 5 (4.5%) 10 (9.1%) 11 (10.0%) 22 (20.0%)

Strict Strict Broad Broad Neither Neither

Figure 2. Appropriateness of cost-effectiveness terminology use by journal impact factor.

176

Marshall et al.

perspective of the CEA. Of these, the primary perspective was societal in three citations (14.3%), a third party payer in 12 (57.1%), a government provider in five (23.8%), and the patient in one (4.8%). The time horizon of analysis was stated explicitly in 46 citations (41.8%) and ranged from 1 wk to 30 yr. Among the 26 citations with time horizons longer than 1 yr, only 15 (57.7%) reported that future costs and/or outcomes had been discounted. Among these, the base-case rates were 3% (four citations), 5% (eight citations), 6% (one citation), and 10% (one citation). One study did not discount future costs in the base-case analysis but tested the effect of discounting in sensitivity analysis. Of 65 citations in which the CEA had assessed costs in the context of a clinical study (i.e., a case series, cohort study, case-control study, or clinical trial), five (7.5%) reported CIs or levels of significance for estimates of cost, 19 did so for estimates of effects, and two did so for both costs and effects. Of the 23 citations that had used decision models to evaluate cost-effectiveness, 17 (73.9%) reported only one-way sensitivity analyses, five (21.7%) also reported two- or multiway sensitivity analyses, and one (4.3%) used a Monte Carlo simulation. The mean total comprehensiveness score assigned by the reviewers was 5.5 (range ⫽ 1–10, SD ⫽ 2.2). Thirty-seven studies (33.6%) were rated as “very” or “fairly” comprehensive by both reviewers in all three categories of effectiveness, costing, and overall analysis. In univariate linear regression models, mean comprehensiveness scores correlated significantly with journal impact factor (p ⬍ 0.005) but not with year of publication. However, multiple linear regression showed both variables to persist as independent predictors of the mean comprehensiveness score, albeit explaining relatively little of the overall variance (R2 ⫽ 0.11, p ⬍ 0.005). Neither journal impact factor nor year of publication predicted mean comprehensiveness scores when the analysis was restricted to the 49 studies for which costeffectiveness was the primary study outcome. However, a post hoc analysis suggested that publication year was a more powerful predictor of comprehensiveness scores among the 25 studies published up to 1990 (R2 ⫽ 0.32, p ⬍ 0.05).

DISCUSSION This systematic review evaluated the use of the terms costeffective and cost-effectiveness in the gastroenterology literature. Seventy percent of the studies evaluated met a predefined broad criterion for appropriateness of use, in that they had given some consideration to both the costs and outcomes of alternative strategies. Furthermore, 55 percent met a predefined strict criterion for appropriateness, which required demonstration of dominance relationship among strategies or consideration of both incremental costs and incremental effects where no strategy was dominant. Although these results are encouraging, they demonstrate that there remains room for improvement among authors, re-

AJG – Vol. 97, No. 1, 2002

viewers, and editors in the application of health economic terminology. It is interesting that the appropriateness of cost-effectiveness terminology use did not seem to vary with either year of publication or journal prominence, as measured by journal impact factor, although an exploratory post hoc analysis suggested some improvement had occurred before 1990. Thus, the release of guidelines for the conduct and reporting of cost-effectiveness analyses in 1996 and a contemporaneous review of the subject have not appeared to influence the use of terminology (4, 13, 16). This is in keeping with the conclusions of other authors, who have observed only modest improvement in the quality of health economics literature with time (6). Several studies (5.5%) judged cost-effectiveness using average cost-effectiveness ratios, or ratios of total costs to total effect. This measure is widely recognized to be inappropriate and meaningless, as it ignores the costs and effects of the alternative strategy a new approach would replace. Indeed, even a do nothing approach can be expected to incur costs and to allow some positive health outcomes. A new strategy must express its costs and effects in the context of those of a standard approach or another reasonable comparator. If the new approach offers greater health benefits at reduced costs, a win-win relationship of dominance is demonstrated and health care policy makers would be well advised to adopt the new strategy. If dominance is not demonstrated, the ICER (the ratio of incremental costs to incremental benefits relative to the comparator) is considered the most useful metric to convey to decision makers the expected trade-off of costs and effects. It is of particular concern that one in 10 studies appeared to judge cost-effectiveness solely on the basis of differences in cost. A strategy that saves money but costs lives is not necessarily cost-effective, unless the degree of cost savings exceeds the perceived value of those lives foregone. In the same way, a strategy that improves outcomes at exorbitant cost would not be considered cost-effective because the gain in health effects does not justify the increment in cost. To label a less costly strategy as cost-effective sends a misleading message to decision makers charged not with minimizing cost, but with maximizing health outcomes within a finite pool of resources. We used a subjective, ordinal grading system to measure the comprehensiveness with which studies reported evaluated and reported their estimates of effectiveness, estimates of cost, and CEA. This grading system, though not well validated, was used in a previous systematic review of cost-benefit analyses in the general medical literature (8). We were unable to demonstrate any change in the comprehensiveness analysis over time or among journals categorized by impact factor. Only one in three studies was rated as fairly or very comprehensive by both reviewers in all three domains, which suggests there is substantial room for improvement in the reporting of cost-effectiveness analyses. The scores reported in this analysis (mean ⫽ 5.5) were

AJG – January, 2002

similar to those reported by Zarnke et al. (8) in their review of cost-benefit analyses (mean ⫽ 4.5); the subjective nature of this scoring system precludes any quantitative comparison among studies. As secondary outcomes, this review also examined three key aspects of CEA methodology. First, recent guidelines have advocated the consideration of costs from a societal perspective, but other analytic viewpoints are accepted if they are explicitly described. In this review, only a minority of eligible studies (21 [19%]) made explicit reference to their perspective of analysis, of which three chose a societal viewpoint for their primary analysis. This is of concern, because the analytic perspective determines which cost components are included in the model and can influence significantly its conclusions. Second, it is important that cost-effectiveness analyses consider how their conclusions are affected by the uncertainty around estimates of costs and effects. Among those cost-effectiveness analyses that accompanied clinical studies, fewer than half reported CIs or levels of significance for estimates and comparisons of cost. Among decision models, the majority used one-way sensitivity analysis, but fewer reported the impact of simultaneous variation in more than one parameter and only one used more advanced techniques such as Monte Carlo simulation. A third methodological consideration is the discounting of future costs and effects to reflect time preference. Of the 26 studies whose reported time horizon exceeded 1 yr, only 15 discounted expected future costs and/or effects. In conclusion, although an exhaustive review of cost-effectiveness methodology was not the primary endpoint of this study, our analysis suggests there is substantial room for improvement in the use and description of key principles of health economics analysis. In summary, this review of the gastroenterology literature demonstrates that the majority of studies that claimed to have evaluated cost-effectiveness did so after some consideration of both the costs and the effects of an intervention and could be considered as bona fide cost-effectiveness analyses. However, many gave no explicit attention to the incremental costs and effects of alternative strategies, and a surprising number evaluated cost-effectiveness without direct consideration of either costs or effects. Despite published guidelines for the conduct and reporting of health economic analyses, the use of cost-effectiveness terminology did not seem to improve with time and did not vary among journals segregated by journal impact factor. Greater attention to the quality and methodology of economic submissions is needed among investigators, authors, and editors.

Cost-Effectiveness Terminology

177

Reprint requests and correspondence: John K. Marshall, M.D., Division of Gastroenterology (4W8), McMaster University Medical Centre, 1200 Main Street West, Hamilton ON L8N 3Z5, Canada. Received May 18, 2001; accepted Sep. 14, 2001.

REFERENCES 1. Elixhauser A, Luce BR, Taylor WR. Health care CBA/CEA: An update of the growth and composition of the literature. Med Care 1993;31:JS1–11. 2. Doubilet P, Weinstein MC, McNeil BJ. Use and misuse of the term “cost-effective” in medicine. N Engl J Med 1986;314: 253– 6. 3. Harris RA, Nease RF. Economic heartburn: Average costeffectiveness and gastroesophageal reflux disease. Gastroenterology 1995;108:303– 6 (letter). 4. Provenzale D, Lipscomb J. Cost-effectiveness: Definitions and use in the gastroenterology literature. Am J Gastroenterol 1996;91:1488 –93. 5. Marshall JK, O’Brien BJ. Cost and cost-effectiveness: Consistent use of terminology is needed. Am J Gastroenterol 1999;94:1108 –9 (letter). 6. Neumann PJ, Stone PW, Chapman RH, et al. The quality of reporting in published cost-utility analyses, 1976-1997. Ann Intern Med 2000;1132:964 –72. 7. Udvarhelyi IS, Colditz GA, Rai A, Epstein AM. Cost-effectiveness and cost-benefit analyses in the medical literature: Are the methods being used correctly? Ann Intern Med 1992; 116:238 – 44. 8. Zarnke KB, Levine MA, O’Brien BJ. Cost-benefit analyses in the health-care literature: Don’t judge a study by its label. J Clin Epidemiol 1997;50:813–22. 9. Drummond MF, O’Brien BJ, Stoddart GL, Torrance GW, eds. Methods for the economic evaluation of health care programmes, 2nd ed. New York: Oxford University Press, 1997. 10. Canadian Coordinating Office for Health Technology Assessment. Guidelines for the economic evaluation of pharmaceuticals: Canada, 2nd ed. Ottawa: Canadian Coordinating Office for Health Technology Assessment, 1997. 11. Russell LB, Gold MR, Siegel JE, et al. The role of costeffectiveness analysis in health and medicine. JAMA 1996; 276:1172–7. 12. Weinstein MC, Siegel JE, Gold MR, et al. Recommendations of the panel on cost-effectiveness in health and medicine. JAMA 1996;276:1253– 8. 13. Siegel JE, Weinstein MC, Russell LB, Gold MR. Recommendations for reporting cost-effectiveness analyses. JAMA 1996; 276:1339 – 41. 14. Jefferson TO, Smith R, Yee Y, et al. Evaluating the BMJ guidelines for economic submissions. JAMA 1998;280:275–7. 15. Journal citation reports (1999). Philadelphia: Institute for Scientific Information, 2000. 16. Gold MR, Siegel JE, Russell LB, Weinstein MC, eds. Cost effectiveness in health and medicine. New York: Oxford University Press, 1996.

178

Marshall et al.

AJG – Vol. 97, No. 1, 2002

Appendix. Summary of Citations Reviewed, With Comprehensiveness Scores and Appropriateness Criteria Authors

Citation

Comprehensiveness Score*

Appropriateness Criterion

Aberg T, et al. Adams PC, et al. Afridi SA, et al. Ashton T Balaban DH, et al. Balan V, et al. Barkin JS, et al. Barkin JS, et al. Barry MJ, et al. Bass EB, et al. Bastid C, et al. Battezatti PM, et al. Bonkovsky HL, et al. Bytzer P, et al. Caliari S, et al. Chandra RK Chu KM, et al. Chung R, et al. Corrao G, et al. Cummings DR, Meshkinpour H D’Albasio G, et al. Davis JM, et al. Deutsch DE, Olson AD Duncan HD, et al. Duphare H, et al. Dusheiko GM, Roberts JA Eddy DM, et al. Eggleston A Elewaut A, et al. Ellenrieder V, et al. Erickson RA Fazio RA, et al. Fendrick AM, et al. Fox N, et al. Graham DY, et al. Greenberg PD, et al. Greenfield SM, et al. Gregor JC, et al. Habu Y, et al. Harris RA, et al. Hay JW, Hay AR Hayashi J, et al. Hoffman MS Hu KQ, et al. Humbert P, et al. Inadomi J, et al. Johnston PW, et al. Jungst D, et al. Kang JY, et al. Kolts BE, et al. Kolts BE, et al. Korula J, et al. Kozarek RA, et al. Laheij RJ, et al. Lapane KL, et al. Lashner BA, et al. Lieberman DA, et al. Lieberman DA Lirussi F, et al. Louis-Jacques O, et al. Low DE, et al.

Endoscopy 1985;17:133–6 Hepatology 1995;22:1720–7 Gastrointest Endosc 1995;41:485–9 Scand J Gastroenterol 1991;26:82–8 Am J Gastroenterol 1997;92:79–83 Gastroenterology 1994;107:453–9 Am J Gastroenterol 1991;86:1437–41 Gastrointest Endosc 1990;36:123–6 Gastroenterology 1987;93:301–10 Gastroenterology 1991;101:189–99 Dig Dis Sci 1994;39:1576–80 Ital J Gastroenterol 1992;24:481–4 Gastroenterology 1993;99:1079–91 Gut 1989;30:1379–84 Scand J Gastroenterol 1996;31:90–4 J Pediatr Gastroenterol Nutr 1997;24:380–8 Am J Gastroenterol 1998;93:1436–42 Gastrointest Endosc 1998;48(1):71–4 Gut 1994;35:771–5 Dig Dis Sci 1985;30:739–41 Am J Gastroenterol 1997;92:1143–7 Aliment Pharmacol Ther 1998;12:207–11 J Pediatr Gastroenterol Nutr 1997;25:26–31 Eur J Gastroenterol Hepatol 1997;9:905–8 J Clin Gastroenterol 1994;18:23–6 Hepatology 1995;22:1863–73 Gastroenterology 1987;92:682–92 Gut 1998;42:13–6 Gut 1993;34:274–8 Eur J Gastroenterol Hepatol 1997;9:1085–9 Gastroenterology 1987;93:835–45 Am J Gastroenterol 1983;78:261–4 Am J Gastroenterol 1997;92:2017–24 Aliment Pharmacol Ther 1988;2:297–309 Am J Gastroenterol 1991;86:1158–62 Am J Gastroenterol 1996;91:228–32 Dig Dis Sci 1986;31:785–91 Gastrointest Endosc 1996;44:118–23 Gastroenterol Hepatol 1998;13:280–7 Am J Gastroenterol 1997;92:2179–87 J Clin Gastroenterol 1992;14:309–17 Dig Dis Sci 1998;43:384–91 J Clin Gastroenterol 1998;26:279–82 Hepatology 1995;21:901–7 Gut 1994;35:1205–8 Gastroenterology 1997;113:1289–94 Gut 1993;34:1158–61 Hepatology 1986;6:239–43 J Gastroenterol Hepatol 1992;7:463–8 Am J Gastroenterol 1993;88:1218–23 Am J Gastroenterol 1993;88:650–5 Gastrointest Endosc 1986;32:1–3 Gastrointest Endosc 1996;43:10–13 Aliment Pharmacol Ther 1998;12:1249–56 Am J Gastroenterol 1998;93:591–6 Am J Gastroenterol 1990;85:1088–95 Am J Gastroenterol 1991;86:1789–94 Gastroenterology 1995;109:1781–90 Scand J Gastroenterol 1993;28:267–73 J Pediatr Gastroenterol Nutr 1997;24:25–32 J Clin Gastroenterol 1989;11:253–9

5.5 9 4.5 6 3.5 6 5 3 8 9 2.5 3 3 7 3.5 6 7.5 4.5 3.5 5 5.5 5 5.5 2.5 2 9 8.5 4 5 4 7 3 7.5 8.5 3 3 4 7.5 9 7 4 3 4.5 3 3.5 8 7 3 7 6 6 5 3 8.5 7.5 7 7 7 3 8 3

Strict Broad Strict Strict Broad Broad Strict Neither Strict Strict Neither Neither Neither Strict Neither Strict Strict Broad Strict Broad Strict Strict Strict Neither Neither Strict Strict Neither Neither Neither Strict Strict Strict Strict Neither Neither Neither Strict Strict Strict Neither Neither Strict Neither Neither Strict Broad Neither Strict Strict Strict Neither Neither Strict Strict Broad Strict Broad Broad Strict Strict continued

AJG – January, 2002

Cost-Effectiveness Terminology

179

Appendix. (continued) Authors Machicado GA, et al. Marks RD, et al. Marshall BJ, et al. Messori A, et al. Moss AA, et al. Nicholl JP, et al. Nishioka NS, Richter JM O’Brien B, et al. Olson AD, et al. Overholt BF, et al. Pasha T, et al. Pawlotsky JM, et al. Prat F, et al. Preston KL, et al. Provenzale D, et al. Puetz T, et al. Pym B, et al. Quirk DM, et al. Rex DK, et al. Rex DK, et al. Richter JM, et al. Rozen P, Ron E Sabat M, et al. Sadowski DC, Rebeneck L Saha SK Sarasin FP, et al. Schreiber JH Segel MC, et al. Shafik A, et al. Smith DW, Rankin RA Sonnenberg A, et al. Sonnenberg A Streitz JM Taniguchi Y, et al. Teran JC, et al. Termanini B, et al. Tham TC, et al. Toth E, et al. Treiber G Trevisani L, et al. Triger DR, et al. Vakil N, Fennerty MB Vakil N, Ashorn M Van Den Boom G, et al. Wagner HJ, et al. Ware AJ, et al. Wilcox CM, et al. Williams JW, et al. Wright TA, et al. * Mean score of two reviewers.

Citation Gastrointest Endosc 1997;45:157–62 Gastroenterology 1994;106:907–15 Am J Gastroenterol 1991;86:438–45 J Clin Gastroenterol 1996;23:269–74 Gastroenterology 1981;80:45–50 Gut 1994;35:1294–300 Gastrointest Endosc 1987;33:277–83 Can J Gastroenterol 1997;11:323–31 Gastrointest Endosc 1996;44:554–61 Gastrointest Endosc 1987;33:1–3 Hepatology 1998;27:1220–6 Hepatology 1998;27:1700–2 Gut 1998;42:76–80 Gastrointest Endosc 1994;40:474–6 Am J Gastroenterol 1994;89:670–80 Am J Gastroenterol 1997;92:254–7 Gastroenterology 1990;99:27–35 Gastrointest Endosc 1997;46:334–7 Gastroenterology 1990;98:855–61 Gastrointest Endosc 1995;42:132–8 Gastrointest Endosc 1995;41:93–8 Am J Gastroenterol 1989;84:1548–51 Am J Gastroenterol 1998;93:2457–62 Am J Gastroenterol 1997;92:608–13 Dig Dis Sci 1996;41:1313–8 Hepatology 1998;28:436–42 Endoscopy 1994;26:292–8 Gastroenterology 1985;89:485–8 Digestion 1995;56:479–82 Gastrointest Endosc 1989;35:90–2 Gastroenterology 1989;96:1445–52 Dig Dis Sci 1991;36:949–56 Am J Gastroenterol 1998;93:911–5 J Gastroenterol 1996;31(S9):37–40 Gastroenterology 1997;112:473–82 Gastroenterology 1997;112:335–47 Gut 1998;42:703–7 Endoscopy 1995;27:654–8 Am J Gastroenterol 1996;91:246–57 Dig Dis Sci 1997;42:2299–303 Gut 1992;33:1553–8 Am J Gastroenterol 1996;91:239–45 Am J Gastroenterol 1998;93:562–8 Scand J Gastroenterol 1996;31:1–9 Endoscopy 1993;25:213–8 Hepatology 1983;3:513–8 Gastroenterology 1996;110:1803–9 Am J Gastroenterol 1987;87:1115–9 Gut 1996;39:574–9

Comprehensiveness Score*

Appropriateness Criterion

5.5 7.5 3 8 6.5 7.5 5 7 7 2 8 5 3 2.5 9.5 5.5 9 3.5 7.5 6 3 6.5 6 7 2 10 3 1 2.5 3 9 8 8 3 7.5 4.5 5.5 3 6.5 7 7.5 6 6.5 7 6 2.5 4.5 5 5

Neither Strict Neither Strict Strict Strict Neither Strict Strict Neither Strict Strict Strict Neither Strict Strict Strict Neither Strict Broad Strict Strict Strict Strict Neither Strict Strict Neither Neither Broad Broad Strict Strict Neither Strict Strict Strict Neither Broad Strict Strict Strict Neither Strict Strict Neither Broad Strict Broad