- Email: [email protected]
Assessing the Appropriateness of Outpatient Abdominopelvic CT and MRI Examinations Using the American College of Radiology Appropriateness Criteria
Assessing the Appropriateness of Outpatient Abdominopelvic CT and MRI Examinations Using the American College of Radiology Appropriateness Criteria Andrew B. Rosenkrantz, MD, MPA, Khalil Marie, BS, Ankur Doshi, MD Rationale and Objectives: To retrospectively assess the appropriateness of outpatient abdominal and pelvic computed tomography (CT) and magnetic resonance imaging (MRI) examinations using the American College of Radiology Appropriateness Criteria (AC). Materials and Methods: A total of 570 adult outpatient abdominopelvic CT (304) and MRI (266) studies performed in a 1-month period with available documentation of the clinical encounter generating the imaging order were included. On the basis of review of the imaging report and patient record, examinations were classified in terms of match to a specific AC variant, appropriateness score, and the presence of a significant result. Data were analyzed using Fisher’s exact test. Results: Forty-five percent of examinations matched an AC variant: 52% of CT and 38% of MRI (P < .001). Ninety-two percent of examinations matching the AC were appropriate: 96% of CT and 86% of MRI (P = .009). Appropriate examinations were more likely to provide a significant result than not appropriate studies (48% vs. 24%, P = .041). Although a significant result was related to the primary study indication more frequently in appropriate than not appropriate examinations, this difference was not significant (93% vs. 80%, respectively, P = .204). The most common indications not matching an AC were colon cancer follow-up (n = 14) and melanoma follow-up (n = 14) among CT, and hepatocellular carcinoma screening (n = 31) and elevated prostate-specific antigen (PSA) without prior biopsy (n = 14) among MRI. Conclusions: Most examinations matching the AC were appropriate, and appropriate examinations were more likely to have a significant result. However, most examinations, including 62% of MRI, had no relevant clinical condition, highlighting a critical area for future AC expansion and modification. Key Words: Radiology; Appropriateness Criteria; utilization; CT; MRI. ªAUR, 2015
I
n response to concerns of escalating use of imaging services and wide variability in such use, the American College of Radiology (ACR) launched an initiative in 1993 to formulate guidelines to assist in appropriate ordering of imaging studies (1,2). The resulting ACR Appropriateness Criteria (AC) seek to standardize which imaging examinations are ordered for a wide array of clinical conditions and thereby reduce variation and overutilization (3,4). The AC strive to optimize the balance between benefit and risk inherent to imaging examinations by maximizing diagnostic yield and Acad Radiol 2015; 22:158–163 From the Department of Radiology, NYU Langone Medical Center, 660 First Avenue, New York, NY 10016 (A.B.R., K.M., A.D.). Received June 23, 2014; accepted August 12, 2014. Funding: None. Conflicts of interest: None. Address correspondence to: A.B.R. e-mail: Andrew.Rosenkrantz@nyumc. org ªAUR, 2015 http://dx.doi.org/10.1016/j.acra.2014.08.005
158
provision of useful information in consideration of the examination’s costs, radiation exposure, other procedural risks, and possibility of incidental findings (2,4–6). The AC were crafted by multidisciplinary expert panels in an evidence-based fashion using a modified Delphi approach (2). AC are organized in terms of specific clinical conditions and associated variants, ranking for each variant the appropriateness of relevant imaging examinations on a 1–9 scale; a score of 7–9 indicates that the given imaging examination is ‘‘usually appropriate’’ for the given condition (6,7). From the time of their introduction, the AC have been continually expanded and updated by the ACR (4), and currently AC exist for over 197 clinical conditions with over 900 variants (8). The AC are the only guidelines in the United States to direct the ordering of diagnostic imaging examinations for a comprehensive spectrum of clinical scenarios, as reflected by the large number of conditions and variants addressed (2). In comparison, other available guidelines, such as the National
Academic Radiology, Vol 22, No 2, February 2015
APPLYING ACR CRITERIA TO ABDOMINOPELVIC CT/MRI
Comprehensive Cancer Network (9) and American Society of Clinical Oncology (10) guidelines, generally direct imaging examinations in limited contexts, such as cancer staging and post-treatment surveillance. The AC have been applied for numerous purposes, such as by private insurers for directing patient benefit policies (11), by academic institutions as the basis for the development of clinical decision support systems (12), and by radiologists as a basis of educating other physicians in appropriate examination ordering (7,13). More recently, the ‘‘Protecting Access to Medicare Act of 2014’’ Act (H.R. 4302, signed into law April 1, 2014) requires that health care providers consult appropriateness use criteria endorsed by national professional medical societies when ordering imaging services (14). It is expected that the ACR AC will be a primary set of guidelines used in fulfilling this requirement (15). This legislation enhances the potential impact of the ACR AC given the large influence of Medicare policy on medical practice in the United States. Thus, as the legislation undergoes implementation, it is important to recognize any limitations or weaknesses of the ACR AC. However, only a small amount of the literature has assessed the overall level of appropriateness of diagnostic imaging examinations relative to the ACR AC or the association of such appropriateness with outcomes (11). Such knowledge would be useful in guiding continued policy development. Therefore, we conducted this study to evaluate the appropriateness of outpatient abdominal and pelvic computed tomography (CT) and magnetic resonance imaging (MRI) examinations using the ACR AC, as well as associations between appropriateness and imaging results.
this retrospective study to reliably make such determinations. This process yielded a final cohort of 570 outpatient studies (304 CT [255 with intravenous (IV) contrast, 49 without IV contrast], 266 MRI [262 with IV contrast, four without IV contrast]).
METHODS Selection of Imaging Examinations
This retrospective study performed at a tertiary care academic medical center was Health Insurance Portability and Accountability Act (HIPAA)-compliant and approved by our institutional review board with a waiver of the requirement for written informed consent. We searched a departmental database for all abdominal or pelvic outpatient CT and MRI examinations performed during a 4-week interval in September 2013, identifying 1743 examinations (968 CT and 775 MRI); Emergency Department examinations were not included. The electronic medical record was then searched to identify whether the full note for the clinical encounter generating the imaging request was available for review. This requirement ensured accurate classification of the appropriateness of each imaging examination in view of the numerous variants comprising the AC and the potentially incomplete clinical history contained only in the imaging reports; this approach was also used in a prior study (16). Exclusion of examinations on this basis did not imply that the examination failed to match an AC variant or was inappropriate, but solely that we lacked sufficient information in
Imaging Examination Review
Available clinical and imaging data were reviewed in consensus by two radiologists, both fellowship-trained abdominal radiologists with one and six years of experience. For each imaging examination, the full note from the clinical encounter generating the request was reviewed to identify the study indication and any other clinical factors (ie, signs and symptoms, laboratory values, known diagnoses) contributing to the reason for imaging. Then, the AC were reviewed to identify a match between the examination indication and a variant of a clinical condition within the AC (8). If no match was identified, then no further analysis of the case was performed. If a match was identified, then the appropriateness score of the performed imaging study was recorded. Examinations receiving a score of 7–9 were considered appropriate. In addition, select examinations receiving a score of 4–6 (‘‘may be appropriate’’) were reclassified as appropriate because of meeting specific exceptions noted in the AC. For cases not classified as appropriate, the imaging examination that would receive the highest appropriateness score for the given indication was recorded; if no imaging examination would receive a score of 7 or greater for the indication, then it was recorded that no imaging was considered appropriate for the indication. For all imaging studies included in the analysis, the presence of a significant result was also recorded. A significant result was defined as any finding described within the report requiring further intervention or follow-up. Finally, for studies with a significant result, whether this result was related to the primary indication for the study was also recorded. Statistical Assessment
The frequency of outpatient studies matching a variant of a clinical condition, of such studies that were considered appropriate, of appropriate studies with a significant result, and of studies with a significant result in which the result was related to the primary indication, were computed using standard summary statistics. The frequencies of indications without a match were tabulated separately for CT and MRI. Fisher’s exact test was used to compare CT and MRI in terms of both frequency of a match and frequency of appropriateness, as well as to compare studies that were and were not appropriate in terms of significance of result and relation to the indication. For performed inappropriate studies, the distribution of examination type and advised alternate imaging examination using the AC were recorded. P values are two-sided and considered statistically significant at P < .05. For tests in which the obtained P value did not indicate statistical significance, post hoc assessment was performed to determine 159
ROSENKRANTZ ET AL
Academic Radiology, Vol 22, No 2, February 2015
TABLE 1. Most Common Indications of Computed Tomography Examinations not Matching an Appropriateness Criteria Variant* Indication
n
Colon cancer follow-up Melanoma follow-up Weight loss; no abdominal pain Abdominal pain not localized to one quadrant; no fever, recent surgery, or laboratory abnormality Lymphoma follow-up Pancreatic cancer follow-up Weight loss and abdominal pain not localized to one quadrant Hepatocellular carcinoma screening Anemia Lung cancer follow-up Breast cancer follow-up, not stage I Neuroendocrine tumor follow-up
14 13 9 8 7 7 6 3 3 3 3 3
*Indications listed if n $ 2.
TABLE 2. Most Common Indications of Magnetic Resonance Imaging Examinations not Matching an Appropriateness Criteria Variant* Indication
n
Hepatocellular carcinoma screening Elevated PSA, no prior prostate biopsy Hepatocellular carcinoma follow-up Solid renal mass, surveillance Lower extremity edema/ulcer, concern for iliac vein compression Fibroids, pretreatment planning; no acute pelvic pain Ureteropelvic junction (UPJ) obstruction Elevated PSA, one prior negative prostate biopsy Pancreatic cyst follow-up Tuberous sclerosis; screening for renal mass
31 14 11 9 9 8 5 5 5 4
*Indications listed if n $ 2.
whether the comparison had sufficient power to detect significance of the observed difference at a power of 20%. Analysis was performed using software (MedCalc for Windows, version 12.7; MedCalc Software, Ostend, Belgium).
RESULTS Forty-five percent of outpatient imaging examinations matched a variant of a clinical condition within the AC: 52% of CT and 34% of MRI (P < .001). All examinations with a match corresponded with conditions and variants described within the gastrointestinal (n = 59), urologic (n = 145), vascular (n = 29), women’s (n = 24), and thoracic (n = 1) topic areas. Tables 1 and 2 list those indications for CT and MRI, respectively, without a match that occurred in at least three instances. Indications of CT examinations without a match 160
generally related to cancer follow-up and symptoms not matching a variant (ie, abdominal pain not localized to one quadrant; weight loss). Indications of MRI examinations without a match represented a broader array of clinical conditions, including screening and follow-up of hepatocellular carcinoma, elevated PSA in a patient with no prior biopsy or one prior negative biopsy, surveillance of known solid renal mass, and others. Of 258 studies with a match, 85% received a score of 7 or greater, indicating appropriateness. However, 13 cases with a score less than 7 were considered appropriate based on exceptions specified in the AC (MRI performed in prostate cancer patient on active surveillance [n = 12]; MRI performed in patient with recurrent urinary tract infections to assess for urethral diverticulum [n = 1]). Thus, 92% of studies matching the AC were appropriate, whereas 8% (21/258) were inappropriate. The use of CTwas appropriate in a significantly higher fraction than MRI (96% vs. 86%, P = .009). (Table 3) Appropriate examinations were significantly more likely to provide a clinically significant result than inappropriate studies (48% vs. 24%, respectively, P = .041). A significant result was related to the indication in 93% of appropriate examinations compared to 80% of not appropriate examinations. Although this difference was not significant (P = .204), post hoc assessment demonstrated that a total sample size of 1238 examinations would have been needed to detect significance for a difference of this magnitude. (Table 4) The 21 inappropriate studies comprised 14 MRI and seven CT. The appropriate test in these cases would have been no imaging (n = 4), ultrasound (n = 7), CT rather than MRI (n = 6), MRI rather than CT (n = 1), same modality (CT) but with contrast (n = 2), or choice of same modality (MRI) but without contrast or of an alternate modality (CT or positron emission tomography rather than MRI) (n = 1).
DISCUSSION We evaluated outpatient CT and MRI examinations performed at a major academic center in terms of the level of appropriateness using the ACR AC as reference. A key aspect of our study design was the inclusion only of studies in which the clinical encounter generating the imaging request was available for review within our electronic medical record. This measure was important given the impact of a broad spectrum of clinical features on the appropriateness scores assigned to various imaging studies for a given condition and helped to ensure the reliability of the assigned scores. Using this approach, we observed that most cases (approximately 90%), which were included, were appropriate. This high fraction is particularly noteworthy given the absence of any formal clinical decision support to guide imaging orders at our institution at the time of this study. An earlier study of outpatient CT and MRI examinations in a single practice reported a lower overall frequency of appropriateness of 74% (11). However, in that study, the reference criteria were proprietary guidelines from a radiology benefit management company
Academic Radiology, Vol 22, No 2, February 2015
APPLYING ACR CRITERIA TO ABDOMINOPELVIC CT/MRI
TABLE 3. Classification of Imaging Examinations Using the ACR AC Characteristic y
Match ACR AC Appropriate examinationz
Entire Cohort
CT
MRI
P*
45% (258/570) 92% (237/258)
52% (158/304) 96% (151/158)
38% (100/266) 86% (86/100)
<.001 .009
AC, Appropriateness Criteria; ACR, American College of Radiology; CT, computed tomography; MRI, magnetic resonance imaging. *Comparison of CT and MRI for given measure. y Fraction of examinations matching variant of clinical condition within ACR AC. z Fraction of conditions matching variant classified as appropriate based on assigned score of 7 or greater or exception specified within the AC text.
TABLE 4. Association Between Imaging Examination Appropriateness and Examination Results
Characteristic
Appropriate Examinations
Not Appropriate Examinations
P
Significant result Related to indication*
48% (113/237) 93% (105/113)
24% (5/21) 80% (4/5)
.041 .204
*Computed as fraction of those examinations with significant result.
that were based on the ACR AC in combination with other guidelines (11). Moreover, the prior study comprised only imaging examinations originating from primary care clinics (11), whereas our sample contains examinations requested by a broad array of primary care and specialist physicians. Primary care physicians deal with a wider range of clinical conditions and imaging examinations, possibly creating greater challenge in maintaining familiarity with the criteria and thereby contributing to these differences in results (11). MRI was not appropriate in a slightly higher fraction of examinations compared to CT. This may relate to the generally lower scores provided for MRI than for CT for many clinical conditions in the absence of clear evidence supporting the greater utility of MRI in the given clinical setting. Such assignments are expected in view of MRI’s greater cost (17,18). However, there are various reasons an MRI may have been ordered in a given patient, for instance, concerns of radiation exposure from CT, goal of receiving a definitive diagnosis, or other individual case-specific factors that may not be anticipated by the guidelines. The complexity of these decisions is reflected by a revision of the ACR AC to currently consider MRI with intravenous contrast an appropriate test in the setting of postoperative surveillance of renal cell carcinoma (8); previously, only contrast-enhanced CTwas considered appropriate in this setting (19). Thus, outpatient MRI may represent a particular area for continued focus, both in terms of education of referring physicians regarding current guidelines and assessment for any potential further modifications of existing AC that may be warranted. Examinations that were not appropriate were associated with a variety of preferred alternative approaches. Frequently, either ultrasound or no imaging at all was appropriate, rather than the requested CTor MRI, consistent with the guidelines’
aim of avoiding excess cost and unnecessary advanced imaging. Nonetheless, it may be difficult to consistently advise physicians to avoid imaging when facing a diagnostic dilemma, suggesting an area in which educational initiatives targeting other specialties is warranted. On the other hand, educating ordering physicians to perform the same modality, but to simply alter the request for contrast, may be easier to implement in practice, even if addressing a less frequent issue in our cohort. Although most examinations with a relevant clinical condition were appropriate, the presence of a relevant AC condition in only 45% of cases is highly concerning. This low frequency may undermine the impact of the recent legislation calling for enhanced implementation of the guidelines. This frequency of cases with a relevant condition is notably lower than that reported by Martin et al. (16), who observed in a study of outpatient imaging examinations at their center that 64% were a complete match of a particular variant. Although the reason for the difference in results is uncertain, the earlier study only included examinations from a general internal medicine clinic (16). It is possible that imaging ordered in such a context may be more likely to deal with common conditions and consequently be more likely to have a relevant condition within the AC. Nevertheless, the lack of a relevant condition in the high fraction observed in our study appears to provide a critical area for improvement of the AC. In many instances, this could be achieved by simply generalizing the clinical conditions and variants associated with existing AC. For instance, the requirement that abdominal pain be localized to a single quadrant and that surveillance after cancer treatment be performed for ‘‘suspected liver metastases’’ resulted in a high volume of CT studies obtained for abdominal pain and cancer follow-up, respectively, being classified as not strictly matching the AC, despite similarity in these indications in a broad sense. The lack of a relevant clinical condition was particularly alarming for MRI examinations, which had no relevant condition in over 60% of instances. A past study assessing only MRI examinations ordered at a military primary care center reported no relevant condition in approximately 50% (1). The particularly high frequency in our present study likely relates to our center’s own practice patterns in which specialists routinely order MRI as their standard practice for specific clinical contexts not addressed by the AC. Examples include 161
ROSENKRANTZ ET AL
hepatocellular carcinoma (HCC) surveillance and follow-up in patients with cirrhosis or other underlying liver disease, and planning of image-guided prostate biopsy in patients with either no prior biopsy or a single prior negative biopsy. These are areas in which improvements in imaging technology have led to evolution in practice patterns over time (20,21), highlighting the need for dynamic AC that evolve at a sufficient pace to maintain broad relevance to modern practice. It is reassuring that a significantly higher fraction of appropriate studies yielded a significant result, comparable to the findings of Lehnert and Bree (11). This observation helps validate the value of the AC in mitigating the use of imaging likely to provide a negative result. Indeed, the AC were designed to ideally select for examinations expected to have maximal diagnostic yield, when balanced with cost and imaging-related risks (2,4). The significant result was related to the examination indication in over 90% of appropriate studies, further demonstrating the value of the AC. Although studies that were not appropriate infrequently had a significant result, it is possible that in such cases the significant result could have been obtained with a cheaper or safer examination. Nonetheless, knowledge of those examinations scored as not appropriate that most commonly yielded a significant result and consideration of whether such examinations are deemed appropriate based on other available imaging guidelines besides the ACR AC could be used to inform future revisions of existing AC. Our findings are particularly relevant given recent legislation calling for clinical decision support based on endorsed appropriateness guidelines in the ordering of imaging studies (14). On the one hand, at our center, only a small (although nontrivial) fraction of examination requests would be impacted: The legislation states that imaging requests for which there is no relevant guideline will be exempt (14), and most examinations with a relevant guideline in our cohort were indeed appropriate. However, for the legislation to achieve its intended effect of generating more effective and standardized ordering of imaging on a national basis, the ACR AC will need to be greatly expanded to substantially lower the fraction of exempt requests. We have identified the most common scenarios at our center in which requested imaging lacks a relevant condition, which may help inform such future efforts. A number of limitations of our study warrant mention. First, our cohort comprised imaging examinations performed in the outpatient setting at a single institution over a 1-month period; the fraction of appropriateness may therefore differ in other settings. In particular, as previously noted, our referral base at a tertiary care academic medical center may not be representative of general medical practices. Nonetheless, as high-end relatively expensive imaging may tend to occur at tertiary care centers, it is important to understand trends for this practice setting. Also, although we compared associations between appropriateness and yield of a significant result on imaging, we did not assess actual long-term clinical outcomes. In addition, as our study was retrospective in nature, we do not know the impact of a prospective intervention, such as that 162
Academic Radiology, Vol 22, No 2, February 2015
enacted with the Protecting Access to Medicare Act of 2014, on appropriateness. Finally, we did not evaluate the appropriateness of outpatient imaging examinations on the basis of existing guidelines from other societies, such as the National Comprehensive Cancer Network (9) and American Society of Clinical Oncology (10). However, these particular guidelines are much narrower in scope, nonuniform in recommendations for a given condition, and depend heavily on knowledge of clinical factors, for instance advising that decisions regarding imaging surveillance for certain cancers be guided by the patient’s presumed risk of recurrence and functional status (9,10).
CONCLUSIONS We applied the ACR AC to outpatient CTand MRI examinations at a single large academic medical center. On the one hand, most examinations with a relevant clinical condition were appropriate, and appropriate examinations were significantly more likely to have a significant result than not appropriate examinations. On the other hand, most examinations, including 62% of MRI examinations, had no relevant clinical condition. Thus, although the ACR AC promote value in imaging utilization, expansion of the AC to apply to a larger fraction of imaging examinations, including through the development of new ACR AC conditions and variants and through modifications of existing AC, is warranted in consideration of federal policy more tightly linking the AC to clinical practice.
REFERENCES 1. Levy G, Blachar A, Goldstein L, et al. Nonradiologist utilization of American College of Radiology Appropriateness Criteria in a preauthorization center for MRI requests: applicability and effects. AJR Am J Roentgenol 2006; 187: 855–858. 2. Sistrom CL, Honeyman JC. Relational data model for the American College of Radiology Appropriateness Criteria. J Digit Imaging 2002; 15: 216–225. 3. Sistrom CL, American College of Radiology. In support of the ACR Appropriateness Criteria. J Am Coll Radiol 2008; 5:630–635; discussion 6–7. 4. Sistrom CL. The ACR Appropriateness Criteria: translation to practice and research. J Am Coll Radiol 2005; 2:61–67. 5. Leschied JR, Knoepp US, Hoff CN, et al. Emergency radiology elective improves second-year medical students’ perceived confidence and knowledge of appropriate imaging utilization. Acad Radiol 2013; 20:1168–1176. 6. Chiunda AB, Mohammed TL. Knowledge of ACR thoracic imaging Appropriateness Criteria(R) among trainees: one institution’s experience. Acad Radiol 2012; 19:635–639. 7. Strother MK, Robert EC, Cobb JG, et al. Reduction in the number and associated costs of unindicated dual-phase head CT examinations after a quality improvement initiative. AJR Am J Roentgenol 2013; 201: 1049–1056. 8. American College of Radiology. ACR Appropriateness Criteria. Available at: http://www.acr.org/quality-safety/appropriateness-criteria. Accessed April 7, 2014. 9. National Comprehensive Cancer Network. NCCN Guidelines for Patients. By cancer type. Available at: http://www.nccn.org/patients/ guidelines/cancers.aspx. Accessed August 4, 2014. 10. American Society of Clinical Oncology. Guidelines. Available at: http:// www.asco.org/quality-guidelines/guidelines. Accessed August 4, 2014. 11. Lehnert BE, Bree RL. Analysis of appropriateness of outpatient CT and MRI referred from primary care clinics at an Academic Medical Center:
Academic Radiology, Vol 22, No 2, February 2015
12.
13.
14.
15.
16.
17.
how critical is the need for improved decision support? J Am Coll Radiol 2010; 7:192–197. Vartanians VM, Sistrom CL, Weilburg JB, et al. Increasing the appropriateness of outpatient imaging: effects of a barrier to ordering low-yield examinations. Radiology 2010; 255:842–849. Covington MF, Agan DL, Liu Y, et al. Teaching cost-conscious medicine: impact of a simple educational intervention on appropriate abdominal imaging at a community-based teaching hospital. J Grad Med Educ 2013; 5:284–288. United Stated Government Printing Office. H.R. 4302: protecting access to Medicare Act of 2014. Available at: http://www.gpo.gov/fdsys/pkg/BILLS113hr4302enr/pdf/BILLS-113hr4302enr.pdf. Accessed April 7, 2014. Healthcare Informatics. Clarity on the new diagnostic imaging ordering mandate in the ‘‘Doc Fix’’/ICD-10 Law. Available at: http://www. healthcare-informatics.com/article/clarity-new-diagnostic-imaging-orderingmandate-doc-fix-icd-10-law. April 7, 2014. Martin TA, Quiroz FA, Rand SD, et al. Applicability of American College of Radiology Appropriateness Criteria in a general internal medicine clinic. AJR Am J Roentgenol 1999; 173:9–11. Margolis NE, Shaver CM, Rosenkrantz AB. Indeterminate liver and renal lesions: comparison of computed tomography and magnetic resonance im-
APPLYING ACR CRITERIA TO ABDOMINOPELVIC CT/MRI
18.
19.
20.
21.
aging in providing a definitive diagnosis and impact on recommendations for additional imaging. Journal of computer assisted tomography 2013; 37: 882–886. McMahon PM, Halpern EF, Fernandez-del Castillo C, et al. Pancreatic cancer: cost-effectiveness of imaging technologies for assessing resectability. Radiology 2001; 221:93–106. United States Department of Health and Human Services, Agency for Healthcare Research and Quality. National guideline clearinghouse guideline summary: ACR Appropriate Criteria follow-up of renal cell carcinoma. Available at: http://www.guideline.gov/content.aspx?id=1576. Accessed April 10, 2014. Becker-Weidman DJ, Kalb B, Sharma P, et al. Hepatocellular carcinoma lesion characterization: single-institution clinical performance review of multiphase gadolinium-enhanced MR imaging—comparison to prior same-center results after MR systems improvements. Radiology 2011; 261:824–833. Pokorny MR, de Rooij M, Duncan E, et al. Prospective study of diagnostic accuracy comparing prostate cancer detection by transrectal ultrasoundguided biopsy versus magnetic resonance (MR) imaging with subsequent MR-guided biopsy in men without previous prostate biopsies. Eur Urol 2014; 66:22–29.
163
I
n response to concerns of escalating use of imaging services and wide variability in such use, the American College of Radiology (ACR) launched an initiative in 1993 to formulate guidelines to assist in appropriate ordering of imaging studies (1,2). The resulting ACR Appropriateness Criteria (AC) seek to standardize which imaging examinations are ordered for a wide array of clinical conditions and thereby reduce variation and overutilization (3,4). The AC strive to optimize the balance between benefit and risk inherent to imaging examinations by maximizing diagnostic yield and Acad Radiol 2015; 22:158–163 From the Department of Radiology, NYU Langone Medical Center, 660 First Avenue, New York, NY 10016 (A.B.R., K.M., A.D.). Received June 23, 2014; accepted August 12, 2014. Funding: None. Conflicts of interest: None. Address correspondence to: A.B.R. e-mail: Andrew.Rosenkrantz@nyumc. org ªAUR, 2015 http://dx.doi.org/10.1016/j.acra.2014.08.005
158
provision of useful information in consideration of the examination’s costs, radiation exposure, other procedural risks, and possibility of incidental findings (2,4–6). The AC were crafted by multidisciplinary expert panels in an evidence-based fashion using a modified Delphi approach (2). AC are organized in terms of specific clinical conditions and associated variants, ranking for each variant the appropriateness of relevant imaging examinations on a 1–9 scale; a score of 7–9 indicates that the given imaging examination is ‘‘usually appropriate’’ for the given condition (6,7). From the time of their introduction, the AC have been continually expanded and updated by the ACR (4), and currently AC exist for over 197 clinical conditions with over 900 variants (8). The AC are the only guidelines in the United States to direct the ordering of diagnostic imaging examinations for a comprehensive spectrum of clinical scenarios, as reflected by the large number of conditions and variants addressed (2). In comparison, other available guidelines, such as the National
Academic Radiology, Vol 22, No 2, February 2015
APPLYING ACR CRITERIA TO ABDOMINOPELVIC CT/MRI
Comprehensive Cancer Network (9) and American Society of Clinical Oncology (10) guidelines, generally direct imaging examinations in limited contexts, such as cancer staging and post-treatment surveillance. The AC have been applied for numerous purposes, such as by private insurers for directing patient benefit policies (11), by academic institutions as the basis for the development of clinical decision support systems (12), and by radiologists as a basis of educating other physicians in appropriate examination ordering (7,13). More recently, the ‘‘Protecting Access to Medicare Act of 2014’’ Act (H.R. 4302, signed into law April 1, 2014) requires that health care providers consult appropriateness use criteria endorsed by national professional medical societies when ordering imaging services (14). It is expected that the ACR AC will be a primary set of guidelines used in fulfilling this requirement (15). This legislation enhances the potential impact of the ACR AC given the large influence of Medicare policy on medical practice in the United States. Thus, as the legislation undergoes implementation, it is important to recognize any limitations or weaknesses of the ACR AC. However, only a small amount of the literature has assessed the overall level of appropriateness of diagnostic imaging examinations relative to the ACR AC or the association of such appropriateness with outcomes (11). Such knowledge would be useful in guiding continued policy development. Therefore, we conducted this study to evaluate the appropriateness of outpatient abdominal and pelvic computed tomography (CT) and magnetic resonance imaging (MRI) examinations using the ACR AC, as well as associations between appropriateness and imaging results.
this retrospective study to reliably make such determinations. This process yielded a final cohort of 570 outpatient studies (304 CT [255 with intravenous (IV) contrast, 49 without IV contrast], 266 MRI [262 with IV contrast, four without IV contrast]).
METHODS Selection of Imaging Examinations
This retrospective study performed at a tertiary care academic medical center was Health Insurance Portability and Accountability Act (HIPAA)-compliant and approved by our institutional review board with a waiver of the requirement for written informed consent. We searched a departmental database for all abdominal or pelvic outpatient CT and MRI examinations performed during a 4-week interval in September 2013, identifying 1743 examinations (968 CT and 775 MRI); Emergency Department examinations were not included. The electronic medical record was then searched to identify whether the full note for the clinical encounter generating the imaging request was available for review. This requirement ensured accurate classification of the appropriateness of each imaging examination in view of the numerous variants comprising the AC and the potentially incomplete clinical history contained only in the imaging reports; this approach was also used in a prior study (16). Exclusion of examinations on this basis did not imply that the examination failed to match an AC variant or was inappropriate, but solely that we lacked sufficient information in
Imaging Examination Review
Available clinical and imaging data were reviewed in consensus by two radiologists, both fellowship-trained abdominal radiologists with one and six years of experience. For each imaging examination, the full note from the clinical encounter generating the request was reviewed to identify the study indication and any other clinical factors (ie, signs and symptoms, laboratory values, known diagnoses) contributing to the reason for imaging. Then, the AC were reviewed to identify a match between the examination indication and a variant of a clinical condition within the AC (8). If no match was identified, then no further analysis of the case was performed. If a match was identified, then the appropriateness score of the performed imaging study was recorded. Examinations receiving a score of 7–9 were considered appropriate. In addition, select examinations receiving a score of 4–6 (‘‘may be appropriate’’) were reclassified as appropriate because of meeting specific exceptions noted in the AC. For cases not classified as appropriate, the imaging examination that would receive the highest appropriateness score for the given indication was recorded; if no imaging examination would receive a score of 7 or greater for the indication, then it was recorded that no imaging was considered appropriate for the indication. For all imaging studies included in the analysis, the presence of a significant result was also recorded. A significant result was defined as any finding described within the report requiring further intervention or follow-up. Finally, for studies with a significant result, whether this result was related to the primary indication for the study was also recorded. Statistical Assessment
The frequency of outpatient studies matching a variant of a clinical condition, of such studies that were considered appropriate, of appropriate studies with a significant result, and of studies with a significant result in which the result was related to the primary indication, were computed using standard summary statistics. The frequencies of indications without a match were tabulated separately for CT and MRI. Fisher’s exact test was used to compare CT and MRI in terms of both frequency of a match and frequency of appropriateness, as well as to compare studies that were and were not appropriate in terms of significance of result and relation to the indication. For performed inappropriate studies, the distribution of examination type and advised alternate imaging examination using the AC were recorded. P values are two-sided and considered statistically significant at P < .05. For tests in which the obtained P value did not indicate statistical significance, post hoc assessment was performed to determine 159
ROSENKRANTZ ET AL
Academic Radiology, Vol 22, No 2, February 2015
TABLE 1. Most Common Indications of Computed Tomography Examinations not Matching an Appropriateness Criteria Variant* Indication
n
Colon cancer follow-up Melanoma follow-up Weight loss; no abdominal pain Abdominal pain not localized to one quadrant; no fever, recent surgery, or laboratory abnormality Lymphoma follow-up Pancreatic cancer follow-up Weight loss and abdominal pain not localized to one quadrant Hepatocellular carcinoma screening Anemia Lung cancer follow-up Breast cancer follow-up, not stage I Neuroendocrine tumor follow-up
14 13 9 8 7 7 6 3 3 3 3 3
*Indications listed if n $ 2.
TABLE 2. Most Common Indications of Magnetic Resonance Imaging Examinations not Matching an Appropriateness Criteria Variant* Indication
n
Hepatocellular carcinoma screening Elevated PSA, no prior prostate biopsy Hepatocellular carcinoma follow-up Solid renal mass, surveillance Lower extremity edema/ulcer, concern for iliac vein compression Fibroids, pretreatment planning; no acute pelvic pain Ureteropelvic junction (UPJ) obstruction Elevated PSA, one prior negative prostate biopsy Pancreatic cyst follow-up Tuberous sclerosis; screening for renal mass
31 14 11 9 9 8 5 5 5 4
*Indications listed if n $ 2.
whether the comparison had sufficient power to detect significance of the observed difference at a power of 20%. Analysis was performed using software (MedCalc for Windows, version 12.7; MedCalc Software, Ostend, Belgium).
RESULTS Forty-five percent of outpatient imaging examinations matched a variant of a clinical condition within the AC: 52% of CT and 34% of MRI (P < .001). All examinations with a match corresponded with conditions and variants described within the gastrointestinal (n = 59), urologic (n = 145), vascular (n = 29), women’s (n = 24), and thoracic (n = 1) topic areas. Tables 1 and 2 list those indications for CT and MRI, respectively, without a match that occurred in at least three instances. Indications of CT examinations without a match 160
generally related to cancer follow-up and symptoms not matching a variant (ie, abdominal pain not localized to one quadrant; weight loss). Indications of MRI examinations without a match represented a broader array of clinical conditions, including screening and follow-up of hepatocellular carcinoma, elevated PSA in a patient with no prior biopsy or one prior negative biopsy, surveillance of known solid renal mass, and others. Of 258 studies with a match, 85% received a score of 7 or greater, indicating appropriateness. However, 13 cases with a score less than 7 were considered appropriate based on exceptions specified in the AC (MRI performed in prostate cancer patient on active surveillance [n = 12]; MRI performed in patient with recurrent urinary tract infections to assess for urethral diverticulum [n = 1]). Thus, 92% of studies matching the AC were appropriate, whereas 8% (21/258) were inappropriate. The use of CTwas appropriate in a significantly higher fraction than MRI (96% vs. 86%, P = .009). (Table 3) Appropriate examinations were significantly more likely to provide a clinically significant result than inappropriate studies (48% vs. 24%, respectively, P = .041). A significant result was related to the indication in 93% of appropriate examinations compared to 80% of not appropriate examinations. Although this difference was not significant (P = .204), post hoc assessment demonstrated that a total sample size of 1238 examinations would have been needed to detect significance for a difference of this magnitude. (Table 4) The 21 inappropriate studies comprised 14 MRI and seven CT. The appropriate test in these cases would have been no imaging (n = 4), ultrasound (n = 7), CT rather than MRI (n = 6), MRI rather than CT (n = 1), same modality (CT) but with contrast (n = 2), or choice of same modality (MRI) but without contrast or of an alternate modality (CT or positron emission tomography rather than MRI) (n = 1).
DISCUSSION We evaluated outpatient CT and MRI examinations performed at a major academic center in terms of the level of appropriateness using the ACR AC as reference. A key aspect of our study design was the inclusion only of studies in which the clinical encounter generating the imaging request was available for review within our electronic medical record. This measure was important given the impact of a broad spectrum of clinical features on the appropriateness scores assigned to various imaging studies for a given condition and helped to ensure the reliability of the assigned scores. Using this approach, we observed that most cases (approximately 90%), which were included, were appropriate. This high fraction is particularly noteworthy given the absence of any formal clinical decision support to guide imaging orders at our institution at the time of this study. An earlier study of outpatient CT and MRI examinations in a single practice reported a lower overall frequency of appropriateness of 74% (11). However, in that study, the reference criteria were proprietary guidelines from a radiology benefit management company
Academic Radiology, Vol 22, No 2, February 2015
APPLYING ACR CRITERIA TO ABDOMINOPELVIC CT/MRI
TABLE 3. Classification of Imaging Examinations Using the ACR AC Characteristic y
Match ACR AC Appropriate examinationz
Entire Cohort
CT
MRI
P*
45% (258/570) 92% (237/258)
52% (158/304) 96% (151/158)
38% (100/266) 86% (86/100)
<.001 .009
AC, Appropriateness Criteria; ACR, American College of Radiology; CT, computed tomography; MRI, magnetic resonance imaging. *Comparison of CT and MRI for given measure. y Fraction of examinations matching variant of clinical condition within ACR AC. z Fraction of conditions matching variant classified as appropriate based on assigned score of 7 or greater or exception specified within the AC text.
TABLE 4. Association Between Imaging Examination Appropriateness and Examination Results
Characteristic
Appropriate Examinations
Not Appropriate Examinations
P
Significant result Related to indication*
48% (113/237) 93% (105/113)
24% (5/21) 80% (4/5)
.041 .204
*Computed as fraction of those examinations with significant result.
that were based on the ACR AC in combination with other guidelines (11). Moreover, the prior study comprised only imaging examinations originating from primary care clinics (11), whereas our sample contains examinations requested by a broad array of primary care and specialist physicians. Primary care physicians deal with a wider range of clinical conditions and imaging examinations, possibly creating greater challenge in maintaining familiarity with the criteria and thereby contributing to these differences in results (11). MRI was not appropriate in a slightly higher fraction of examinations compared to CT. This may relate to the generally lower scores provided for MRI than for CT for many clinical conditions in the absence of clear evidence supporting the greater utility of MRI in the given clinical setting. Such assignments are expected in view of MRI’s greater cost (17,18). However, there are various reasons an MRI may have been ordered in a given patient, for instance, concerns of radiation exposure from CT, goal of receiving a definitive diagnosis, or other individual case-specific factors that may not be anticipated by the guidelines. The complexity of these decisions is reflected by a revision of the ACR AC to currently consider MRI with intravenous contrast an appropriate test in the setting of postoperative surveillance of renal cell carcinoma (8); previously, only contrast-enhanced CTwas considered appropriate in this setting (19). Thus, outpatient MRI may represent a particular area for continued focus, both in terms of education of referring physicians regarding current guidelines and assessment for any potential further modifications of existing AC that may be warranted. Examinations that were not appropriate were associated with a variety of preferred alternative approaches. Frequently, either ultrasound or no imaging at all was appropriate, rather than the requested CTor MRI, consistent with the guidelines’
aim of avoiding excess cost and unnecessary advanced imaging. Nonetheless, it may be difficult to consistently advise physicians to avoid imaging when facing a diagnostic dilemma, suggesting an area in which educational initiatives targeting other specialties is warranted. On the other hand, educating ordering physicians to perform the same modality, but to simply alter the request for contrast, may be easier to implement in practice, even if addressing a less frequent issue in our cohort. Although most examinations with a relevant clinical condition were appropriate, the presence of a relevant AC condition in only 45% of cases is highly concerning. This low frequency may undermine the impact of the recent legislation calling for enhanced implementation of the guidelines. This frequency of cases with a relevant condition is notably lower than that reported by Martin et al. (16), who observed in a study of outpatient imaging examinations at their center that 64% were a complete match of a particular variant. Although the reason for the difference in results is uncertain, the earlier study only included examinations from a general internal medicine clinic (16). It is possible that imaging ordered in such a context may be more likely to deal with common conditions and consequently be more likely to have a relevant condition within the AC. Nevertheless, the lack of a relevant condition in the high fraction observed in our study appears to provide a critical area for improvement of the AC. In many instances, this could be achieved by simply generalizing the clinical conditions and variants associated with existing AC. For instance, the requirement that abdominal pain be localized to a single quadrant and that surveillance after cancer treatment be performed for ‘‘suspected liver metastases’’ resulted in a high volume of CT studies obtained for abdominal pain and cancer follow-up, respectively, being classified as not strictly matching the AC, despite similarity in these indications in a broad sense. The lack of a relevant clinical condition was particularly alarming for MRI examinations, which had no relevant condition in over 60% of instances. A past study assessing only MRI examinations ordered at a military primary care center reported no relevant condition in approximately 50% (1). The particularly high frequency in our present study likely relates to our center’s own practice patterns in which specialists routinely order MRI as their standard practice for specific clinical contexts not addressed by the AC. Examples include 161
ROSENKRANTZ ET AL
hepatocellular carcinoma (HCC) surveillance and follow-up in patients with cirrhosis or other underlying liver disease, and planning of image-guided prostate biopsy in patients with either no prior biopsy or a single prior negative biopsy. These are areas in which improvements in imaging technology have led to evolution in practice patterns over time (20,21), highlighting the need for dynamic AC that evolve at a sufficient pace to maintain broad relevance to modern practice. It is reassuring that a significantly higher fraction of appropriate studies yielded a significant result, comparable to the findings of Lehnert and Bree (11). This observation helps validate the value of the AC in mitigating the use of imaging likely to provide a negative result. Indeed, the AC were designed to ideally select for examinations expected to have maximal diagnostic yield, when balanced with cost and imaging-related risks (2,4). The significant result was related to the examination indication in over 90% of appropriate studies, further demonstrating the value of the AC. Although studies that were not appropriate infrequently had a significant result, it is possible that in such cases the significant result could have been obtained with a cheaper or safer examination. Nonetheless, knowledge of those examinations scored as not appropriate that most commonly yielded a significant result and consideration of whether such examinations are deemed appropriate based on other available imaging guidelines besides the ACR AC could be used to inform future revisions of existing AC. Our findings are particularly relevant given recent legislation calling for clinical decision support based on endorsed appropriateness guidelines in the ordering of imaging studies (14). On the one hand, at our center, only a small (although nontrivial) fraction of examination requests would be impacted: The legislation states that imaging requests for which there is no relevant guideline will be exempt (14), and most examinations with a relevant guideline in our cohort were indeed appropriate. However, for the legislation to achieve its intended effect of generating more effective and standardized ordering of imaging on a national basis, the ACR AC will need to be greatly expanded to substantially lower the fraction of exempt requests. We have identified the most common scenarios at our center in which requested imaging lacks a relevant condition, which may help inform such future efforts. A number of limitations of our study warrant mention. First, our cohort comprised imaging examinations performed in the outpatient setting at a single institution over a 1-month period; the fraction of appropriateness may therefore differ in other settings. In particular, as previously noted, our referral base at a tertiary care academic medical center may not be representative of general medical practices. Nonetheless, as high-end relatively expensive imaging may tend to occur at tertiary care centers, it is important to understand trends for this practice setting. Also, although we compared associations between appropriateness and yield of a significant result on imaging, we did not assess actual long-term clinical outcomes. In addition, as our study was retrospective in nature, we do not know the impact of a prospective intervention, such as that 162
Academic Radiology, Vol 22, No 2, February 2015
enacted with the Protecting Access to Medicare Act of 2014, on appropriateness. Finally, we did not evaluate the appropriateness of outpatient imaging examinations on the basis of existing guidelines from other societies, such as the National Comprehensive Cancer Network (9) and American Society of Clinical Oncology (10). However, these particular guidelines are much narrower in scope, nonuniform in recommendations for a given condition, and depend heavily on knowledge of clinical factors, for instance advising that decisions regarding imaging surveillance for certain cancers be guided by the patient’s presumed risk of recurrence and functional status (9,10).
CONCLUSIONS We applied the ACR AC to outpatient CTand MRI examinations at a single large academic medical center. On the one hand, most examinations with a relevant clinical condition were appropriate, and appropriate examinations were significantly more likely to have a significant result than not appropriate examinations. On the other hand, most examinations, including 62% of MRI examinations, had no relevant clinical condition. Thus, although the ACR AC promote value in imaging utilization, expansion of the AC to apply to a larger fraction of imaging examinations, including through the development of new ACR AC conditions and variants and through modifications of existing AC, is warranted in consideration of federal policy more tightly linking the AC to clinical practice.
REFERENCES 1. Levy G, Blachar A, Goldstein L, et al. Nonradiologist utilization of American College of Radiology Appropriateness Criteria in a preauthorization center for MRI requests: applicability and effects. AJR Am J Roentgenol 2006; 187: 855–858. 2. Sistrom CL, Honeyman JC. Relational data model for the American College of Radiology Appropriateness Criteria. J Digit Imaging 2002; 15: 216–225. 3. Sistrom CL, American College of Radiology. In support of the ACR Appropriateness Criteria. J Am Coll Radiol 2008; 5:630–635; discussion 6–7. 4. Sistrom CL. The ACR Appropriateness Criteria: translation to practice and research. J Am Coll Radiol 2005; 2:61–67. 5. Leschied JR, Knoepp US, Hoff CN, et al. Emergency radiology elective improves second-year medical students’ perceived confidence and knowledge of appropriate imaging utilization. Acad Radiol 2013; 20:1168–1176. 6. Chiunda AB, Mohammed TL. Knowledge of ACR thoracic imaging Appropriateness Criteria(R) among trainees: one institution’s experience. Acad Radiol 2012; 19:635–639. 7. Strother MK, Robert EC, Cobb JG, et al. Reduction in the number and associated costs of unindicated dual-phase head CT examinations after a quality improvement initiative. AJR Am J Roentgenol 2013; 201: 1049–1056. 8. American College of Radiology. ACR Appropriateness Criteria. Available at: http://www.acr.org/quality-safety/appropriateness-criteria. Accessed April 7, 2014. 9. National Comprehensive Cancer Network. NCCN Guidelines for Patients. By cancer type. Available at: http://www.nccn.org/patients/ guidelines/cancers.aspx. Accessed August 4, 2014. 10. American Society of Clinical Oncology. Guidelines. Available at: http:// www.asco.org/quality-guidelines/guidelines. Accessed August 4, 2014. 11. Lehnert BE, Bree RL. Analysis of appropriateness of outpatient CT and MRI referred from primary care clinics at an Academic Medical Center:
Academic Radiology, Vol 22, No 2, February 2015
12.
13.
14.
15.
16.
17.
how critical is the need for improved decision support? J Am Coll Radiol 2010; 7:192–197. Vartanians VM, Sistrom CL, Weilburg JB, et al. Increasing the appropriateness of outpatient imaging: effects of a barrier to ordering low-yield examinations. Radiology 2010; 255:842–849. Covington MF, Agan DL, Liu Y, et al. Teaching cost-conscious medicine: impact of a simple educational intervention on appropriate abdominal imaging at a community-based teaching hospital. J Grad Med Educ 2013; 5:284–288. United Stated Government Printing Office. H.R. 4302: protecting access to Medicare Act of 2014. Available at: http://www.gpo.gov/fdsys/pkg/BILLS113hr4302enr/pdf/BILLS-113hr4302enr.pdf. Accessed April 7, 2014. Healthcare Informatics. Clarity on the new diagnostic imaging ordering mandate in the ‘‘Doc Fix’’/ICD-10 Law. Available at: http://www. healthcare-informatics.com/article/clarity-new-diagnostic-imaging-orderingmandate-doc-fix-icd-10-law. April 7, 2014. Martin TA, Quiroz FA, Rand SD, et al. Applicability of American College of Radiology Appropriateness Criteria in a general internal medicine clinic. AJR Am J Roentgenol 1999; 173:9–11. Margolis NE, Shaver CM, Rosenkrantz AB. Indeterminate liver and renal lesions: comparison of computed tomography and magnetic resonance im-
APPLYING ACR CRITERIA TO ABDOMINOPELVIC CT/MRI
18.
19.
20.
21.
aging in providing a definitive diagnosis and impact on recommendations for additional imaging. Journal of computer assisted tomography 2013; 37: 882–886. McMahon PM, Halpern EF, Fernandez-del Castillo C, et al. Pancreatic cancer: cost-effectiveness of imaging technologies for assessing resectability. Radiology 2001; 221:93–106. United States Department of Health and Human Services, Agency for Healthcare Research and Quality. National guideline clearinghouse guideline summary: ACR Appropriate Criteria follow-up of renal cell carcinoma. Available at: http://www.guideline.gov/content.aspx?id=1576. Accessed April 10, 2014. Becker-Weidman DJ, Kalb B, Sharma P, et al. Hepatocellular carcinoma lesion characterization: single-institution clinical performance review of multiphase gadolinium-enhanced MR imaging—comparison to prior same-center results after MR systems improvements. Radiology 2011; 261:824–833. Pokorny MR, de Rooij M, Duncan E, et al. Prospective study of diagnostic accuracy comparing prostate cancer detection by transrectal ultrasoundguided biopsy versus magnetic resonance (MR) imaging with subsequent MR-guided biopsy in men without previous prostate biopsies. Eur Urol 2014; 66:22–29.
163