- Email: [email protected]
Institutional Implementation of a Structured Reporting System: Our Experience with the Brain Tumor Reporting and Data System
ARTICLE IN PRESS
Original Investigation
Institutional Implementation of a Structured Reporting System: Our Experience with the Brain Tumor Reporting and Data System Ashwani Gore, MD, Michael J. Hoch, MD, Hui-Kuo G. Shu, MD, PhD, Jeffrey J. Olson, MD, Alfredo D. Voloschin, MD, Brent D. Weinberg, MD, PhD
Abbreviations BT-RADS Brain Tumor Reporting and Data System FLAIR fluid attenuation inversion recovery MRI magnetic resonance imaging RANO Response Assessment in Neuro-Oncology WHO World Health Organization
Rationale and Objectives: Analyze the impact of implementing a structured reporting system for primary brain tumors, the Brain Tumor Reporting and Data System, on attitudes toward radiology reports at a single institution. Materials and Methods: Following Institutional Review Board approval, an initial 22 question, 5 point (1—worst to 5—best), survey was sent to faculty members, house staff members, and nonphysician providers at our institution who participate in the direct care of brain tumor patients. Results were used to develop a structured reporting strategy for brain tumors which was implemented across an entire neuroradiology section in a staged approach. Nine months following structured reporting implementation, a follow-up 27 question survey was sent to the same group of providers. Keyword search of radiology reports was used to assess usage of Brain Tumor Reporting and Data System over time. Results: Fifty-three brain tumor care providers responded to the initial survey and 38 to the follow-up survey. After implementing BT-RADS, respondents reported improved attitudes across multiple areas including: report consistency (4.3 vs. 3.4; p < 0.001), report ambiguity (4.2 vs. 3.2, p < 0.001), radiologist/physician communication (4.5 vs. 3.8; p < 0.001), facilitation of patient management (4.2 vs. 3.6; p = 0.003), and confidence in reports (4.3 vs. 3.5; p < 0.001). Providers were more satisfied with the BTRADS structured reporting system (4.3 vs. 3.7; p = 0.04). Use of the reporting template progressively increased with 81% of brain tumor reports dictated using the new template by 9 months. Conclusion: Implementing a structured template for brain tumor imaging improves perception of radiology reports among radiologists and referring providers involved in the care of brain tumor patients. Key Words: Structured template; Patient care; Response criteria; Brain tumor; Glioblastoma. © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.
INTRODUCTION
T
he radiology report is an invaluable source of information to the ordering clinician. Clear and accurate communication of imaging results plays a crucial role in
Acad Radiol 2019; &:1 7 From the Department of Radiology and Imaging Sciences, Emory University Hospital, 1364 Clifton Rd. NE, Atlanta, GA (A.G., M.J.H., B.D.W.); Department of Radiation Oncology, Emory University Hospital, Atlanta, Georgia (H.-K.G.S.); Department of Neurosurgery, Emory University Hospital, Atlanta, Georgia (J.J.O.); Department of Hematology and Medical Oncology, Emory University Hospital, Atlanta, Georgia (A.D.V.). Received December 10, 2018; revised December 17, 2018; accepted December 18, 2018. Financial Disclosure: None. Address correspondence to: B.D.W. e-mails: [email protected], [email protected] © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.acra.2018.12.023
guiding appropriate patient management. Traditionally, reports are created in a narrative style, free-prose format that can sometimes be disorganized, incomplete, or ambiguous, particularly considering interobserver variability and stylistic differences amongst radiologists (1 5). Currently, more attention is being made to the report end-product with a practice shift toward structured reporting (6). Structured reporting focuses on uniformity and consistency by implementing template macros with subheadings, standardized language, and prepopulated pick lists (7). Advantages of this format include improved clarity and efficiency, reduced diagnostic errors, and facilitation of research (6). Disease specific structured reporting as seen with Breast Imaging Reporting and Data System ( and Neck Imaging Reporting and Data System are widely accepted to examples of value-added radiology (8,9). Such reports are appreciated by referring
1
ARTICLE IN PRESS GORE ET AL
clinicians and assist with more confident patient treatment. The development of a structured reporting system with associated clinical decision support for primary brain tumor patients is an opportunity to significantly impact patient-centered care. The poor prognosis of patients with malignant brain tumors underscores the importance of clear and accurate radiology reporting in order to maximize treatment outcomes. Assessment of post-treatment brain tumor response is challenging for the practicing radiologist, as overlap between the appearance of tumor progression and treatment effects can cause ambiguous interpretations which are difficult to translate into management decisions (10,11). Although a variety of response criteria have evolved to assess brain tumor response such as the Levin, World Health Organization, MacDonald, and the Response Assessment in Neuro-Oncology criteria, they are more often utilized in clinical trials and not routinely incorporated in the radiology reports of clinical practice (10,12 14). Factors limiting their routine clinical use include complex criteria, need for multiple measurements, high interobserver variability, and limited understanding among practicing physicians (15). Thus, there is an unmet clinical need for a practical disease-specific structured reporting system for magnetic resonance imaging (MRI) surveillance of post-treatment glioma patients. The Brain Tumor Reporting and Data System (BT-RADS) was designed and implemented to establish a standardized surveillance algorithm for interpreting MRI exams of treatment response in brain tumor patients (16). It was created by a multidisciplinary team of physicians with expertise in neuro-oncology and treatment of patients with brain tumors, in particular gliomas. The BT-RADS structured template includes a managementbased numeric assessment (score 0 4) of MRI findings associated with simple management recommendations that serve to help guide clinical treatment. The objective was to create more concise and understandable reports and allow radiologists to fully commit to a clearly defined interpretation. We have implemented BT-RADS at our academic institution where we are a part of a busy neurooncology service. The goal of our study is to assess the initial usage and impact of BT-RADS at our institution following its implementation.
MATERIALS AND METHODS Initial Attitude Survey
Following Institutional Review Board approval, a 22 question survey was developed to assess attitudes toward brain tumor radiology reports. The survey was based on a five-point Likerttype scale with five representing the most positive response and one representing the least positive response. Nonradiology survey recipients included participants in multidisciplinary brain tumor board from neuro-oncology, neurosurgery, neuropathology, and radiation oncology. Neuroradiology faculty and fellows and all radiology residents were included. The survey was distributed electronically to a total of 136 individuals. 2
Academic Radiology, Vol &, No &&, && 2019
Structured Template Design and Implementation
Considering weaknesses in the conventional reporting strategy based on the initial survey results, a structured reporting template with standardized management suggestions was designed and implemented to better assist with post-treatment brain tumor MRI interpretation. Structured reporting materials included a table of reporting categories, imaging criteria for assigning studies to specific categories, a flowchart to assist with category implementation, and a dictation template (Powerscribe 360). All of these resources are freely accessible online at: www.btrads.com. This reporting system was created as a collaborative effort between radiology, neurosurgery, neurooncology, and radiation oncology. The ultimate goals of this approach were to facilitate more effective communication and improve overall care of brain tumor patients. The details of the BT-RADS scoring system have been reported previously (16). The BT-RADS and structured/management-based reporting template was implemented at our institution in a staged approach. In October 2017, the structured approach was implemented by two attending radiologists on a trial basis. After making iterative changes to the reporting system, the structured reporting method was incorporated in site-wide templates for use by all faculty 4 months after initial introduction in March 2018. A sample report using BT-RADS is illustrated in Supplemental Material A. Template Usage
Keyword search of radiology reports (Philips PerformanceBridge Report Search Tool) was used to assess usage of BTRADS from October 1, 2017 to June 30, 2018. The total numbers of brain tumor MRI reports were queried using MR as the modality with search queries of “brain tumor” or “glioblastoma” or “GBM” or “astrocytoma” or “oligodendroglioma” or “glioma.” The number of structured reports was determined by searching for structured codes in the report (“bt-0” OR “bt-1a” OR “bt-1b” OR “bt-2” OR “bt-3a” OR “bt-3b” OR “bt-3c” OR “bt-4”). The output represented the total number of brain MRI reports that used the BT-RADS structured reporting system. These outputs were separated based on month of usage from October 2017 to June 2018 (9 months) and the percent usage over time was calculated. Additionally, individually reported BT-RADS categories were tabulated during this 9-month period. Follow-up Attitude Survey
A 27-question survey was designed to assess evolving user attitudes toward the BT-RADS structured reporting system. This survey contained the same 22 questions on the original study with the addition of five new questions asking participants to directly compare the prior and new reporting system with regards to uniformity and consistency of reports, ambiguity in interpretations, promoting research, overall satisfaction, and a
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
IMPLEMENTATION OF BRAIN TUMOR STRUCTURED REPORTING
survey. Although participants felt the conventional reporting system communicated relevant findings (mean Likert score of 3.9), lower scores were seen in maintaining consistency across reports (3.4), refraining from ambiguity (3.2), and promoting clear communication between patient and physician (3.4). A general perception was that patients were only sometimes satisfied with their MRI reports (3.4) (Fig 1). Compared to nonradiologists, radiologists felt their reports did not always promote clear communication to ordering physicians (radiologists 3.5 vs. nonradiologists 4.1, p < 0.01). Radiologists felt their conventional reports frequently did not facilitate decision-making for treatment (3.3 vs. 3.9, p < 0.01) and were less conducive to trainee education (3.3 vs. 3.9, p = 0.01). Overall radiologists were not as satisfied compared to nonradiologists with primary brain tumor MRI reports (3.5 vs. 4.0, p = 0.03). Nonetheless, both groups strongly favored findings (4.3 vs. 4.1, p = 0.38) and impression sections (4.4 vs. 4.4, p = 0.77) adhering to a structured format. Following implementation of the structured reporting template, the institutional report search tool yielded a total of 1078 brain tumor MRI dictations from October 2017 to June 2018. Out of these, 536 reports were dictated using the BT-RADS structured template. Percent use of the BT-RADS template was
free response/comment section (Supplemental Material B). On questions asking respondents to directly compare reports before and after template implementation, a five-point scale was again used where 1 indicated much worse, 3 indicated the same, and 5 indicated much better. Statistical Analysis
Results of the post BT-RADS implementation survey was then compared to the initial, preimplementation, free dictation reporting system. Mean differences between survey categories were calculated using a two-tailed t test with equal variances (17). Corrected p values using Bonferroni correction with p < 0.05 was deemed statistically significant. RESULTS The initial survey was divided into four main categories which included the following: details of the existing conventional reporting system, overall satisfaction with existing brain tumor reports, changes to the reporting system, and a free responses/suggestions section (Table 1). A total of 53 providers (27 radiologists and 26 nonradiologists; 40% response rate) responded to the initial
TABLE 1. Opinions Regarding Conventional Brain Tumor Surveillance MRI Reports
How well do the current radiology reports Clearly communicate the relevant findings Maintain consistency Refrain from ambiguity Promote clear communication between patient and physician Promote clear communication between radiologist and ordering physician Facilitate decision-making for treatment Match your practice patterns Facilitate research Facilitate trainee (resident and fellow) education Promote confidence in report findings How strongly do you agree with the following statements Finding sections should adhere to a structured format Impression sections in radiology reports should adhere to a structured format It would be helpful if report impressions provided categories which corresponded to suggested management decisions Overall satisfaction Overall, how satisfied are you with the primary brain tumor MRI reports? Overall, what is your perception of patient satisfaction with brain tumor MRI reports? What is your overall level of trust in the information contained in radiology reports for MRIs on brain tumor patients?
All
Radiologist
Nonradiologist
p value
3.9 3.4 3.2 3.4
3.7 3.2 2.9 3.1
4.0 3.7 3.4 3.7
0.0587 0.0841 0.0979 0.0078
3.8
3.5
4.1
0.0020
3.6 3.7 3.1 3.6 3.5
3.3 3.4 2.5 3.3 3.2
3.9 4.0 3.7 3.9 3.9
0.0042 0.0062 <0.001 0.0086 0.0057
4.2 4.4
4.3 4.4
4.1 4.4
0.3785 0.7733
4.3
4.6
3.9
0.0035
3.7
3.5
4.0
0.0264
3.4
3.1
3.6
0.0378
3.9
3.7
4.2
0.0345
Values are based on a 5-point Likert scale (1-Never, 2-Rarely, 3-Sometimes, 4-Often, 5-Always). Values in bold are less than 3.5 and in italic are greater than 4. p values are for comparison between radiologists and nonradiologists (p < 0.05 is significant; bold).
3
ARTICLE IN PRESS GORE ET AL
Academic Radiology, Vol &, No &&, && 2019
Figure 1. Comparison of pre- and post BT-RADS survey results. Mean Likert scores are listed with 95% confidence intervals. Asterisks (*) indicate statistical difference (p < 0.05) between preimplementation and postimplementation survey. BT-RADS, Brain Tumor Reporting and Data System.
recorded on a month-by-month basis with progressive use over time (Fig 2). Reported BT-RADS score categories during this time period was also tabulated (Fig 3). The majority of reported studies fell into category 2 (51%), or no significant change, with the remaining studies divided across the other categories. A total of 38 providers (21 radiologist and 17 nonradiologists; 28% response rate) responded to our follow-up survey. After implementation of the new reporting strategy, there was improved perception of MRI reports in multiple areas (Fig 1) including report consistency (4.3 vs. 3.4; p < 0.01), radiologist/ physician communication (4.5 vs. 3.8; p < 0.01), correlation with patient management (4.2 vs. 3.6; p < 0.01), and report ambiguity (4.2 vs. 3.2, p < 0.01). Survey respondents continued to favor a structured findings and impression section on MRI brain tumor reports (4.1 vs. 4.2; p = not significant and 4.3 vs. 4.4; p = not significant, respectively) after the implementation of BT-RADS. Overall, providers were more satisfied with the BT-RADS structured reporting system (4.3 vs. 3.7; p = 0.04) and there was a perceived increase in patient satisfaction with the new reporting (4.0 vs. 3.4; p = 0.01). 4
Results from questions directly comparing the conventional and structured reports are seen in Figure 4. All responses were positive regarding the change toward structured reporting (mean Likert scores >3). Radiologists reported more positive scores (Fig 4) compared to nonradiologists (all results statistically significant). DISCUSSION Our initial survey identified areas of potential weakness in the conventional/free dictation of brain tumor MRI reports at our institution. There was a general consensus that brain tumor reports needed improvement in report consistency and refraining from ambiguity. Some participants indicated that there was “too much variability in the language and structure of final reports” and that a “more consistent structure is always appreciated.” There was a perception that patients were only sometimes satisfied with their MRI reports. Additionally, both radiologists and nonradiologists advocated for structured reporting which was reflected in many participants comments.
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
IMPLEMENTATION OF BRAIN TUMOR STRUCTURED REPORTING
Figure 2. Percent use of the BT-RADS reporting system from October 2017 to June 2018. BT-RADS, Brain Tumor Reporting and Data System.
Differences in opinion between radiologists and nonradiologists were also present, as radiologists were more critical of their own dictations, feeling less satisfied with their final reports. A structured reporting template was implemented in a staged manner to address weaknesses of the conventional reporting system. Despite the lack of added incentives among trainees or attendings to implement the structured reporting system, by the end of the study period, greater than 80% of MRI brain tumor reports were dictated using the new
format. Based on feedback from radiology attendings, trainees, and referring clinicians, the increased use of BT-RADS over time was most likely attributed to self-implementation, awareness, and favorability among referring providers. The new reports provided a framework for discussion and became a part of the vernacular during tumor boards and other interactions with clinicians. This was facilitated by easy-to-use tools, such as charts summarizing imaging characteristics, a flow diagram to aid with decision making, and a customized
Figure 3. Number of reported BT-RADS scores from October 2017 to June 2018. BT-RADS, Brain Tumor Reporting and Data System.
5
ARTICLE IN PRESS GORE ET AL
Academic Radiology, Vol &, No &&, && 2019
Figure 4. Questions directly comparing the conventional and BT-RADS structured reporting between radiologists and nonradiologists with Likert scores ranging from much worse (1) to much better (5). All responses were statistically significant between radiologists and nonradiologists with p <0.05.
dictation template, all of which assisted with scoring posttreatment brain tumor MRIs. Following implementation of the BT-RADS template, our postsurvey findings reflected improvement in the previously mentioned weaknesses. Participants noted decreased variability and ambiguity among reports, which they felt better facilitated decision making and promoted confidence in the reports. Survey results suggested that the new format promoted education, with resident and fellow trainees suggesting that the more succinct, clear and convenient BT-RADS template made it easier to understand imaging findings and know their associated management implications. Overall, there was an increased satisfaction with the BT-RADS structured reporting among providers at our institution. These findings underscore the advantages of structured reporting when interpreting complex MRI reports in patients with posttreatment brain tumors. Not only were referring providers more satisfied with the reports, most respondents believed the BT-RADS reporting method would facilitate research. The structured template was created such that it would be easily understood and machine readable. Because the structured reporting template has impression categories that contain the structured score, reports can easily be collected for research and data mining. Additionally, there was a perceived increase in patient satisfaction with the new brain tumor reports. Despite the aforementioned benefits of implementing a structured reporting system like BT-RADS, controversy exists regarding structured reporting. Johnson et al. suggested that structured reporting can be more time consuming pi re 3jand may not increase report accuracy or completeness compared to conventional free prose dictation (18). In contrast, 6
other studies have indicated structured reporting improves communication and content in diagnostic findings, contributes to quality assurance, and promotes widespread use of scoring systems and implementation in large scale multicenter studies (19 21). Another study indicated that structured reporting did not change physician perceptions of report clarity, which may be more closely tied to experience level (22). It is important to note that a learning curve may exist when using new templates/structured reporting which may initially decrease productivity. This may be a reason why individuals may resist change and become uncertain about switching to a structured reporting system. Critiques received included frustration with difficulty finding results in the expected location in the report, concerns that patients would be confused by the reports, and dissatisfaction with reports that were perceived to be longer. Structured reporting has the potential to improve communication by limiting variability in language, style, and length of dictations (6). It can provide a well-organized reporting structure that makes results easy to read and compare to prior results (1,23). With BT-RADS, we noticed multiple benefits and favorability among our referring clinicians to adopt such a reporting system. Compared to other grading criteria for brain tumor interpretation, such as the Response Assessment in Neuro-Oncology criteria, BT-RADS is relatively simplified while still accounting for most imaging findings routinely needed to guide patient care in brain tumor patients. The authors recognize that there are limitations to our study. The pre- and post-BT-RADS surveys demonstrated response rates of 40% and 28%. While low, other studies have demonstrated that survey participation rates are significantly falling over a number of fields for reasons including a
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
IMPLEMENTATION OF BRAIN TUMOR STRUCTURED REPORTING
general decrease in volunteerism, time requirements for survey completion, survey fatigue, and increasing life complexity among participants (24,25). However, a cohort study looking at relative risk estimates from studies of the same population with contrasting response rates of 18% vs. 60% were extremely similar, suggesting that studies with relatively low response rates remain valid (26). Another significant limitation is implementation at a single institution, which may limit generalizability to institutions with different patient and physician populations. Practice patterns vary by institution, and some variation is to be expected when implementing such a strategy on a wider scale. Additionally, reported values for criteria such as consistency and ambiguity were not directly measured and were based upon physician perception of these factors. Further work will be needed to determine if the structured reporting strategy quantitatively improves these measures. The next steps for this work will include determining repeat and inter-rater reliability for this reporting strategy and correlating results to patient outcomes. In conclusion, we introduced a structured system for reporting brain MRIs in brain tumor patients which was widely implemented by neuroradiologists in an academic department. This change was well-received by both radiologists and nonradiologists taking care of brain tumor patients and suggests that this reporting method is a significant way of increasing radiologist value to physicians and patients alike. Further study will enable more quantitative understanding of exactly how this system can further contribute to patient care.
8.
9.
10. 11.
12. 13.
14.
15.
16.
17. 18.
19.
20. 21.
22.
REFERENCES 1. Bosmans JML, Weyler JJ, Parizel PM. Structure and content of radiology reports, a quantitative and qualitative study in eight medical centers. Eur J Radiol 2009; 72(2):354–358. 2. Heikkinen K, Loyttyniemi M, Kormano M. Structure and content of 400 CT reports in four teaching hospitals using a new, Windows-based software tool. Acta Radiol 2000; 41(1):102–105. 3. Naik SS, Hanbidge A, Wilson SR. Radiology reports: examining radiologist and clinician preferences regarding style and content. AJR Am J Roentgenol 2001; 176(3):591–598. 4. Wallis A, McCoubrie P. The radiology report—are we getting the message across? Clin Radiol 2011; 66(11):1015–1022. 5. Schwartz LH, Panicek DM, Berk AR, et al. Improving communication of diagnostic radiology findings through structured reporting. Radiology 2011; 260(1):174–181. 6. Ganeshan D, Duong P-AT, Probyn L, et al. Structured reporting in radiology. Acad Radiol 2018; 25(1):66–73. 7. Cramer J, Eisenmenger LB, Pierson NS, et al. Structured and templated reporting: an overview. Appl Radiol 2014; 43(8):18–21.
23. 24. 25. 26.
Aiken AH, Farley A, Baugnon KL, et al. Implementation of a novel surveillance template for head and neck cancer: Neck Imaging Reporting and Data System (NI-RADS). J Am Coll Radiol 2016; 13(6):743–746. e1. Lazarus E, Mainiero MB, Schepps B, et al. BI-RADS Lexicon for US and mammography: interobserver variability and positive predictive value. Radiology 2006; 239(2):385–391. Quant EC, Wen PY. Response assessment in neuro-oncology. Curr Oncol Rep 2011; 13(1):50–56. van den Bent MJ, Afra D, de Witte O, et al. Long-term efficacy of early versus delayed radiotherapy for low-grade astrocytoma and oligodendroglioma in adults: the EORTC 22845 randomised trial. Lancet 2005; 366(9490):985–990. Huang RY, Wen PY. Response assessment in neuro-oncology criteria and clinical endpoints. Magn Reson Imaging Clin N Am 2016; 24(4):705–718. Macdonald DR, Cascino TL, Schold SC Jr, et al. Response criteria for phase II studies of supratentorial malignant glioma. J Clin Oncol 1990; 8(7): 1277–1280. Wen PY, Macdonald DR, Reardon DA, et al. Updated response assessment criteria for high-grade gliomas: response assessment in neurooncology working group. J Clin Oncol 2010; 28(11):1963–1972. James K, Eisenhauer E, Christian M, et al. Measuring response in solid tumors: unidimensional versus bidimensional measurement. JNCI 1999; 91(6):523–528. Weinberg BD, Gore A, Shu H-KG, et al. Management-based structured reporting of posttreatment Glioma response with the Brain Tumor Reporting and Data System. J Am Coll Radiol 2018; 15(5):767–771. Sullivan GM, Artino AR. Analyzing and interpreting data from Likert-type scales. J Grad Med Educ 2013; 5(4):541–542. Johnson AJ, Chen MYM, Swan JS, et al. Cohort study of structured reporting compared with conventional dictation. Radiology 2009; 253 (1):74–80. Schwartz LH, Panicek DM, Berk AR, et al. Improving communication of diagnostic radiology findings through structured reporting. Radiology 2011; 260(1):174–181. Rothke M, Blondin D, Schlemmer HP, et al. PI-RADS classification: structured reporting for MRI of the prostate. Rofo 2013; 185(3):253–261. Larson DB, Towbin AJ, Pryor RM, et al. Improving consistency in radiology reporting through the use of Department-wide Standardized Structured Reporting. Radiology 2013; 267(1):240–250. Johnson AJ, Chen MYM, Zapadka ME, et al. Radiology report clarity: a cohort study of structured reporting compared with conventional dictation. J Am Coll Radiol 2010; 7(7):501–506. Weiss DL, Langlotz CP. Structured reporting: patient care enhancement or productivity nightmare? Radiology 2008; 249(3):739–747. Morton SMB, Bandara DK, Robinson EM, et al. In the 21st century, what is an acceptable response rate? Aust N Z J Pub Health 2012; 36(2):106–108. Nulty DD. The adequacy of response rates to online and paper surveys: what can be done? Assess Eval High Educ 2008; 33(3):301–314. Curtin R, Presser S, Singer E. The effects of response rate changes on the Index of Consumer Sentiment*. Pub Opin Quarterly 2000; 64(4):413–428.
SUPPLEMENTARY MATERIAL Supplementary material associated with this article can be found in the online version at doi:10.1016/j.acra.2018.12.023.
7
Original Investigation
Institutional Implementation of a Structured Reporting System: Our Experience with the Brain Tumor Reporting and Data System Ashwani Gore, MD, Michael J. Hoch, MD, Hui-Kuo G. Shu, MD, PhD, Jeffrey J. Olson, MD, Alfredo D. Voloschin, MD, Brent D. Weinberg, MD, PhD
Abbreviations BT-RADS Brain Tumor Reporting and Data System FLAIR fluid attenuation inversion recovery MRI magnetic resonance imaging RANO Response Assessment in Neuro-Oncology WHO World Health Organization
Rationale and Objectives: Analyze the impact of implementing a structured reporting system for primary brain tumors, the Brain Tumor Reporting and Data System, on attitudes toward radiology reports at a single institution. Materials and Methods: Following Institutional Review Board approval, an initial 22 question, 5 point (1—worst to 5—best), survey was sent to faculty members, house staff members, and nonphysician providers at our institution who participate in the direct care of brain tumor patients. Results were used to develop a structured reporting strategy for brain tumors which was implemented across an entire neuroradiology section in a staged approach. Nine months following structured reporting implementation, a follow-up 27 question survey was sent to the same group of providers. Keyword search of radiology reports was used to assess usage of Brain Tumor Reporting and Data System over time. Results: Fifty-three brain tumor care providers responded to the initial survey and 38 to the follow-up survey. After implementing BT-RADS, respondents reported improved attitudes across multiple areas including: report consistency (4.3 vs. 3.4; p < 0.001), report ambiguity (4.2 vs. 3.2, p < 0.001), radiologist/physician communication (4.5 vs. 3.8; p < 0.001), facilitation of patient management (4.2 vs. 3.6; p = 0.003), and confidence in reports (4.3 vs. 3.5; p < 0.001). Providers were more satisfied with the BTRADS structured reporting system (4.3 vs. 3.7; p = 0.04). Use of the reporting template progressively increased with 81% of brain tumor reports dictated using the new template by 9 months. Conclusion: Implementing a structured template for brain tumor imaging improves perception of radiology reports among radiologists and referring providers involved in the care of brain tumor patients. Key Words: Structured template; Patient care; Response criteria; Brain tumor; Glioblastoma. © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.
INTRODUCTION
T
he radiology report is an invaluable source of information to the ordering clinician. Clear and accurate communication of imaging results plays a crucial role in
Acad Radiol 2019; &:1 7 From the Department of Radiology and Imaging Sciences, Emory University Hospital, 1364 Clifton Rd. NE, Atlanta, GA (A.G., M.J.H., B.D.W.); Department of Radiation Oncology, Emory University Hospital, Atlanta, Georgia (H.-K.G.S.); Department of Neurosurgery, Emory University Hospital, Atlanta, Georgia (J.J.O.); Department of Hematology and Medical Oncology, Emory University Hospital, Atlanta, Georgia (A.D.V.). Received December 10, 2018; revised December 17, 2018; accepted December 18, 2018. Financial Disclosure: None. Address correspondence to: B.D.W. e-mails: [email protected], [email protected] © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.acra.2018.12.023
guiding appropriate patient management. Traditionally, reports are created in a narrative style, free-prose format that can sometimes be disorganized, incomplete, or ambiguous, particularly considering interobserver variability and stylistic differences amongst radiologists (1 5). Currently, more attention is being made to the report end-product with a practice shift toward structured reporting (6). Structured reporting focuses on uniformity and consistency by implementing template macros with subheadings, standardized language, and prepopulated pick lists (7). Advantages of this format include improved clarity and efficiency, reduced diagnostic errors, and facilitation of research (6). Disease specific structured reporting as seen with Breast Imaging Reporting and Data System ( and Neck Imaging Reporting and Data System are widely accepted to examples of value-added radiology (8,9). Such reports are appreciated by referring
1
ARTICLE IN PRESS GORE ET AL
clinicians and assist with more confident patient treatment. The development of a structured reporting system with associated clinical decision support for primary brain tumor patients is an opportunity to significantly impact patient-centered care. The poor prognosis of patients with malignant brain tumors underscores the importance of clear and accurate radiology reporting in order to maximize treatment outcomes. Assessment of post-treatment brain tumor response is challenging for the practicing radiologist, as overlap between the appearance of tumor progression and treatment effects can cause ambiguous interpretations which are difficult to translate into management decisions (10,11). Although a variety of response criteria have evolved to assess brain tumor response such as the Levin, World Health Organization, MacDonald, and the Response Assessment in Neuro-Oncology criteria, they are more often utilized in clinical trials and not routinely incorporated in the radiology reports of clinical practice (10,12 14). Factors limiting their routine clinical use include complex criteria, need for multiple measurements, high interobserver variability, and limited understanding among practicing physicians (15). Thus, there is an unmet clinical need for a practical disease-specific structured reporting system for magnetic resonance imaging (MRI) surveillance of post-treatment glioma patients. The Brain Tumor Reporting and Data System (BT-RADS) was designed and implemented to establish a standardized surveillance algorithm for interpreting MRI exams of treatment response in brain tumor patients (16). It was created by a multidisciplinary team of physicians with expertise in neuro-oncology and treatment of patients with brain tumors, in particular gliomas. The BT-RADS structured template includes a managementbased numeric assessment (score 0 4) of MRI findings associated with simple management recommendations that serve to help guide clinical treatment. The objective was to create more concise and understandable reports and allow radiologists to fully commit to a clearly defined interpretation. We have implemented BT-RADS at our academic institution where we are a part of a busy neurooncology service. The goal of our study is to assess the initial usage and impact of BT-RADS at our institution following its implementation.
MATERIALS AND METHODS Initial Attitude Survey
Following Institutional Review Board approval, a 22 question survey was developed to assess attitudes toward brain tumor radiology reports. The survey was based on a five-point Likerttype scale with five representing the most positive response and one representing the least positive response. Nonradiology survey recipients included participants in multidisciplinary brain tumor board from neuro-oncology, neurosurgery, neuropathology, and radiation oncology. Neuroradiology faculty and fellows and all radiology residents were included. The survey was distributed electronically to a total of 136 individuals. 2
Academic Radiology, Vol &, No &&, && 2019
Structured Template Design and Implementation
Considering weaknesses in the conventional reporting strategy based on the initial survey results, a structured reporting template with standardized management suggestions was designed and implemented to better assist with post-treatment brain tumor MRI interpretation. Structured reporting materials included a table of reporting categories, imaging criteria for assigning studies to specific categories, a flowchart to assist with category implementation, and a dictation template (Powerscribe 360). All of these resources are freely accessible online at: www.btrads.com. This reporting system was created as a collaborative effort between radiology, neurosurgery, neurooncology, and radiation oncology. The ultimate goals of this approach were to facilitate more effective communication and improve overall care of brain tumor patients. The details of the BT-RADS scoring system have been reported previously (16). The BT-RADS and structured/management-based reporting template was implemented at our institution in a staged approach. In October 2017, the structured approach was implemented by two attending radiologists on a trial basis. After making iterative changes to the reporting system, the structured reporting method was incorporated in site-wide templates for use by all faculty 4 months after initial introduction in March 2018. A sample report using BT-RADS is illustrated in Supplemental Material A. Template Usage
Keyword search of radiology reports (Philips PerformanceBridge Report Search Tool) was used to assess usage of BTRADS from October 1, 2017 to June 30, 2018. The total numbers of brain tumor MRI reports were queried using MR as the modality with search queries of “brain tumor” or “glioblastoma” or “GBM” or “astrocytoma” or “oligodendroglioma” or “glioma.” The number of structured reports was determined by searching for structured codes in the report (“bt-0” OR “bt-1a” OR “bt-1b” OR “bt-2” OR “bt-3a” OR “bt-3b” OR “bt-3c” OR “bt-4”). The output represented the total number of brain MRI reports that used the BT-RADS structured reporting system. These outputs were separated based on month of usage from October 2017 to June 2018 (9 months) and the percent usage over time was calculated. Additionally, individually reported BT-RADS categories were tabulated during this 9-month period. Follow-up Attitude Survey
A 27-question survey was designed to assess evolving user attitudes toward the BT-RADS structured reporting system. This survey contained the same 22 questions on the original study with the addition of five new questions asking participants to directly compare the prior and new reporting system with regards to uniformity and consistency of reports, ambiguity in interpretations, promoting research, overall satisfaction, and a
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
IMPLEMENTATION OF BRAIN TUMOR STRUCTURED REPORTING
survey. Although participants felt the conventional reporting system communicated relevant findings (mean Likert score of 3.9), lower scores were seen in maintaining consistency across reports (3.4), refraining from ambiguity (3.2), and promoting clear communication between patient and physician (3.4). A general perception was that patients were only sometimes satisfied with their MRI reports (3.4) (Fig 1). Compared to nonradiologists, radiologists felt their reports did not always promote clear communication to ordering physicians (radiologists 3.5 vs. nonradiologists 4.1, p < 0.01). Radiologists felt their conventional reports frequently did not facilitate decision-making for treatment (3.3 vs. 3.9, p < 0.01) and were less conducive to trainee education (3.3 vs. 3.9, p = 0.01). Overall radiologists were not as satisfied compared to nonradiologists with primary brain tumor MRI reports (3.5 vs. 4.0, p = 0.03). Nonetheless, both groups strongly favored findings (4.3 vs. 4.1, p = 0.38) and impression sections (4.4 vs. 4.4, p = 0.77) adhering to a structured format. Following implementation of the structured reporting template, the institutional report search tool yielded a total of 1078 brain tumor MRI dictations from October 2017 to June 2018. Out of these, 536 reports were dictated using the BT-RADS structured template. Percent use of the BT-RADS template was
free response/comment section (Supplemental Material B). On questions asking respondents to directly compare reports before and after template implementation, a five-point scale was again used where 1 indicated much worse, 3 indicated the same, and 5 indicated much better. Statistical Analysis
Results of the post BT-RADS implementation survey was then compared to the initial, preimplementation, free dictation reporting system. Mean differences between survey categories were calculated using a two-tailed t test with equal variances (17). Corrected p values using Bonferroni correction with p < 0.05 was deemed statistically significant. RESULTS The initial survey was divided into four main categories which included the following: details of the existing conventional reporting system, overall satisfaction with existing brain tumor reports, changes to the reporting system, and a free responses/suggestions section (Table 1). A total of 53 providers (27 radiologists and 26 nonradiologists; 40% response rate) responded to the initial
TABLE 1. Opinions Regarding Conventional Brain Tumor Surveillance MRI Reports
How well do the current radiology reports Clearly communicate the relevant findings Maintain consistency Refrain from ambiguity Promote clear communication between patient and physician Promote clear communication between radiologist and ordering physician Facilitate decision-making for treatment Match your practice patterns Facilitate research Facilitate trainee (resident and fellow) education Promote confidence in report findings How strongly do you agree with the following statements Finding sections should adhere to a structured format Impression sections in radiology reports should adhere to a structured format It would be helpful if report impressions provided categories which corresponded to suggested management decisions Overall satisfaction Overall, how satisfied are you with the primary brain tumor MRI reports? Overall, what is your perception of patient satisfaction with brain tumor MRI reports? What is your overall level of trust in the information contained in radiology reports for MRIs on brain tumor patients?
All
Radiologist
Nonradiologist
p value
3.9 3.4 3.2 3.4
3.7 3.2 2.9 3.1
4.0 3.7 3.4 3.7
0.0587 0.0841 0.0979 0.0078
3.8
3.5
4.1
0.0020
3.6 3.7 3.1 3.6 3.5
3.3 3.4 2.5 3.3 3.2
3.9 4.0 3.7 3.9 3.9
0.0042 0.0062 <0.001 0.0086 0.0057
4.2 4.4
4.3 4.4
4.1 4.4
0.3785 0.7733
4.3
4.6
3.9
0.0035
3.7
3.5
4.0
0.0264
3.4
3.1
3.6
0.0378
3.9
3.7
4.2
0.0345
Values are based on a 5-point Likert scale (1-Never, 2-Rarely, 3-Sometimes, 4-Often, 5-Always). Values in bold are less than 3.5 and in italic are greater than 4. p values are for comparison between radiologists and nonradiologists (p < 0.05 is significant; bold).
3
ARTICLE IN PRESS GORE ET AL
Academic Radiology, Vol &, No &&, && 2019
Figure 1. Comparison of pre- and post BT-RADS survey results. Mean Likert scores are listed with 95% confidence intervals. Asterisks (*) indicate statistical difference (p < 0.05) between preimplementation and postimplementation survey. BT-RADS, Brain Tumor Reporting and Data System.
recorded on a month-by-month basis with progressive use over time (Fig 2). Reported BT-RADS score categories during this time period was also tabulated (Fig 3). The majority of reported studies fell into category 2 (51%), or no significant change, with the remaining studies divided across the other categories. A total of 38 providers (21 radiologist and 17 nonradiologists; 28% response rate) responded to our follow-up survey. After implementation of the new reporting strategy, there was improved perception of MRI reports in multiple areas (Fig 1) including report consistency (4.3 vs. 3.4; p < 0.01), radiologist/ physician communication (4.5 vs. 3.8; p < 0.01), correlation with patient management (4.2 vs. 3.6; p < 0.01), and report ambiguity (4.2 vs. 3.2, p < 0.01). Survey respondents continued to favor a structured findings and impression section on MRI brain tumor reports (4.1 vs. 4.2; p = not significant and 4.3 vs. 4.4; p = not significant, respectively) after the implementation of BT-RADS. Overall, providers were more satisfied with the BT-RADS structured reporting system (4.3 vs. 3.7; p = 0.04) and there was a perceived increase in patient satisfaction with the new reporting (4.0 vs. 3.4; p = 0.01). 4
Results from questions directly comparing the conventional and structured reports are seen in Figure 4. All responses were positive regarding the change toward structured reporting (mean Likert scores >3). Radiologists reported more positive scores (Fig 4) compared to nonradiologists (all results statistically significant). DISCUSSION Our initial survey identified areas of potential weakness in the conventional/free dictation of brain tumor MRI reports at our institution. There was a general consensus that brain tumor reports needed improvement in report consistency and refraining from ambiguity. Some participants indicated that there was “too much variability in the language and structure of final reports” and that a “more consistent structure is always appreciated.” There was a perception that patients were only sometimes satisfied with their MRI reports. Additionally, both radiologists and nonradiologists advocated for structured reporting which was reflected in many participants comments.
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
IMPLEMENTATION OF BRAIN TUMOR STRUCTURED REPORTING
Figure 2. Percent use of the BT-RADS reporting system from October 2017 to June 2018. BT-RADS, Brain Tumor Reporting and Data System.
Differences in opinion between radiologists and nonradiologists were also present, as radiologists were more critical of their own dictations, feeling less satisfied with their final reports. A structured reporting template was implemented in a staged manner to address weaknesses of the conventional reporting system. Despite the lack of added incentives among trainees or attendings to implement the structured reporting system, by the end of the study period, greater than 80% of MRI brain tumor reports were dictated using the new
format. Based on feedback from radiology attendings, trainees, and referring clinicians, the increased use of BT-RADS over time was most likely attributed to self-implementation, awareness, and favorability among referring providers. The new reports provided a framework for discussion and became a part of the vernacular during tumor boards and other interactions with clinicians. This was facilitated by easy-to-use tools, such as charts summarizing imaging characteristics, a flow diagram to aid with decision making, and a customized
Figure 3. Number of reported BT-RADS scores from October 2017 to June 2018. BT-RADS, Brain Tumor Reporting and Data System.
5
ARTICLE IN PRESS GORE ET AL
Academic Radiology, Vol &, No &&, && 2019
Figure 4. Questions directly comparing the conventional and BT-RADS structured reporting between radiologists and nonradiologists with Likert scores ranging from much worse (1) to much better (5). All responses were statistically significant between radiologists and nonradiologists with p <0.05.
dictation template, all of which assisted with scoring posttreatment brain tumor MRIs. Following implementation of the BT-RADS template, our postsurvey findings reflected improvement in the previously mentioned weaknesses. Participants noted decreased variability and ambiguity among reports, which they felt better facilitated decision making and promoted confidence in the reports. Survey results suggested that the new format promoted education, with resident and fellow trainees suggesting that the more succinct, clear and convenient BT-RADS template made it easier to understand imaging findings and know their associated management implications. Overall, there was an increased satisfaction with the BT-RADS structured reporting among providers at our institution. These findings underscore the advantages of structured reporting when interpreting complex MRI reports in patients with posttreatment brain tumors. Not only were referring providers more satisfied with the reports, most respondents believed the BT-RADS reporting method would facilitate research. The structured template was created such that it would be easily understood and machine readable. Because the structured reporting template has impression categories that contain the structured score, reports can easily be collected for research and data mining. Additionally, there was a perceived increase in patient satisfaction with the new brain tumor reports. Despite the aforementioned benefits of implementing a structured reporting system like BT-RADS, controversy exists regarding structured reporting. Johnson et al. suggested that structured reporting can be more time consuming pi re 3jand may not increase report accuracy or completeness compared to conventional free prose dictation (18). In contrast, 6
other studies have indicated structured reporting improves communication and content in diagnostic findings, contributes to quality assurance, and promotes widespread use of scoring systems and implementation in large scale multicenter studies (19 21). Another study indicated that structured reporting did not change physician perceptions of report clarity, which may be more closely tied to experience level (22). It is important to note that a learning curve may exist when using new templates/structured reporting which may initially decrease productivity. This may be a reason why individuals may resist change and become uncertain about switching to a structured reporting system. Critiques received included frustration with difficulty finding results in the expected location in the report, concerns that patients would be confused by the reports, and dissatisfaction with reports that were perceived to be longer. Structured reporting has the potential to improve communication by limiting variability in language, style, and length of dictations (6). It can provide a well-organized reporting structure that makes results easy to read and compare to prior results (1,23). With BT-RADS, we noticed multiple benefits and favorability among our referring clinicians to adopt such a reporting system. Compared to other grading criteria for brain tumor interpretation, such as the Response Assessment in Neuro-Oncology criteria, BT-RADS is relatively simplified while still accounting for most imaging findings routinely needed to guide patient care in brain tumor patients. The authors recognize that there are limitations to our study. The pre- and post-BT-RADS surveys demonstrated response rates of 40% and 28%. While low, other studies have demonstrated that survey participation rates are significantly falling over a number of fields for reasons including a
ARTICLE IN PRESS Academic Radiology, Vol &, No &&, && 2019
IMPLEMENTATION OF BRAIN TUMOR STRUCTURED REPORTING
general decrease in volunteerism, time requirements for survey completion, survey fatigue, and increasing life complexity among participants (24,25). However, a cohort study looking at relative risk estimates from studies of the same population with contrasting response rates of 18% vs. 60% were extremely similar, suggesting that studies with relatively low response rates remain valid (26). Another significant limitation is implementation at a single institution, which may limit generalizability to institutions with different patient and physician populations. Practice patterns vary by institution, and some variation is to be expected when implementing such a strategy on a wider scale. Additionally, reported values for criteria such as consistency and ambiguity were not directly measured and were based upon physician perception of these factors. Further work will be needed to determine if the structured reporting strategy quantitatively improves these measures. The next steps for this work will include determining repeat and inter-rater reliability for this reporting strategy and correlating results to patient outcomes. In conclusion, we introduced a structured system for reporting brain MRIs in brain tumor patients which was widely implemented by neuroradiologists in an academic department. This change was well-received by both radiologists and nonradiologists taking care of brain tumor patients and suggests that this reporting method is a significant way of increasing radiologist value to physicians and patients alike. Further study will enable more quantitative understanding of exactly how this system can further contribute to patient care.
8.
9.
10. 11.
12. 13.
14.
15.
16.
17. 18.
19.
20. 21.
22.
REFERENCES 1. Bosmans JML, Weyler JJ, Parizel PM. Structure and content of radiology reports, a quantitative and qualitative study in eight medical centers. Eur J Radiol 2009; 72(2):354–358. 2. Heikkinen K, Loyttyniemi M, Kormano M. Structure and content of 400 CT reports in four teaching hospitals using a new, Windows-based software tool. Acta Radiol 2000; 41(1):102–105. 3. Naik SS, Hanbidge A, Wilson SR. Radiology reports: examining radiologist and clinician preferences regarding style and content. AJR Am J Roentgenol 2001; 176(3):591–598. 4. Wallis A, McCoubrie P. The radiology report—are we getting the message across? Clin Radiol 2011; 66(11):1015–1022. 5. Schwartz LH, Panicek DM, Berk AR, et al. Improving communication of diagnostic radiology findings through structured reporting. Radiology 2011; 260(1):174–181. 6. Ganeshan D, Duong P-AT, Probyn L, et al. Structured reporting in radiology. Acad Radiol 2018; 25(1):66–73. 7. Cramer J, Eisenmenger LB, Pierson NS, et al. Structured and templated reporting: an overview. Appl Radiol 2014; 43(8):18–21.
23. 24. 25. 26.
Aiken AH, Farley A, Baugnon KL, et al. Implementation of a novel surveillance template for head and neck cancer: Neck Imaging Reporting and Data System (NI-RADS). J Am Coll Radiol 2016; 13(6):743–746. e1. Lazarus E, Mainiero MB, Schepps B, et al. BI-RADS Lexicon for US and mammography: interobserver variability and positive predictive value. Radiology 2006; 239(2):385–391. Quant EC, Wen PY. Response assessment in neuro-oncology. Curr Oncol Rep 2011; 13(1):50–56. van den Bent MJ, Afra D, de Witte O, et al. Long-term efficacy of early versus delayed radiotherapy for low-grade astrocytoma and oligodendroglioma in adults: the EORTC 22845 randomised trial. Lancet 2005; 366(9490):985–990. Huang RY, Wen PY. Response assessment in neuro-oncology criteria and clinical endpoints. Magn Reson Imaging Clin N Am 2016; 24(4):705–718. Macdonald DR, Cascino TL, Schold SC Jr, et al. Response criteria for phase II studies of supratentorial malignant glioma. J Clin Oncol 1990; 8(7): 1277–1280. Wen PY, Macdonald DR, Reardon DA, et al. Updated response assessment criteria for high-grade gliomas: response assessment in neurooncology working group. J Clin Oncol 2010; 28(11):1963–1972. James K, Eisenhauer E, Christian M, et al. Measuring response in solid tumors: unidimensional versus bidimensional measurement. JNCI 1999; 91(6):523–528. Weinberg BD, Gore A, Shu H-KG, et al. Management-based structured reporting of posttreatment Glioma response with the Brain Tumor Reporting and Data System. J Am Coll Radiol 2018; 15(5):767–771. Sullivan GM, Artino AR. Analyzing and interpreting data from Likert-type scales. J Grad Med Educ 2013; 5(4):541–542. Johnson AJ, Chen MYM, Swan JS, et al. Cohort study of structured reporting compared with conventional dictation. Radiology 2009; 253 (1):74–80. Schwartz LH, Panicek DM, Berk AR, et al. Improving communication of diagnostic radiology findings through structured reporting. Radiology 2011; 260(1):174–181. Rothke M, Blondin D, Schlemmer HP, et al. PI-RADS classification: structured reporting for MRI of the prostate. Rofo 2013; 185(3):253–261. Larson DB, Towbin AJ, Pryor RM, et al. Improving consistency in radiology reporting through the use of Department-wide Standardized Structured Reporting. Radiology 2013; 267(1):240–250. Johnson AJ, Chen MYM, Zapadka ME, et al. Radiology report clarity: a cohort study of structured reporting compared with conventional dictation. J Am Coll Radiol 2010; 7(7):501–506. Weiss DL, Langlotz CP. Structured reporting: patient care enhancement or productivity nightmare? Radiology 2008; 249(3):739–747. Morton SMB, Bandara DK, Robinson EM, et al. In the 21st century, what is an acceptable response rate? Aust N Z J Pub Health 2012; 36(2):106–108. Nulty DD. The adequacy of response rates to online and paper surveys: what can be done? Assess Eval High Educ 2008; 33(3):301–314. Curtin R, Presser S, Singer E. The effects of response rate changes on the Index of Consumer Sentiment*. Pub Opin Quarterly 2000; 64(4):413–428.
SUPPLEMENTARY MATERIAL Supplementary material associated with this article can be found in the online version at doi:10.1016/j.acra.2018.12.023.
7