- Email: [email protected]
Detailed prospective peer review in a community radiation oncology clinic
Practical Radiation Oncology (2017) 7, 50-56
www.practicalradonc.org
Basic Original Report
Detailed prospective peer review in a community radiation oncology clinic James D. Mitchell MD a,⁎, Thomas J. Chesnut MS b , David V. Eastham MD a , Carlo N. Demandante MD a , David J. Hoopes MD c a
Joint Radiation Oncology Center, David Grant Medical Center, Travis Air Force Base, California U.S. Census Bureau, Washington, DC c Department of Radiation Oncology, University of California San Diego, San Diego, California b
Received 27 April 2016; revised 19 July 2016; accepted 23 August 2016
Abstract Purpose: In 2012, we instituted detailed prospective peer review of new cases. We present the outcomes of peer review on patient management and time required for peer review. Methods and materials: Peer review rounds were held 3 to 4 days weekly and required 2 physicians to review pertinent information from the electronic medical record and treatment planning system. Eight aspects were reviewed for each case: 1) workup and staging; 2) treatment intent and prescription; 3) position, immobilization, and simulation; 4) motion assessment and management; 5) target contours; 6) normal tissue contours; 7) target dosimetry; and 8) normal tissue dosimetry. Cases were marked as, “Meets standard of care,” “Variation,” or “Major deviation.” Changes in treatment plan were noted. As our process evolved, we recorded the time spent reviewing each case. Results: From 2012 to 2014, we collected peer review data on 442 of 465 (95%) radiation therapy patients treated in our hospital-based clinic. Overall, 91 (20.6%) of the cases were marked as having a variation, and 3 (0.7%) as major deviation. Forty-two (9.5%) of the cases were altered after peer review. An overall peer review score of “Variation” or “Major deviation” was highly associated with a change in treatment plan (P b .01). Changes in target contours were recommended in 10% of cases. Gastrointestinal cases were significantly associated with a change in treatment plan after peer review. Indicators on position, immobilization, simulation, target contours, target dosimetry, motion management, normal tissue contours, and normal tissue dosimetry were significantly associated with a change in treatment plan. The mean time spent on each case was 7 minutes. Conclusions: Prospective peer review is feasible in a community radiation oncology practice. Our process led to changes in 9.5% of cases. Peer review should focus on technical factors such as target contours and dosimetry. Peer review required 7 minutes per case. Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
Presented at the 2013 ASTRO Annual Meeting, September 22-25, 2013, Atlanta, Georgia. Conflicts of interest: None. The views expressed in this material are those of the authors and do not reflect the official policy or position of the U.S. Government, the Department of Defense, or the Department of the Air Force. The work reported herein was performed under United States Air Force Surgeon General approved Clinical Investigation Number FDG20130018E. ⁎ Corresponding author. Joint Radiation Oncology Center, David Grant USAF Medical Center, 101 Bodin Circle, Travis Air Force Base, CA 94535. E-mail address: [email protected] (J.D. Mitchell). http://dx.doi.org/10.1016/j.prro.2016.08.011 1879-8500/Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
Practical Radiation Oncology: January-February 2017
Peer Review, Early Patient: DATE: Treating MD: --Review MD: ---
Peer Review in a Community Clinic
DOB: Site CNS/Peds H&N Lung/Sarc GI Breast GYN GU Heme Skin Palliative
1. Workup and Staging Comments:
2. Treatment Intent and Prescription
MRN: No Change
3. Position, immobilization, simulation
Variation
Meets Standard of Care
Major Deviation
Does NOT Meet Standard of Care I would manage this case differently. The current management plan is not reasonable. I recommend changes be made. I would:
I would not change the management in this case.
I would manage this case differently but the current management is reasonable. I would:
No Change
Do additional workup Do less workup
Do additional workup
Not Rx RT ↑Fraction Size ↓Fraction Size ↑Total Dose ↓Total Dose Δ RT Schedule Δ RT Modality
Not Rx RT ↑Fraction Size ↓Fraction Size ↑Total Dose ↓Total Dose Δ RT Schedule Δ RT Modality
Comments:
Comments:
Comments:
Comments:
↑Target Contour size ↓Target Contour size Include other targets Use Fusion Modality
↑Target Contour size ↓Target Contour size Include other targets Use Fusion Modality
↑Avoid Contour size ↓Avoid Contour size Include other Avoids Use Fusion Modality
↑Avoid Contour size ↓Avoid Contour size Include other Avoids Use Fusion Modality
↑Target Coverage Fill target cold spot ↓ Plan Max Dose
↑Target Coverage Fill target cold spot ↓ Plan Max Dose
↓ Normal Tissue Dose
↓ Normal Tissue Dose
Brain Brainstem Optics Cochlea Cord Parotid Larynx B. Plexus
Brain Brainstem Optics Cochlea Cord Parotid Larynx B. Plexus
No Change
Comments:
No Change
4. Motion assessment and management 5. Target Contours
No Change
No Change
Comments:
6. Normal Tissue (Avoidance) Contours Comments:
7. Target Dosimetry/HotSpot (DVH&Isodose) Comments:
8. Normal Tissue Dosimetry (DVH &Isodose)
No Change
No Change
Comments:
No Change
Overall Peer Review
No Change
Prospective Peer Review changed Tx plan? ---
Lung Heart Esophagus Liver Small Bowel Kidney Bladder Rectum
Variation
How?
Additional Comments:
Figure 1
51
Peer review document.
Lung Heart Esophagus Liver SmallBowel Kidney Bladder Rectum
Major Deviation
52
J.D. Mitchell et al
Introduction The American Society for Radiation Oncology (ASTRO) has identified peer review as a critical safety and quality improvement tool for radiation oncology departments. In 2013, Marks et al reported the ASTRO Executive Summary on case-oriented peer review, highlighting important safety measures that should be included in departmental policies and procedures. 1 In an ASTRO survey, 83% of radiation oncologists reported participating in a peer review program, and 65% reported reviewing cases before starting treatment. 2 Despite near-universal adoption in US academic and community radiation oncology departments, there are few prospective reports on the outcomes of peer review programs. Early reports came from Canada, Australia, and Singapore, with few data from centers within the United States. 3-6 Recently, the MD Anderson Cancer Center reported on a peer review program for their outpatient radiation therapy centers 7; in the accompanying editorial, Chen argues for more research into the costs and efficacy of peer review. 8 Large gaps remain in our understanding of the efficacy, yield, and costs of peer review. Specifically, there is no conclusive data showing which types of cases require peer review, which cases can safely forego peer review, and what are the critical aspects to review for each case. Additionally, there is little information on the time required for quality peer review. 9 In an attempt to address these knowledge gaps, our department standardized our peer review process and documentation, systematically collected data on prospective peer review of new cases, and recorded those changes. We later began collecting data on the amount of time spent on each case. In this report, we present data on the outcomes of peer review on clinical practice and decision-making as well as time required for review.
Methods and materials Beginning in 2012, the Joint Radiation Oncology Center at Travis Air Force Base optimized its departmental continuous quality improvement (CQI) program. This overhaul was performed as preparation for American College of Radiology accreditation and in response to internal safety concerns. A major part of this program included standardized, prospective peer review of new radiation therapy cases. Peer review was performed 3 to 4 times per week during our morning departmental “huddle,” in a setting where the electronic medical record and treatment planning system could be viewed simultaneously. At a minimum, 2 physicians, 1 physicist, 1 dosimetrist, and 1 radiation therapist were present. The treating physicians were all general radiation oncologists treating all sites of disease, and new cases were assigned to each physician at random. The reviewing physicians were also chosen at
Practical Radiation Oncology: January-February 2017
random based on who was present during each conference. The goal of the CQI program was to perform peer review on at least 50% of cases before beginning treatment. Cases that did not undergo peer review consisted of urgent cases that required immediate initiation of therapy. Prospective peer review was documented electronically using a standardized form (Fig 1). Each case was identified based on the primary site of treatment: central nervous system/ pediatrics, head and neck, lung/sarcoma, gastrointestinal, breast, gynecologic, genitourinary, hematologic, and palliative. Each case was presented by the treating physician, and the treatment plan was evaluated in real time by the reviewing physician. Eight separate aspects were reviewed for each case: 1) workup and staging (including evaluation and management up to the date of peer review); 2) treatment intent and prescription; 3) position, immobilization, and simulation; 4) motion assessment and management; 5) target contours; 6) normal tissue (avoidance) contours; 7) target dosimetry; and 8) normal tissue dosimetry. Review of the contours, treatment plan, and dose-volume histogram was performed in real-time within the treatment planning system. Therefore, slice-by-slice review of contours and dose distributions was performed as was interactive review of the dose-volume histogram. For each of the 8 aspects of a new case, the peer reviewer marked either, “No Change: I would not change the management in this case,” “Variation: I would manage this case differently but the current management plan is reasonable,” or “Major Deviation: I would manage this case differently. The current plan is not reasonable. I recommend changes be made.” A primary outcome of this work was to record how often peer review led to a change in the clinical management of a patient. All treatment changes prompted by peer review were recorded. Beginning in 2013, we made an improvement to the peer review documentation system that allowed it to track the time taken for each individual peer review case. An institutional review board exemption was obtained to study our peer review program (United States Air Force Surgeon General approved Clinical Investigation Number FDG20130018E.) We identified all cases that underwent prospective peer review between January 2012 and April 2014. Data were collected on 10 domains: the primary site of treatment, the 8 aspects of peer review listed previously, and whether or not a change was made based on the peer review process. The data were placed in an anonymous database for analysis.
Statistical analysis Given the categorical nature of the data collected with the exception of peer review time, we used a non-model-based statistical methodology using categorical data analysis 10 to explore the patient case characteristics as they relate to the peer review process and change in treatment outcome. Defining the patient characteristics as our independent variables and “change in treatment” as our dependent
Practical Radiation Oncology: January-February 2017 Table 1
Peer Review in a Community Clinic
53
Distribution of cases and peer review yield for each physician
Physician 1 2 3 4 5
Number of cases treated
Number of cases reviewed
Number of reviewed cases changed after peer review
Relative risk of treatment change after peer review
36 127 118 119 41
25 179 91 122 24
2 26 3 11 0
0.83 (P = 1.0) 2.4 (P = .005) 0.30 (P = .04) 0.92 (P = .95) NA
NA, not available.
variable, we examine the relationship between each independent variable and the change in treatment outcome using 2-way contingency tables. Using the Pearson χ 2 statistical test of independence and the resulting P values, we identify those patient characteristics that influence the change in treatment outcomes as supported by the given data.
P = .005) and review by physician 3 was less likely to lead to a change (RR = 0.3, P = .04). Overall, 91 (20.6%) cases were marked as having a variation, and 42 (9.5%) were altered after peer review. Of the 42 cases that were altered, 40 (95%) were marked as having a variation. Only 3 cases (0.7%) were marked as having a major deviation. The distribution of recommended changes based on primary disease site is listed in Table 2. Of the different primary cancer sites, gastrointestinal cases were 3.2 times more likely to have a change in treatment compared with all other cases (P = .002). Additionally, there was a trend toward an increase in changes in treatment plans for hematologic malignancies (RR = 2.8; P = .086), and a decrease in changes for genitourinary malignancies (RR = 0.5, P = .095). As would be expected, the indicator “Variation” or “Major Deviation” was highly associated with a change in treatment plan (P b .001). Of the different sections of the peer review process, target contours were most commonly marked as “Variation” or “Major Deviation.” Cases that were marked as having a variation or major deviation in target contours were more likely to lead to a change in treatment when compared with cases that were not marked as such (RR = 6.7, P b .001). Additional domains associated with a change in treatment plan included normal tissue dosimetry (RR = 5.3, P b .001); target dosimetry (RR = 5.3, P b .001); position, immobilization, and simulation (RR = 5.0, P b .001); normal tissue contours (RR = 8.0, P b .001); and motion assessment and management (RR = 4.1, P = .001). The overall results
Results During the years 2012 through 2014, we collected peer review data on 442 of 465 patients. This represents 95% of new treatment plans in the department that underwent peer review before initiating treatment. All of these are included in this analysis. Additionally, we collected data regarding the time required to perform peer review on the most recent 121 patients. All cases consisted of external beam radiation therapy, not including stereotactic radiosurgery. Five physicians were included in the study, and the distribution of cases treated and reviewed by each physician is listed in Table 1. Only physicians 2 and 3 were present in the department for the entire duration of the study. Table 1 also shows the number of cases changed after review by each physician as well as the relative risk of a treatment change for each physician reviewer. The majority of changes came after peer review by physicians 2 and 4. Peer review by physician 2 was more likely to lead to a change in treatment plan (relative risk [RR] = 2.4,
Table 2
Change in treatment plan based on primary disease site
Disease site Central nervous system Lung and sarcoma Breast Genitourinary Palliative Head and neck Gastrointestinal Hematologic
Number of cases (% of total) 8 59 38 128 115 45 30 16
(1.8) (13) (8.6) (29) (26) (10.2) (6.8) (3.6)
Number of cases with change in treatment (%)
P value for association with change
0 6 3 7 7 7 6 4
.75 1 .95 .09 .2 .23 .003 .086
(0) (10) (7.9) (5.4) (6.1) (15.6) (27) (25)
54
J.D. Mitchell et al
Practical Radiation Oncology: January-February 2017
for all peer review domains are listed in Table 3. Of the more targeted subsections of the peer review document, the most frequently cited variations included increase target size (n = 33), decrease target size (n = 11), increase target coverage (n = 6), and add additional workup (n = 6). Of the normal tissues, the most frequently cited recommendation was to decrease the dose to small bowel (n = 5). Table 4 lists the most frequently marked variations. For the most recent 121 patients, the mean time spent on peer review was 7.05 minutes (range, 2-24 minutes). The amount of time spent per case based on primary site is shown in Fig 2. Cases where peer review led to a change in therapy took 5.1 minutes longer to review than those cases that did not lead to a change (11.8 vs 6.7 minutes, P = .1). Head and neck cases required the most time for peer review, but the differences were not statistically significant.
Discussion As a field of medicine, radiation oncology has benefited greatly from advances in technology. Through integration of advanced treatment delivery machines, brachytherapy, treatment planning systems, and electronic medical records, we are able to deliver precision treatment for complex diseases. These technologic milestones present opportunities and challenges for patient safety. It follows that departments should have internal CQI programs outlining safety related policies and procedures. Peer review is an important aspect of departmental safety culture and has been recognized as such by ASTRO. The ASTRO Executive Summary on peer review states that, “Peer review is one of the most effective means for assuring the quality of qualitative, and potentially controversial, patient-specific decisions in radiation oncology.” The authors go on to outline important aspects to include in a peer review program spanning from the initial decision to treat with radiation therapy through delivery of treatment. 1 However, despite guidance from ASTRO, and a widespread use of peer review in at least some form, we are just beginning to scientifically examine how peer Table 3
review impacts patient care and departmental operations. In an editorial, Chen outlines several areas that demand further research including identifying which patients require peer review, when should peer review occur, and the true costs. 8 This report adds to our understanding of the impact of prospective peer review through a detailed analysis of 442 nearly serial patients treated in a nonacademic, hospital-based department. A potential barrier to quality peer review may include the amount of time required. To our knowledge, we are the first to report prospective data regarding the amount of time spent on peer review for each case. Our data show a mean time of 7 minutes per case with a range of 2 to 24 minutes. In a survey of academic radiation oncology departments, Lawrence reported a mean time of 2.7 minutes spent on each case. 9 However, these data were not collected prospectively during peer review conferences, but rather through survey questions that relied on recall and estimation. Our data show that meaningful peer review requires time, and a busy radiation oncology clinic may have difficulty setting aside enough time for peer review of every case. In a survey of Canadian radiation oncologists, Hamilton reported that 40% found it moderately difficult and 22% extremely difficult to incorporate peer review into their schedules. 11 The question then arises: does every case require peer review? Our data show that gastrointestinal cases were significantly associated with treatment changes based on peer review, but there was no clear subset that we identified as not benefiting from peer review. The group from MD Anderson Cancer Center also identified gastrointestinal sites as well as head and neck and gynecologic malignancies as accounting for the majority of changes made. 7 In a report from Canada, Lefresne reported that gastrointestinal malignancies, lung cancer, and lymphoma represented the highest proportion of cases marked as needing either a minor or major change prior to treatment. 12 Although gastrointestinal cases appear to have a high rate of changes recommended through peer review, clearly we need more research to better identify cases that benefit the most.
Change in treatment plan based on peer review results of technical factors
Workup and staging Treatment intent and prescription Position, immobilization, and simulation Motion assessment and management Target contours Normal tissue contours Target dosimetry Normal tissue dosimetry
Number of cases marked as variation (% of total)
Number of cases with change in treatment (% of category)
P value for association with change
9 9 6 6 44 9 14 13
2 (22) 2 (22) 3 (50) 2 (33.3) 18 (40.9) 6 (67) 5 (35.7) 9 (69.2)
.459 .459 b.001 .001 b.001 b.001 b.001 b.001
(2.0) (2.0) (1.3) (1.3) (9.96) (2.0) (3.2) (2.9)
Practical Radiation Oncology: January-February 2017 Table 4 Most frequently marked reasons for labelling a case as “variation” Reason for variation Increase target size Decrease target size Increase target coverage Add additional workup Decrease small bowel dose Decrease plan maximum dose Decrease prescription dose Add additional avoidance structure Decrease size of avoidance structure Decrease spinal cord dose Decrease lung dose Increase prescription dose Add motion management (4-dimensional computed tomography) Do not use radiation therapy
Number of cases marked as variation 33 11 6 6 5 5 4 3 3 2 2 2 2 2
Our data show a trend toward a decreased number of genitourinary cases with changes after peer review, but this difference was not statistically significant. The vast majority of our genitourinary cases are prostate cancer, which is the most frequently treated diagnosis in our department. Because of our large volume of prostate patients, we have a standardized approach to planning
Peer Review in a Community Clinic
55
prostate radiation therapy. This standardization, in combination with our collective experience treating prostate cancer, may explain why we have fewer changes recommended at peer review. On the other hand, our gastrointestinal practice represents a wide variety of cases including cancers of the esophagus, stomach, pancreas, liver, rectum, and anus. It may be that peer review is more important for treatment that is performed less frequently, but determining this requires further investigation. Some departments may be inclined to eliminate palliative cases from peer review because of a perception of simplicity and decreased risk of harm. However, our data do not show a decreased peer review yield for palliative cases, and we recommend continuing peer review for palliative cases. Another goal of our study was to identify which aspects of a radiation therapy case are most important from a peer review standpoint. The planning steps that we found were most closely associated with a change in treatment included: 1) position, immobilization, and simulation; 2) target contours; 3) target dosimetry; 4) motion management; 5) normal tissue dosimetry; and 6) normal tissue contours. Similarly, in the Canadian series reported by Lefresne, they noted that the most common causes for plans to be marked as requiring a significant change included, “inadequate target volume coverage (36%), suboptimal dose or fractionation (27%), errors in patient setup (27%), and overtreatment of normal tissue (9%).” 12 It may therefore stand to reason that in an effort to improve efficiency, peer review should focus on these technical factors.
Figure 2 Peer review time for different disease sites. (Tukey-style box plot: the box bounds the interquartile range [IQR] divided by the median, and whiskers extend to a maximum of 1.5 × IQR beyond the box. Time is expressed in minutes.)
56
J.D. Mitchell et al
One weakness of our study is the limited number of gynecologic cases. Because of our patient population, we see very few gynecologic patients and therefore we cannot make any meaningful conclusions regarding the usefulness of, or time required for, peer review. Additionally, we did not collect data on peer review for brachytherapy or radiosurgery, two areas where peer review has been understudied and underutilized. Last, the total number of events (cases changed by peer review) was low, and it is possible that our study was underpowered to detect more subtle differences in peer review outcomes.
Conclusions A robust peer review program is feasible in a community radiation oncology practice, and our program led to changes in 9.5% of treatment plans. Peer review should focus on technical factors such as target contours and dosimetry results. On average cases require 7 minutes for peer review. Further research is needed to identify cases most appropriate for peer review.
References 1. Marks LB, Adams RD, Pawlicki T, et al. Enhancing the role of case-oriented peer review to improve quality and safety in radiation oncology: Executive summary. Pract Radiat Oncol. 2013;3:149-156.
Practical Radiation Oncology: January-February 2017 2. Hoopes DJ, Johnstone PA, Chapin PS, et al. Practice patterns for peer review in radiation oncology. Pract Radiat Oncol. 2015;5: 32-38. 3. Brundage MD, Dixon PF, Mackillop WJ, et al. A real-time audit of radiation therapy in a regional cancer center. Int J Radiat Oncol Biol Phys. 1999;43:115-124. 4. Fogarty GB, Hornby C, Ferguson HM, et al. Quality assurance in a radiation oncology unit: the chart round experience. Australas Radiol. 2001;45:189-194. 5. Boxer M, Forstner D, Kneebone A, et al. Impact of a real-time peer review audit on patient management in a radiation oncology department. J Med Imaging Radiat Oncol. 2009;53:405-411. 6. Leong CN, Shakespeare TP, Mukherjee RK, et al. Efficacy of an integrated continuing medical education (CME) and quality improvement (QI) program on radiation oncologist (RO) clinical practice. Int J Radiat Oncol Biol Phys. 2006;66:1457-1460. 7. Ballo MT, Chronowski GM, Schlembach PJ, et al. Prospective peer review quality assurance for outpatient radiation therapy. Pract Radiat Oncol. 2014;4:279-284. 8. Chen RC. Commentary: Toward safe and high quality care through peer review in radiation oncology: Need for more evidence. Pract Radiat Oncol. 2014;4:285-287. 9. Lawrence YR, Whiton MA, Symon Z, et al. Quality assurance peer review chart rounds in 2011: A survey of academic institutions in the United States. Int J Radiat Oncol Biol Phys. 2012;84:590-595. 10. Agresti A. An Introduction to Categorical Data Analysis. New York: John Wiley & Sons. 1996. 11. Hamilton SN, Hasan H, Parsons C, et al. Canadian radiation oncologists' opinions regarding peer review: A national survey. Pract Radiat Oncol. 2015;5:120-126. 12. Lefresne S, Olivotto IA, Joe H, et al. Impact of quality assurance rounds in a Canadian radiation therapy department. Int J Radiat Oncol Biol Phys. 2013.
www.practicalradonc.org
Basic Original Report
Detailed prospective peer review in a community radiation oncology clinic James D. Mitchell MD a,⁎, Thomas J. Chesnut MS b , David V. Eastham MD a , Carlo N. Demandante MD a , David J. Hoopes MD c a
Joint Radiation Oncology Center, David Grant Medical Center, Travis Air Force Base, California U.S. Census Bureau, Washington, DC c Department of Radiation Oncology, University of California San Diego, San Diego, California b
Received 27 April 2016; revised 19 July 2016; accepted 23 August 2016
Abstract Purpose: In 2012, we instituted detailed prospective peer review of new cases. We present the outcomes of peer review on patient management and time required for peer review. Methods and materials: Peer review rounds were held 3 to 4 days weekly and required 2 physicians to review pertinent information from the electronic medical record and treatment planning system. Eight aspects were reviewed for each case: 1) workup and staging; 2) treatment intent and prescription; 3) position, immobilization, and simulation; 4) motion assessment and management; 5) target contours; 6) normal tissue contours; 7) target dosimetry; and 8) normal tissue dosimetry. Cases were marked as, “Meets standard of care,” “Variation,” or “Major deviation.” Changes in treatment plan were noted. As our process evolved, we recorded the time spent reviewing each case. Results: From 2012 to 2014, we collected peer review data on 442 of 465 (95%) radiation therapy patients treated in our hospital-based clinic. Overall, 91 (20.6%) of the cases were marked as having a variation, and 3 (0.7%) as major deviation. Forty-two (9.5%) of the cases were altered after peer review. An overall peer review score of “Variation” or “Major deviation” was highly associated with a change in treatment plan (P b .01). Changes in target contours were recommended in 10% of cases. Gastrointestinal cases were significantly associated with a change in treatment plan after peer review. Indicators on position, immobilization, simulation, target contours, target dosimetry, motion management, normal tissue contours, and normal tissue dosimetry were significantly associated with a change in treatment plan. The mean time spent on each case was 7 minutes. Conclusions: Prospective peer review is feasible in a community radiation oncology practice. Our process led to changes in 9.5% of cases. Peer review should focus on technical factors such as target contours and dosimetry. Peer review required 7 minutes per case. Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
Presented at the 2013 ASTRO Annual Meeting, September 22-25, 2013, Atlanta, Georgia. Conflicts of interest: None. The views expressed in this material are those of the authors and do not reflect the official policy or position of the U.S. Government, the Department of Defense, or the Department of the Air Force. The work reported herein was performed under United States Air Force Surgeon General approved Clinical Investigation Number FDG20130018E. ⁎ Corresponding author. Joint Radiation Oncology Center, David Grant USAF Medical Center, 101 Bodin Circle, Travis Air Force Base, CA 94535. E-mail address: [email protected] (J.D. Mitchell). http://dx.doi.org/10.1016/j.prro.2016.08.011 1879-8500/Published by Elsevier Inc. on behalf of American Society for Radiation Oncology.
Practical Radiation Oncology: January-February 2017
Peer Review, Early Patient: DATE: Treating MD: --Review MD: ---
Peer Review in a Community Clinic
DOB: Site CNS/Peds H&N Lung/Sarc GI Breast GYN GU Heme Skin Palliative
1. Workup and Staging Comments:
2. Treatment Intent and Prescription
MRN: No Change
3. Position, immobilization, simulation
Variation
Meets Standard of Care
Major Deviation
Does NOT Meet Standard of Care I would manage this case differently. The current management plan is not reasonable. I recommend changes be made. I would:
I would not change the management in this case.
I would manage this case differently but the current management is reasonable. I would:
No Change
Do additional workup Do less workup
Do additional workup
Not Rx RT ↑Fraction Size ↓Fraction Size ↑Total Dose ↓Total Dose Δ RT Schedule Δ RT Modality
Not Rx RT ↑Fraction Size ↓Fraction Size ↑Total Dose ↓Total Dose Δ RT Schedule Δ RT Modality
Comments:
Comments:
Comments:
Comments:
↑Target Contour size ↓Target Contour size Include other targets Use Fusion Modality
↑Target Contour size ↓Target Contour size Include other targets Use Fusion Modality
↑Avoid Contour size ↓Avoid Contour size Include other Avoids Use Fusion Modality
↑Avoid Contour size ↓Avoid Contour size Include other Avoids Use Fusion Modality
↑Target Coverage Fill target cold spot ↓ Plan Max Dose
↑Target Coverage Fill target cold spot ↓ Plan Max Dose
↓ Normal Tissue Dose
↓ Normal Tissue Dose
Brain Brainstem Optics Cochlea Cord Parotid Larynx B. Plexus
Brain Brainstem Optics Cochlea Cord Parotid Larynx B. Plexus
No Change
Comments:
No Change
4. Motion assessment and management 5. Target Contours
No Change
No Change
Comments:
6. Normal Tissue (Avoidance) Contours Comments:
7. Target Dosimetry/HotSpot (DVH&Isodose) Comments:
8. Normal Tissue Dosimetry (DVH &Isodose)
No Change
No Change
Comments:
No Change
Overall Peer Review
No Change
Prospective Peer Review changed Tx plan? ---
Lung Heart Esophagus Liver Small Bowel Kidney Bladder Rectum
Variation
How?
Additional Comments:
Figure 1
51
Peer review document.
Lung Heart Esophagus Liver SmallBowel Kidney Bladder Rectum
Major Deviation
52
J.D. Mitchell et al
Introduction The American Society for Radiation Oncology (ASTRO) has identified peer review as a critical safety and quality improvement tool for radiation oncology departments. In 2013, Marks et al reported the ASTRO Executive Summary on case-oriented peer review, highlighting important safety measures that should be included in departmental policies and procedures. 1 In an ASTRO survey, 83% of radiation oncologists reported participating in a peer review program, and 65% reported reviewing cases before starting treatment. 2 Despite near-universal adoption in US academic and community radiation oncology departments, there are few prospective reports on the outcomes of peer review programs. Early reports came from Canada, Australia, and Singapore, with few data from centers within the United States. 3-6 Recently, the MD Anderson Cancer Center reported on a peer review program for their outpatient radiation therapy centers 7; in the accompanying editorial, Chen argues for more research into the costs and efficacy of peer review. 8 Large gaps remain in our understanding of the efficacy, yield, and costs of peer review. Specifically, there is no conclusive data showing which types of cases require peer review, which cases can safely forego peer review, and what are the critical aspects to review for each case. Additionally, there is little information on the time required for quality peer review. 9 In an attempt to address these knowledge gaps, our department standardized our peer review process and documentation, systematically collected data on prospective peer review of new cases, and recorded those changes. We later began collecting data on the amount of time spent on each case. In this report, we present data on the outcomes of peer review on clinical practice and decision-making as well as time required for review.
Methods and materials Beginning in 2012, the Joint Radiation Oncology Center at Travis Air Force Base optimized its departmental continuous quality improvement (CQI) program. This overhaul was performed as preparation for American College of Radiology accreditation and in response to internal safety concerns. A major part of this program included standardized, prospective peer review of new radiation therapy cases. Peer review was performed 3 to 4 times per week during our morning departmental “huddle,” in a setting where the electronic medical record and treatment planning system could be viewed simultaneously. At a minimum, 2 physicians, 1 physicist, 1 dosimetrist, and 1 radiation therapist were present. The treating physicians were all general radiation oncologists treating all sites of disease, and new cases were assigned to each physician at random. The reviewing physicians were also chosen at
Practical Radiation Oncology: January-February 2017
random based on who was present during each conference. The goal of the CQI program was to perform peer review on at least 50% of cases before beginning treatment. Cases that did not undergo peer review consisted of urgent cases that required immediate initiation of therapy. Prospective peer review was documented electronically using a standardized form (Fig 1). Each case was identified based on the primary site of treatment: central nervous system/ pediatrics, head and neck, lung/sarcoma, gastrointestinal, breast, gynecologic, genitourinary, hematologic, and palliative. Each case was presented by the treating physician, and the treatment plan was evaluated in real time by the reviewing physician. Eight separate aspects were reviewed for each case: 1) workup and staging (including evaluation and management up to the date of peer review); 2) treatment intent and prescription; 3) position, immobilization, and simulation; 4) motion assessment and management; 5) target contours; 6) normal tissue (avoidance) contours; 7) target dosimetry; and 8) normal tissue dosimetry. Review of the contours, treatment plan, and dose-volume histogram was performed in real-time within the treatment planning system. Therefore, slice-by-slice review of contours and dose distributions was performed as was interactive review of the dose-volume histogram. For each of the 8 aspects of a new case, the peer reviewer marked either, “No Change: I would not change the management in this case,” “Variation: I would manage this case differently but the current management plan is reasonable,” or “Major Deviation: I would manage this case differently. The current plan is not reasonable. I recommend changes be made.” A primary outcome of this work was to record how often peer review led to a change in the clinical management of a patient. All treatment changes prompted by peer review were recorded. Beginning in 2013, we made an improvement to the peer review documentation system that allowed it to track the time taken for each individual peer review case. An institutional review board exemption was obtained to study our peer review program (United States Air Force Surgeon General approved Clinical Investigation Number FDG20130018E.) We identified all cases that underwent prospective peer review between January 2012 and April 2014. Data were collected on 10 domains: the primary site of treatment, the 8 aspects of peer review listed previously, and whether or not a change was made based on the peer review process. The data were placed in an anonymous database for analysis.
Statistical analysis Given the categorical nature of the data collected with the exception of peer review time, we used a non-model-based statistical methodology using categorical data analysis 10 to explore the patient case characteristics as they relate to the peer review process and change in treatment outcome. Defining the patient characteristics as our independent variables and “change in treatment” as our dependent
Practical Radiation Oncology: January-February 2017 Table 1
Peer Review in a Community Clinic
53
Distribution of cases and peer review yield for each physician
Physician 1 2 3 4 5
Number of cases treated
Number of cases reviewed
Number of reviewed cases changed after peer review
Relative risk of treatment change after peer review
36 127 118 119 41
25 179 91 122 24
2 26 3 11 0
0.83 (P = 1.0) 2.4 (P = .005) 0.30 (P = .04) 0.92 (P = .95) NA
NA, not available.
variable, we examine the relationship between each independent variable and the change in treatment outcome using 2-way contingency tables. Using the Pearson χ 2 statistical test of independence and the resulting P values, we identify those patient characteristics that influence the change in treatment outcomes as supported by the given data.
P = .005) and review by physician 3 was less likely to lead to a change (RR = 0.3, P = .04). Overall, 91 (20.6%) cases were marked as having a variation, and 42 (9.5%) were altered after peer review. Of the 42 cases that were altered, 40 (95%) were marked as having a variation. Only 3 cases (0.7%) were marked as having a major deviation. The distribution of recommended changes based on primary disease site is listed in Table 2. Of the different primary cancer sites, gastrointestinal cases were 3.2 times more likely to have a change in treatment compared with all other cases (P = .002). Additionally, there was a trend toward an increase in changes in treatment plans for hematologic malignancies (RR = 2.8; P = .086), and a decrease in changes for genitourinary malignancies (RR = 0.5, P = .095). As would be expected, the indicator “Variation” or “Major Deviation” was highly associated with a change in treatment plan (P b .001). Of the different sections of the peer review process, target contours were most commonly marked as “Variation” or “Major Deviation.” Cases that were marked as having a variation or major deviation in target contours were more likely to lead to a change in treatment when compared with cases that were not marked as such (RR = 6.7, P b .001). Additional domains associated with a change in treatment plan included normal tissue dosimetry (RR = 5.3, P b .001); target dosimetry (RR = 5.3, P b .001); position, immobilization, and simulation (RR = 5.0, P b .001); normal tissue contours (RR = 8.0, P b .001); and motion assessment and management (RR = 4.1, P = .001). The overall results
Results During the years 2012 through 2014, we collected peer review data on 442 of 465 patients. This represents 95% of new treatment plans in the department that underwent peer review before initiating treatment. All of these are included in this analysis. Additionally, we collected data regarding the time required to perform peer review on the most recent 121 patients. All cases consisted of external beam radiation therapy, not including stereotactic radiosurgery. Five physicians were included in the study, and the distribution of cases treated and reviewed by each physician is listed in Table 1. Only physicians 2 and 3 were present in the department for the entire duration of the study. Table 1 also shows the number of cases changed after review by each physician as well as the relative risk of a treatment change for each physician reviewer. The majority of changes came after peer review by physicians 2 and 4. Peer review by physician 2 was more likely to lead to a change in treatment plan (relative risk [RR] = 2.4,
Table 2
Change in treatment plan based on primary disease site
Disease site Central nervous system Lung and sarcoma Breast Genitourinary Palliative Head and neck Gastrointestinal Hematologic
Number of cases (% of total) 8 59 38 128 115 45 30 16
(1.8) (13) (8.6) (29) (26) (10.2) (6.8) (3.6)
Number of cases with change in treatment (%)
P value for association with change
0 6 3 7 7 7 6 4
.75 1 .95 .09 .2 .23 .003 .086
(0) (10) (7.9) (5.4) (6.1) (15.6) (27) (25)
54
J.D. Mitchell et al
Practical Radiation Oncology: January-February 2017
for all peer review domains are listed in Table 3. Of the more targeted subsections of the peer review document, the most frequently cited variations included increase target size (n = 33), decrease target size (n = 11), increase target coverage (n = 6), and add additional workup (n = 6). Of the normal tissues, the most frequently cited recommendation was to decrease the dose to small bowel (n = 5). Table 4 lists the most frequently marked variations. For the most recent 121 patients, the mean time spent on peer review was 7.05 minutes (range, 2-24 minutes). The amount of time spent per case based on primary site is shown in Fig 2. Cases where peer review led to a change in therapy took 5.1 minutes longer to review than those cases that did not lead to a change (11.8 vs 6.7 minutes, P = .1). Head and neck cases required the most time for peer review, but the differences were not statistically significant.
Discussion As a field of medicine, radiation oncology has benefited greatly from advances in technology. Through integration of advanced treatment delivery machines, brachytherapy, treatment planning systems, and electronic medical records, we are able to deliver precision treatment for complex diseases. These technologic milestones present opportunities and challenges for patient safety. It follows that departments should have internal CQI programs outlining safety related policies and procedures. Peer review is an important aspect of departmental safety culture and has been recognized as such by ASTRO. The ASTRO Executive Summary on peer review states that, “Peer review is one of the most effective means for assuring the quality of qualitative, and potentially controversial, patient-specific decisions in radiation oncology.” The authors go on to outline important aspects to include in a peer review program spanning from the initial decision to treat with radiation therapy through delivery of treatment. 1 However, despite guidance from ASTRO, and a widespread use of peer review in at least some form, we are just beginning to scientifically examine how peer Table 3
review impacts patient care and departmental operations. In an editorial, Chen outlines several areas that demand further research including identifying which patients require peer review, when should peer review occur, and the true costs. 8 This report adds to our understanding of the impact of prospective peer review through a detailed analysis of 442 nearly serial patients treated in a nonacademic, hospital-based department. A potential barrier to quality peer review may include the amount of time required. To our knowledge, we are the first to report prospective data regarding the amount of time spent on peer review for each case. Our data show a mean time of 7 minutes per case with a range of 2 to 24 minutes. In a survey of academic radiation oncology departments, Lawrence reported a mean time of 2.7 minutes spent on each case. 9 However, these data were not collected prospectively during peer review conferences, but rather through survey questions that relied on recall and estimation. Our data show that meaningful peer review requires time, and a busy radiation oncology clinic may have difficulty setting aside enough time for peer review of every case. In a survey of Canadian radiation oncologists, Hamilton reported that 40% found it moderately difficult and 22% extremely difficult to incorporate peer review into their schedules. 11 The question then arises: does every case require peer review? Our data show that gastrointestinal cases were significantly associated with treatment changes based on peer review, but there was no clear subset that we identified as not benefiting from peer review. The group from MD Anderson Cancer Center also identified gastrointestinal sites as well as head and neck and gynecologic malignancies as accounting for the majority of changes made. 7 In a report from Canada, Lefresne reported that gastrointestinal malignancies, lung cancer, and lymphoma represented the highest proportion of cases marked as needing either a minor or major change prior to treatment. 12 Although gastrointestinal cases appear to have a high rate of changes recommended through peer review, clearly we need more research to better identify cases that benefit the most.
Change in treatment plan based on peer review results of technical factors
Workup and staging Treatment intent and prescription Position, immobilization, and simulation Motion assessment and management Target contours Normal tissue contours Target dosimetry Normal tissue dosimetry
Number of cases marked as variation (% of total)
Number of cases with change in treatment (% of category)
P value for association with change
9 9 6 6 44 9 14 13
2 (22) 2 (22) 3 (50) 2 (33.3) 18 (40.9) 6 (67) 5 (35.7) 9 (69.2)
.459 .459 b.001 .001 b.001 b.001 b.001 b.001
(2.0) (2.0) (1.3) (1.3) (9.96) (2.0) (3.2) (2.9)
Practical Radiation Oncology: January-February 2017 Table 4 Most frequently marked reasons for labelling a case as “variation” Reason for variation Increase target size Decrease target size Increase target coverage Add additional workup Decrease small bowel dose Decrease plan maximum dose Decrease prescription dose Add additional avoidance structure Decrease size of avoidance structure Decrease spinal cord dose Decrease lung dose Increase prescription dose Add motion management (4-dimensional computed tomography) Do not use radiation therapy
Number of cases marked as variation 33 11 6 6 5 5 4 3 3 2 2 2 2 2
Our data show a trend toward a decreased number of genitourinary cases with changes after peer review, but this difference was not statistically significant. The vast majority of our genitourinary cases are prostate cancer, which is the most frequently treated diagnosis in our department. Because of our large volume of prostate patients, we have a standardized approach to planning
Peer Review in a Community Clinic
55
prostate radiation therapy. This standardization, in combination with our collective experience treating prostate cancer, may explain why we have fewer changes recommended at peer review. On the other hand, our gastrointestinal practice represents a wide variety of cases including cancers of the esophagus, stomach, pancreas, liver, rectum, and anus. It may be that peer review is more important for treatment that is performed less frequently, but determining this requires further investigation. Some departments may be inclined to eliminate palliative cases from peer review because of a perception of simplicity and decreased risk of harm. However, our data do not show a decreased peer review yield for palliative cases, and we recommend continuing peer review for palliative cases. Another goal of our study was to identify which aspects of a radiation therapy case are most important from a peer review standpoint. The planning steps that we found were most closely associated with a change in treatment included: 1) position, immobilization, and simulation; 2) target contours; 3) target dosimetry; 4) motion management; 5) normal tissue dosimetry; and 6) normal tissue contours. Similarly, in the Canadian series reported by Lefresne, they noted that the most common causes for plans to be marked as requiring a significant change included, “inadequate target volume coverage (36%), suboptimal dose or fractionation (27%), errors in patient setup (27%), and overtreatment of normal tissue (9%).” 12 It may therefore stand to reason that in an effort to improve efficiency, peer review should focus on these technical factors.
Figure 2 Peer review time for different disease sites. (Tukey-style box plot: the box bounds the interquartile range [IQR] divided by the median, and whiskers extend to a maximum of 1.5 × IQR beyond the box. Time is expressed in minutes.)
56
J.D. Mitchell et al
One weakness of our study is the limited number of gynecologic cases. Because of our patient population, we see very few gynecologic patients and therefore we cannot make any meaningful conclusions regarding the usefulness of, or time required for, peer review. Additionally, we did not collect data on peer review for brachytherapy or radiosurgery, two areas where peer review has been understudied and underutilized. Last, the total number of events (cases changed by peer review) was low, and it is possible that our study was underpowered to detect more subtle differences in peer review outcomes.
Conclusions A robust peer review program is feasible in a community radiation oncology practice, and our program led to changes in 9.5% of treatment plans. Peer review should focus on technical factors such as target contours and dosimetry results. On average cases require 7 minutes for peer review. Further research is needed to identify cases most appropriate for peer review.
References 1. Marks LB, Adams RD, Pawlicki T, et al. Enhancing the role of case-oriented peer review to improve quality and safety in radiation oncology: Executive summary. Pract Radiat Oncol. 2013;3:149-156.
Practical Radiation Oncology: January-February 2017 2. Hoopes DJ, Johnstone PA, Chapin PS, et al. Practice patterns for peer review in radiation oncology. Pract Radiat Oncol. 2015;5: 32-38. 3. Brundage MD, Dixon PF, Mackillop WJ, et al. A real-time audit of radiation therapy in a regional cancer center. Int J Radiat Oncol Biol Phys. 1999;43:115-124. 4. Fogarty GB, Hornby C, Ferguson HM, et al. Quality assurance in a radiation oncology unit: the chart round experience. Australas Radiol. 2001;45:189-194. 5. Boxer M, Forstner D, Kneebone A, et al. Impact of a real-time peer review audit on patient management in a radiation oncology department. J Med Imaging Radiat Oncol. 2009;53:405-411. 6. Leong CN, Shakespeare TP, Mukherjee RK, et al. Efficacy of an integrated continuing medical education (CME) and quality improvement (QI) program on radiation oncologist (RO) clinical practice. Int J Radiat Oncol Biol Phys. 2006;66:1457-1460. 7. Ballo MT, Chronowski GM, Schlembach PJ, et al. Prospective peer review quality assurance for outpatient radiation therapy. Pract Radiat Oncol. 2014;4:279-284. 8. Chen RC. Commentary: Toward safe and high quality care through peer review in radiation oncology: Need for more evidence. Pract Radiat Oncol. 2014;4:285-287. 9. Lawrence YR, Whiton MA, Symon Z, et al. Quality assurance peer review chart rounds in 2011: A survey of academic institutions in the United States. Int J Radiat Oncol Biol Phys. 2012;84:590-595. 10. Agresti A. An Introduction to Categorical Data Analysis. New York: John Wiley & Sons. 1996. 11. Hamilton SN, Hasan H, Parsons C, et al. Canadian radiation oncologists' opinions regarding peer review: A national survey. Pract Radiat Oncol. 2015;5:120-126. 12. Lefresne S, Olivotto IA, Joe H, et al. Impact of quality assurance rounds in a Canadian radiation therapy department. Int J Radiat Oncol Biol Phys. 2013.