- Email: [email protected]
Early Results From the ASTRO Practice Patterns in Peer Review Survey Project
Oral Scientific Sessions S119
Volume 87 Number 2S Supplement 2013 Oral Scientific Abstract 290; Table Descriptive data and paired t test statistics P value for workload and sub-tasks
Measures (Dimensions/Sub-tasks) Workload
Performance
NASA-TLX Mental Demand Temporal Demand Effort Frustration Total completion-time of all the sub-tasks (sec) Completion-time for Obtaining Information Completion-time for Searching for Information Completion-time for Inputting Information
Regular coverage mean (std dev)
Cross coverage mean (std dev)
27 31 27 36 19 151
49 60 55 54 49 228
(15) (22) (17) (18) (12) (34)
(11) (21) (24) (18) (28) (102)
P value .01 .013 .006 .006 .02 .015
33 (45)
85 (119)
.07
47 (25)
57 (32)
.3
72 (34)
86 (29)
.28
significantly longer for CC vs RC scenario (85 [119] vs 33 [45], p Z 0.07, Table). Conclusions: Participants took significantly longer time to obtain information, and scored double for mental and temporal dimensions of NASATLX, and were more frustrated during CC vs RC scenario. Effort must be done to properly design the EMRs to facilitate easy access to information during CC and hence to reduce unnecessary WL. This study is a first step towards quantifying CC effects, during treatment planning tasks in radiation oncology. We recognize that there are inherent limitations to this study (e.g., simulated environment with a limited number of participants from one clinic using specific tools). Nevertheless, the designed experiment scenario and the environment mock-up were similar to the real-life. As there is no well-accepted model to study CC in the medical setting, the simulated model system used seems reasonable and the data provide some quantification for a common challenge in healthcare. Author Disclosure: P. Mosaly: None. L.M. Mazur: None. E.L. Jones: None. L.M. Hoyle: None. B.S. Chera: None. T. Zagar: None. L.B. Marks: None.
291 Early Results From the ASTRO Practice Patterns in Peer Review Survey Project D.J. Hoopes,1 P.A. Johnstone,2 P.S. Chapin,3 C.M. Schubert Kabban,3 W.R. Lee,4 A.B. Chen,5 B.A. Fraass,6 W.J.K. Skinner,7 and L.B. Marks8; 1 David Grant Medical Center, Travis AFB, CA, 2Indiana University, Indianapolis, IN, 3Air Force Institute of Technology, Wright-Patterson AFB, OH, 4Duke University School of Medicine, Durham, NC, 5DanaFarber Cancer Institute, Boston, MA, 6Cedars-Sinai Medical Center, Los Angeles, CA, 7Walter Reed Bethesda National Military Medical, Bethesda, MD, 8University of North Carolina, Chapel Hill, NC Purpose/Objective(s): Peer review is an important aspect of medical quality assurance and patient safety. There is little published literature on peer review in Radiation Oncology. To establish current peer review practice patterns, evaluate interest in formal recommendation for peer review, and establish a framework for future recommendations, ASTRO proposed a survey of physician members. Materials/Methods: A Radiation Oncology-specific peer review survey instrument was developed, formally tested, and found to meet established levels of reliability and validity. The final instrument consisted of 8 demographic items and 25 peer/practice review items delivered using a web-based survey platform including reminders. All ASTRO physicianmembers and members-in-training worldwide were invited by email to participate in the survey. Results: A total of 5,674 physicians were contacted starting in January 2013. During days 1-23 of the 60 day survey, 561 physicians participated (10%) yielding a +/- 4% margin of error (95% CI). Those responding included 509 staff radiation oncologists, 35 residents/trainees, 265
academic physicians, 25 Government/DoD physicians, and 112 non-US physicians. The mean years-in-practice Z 13 (SD 10), mean number of new patients/year Z 238 (SD 128), and median practice size Z 6 physicians. Overall, 83% of respondents were involved in a peer review program, 75% were comfortable with their program, and 80% felt that peer review was encouraged. Of those involved in peer review, 65% report doing at least some peer review before radiation therapy starts. Of patients treated by these physicians, 56% (SD 35) are reviewed before treatment. Peer review elements reviewed include overall treatment strategy (88%), dose/fractionation (89%), contouring (58%), and isodose/DVH (75%). Peer review is performed using hard copy charts (34%), electronic medical records (63%), and images/isodose plots/DVHs on screen (73%). Eightyseven percent of physicians have changed radiation treatment plans because of peer review. These providers describe making changes in 9% (SD 14) of cases and report that 3% (SD 5) of cases fall outside the standard of care. Seventy-four percent of physicians agree or strongly agree that ASTRO should make formal peer review recommendations with 7% in opposition to this proposal. Conclusions: Preliminary results from this survey suggest that peer review in Radiation Oncology is common and leads to changes in clinical management in a meaningful fraction of cases. There is much variation in the manner of conducting (e.g. content, timing), and reported utility of, peer review. The vast majority of ASTRO physician members support formal recommendations and guidance on peer review. Full results and analysis will be available for the ASTRO 2013 Annual Meeting. Author Disclosure: D.J. Hoopes: None. P.A. Johnstone: None. P.S. Chapin: None. C.M. Schubert Kabban: None. W.R. Lee: None. A.B. Chen: None. B.A. Fraass: G. Consultant; Varian Oncology Systems as part of the Varian Patient Safety Council. H. Travel Expenses; Varian Oncology Systems as part of the Varian Patient Safety Council. W.J.K. Skinner: None. L.B. Marks: None.
292 A Real-Time Safety and Quality Reporting System: Initial Clinical Outcomes and Survey Results D.A. Rahn, G.G. Kim, T. Pawlicki, and A.J. Mundt; University of California San Diego, La Jolla, CA Purpose/Objective(s): With the imminent deployment of a national incident learning system for radiation therapy our purpose was to provide an independent clinical experience of online incident capture and analysis together with staff perception of the system’s utility. Materials/Methods: On September 24, 2010, our Department initiated an online real-time voluntary reporting system for safety issues called the Radiation Oncology Quality Reporting System (ROQRS). Any staff member can submit an online report either signed or anonymously. The system immediately notifies members via email of reports and initiates problem solving. A separate system (not reported here) logs machine downtime and mechanical issues. We retrospectively reviewed the reports from the ROQRS program’s inception through January 18, 2013 (2 years, 3 months, and 25 days). “Near miss” events were defined as situations where an individual reported catching an error that, if uncorrected, would have resulted in deviation from the radiation therapy prescription. “Errors” were defined as patient treatment that actually deviated from the radiation therapy prescription. In addition, a survey of the physics, dosimetry, nursing and radiation therapy staff using the ROQRS program was conducted to assess staff values pertaining to safety and use of the ROQRS system. Results: During the study interval there were 60,168 fractions of external beam RT and 955 brachytherapy procedures. There were 298 entries in the ROQRS system, of which 108 problems were reported. There were 31 patients with “near misses” and 27 patients with “errors” reported. Errors occurred in 68 total treatment fractions (0.11% of treatments delivered during the study interval). None of these errors resulted in a medical event of deviation from the prescription by 5% or more. A solution to the error was documented in ROQRS in 65% of the cases. Errors were documented as a repeated error in 22% of the cases. On Chi square analysis a disproportionate number of these errors were due to improper patient
Volume 87 Number 2S Supplement 2013 Oral Scientific Abstract 290; Table Descriptive data and paired t test statistics P value for workload and sub-tasks
Measures (Dimensions/Sub-tasks) Workload
Performance
NASA-TLX Mental Demand Temporal Demand Effort Frustration Total completion-time of all the sub-tasks (sec) Completion-time for Obtaining Information Completion-time for Searching for Information Completion-time for Inputting Information
Regular coverage mean (std dev)
Cross coverage mean (std dev)
27 31 27 36 19 151
49 60 55 54 49 228
(15) (22) (17) (18) (12) (34)
(11) (21) (24) (18) (28) (102)
P value .01 .013 .006 .006 .02 .015
33 (45)
85 (119)
.07
47 (25)
57 (32)
.3
72 (34)
86 (29)
.28
significantly longer for CC vs RC scenario (85 [119] vs 33 [45], p Z 0.07, Table). Conclusions: Participants took significantly longer time to obtain information, and scored double for mental and temporal dimensions of NASATLX, and were more frustrated during CC vs RC scenario. Effort must be done to properly design the EMRs to facilitate easy access to information during CC and hence to reduce unnecessary WL. This study is a first step towards quantifying CC effects, during treatment planning tasks in radiation oncology. We recognize that there are inherent limitations to this study (e.g., simulated environment with a limited number of participants from one clinic using specific tools). Nevertheless, the designed experiment scenario and the environment mock-up were similar to the real-life. As there is no well-accepted model to study CC in the medical setting, the simulated model system used seems reasonable and the data provide some quantification for a common challenge in healthcare. Author Disclosure: P. Mosaly: None. L.M. Mazur: None. E.L. Jones: None. L.M. Hoyle: None. B.S. Chera: None. T. Zagar: None. L.B. Marks: None.
291 Early Results From the ASTRO Practice Patterns in Peer Review Survey Project D.J. Hoopes,1 P.A. Johnstone,2 P.S. Chapin,3 C.M. Schubert Kabban,3 W.R. Lee,4 A.B. Chen,5 B.A. Fraass,6 W.J.K. Skinner,7 and L.B. Marks8; 1 David Grant Medical Center, Travis AFB, CA, 2Indiana University, Indianapolis, IN, 3Air Force Institute of Technology, Wright-Patterson AFB, OH, 4Duke University School of Medicine, Durham, NC, 5DanaFarber Cancer Institute, Boston, MA, 6Cedars-Sinai Medical Center, Los Angeles, CA, 7Walter Reed Bethesda National Military Medical, Bethesda, MD, 8University of North Carolina, Chapel Hill, NC Purpose/Objective(s): Peer review is an important aspect of medical quality assurance and patient safety. There is little published literature on peer review in Radiation Oncology. To establish current peer review practice patterns, evaluate interest in formal recommendation for peer review, and establish a framework for future recommendations, ASTRO proposed a survey of physician members. Materials/Methods: A Radiation Oncology-specific peer review survey instrument was developed, formally tested, and found to meet established levels of reliability and validity. The final instrument consisted of 8 demographic items and 25 peer/practice review items delivered using a web-based survey platform including reminders. All ASTRO physicianmembers and members-in-training worldwide were invited by email to participate in the survey. Results: A total of 5,674 physicians were contacted starting in January 2013. During days 1-23 of the 60 day survey, 561 physicians participated (10%) yielding a +/- 4% margin of error (95% CI). Those responding included 509 staff radiation oncologists, 35 residents/trainees, 265
academic physicians, 25 Government/DoD physicians, and 112 non-US physicians. The mean years-in-practice Z 13 (SD 10), mean number of new patients/year Z 238 (SD 128), and median practice size Z 6 physicians. Overall, 83% of respondents were involved in a peer review program, 75% were comfortable with their program, and 80% felt that peer review was encouraged. Of those involved in peer review, 65% report doing at least some peer review before radiation therapy starts. Of patients treated by these physicians, 56% (SD 35) are reviewed before treatment. Peer review elements reviewed include overall treatment strategy (88%), dose/fractionation (89%), contouring (58%), and isodose/DVH (75%). Peer review is performed using hard copy charts (34%), electronic medical records (63%), and images/isodose plots/DVHs on screen (73%). Eightyseven percent of physicians have changed radiation treatment plans because of peer review. These providers describe making changes in 9% (SD 14) of cases and report that 3% (SD 5) of cases fall outside the standard of care. Seventy-four percent of physicians agree or strongly agree that ASTRO should make formal peer review recommendations with 7% in opposition to this proposal. Conclusions: Preliminary results from this survey suggest that peer review in Radiation Oncology is common and leads to changes in clinical management in a meaningful fraction of cases. There is much variation in the manner of conducting (e.g. content, timing), and reported utility of, peer review. The vast majority of ASTRO physician members support formal recommendations and guidance on peer review. Full results and analysis will be available for the ASTRO 2013 Annual Meeting. Author Disclosure: D.J. Hoopes: None. P.A. Johnstone: None. P.S. Chapin: None. C.M. Schubert Kabban: None. W.R. Lee: None. A.B. Chen: None. B.A. Fraass: G. Consultant; Varian Oncology Systems as part of the Varian Patient Safety Council. H. Travel Expenses; Varian Oncology Systems as part of the Varian Patient Safety Council. W.J.K. Skinner: None. L.B. Marks: None.
292 A Real-Time Safety and Quality Reporting System: Initial Clinical Outcomes and Survey Results D.A. Rahn, G.G. Kim, T. Pawlicki, and A.J. Mundt; University of California San Diego, La Jolla, CA Purpose/Objective(s): With the imminent deployment of a national incident learning system for radiation therapy our purpose was to provide an independent clinical experience of online incident capture and analysis together with staff perception of the system’s utility. Materials/Methods: On September 24, 2010, our Department initiated an online real-time voluntary reporting system for safety issues called the Radiation Oncology Quality Reporting System (ROQRS). Any staff member can submit an online report either signed or anonymously. The system immediately notifies members via email of reports and initiates problem solving. A separate system (not reported here) logs machine downtime and mechanical issues. We retrospectively reviewed the reports from the ROQRS program’s inception through January 18, 2013 (2 years, 3 months, and 25 days). “Near miss” events were defined as situations where an individual reported catching an error that, if uncorrected, would have resulted in deviation from the radiation therapy prescription. “Errors” were defined as patient treatment that actually deviated from the radiation therapy prescription. In addition, a survey of the physics, dosimetry, nursing and radiation therapy staff using the ROQRS program was conducted to assess staff values pertaining to safety and use of the ROQRS system. Results: During the study interval there were 60,168 fractions of external beam RT and 955 brachytherapy procedures. There were 298 entries in the ROQRS system, of which 108 problems were reported. There were 31 patients with “near misses” and 27 patients with “errors” reported. Errors occurred in 68 total treatment fractions (0.11% of treatments delivered during the study interval). None of these errors resulted in a medical event of deviation from the prescription by 5% or more. A solution to the error was documented in ROQRS in 65% of the cases. Errors were documented as a repeated error in 22% of the cases. On Chi square analysis a disproportionate number of these errors were due to improper patient