- Email: [email protected]
RADPEER Peer Review: Relevance, Use, Concerns, Challenges, and Direction Forward
RADPEER Peer Review: Relevance, Use, Concerns, Challenges, and Direction Forward Hani Abujudeh, MD, MBAa, Robert S. Pyatt Jr, MDb, Michael A. Bruno, MDc, Alison L. Chetlen, DOc, David Buck, MDd, Susan K. Hobbs, MD, PhDe, Christopher Roth, MDf, Charles Truwit, MDg, Rajan Agarwal, MD, MBAh, Scott T. O. Kennedy, MD, MBAi, Lucille Glenn, MDj
RADPEER is a product developed by the ACR that aims to assist radiologists with quality assessment and improvement through peer review. The program opened in 2002, was initially offered to physician groups in 2003, developed an electronic version in 2005 (eRADPEER), revised the scoring system in 2009, and first surveyed the RADPEER membership in 2010. In 2012, a survey was sent to 16,000 ACR member radiologists, both users and nonusers of RADPEER, with the goal of understanding how to make RADPEER more relevant to its members. A total of 31 questions were used, some of which were repeated from the 2010 survey. The ACR’s RADPEER committee has published 3 papers on the program since its inception. In this report, the authors summarize the survey results and suggest future opportunities for making RADPEER more useful to its membership. Key Words: RADPEER, peer review, OPPE, quality and safety, PQI J Am Coll Radiol 2014;-:---. Copyright © 2014 American College of Radiology
INTRODUCTION
RADPEER was designed to be a simple, cost-effective process that allows radiologist peer review to be performed during the routine interpretation of current images [1]. If prior images and reports are available at the time a new study is being interpreted, these prior studies and the accuracy of their interpretation would typically be evaluated at the time the radiologist interprets the current study. RADPEER was first offered to physician groups in 2003, and in January 2013 it celebrated its 10th anniversary. The original RADPEER scoring system had 4 possible scores: 1 ¼ concur with interpretation; 2 ¼ difficult
a
Massachusetts General Hospital, Boston, Massachusetts.
b
Chambersburg Imaging Associates, Chambersburg, Pennsylvania.
c
Penn State Milton Hershey Medical Center, Hershey, Pennsylvania. d Indiana Regional Imaging, Indiana, Pennsylvania. e
University of Rochester, Rochester, New York.
f
Duke University, Durham, North Carolina.
g
Hennepin Health System, Minneapolis, Minnesota.
h
Abington Health, Lansdale Hospital, Lansdale, Pennsylvania.
i
University of California San Diego Medical Center, San Diego, California.
j
Virginia Mason Medical Center, Seattle, Washington.
Corresponding author and reprints: Hani Abujudeh, MD, MBA, Massachusetts General Hospital, Founders Building 213 D, 55 Fruit Street, Boston, MA 02114; e-mail: [email protected]. ª 2014 American College of Radiology 1546-1440/14/$36.00 http://dx.doi.org/10.1016/j.jacr.2014.02.004
diagnosis, not ordinarily expected to be made; 3 ¼ diagnosis should be made most of the time; and 4 ¼ diagnosis should be made almost every time, misinterpretation of findings. In 2009, the second RADPEER committee white paper was published [2], and the scoring system was revised to include optional categories of (a) unlikely to be clinically significant and (b) likely to be clinically significant, for scores of 2, 3, and 4. The 2009 paper gave examples of scoring. Also in 2009, RADPEER, as a practice quality improvement (PQI) project, was accepted by the ABR. In 2010, a survey was sent to all RADPEER members in an attempt to understand the motivation of the membership body for using RADPEER. As a result, the committee published a paper titled “Getting the Most Out of RADPEER” [3], which explained the process and offered specific suggestions on how to improve the usefulness of RADPEER. Specific suggestions included improving the culture and acceptance of RADPEER, defining categories of discrepancies, describing which imaging modalities should be reviewed, and developing an appeals process. To better understand the radiology practice use of peer review and RADPEER, a questionnaire was developed and distributed in September 2012, with the goal of understanding how to make the RADPEER program more relevant. This was a web-based survey 1
2 Journal of the American College of Radiology/Vol.
-
No.
-
Month 2014
with a total of 31 questions, which was sent to more than 15,000 recipients. A total of 1,589 responses were received, for a response rate of about 10%. As of midNovember 2013, RADPEER had a total of 1,147 groups participating in the program, with a total of 17,037 physicians, with 1,679 of those being RADPEER PQI participants. To date, the median number of cases reviewed in RADPEER is 776 each year, which translates into 3 to 4 cases reviewed per working day. We believe that this number of cases may reasonably be achieved by most practicing radiologists in a peer review process. RESULTS Characteristics of Respondents
The 2012 survey participants came from a variety of practice settings, including 46.3% from hospital-based private practices, 23.2% from academic practices, 19.3% from outpatient-based private practices, and 17.7% from hospital employee practices. More than three-fifths (61.4%) of practices used RADPEER, with 4.9% of responding radiologists unsure if their practices participated. Of practices that participated in RADPEER, 85.2% had all radiologists in the practices participate in RADPEER. The vast majority of radiologists who responded to the survey had been using RADPEER for more than 3 years (73.8%). The survey demonstrated that radiologists are using multiple IT platforms and vendors to participate in RADPEER including PowerScribe (7.6%), PeerVue (4.8%), and Primordial (4.7%). More than a quarter (26.4%) of responders use other systems to record RADPEER findings, of whom 44% have home-grown systems. Why Facilities Do Peer Review
Radiologists have several data-reporting requirements to entities outside and inside their hospitals and practices. To meet CMS requirements for reimbursement policies, radiologists often seek the accreditation services offered by the ACR. This is reflected in the increase in
RADPEER participation seen in 2007 and 2008, soon after the incorporation of peer review into the ACR accreditation process. RADPEER can satisfy both hospital and Joint Commission credentialing requirements. Part of the accreditation process includes a testament from a department about the performance of a radiologist. The RADPEER survey asked, “Does your hospital require peer review data for re-credentialing?” and 43.8% responded “yes” (Table 1). In those situations, the radiologists must perform peer review to be credentialed and recredentialed by the hospital to be able practice and continue to practice within the hospital. Radiologists are also involved in the hospital credentialing process by The Joint Commission, which requires an ongoing physician performance evaluation (OPPE) and a focused physician performance evaluation [4]. Peer review is one way to perform both OPPE and focused physician performance evaluation. Radiologists must keep their board certifications current with the ABR. The ABR now requires board-certified physicians to participate in its Maintenance of Certification program, and the ACR’s RADPEER program is an accepted PQI project. When participants were asked, “Why does your practice perform peer review?” the two most common responses referred either to the ACR requirements or to the hospital requirements, at 79.9% and 69.6%, respectively (Table 1). The PQI requirements accounted for (29.5%), and other entities’ requirements, such as those of the US Department of Veterans Affairs, and the desire to improve patient care quality via selfmonitoring and education accounted for (13.2%). Enforcement of and Satisfaction With Peer Review
The majority of practices have set peer review targets (71.8%) (Table 2). These targets for the majority of respondents involve absolute numbers of cases rather than percentages of cases. Of those physicians who responded, 50% prefer an absolute number of cases to report. In terms of the time period of review, respondents demonstrated a
Table 1. Why facilities do peer review Hospital requirement Yes No Don’t know Reasons stated for use of peer review To meet accreditation requirements, such as those of the ACR It satisfies hospital recredentialing requirements, such as those of The Joint Commission It is a requirement for other reasons For MOC PQI project For other reasons (not a requirement) Themes observed in free-text responses Quality improvement in patient care Education Required Note: MOC ¼ Maintenance of Certification; PQI ¼ practice quality improvement.
Response Percentage
Numerator/Denominator
43.8 23.7 32.5
615/1,405 333/1,405 457/1,405
79.9 69.6
1122/1,405 978/1,405
16.2 29.5 13.2
228/1,405 415/1,405 186/1,405
78.4 5.7 11.9
138/176 10/176 21/176
Abujudeh et al/RADPEER Peer Review 3
Table 2. Enforcement of and satisfaction with peer review Response Numerator/ Percentage Denominator Are targets used? Yes No Not available
71.8 23.8 4.3
Response Percentage
1,009/1,405 335/1,405 61/1,405 Actual
Type of targets Number Percent No opinion Period Per day Per month Per quarter Per year No opinion
Numerator/ Denominator
Desired
80.6 19.4
662/821 159/821
49.7 29.3 21.1
375/755 221/755 159/755
19.8 44.0 21.7 14.6
160/810 356/810 176/810 118/810
18.0 39.6 18.6 15.8 8.0
116/646 256/646 120/646 102/646 52/646
moderate preference for monthly reviews, with daily, quarterly, and yearly reviews fairly equally represented. Monthly review intervals (44%) are moderately preferred over other intervals, such as quarterly (21.7%). Concerns and Challenges
Radiologists have numerous concerns when it comes to peer review processes. Concerns such as the potential discoverability of the data, the awkwardness of being graded, the possible ramifications of institutional politics with possible misuse of the data, the validity of the results, potential harm to the reputation of the radiologists involved, the use of time and resources involved, and the possibility of punitive outcomes are among many other concerns. When asked, “Do you feel there is an under reporting of significant disagreements in the peer review process at your practice?” 33% of respondents said “yes” (Table 3). This may be directly related to the concerns of discoverability, at 65.4%. The survey results suggest that anonymity may improve reporting of disagreements, at 43%. This stands in contrast to the survey results suggesting that only 38.7% of peer review processes are anonymous. The RADPEER report contains the identities of the user group, the two radiologists, and the modality type, as well as the score. It does not contain information about the patient or the specific examination. Information sent to the ACR is protected under the Virginia’s medical malpractice law, one of the strongest peer review statutes in the country. Radiology groups should consult lawyers familiar with local statues regarding peer review data. In most, but regrettably not all, states, the peer review process is considered protected and therefore not discoverable. As noted in Table 3, discoverability in the medicolegal sense of the word was noted as a concern by 65% of respondents. Unfortunately, in several very visible and litigious states, the process is not protected, which poses a seemingly insurmountable hurdle to practices seeking to engage in peer review.
Regrettably, nearly half of respondents still feel that the process of peer review remains too time consuming, and nearly a third struggle to see quality improvement as a legitimate outcome of the current systems of peer review. More than half of respondents thought that better integration with their PACS or voice recognition daily workflow would improve compliance with peer review in their practices, presumably correlating with the response of “too time consuming.” Other concerns included the notion of “being graded” by one’s peers, awkwardness of grading one’s senior (eg, medical director), capturing of peer review information by credentialing committees in a hospital (with the potential for decredentialing of radiologists), and public embarrassment. Although these concerns may be legitimate, there are ways to overcome most of them. By anonymizing the process, the “public grading” and embarrassment issues can be obviated. As seen in Table 3, 43% of respondents think that anonymizing the process would improve reporting of disagreements, or more important, only 27% did not think it would help. This finding is interesting in the face of only 38.7% of respondents’ reporting that their peer review systems are currently anonymous. Despite the potential concern for “grading” (or being graded by) a fellow radiologist, the committee feels that identifying important errors can have a significant effect on patient care and requires participants to act. When a significant error is identified, it becomes incumbent upon the radiologists to take corrective action. In particular, interpretations may require report addenda; communications with clinicians often are needed, and in many cases, discussions with patients may be appropriate. As a result, the issue of litigation remains a legitimate concern. However, not addressing the errors is not in patients’ best interests. Moreover, as has been noted in the literature, transparency and disclosure are far more likely to prevent litigation than to precipitate it [5].
4 Journal of the American College of Radiology/Vol.
-
No.
-
Month 2014
Table 3. Concerns regarding peer review reporting Concerns reported Discoverability (medicolegal) of data Being graded Awkwardness of grading medical director or supervising physician Information being used for decredentialing Public embarrassment Other Are disagreements underreported? Yes No Don’t know Would anonymity help improve reporting of disagreements Yes No Don’t know Is your peer review system anonymous? Yes No Don’t know Best reason not to do peer review Too time consuming Do not see it as helpful to quality improvement Other Analysis of free-text responses No good reason not to Discovery/legal concerns Misuse of data Waste of time/too time consuming No relevant data or feedback Would better integration with vendor’s system encourage peer review? Yes No Don’t know
In addition to anonymizing the process, respondents generally felt that improvements in the integration of peer review into the PACS would be helpful and would encourage the use of peer review. In some cases, voice dictation and PACS “prioritizing” and presenting examinations for peer review that were performed recently might improve utilization because finding a clinically significant missed finding very recently after an imaging procedure could be much easier to reconcile for a radiologist or practice than older findings. RADPEER should not be the sole tool for quality improvement in a practice setting but rather should be one of several assets in a comprehensive quality improvement program. Although the RADPEER system is not designed to be used as part of an OPPE, it can be incorporated into such programs, for example, by facilitating the tracking of physician participation in peer review as a monitored performance measure. We would strongly caution against designing an OPPE regime that incorporates the actual RADPEER scores (eg, 2b, 3a) or fractions of scores (eg, percentages of scores of 3 and 4) as a means of competency assessment however. Such a plan, whether based on a maximum average RADPEER score or some absolute threshold RADPEER score, when used to evaluate
Response Percentage
Numerator/Denominator
65.4 49.4 29.5 35.2 29.9 20.1
833/1,273 629/1,273 376/1,273 448/1,273 380/1,273 256/1,273
33.5 45.3 21.2
426/1,273 577/1,273 270/1,273
43.0 27.1 29.9
547/1,273 345/1,273 381/1,273
38.7 51.1 10.2
194/501 256/501 51/501
47.7 32.3 32.0
607/1,273 411/1,273 407/1,273
45.8 7.6 6.5 6.3 15.4
175/382 29/382 25/382 24/382 59/382
52.3 20.0 27.7
652/1,247 250/1,247 345/1,247
individual physician performance for maintenance of clinical privileges, is likely to produce the undesired consequence of undermining the quality improvement goals of peer review. Although this has indeed been done at some institutions, there is a high risk for undermining users’ trust in the “just culture” element needed to make RADPEER optimally effective. This is because when peer reviewers are aware that the actual scores they assign may have potential adverse consequences for those reviewed, this knowledge would be anticipated to result in inappropriate downward pressure on their scoring, thus undermining the effectiveness of RADPEER as a quality improvement metric. There is also a worrisome potential for malicious up-scoring of individuals by disruptive group members who may abuse RADPEER deliberately with the intent of targeting a reviewed peer. In these ways, the most basic design features of RADPEER, its nonpunitive nature and anonymity in scoring, may be undermined and its core purpose thwarted. It is important to keep in mind that RADPEER is intended only as a quality improvement tool, and not as a means for “policing” a practice, and it is certainly not appropriate as a management tool for “weeding out” those members whose performance is deemed substandard.
Abujudeh et al/RADPEER Peer Review 5
of the program, we view it as an opportunity for improvement. As part of the quality assurance cycle, users should strive to decrease the variation of level assignment within their reader groups. A recent paper [3] published in 2011 recognizes this weakness in RADPEER and attempts to remedy and improve the system by giving multiple examples to decrease variability. In addition the authors provided 11 specific suggestions to optimize the performance of RADPEER and suggested opportunities for future improvement of the program (Table 4).
Table 4. Eleven suggestions to optimize the performance of RADPEER and opportunities for future improvement of the program 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
Create a culture of acceptance Define categories of discrepancy on a local basis Define an appropriate level of case submission Define which cases will be reviewed Decide if outside radiologists will be reviewed Decide if all modalities will be reviewed Ensure a broad sampling of modalities Establish an appeals process Ensure notification of correct interpretation Consider incentives to participate Encourage the use of RADPEER in practice quality improvement projects
SUMMARY OPINION
Source: Larson et al [3].
The value of the RADPEER program is directly proportional to the quality of information that is entered. If concerns regarding potential consequences of the scores given are allowed to distort the data, the value of the tool is diminished. The ACR allows considerable flexibility in its implementation, which may lead to suboptimal implementation of the program. Some of the benefits of this flexibility include the potential for this program in being inclusive of most types of imaging practices and also allowing participating radiologists and groups to further enhance the program according to their needs. One of the expressed concerns about RADPEER, and also an opportunity for improvement in the system, is the discrepancies that exist in score assignment between readers with regard to the level of disagreement. For example, should a discrepancy be scored 2a or 2b? Many practices use oversight committees in an attempt to standardize those scores with independent review, but this is time-consuming, and uniformity of results is not ensured. Although this will continue to be a limitation
On the basis of the RADPEER survey data of September 2012, there is no consensus among users as to whether their participation in RADPEER has resulted in changes or improvements to their groups’ practice patterns. Forty-seven percent of respondents answered that their practice patters had not changed, 33.1% were unsure, and only 19.8% answered that their practice patterns had changed (Table 5). Of the 132 respondents who commented on the usefulness of peer review, about half (52.3%) noted improvements in their radiologic interpretations as a result of increased awareness of quality oversight. Regarding the question of what users like and dislike about RADPEER, of the 1,247 respondents who commented on this question, a significant fraction of RADPEER users (18%) indicated that they valued the ease of usefulness of the eRADPEER web interface. This was a much larger group than the respondents who criticized RADPEER as being not well integrated into the daily workflow or being difficult to use (2.4%). Criticisms of the peer review system included concerns about the reviews’ being ineffective or irrelevant (12.5%), and a small percentage felt that the peer review process was overly time-consuming (7%). There were
Table 5. Summary opinions on peer review Have practice patterns changed with peer review? Yes No Don’t know Analysis of free-text response Improvement in reads/QA awareness Education Catching misses What do people like or dislike about peer review? Analysis of free-text responses Likes Ease of use Dislikes Time consuming Ineffective/irrelevant Not integrated Not random or anonymous Legal issues
Response Percentage
Numerator/Denominator
19.8 47.0 33.1
246/1,240 583/1,240 411/1,240
52.3 8.3 6.8
69/132 11/132 9/132
18.0
224/1,247
7.1 12.5 2.4 2.4 0.7
88/1,247 156/1,247 30/1,247 30/1,247 9/1,247
6 Journal of the American College of Radiology/Vol.
-
No.
-
Month 2014
also concerns expressed among a small number of respondents regarding insufficient randomization and anonymity of the peer review process using RADPEER (2.4%), and a minority of responders cited concerns over legal issues (0.7%). Recent discussions at national quality meetings have centered on whether RADPEER, which is designed to measure a statistical sampling of error, should be used in parallel with a more traditional morbidity and mortality conference, in which all errors discovered in a practice are reviewed in an all-inclusive, open-discussion forum in the hope of the entire group’s being able to learn from any and all errors that are identified, whether through RADPEER or by other means [6]. The committee members suggested adding additional assessments to the existing RADPEER scores. These included a question related to whether a comparison was made with a prior study when available, when a recommendation was made, whether the new reader agrees with the recommendation, and whether the new reader agrees with the method of communication of the prior examination (ie, when a critical result was handled as a routine communication). These assessments are part of the ACR communication guidelines [6], and measuring compliance potentially could enhance the standard of care. Finally, the committee members suggested adding a choice for making peer review cases as “errors in processes.” The committee felt that RADPEER could be expanded and used as a means to collect data on other aspects of quality and safety in radiology. The committee also felt that there needs to be consideration to balance additional value against the added burden of submitting the data and its implications for workflow. TAKE-HOME POINTS
RADPEER was designed to be a simple, cost-effective process that allows radiologist peer review to be per-
formed during the routine interpretation of current images. The median number of cases reviewed in RADPEER is 776 each year, which translates into 3 to 4 cases reviewed per working day. The RADPEER system is not designed to be a sole OPPE measure, but it can be incorporated into such programs. The committee discourages the use of scores as a means of competency assessment and encourages the maintenance of the nonpunitive nature and anonymity in scoring. RADPEER could be expanded and used as a means to collect data on other aspects of quality and safety in radiology. ACKNOWLEDGMENTS The authors thank Fern Jackson, Mythreyi Chatfield, and Pamela Wilcox of the ACR’s Department of Quality and Safety for their review and commentary on this paper and for their ongoing work to maintain and improve the RADPEER program. REFERENCES 1. Borgstede J, Lewis R, Bhargavan M, Sunshine J. RADPEER quality assurance program: a multifacility study of interpretive disagreement rates. J Am Coll Radiol 2004;1:59-65. 2. Jackson V, Cushing T, Abujudeh H, et al. RADPEER scoring white paper. J Am Coll Radiol 2009;6:21-5. 3. Larson P, Pyatt R, Grimes C, Abujudeh H, Chin K, Roth C. Getting the most out of RADPEER. J Am Coll Radiol 2011;8:543-8. 4. Steele J. The role of RADPEER in the Joint Commission ongoing practice performance evaluation. J Am Coll Radiol 2011;8:6-7. 5. Berlin L. To disclose or not to disclose radiologic errors: should “patientfirst” supersede radiologists’ self-interest? Radiology 2013;268:4-7. 6. American College of Radiology. ACR practice guideline for communication of diagnostic imaging findings (Resolution 11). Available at: http:// www.acr.org/w/media/C5D1443C9EA4424AA12477D1AD1D927D. pdf. Accessed November 11, 2013.
RADPEER is a product developed by the ACR that aims to assist radiologists with quality assessment and improvement through peer review. The program opened in 2002, was initially offered to physician groups in 2003, developed an electronic version in 2005 (eRADPEER), revised the scoring system in 2009, and first surveyed the RADPEER membership in 2010. In 2012, a survey was sent to 16,000 ACR member radiologists, both users and nonusers of RADPEER, with the goal of understanding how to make RADPEER more relevant to its members. A total of 31 questions were used, some of which were repeated from the 2010 survey. The ACR’s RADPEER committee has published 3 papers on the program since its inception. In this report, the authors summarize the survey results and suggest future opportunities for making RADPEER more useful to its membership. Key Words: RADPEER, peer review, OPPE, quality and safety, PQI J Am Coll Radiol 2014;-:---. Copyright © 2014 American College of Radiology
INTRODUCTION
RADPEER was designed to be a simple, cost-effective process that allows radiologist peer review to be performed during the routine interpretation of current images [1]. If prior images and reports are available at the time a new study is being interpreted, these prior studies and the accuracy of their interpretation would typically be evaluated at the time the radiologist interprets the current study. RADPEER was first offered to physician groups in 2003, and in January 2013 it celebrated its 10th anniversary. The original RADPEER scoring system had 4 possible scores: 1 ¼ concur with interpretation; 2 ¼ difficult
a
Massachusetts General Hospital, Boston, Massachusetts.
b
Chambersburg Imaging Associates, Chambersburg, Pennsylvania.
c
Penn State Milton Hershey Medical Center, Hershey, Pennsylvania. d Indiana Regional Imaging, Indiana, Pennsylvania. e
University of Rochester, Rochester, New York.
f
Duke University, Durham, North Carolina.
g
Hennepin Health System, Minneapolis, Minnesota.
h
Abington Health, Lansdale Hospital, Lansdale, Pennsylvania.
i
University of California San Diego Medical Center, San Diego, California.
j
Virginia Mason Medical Center, Seattle, Washington.
Corresponding author and reprints: Hani Abujudeh, MD, MBA, Massachusetts General Hospital, Founders Building 213 D, 55 Fruit Street, Boston, MA 02114; e-mail: [email protected]. ª 2014 American College of Radiology 1546-1440/14/$36.00 http://dx.doi.org/10.1016/j.jacr.2014.02.004
diagnosis, not ordinarily expected to be made; 3 ¼ diagnosis should be made most of the time; and 4 ¼ diagnosis should be made almost every time, misinterpretation of findings. In 2009, the second RADPEER committee white paper was published [2], and the scoring system was revised to include optional categories of (a) unlikely to be clinically significant and (b) likely to be clinically significant, for scores of 2, 3, and 4. The 2009 paper gave examples of scoring. Also in 2009, RADPEER, as a practice quality improvement (PQI) project, was accepted by the ABR. In 2010, a survey was sent to all RADPEER members in an attempt to understand the motivation of the membership body for using RADPEER. As a result, the committee published a paper titled “Getting the Most Out of RADPEER” [3], which explained the process and offered specific suggestions on how to improve the usefulness of RADPEER. Specific suggestions included improving the culture and acceptance of RADPEER, defining categories of discrepancies, describing which imaging modalities should be reviewed, and developing an appeals process. To better understand the radiology practice use of peer review and RADPEER, a questionnaire was developed and distributed in September 2012, with the goal of understanding how to make the RADPEER program more relevant. This was a web-based survey 1
2 Journal of the American College of Radiology/Vol.
-
No.
-
Month 2014
with a total of 31 questions, which was sent to more than 15,000 recipients. A total of 1,589 responses were received, for a response rate of about 10%. As of midNovember 2013, RADPEER had a total of 1,147 groups participating in the program, with a total of 17,037 physicians, with 1,679 of those being RADPEER PQI participants. To date, the median number of cases reviewed in RADPEER is 776 each year, which translates into 3 to 4 cases reviewed per working day. We believe that this number of cases may reasonably be achieved by most practicing radiologists in a peer review process. RESULTS Characteristics of Respondents
The 2012 survey participants came from a variety of practice settings, including 46.3% from hospital-based private practices, 23.2% from academic practices, 19.3% from outpatient-based private practices, and 17.7% from hospital employee practices. More than three-fifths (61.4%) of practices used RADPEER, with 4.9% of responding radiologists unsure if their practices participated. Of practices that participated in RADPEER, 85.2% had all radiologists in the practices participate in RADPEER. The vast majority of radiologists who responded to the survey had been using RADPEER for more than 3 years (73.8%). The survey demonstrated that radiologists are using multiple IT platforms and vendors to participate in RADPEER including PowerScribe (7.6%), PeerVue (4.8%), and Primordial (4.7%). More than a quarter (26.4%) of responders use other systems to record RADPEER findings, of whom 44% have home-grown systems. Why Facilities Do Peer Review
Radiologists have several data-reporting requirements to entities outside and inside their hospitals and practices. To meet CMS requirements for reimbursement policies, radiologists often seek the accreditation services offered by the ACR. This is reflected in the increase in
RADPEER participation seen in 2007 and 2008, soon after the incorporation of peer review into the ACR accreditation process. RADPEER can satisfy both hospital and Joint Commission credentialing requirements. Part of the accreditation process includes a testament from a department about the performance of a radiologist. The RADPEER survey asked, “Does your hospital require peer review data for re-credentialing?” and 43.8% responded “yes” (Table 1). In those situations, the radiologists must perform peer review to be credentialed and recredentialed by the hospital to be able practice and continue to practice within the hospital. Radiologists are also involved in the hospital credentialing process by The Joint Commission, which requires an ongoing physician performance evaluation (OPPE) and a focused physician performance evaluation [4]. Peer review is one way to perform both OPPE and focused physician performance evaluation. Radiologists must keep their board certifications current with the ABR. The ABR now requires board-certified physicians to participate in its Maintenance of Certification program, and the ACR’s RADPEER program is an accepted PQI project. When participants were asked, “Why does your practice perform peer review?” the two most common responses referred either to the ACR requirements or to the hospital requirements, at 79.9% and 69.6%, respectively (Table 1). The PQI requirements accounted for (29.5%), and other entities’ requirements, such as those of the US Department of Veterans Affairs, and the desire to improve patient care quality via selfmonitoring and education accounted for (13.2%). Enforcement of and Satisfaction With Peer Review
The majority of practices have set peer review targets (71.8%) (Table 2). These targets for the majority of respondents involve absolute numbers of cases rather than percentages of cases. Of those physicians who responded, 50% prefer an absolute number of cases to report. In terms of the time period of review, respondents demonstrated a
Table 1. Why facilities do peer review Hospital requirement Yes No Don’t know Reasons stated for use of peer review To meet accreditation requirements, such as those of the ACR It satisfies hospital recredentialing requirements, such as those of The Joint Commission It is a requirement for other reasons For MOC PQI project For other reasons (not a requirement) Themes observed in free-text responses Quality improvement in patient care Education Required Note: MOC ¼ Maintenance of Certification; PQI ¼ practice quality improvement.
Response Percentage
Numerator/Denominator
43.8 23.7 32.5
615/1,405 333/1,405 457/1,405
79.9 69.6
1122/1,405 978/1,405
16.2 29.5 13.2
228/1,405 415/1,405 186/1,405
78.4 5.7 11.9
138/176 10/176 21/176
Abujudeh et al/RADPEER Peer Review 3
Table 2. Enforcement of and satisfaction with peer review Response Numerator/ Percentage Denominator Are targets used? Yes No Not available
71.8 23.8 4.3
Response Percentage
1,009/1,405 335/1,405 61/1,405 Actual
Type of targets Number Percent No opinion Period Per day Per month Per quarter Per year No opinion
Numerator/ Denominator
Desired
80.6 19.4
662/821 159/821
49.7 29.3 21.1
375/755 221/755 159/755
19.8 44.0 21.7 14.6
160/810 356/810 176/810 118/810
18.0 39.6 18.6 15.8 8.0
116/646 256/646 120/646 102/646 52/646
moderate preference for monthly reviews, with daily, quarterly, and yearly reviews fairly equally represented. Monthly review intervals (44%) are moderately preferred over other intervals, such as quarterly (21.7%). Concerns and Challenges
Radiologists have numerous concerns when it comes to peer review processes. Concerns such as the potential discoverability of the data, the awkwardness of being graded, the possible ramifications of institutional politics with possible misuse of the data, the validity of the results, potential harm to the reputation of the radiologists involved, the use of time and resources involved, and the possibility of punitive outcomes are among many other concerns. When asked, “Do you feel there is an under reporting of significant disagreements in the peer review process at your practice?” 33% of respondents said “yes” (Table 3). This may be directly related to the concerns of discoverability, at 65.4%. The survey results suggest that anonymity may improve reporting of disagreements, at 43%. This stands in contrast to the survey results suggesting that only 38.7% of peer review processes are anonymous. The RADPEER report contains the identities of the user group, the two radiologists, and the modality type, as well as the score. It does not contain information about the patient or the specific examination. Information sent to the ACR is protected under the Virginia’s medical malpractice law, one of the strongest peer review statutes in the country. Radiology groups should consult lawyers familiar with local statues regarding peer review data. In most, but regrettably not all, states, the peer review process is considered protected and therefore not discoverable. As noted in Table 3, discoverability in the medicolegal sense of the word was noted as a concern by 65% of respondents. Unfortunately, in several very visible and litigious states, the process is not protected, which poses a seemingly insurmountable hurdle to practices seeking to engage in peer review.
Regrettably, nearly half of respondents still feel that the process of peer review remains too time consuming, and nearly a third struggle to see quality improvement as a legitimate outcome of the current systems of peer review. More than half of respondents thought that better integration with their PACS or voice recognition daily workflow would improve compliance with peer review in their practices, presumably correlating with the response of “too time consuming.” Other concerns included the notion of “being graded” by one’s peers, awkwardness of grading one’s senior (eg, medical director), capturing of peer review information by credentialing committees in a hospital (with the potential for decredentialing of radiologists), and public embarrassment. Although these concerns may be legitimate, there are ways to overcome most of them. By anonymizing the process, the “public grading” and embarrassment issues can be obviated. As seen in Table 3, 43% of respondents think that anonymizing the process would improve reporting of disagreements, or more important, only 27% did not think it would help. This finding is interesting in the face of only 38.7% of respondents’ reporting that their peer review systems are currently anonymous. Despite the potential concern for “grading” (or being graded by) a fellow radiologist, the committee feels that identifying important errors can have a significant effect on patient care and requires participants to act. When a significant error is identified, it becomes incumbent upon the radiologists to take corrective action. In particular, interpretations may require report addenda; communications with clinicians often are needed, and in many cases, discussions with patients may be appropriate. As a result, the issue of litigation remains a legitimate concern. However, not addressing the errors is not in patients’ best interests. Moreover, as has been noted in the literature, transparency and disclosure are far more likely to prevent litigation than to precipitate it [5].
4 Journal of the American College of Radiology/Vol.
-
No.
-
Month 2014
Table 3. Concerns regarding peer review reporting Concerns reported Discoverability (medicolegal) of data Being graded Awkwardness of grading medical director or supervising physician Information being used for decredentialing Public embarrassment Other Are disagreements underreported? Yes No Don’t know Would anonymity help improve reporting of disagreements Yes No Don’t know Is your peer review system anonymous? Yes No Don’t know Best reason not to do peer review Too time consuming Do not see it as helpful to quality improvement Other Analysis of free-text responses No good reason not to Discovery/legal concerns Misuse of data Waste of time/too time consuming No relevant data or feedback Would better integration with vendor’s system encourage peer review? Yes No Don’t know
In addition to anonymizing the process, respondents generally felt that improvements in the integration of peer review into the PACS would be helpful and would encourage the use of peer review. In some cases, voice dictation and PACS “prioritizing” and presenting examinations for peer review that were performed recently might improve utilization because finding a clinically significant missed finding very recently after an imaging procedure could be much easier to reconcile for a radiologist or practice than older findings. RADPEER should not be the sole tool for quality improvement in a practice setting but rather should be one of several assets in a comprehensive quality improvement program. Although the RADPEER system is not designed to be used as part of an OPPE, it can be incorporated into such programs, for example, by facilitating the tracking of physician participation in peer review as a monitored performance measure. We would strongly caution against designing an OPPE regime that incorporates the actual RADPEER scores (eg, 2b, 3a) or fractions of scores (eg, percentages of scores of 3 and 4) as a means of competency assessment however. Such a plan, whether based on a maximum average RADPEER score or some absolute threshold RADPEER score, when used to evaluate
Response Percentage
Numerator/Denominator
65.4 49.4 29.5 35.2 29.9 20.1
833/1,273 629/1,273 376/1,273 448/1,273 380/1,273 256/1,273
33.5 45.3 21.2
426/1,273 577/1,273 270/1,273
43.0 27.1 29.9
547/1,273 345/1,273 381/1,273
38.7 51.1 10.2
194/501 256/501 51/501
47.7 32.3 32.0
607/1,273 411/1,273 407/1,273
45.8 7.6 6.5 6.3 15.4
175/382 29/382 25/382 24/382 59/382
52.3 20.0 27.7
652/1,247 250/1,247 345/1,247
individual physician performance for maintenance of clinical privileges, is likely to produce the undesired consequence of undermining the quality improvement goals of peer review. Although this has indeed been done at some institutions, there is a high risk for undermining users’ trust in the “just culture” element needed to make RADPEER optimally effective. This is because when peer reviewers are aware that the actual scores they assign may have potential adverse consequences for those reviewed, this knowledge would be anticipated to result in inappropriate downward pressure on their scoring, thus undermining the effectiveness of RADPEER as a quality improvement metric. There is also a worrisome potential for malicious up-scoring of individuals by disruptive group members who may abuse RADPEER deliberately with the intent of targeting a reviewed peer. In these ways, the most basic design features of RADPEER, its nonpunitive nature and anonymity in scoring, may be undermined and its core purpose thwarted. It is important to keep in mind that RADPEER is intended only as a quality improvement tool, and not as a means for “policing” a practice, and it is certainly not appropriate as a management tool for “weeding out” those members whose performance is deemed substandard.
Abujudeh et al/RADPEER Peer Review 5
of the program, we view it as an opportunity for improvement. As part of the quality assurance cycle, users should strive to decrease the variation of level assignment within their reader groups. A recent paper [3] published in 2011 recognizes this weakness in RADPEER and attempts to remedy and improve the system by giving multiple examples to decrease variability. In addition the authors provided 11 specific suggestions to optimize the performance of RADPEER and suggested opportunities for future improvement of the program (Table 4).
Table 4. Eleven suggestions to optimize the performance of RADPEER and opportunities for future improvement of the program 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.
Create a culture of acceptance Define categories of discrepancy on a local basis Define an appropriate level of case submission Define which cases will be reviewed Decide if outside radiologists will be reviewed Decide if all modalities will be reviewed Ensure a broad sampling of modalities Establish an appeals process Ensure notification of correct interpretation Consider incentives to participate Encourage the use of RADPEER in practice quality improvement projects
SUMMARY OPINION
Source: Larson et al [3].
The value of the RADPEER program is directly proportional to the quality of information that is entered. If concerns regarding potential consequences of the scores given are allowed to distort the data, the value of the tool is diminished. The ACR allows considerable flexibility in its implementation, which may lead to suboptimal implementation of the program. Some of the benefits of this flexibility include the potential for this program in being inclusive of most types of imaging practices and also allowing participating radiologists and groups to further enhance the program according to their needs. One of the expressed concerns about RADPEER, and also an opportunity for improvement in the system, is the discrepancies that exist in score assignment between readers with regard to the level of disagreement. For example, should a discrepancy be scored 2a or 2b? Many practices use oversight committees in an attempt to standardize those scores with independent review, but this is time-consuming, and uniformity of results is not ensured. Although this will continue to be a limitation
On the basis of the RADPEER survey data of September 2012, there is no consensus among users as to whether their participation in RADPEER has resulted in changes or improvements to their groups’ practice patterns. Forty-seven percent of respondents answered that their practice patters had not changed, 33.1% were unsure, and only 19.8% answered that their practice patterns had changed (Table 5). Of the 132 respondents who commented on the usefulness of peer review, about half (52.3%) noted improvements in their radiologic interpretations as a result of increased awareness of quality oversight. Regarding the question of what users like and dislike about RADPEER, of the 1,247 respondents who commented on this question, a significant fraction of RADPEER users (18%) indicated that they valued the ease of usefulness of the eRADPEER web interface. This was a much larger group than the respondents who criticized RADPEER as being not well integrated into the daily workflow or being difficult to use (2.4%). Criticisms of the peer review system included concerns about the reviews’ being ineffective or irrelevant (12.5%), and a small percentage felt that the peer review process was overly time-consuming (7%). There were
Table 5. Summary opinions on peer review Have practice patterns changed with peer review? Yes No Don’t know Analysis of free-text response Improvement in reads/QA awareness Education Catching misses What do people like or dislike about peer review? Analysis of free-text responses Likes Ease of use Dislikes Time consuming Ineffective/irrelevant Not integrated Not random or anonymous Legal issues
Response Percentage
Numerator/Denominator
19.8 47.0 33.1
246/1,240 583/1,240 411/1,240
52.3 8.3 6.8
69/132 11/132 9/132
18.0
224/1,247
7.1 12.5 2.4 2.4 0.7
88/1,247 156/1,247 30/1,247 30/1,247 9/1,247
6 Journal of the American College of Radiology/Vol.
-
No.
-
Month 2014
also concerns expressed among a small number of respondents regarding insufficient randomization and anonymity of the peer review process using RADPEER (2.4%), and a minority of responders cited concerns over legal issues (0.7%). Recent discussions at national quality meetings have centered on whether RADPEER, which is designed to measure a statistical sampling of error, should be used in parallel with a more traditional morbidity and mortality conference, in which all errors discovered in a practice are reviewed in an all-inclusive, open-discussion forum in the hope of the entire group’s being able to learn from any and all errors that are identified, whether through RADPEER or by other means [6]. The committee members suggested adding additional assessments to the existing RADPEER scores. These included a question related to whether a comparison was made with a prior study when available, when a recommendation was made, whether the new reader agrees with the recommendation, and whether the new reader agrees with the method of communication of the prior examination (ie, when a critical result was handled as a routine communication). These assessments are part of the ACR communication guidelines [6], and measuring compliance potentially could enhance the standard of care. Finally, the committee members suggested adding a choice for making peer review cases as “errors in processes.” The committee felt that RADPEER could be expanded and used as a means to collect data on other aspects of quality and safety in radiology. The committee also felt that there needs to be consideration to balance additional value against the added burden of submitting the data and its implications for workflow. TAKE-HOME POINTS
RADPEER was designed to be a simple, cost-effective process that allows radiologist peer review to be per-
formed during the routine interpretation of current images. The median number of cases reviewed in RADPEER is 776 each year, which translates into 3 to 4 cases reviewed per working day. The RADPEER system is not designed to be a sole OPPE measure, but it can be incorporated into such programs. The committee discourages the use of scores as a means of competency assessment and encourages the maintenance of the nonpunitive nature and anonymity in scoring. RADPEER could be expanded and used as a means to collect data on other aspects of quality and safety in radiology. ACKNOWLEDGMENTS The authors thank Fern Jackson, Mythreyi Chatfield, and Pamela Wilcox of the ACR’s Department of Quality and Safety for their review and commentary on this paper and for their ongoing work to maintain and improve the RADPEER program. REFERENCES 1. Borgstede J, Lewis R, Bhargavan M, Sunshine J. RADPEER quality assurance program: a multifacility study of interpretive disagreement rates. J Am Coll Radiol 2004;1:59-65. 2. Jackson V, Cushing T, Abujudeh H, et al. RADPEER scoring white paper. J Am Coll Radiol 2009;6:21-5. 3. Larson P, Pyatt R, Grimes C, Abujudeh H, Chin K, Roth C. Getting the most out of RADPEER. J Am Coll Radiol 2011;8:543-8. 4. Steele J. The role of RADPEER in the Joint Commission ongoing practice performance evaluation. J Am Coll Radiol 2011;8:6-7. 5. Berlin L. To disclose or not to disclose radiologic errors: should “patientfirst” supersede radiologists’ self-interest? Radiology 2013;268:4-7. 6. American College of Radiology. ACR practice guideline for communication of diagnostic imaging findings (Resolution 11). Available at: http:// www.acr.org/w/media/C5D1443C9EA4424AA12477D1AD1D927D. pdf. Accessed November 11, 2013.