- Email: [email protected]
Patient Compliance With Electronic Patient Reported Outcomes Following Shoulder Arthroscopy
Patient Compliance With Electronic Patient Reported Outcomes Following Shoulder Arthroscopy Eric C. Makhni, M.D., M.B.A., John D. Higgins, B.A., Jason T. Hamamoto, B.S., Brian J. Cole, M.D., M.B.A., Anthony A. Romeo, M.D., and Nikhil N. Verma, M.D.
Purpose: To determine the patient compliance in completing electronically administered patient-reported outcome (PRO) scores following shoulder arthroscopy, and to determine if dedicated research assistants improve patient compliance. Methods: Patients undergoing arthroscopic shoulder surgery from January 1, 2014, to December 31, 2014, were prospectively enrolled into an electronic data collection system with retrospective review of compliance data. A total of 143 patients were included in this study; 406 patients were excluded (for any or all of the following reasons, such as incomplete follow-up, inaccessibility to the order sets, and inability to complete the order sets). All patients were assigned an order set of PROs through an electronic reporting system, with order sets to be completed prior to surgery, as well as 6 and 12 months postoperatively. Compliance rates of form completion were documented. Patients who underwent arthroscopic anterior and/or posterior stabilization were excluded. Results: The average age of the patients was 53.1 years, ranging from 20 to 83. Compliance of form completion was highest preoperatively (76%), and then dropped subsequently at 6 months postoperatively (57%) and 12 months postoperatively (45%). Use of research assistants improved compliance by approximately 20% at each time point. No differences were found according to patient gender and age group. Of those completing forms, a majority completed forms at home or elsewhere prior to returning to the office for the clinic visit. Conclusions: Electronic administration of PRO may decrease the amount of time required in the office setting for PRO completion by patients. This may be mutually beneficial to providers and patients. It is unclear if an electronic system improves patient compliance in voluntary completion PRO. Compliance rates at final follow-up remain a concern if data are to be used for establishing quality or outcome metrics. Level of Evidence: Level IV, case series.
I
From the Department of Orthopedics, Henry Ford Health System (E.C.M.), Detroit, Michigan; Department of Sports Medicine, Rush University Medical Center (J.D.H., J.T.H., B.J.C., A.A.R., N.N.V.), Chicago, Illinois, U.S.A. The authors report the following potential conflicts of interest or sources of funding: J.T.H. has stock/stock options in Novartis and Nuvasive. B.J.C. receives consulting fees from Arthrex, Regentis, and Zimmer; payment for lectures including service on speakers bureaus from Arthrex; royalties from Arthrex, SLACK, DJ Orthopaedics, Elsevier Publishing, and Saunders/ Mosby-Elsevier; has stock/stock options in Carticept and Regentis; other from Tornier, Ossur, Medipost, Cytori, and Aesculap/Braun. A.A.R. receives consulting fees from Arthrex; payment for lectures including service on speakers bureaus from Arthrex; royalties from Arthrex and SLACK; other from DJO Surgical, Ossur, Saunders/Mosby-Elsevier, and Smith & Nephew. N.N.V. receives consulting fees from Orthospace and Smith & Nephew; royalties from Arthroscopy, Smith & Nephew, Vindico Medical-Orthopedics Hyperguide; Stock/stock options in Cymedica and Omeros; other from Ossur, Minivasive, DJ orthopaedics, and Arthrosurface. Full ICMJE author disclosure forms are available for this article online, as supplementary material. Received October 13, 2016; accepted June 2, 2017. Address correspondence to Eric C. Makhni, M.D., M.B.A., Department of Orthopedics, Henry Ford Health System, 6777 West Maple Road, West Bloomfield, MI 48322, U.S.A. E-mail: [email protected] Ó 2017 by the Arthroscopy Association of North America 0749-8063/16967/$36.00 http://dx.doi.org/10.1016/j.arthro.2017.06.016
t is estimated that physicians in the United States spend greater than 15 hours per week reporting quality measures, totaling 785 hours per physician, and an alarming $15.4 billion of spending1 per year. The shift from volume-based to value-based care has mandated an increased effort on reporting outcomes,1 and in particular, patient-reported outcomes (PROs). The incentive to make the reporting process more efficient for both patient and physician, therefore, is high. Unfortunately, significant hurdles exist when considering implementation of standardized outcome reporting guidelines and mandates. These include logistical and financial constraints incurred by the practitioner, as well as additional time and energy required by patients. Moreover, there has been a lack of consistency with regard to selection and administration of common PRO for disease-specific orthopaedic conditions.2,3 Traditionally, outcomes metrics have been administered with paper and pen to patients. Such administration has the advantage of being convenient to the patient, in that it is relatively cheap and can be done in
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol
-,
No
-
(Month), 2017: pp 1-7
1
2
E. C. MAKHNI ET AL.
any type of clinical setting. However, paper outcome reporting is difficult to administer remotely, therefore requiring the patient to physically be with the researcher or clinician. These forms must also be cleaned, scanned into a medical record or reporting system, and ultimately scored, thereby limiting seamless patient searching and “databasing.” Recently, electronic patient-reported outcomes (ePROM) have become more readily available to clinicians and researchers.4 This form of administration has many advantages over traditional paper-and-pen methods, including the ability to readily administer forms remotely (thereby allowing patients to complete forms at home prior to their visit), as well as improved sorting and storage of completed forms. Moreover, such a system may foster improved ability to conduct multicenter trials, as well as incorporate new computer adaptive testing methods.4-6 However, little data exist regarding the actual benefit of electronic forms, and whether or not patients readily complete these forms prior to their clinic visits (and thus decompressing the paperwork burden in the office). Electronic outcome reporting has shown potential for success in fields outside of orthopaedics.7,8 The purpose of this study was to determine patient compliance in completing electronically administered PRO scores following shoulder arthroscopy, and to determine if dedicated research assistants improve patient compliance. We hypothesized that the compliance rate will generally be below acceptable limits for publication of outcome studies using prospectively collected data.
Methods Institutional review board approval was obtained from our institution in order to retrieve outcome reporting data from patients presenting to our shoulder and sports medicine clinic. All patients who underwent arthroscopic shoulder surgery (including rotator cuff repair, subacromial decompression/acromioplasty, biceps tenotomy/tenodesis, distal clavicle excision, acromioclavicular joint reconstruction, or glenohumeral debridement) from January 1, 2014, to December 31, 2014, by 2 of the senior authors (B.J.C. and N.N.V.) were retrospectively screened for study inclusion. Survey protocols were not instituted for patients with shoulder stabilization at the time of study commencement; therefore these patients were not included in the study. Patients were excluded if they were younger than 18 years of age, did not understand the English language, or did not electronically receive outcome measures at all 3 time points (e.g., were enrolled into the PRO electronic registry postoperatively and therefore without preoperative data). In our practice, outcomes reporting transitioned to an electronic system in the middle of 2013. Therefore, enrollment into the study began in 2014 following
successful initiation of the electronic system. Piloting and iterations of the registry/order set components was continued after initiation of the registry, and therefore not all patients operated on during this time period were included in the study. Upon indication for surgery, each patient was enrolled into the electronic outcome reporting system (Outcomes Based Electronic Research Database [OBERD], Columbia, MO) and assigned an order set of outcome measures. Patients who underwent rotator cuff repair were assigned a Rotator Cuff order set, whereas those who underwent subacromial decompression/acromioplasty, biceps tenotomy/tenodesis, distal clavicle excision, acromioclavicular joint reconstruction, or glenohumeral debridement were assigned the Miscellaneous order set. Components of each order set can be found in Table 1. Patients were assigned to complete the order set prior to surgery, at 6 months postoperation, and at 12 months postoperation. The OBERD outcomes system was designed to provide improved, secure accessibility for patients and providers to clinical data forms. Users are assigned a user name and password, which is used for site entry. Personal security questions follow the login page for additional security. Patients are guided to their personal file, which shows previously completed surveys and current outstanding surveys. They may access the registry either through a traditional desktop web version or through a mobile version on their phones. Upon initiating a survey, questions are presented in a large font with answers that are outlined in colors corresponding with their connotation (red meaning poor, green meaning positive). Patients have the option of skipping questions, submitting their survey at any point or returning to previous questions. Once a survey is complete, patients are prompted to start the next survey or exit to the main menu if they have completed all assigned surveys. Patients enrolled in the electronic system received electronic reminders to their personal e-mail prior to their postoperative deadlines. These reminders are sent
Table 1. Order Set Components Order Set Rotator Cuff
Miscellaneous
Components ASESdStandardized Shoulder Assessment Form Constant-Murley Score Single Assessment Numeric Evaluation (SANE) score Western Ontario Rotator Cuff Index (WORC) ASESdShoulder Assessment Form Constant-Murley Score Single Assessment Numeric Evaluation (SANE) score
ASES, American Shoulder and Elbow Surgeons.
3
PATIENT COMPLIANCE WITH ELECTRONIC PROS Table 2. Patient Cohort Demographic Information Females, n (%) Average age, years, mean (SD) Rotator Cuff order set patients, n (%) Miscellaneous order set patients, n (%)
Both MDs 49 (34) 53.1 (12.0) 85 58
MD 1 20 (37) 54.7 (12.8) 41 (48) 13 (22)
MD 2 29 (33) 52.1 (11.5) 44 (52) 45 (78)
MD 1, enrolling surgeon 1; MD 2, enrolling surgeon 2; SD, standard deviation.
every 5 days. Patients have the opportunity to complete their assigned order set 1 month prior to and 1 month after their 6- and 12-month postoperative dates. Reminder e-mails are discontinued once patients have completed their order set or their 2-month completion period had expired. Additional opportunity to complete forms was presented during the office visit, but only if there was sufficient time between patient registration and when the patient was called to the examination room. A patient was considered to be compliant if he or she completed all survey forms in full at the requested time points (preoperative, 6 months postoperatively, or 12 months postoperatively). Those who partially completed their forms at any given time point were considered to be partially compliant, whereas those who failed to open or answer any of the questions in the forms were considered to be noncompliant at that respective time point. Compliance was documented at each of the 3 time points according to order set received (Rotator Cuff vs Miscellaneous), as well as by attending surgeon (B.J.C. vs N.N.V.), patient gender, and patient age (<50, 50-60, and 60 years). Moreover, the proportion of patients who completed their forms at home (or before the clinic visit) was reported compared with the proportion that completed the forms in the office on the day of the visit. Both senior authors (B.J.C. and N.N.V.) had a research assistant who enrolled patients into the respective order sets. Once enrolled into the order set, the patient received automatic reminders to complete their forms. One of the senior authors (B.J.C.) had additional dedicated research assistants who actively sent extra reminders to patients who were noncompliant with outcome reporting and monitored the clinic schedule to identify noncompliant patients. The impact of overall survey compliance related to additional personnel who provided personal reminders was evaluated with the current study. Statistical Analysis In this study, comparisons between categorical data were made using Fisher exact test or chi-square test for groups of 2 or 3 or more comparators, respectively. All comparisons of continuous data were made using analysis of variance using readily available web-based software (www.openepi.com).
Results There were 509 shoulder arthroscopies performed during the study period. A total of 143 patients were included in the study. There were 406 patients who were excluded (for any or all of the following reasons, such as incomplete follow-up, inaccessibility to the order sets, and inability to complete the order sets). There were no statistically significant differences between enrolling surgeons based on age or gender (Table 2). In total, there were 509 shoulder arthroscopies performed during this time. However, only patients who met inclusion criteria were included. Preoperatively, 76% of all 143 patients (n ¼ 109 patients) were compliant in completing their questionnaires. Compliance decreased to 57% (n ¼ 81 patients) and 45% (n ¼ 64 patients) at the 6- and 12-month postoperative time points, respectively (Fig 1; P < .001). When considering order set assigned, there was no statistically significant difference in compliance between patients in the Rotator Cuff versus the Miscellaneous order sets (Fig 2; P ¼ .2 or higher across all time points). With regard to timing of forms completion (Fig 3), all patients who completed preoperative forms completed them at home (or elsewhere prior to coming to the hospital for surgery; 109 of 109 patients). At the 6month postoperative mark, of the 81 patients who completed forms, a majority (85%; n ¼ 69 patients) completed them prior to coming for their office visit. At the 12-month postoperative mark, 55 of 64 patients who completed their forms did so prior to the visit (86%). When considering noncompliant patients (Fig 4), at the preoperative time point, there were 27 (19%) patients who partially completed their surveys whereas 7 (5%) patients did not start their order set surveys. At 6 and 12 months postoperatively, the number of patients who partially completed the surveys was 10 and 14 (7% and 10%), respectively. There were 52 (36%) and 65 (45%) patients who never started their order set surveys at the 6- and 12-month postoperative time points, respectively. There was a statistically significant increase over the course of the follow-up times in the proportion of patients who did not attempt to answer any order set surveys compared with those who partially answered their assigned order set (P < .001).
4
E. C. MAKHNI ET AL. 100% 90% 80%
70% 60% 50% 40% 30% 20% 10% 0%
Pre-Op
6 Month Post-Op
12 Month Post-Op
Fig 1. Compliance rates at the preoperative, 6-month, and 12-month time points.
Patient compliance was also reported according to demographic factors. There was no statistically significant difference found in compliance between male patients and female patients (Fig 5; P > .3). There was also no statistically significant difference in compliance rates found according to patient age (Fig 6; P ¼ .73). Finally, the impact of a dedicated research assistant was analyzed with regard to survey compliance (Fig 7). One of the senior contributing authors (MD 1) did have a research team that called and e-mailed patients who had not completed their surveys despite an automatically generated e-mail system from the outcome collection system. These research assistants also tracked the senior contributing author’s clinic schedule to identify patients who were noncompliant to administer surveys in clinic. The compliance of these patients in completing their surveys was 94% preoperatively (51 of 54 patients), 72% at 6 months postoperative (39 of 54 patients), and 56% at the 12 months
Fig 3. The location where patients completed their assigned survey forms. Patients either completed their survey at home (or elsewhere other than the office/clinic) or in the office/ clinic setting. Home: patients who completed surveys at home (or elsewhere other than the office/clinic); In office/ clinic: patients who completed surveys in the office/clinic setting.
postoperative (30 of 54 patients). In comparison, the second surgeon, who did not have a research assistant providing additional compliance support, had a compliance of 65% preoperatively (58 of 89 patients). At 6 and 12 months postoperatively, the compliance decreased to 46% (41 of 89 patients) and 38% (34 of 89 patients), respectively. The difference between these 2 surgeons was statistically significant (P < .01) over the course of the follow-up period.
Discussion Although compliance decreased with increasing follow-up times, the proportion of patients completing
100% 90% 80% 70% 60% Miscellaneous (n=58)
50%
Rotator Cuff (n=85) 40% 30%
20% 10% 0%
Pre-Op
6 Month Post-Op
12 Month Post-Op
Fig 2. Order set compliance rates at the preoperative, 6-month, and 12-month time points. Miscellaneous: patients who completed the assigned Miscellaneous order set; Rotator Cuff: patients who completed the assigned Rotator Cuff order set.
Fig 4. Patient noncompliance rates. Patients who were noncompliant in completing their surveys were categorized into partially compliant and noncompliant. There was a statistically significant difference in the proportion of patients who did not attempt to answer their assigned order sets compared with those who partially completed their assigned order set. * P < .001. Partial: patients who partially completed their order sets; Unopened: patients who never attempted or opened their assigned order sets.
5
PATIENT COMPLIANCE WITH ELECTRONIC PROS 100%
100%
90%
90%
80%
80%
70%
70%
60%
60%
Female (n=49)
50%
Male (n=94)
40%
30%
20%
20%
10%
10% 0%
Pre-Op
6 Month Post-Op
12 Month Post-Op
Fig 5. Male and female compliance rates. There was no statistically significant difference between male and female compliance rates at any time point. Female: female patients who completed their assigned order set; Male: male patients who completed their assigned order set.
forms prior to coming to the clinic was excellent when a team of research coordinators were tasked with ensuring compliance (94%). Unfortunately, compliance was only 65% when the forms were administered electronically and there was no additional research staff to improve compliance. In addition, compliance at postoperative follow-up, particularly 1-year, remains a significant concern. The results of our study indicate a decrease in compliance from preoperative to 6 months postoperative, and from 6 months to 12 months postoperative. Therefore, despite electronic reminders and the ability to complete forms away from the clinic, there was still a majority (55%) of patients who did not fully complete their surveys 1 year postoperatively. However, 56% of patients at least partially completed their surveys at this time point. These compliance results are lower than those of other studies that have documented similar findings. For example, in a study of hip arthroplasty patients in the United Kingdom, Imam and colleagues reported a PRO compliance rate with written forms of 89% and 83% at the 6- and 12-month postoperative marks.9 This increase in compliance could be due to a greater willingness to complete and return paper forms (as was used in this study), a greater willingness among arthroplasty patients, a greater willingness among patients in a nationalized health care setting, or the administration of the paper form questionnaire immediately prior to the physician evaluation in the office. Electronic outcome reporting has been successfully used in other medical specialties as well.10-14 In our study, we also found that patients were likely to not even begin completing their electronic forms as they approached their 12-month postoperative time point. This suggests that the patient’s interest levels in answering survey questions on an electronic form decreases as they progress farther from surgery.
50-59 years (n=47) ≥60 years (n=48)
40%
30%
0%
>50 years (n=48)
50%
Pre-Op
6 Month Post-Op
12 Month Post-Op
Fig 6. Patient compliance rates by age group. Patients were grouped by age at the date of service. No statistically significant differences were found between patient age groups. >50 years: patients younger than 50 years who completed their assigned order set; 50-59: patients aged 50 to 59 years who completed their assigned order set; 60: patients aged 60 years or older who completed their assigned order set.
Interestingly, a majority of patients who completed surveys did so prior to coming to their office visit. This finding supports the concept that use of electronic outcome reporting may decrease the burden of survey completion in the office. Another important finding in our study is the impact of dedicated research personnel who made additional personal reminders to patients to complete their surveys. These reminders were in addition to automatic reminders generated by the electronic outcome reporting system. Research staff also monitored the surgeon’s clinic schedule to identify noncompliant patients. The research assistant would then have the patient complete their surveys as they waited to be seen by the surgeon. Using this additional support staff and
Fig 7. Compliance rates for each enrolling physician. There was a statistically significant difference between the enrolling physicians at each time point (*P < .001). MD 1: patients who received care from Surgeon 1, who completed their assigned order set; MD 2: patients who received care from Surgeon 2, who completed their assigned order set.
6
E. C. MAKHNI ET AL.
verification of compliance resulted in an increase of approximately 20% at each time point. Outcome reporting appears to be significantly enhanced by dedicated support staff that is actively working with patients to increase compliance. In clinicians with busy outpatient office settings, additional dedicated research staff may be essential in maintaining efficient office clinical care while still ensuring high rates of compliance with outcome reporting. However, the addition of staff results in increased costs to the practice, and currently there is no additional compensation for the implementation of the electronic outcomes system, or the staff required to maintain and monitor its use. This likely represents a major barrier to entry for smaller private practice groups, and significant increased costs to the health care system that are currently not accounted for within payment models. With the mandate that all practices report outcomes to define value-based care, the government has imposed a system that may not effectively provide the information required by the surgeon without substantial additional infrastructure and personnel costs throughout their practice. Our study has demonstrated that despite implementation of an accessible electronic outcomes system, compliance among patients remains low at the 1-year follow-up time frame. Although not included in the current study, one may hypothesize that further decreases would be expected at longer follow-up time periods. As most are aware of from the research arena, the current compliance rates as demonstrated in this study are well below currently accepted norms for drawing any valid conclusions regarding the outcome of a given procedure and below the rate at which peer review publications can reliably occur. However, this important limitation of outcomes collection receives little or no attention in the current discussion regarding outcomes-based payments and the realities of implementing such a process. In moving forward, physicians should insist that these value-based payment schemes include funding for implementation and monitoring of outcomes collection as well as requirements for patient participation, with penalties for noncompliance such as increased deductibles or copayments. Lastly the currently used PROs have no ability to account for patient compliance or comorbidities in determining outcome, another limitation that has yet to be addressed. Limitations This study is not without limitations. Primarily, the compliance rates of patients receiving paper forms was not studied, and therefore no control group exists for our study population. Although these data (as well as data regarding time required for survey completion) would have been helpful to use as a comparison, they
were not available to the study team, as all patients transitioned to electronic reporting prior to the study collection period. However, we do not believe that paper-based collection is a realistic solution for largescale outcomes collection as would be required for value-based payment models. Additionally, the impact of multiple PROs administered on overall compliance was not documented, and therefore the impact of “survey fatigue” is not known. Second, the collection of PROs used in our order sets may be different from those collected by other practitioners. Order sets with fewer components may result in higher compliance rates, but this could not be determined from the current study. No current standardization exists for outcome sets for a given orthopaedic disease state. Our study population was limited to those undergoing shoulder arthroscopy. Additionally, the automatically generated electronic reminders to patients from the outcome reporting system may have been ineffective in increasing survey compliance. Another limitation is that definitive information regarding weekly and hourly efforts by research assistants was not available. As this was a retrospective study, the goal was to provide descriptive information regarding PRO compliance in our practice, which employs providers who are dedicated to collecting this information. Therefore, we are unable to provide quantitative data regarding number of hours spent by each member of the research team. It is estimated, however, that approximately 10% to 20% of the assistants’ efforts would go toward outcome administration and collection on any given week during the study period. Finally, determining reasons for noncompliance would be important, including a determination of changing contact information, the role of worker’s compensation or third-party liability, age, the impact of the clinical outcome at the time of evaluation, and other variables that are yet to be elucidated.
Conclusions Electronic administration of PRO may decrease the amount of time required in the office setting for PRO completion by patients. This may be mutually beneficial to providers and patients. It is unclear if an electronic system improves patient compliance in voluntary completion PRO. Compliance rates at final follow-up remain a concern if data are to be used for establishing quality or outcome metrics.
References 1. Bryan S, Davis J, Broesch J, et al. Choosing your partner for the PROM: A review of evidence on patient-reported outcome measures for use in primary and community care. Healthc Policy 2014;10:38-51. 2. Makhni EC, Padaki AS, Petridis PD, et al. High variability in outcome reporting patterns in high-impact ACL literature. J Bone Joint Surg Am 2015;97:1529-1542.
PATIENT COMPLIANCE WITH ELECTRONIC PROS 3. Makhni EC, Steinhaus ME, Morrow ZS, et al. Outcomes assessment in rotator cuff pathology: What are we measuring? J Shoulder Elbow Surg 2015;24:2008-2015. 4. Okike K, Lee OC, Makanji H, Morgan JH, Harris MB, Vrahas MS. Comparison of locked plate fixation and nonoperative management for displaced proximal humerus fractures in elderly patients. Am J Orthop (Belle Mead NJ) 2015;44:E106-E112. 5. Beckmann JT, Hung M, Bounsanga J, Wylie JD, Granger EK, Tashjian RZ. Psychometric evaluation of the PROMIS Physical Function Computerized Adaptive Test in comparison to the American Shoulder and Elbow Surgeons score and Simple Shoulder Test in patients with rotator cuff disease. J Shoulder Elbow Surg 2015;24: 1961-1967. 6. Overbeek CL, Nota SP, Jayakumar P, Hageman MG, Ring D. The PROMIS physical function correlates with the QuickDASH in patients with upper extremity illness. Clin Orthop Relat Res 2015;473:311-317. 7. Nippita S, Oviedo JD, Velasco MG, Westhoff CL, Davis AR, Castano PM. A randomized controlled trial of daily text messages versus monthly paper diaries to collect bleeding data after intrauterine device insertion. Contraception 2015;92:578-584. 8. Newell D, Diment E, Bolton JE. An electronic patientreported outcome measures system in UK chiropractic practices: A feasibility study of routine collection of outcomes and costs. J Manipulative Physiol Ther 2016;39:31-41.
7
9. Imam MA, Barke S, Stafford GH, Parkin D, Field RE. Loss to follow-up after total hip replacement: A source of bias in patient reported outcome measures and registry datasets? Hip Int 2014;24:465-472. 10. Cowan RA, Suidan RS, Andikyan V, et al. Electronic patient-reported outcomes from home in patients recovering from major gynecologic cancer surgery: A prospective study measuring symptoms and health-related quality of life. Gynecol Oncol 2016;143:362-366. 11. Jensen RE, Rothrock NE, DeWitt EM, et al. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care 2015;53:153-159. 12. Jensen RE, Snyder CF, Basch E, Frank L, Wu AW. All together now: Findings from a PCORI workshop to align patient-reported outcomes in the electronic health record. J Comp Eff Res 2016;5:561-567. 13. Pincus T. Electronic Multidimensional Health Assessment Questionnaire (eMDHAQ): Past, present and future of a proposed single data management system for clinical care, research, quality improvement, and monitoring of long-term outcomes. Clin Exp Rheumatol 2016;34:S17-S33. 14. Stefanovic S, Wallwiener M, Karic U, et al. Patientreported outcomes (PRO) focused on adverse events (PRO-AEs) in adjuvant and metastatic breast cancer: clinical and translational implications. Support Care Cancer 2017;25:549-558.
Purpose: To determine the patient compliance in completing electronically administered patient-reported outcome (PRO) scores following shoulder arthroscopy, and to determine if dedicated research assistants improve patient compliance. Methods: Patients undergoing arthroscopic shoulder surgery from January 1, 2014, to December 31, 2014, were prospectively enrolled into an electronic data collection system with retrospective review of compliance data. A total of 143 patients were included in this study; 406 patients were excluded (for any or all of the following reasons, such as incomplete follow-up, inaccessibility to the order sets, and inability to complete the order sets). All patients were assigned an order set of PROs through an electronic reporting system, with order sets to be completed prior to surgery, as well as 6 and 12 months postoperatively. Compliance rates of form completion were documented. Patients who underwent arthroscopic anterior and/or posterior stabilization were excluded. Results: The average age of the patients was 53.1 years, ranging from 20 to 83. Compliance of form completion was highest preoperatively (76%), and then dropped subsequently at 6 months postoperatively (57%) and 12 months postoperatively (45%). Use of research assistants improved compliance by approximately 20% at each time point. No differences were found according to patient gender and age group. Of those completing forms, a majority completed forms at home or elsewhere prior to returning to the office for the clinic visit. Conclusions: Electronic administration of PRO may decrease the amount of time required in the office setting for PRO completion by patients. This may be mutually beneficial to providers and patients. It is unclear if an electronic system improves patient compliance in voluntary completion PRO. Compliance rates at final follow-up remain a concern if data are to be used for establishing quality or outcome metrics. Level of Evidence: Level IV, case series.
I
From the Department of Orthopedics, Henry Ford Health System (E.C.M.), Detroit, Michigan; Department of Sports Medicine, Rush University Medical Center (J.D.H., J.T.H., B.J.C., A.A.R., N.N.V.), Chicago, Illinois, U.S.A. The authors report the following potential conflicts of interest or sources of funding: J.T.H. has stock/stock options in Novartis and Nuvasive. B.J.C. receives consulting fees from Arthrex, Regentis, and Zimmer; payment for lectures including service on speakers bureaus from Arthrex; royalties from Arthrex, SLACK, DJ Orthopaedics, Elsevier Publishing, and Saunders/ Mosby-Elsevier; has stock/stock options in Carticept and Regentis; other from Tornier, Ossur, Medipost, Cytori, and Aesculap/Braun. A.A.R. receives consulting fees from Arthrex; payment for lectures including service on speakers bureaus from Arthrex; royalties from Arthrex and SLACK; other from DJO Surgical, Ossur, Saunders/Mosby-Elsevier, and Smith & Nephew. N.N.V. receives consulting fees from Orthospace and Smith & Nephew; royalties from Arthroscopy, Smith & Nephew, Vindico Medical-Orthopedics Hyperguide; Stock/stock options in Cymedica and Omeros; other from Ossur, Minivasive, DJ orthopaedics, and Arthrosurface. Full ICMJE author disclosure forms are available for this article online, as supplementary material. Received October 13, 2016; accepted June 2, 2017. Address correspondence to Eric C. Makhni, M.D., M.B.A., Department of Orthopedics, Henry Ford Health System, 6777 West Maple Road, West Bloomfield, MI 48322, U.S.A. E-mail: [email protected] Ó 2017 by the Arthroscopy Association of North America 0749-8063/16967/$36.00 http://dx.doi.org/10.1016/j.arthro.2017.06.016
t is estimated that physicians in the United States spend greater than 15 hours per week reporting quality measures, totaling 785 hours per physician, and an alarming $15.4 billion of spending1 per year. The shift from volume-based to value-based care has mandated an increased effort on reporting outcomes,1 and in particular, patient-reported outcomes (PROs). The incentive to make the reporting process more efficient for both patient and physician, therefore, is high. Unfortunately, significant hurdles exist when considering implementation of standardized outcome reporting guidelines and mandates. These include logistical and financial constraints incurred by the practitioner, as well as additional time and energy required by patients. Moreover, there has been a lack of consistency with regard to selection and administration of common PRO for disease-specific orthopaedic conditions.2,3 Traditionally, outcomes metrics have been administered with paper and pen to patients. Such administration has the advantage of being convenient to the patient, in that it is relatively cheap and can be done in
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol
-,
No
-
(Month), 2017: pp 1-7
1
2
E. C. MAKHNI ET AL.
any type of clinical setting. However, paper outcome reporting is difficult to administer remotely, therefore requiring the patient to physically be with the researcher or clinician. These forms must also be cleaned, scanned into a medical record or reporting system, and ultimately scored, thereby limiting seamless patient searching and “databasing.” Recently, electronic patient-reported outcomes (ePROM) have become more readily available to clinicians and researchers.4 This form of administration has many advantages over traditional paper-and-pen methods, including the ability to readily administer forms remotely (thereby allowing patients to complete forms at home prior to their visit), as well as improved sorting and storage of completed forms. Moreover, such a system may foster improved ability to conduct multicenter trials, as well as incorporate new computer adaptive testing methods.4-6 However, little data exist regarding the actual benefit of electronic forms, and whether or not patients readily complete these forms prior to their clinic visits (and thus decompressing the paperwork burden in the office). Electronic outcome reporting has shown potential for success in fields outside of orthopaedics.7,8 The purpose of this study was to determine patient compliance in completing electronically administered PRO scores following shoulder arthroscopy, and to determine if dedicated research assistants improve patient compliance. We hypothesized that the compliance rate will generally be below acceptable limits for publication of outcome studies using prospectively collected data.
Methods Institutional review board approval was obtained from our institution in order to retrieve outcome reporting data from patients presenting to our shoulder and sports medicine clinic. All patients who underwent arthroscopic shoulder surgery (including rotator cuff repair, subacromial decompression/acromioplasty, biceps tenotomy/tenodesis, distal clavicle excision, acromioclavicular joint reconstruction, or glenohumeral debridement) from January 1, 2014, to December 31, 2014, by 2 of the senior authors (B.J.C. and N.N.V.) were retrospectively screened for study inclusion. Survey protocols were not instituted for patients with shoulder stabilization at the time of study commencement; therefore these patients were not included in the study. Patients were excluded if they were younger than 18 years of age, did not understand the English language, or did not electronically receive outcome measures at all 3 time points (e.g., were enrolled into the PRO electronic registry postoperatively and therefore without preoperative data). In our practice, outcomes reporting transitioned to an electronic system in the middle of 2013. Therefore, enrollment into the study began in 2014 following
successful initiation of the electronic system. Piloting and iterations of the registry/order set components was continued after initiation of the registry, and therefore not all patients operated on during this time period were included in the study. Upon indication for surgery, each patient was enrolled into the electronic outcome reporting system (Outcomes Based Electronic Research Database [OBERD], Columbia, MO) and assigned an order set of outcome measures. Patients who underwent rotator cuff repair were assigned a Rotator Cuff order set, whereas those who underwent subacromial decompression/acromioplasty, biceps tenotomy/tenodesis, distal clavicle excision, acromioclavicular joint reconstruction, or glenohumeral debridement were assigned the Miscellaneous order set. Components of each order set can be found in Table 1. Patients were assigned to complete the order set prior to surgery, at 6 months postoperation, and at 12 months postoperation. The OBERD outcomes system was designed to provide improved, secure accessibility for patients and providers to clinical data forms. Users are assigned a user name and password, which is used for site entry. Personal security questions follow the login page for additional security. Patients are guided to their personal file, which shows previously completed surveys and current outstanding surveys. They may access the registry either through a traditional desktop web version or through a mobile version on their phones. Upon initiating a survey, questions are presented in a large font with answers that are outlined in colors corresponding with their connotation (red meaning poor, green meaning positive). Patients have the option of skipping questions, submitting their survey at any point or returning to previous questions. Once a survey is complete, patients are prompted to start the next survey or exit to the main menu if they have completed all assigned surveys. Patients enrolled in the electronic system received electronic reminders to their personal e-mail prior to their postoperative deadlines. These reminders are sent
Table 1. Order Set Components Order Set Rotator Cuff
Miscellaneous
Components ASESdStandardized Shoulder Assessment Form Constant-Murley Score Single Assessment Numeric Evaluation (SANE) score Western Ontario Rotator Cuff Index (WORC) ASESdShoulder Assessment Form Constant-Murley Score Single Assessment Numeric Evaluation (SANE) score
ASES, American Shoulder and Elbow Surgeons.
3
PATIENT COMPLIANCE WITH ELECTRONIC PROS Table 2. Patient Cohort Demographic Information Females, n (%) Average age, years, mean (SD) Rotator Cuff order set patients, n (%) Miscellaneous order set patients, n (%)
Both MDs 49 (34) 53.1 (12.0) 85 58
MD 1 20 (37) 54.7 (12.8) 41 (48) 13 (22)
MD 2 29 (33) 52.1 (11.5) 44 (52) 45 (78)
MD 1, enrolling surgeon 1; MD 2, enrolling surgeon 2; SD, standard deviation.
every 5 days. Patients have the opportunity to complete their assigned order set 1 month prior to and 1 month after their 6- and 12-month postoperative dates. Reminder e-mails are discontinued once patients have completed their order set or their 2-month completion period had expired. Additional opportunity to complete forms was presented during the office visit, but only if there was sufficient time between patient registration and when the patient was called to the examination room. A patient was considered to be compliant if he or she completed all survey forms in full at the requested time points (preoperative, 6 months postoperatively, or 12 months postoperatively). Those who partially completed their forms at any given time point were considered to be partially compliant, whereas those who failed to open or answer any of the questions in the forms were considered to be noncompliant at that respective time point. Compliance was documented at each of the 3 time points according to order set received (Rotator Cuff vs Miscellaneous), as well as by attending surgeon (B.J.C. vs N.N.V.), patient gender, and patient age (<50, 50-60, and 60 years). Moreover, the proportion of patients who completed their forms at home (or before the clinic visit) was reported compared with the proportion that completed the forms in the office on the day of the visit. Both senior authors (B.J.C. and N.N.V.) had a research assistant who enrolled patients into the respective order sets. Once enrolled into the order set, the patient received automatic reminders to complete their forms. One of the senior authors (B.J.C.) had additional dedicated research assistants who actively sent extra reminders to patients who were noncompliant with outcome reporting and monitored the clinic schedule to identify noncompliant patients. The impact of overall survey compliance related to additional personnel who provided personal reminders was evaluated with the current study. Statistical Analysis In this study, comparisons between categorical data were made using Fisher exact test or chi-square test for groups of 2 or 3 or more comparators, respectively. All comparisons of continuous data were made using analysis of variance using readily available web-based software (www.openepi.com).
Results There were 509 shoulder arthroscopies performed during the study period. A total of 143 patients were included in the study. There were 406 patients who were excluded (for any or all of the following reasons, such as incomplete follow-up, inaccessibility to the order sets, and inability to complete the order sets). There were no statistically significant differences between enrolling surgeons based on age or gender (Table 2). In total, there were 509 shoulder arthroscopies performed during this time. However, only patients who met inclusion criteria were included. Preoperatively, 76% of all 143 patients (n ¼ 109 patients) were compliant in completing their questionnaires. Compliance decreased to 57% (n ¼ 81 patients) and 45% (n ¼ 64 patients) at the 6- and 12-month postoperative time points, respectively (Fig 1; P < .001). When considering order set assigned, there was no statistically significant difference in compliance between patients in the Rotator Cuff versus the Miscellaneous order sets (Fig 2; P ¼ .2 or higher across all time points). With regard to timing of forms completion (Fig 3), all patients who completed preoperative forms completed them at home (or elsewhere prior to coming to the hospital for surgery; 109 of 109 patients). At the 6month postoperative mark, of the 81 patients who completed forms, a majority (85%; n ¼ 69 patients) completed them prior to coming for their office visit. At the 12-month postoperative mark, 55 of 64 patients who completed their forms did so prior to the visit (86%). When considering noncompliant patients (Fig 4), at the preoperative time point, there were 27 (19%) patients who partially completed their surveys whereas 7 (5%) patients did not start their order set surveys. At 6 and 12 months postoperatively, the number of patients who partially completed the surveys was 10 and 14 (7% and 10%), respectively. There were 52 (36%) and 65 (45%) patients who never started their order set surveys at the 6- and 12-month postoperative time points, respectively. There was a statistically significant increase over the course of the follow-up times in the proportion of patients who did not attempt to answer any order set surveys compared with those who partially answered their assigned order set (P < .001).
4
E. C. MAKHNI ET AL. 100% 90% 80%
70% 60% 50% 40% 30% 20% 10% 0%
Pre-Op
6 Month Post-Op
12 Month Post-Op
Fig 1. Compliance rates at the preoperative, 6-month, and 12-month time points.
Patient compliance was also reported according to demographic factors. There was no statistically significant difference found in compliance between male patients and female patients (Fig 5; P > .3). There was also no statistically significant difference in compliance rates found according to patient age (Fig 6; P ¼ .73). Finally, the impact of a dedicated research assistant was analyzed with regard to survey compliance (Fig 7). One of the senior contributing authors (MD 1) did have a research team that called and e-mailed patients who had not completed their surveys despite an automatically generated e-mail system from the outcome collection system. These research assistants also tracked the senior contributing author’s clinic schedule to identify patients who were noncompliant to administer surveys in clinic. The compliance of these patients in completing their surveys was 94% preoperatively (51 of 54 patients), 72% at 6 months postoperative (39 of 54 patients), and 56% at the 12 months
Fig 3. The location where patients completed their assigned survey forms. Patients either completed their survey at home (or elsewhere other than the office/clinic) or in the office/ clinic setting. Home: patients who completed surveys at home (or elsewhere other than the office/clinic); In office/ clinic: patients who completed surveys in the office/clinic setting.
postoperative (30 of 54 patients). In comparison, the second surgeon, who did not have a research assistant providing additional compliance support, had a compliance of 65% preoperatively (58 of 89 patients). At 6 and 12 months postoperatively, the compliance decreased to 46% (41 of 89 patients) and 38% (34 of 89 patients), respectively. The difference between these 2 surgeons was statistically significant (P < .01) over the course of the follow-up period.
Discussion Although compliance decreased with increasing follow-up times, the proportion of patients completing
100% 90% 80% 70% 60% Miscellaneous (n=58)
50%
Rotator Cuff (n=85) 40% 30%
20% 10% 0%
Pre-Op
6 Month Post-Op
12 Month Post-Op
Fig 2. Order set compliance rates at the preoperative, 6-month, and 12-month time points. Miscellaneous: patients who completed the assigned Miscellaneous order set; Rotator Cuff: patients who completed the assigned Rotator Cuff order set.
Fig 4. Patient noncompliance rates. Patients who were noncompliant in completing their surveys were categorized into partially compliant and noncompliant. There was a statistically significant difference in the proportion of patients who did not attempt to answer their assigned order sets compared with those who partially completed their assigned order set. * P < .001. Partial: patients who partially completed their order sets; Unopened: patients who never attempted or opened their assigned order sets.
5
PATIENT COMPLIANCE WITH ELECTRONIC PROS 100%
100%
90%
90%
80%
80%
70%
70%
60%
60%
Female (n=49)
50%
Male (n=94)
40%
30%
20%
20%
10%
10% 0%
Pre-Op
6 Month Post-Op
12 Month Post-Op
Fig 5. Male and female compliance rates. There was no statistically significant difference between male and female compliance rates at any time point. Female: female patients who completed their assigned order set; Male: male patients who completed their assigned order set.
forms prior to coming to the clinic was excellent when a team of research coordinators were tasked with ensuring compliance (94%). Unfortunately, compliance was only 65% when the forms were administered electronically and there was no additional research staff to improve compliance. In addition, compliance at postoperative follow-up, particularly 1-year, remains a significant concern. The results of our study indicate a decrease in compliance from preoperative to 6 months postoperative, and from 6 months to 12 months postoperative. Therefore, despite electronic reminders and the ability to complete forms away from the clinic, there was still a majority (55%) of patients who did not fully complete their surveys 1 year postoperatively. However, 56% of patients at least partially completed their surveys at this time point. These compliance results are lower than those of other studies that have documented similar findings. For example, in a study of hip arthroplasty patients in the United Kingdom, Imam and colleagues reported a PRO compliance rate with written forms of 89% and 83% at the 6- and 12-month postoperative marks.9 This increase in compliance could be due to a greater willingness to complete and return paper forms (as was used in this study), a greater willingness among arthroplasty patients, a greater willingness among patients in a nationalized health care setting, or the administration of the paper form questionnaire immediately prior to the physician evaluation in the office. Electronic outcome reporting has been successfully used in other medical specialties as well.10-14 In our study, we also found that patients were likely to not even begin completing their electronic forms as they approached their 12-month postoperative time point. This suggests that the patient’s interest levels in answering survey questions on an electronic form decreases as they progress farther from surgery.
50-59 years (n=47) ≥60 years (n=48)
40%
30%
0%
>50 years (n=48)
50%
Pre-Op
6 Month Post-Op
12 Month Post-Op
Fig 6. Patient compliance rates by age group. Patients were grouped by age at the date of service. No statistically significant differences were found between patient age groups. >50 years: patients younger than 50 years who completed their assigned order set; 50-59: patients aged 50 to 59 years who completed their assigned order set; 60: patients aged 60 years or older who completed their assigned order set.
Interestingly, a majority of patients who completed surveys did so prior to coming to their office visit. This finding supports the concept that use of electronic outcome reporting may decrease the burden of survey completion in the office. Another important finding in our study is the impact of dedicated research personnel who made additional personal reminders to patients to complete their surveys. These reminders were in addition to automatic reminders generated by the electronic outcome reporting system. Research staff also monitored the surgeon’s clinic schedule to identify noncompliant patients. The research assistant would then have the patient complete their surveys as they waited to be seen by the surgeon. Using this additional support staff and
Fig 7. Compliance rates for each enrolling physician. There was a statistically significant difference between the enrolling physicians at each time point (*P < .001). MD 1: patients who received care from Surgeon 1, who completed their assigned order set; MD 2: patients who received care from Surgeon 2, who completed their assigned order set.
6
E. C. MAKHNI ET AL.
verification of compliance resulted in an increase of approximately 20% at each time point. Outcome reporting appears to be significantly enhanced by dedicated support staff that is actively working with patients to increase compliance. In clinicians with busy outpatient office settings, additional dedicated research staff may be essential in maintaining efficient office clinical care while still ensuring high rates of compliance with outcome reporting. However, the addition of staff results in increased costs to the practice, and currently there is no additional compensation for the implementation of the electronic outcomes system, or the staff required to maintain and monitor its use. This likely represents a major barrier to entry for smaller private practice groups, and significant increased costs to the health care system that are currently not accounted for within payment models. With the mandate that all practices report outcomes to define value-based care, the government has imposed a system that may not effectively provide the information required by the surgeon without substantial additional infrastructure and personnel costs throughout their practice. Our study has demonstrated that despite implementation of an accessible electronic outcomes system, compliance among patients remains low at the 1-year follow-up time frame. Although not included in the current study, one may hypothesize that further decreases would be expected at longer follow-up time periods. As most are aware of from the research arena, the current compliance rates as demonstrated in this study are well below currently accepted norms for drawing any valid conclusions regarding the outcome of a given procedure and below the rate at which peer review publications can reliably occur. However, this important limitation of outcomes collection receives little or no attention in the current discussion regarding outcomes-based payments and the realities of implementing such a process. In moving forward, physicians should insist that these value-based payment schemes include funding for implementation and monitoring of outcomes collection as well as requirements for patient participation, with penalties for noncompliance such as increased deductibles or copayments. Lastly the currently used PROs have no ability to account for patient compliance or comorbidities in determining outcome, another limitation that has yet to be addressed. Limitations This study is not without limitations. Primarily, the compliance rates of patients receiving paper forms was not studied, and therefore no control group exists for our study population. Although these data (as well as data regarding time required for survey completion) would have been helpful to use as a comparison, they
were not available to the study team, as all patients transitioned to electronic reporting prior to the study collection period. However, we do not believe that paper-based collection is a realistic solution for largescale outcomes collection as would be required for value-based payment models. Additionally, the impact of multiple PROs administered on overall compliance was not documented, and therefore the impact of “survey fatigue” is not known. Second, the collection of PROs used in our order sets may be different from those collected by other practitioners. Order sets with fewer components may result in higher compliance rates, but this could not be determined from the current study. No current standardization exists for outcome sets for a given orthopaedic disease state. Our study population was limited to those undergoing shoulder arthroscopy. Additionally, the automatically generated electronic reminders to patients from the outcome reporting system may have been ineffective in increasing survey compliance. Another limitation is that definitive information regarding weekly and hourly efforts by research assistants was not available. As this was a retrospective study, the goal was to provide descriptive information regarding PRO compliance in our practice, which employs providers who are dedicated to collecting this information. Therefore, we are unable to provide quantitative data regarding number of hours spent by each member of the research team. It is estimated, however, that approximately 10% to 20% of the assistants’ efforts would go toward outcome administration and collection on any given week during the study period. Finally, determining reasons for noncompliance would be important, including a determination of changing contact information, the role of worker’s compensation or third-party liability, age, the impact of the clinical outcome at the time of evaluation, and other variables that are yet to be elucidated.
Conclusions Electronic administration of PRO may decrease the amount of time required in the office setting for PRO completion by patients. This may be mutually beneficial to providers and patients. It is unclear if an electronic system improves patient compliance in voluntary completion PRO. Compliance rates at final follow-up remain a concern if data are to be used for establishing quality or outcome metrics.
References 1. Bryan S, Davis J, Broesch J, et al. Choosing your partner for the PROM: A review of evidence on patient-reported outcome measures for use in primary and community care. Healthc Policy 2014;10:38-51. 2. Makhni EC, Padaki AS, Petridis PD, et al. High variability in outcome reporting patterns in high-impact ACL literature. J Bone Joint Surg Am 2015;97:1529-1542.
PATIENT COMPLIANCE WITH ELECTRONIC PROS 3. Makhni EC, Steinhaus ME, Morrow ZS, et al. Outcomes assessment in rotator cuff pathology: What are we measuring? J Shoulder Elbow Surg 2015;24:2008-2015. 4. Okike K, Lee OC, Makanji H, Morgan JH, Harris MB, Vrahas MS. Comparison of locked plate fixation and nonoperative management for displaced proximal humerus fractures in elderly patients. Am J Orthop (Belle Mead NJ) 2015;44:E106-E112. 5. Beckmann JT, Hung M, Bounsanga J, Wylie JD, Granger EK, Tashjian RZ. Psychometric evaluation of the PROMIS Physical Function Computerized Adaptive Test in comparison to the American Shoulder and Elbow Surgeons score and Simple Shoulder Test in patients with rotator cuff disease. J Shoulder Elbow Surg 2015;24: 1961-1967. 6. Overbeek CL, Nota SP, Jayakumar P, Hageman MG, Ring D. The PROMIS physical function correlates with the QuickDASH in patients with upper extremity illness. Clin Orthop Relat Res 2015;473:311-317. 7. Nippita S, Oviedo JD, Velasco MG, Westhoff CL, Davis AR, Castano PM. A randomized controlled trial of daily text messages versus monthly paper diaries to collect bleeding data after intrauterine device insertion. Contraception 2015;92:578-584. 8. Newell D, Diment E, Bolton JE. An electronic patientreported outcome measures system in UK chiropractic practices: A feasibility study of routine collection of outcomes and costs. J Manipulative Physiol Ther 2016;39:31-41.
7
9. Imam MA, Barke S, Stafford GH, Parkin D, Field RE. Loss to follow-up after total hip replacement: A source of bias in patient reported outcome measures and registry datasets? Hip Int 2014;24:465-472. 10. Cowan RA, Suidan RS, Andikyan V, et al. Electronic patient-reported outcomes from home in patients recovering from major gynecologic cancer surgery: A prospective study measuring symptoms and health-related quality of life. Gynecol Oncol 2016;143:362-366. 11. Jensen RE, Rothrock NE, DeWitt EM, et al. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care 2015;53:153-159. 12. Jensen RE, Snyder CF, Basch E, Frank L, Wu AW. All together now: Findings from a PCORI workshop to align patient-reported outcomes in the electronic health record. J Comp Eff Res 2016;5:561-567. 13. Pincus T. Electronic Multidimensional Health Assessment Questionnaire (eMDHAQ): Past, present and future of a proposed single data management system for clinical care, research, quality improvement, and monitoring of long-term outcomes. Clin Exp Rheumatol 2016;34:S17-S33. 14. Stefanovic S, Wallwiener M, Karic U, et al. Patientreported outcomes (PRO) focused on adverse events (PRO-AEs) in adjuvant and metastatic breast cancer: clinical and translational implications. Support Care Cancer 2017;25:549-558.