- Email: [email protected]
Patient-Reported Outcomes Measurement Information System (PROMIS) Instruments Correlate Better With Legacy Measures in Knee Cartilage Patients at Postoperative Than at Preoperative Assessment
Patient-Reported Outcomes Measurement Information System (PROMIS) Instruments Correlate Better With Legacy Measures in Knee Cartilage Patients at Postoperative Than at Preoperative Assessment Benedict U. Nwachukwu, M.D., M.B.A., Alexander Beletsky, B.A., Neal Naveen, B.S., Taylor M. Southworth, B.S., Kelechi R. Okoroha, M.D., Brian Forsythe, M.D., Nikhil Verma, M.D., Adam Yanke, M.D., Ph.D., and Brian J. Cole, M.D., M.B.A.
Purpose: To define the psychometric properties of the Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function (PF), Pain Interference (PI), and Depression computer adaptive tests (CATs) in patients undergoing knee cartilage surgical procedures. Methods: The PROMIS PF, PI, and Depression CATs were administered preoperatively and at 6 months alongside legacy knee patient-reported outcome measures (PROMs) in patients undergoing knee cartilage surgical procedures. Statistical analysis consisted of the time to completion, psychometric analysis for correlative strengths, absolute and relative floor and ceiling effects, and Cohen effect size. Results: Our study included 250 patients (57.2% male patients), averaging 1.87, 1.53, and 1.91 minutes for completion of the PF, PI, and Depression CATs, respectively. Preoperatively, the PROMIS PF and PI CATs showed wide ranges of correlation coefficients with respect to function (r ¼ 0.14-0.72 and r ¼ 0.29-0.77, respectively) and health-related quality-of-life PROMs (r ¼ 0.640.70). At 6 months, the PROMIS PF CAT (r ¼ 0.82-0.93) and PI CAT (r ¼ 0.77-0.93) both exhibited excellent correlations with respect to legacy function and health-related quality-of-life PROMs except for the Marx Activity Rating Scale (r ¼ 0.36-0.44). None of the PROMIS instruments exhibited any significant floor or ceiling effects. Conclusions: The PROMIS PF, PI, and Depression CATs performed better with respect to legacy PROMs in the postoperative period than the preoperative period. In addition, the PROMIS PF and PI measures performed best with respect to the International Knee Documentation Committee questionnaire, and no floor or ceiling effects were identified for the PROMIS instruments. The PROMIS instruments may be more suited to track outcomes postoperatively than to establish preoperative baselines in cartilage surgery patients. Level of Evidence: Level IV, retrospective case series.
See commentary on page 1429
From the Division of Sports Medicine, Midwest Orthopaedics at Rush, Rush University Medical Center, Chicago, Illinois, U.S.A. The authors report the following potential conflicts of interest or sources of funding: B.F. receives paid royalties from Elsevier; research support from Arthrex; owns stock or stock options in Jace Medical; receives fellowship support from Smith & Nephew and Ossur; and is a consultant for and receives research support from Stryker. N.V. receives research support from Arthrex and DJ Orthopaedics; receives publishing royalties and material support from Arthroscopy; receives personal fees from Orthospace; and receives publishing royalties from Vindico Medical-Orthopedics Hyperguide. A.Y. receives research support from Arthrex and NuTech and hospitality fees from Smith & Nephew. B.J.C. receives research support from Aesculap/B.Braun, National Institutes of Health (NIAMS and NICHD), and Zimmer; is on the editorial or governing board of the American Journal of Orthopedics, American Journal of Sports Medicine, Cartilage, Journal of Shoulder and Elbow Surgery, and Journal of the American Academy of Orthopaedic Surgeons; receives IP royalties and research support from Arthrex; is a board or committee
member of the Arthroscopy Association of North America and International Cartilage Repair Society; receives other financial or material support from Athletico, JRF Ortho, and Smith & Nephew; receives IP royalties from Elsevier Publishing; receives publishing royalties from Operative Techniques in Sports Medicine; receives stock or stock options from Ossio; and receives stock or stock options and research support from Regentis. Full ICMJE author disclosure forms are available for this article online, as supplementary material. Received May 23, 2019; accepted January 16, 2020. Address correspondence to Nikhil Verma, M.D., Midwest Orthopaedics at Rush, Rush University Medical Center, 1611 W Harrison St, Ste 300, Chicago, IL 60612, U.S.A. E-mail: [email protected] Ó 2020 by the Arthroscopy Association of North America 0749-8063/19641/$36.00 https://doi.org/10.1016/j.arthro.2020.01.036
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol 36, No 5 (May), 2020: pp 1419-1428
1419
1420
O
B. U. NWACHUKWU ET AL.
utcomes research in orthopaedic surgery has evolved significantly in recent history, improving from variable single-surgeon reports to standardized, domain-based questionnaires completed by individual patients.1 Patient-reported outcome measure (PROM) data have been used in conjunction with physical examination and imaging techniques to define clinical outcomes after cartilage procedures.1-5 Furthermore, by connecting PROM data with specific questionnaires defining self-perceived improvement and satisfaction, researchers have been able to define PROM threshold values above which most patients report true subjective improvement after specific cartilage surgical procedures and viscosupplementation.6-9 However, significant interest in domain-based PROM data has led to important secondary limitations including (1) increased variability in PROM selection between studies, (2) variable use of PROM data longitudinally to track outcomes, (3) failure to include all relevant domains of interest (e.g., function, health-related quality of life [HRQoL], and mental health), and (4) significant question burden for patients completing numerous PROMs at once. The Patient-Reported Outcomes Measurement Information System (PROMIS), an initiative introduced in 2004, aims to address some of the limitations in patientreported outcome research. Specifically, certain legacy PROMs may have significant floor or ceiling effects in specific patient populations, require full questionnaire completion to produce a score, and ask predefined questions of unknown discriminative value.1,10-12 PROMIS measures such as the Physical Function (PF), Pain Interference (PI), and Upper Extremity computer adaptive tests (CATs) have shown decreased question requirements, high instrument responsiveness, decreased time to completion (TTC), and possibly early responsiveness to changes in clinical status.13-18 In addition, by using CATs, the PROMIS instruments select questions of high discriminative value based on previous responses.13,19-21 Given the inherent appeal of PROMIS, there is increasing interest in evaluating the PROMIS instruments for musculoskeletal conditions.22 Several published studies have shown that PROMIS correlates well at preoperative time points with legacy measures for anterior cruciate ligament injury, meniscal injury, rotator cuff pathology, femoroacetabular impingement, and glenohumeral arthritis.18,23-28 No prior study has examined the psychometric properties of the PROMIS instruments longitudinally in knee cartilage patients, and only a single study has examined the performance of CATs in cartilage procedures.21 Thus, for knee cartilage surgical procedures, there is a need to examine the performance of the PROMIS instruments relative to legacy measures to (1) validate these instruments in this patient population; (2)
investigate any time-dependent nature associated with PROMIS (e.g., preoperative vs postoperative); and (3) determine whether previously defined relations with respect to question burden, responsiveness, and TTC are maintained.22 The purpose of this study was to define the psychometric properties of the PROMIS PF, PI, and Depression CATs in patients undergoing knee cartilage surgical procedures. We hypothesized that (1) both the PROMIS PF CAT and PI CAT would exhibit good to excellent correlations relative to function and HRQoL legacy instruments at both preoperative and 6-month time points,29-31 (2) the PF and PI CATs would exhibit fair to good correlations with mental health legacy PROMs across time points,1,13 (3) the PROMIS Depression CAT would exhibit fair to very good correlations with legacy PROMs across time points,29 and (4) all PROMIS measures would not show any significant absolute or relative floor or ceiling effects.29,30,32
Methods Study Design and Cohort Establishment Prior to study initiation, the study protocol was reviewed and approved by an institutional review board with a corresponding waiver of consent. Prospectively collected PROM data across multiple orthopaedic surgeons at a single institution were aggregated via an institutional electronic registry (Outcome Based Electronic Research Database; Universal Research Solutions, Columbia, MO). The cartilage cohort was constructed by using Current Procedural Terminology (CPT) codes to identify patients undergoing articular cartilage debridement (CPT codes 29876 and 29877), osteochondral allograft transplant (OCA) (CPT code 27415), or microfracture (CPT code 29879) between December 2017 and August 2018. The inclusion criteria included receipt of a primary cartilage procedure (i.e., debridement, allograft transplantation, microfracture, or autologous chondrocyte implantation [ACI]) for a focal cartilage defect and full completion of preoperative and 6-month PROM data. The 6-month follow-up time point was defined based on the date of surgery, with patients undergoing follow-up as early as 3 weeks prior to 6 months and as late as 3 weeks after the 6-month follow-up point. The exclusion criteria included concomitant ligamentous, meniscal, or bony procedures; biological augmentation; receipt of total knee arthroplasty within the follow-up period; or failure to complete preoperative PROMs. Demographic data collected included age, sex, insurance provider, and Workers’ Compensation status. During the study time frame, no patient undergoing ACI (CPT code 27412) or osteochondral autograft transplantation (OAT) (CPT code 29866) met the inclusion criteria.
1421
OUTCOMES IN KNEE CARTILAGE PATIENTS Table 1. Demographic Characteristics by Cartilage Procedure Age, yr WC Sex Male Female Laterality Right Left Preoperative PROM PROMIS PF Pain Depression BRS IKDC KOOS ADL JR Pain PS QOL Sport Symptom Marx Activity Rating Scale SF-12 MCS PCS VR-12 MCS PCS VR6D WOMAC Function Pain Stiffness Total
ACD OCA 41.2 15.1 29.8 9.2 22 (11.3) 2 (6.1) 113 (58.2) 81 (41.8)
14 (42.4) 19 (57.6)
105 (54.1) 89 (45.9)
18 (54.5) 15 (45.5)
MFX P Value 26.2 11.0 <.01* 0 (0.0) .22 .11 16 (69.6) 7 (30.4) .09 7 (30.4) 16 (69.6)
41.6 59.4 46.0 3.9 43.2
7.4 6.8 7.3 0.6 16.5
38.5 62.1 48.9 3.9 36.0
6.5 7.0 7.4 0.5 16.3
41.7 59.2 51.8 3.8 43.0
11.4 8.5 5.7 0.6 16.6
.14 .06 .01* .71 .15
66.0 57.9 58.2 40.2 29.6 32.9 56.1 7.5
20.2 15.3 18.4 15.3 19.4 24.2 20.3 6.5
64.3 55.0 55.0 42.7 15.0 23.3 59.5 6.5
21.7 14.2 19.2 14.7 15.3 19.6 17.3 7.1
70.2 59.1 59.6 39.1 29.5 32.5 61.7 11.4
25.7 16.3 22.1 14.5 20.3 21.2 17.5 5.6
.63 .61 .62 .64 <.01* .10 .39 .04*
Outcome Score (KOOS) Joint Replacement (JR), Physical Function, Symptoms, Pain, Activities of Daily Living, Sport, and Quality of Life; Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) Pain, Stiffness, Function, and Total; Marx Activity Rating Scale; Brief Resilience Scale; Veteran-RAND 12item Health Survey (VR-12); 12-item Short Form Health Survey (Short Form 12 [SF-12]); and Veterans RAND 6 Domain (VR6D). The VR-12 and SF-12 instruments were split into the Physical Component Scores and Mental Component Scores (MCSs) for subsequent analysis. PROMs were organized into function, HRQoL, and mental health domains to allow for appropriate comparison.1,33
NOTE. Data are presented as mean standard deviation or number (percentage). P values represent results from the t test or c2 test for demographic variables and 1-way analysis of variance for preoperative score values. ACD, articular cartilage debridement; ADL, Activities of Daily Living; BRS, Brief Resilience Scale; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; MFX, microfracture; OCA, osteochondral allograft transplant; PCS, Physical Component Score; PF, Physical Function; PROM, patient-reported outcome measure; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WC, Workers’ Compensation; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Statistically significant (P < .05).
Statistical Analysis: Instrument Performance, Responsiveness, and Floor and Ceiling Effects A priori sample size calculations assuming a type I error rate of 5% and a power (1 e b) of 0.90 yielded a minimum sample size of 164 assuming a small effect size. Institutional registry data were first reviewed for calculation of the average number of questions completed per outcome instrument (i.e., question burden) as well as the average TTC for a given instrument in minutes. Spearman rank correlation coefficients were used to examine the psychometric properties of the PROMIS PF, PI, or Depression CAT relative to legacy PROMs. Correlation coefficients were classified by the strength of association exhibited, with greater than 0.8 equating to excellent; 0.71 to 0.8, very good; 0.61 to 0.7, good; 0.41 to 0.6, fair; and 0.21 to 0.4, poor.1,34 Both absolute and relative floor and ceiling effects were assessed by calculating the percentage of respondents who obtained the absolute and relative highest and lowest possible scores, respectively, on a given PROM. In the case of relative floor and ceiling effects, a score distribution of 5% from the relative minimum and maximum scores was calculated to qualify a score in the relative floor or ceiling range. A percentage of 15% or greater was designated as a significant absolute or relative floor or ceiling effect.34-36 Internal responsiveness was evaluated by calculation of the Cohen d, with an effect size of less than 0.20 corresponding to a negligible effect size; 0.20 to 0.49, small effect size; 0.50 to 0.80, moderate effect size; and greater than 0.80, large effect size.37,38 Subgroup analysis was performed for PROMs showing ceiling or floor effects to examine the effect of age and sex on relative and absolute ceiling achievement.
Patient-Reported Outcome Measures The PROMs examined in this study included the following: PROMIS PF (version 2.0) CAT; PROMIS Pain (version 1.1) CAT; PROMIS Depression (version 1.0) CAT; International Knee Documentation Committee (IKDC) questionnaire; Knee Injury and Osteoarthritis
A total of 250 patients were included in this study. Demographic data and preoperative PROM scores are summarized in Table 1. The articular cartilage debridement cohort was significantly older than the OCA and microfracture cohorts (P < .01). No significant
54.1 9.7 36.8 9.6
53.8 7.8 33.4 9.2
37.6 9.7 37.2 6.8
56.5 9.2 55.4 8.0 58.0 8.1 39.0 10.0 35.1 10.3 39.4 10.1 0.67 0.11 0.63 0.1 0.68 0.1 65.1 65.8 55.7 64.0
23.2 21.7 28.1 23.4
67.3 64.5 60.3 64.4
19.7 20.4 22.7 18.4
60.5 58.3 63.5 64.7
26.4 23.4 20.3 20.0
.75 .20 .65 .17 .24 .69 .52 .48 .91
Results
1422
B. U. NWACHUKWU ET AL.
Table 2. Performance of PROMIS Physical Function CAT in Cartilage Patients Preoperative PROM Function PROMIS Pain IKDC KOOS ADL JR Pain PS Sport Symptom Marx Activity Rating Scale WOMAC Function Pain Stiffness Total HRQoL KOOS QOL SF-12 PCS VR-12 PCS VR6D Mental health PROMIS Depression VR-12 MCS SF-12 MCS BRS
6 mo
r
Correlative Classification
r
Correlative Classification
e0.64* 0.72*
Good Very good
e0.85* 0.90*
Excellent Excellent
0.61* 0.55* 0.57* e0.60* 0.66* 0.32* 0.14*
Good Fair Fair Fair Good Poor Very poor
0.88* 0.92* 0.88* e0.82* 0.82* 0.80* 0.44*
Excellent Excellent Excellent Excellent Excellent Very good Fair
0.58* 0.52* 0.37* 0.61*
Fair Fair Poor Good
0.88* 0.93* 0.80* 0.90*
Excellent Excellent Very good Excellent
0.65* 0.69* 0.70* 0.64*
Good Good Good Good
0.91* 0.86* 0.87* 0.83*
Excellent Excellent Excellent Excellent
e0.28* 0.32* 0.12* 0.13*
Poor Poor Very poor Very poor
e0.45* 0.47* 0.27*
Fair Fair Poor
ADL, Activities of Daily Living; BRS, Brief Resilience Scale; CAT, computer adaptive test; HRQoL, health related quality of life; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PROM, patient-reported outcome measure; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR-12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Significant correlation (P < .05).
differences were observed with respect to Workers’ Compensation status, sex, or laterality. Regarding questionnaire burden, patients averaged a total of 4.59 questions for the PROMIS PF (mean TTC, 1.87 minutes), 4.28 questions for the PROMIS PI (mean TTC, 1.52 minutes), 4 questions for the Marx Activity Rating Scale (mean TTC, 1.53 minutes), and 6.25 questions for the PROMIS Depression (mean TTC, 1.91 minutes) and Brief Resilience Scale (mean TTC, 2.10 minutes) instruments. The IKDC questionnaire (mean TTC, 5.61 minutes), KOOS instruments (mean TTC, 6.52 minutes), and VR-12 or SF-12 instruments (mean TTC, 6.12 minutes) showed average TTC values in excess of 5 minutes (Table 1). Preoperative and 6-month postoperative correlation coefficients between legacy PROMs and the PROMIS PF CAT are summarized in Table 2. With respect to function PROMs, the IKDC questionnaire showed the best (very good) preoperative correlation (r ¼ 0.72) whereas the WOMAC instruments (r ¼ 0.37-0.61) and KOOS instruments (r ¼ 0.32-0.66) exhibited poor to good correlations relative to the PF CAT. All HRQoL PROMs exhibited good preoperative correlations with respect to
the PROMIS PF CAT (r ¼ 0.64-0.70). Mental health measures exhibited very poor to poor preoperative correlations relative to the PROMIS PF CAT (r ¼ 0.12-0.32). Every single PROM showed improved correlation with the PROMIS PF CAT at the 6-month assessment. At 6 months postoperatively, the PROMIS PF CAT showed very good to excellent correlations (r ¼ 0.82-0.93 [absolute value]) with legacy functional outcome PROMs except for the Marx Activity Rating Scale, which exhibited a fair correlation (r ¼ 0.44). All HRQoL PROMs exhibited excellent correlations (r ¼ 0.83-0.91) with the PROMIS PF CAT, whereas the mental health measures showed fair or poor correlations (r ¼ 0.27-0.47) (Table 2). The IKDC questionnaire exhibited the strongest preoperative correlation (very good) with respect to the PROMIS PI CAT (r ¼ 0.77). The KOOS instruments exhibited fair to very good preoperative correlations with the PROMIS PI CAT (r ¼ 0.59-0.77). The WOMAC instruments exhibited good to very good correlations relative to the PI CAT (r ¼ 0.62-0.72), and the Marx Activity Rating Scale was the only function measure to exhibit a poor correlation preoperatively (r ¼ 0.29).
1423
OUTCOMES IN KNEE CARTILAGE PATIENTS Table 3. Performance of PROMIS Pain Interference CAT in Cartilage Patients Preoperative PROM Function PROMIS PF CAT IKDC KOOS ADL JR Pain PS Sport Symptom Marx Activity Rating Scale WOMAC Function Pain Stiffness Total HRQoL KOOS QOL SF-12 PCS VR-12 PCS VR6D Mental health PROMIS Depression VR-12 MCS SF-12 MCS BRS
6 mo
r
Correlative Classification
r
Correlative Classification
e0.64* e0.77*
Good Very good
e0.85* e0.88*
Excellent Excellent
e0.71* e0.73* e0.77* 0.71* e0.72* e0.59* e0.29*
Very good Very good Very good Very good Very good Fair Poor
e0.93* e0.90* e0.83* 0.85* e0.78* e0.83* e0.36*
Excellent Excellent Excellent Excellent Very good Excellent Poor
e0.71* e0.71* e0.62* e0.72*
Very good Very good Good Very good
e0.93* e0.88* e0.81* e0.92*
Excellent Excellent Excellent Excellent
e0.64* e0.65* e0.69* e0.70*
Good Good Good Good
e0.86* e0.81* e0.84* e0.77*
Excellent Excellent Excellent Very good
0.27* e0.44* e0.32* e0.16*
Poor Fair Poor Very poor
0.48* e0.45* e0.26*
Fair Fair Poor
ADL, Activities of Daily Living; BRS, Brief Resilience Scale; CAT, computer adaptive test; HRQoL, health related quality of life; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PF, Physical Function; PROM, patient-reported outcome measure; PROMIS, PatientReported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Significant correlation (P < .05).
HRQoL instruments exhibited good correlations with the PROMIS PI CAT (r ¼ 0.64-0.70). Correlation coefficients for each function and HRQoL measure relative to the PROMIS PI CAT improved from preoperatively to 6 months postoperatively. The IKDC questionnaire (r ¼ 0.88) and WOMAC instruments (r ¼ 0.81-0.93) all exhibited excellent correlations relative to the PI CAT at 6 months. The KOOS subscales (r ¼ 0.78-0.93) and HRQoL PROMs (r ¼ 0.77-0.86) exhibited very good to excellent correlations at 6 Table 4. Performance of PROMIS Depression CAT in Cartilage Patients Preoperative PROM BRS SF-12 MCS VR-12 MCS
r e0.41* e0.49* e0.58*
Correlative Classification Fair Fair Fair
6 mo r
Correlative Classification
e0.70* e0.78*
Good Very good
BRS, Brief Resilience Scale; CAT, computer adaptive test; MCS, Mental Component Score; PROM, patient-reported outcome measure; PROMIS, Patient-Reported Outcomes Measurement Information System; SF-12, Short Form 12; VR-12, Veterans RAND 12. *Significant correlation (P < .05).
months. Mental health PROMs exhibited very poor to fair correlations preoperatively (r ¼ 0.16-0.44) and poor to fair correlations at 6 months (r ¼ e0.26 to 0.45) (Table 3). Table 4 summarizes the performance of the PROMIS Depression CAT relative to mental health legacy instruments. Preoperatively, mental health PROMs exhibited fair correlations relative to the PROMIS Depression CAT (r ¼ 0.41-0.58). At 6 months, correlation coefficients improved to good or very good (r ¼ 0.70-0.78) (Table 4). No significant absolute or relative floor or ceiling effects were observed for the PROMIS Depression, PF, or PI CAT. The Marx Activity Rating Scale showed a significant absolute floor effect (n ¼ 75, 33%) preoperatively, as well as an absolute floor effect at 6 months postoperatively (n ¼ 24, 35.8%). The Marx instrument showed absolute ceiling effects preoperatively and at 6 months (Table 5). A relative floor effect was observed in KOOS Sport preoperatively (n ¼ 53, 20.5%) as well as at 6 months (Table 6). The IKDC questionnaire (0.53), KOOS Quality of Life (0.52), and VR-12 Physical Component Score (0.51) showed medium effect sizes (Table 7).
1424
B. U. NWACHUKWU ET AL.
Table 5. Absolute Floor and Ceiling Effects Preoperative (%) Instrument
Absolute Floor
PROMIS Depression PF PI BRS IKDC KOOS ADL JR Pain PS QOL Sport Symptom Marx Activity Rating Scale SF-12 PCS MCS VR-12 PCS MCS VR6D WOMAC Function Stiffness Pain Total
0 0 0 0 0 1 2 2 1 29 34 0 75
6 mo (%)
Absolute Ceiling
(0.0) (0.0) (0.0) (0.0) (0.0)
0 0 0 26 0
(0.0) (0.0) (0.0) (10.4) (0.0)
(0.4) (0.8) (0.8) (0.4) (11.6) (13.6) (0.0) (30.0)*
6 2 1 4 0 2 3 55
(2.4) (0.8) (0.4) (1.6) (0.0) (0.8) (1.2) (22.0)*
Absolute Floor
Absolute Ceiling
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0)
0 (0.0)
0 0 0 0 2 5 0 24
(0.0) (0.0) (0.0) (0.0) (8.0) (20)* (0.0) (35.8)*
1 0 1 0 0 0 0 8
(4.0) (0.0) (4.0) (0.0) (0.0) (0.0) (0.0) (12)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 0 0 0
1 1 3 1
2 4 10 1
(0.8) (1.6) (4.0) (0.4)
7 4 20 2
(2.8) (1.6) (8.0) (0.8)
(0.0) (0.0) (0.0) (0.0)
(4.0) (4.0) (12) (4.0)
ADL, Activities of Daily Living; BRS, Brief Resilience Scale; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PF, Physical Function; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR-12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Significant floor or ceiling effect.
Discussion The main finding of this study was that the correlation of the PROMIS PF and PI CATs relative to legacy function and HRQoL instruments improves from preoperative assessment to 6-month postoperative assessment. Specifically, the correlation of the PROMIS PF CAT compared with legacy function PROMs improved from a range of fair to very good to a range of very good to excellent (excluding the Marx Activity Rating Scale). The performance of the PROMIS PF CAT compared with legacy HRQoL measures improved from good to excellent. A similar trend was found with the PROMIS PI CAT. Notably, of the assessed legacy PROMs, the PROMIS tracked the most consistently with the IKDC questionnaire. In addition, we have confirmed that PROMIS PF, PI, and Depression CATs maintained low questionnaire burden relative to the IKDC, KOOS, and VR-12 or SF-12 instruments, and average completion times were less than 2 minutes for each PROMIS CAT. Finally, none of the 3 PROMIS CATs showed absolute or relative floor or ceiling effects in our cohort, and each showed a small effect size. These findings allow us
to refine our hypothesis and advance our understanding of the performance of the PROMIS instruments in patients undergoing knee cartilage surgery. The psychometric properties of the PROMIS PF CAT have been studied across a variety of orthopaedic patient populations.13,17,39,40 In this study, the PROMIS PF CAT showed a very good correlation with the IKDC questionnaire, poor to good correlations with the KOOS and WOMAC instruments, and good correlations with the HRQoL instruments preoperatively. At 6 months, correlations improved to very good to excellent for all function and HRQoL instruments, excluding the Marx Activity Rating Scale, which improved from very poor to fair. Hancock et al.41 reported comparable performance metrics for the PROMIS PF CAT relative to KOOS subscores in an anterior cruciate ligament reconstruction (ACLR) cohort, with strength ranging from fair to very good. They also showed that the Marx Activity Rating Scale exhibited the weakest correlation coefficient with the PROMIS PF CAT, a relation most likely resulting from differences in instrument design and domain of assessment. The PROMIS PF CAT,
1425
OUTCOMES IN KNEE CARTILAGE PATIENTS Table 6. Relative Floor and Ceiling Effects Preoperative (%) Instrument PROMIS Depression PF PI BRS IKDC KOOS ADL JR Pain PS QOL Sport Symptom Marx Activity Rating Scale SF-12 PCS MCS VR-12 PCS MCS VR6D WOMAC Function Stiffness Pain Total
Relative Floor 0 0 0 1 1 3 2 2 1 29 53 0 86
6 mo (%)
Relative Ceiling
(0.0) (0.0) (0.0) (0.4) (0.4)
0 0 0 38 0
(0.0) (0.0) (0.0) (14.7) (0.0)
(1.2) (0.8) (0.8) (0.4) (11.2) (20.5)* (0.0) (33.3)*
18 2 1 4 0 3 6 65
(7.0) (0.8) (0.4) (1.6) (0.0) (1.2) (2.3) (25.2)*
Relative Floor
Relative Ceiling
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
1 (0.4)
0 (0.0)
0 0 0 0 2 5 0 24
(0.0) (0.0) (0.0) (0.0) (8.0) (20)* (0.0) (35.8)*
4 0 2 0 0 1 0 8
(16)* (0.0) (8.0) (0.0) (0.0) (4.0) (0.0) (12)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 0 0 0
4 1 6 2
3 10 4 3
(1.2) (3.9) (1.6) (1.2)
15 20 16 9
(5.8) (7.8) (9.0) (3.5)
(0.0) (0.0) (0.0) (0.0)
(16)* (4.0) (24)* (8.0)
ADL, Activities of Daily Living; BRS, Brief Resilience Scale; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PF, Physical Function; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR-12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Significant floor or ceiling effect.
similarly to the IKDC questionnaire, assesses function by asking patients how much difficulty they experience performing specific activities (e.g., standing, pulling, and exercising), whereas the Marx instrument consists of 4 questions assessing activity level based on the frequency of sport-related activities (e.g., cutting or pivoting). In a similar study, Scott et al.16 examined the psychometric properties of the PF CAT over a 2-year follow-up period in ACLR patients, reporting good to excellent correlations with the KOOS Sport, KOOS Activities of Daily Living, and SF-36 Physical Function instruments. Both the PROMIS PF CAT and PI CAT exhibited the weakest correlations with mental health measures in our study, a finding consistent with previous work by Hancock et al.1 examining the psychometric properties of the PF CAT in patients undergoing meniscal surgery. We found that the most consistent correlation between PROMIS and legacy function PROMs was for the IKDC questionnaire. Similarities in instrument design may in part explain this relation, given that both instruments have 4 to 5 graded responses corresponding
to increasing difficulty performing specific activities. Prior literature has shown favorable psychometric qualities for the IKDC questionnaire; however, the KOOS and KOOS JR have been recommended in the assessment of degenerative knee conditions.42-44 Our study findings raise questions about the role of PROMIS in relation to legacy PROMs. In theory, PROMIS seeks to replace legacy PROMs that present high questionnaire burden and fatigue. We found variable correlations for the PROMIS instruments with legacy measures at preoperative time points. This finding may suggest that the PROMIS instruments can appropriately substitute for instruments with demonstrable very good correlations (e.g., IKDC questionnaire, with very good and excellent correlations) but should not replace instruments with less favorable correlations (e.g., KOOS JR, with fair correlation at preoperative assessment). In this regard, our study differs from prior work on PROMIS because we do not recommend the wholescale abandonment of legacy measures in favor of PROMIS. Rather, the PROMIS instruments may more appropriately be administered alongside select disease-
1426
B. U. NWACHUKWU ET AL.
Table 7. Effect Sizes Instrument PROMIS Depression PF PI IKDC KOOS ADL JR Pain PS QOL Sport Symptom Marx Activity Rating Scale SF-12 PCS MCS VR-12 PCS MCS VR6D WOMAC Function Stiffness Pain Total
Mean Effect Size (95% CI)
Magnitude
e0.41 e0.30 0.21 e0.53
(e1.02 (e0.64 (e0.41 (e1.06
to to to to
0.21) 0.04) 0.84) 0.01)
Small Small Small Medium*
e0.38 e0.40 e0.50 e0.09 e0.52 e0.37 e0.41 0.27
(e0.96 (e0.99 (e1.08 (e0.44 (e1.09 (e0.95 (e0.97 (e0.08
to to to to to to to to
0.19) 0.18) 0.08) 0.26) 0.06) 0.20) 0.15) 0.61)
Small Small Small Negligible Medium* Small Small Small
e0.44 (e0.80 to e0.09) 0.24 (e0.11 to 0.59)
Small Small
e0.51 (e0.87 to e0.15) 0.05 (e0.31 to 0.40) e0.38 (e0.74 to e0.02)
Medium* Negligible Small
e0.28 e0.15 e0.31 e0.29
(e0.89 (e0.76 (e0.92 (e0.90
to to to to
0.33) 0.46) 0.30) 0.33)
Small Negligible Small Small
ADL, Activities of Daily Living; CI, confidence interval; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PF, Physical Function; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR-12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Medium or large effect size.
and function-specific PROMs to aid in outcome discrimination, particularly if instruments with a propensity toward floor and ceiling effects are used. It is worth noting that several of the evaluated legacy PROMs showed significant floor and/or ceiling effects (i.e., Marx Activity Rating Scale and KOOS subscales). This finding confirms the ability of PROMIS to distinguish outcomes between particularly high-functioning and low-functioning patients. Scott et al.16 suggested that because of limited floor and ceiling effects, the PROMIS PF measure was particularly well suited to tracking outcomes after ACLR, particularly in a young, active population known to experience significant ceiling effects when tracked with legacy measures. Conversely, they noted that there were particular legacy PROMs that may be used to assess specific domains of interest (e.g., use of the Marx Activity Rating Scale to evaluate postoperative rehabilitation).16 This notion is supported by our study findings in that legacy measures appear to be better suited for the preoperative assessment of joint-specific disability. Postoperatively,
however, numerous legacy PROMs included in our study showed floor and ceiling effects, whereas the PROMIS instruments were absent of this finding. Similarly, with respect to the PROMIS Depression CAT, legacy measures such as VR-12 MCS and SF-12 MCS may more appropriately establish baseline mental health states in cartilage patients whereas the Depression CAT may be better used to track postoperative changes in mental health. Questionnaire burden and TTC analysis revealed an average TTC of less than 2 minutes for each PROMIS instrument in our study. In addition, the average question burden was 4.28 for the PI CAT, 4.59 for the PF CAT, and 6.25 for the Depression CAT. The PROMIS PF CAT has previously been shown to have an average completion time of 54.1 seconds across 10,344 patients presenting to a hand clinic.17 The average question burden for the PROMIS PF CAT has been reported to be between 4 and 4.54 questions, dependent on the population of interest.16,41,45 For the PROMIS PI CAT, the average question requirement has been reported as 4.27, with a TTC of 45.05 seconds.14,45 Reported TTC data for the Depression CAT have shown values of 0.57 minutes and 0.80 minutes (48.11 seconds) in ambulatory sports medicine and hand clinics, respectively.14,17 Although our questionnaire burden data align well with previous literature, the average TTC data in our cohort for the PROMIS PF, PI, and Depression CATs showed completion times approximately 1 minute longer than those reported in prior studies. Limitations Our study has specific limitations readers must consider. Our cohort may be susceptible to selection bias by virtue of our inclusion criteria: Our data set was limited to primary surgical patients, and as such, our study conclusions do not extend to patients undergoing nonoperative treatment. In addition, the cartilage surgery cases from which our data were drawn were obtained across 3 surgeons at a single, high-volume academic orthopaedic institution. As a result, our data may not necessarily be generalizable to all patient populations and surgeons. Moreover, included in our case series were patients receiving cartilage debridement, OCA, and microfracture, suggesting variability in focal cartilage defect size. Because of limitations in our database, we are unable to stratify the cohort by defect size, lesion location, or lesion grade or to subsequently examine the impact of these operative factors on performance. Our patient-reported collection system used tablet computers to administer patient-reported outcome instruments. Consequently, we are unable to confirm whether patients completed a survey in question by themselves or perhaps a family member or associate assisted in the interpretation and completion of PROMs on their behalf. Finally, although
OUTCOMES IN KNEE CARTILAGE PATIENTS
8 individuals received ACI for a focal autologous cartilage defect within the study period, none of the recipients met the inclusion criteria because of previous surgical intervention (n ¼ 3), subsequent total knee arthroplasty within the follow-up period (n ¼ 1), or lack of PROM completion (n ¼ 4). No patients received osteochondral autograft transplantation within the study period.
Conclusions The PROMIS PF, PI, and Depression CATs performed better with respect to legacy PROMs in the postoperative period than the preoperative period. In addition, the PROMIS PF and PI measures performed best with respect to the IKDC questionnaire, and no floor or ceiling effects were identified for the PROMIS instruments. The PROMIS instruments may be more suited to track outcomes postoperatively than to establish preoperative baselines in cartilage surgery patients.
References 1. Hancock KJ, Glass N, Anthony CA, et al. Performance of PROMIS for healthy patients undergoing meniscal surgery. J Bone Joint Surg Am 2017;99:954-958. 2. MOTION Group. Patient-reported outcomes in orthopaedics. J Bone Joint Surg Am 2018;100:436-442. 3. Carr AJ. Evidence-based orthopaedic surgery: What type of research will best improve clinical practice? J Bone Joint Surg Br 2005;87:1593-1594. 4. Wang T, Wang DX, Burge AJ, et al. Clinical and MRI outcomes of fresh osteochondral allograft transplantation after failed cartilage repair surgery in the knee. J Bone Joint Surg Am 2018;100:1949-1959. 5. Tirico LEP, McCauley JC, Pulido PA, Demange MK, Bugbee WD. Is patient satisfaction associated with clinical outcomes after osteochondral allograft transplantation in the knee? Am J Sports Med 2018:363546518812420. 6. Ogura T, Ackermann J, Mestriner AB, Merkely G, Gomoll AH. The minimal clinically important difference and substantial clinical benefit in the patient-reported outcome measures of patients undergoing osteochondral allograft transplantation in the knee. Cartilage 2018: 1947603518812552. 7. Wang D, Chang B, Coxe FR, et al. Clinically meaningful improvement after treatment of cartilage defects of the knee with osteochondral grafts. Am J Sports Med 2018: 363546518808030. 8. Ogura T, Ackermann J, Barbieri Mestriner A, Merkely G, Gomoll AH. Minimal clinically important differences and substantial clinical benefit in patient-reported outcome measures after autologous chondrocyte implantation. Cartilage 2018:1947603518799839. 9. Conrozier T, Monet M, Lohse A, Raman R. Getting better or getting well? The patient acceptable symptom state (PASS) better predicts patient’s satisfaction than the decrease of pain, in knee osteoarthritis subjects treated with viscosupplementation. Cartilage 2018;9:370-377.
1427
10. Bekkers JE, de Windt TS, Raijmakers NJ, Dhert WJ, Saris DB. Validation of the Knee Injury and Osteoarthritis Outcome Score (KOOS) for the treatment of focal cartilage lesions. Osteoarthritis Cartilage 2009;17:1434-1439. 11. Roos EM, Toksvig-Larsen S. Knee injury and Osteoarthritis Outcome Score (KOOS)dValidation and comparison to the WOMAC in total knee replacement. Health Qual Life Outcomes 2003;1:17. 12. Escobar A, Quintana JM, Bilbao A, Arostegui I, Lafuente I, Vidaurreta I. Responsiveness and clinically important differences for the WOMAC and SF-36 after total knee replacement. Osteoarthritis Cartilage 2007;15:273-280. 13. Dowdle SB, Glass N, Anthony CA, Hettrich CM. Use of PROMIS for patients undergoing primary total shoulder arthroplasty. Orthop J Sports Med 2017;5: 2325967117726044. 14. Kadri O, Jildeh TR, Meldau JE, et al. How long does it take for patients to complete PROMIS scores?: An assessment of PROMIS CAT questionnaires administered at an ambulatory sports medicine clinic. Orthop J Sports Med 2018;6:2325967118791180. 15. Hung M, Saltzman CL, Greene T, et al. The responsiveness of the PROMIS instruments and the qDASH in an upper extremity population. J Patient Rep Outcomes 2017;1:12. 16. Scott EJ, Westermann R, Glass NA, Hettrich C, Wolf BR, Bollier MJ. Performance of the PROMIS in patients after anterior cruciate ligament reconstruction. Orthop J Sports Med 2018;6:2325967118774509. 17. Bernstein DN, Houck JR, Hammert WC. A comparison of PROMIS UE versus PF: Correlation to PROMIS PI and depression, ceiling and floor effects, and time to completion. J Hand Surg Am 2019;44:901.e1-901.e7. 18. Bernholt D, Wright RW, Matava MJ, Brophy RH, Bogunovic L, Smith MV. Patient Reported Outcomes Measurement Information System scores are responsive to early changes in patient outcomes following arthroscopic partial meniscectomy. Arthroscopy 2018;34: 1113-1117. 19. Senders A, Hanes D, Bourdette D, Whitham R, Shinto L. Reducing survey burden: Feasibility and validity of PROMIS measures in multiple sclerosis. Mult Scler 2014;20:1102-1111. 20. Amtmann D, Cook KF, Jensen MP, et al. Development of a PROMIS item bank to measure pain interference. Pain 2010;150:173-182. 21. Sherman SLG, Andrew J, Voss EE. Correlation of Patient Reported Outcome Measurement Information System Computer Adaptive Test (PROMIS-CAT) and Knee Injury and Osteoarthritis Outcome Score (KOOS) in knee osteotomy and cartilage procedures. Orthop J Sports Med 2018;6:2325967118S00102 (suppl 4). 22. Fabricant PD. Editorial Commentary: We PROMIS...one more patient-reported outcome questionnaire! Arthroscopy 2018;34:1118-1120. 23. Kohan EM, Hill JR, Schwabe M, Aleem AW, Keener JD, Chamberlain AM. The influence of mental health on Patient-Reported Outcomes Measurement Information System (PROMIS) and traditional outcome instruments in patients with symptomatic glenohumeral arthritis. J Shoulder Elbow Surg 2019;28:e40-e48.
1428
B. U. NWACHUKWU ET AL.
24. Chen RE, Papuga MO, Voloshin I, et al. Preoperative PROMIS scores predict postoperative outcomes after primary ACL reconstruction. Orthop J Sports Med 2018;6: 2325967118771286. 25. Karns MR, Jones DL, Todd DC, et al. Patient- and procedure-specific variables driving total direct costs of outpatient anterior cruciate ligament reconstruction. Orthop J Sports Med 2018;6:2325967118788543. 26. Patterson BM, Orvets ND, Aleem AW, et al. Correlation of Patient-Reported Outcomes Measurement Information System (PROMIS) scores with legacy patient-reported outcome scores in patients undergoing rotator cuff repair. J Shoulder Elbow Surg 2018;27:S17-S23. 27. Anthony CA, Glass N, Hancock K, Bollier M, Hettrich CM, Wolf BR. Preoperative Performance of the PatientReported Outcomes Measurement Information System in patients with rotator cuff pathology. Arthroscopy 2017;33:1770-1774.e1. 28. Sheean AJ, Schmitz MR, Ward CL, et al. Assessment of disability related to femoroacetabular impingement syndrome by use of the Patient-Reported Outcome Measure Information System (PROMIS) and objective measures of physical performance. Am J Sports Med 2017;45: 2476-2482. 29. Beletsky A, Nwachukwu BU, Manderle BJ, et al. The impact of workers’ compensation on Patient-Reported Outcomes Measurement Information System upper extremity and legacy outcome measures in patients undergoing arthroscopic rotator cuff repair. Arthroscopy 2019;35: 2817-2824. 30. Carender CN, Bollier MJ, Wolf BR, Duchman KR, An Q, Westermann RW. Preoperative performance of PROMIS in patients with patellofemoral malalignment and chondral disease. Orthop J Sports Med 2019;7: 2325967119855001. 31. Makhni EC, Meldau JE, Blanchett J, et al. Correlation of PROMIS physical function, pain interference, and depression in pediatric and adolescent patients in the ambulatory sports medicine clinic. Orthop J Sports Med 2019;7:2325967119851100. 32. Fries J, Rose M, Krishnan E. The PROMIS of better outcome assessment: Responsiveness, floor and ceiling effects, and Internet administration. J Rheumatol 2011;38: 1759-1764. 33. Hey HWD, Luo N, Chin SY, et al. The predictive value of preoperative health-related quality-of-life scores on postoperative patient-reported outcome scores in lumbar spine surgery. Global Spine J 2018;8:156-163. 34. Anthony CA, Glass NA, Hancock K, Bollier M, Wolf BR, Hettrich CM. Performance of PROMIS instruments in patients with shoulder instability. Am J Sports Med 2017;45:449-453. 35. Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60: 34-42.
36. Selim AJ, Rogers W, Qian SX, Brazier J, Kazis LE. A preference-based measure of health: The VR-6D derived from the veterans RAND 12-Item Health Survey. Qual Life Res 2011;20:1337-1347. 37. Oak SR, Strnad GJ, Bena J, et al. Responsiveness comparison of the EQ-5D, PROMIS Global Health, and VR-12 questionnaires in knee arthroscopy. Orthop J Sports Med 2016;4:2325967116674714. 38. Middel B, van Sonderen E. Statistical significant change versus relevant or important change in (quasi) experimental design: Some conceptual and methodological problems in estimating magnitude of intervention-related change in health services research. Int J Integr Care 2002;2:e15. 39. Kollmorgen RC, Hutyra CA, Green C, Lewis B, Olson SA, Mather RC III. Relationship between PROMIS computer adaptive tests and legacy hip measures among patients presenting to a tertiary care hip preservation center. Am J Sports Med 2019:363546518825252. 40. Hung M, Voss MW, Bounsanga J, Gu Y, Granger EK, Tashjian RZ. Psychometrics of the Patient-Reported Outcomes Measurement Information System physical function instrument administered by computerized adaptive testing and the Disabilities of Arm, Shoulder and Hand in the orthopedic elbow patient population. J Shoulder Elbow Surg 2018;27:515-522. 41. Hancock KJ, Glass N, Anthony CA, et al. PROMIS: A valid and efficient outcomes instrument for patients with ACL tears. Knee Surg Sports Traumatol Arthrosc 2019;27: 100-104. 42. Lyman S, Lee YY, Franklin PD, Li W, Cross MB, Padgett DE. Validation of the KOOS, JR: A short-form knee arthroplasty outcomes survey. Clin Orthop Relat Res 2016;474:1461-1471. 43. Collins NJ, Misra D, Felson DT, Crossley KM, Roos EM. Measures of knee function: International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form, Knee Injury and Osteoarthritis Outcome Score (KOOS), Knee Injury and Osteoarthritis Outcome Score Physical Function Short Form (KOOS-PS), Knee Outcome Survey Activities of Daily Living Scale (KOS-ADL), Lysholm Knee Scoring Scale, Oxford Knee Score (OKS), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), Activity Rating Scale (ARS), and Tegner Activity Score (TAS). Arthritis Care Res (Hoboken) 2011;63:S208-S228 (suppl 11). 44. van de Graaf VA, Wolterbeek N, Scholtes VA, Mutsaerts EL, Poolman RW. Reliability and validity of the IKDC, KOOS, and WOMAC for patients with meniscal injuries. Am J Sports Med 2014;42: 1408-1416. 45. Strong B, Maloney M, Baumhauer J, et al. Psychometric evaluation of the Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function and Pain Interference Computer Adaptive Test for subacromial impingement syndrome. J Shoulder Elbow Surg 2019;28:324-329.
Purpose: To define the psychometric properties of the Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function (PF), Pain Interference (PI), and Depression computer adaptive tests (CATs) in patients undergoing knee cartilage surgical procedures. Methods: The PROMIS PF, PI, and Depression CATs were administered preoperatively and at 6 months alongside legacy knee patient-reported outcome measures (PROMs) in patients undergoing knee cartilage surgical procedures. Statistical analysis consisted of the time to completion, psychometric analysis for correlative strengths, absolute and relative floor and ceiling effects, and Cohen effect size. Results: Our study included 250 patients (57.2% male patients), averaging 1.87, 1.53, and 1.91 minutes for completion of the PF, PI, and Depression CATs, respectively. Preoperatively, the PROMIS PF and PI CATs showed wide ranges of correlation coefficients with respect to function (r ¼ 0.14-0.72 and r ¼ 0.29-0.77, respectively) and health-related quality-of-life PROMs (r ¼ 0.640.70). At 6 months, the PROMIS PF CAT (r ¼ 0.82-0.93) and PI CAT (r ¼ 0.77-0.93) both exhibited excellent correlations with respect to legacy function and health-related quality-of-life PROMs except for the Marx Activity Rating Scale (r ¼ 0.36-0.44). None of the PROMIS instruments exhibited any significant floor or ceiling effects. Conclusions: The PROMIS PF, PI, and Depression CATs performed better with respect to legacy PROMs in the postoperative period than the preoperative period. In addition, the PROMIS PF and PI measures performed best with respect to the International Knee Documentation Committee questionnaire, and no floor or ceiling effects were identified for the PROMIS instruments. The PROMIS instruments may be more suited to track outcomes postoperatively than to establish preoperative baselines in cartilage surgery patients. Level of Evidence: Level IV, retrospective case series.
See commentary on page 1429
From the Division of Sports Medicine, Midwest Orthopaedics at Rush, Rush University Medical Center, Chicago, Illinois, U.S.A. The authors report the following potential conflicts of interest or sources of funding: B.F. receives paid royalties from Elsevier; research support from Arthrex; owns stock or stock options in Jace Medical; receives fellowship support from Smith & Nephew and Ossur; and is a consultant for and receives research support from Stryker. N.V. receives research support from Arthrex and DJ Orthopaedics; receives publishing royalties and material support from Arthroscopy; receives personal fees from Orthospace; and receives publishing royalties from Vindico Medical-Orthopedics Hyperguide. A.Y. receives research support from Arthrex and NuTech and hospitality fees from Smith & Nephew. B.J.C. receives research support from Aesculap/B.Braun, National Institutes of Health (NIAMS and NICHD), and Zimmer; is on the editorial or governing board of the American Journal of Orthopedics, American Journal of Sports Medicine, Cartilage, Journal of Shoulder and Elbow Surgery, and Journal of the American Academy of Orthopaedic Surgeons; receives IP royalties and research support from Arthrex; is a board or committee
member of the Arthroscopy Association of North America and International Cartilage Repair Society; receives other financial or material support from Athletico, JRF Ortho, and Smith & Nephew; receives IP royalties from Elsevier Publishing; receives publishing royalties from Operative Techniques in Sports Medicine; receives stock or stock options from Ossio; and receives stock or stock options and research support from Regentis. Full ICMJE author disclosure forms are available for this article online, as supplementary material. Received May 23, 2019; accepted January 16, 2020. Address correspondence to Nikhil Verma, M.D., Midwest Orthopaedics at Rush, Rush University Medical Center, 1611 W Harrison St, Ste 300, Chicago, IL 60612, U.S.A. E-mail: [email protected] Ó 2020 by the Arthroscopy Association of North America 0749-8063/19641/$36.00 https://doi.org/10.1016/j.arthro.2020.01.036
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol 36, No 5 (May), 2020: pp 1419-1428
1419
1420
O
B. U. NWACHUKWU ET AL.
utcomes research in orthopaedic surgery has evolved significantly in recent history, improving from variable single-surgeon reports to standardized, domain-based questionnaires completed by individual patients.1 Patient-reported outcome measure (PROM) data have been used in conjunction with physical examination and imaging techniques to define clinical outcomes after cartilage procedures.1-5 Furthermore, by connecting PROM data with specific questionnaires defining self-perceived improvement and satisfaction, researchers have been able to define PROM threshold values above which most patients report true subjective improvement after specific cartilage surgical procedures and viscosupplementation.6-9 However, significant interest in domain-based PROM data has led to important secondary limitations including (1) increased variability in PROM selection between studies, (2) variable use of PROM data longitudinally to track outcomes, (3) failure to include all relevant domains of interest (e.g., function, health-related quality of life [HRQoL], and mental health), and (4) significant question burden for patients completing numerous PROMs at once. The Patient-Reported Outcomes Measurement Information System (PROMIS), an initiative introduced in 2004, aims to address some of the limitations in patientreported outcome research. Specifically, certain legacy PROMs may have significant floor or ceiling effects in specific patient populations, require full questionnaire completion to produce a score, and ask predefined questions of unknown discriminative value.1,10-12 PROMIS measures such as the Physical Function (PF), Pain Interference (PI), and Upper Extremity computer adaptive tests (CATs) have shown decreased question requirements, high instrument responsiveness, decreased time to completion (TTC), and possibly early responsiveness to changes in clinical status.13-18 In addition, by using CATs, the PROMIS instruments select questions of high discriminative value based on previous responses.13,19-21 Given the inherent appeal of PROMIS, there is increasing interest in evaluating the PROMIS instruments for musculoskeletal conditions.22 Several published studies have shown that PROMIS correlates well at preoperative time points with legacy measures for anterior cruciate ligament injury, meniscal injury, rotator cuff pathology, femoroacetabular impingement, and glenohumeral arthritis.18,23-28 No prior study has examined the psychometric properties of the PROMIS instruments longitudinally in knee cartilage patients, and only a single study has examined the performance of CATs in cartilage procedures.21 Thus, for knee cartilage surgical procedures, there is a need to examine the performance of the PROMIS instruments relative to legacy measures to (1) validate these instruments in this patient population; (2)
investigate any time-dependent nature associated with PROMIS (e.g., preoperative vs postoperative); and (3) determine whether previously defined relations with respect to question burden, responsiveness, and TTC are maintained.22 The purpose of this study was to define the psychometric properties of the PROMIS PF, PI, and Depression CATs in patients undergoing knee cartilage surgical procedures. We hypothesized that (1) both the PROMIS PF CAT and PI CAT would exhibit good to excellent correlations relative to function and HRQoL legacy instruments at both preoperative and 6-month time points,29-31 (2) the PF and PI CATs would exhibit fair to good correlations with mental health legacy PROMs across time points,1,13 (3) the PROMIS Depression CAT would exhibit fair to very good correlations with legacy PROMs across time points,29 and (4) all PROMIS measures would not show any significant absolute or relative floor or ceiling effects.29,30,32
Methods Study Design and Cohort Establishment Prior to study initiation, the study protocol was reviewed and approved by an institutional review board with a corresponding waiver of consent. Prospectively collected PROM data across multiple orthopaedic surgeons at a single institution were aggregated via an institutional electronic registry (Outcome Based Electronic Research Database; Universal Research Solutions, Columbia, MO). The cartilage cohort was constructed by using Current Procedural Terminology (CPT) codes to identify patients undergoing articular cartilage debridement (CPT codes 29876 and 29877), osteochondral allograft transplant (OCA) (CPT code 27415), or microfracture (CPT code 29879) between December 2017 and August 2018. The inclusion criteria included receipt of a primary cartilage procedure (i.e., debridement, allograft transplantation, microfracture, or autologous chondrocyte implantation [ACI]) for a focal cartilage defect and full completion of preoperative and 6-month PROM data. The 6-month follow-up time point was defined based on the date of surgery, with patients undergoing follow-up as early as 3 weeks prior to 6 months and as late as 3 weeks after the 6-month follow-up point. The exclusion criteria included concomitant ligamentous, meniscal, or bony procedures; biological augmentation; receipt of total knee arthroplasty within the follow-up period; or failure to complete preoperative PROMs. Demographic data collected included age, sex, insurance provider, and Workers’ Compensation status. During the study time frame, no patient undergoing ACI (CPT code 27412) or osteochondral autograft transplantation (OAT) (CPT code 29866) met the inclusion criteria.
1421
OUTCOMES IN KNEE CARTILAGE PATIENTS Table 1. Demographic Characteristics by Cartilage Procedure Age, yr WC Sex Male Female Laterality Right Left Preoperative PROM PROMIS PF Pain Depression BRS IKDC KOOS ADL JR Pain PS QOL Sport Symptom Marx Activity Rating Scale SF-12 MCS PCS VR-12 MCS PCS VR6D WOMAC Function Pain Stiffness Total
ACD OCA 41.2 15.1 29.8 9.2 22 (11.3) 2 (6.1) 113 (58.2) 81 (41.8)
14 (42.4) 19 (57.6)
105 (54.1) 89 (45.9)
18 (54.5) 15 (45.5)
MFX P Value 26.2 11.0 <.01* 0 (0.0) .22 .11 16 (69.6) 7 (30.4) .09 7 (30.4) 16 (69.6)
41.6 59.4 46.0 3.9 43.2
7.4 6.8 7.3 0.6 16.5
38.5 62.1 48.9 3.9 36.0
6.5 7.0 7.4 0.5 16.3
41.7 59.2 51.8 3.8 43.0
11.4 8.5 5.7 0.6 16.6
.14 .06 .01* .71 .15
66.0 57.9 58.2 40.2 29.6 32.9 56.1 7.5
20.2 15.3 18.4 15.3 19.4 24.2 20.3 6.5
64.3 55.0 55.0 42.7 15.0 23.3 59.5 6.5
21.7 14.2 19.2 14.7 15.3 19.6 17.3 7.1
70.2 59.1 59.6 39.1 29.5 32.5 61.7 11.4
25.7 16.3 22.1 14.5 20.3 21.2 17.5 5.6
.63 .61 .62 .64 <.01* .10 .39 .04*
Outcome Score (KOOS) Joint Replacement (JR), Physical Function, Symptoms, Pain, Activities of Daily Living, Sport, and Quality of Life; Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) Pain, Stiffness, Function, and Total; Marx Activity Rating Scale; Brief Resilience Scale; Veteran-RAND 12item Health Survey (VR-12); 12-item Short Form Health Survey (Short Form 12 [SF-12]); and Veterans RAND 6 Domain (VR6D). The VR-12 and SF-12 instruments were split into the Physical Component Scores and Mental Component Scores (MCSs) for subsequent analysis. PROMs were organized into function, HRQoL, and mental health domains to allow for appropriate comparison.1,33
NOTE. Data are presented as mean standard deviation or number (percentage). P values represent results from the t test or c2 test for demographic variables and 1-way analysis of variance for preoperative score values. ACD, articular cartilage debridement; ADL, Activities of Daily Living; BRS, Brief Resilience Scale; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; MFX, microfracture; OCA, osteochondral allograft transplant; PCS, Physical Component Score; PF, Physical Function; PROM, patient-reported outcome measure; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WC, Workers’ Compensation; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Statistically significant (P < .05).
Statistical Analysis: Instrument Performance, Responsiveness, and Floor and Ceiling Effects A priori sample size calculations assuming a type I error rate of 5% and a power (1 e b) of 0.90 yielded a minimum sample size of 164 assuming a small effect size. Institutional registry data were first reviewed for calculation of the average number of questions completed per outcome instrument (i.e., question burden) as well as the average TTC for a given instrument in minutes. Spearman rank correlation coefficients were used to examine the psychometric properties of the PROMIS PF, PI, or Depression CAT relative to legacy PROMs. Correlation coefficients were classified by the strength of association exhibited, with greater than 0.8 equating to excellent; 0.71 to 0.8, very good; 0.61 to 0.7, good; 0.41 to 0.6, fair; and 0.21 to 0.4, poor.1,34 Both absolute and relative floor and ceiling effects were assessed by calculating the percentage of respondents who obtained the absolute and relative highest and lowest possible scores, respectively, on a given PROM. In the case of relative floor and ceiling effects, a score distribution of 5% from the relative minimum and maximum scores was calculated to qualify a score in the relative floor or ceiling range. A percentage of 15% or greater was designated as a significant absolute or relative floor or ceiling effect.34-36 Internal responsiveness was evaluated by calculation of the Cohen d, with an effect size of less than 0.20 corresponding to a negligible effect size; 0.20 to 0.49, small effect size; 0.50 to 0.80, moderate effect size; and greater than 0.80, large effect size.37,38 Subgroup analysis was performed for PROMs showing ceiling or floor effects to examine the effect of age and sex on relative and absolute ceiling achievement.
Patient-Reported Outcome Measures The PROMs examined in this study included the following: PROMIS PF (version 2.0) CAT; PROMIS Pain (version 1.1) CAT; PROMIS Depression (version 1.0) CAT; International Knee Documentation Committee (IKDC) questionnaire; Knee Injury and Osteoarthritis
A total of 250 patients were included in this study. Demographic data and preoperative PROM scores are summarized in Table 1. The articular cartilage debridement cohort was significantly older than the OCA and microfracture cohorts (P < .01). No significant
54.1 9.7 36.8 9.6
53.8 7.8 33.4 9.2
37.6 9.7 37.2 6.8
56.5 9.2 55.4 8.0 58.0 8.1 39.0 10.0 35.1 10.3 39.4 10.1 0.67 0.11 0.63 0.1 0.68 0.1 65.1 65.8 55.7 64.0
23.2 21.7 28.1 23.4
67.3 64.5 60.3 64.4
19.7 20.4 22.7 18.4
60.5 58.3 63.5 64.7
26.4 23.4 20.3 20.0
.75 .20 .65 .17 .24 .69 .52 .48 .91
Results
1422
B. U. NWACHUKWU ET AL.
Table 2. Performance of PROMIS Physical Function CAT in Cartilage Patients Preoperative PROM Function PROMIS Pain IKDC KOOS ADL JR Pain PS Sport Symptom Marx Activity Rating Scale WOMAC Function Pain Stiffness Total HRQoL KOOS QOL SF-12 PCS VR-12 PCS VR6D Mental health PROMIS Depression VR-12 MCS SF-12 MCS BRS
6 mo
r
Correlative Classification
r
Correlative Classification
e0.64* 0.72*
Good Very good
e0.85* 0.90*
Excellent Excellent
0.61* 0.55* 0.57* e0.60* 0.66* 0.32* 0.14*
Good Fair Fair Fair Good Poor Very poor
0.88* 0.92* 0.88* e0.82* 0.82* 0.80* 0.44*
Excellent Excellent Excellent Excellent Excellent Very good Fair
0.58* 0.52* 0.37* 0.61*
Fair Fair Poor Good
0.88* 0.93* 0.80* 0.90*
Excellent Excellent Very good Excellent
0.65* 0.69* 0.70* 0.64*
Good Good Good Good
0.91* 0.86* 0.87* 0.83*
Excellent Excellent Excellent Excellent
e0.28* 0.32* 0.12* 0.13*
Poor Poor Very poor Very poor
e0.45* 0.47* 0.27*
Fair Fair Poor
ADL, Activities of Daily Living; BRS, Brief Resilience Scale; CAT, computer adaptive test; HRQoL, health related quality of life; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PROM, patient-reported outcome measure; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR-12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Significant correlation (P < .05).
differences were observed with respect to Workers’ Compensation status, sex, or laterality. Regarding questionnaire burden, patients averaged a total of 4.59 questions for the PROMIS PF (mean TTC, 1.87 minutes), 4.28 questions for the PROMIS PI (mean TTC, 1.52 minutes), 4 questions for the Marx Activity Rating Scale (mean TTC, 1.53 minutes), and 6.25 questions for the PROMIS Depression (mean TTC, 1.91 minutes) and Brief Resilience Scale (mean TTC, 2.10 minutes) instruments. The IKDC questionnaire (mean TTC, 5.61 minutes), KOOS instruments (mean TTC, 6.52 minutes), and VR-12 or SF-12 instruments (mean TTC, 6.12 minutes) showed average TTC values in excess of 5 minutes (Table 1). Preoperative and 6-month postoperative correlation coefficients between legacy PROMs and the PROMIS PF CAT are summarized in Table 2. With respect to function PROMs, the IKDC questionnaire showed the best (very good) preoperative correlation (r ¼ 0.72) whereas the WOMAC instruments (r ¼ 0.37-0.61) and KOOS instruments (r ¼ 0.32-0.66) exhibited poor to good correlations relative to the PF CAT. All HRQoL PROMs exhibited good preoperative correlations with respect to
the PROMIS PF CAT (r ¼ 0.64-0.70). Mental health measures exhibited very poor to poor preoperative correlations relative to the PROMIS PF CAT (r ¼ 0.12-0.32). Every single PROM showed improved correlation with the PROMIS PF CAT at the 6-month assessment. At 6 months postoperatively, the PROMIS PF CAT showed very good to excellent correlations (r ¼ 0.82-0.93 [absolute value]) with legacy functional outcome PROMs except for the Marx Activity Rating Scale, which exhibited a fair correlation (r ¼ 0.44). All HRQoL PROMs exhibited excellent correlations (r ¼ 0.83-0.91) with the PROMIS PF CAT, whereas the mental health measures showed fair or poor correlations (r ¼ 0.27-0.47) (Table 2). The IKDC questionnaire exhibited the strongest preoperative correlation (very good) with respect to the PROMIS PI CAT (r ¼ 0.77). The KOOS instruments exhibited fair to very good preoperative correlations with the PROMIS PI CAT (r ¼ 0.59-0.77). The WOMAC instruments exhibited good to very good correlations relative to the PI CAT (r ¼ 0.62-0.72), and the Marx Activity Rating Scale was the only function measure to exhibit a poor correlation preoperatively (r ¼ 0.29).
1423
OUTCOMES IN KNEE CARTILAGE PATIENTS Table 3. Performance of PROMIS Pain Interference CAT in Cartilage Patients Preoperative PROM Function PROMIS PF CAT IKDC KOOS ADL JR Pain PS Sport Symptom Marx Activity Rating Scale WOMAC Function Pain Stiffness Total HRQoL KOOS QOL SF-12 PCS VR-12 PCS VR6D Mental health PROMIS Depression VR-12 MCS SF-12 MCS BRS
6 mo
r
Correlative Classification
r
Correlative Classification
e0.64* e0.77*
Good Very good
e0.85* e0.88*
Excellent Excellent
e0.71* e0.73* e0.77* 0.71* e0.72* e0.59* e0.29*
Very good Very good Very good Very good Very good Fair Poor
e0.93* e0.90* e0.83* 0.85* e0.78* e0.83* e0.36*
Excellent Excellent Excellent Excellent Very good Excellent Poor
e0.71* e0.71* e0.62* e0.72*
Very good Very good Good Very good
e0.93* e0.88* e0.81* e0.92*
Excellent Excellent Excellent Excellent
e0.64* e0.65* e0.69* e0.70*
Good Good Good Good
e0.86* e0.81* e0.84* e0.77*
Excellent Excellent Excellent Very good
0.27* e0.44* e0.32* e0.16*
Poor Fair Poor Very poor
0.48* e0.45* e0.26*
Fair Fair Poor
ADL, Activities of Daily Living; BRS, Brief Resilience Scale; CAT, computer adaptive test; HRQoL, health related quality of life; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PF, Physical Function; PROM, patient-reported outcome measure; PROMIS, PatientReported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Significant correlation (P < .05).
HRQoL instruments exhibited good correlations with the PROMIS PI CAT (r ¼ 0.64-0.70). Correlation coefficients for each function and HRQoL measure relative to the PROMIS PI CAT improved from preoperatively to 6 months postoperatively. The IKDC questionnaire (r ¼ 0.88) and WOMAC instruments (r ¼ 0.81-0.93) all exhibited excellent correlations relative to the PI CAT at 6 months. The KOOS subscales (r ¼ 0.78-0.93) and HRQoL PROMs (r ¼ 0.77-0.86) exhibited very good to excellent correlations at 6 Table 4. Performance of PROMIS Depression CAT in Cartilage Patients Preoperative PROM BRS SF-12 MCS VR-12 MCS
r e0.41* e0.49* e0.58*
Correlative Classification Fair Fair Fair
6 mo r
Correlative Classification
e0.70* e0.78*
Good Very good
BRS, Brief Resilience Scale; CAT, computer adaptive test; MCS, Mental Component Score; PROM, patient-reported outcome measure; PROMIS, Patient-Reported Outcomes Measurement Information System; SF-12, Short Form 12; VR-12, Veterans RAND 12. *Significant correlation (P < .05).
months. Mental health PROMs exhibited very poor to fair correlations preoperatively (r ¼ 0.16-0.44) and poor to fair correlations at 6 months (r ¼ e0.26 to 0.45) (Table 3). Table 4 summarizes the performance of the PROMIS Depression CAT relative to mental health legacy instruments. Preoperatively, mental health PROMs exhibited fair correlations relative to the PROMIS Depression CAT (r ¼ 0.41-0.58). At 6 months, correlation coefficients improved to good or very good (r ¼ 0.70-0.78) (Table 4). No significant absolute or relative floor or ceiling effects were observed for the PROMIS Depression, PF, or PI CAT. The Marx Activity Rating Scale showed a significant absolute floor effect (n ¼ 75, 33%) preoperatively, as well as an absolute floor effect at 6 months postoperatively (n ¼ 24, 35.8%). The Marx instrument showed absolute ceiling effects preoperatively and at 6 months (Table 5). A relative floor effect was observed in KOOS Sport preoperatively (n ¼ 53, 20.5%) as well as at 6 months (Table 6). The IKDC questionnaire (0.53), KOOS Quality of Life (0.52), and VR-12 Physical Component Score (0.51) showed medium effect sizes (Table 7).
1424
B. U. NWACHUKWU ET AL.
Table 5. Absolute Floor and Ceiling Effects Preoperative (%) Instrument
Absolute Floor
PROMIS Depression PF PI BRS IKDC KOOS ADL JR Pain PS QOL Sport Symptom Marx Activity Rating Scale SF-12 PCS MCS VR-12 PCS MCS VR6D WOMAC Function Stiffness Pain Total
0 0 0 0 0 1 2 2 1 29 34 0 75
6 mo (%)
Absolute Ceiling
(0.0) (0.0) (0.0) (0.0) (0.0)
0 0 0 26 0
(0.0) (0.0) (0.0) (10.4) (0.0)
(0.4) (0.8) (0.8) (0.4) (11.6) (13.6) (0.0) (30.0)*
6 2 1 4 0 2 3 55
(2.4) (0.8) (0.4) (1.6) (0.0) (0.8) (1.2) (22.0)*
Absolute Floor
Absolute Ceiling
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0)
0 (0.0)
0 0 0 0 2 5 0 24
(0.0) (0.0) (0.0) (0.0) (8.0) (20)* (0.0) (35.8)*
1 0 1 0 0 0 0 8
(4.0) (0.0) (4.0) (0.0) (0.0) (0.0) (0.0) (12)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 0 0 0
1 1 3 1
2 4 10 1
(0.8) (1.6) (4.0) (0.4)
7 4 20 2
(2.8) (1.6) (8.0) (0.8)
(0.0) (0.0) (0.0) (0.0)
(4.0) (4.0) (12) (4.0)
ADL, Activities of Daily Living; BRS, Brief Resilience Scale; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PF, Physical Function; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR-12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Significant floor or ceiling effect.
Discussion The main finding of this study was that the correlation of the PROMIS PF and PI CATs relative to legacy function and HRQoL instruments improves from preoperative assessment to 6-month postoperative assessment. Specifically, the correlation of the PROMIS PF CAT compared with legacy function PROMs improved from a range of fair to very good to a range of very good to excellent (excluding the Marx Activity Rating Scale). The performance of the PROMIS PF CAT compared with legacy HRQoL measures improved from good to excellent. A similar trend was found with the PROMIS PI CAT. Notably, of the assessed legacy PROMs, the PROMIS tracked the most consistently with the IKDC questionnaire. In addition, we have confirmed that PROMIS PF, PI, and Depression CATs maintained low questionnaire burden relative to the IKDC, KOOS, and VR-12 or SF-12 instruments, and average completion times were less than 2 minutes for each PROMIS CAT. Finally, none of the 3 PROMIS CATs showed absolute or relative floor or ceiling effects in our cohort, and each showed a small effect size. These findings allow us
to refine our hypothesis and advance our understanding of the performance of the PROMIS instruments in patients undergoing knee cartilage surgery. The psychometric properties of the PROMIS PF CAT have been studied across a variety of orthopaedic patient populations.13,17,39,40 In this study, the PROMIS PF CAT showed a very good correlation with the IKDC questionnaire, poor to good correlations with the KOOS and WOMAC instruments, and good correlations with the HRQoL instruments preoperatively. At 6 months, correlations improved to very good to excellent for all function and HRQoL instruments, excluding the Marx Activity Rating Scale, which improved from very poor to fair. Hancock et al.41 reported comparable performance metrics for the PROMIS PF CAT relative to KOOS subscores in an anterior cruciate ligament reconstruction (ACLR) cohort, with strength ranging from fair to very good. They also showed that the Marx Activity Rating Scale exhibited the weakest correlation coefficient with the PROMIS PF CAT, a relation most likely resulting from differences in instrument design and domain of assessment. The PROMIS PF CAT,
1425
OUTCOMES IN KNEE CARTILAGE PATIENTS Table 6. Relative Floor and Ceiling Effects Preoperative (%) Instrument PROMIS Depression PF PI BRS IKDC KOOS ADL JR Pain PS QOL Sport Symptom Marx Activity Rating Scale SF-12 PCS MCS VR-12 PCS MCS VR6D WOMAC Function Stiffness Pain Total
Relative Floor 0 0 0 1 1 3 2 2 1 29 53 0 86
6 mo (%)
Relative Ceiling
(0.0) (0.0) (0.0) (0.4) (0.4)
0 0 0 38 0
(0.0) (0.0) (0.0) (14.7) (0.0)
(1.2) (0.8) (0.8) (0.4) (11.2) (20.5)* (0.0) (33.3)*
18 2 1 4 0 3 6 65
(7.0) (0.8) (0.4) (1.6) (0.0) (1.2) (2.3) (25.2)*
Relative Floor
Relative Ceiling
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
1 (0.4)
0 (0.0)
0 0 0 0 2 5 0 24
(0.0) (0.0) (0.0) (0.0) (8.0) (20)* (0.0) (35.8)*
4 0 2 0 0 1 0 8
(16)* (0.0) (8.0) (0.0) (0.0) (4.0) (0.0) (12)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 (0.0) 0 (0.0) 0 (0.0)
0 0 0 0
4 1 6 2
3 10 4 3
(1.2) (3.9) (1.6) (1.2)
15 20 16 9
(5.8) (7.8) (9.0) (3.5)
(0.0) (0.0) (0.0) (0.0)
(16)* (4.0) (24)* (8.0)
ADL, Activities of Daily Living; BRS, Brief Resilience Scale; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PF, Physical Function; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR-12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Significant floor or ceiling effect.
similarly to the IKDC questionnaire, assesses function by asking patients how much difficulty they experience performing specific activities (e.g., standing, pulling, and exercising), whereas the Marx instrument consists of 4 questions assessing activity level based on the frequency of sport-related activities (e.g., cutting or pivoting). In a similar study, Scott et al.16 examined the psychometric properties of the PF CAT over a 2-year follow-up period in ACLR patients, reporting good to excellent correlations with the KOOS Sport, KOOS Activities of Daily Living, and SF-36 Physical Function instruments. Both the PROMIS PF CAT and PI CAT exhibited the weakest correlations with mental health measures in our study, a finding consistent with previous work by Hancock et al.1 examining the psychometric properties of the PF CAT in patients undergoing meniscal surgery. We found that the most consistent correlation between PROMIS and legacy function PROMs was for the IKDC questionnaire. Similarities in instrument design may in part explain this relation, given that both instruments have 4 to 5 graded responses corresponding
to increasing difficulty performing specific activities. Prior literature has shown favorable psychometric qualities for the IKDC questionnaire; however, the KOOS and KOOS JR have been recommended in the assessment of degenerative knee conditions.42-44 Our study findings raise questions about the role of PROMIS in relation to legacy PROMs. In theory, PROMIS seeks to replace legacy PROMs that present high questionnaire burden and fatigue. We found variable correlations for the PROMIS instruments with legacy measures at preoperative time points. This finding may suggest that the PROMIS instruments can appropriately substitute for instruments with demonstrable very good correlations (e.g., IKDC questionnaire, with very good and excellent correlations) but should not replace instruments with less favorable correlations (e.g., KOOS JR, with fair correlation at preoperative assessment). In this regard, our study differs from prior work on PROMIS because we do not recommend the wholescale abandonment of legacy measures in favor of PROMIS. Rather, the PROMIS instruments may more appropriately be administered alongside select disease-
1426
B. U. NWACHUKWU ET AL.
Table 7. Effect Sizes Instrument PROMIS Depression PF PI IKDC KOOS ADL JR Pain PS QOL Sport Symptom Marx Activity Rating Scale SF-12 PCS MCS VR-12 PCS MCS VR6D WOMAC Function Stiffness Pain Total
Mean Effect Size (95% CI)
Magnitude
e0.41 e0.30 0.21 e0.53
(e1.02 (e0.64 (e0.41 (e1.06
to to to to
0.21) 0.04) 0.84) 0.01)
Small Small Small Medium*
e0.38 e0.40 e0.50 e0.09 e0.52 e0.37 e0.41 0.27
(e0.96 (e0.99 (e1.08 (e0.44 (e1.09 (e0.95 (e0.97 (e0.08
to to to to to to to to
0.19) 0.18) 0.08) 0.26) 0.06) 0.20) 0.15) 0.61)
Small Small Small Negligible Medium* Small Small Small
e0.44 (e0.80 to e0.09) 0.24 (e0.11 to 0.59)
Small Small
e0.51 (e0.87 to e0.15) 0.05 (e0.31 to 0.40) e0.38 (e0.74 to e0.02)
Medium* Negligible Small
e0.28 e0.15 e0.31 e0.29
(e0.89 (e0.76 (e0.92 (e0.90
to to to to
0.33) 0.46) 0.30) 0.33)
Small Negligible Small Small
ADL, Activities of Daily Living; CI, confidence interval; IKDC, International Knee Documentation Committee; JR, Joint Replacement; KOOS, Knee Injury and Osteoarthritis Outcome Score; MCS, Mental Component Score; PCS, Physical Component Score; PF, Physical Function; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System; PS, Physical Function Short Form; QOL, Quality of Life; SF-12, Short Form 12; VR-12, Veterans RAND 12; VR6D, Veterans RAND 6 Domain; WOMAC, Western Ontario and McMaster Osteoarthritis Index. *Medium or large effect size.
and function-specific PROMs to aid in outcome discrimination, particularly if instruments with a propensity toward floor and ceiling effects are used. It is worth noting that several of the evaluated legacy PROMs showed significant floor and/or ceiling effects (i.e., Marx Activity Rating Scale and KOOS subscales). This finding confirms the ability of PROMIS to distinguish outcomes between particularly high-functioning and low-functioning patients. Scott et al.16 suggested that because of limited floor and ceiling effects, the PROMIS PF measure was particularly well suited to tracking outcomes after ACLR, particularly in a young, active population known to experience significant ceiling effects when tracked with legacy measures. Conversely, they noted that there were particular legacy PROMs that may be used to assess specific domains of interest (e.g., use of the Marx Activity Rating Scale to evaluate postoperative rehabilitation).16 This notion is supported by our study findings in that legacy measures appear to be better suited for the preoperative assessment of joint-specific disability. Postoperatively,
however, numerous legacy PROMs included in our study showed floor and ceiling effects, whereas the PROMIS instruments were absent of this finding. Similarly, with respect to the PROMIS Depression CAT, legacy measures such as VR-12 MCS and SF-12 MCS may more appropriately establish baseline mental health states in cartilage patients whereas the Depression CAT may be better used to track postoperative changes in mental health. Questionnaire burden and TTC analysis revealed an average TTC of less than 2 minutes for each PROMIS instrument in our study. In addition, the average question burden was 4.28 for the PI CAT, 4.59 for the PF CAT, and 6.25 for the Depression CAT. The PROMIS PF CAT has previously been shown to have an average completion time of 54.1 seconds across 10,344 patients presenting to a hand clinic.17 The average question burden for the PROMIS PF CAT has been reported to be between 4 and 4.54 questions, dependent on the population of interest.16,41,45 For the PROMIS PI CAT, the average question requirement has been reported as 4.27, with a TTC of 45.05 seconds.14,45 Reported TTC data for the Depression CAT have shown values of 0.57 minutes and 0.80 minutes (48.11 seconds) in ambulatory sports medicine and hand clinics, respectively.14,17 Although our questionnaire burden data align well with previous literature, the average TTC data in our cohort for the PROMIS PF, PI, and Depression CATs showed completion times approximately 1 minute longer than those reported in prior studies. Limitations Our study has specific limitations readers must consider. Our cohort may be susceptible to selection bias by virtue of our inclusion criteria: Our data set was limited to primary surgical patients, and as such, our study conclusions do not extend to patients undergoing nonoperative treatment. In addition, the cartilage surgery cases from which our data were drawn were obtained across 3 surgeons at a single, high-volume academic orthopaedic institution. As a result, our data may not necessarily be generalizable to all patient populations and surgeons. Moreover, included in our case series were patients receiving cartilage debridement, OCA, and microfracture, suggesting variability in focal cartilage defect size. Because of limitations in our database, we are unable to stratify the cohort by defect size, lesion location, or lesion grade or to subsequently examine the impact of these operative factors on performance. Our patient-reported collection system used tablet computers to administer patient-reported outcome instruments. Consequently, we are unable to confirm whether patients completed a survey in question by themselves or perhaps a family member or associate assisted in the interpretation and completion of PROMs on their behalf. Finally, although
OUTCOMES IN KNEE CARTILAGE PATIENTS
8 individuals received ACI for a focal autologous cartilage defect within the study period, none of the recipients met the inclusion criteria because of previous surgical intervention (n ¼ 3), subsequent total knee arthroplasty within the follow-up period (n ¼ 1), or lack of PROM completion (n ¼ 4). No patients received osteochondral autograft transplantation within the study period.
Conclusions The PROMIS PF, PI, and Depression CATs performed better with respect to legacy PROMs in the postoperative period than the preoperative period. In addition, the PROMIS PF and PI measures performed best with respect to the IKDC questionnaire, and no floor or ceiling effects were identified for the PROMIS instruments. The PROMIS instruments may be more suited to track outcomes postoperatively than to establish preoperative baselines in cartilage surgery patients.
References 1. Hancock KJ, Glass N, Anthony CA, et al. Performance of PROMIS for healthy patients undergoing meniscal surgery. J Bone Joint Surg Am 2017;99:954-958. 2. MOTION Group. Patient-reported outcomes in orthopaedics. J Bone Joint Surg Am 2018;100:436-442. 3. Carr AJ. Evidence-based orthopaedic surgery: What type of research will best improve clinical practice? J Bone Joint Surg Br 2005;87:1593-1594. 4. Wang T, Wang DX, Burge AJ, et al. Clinical and MRI outcomes of fresh osteochondral allograft transplantation after failed cartilage repair surgery in the knee. J Bone Joint Surg Am 2018;100:1949-1959. 5. Tirico LEP, McCauley JC, Pulido PA, Demange MK, Bugbee WD. Is patient satisfaction associated with clinical outcomes after osteochondral allograft transplantation in the knee? Am J Sports Med 2018:363546518812420. 6. Ogura T, Ackermann J, Mestriner AB, Merkely G, Gomoll AH. The minimal clinically important difference and substantial clinical benefit in the patient-reported outcome measures of patients undergoing osteochondral allograft transplantation in the knee. Cartilage 2018: 1947603518812552. 7. Wang D, Chang B, Coxe FR, et al. Clinically meaningful improvement after treatment of cartilage defects of the knee with osteochondral grafts. Am J Sports Med 2018: 363546518808030. 8. Ogura T, Ackermann J, Barbieri Mestriner A, Merkely G, Gomoll AH. Minimal clinically important differences and substantial clinical benefit in patient-reported outcome measures after autologous chondrocyte implantation. Cartilage 2018:1947603518799839. 9. Conrozier T, Monet M, Lohse A, Raman R. Getting better or getting well? The patient acceptable symptom state (PASS) better predicts patient’s satisfaction than the decrease of pain, in knee osteoarthritis subjects treated with viscosupplementation. Cartilage 2018;9:370-377.
1427
10. Bekkers JE, de Windt TS, Raijmakers NJ, Dhert WJ, Saris DB. Validation of the Knee Injury and Osteoarthritis Outcome Score (KOOS) for the treatment of focal cartilage lesions. Osteoarthritis Cartilage 2009;17:1434-1439. 11. Roos EM, Toksvig-Larsen S. Knee injury and Osteoarthritis Outcome Score (KOOS)dValidation and comparison to the WOMAC in total knee replacement. Health Qual Life Outcomes 2003;1:17. 12. Escobar A, Quintana JM, Bilbao A, Arostegui I, Lafuente I, Vidaurreta I. Responsiveness and clinically important differences for the WOMAC and SF-36 after total knee replacement. Osteoarthritis Cartilage 2007;15:273-280. 13. Dowdle SB, Glass N, Anthony CA, Hettrich CM. Use of PROMIS for patients undergoing primary total shoulder arthroplasty. Orthop J Sports Med 2017;5: 2325967117726044. 14. Kadri O, Jildeh TR, Meldau JE, et al. How long does it take for patients to complete PROMIS scores?: An assessment of PROMIS CAT questionnaires administered at an ambulatory sports medicine clinic. Orthop J Sports Med 2018;6:2325967118791180. 15. Hung M, Saltzman CL, Greene T, et al. The responsiveness of the PROMIS instruments and the qDASH in an upper extremity population. J Patient Rep Outcomes 2017;1:12. 16. Scott EJ, Westermann R, Glass NA, Hettrich C, Wolf BR, Bollier MJ. Performance of the PROMIS in patients after anterior cruciate ligament reconstruction. Orthop J Sports Med 2018;6:2325967118774509. 17. Bernstein DN, Houck JR, Hammert WC. A comparison of PROMIS UE versus PF: Correlation to PROMIS PI and depression, ceiling and floor effects, and time to completion. J Hand Surg Am 2019;44:901.e1-901.e7. 18. Bernholt D, Wright RW, Matava MJ, Brophy RH, Bogunovic L, Smith MV. Patient Reported Outcomes Measurement Information System scores are responsive to early changes in patient outcomes following arthroscopic partial meniscectomy. Arthroscopy 2018;34: 1113-1117. 19. Senders A, Hanes D, Bourdette D, Whitham R, Shinto L. Reducing survey burden: Feasibility and validity of PROMIS measures in multiple sclerosis. Mult Scler 2014;20:1102-1111. 20. Amtmann D, Cook KF, Jensen MP, et al. Development of a PROMIS item bank to measure pain interference. Pain 2010;150:173-182. 21. Sherman SLG, Andrew J, Voss EE. Correlation of Patient Reported Outcome Measurement Information System Computer Adaptive Test (PROMIS-CAT) and Knee Injury and Osteoarthritis Outcome Score (KOOS) in knee osteotomy and cartilage procedures. Orthop J Sports Med 2018;6:2325967118S00102 (suppl 4). 22. Fabricant PD. Editorial Commentary: We PROMIS...one more patient-reported outcome questionnaire! Arthroscopy 2018;34:1118-1120. 23. Kohan EM, Hill JR, Schwabe M, Aleem AW, Keener JD, Chamberlain AM. The influence of mental health on Patient-Reported Outcomes Measurement Information System (PROMIS) and traditional outcome instruments in patients with symptomatic glenohumeral arthritis. J Shoulder Elbow Surg 2019;28:e40-e48.
1428
B. U. NWACHUKWU ET AL.
24. Chen RE, Papuga MO, Voloshin I, et al. Preoperative PROMIS scores predict postoperative outcomes after primary ACL reconstruction. Orthop J Sports Med 2018;6: 2325967118771286. 25. Karns MR, Jones DL, Todd DC, et al. Patient- and procedure-specific variables driving total direct costs of outpatient anterior cruciate ligament reconstruction. Orthop J Sports Med 2018;6:2325967118788543. 26. Patterson BM, Orvets ND, Aleem AW, et al. Correlation of Patient-Reported Outcomes Measurement Information System (PROMIS) scores with legacy patient-reported outcome scores in patients undergoing rotator cuff repair. J Shoulder Elbow Surg 2018;27:S17-S23. 27. Anthony CA, Glass N, Hancock K, Bollier M, Hettrich CM, Wolf BR. Preoperative Performance of the PatientReported Outcomes Measurement Information System in patients with rotator cuff pathology. Arthroscopy 2017;33:1770-1774.e1. 28. Sheean AJ, Schmitz MR, Ward CL, et al. Assessment of disability related to femoroacetabular impingement syndrome by use of the Patient-Reported Outcome Measure Information System (PROMIS) and objective measures of physical performance. Am J Sports Med 2017;45: 2476-2482. 29. Beletsky A, Nwachukwu BU, Manderle BJ, et al. The impact of workers’ compensation on Patient-Reported Outcomes Measurement Information System upper extremity and legacy outcome measures in patients undergoing arthroscopic rotator cuff repair. Arthroscopy 2019;35: 2817-2824. 30. Carender CN, Bollier MJ, Wolf BR, Duchman KR, An Q, Westermann RW. Preoperative performance of PROMIS in patients with patellofemoral malalignment and chondral disease. Orthop J Sports Med 2019;7: 2325967119855001. 31. Makhni EC, Meldau JE, Blanchett J, et al. Correlation of PROMIS physical function, pain interference, and depression in pediatric and adolescent patients in the ambulatory sports medicine clinic. Orthop J Sports Med 2019;7:2325967119851100. 32. Fries J, Rose M, Krishnan E. The PROMIS of better outcome assessment: Responsiveness, floor and ceiling effects, and Internet administration. J Rheumatol 2011;38: 1759-1764. 33. Hey HWD, Luo N, Chin SY, et al. The predictive value of preoperative health-related quality-of-life scores on postoperative patient-reported outcome scores in lumbar spine surgery. Global Spine J 2018;8:156-163. 34. Anthony CA, Glass NA, Hancock K, Bollier M, Wolf BR, Hettrich CM. Performance of PROMIS instruments in patients with shoulder instability. Am J Sports Med 2017;45:449-453. 35. Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60: 34-42.
36. Selim AJ, Rogers W, Qian SX, Brazier J, Kazis LE. A preference-based measure of health: The VR-6D derived from the veterans RAND 12-Item Health Survey. Qual Life Res 2011;20:1337-1347. 37. Oak SR, Strnad GJ, Bena J, et al. Responsiveness comparison of the EQ-5D, PROMIS Global Health, and VR-12 questionnaires in knee arthroscopy. Orthop J Sports Med 2016;4:2325967116674714. 38. Middel B, van Sonderen E. Statistical significant change versus relevant or important change in (quasi) experimental design: Some conceptual and methodological problems in estimating magnitude of intervention-related change in health services research. Int J Integr Care 2002;2:e15. 39. Kollmorgen RC, Hutyra CA, Green C, Lewis B, Olson SA, Mather RC III. Relationship between PROMIS computer adaptive tests and legacy hip measures among patients presenting to a tertiary care hip preservation center. Am J Sports Med 2019:363546518825252. 40. Hung M, Voss MW, Bounsanga J, Gu Y, Granger EK, Tashjian RZ. Psychometrics of the Patient-Reported Outcomes Measurement Information System physical function instrument administered by computerized adaptive testing and the Disabilities of Arm, Shoulder and Hand in the orthopedic elbow patient population. J Shoulder Elbow Surg 2018;27:515-522. 41. Hancock KJ, Glass N, Anthony CA, et al. PROMIS: A valid and efficient outcomes instrument for patients with ACL tears. Knee Surg Sports Traumatol Arthrosc 2019;27: 100-104. 42. Lyman S, Lee YY, Franklin PD, Li W, Cross MB, Padgett DE. Validation of the KOOS, JR: A short-form knee arthroplasty outcomes survey. Clin Orthop Relat Res 2016;474:1461-1471. 43. Collins NJ, Misra D, Felson DT, Crossley KM, Roos EM. Measures of knee function: International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form, Knee Injury and Osteoarthritis Outcome Score (KOOS), Knee Injury and Osteoarthritis Outcome Score Physical Function Short Form (KOOS-PS), Knee Outcome Survey Activities of Daily Living Scale (KOS-ADL), Lysholm Knee Scoring Scale, Oxford Knee Score (OKS), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), Activity Rating Scale (ARS), and Tegner Activity Score (TAS). Arthritis Care Res (Hoboken) 2011;63:S208-S228 (suppl 11). 44. van de Graaf VA, Wolterbeek N, Scholtes VA, Mutsaerts EL, Poolman RW. Reliability and validity of the IKDC, KOOS, and WOMAC for patients with meniscal injuries. Am J Sports Med 2014;42: 1408-1416. 45. Strong B, Maloney M, Baumhauer J, et al. Psychometric evaluation of the Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function and Pain Interference Computer Adaptive Test for subacromial impingement syndrome. J Shoulder Elbow Surg 2019;28:324-329.