- Email: [email protected]
Characteristics of the MATRICS Consensus Cognitive Battery in a 29-site antipsychotic schizophrenia clinical trial
Schizophrenia Research 125 (2011) 161–168
Contents lists available at ScienceDirect
Schizophrenia Research j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / s c h r e s
Characteristics of the MATRICS Consensus Cognitive Battery in a 29-site antipsychotic schizophrenia clinical trial Richard S.E. Keefe a,b,⁎, Kolleen Hurley Fox b, Philip D. Harvey c, Josephine Cucchiaro d, Cynthia Siu e, Antony Loebel d a b c d e
Department of Psychiatry, Box 3270, Duke University Medical Center, Durham, NC, 27710, USA NeuroCog Trials, Inc., 3211 Shannon Rd., Suite 300, Durham, NC, 27707, USA Department of Psychiatry and Behavioral Sciences, University of Miami Miller School of Medicine, 1120 NW 14th Street, Suite 1450, Miami, FL 33136, USA Dainippon Sumitomo Pharma USA/Sepracor/Sunovion, 1 Bridge Plaza N #510, Fort Lee, NJ, 07024, USA Data Power (DP), Inc., One Palomino Ct., Ringoes, NJ, 8551, USA
a r t i c l e
i n f o
Article history: Received 30 August 2010 Accepted 9 September 2010 Available online 31 December 2010
Keywords: Cognition Neuropsychology Reliability MCCB Schizophrenia Clinical trials
a b s t r a c t Objective: The Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Project produced a battery of tests, the MATRICS Consensus Cognitive Battery (MCCB), designed to assess cognitive treatment effects in clinical trials of patients with schizophrenia. In validation studies, the MCCB demonstrated excellent reliability, minimal practice effects and significant correlations with measures of functional capacity. This study addresses whether the MCCB demonstrates these favorable characteristics when administered in the context of the type of large multi-site industry trial for which it was designed. Methods: In a clinical trial comparing risperidone and lurasidone, 323 clinically-stable outpatients with schizophrenia at 29 sites were assessed with MCCB at screening and a median of 15 days later at baseline. A measure of functional capacity, the UCSD Performance-based Skills Assessment – Brief (UPSA-B) was administered at baseline. Results: All 323 (100%) patients had sufficient data for computing a composite score according to the MCCB criteria. The test–retest reliability of the MCCB composite score was excellent (ICC = 0.88). The severity of cognitive impairment was T = 24.7 (SD = 12.1) at screening and T = 26.7 (SD = 12.4) at baseline. The MCCB composite score demonstrated a large correlation with the UPSA-B composite score (r = .60, df= 304, p b .001). The practice effect on the composite score was small (z = 0.18). Discussion: In the context of a 29-site antipsychotic trial in stable outpatients with schizophrenia, the MCCB is sensitive to cognitive deficits in all domains, demonstrates excellent test–retest reliability and small practice effects, and is strongly correlated with a leading measure of functional capacity. © 2010 Elsevier B.V. All rights reserved.
1. Introduction ⁎ Corresponding author. Department of Psychiatry, Box 3270, Duke University Medical Center, Durham, NC, 27710, USA. Tel.: +1 919 684 4306; fax: +1 919 684 2632. E-mail addresses: [email protected] (R.S.E. Keefe), [email protected] (K.H. Fox), [email protected] (P.D. Harvey), [email protected] (J. Cucchiaro), [email protected] (C. Siu), [email protected] (A. Loebel). 0920-9964/$ – see front matter © 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.schres.2010.09.015
One of the major objectives of the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Project was to develop a battery of tests that could be used to assess cognitive treatment effects in clinical trials of patients with schizophrenia (Green et al., 2004; Nuechterlein and Green, 2006). The MATRICS Neurocognition Committee
162
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
established criteria for the test battery emphasizing characteristics required for cognitive measures in the context of clinical trials: test–retest reliability; utility as a repeated measure; relationship to functional status; potential changeability in response to pharmacological agents; and practicality for clinical trials and tolerability for patients (Nuechterlein et al., 2008). The final battery of tests, the MATRICS Consensus Cognitive Battery (MCCB), was vetted through a detailed process involving leaders from academia, industry and NIMH, and validated through a 5-site academic consortium, the MATRICS Psychometric and Statistics Study (PASS). Normative data were collected on 300 healthy controls matched on key demographic factors to the United States population (Kern et al., 2008). The MCCB was approved by FDA for use in clinical trials for cognitive improvement in schizophrenia (Nuechterlein et al., 2008; Buchanan et al., 2005; Buchanan et al, 2010). In the MATRICS PASS validation study, conducted at 5 medical centers with schizophrenia cognition expertise, the MCCB tests demonstrated excellent reliability, minimal practice effects, and significant correlations with measures of functional capacity (Nuechterlein et al., 2008). However, it has been an empirical question whether the MCCB would demonstrate these favorable characteristics when administered in the context of the type of large multi-site industry trial for which it was designed. The current study is the first attempt to address this question; it describes cognitive data collected with the MCCB in a 29-site clinical trial assessing the comparative effects of antipsychotic treatment with lurasidone or risperidone in patients with schizophrenia. The design of this treatment trial involved a multiple-baseline assessment period during which patients were assessed twice with the MCCB, providing the opportunity to investigate the sensitivity, test–retest reliability, and predictive validity of the MCCB in a schizophrenia clinical trial setting. For the MCCB to be useful in clinical trials, it is expected that it will demonstrate high test–retest reliability, minimal practice effects, and substantial correlations with functional capacity co-primary measures such as the UCSD Performance-based Skills Assessment, Brief (UPSA-B) which has been identified as a leading co-primary measure for schizophrenia cognition trials. This study provides a test–retest convenience sample to assess these psychometric characteristics of the MCCB. 2. Methods 2.1. Patients Three hundred twenty-three clinically stable outpatients with schizophrenia in the United States were randomized 2:1 to lurasidone and risperidone flexible dose treatment
Table 1 Demographics and baseline characteristics. All subjects (N = 323) Age Gender Race
PANSS total score
43.1 (SD = 10.4) Male 231 (72%); female 92 (28%) White 99 (31%) African American 206 (64%) Other 18 (5%) 67.5 (SD = 11.7)
(clinicaltrials.gov ID: NCT00641745). Table 1 describes the demographic and baseline characteristics of this patient sample. The treatment protocol was approved by the Human Subjects Committees at the 29 sites. As part of the protocol screening process, patients received the full MCCB. Patients met the following inclusion criteria, which closely follow the MATRICS guidelines presented at a panel sponsored by FDA, NIMH and MATRICS (Buchanan et al., 2005): age 18–65; diagnosis of schizophrenia; CGI-S b 4 at both screening and baseline; no major change in antipsychotic medications for at least 6 weeks prior to screening; no hospitalization for psychiatric illness for at least 8 weeks prior to screening; and moderate or less (b4) severity rating on selected PANSS positive scale items at both screening and baseline. Patients were excluded from the study if they demonstrated any of the major exclusion criteria: resistance to antipsychotic treatment defined as failure to respond to 2 or more antipsychotic agents administered at an adequate dose for at least 6 weeks or a history of treatment intolerance to 3 or more antipsychotic agents; previous treatment with risperidone within 6 weeks prior to baseline; history of poor response to risperidone or intolerability to risperidone treatment; history of treatment with clozapine within 4 months of baseline visit or treatment with clozapine for refractory psychosis; history of antipsychotic withdrawal for N3 months without evidence of psychosis; unstable medical illness, or a history of hypersensitivity to risperidone or an allergic reaction to more than 2 chemical classes of drugs (prescription or non-prescription). Subjects were considered ineligible for participation in the cognitive component of the study described here if they met any of the following exclusion criteria relevant to the cognitive data analysis: treatment with N10 mg of haloperidol per day or equivalent dose of other conventional antipsychotics within 30 days prior to screening; history of moderate or greater head injury; clinically significant or history of alcohol abuse/ alcoholism or drug abuse/dependence within the last 6 months; history of previous psychosurgery; history of neuroleptic malignant syndrome; evidence of severe tardive dyskinesia, tardive dystonia, or other severe chronic movement disorder; clinically significant suicidal ideation or behavior or violent behavior in the last 6 months; electroconvulsive therapy within the past 3 months; treatment with mood stabilizers or antidepressants within 1 week, fluoxetine within 1 month, or MAO inhibitor within 3 weeks of baseline visit. Subjects voluntarily agreed to participate in the study by giving written informed consent. 2.2. MATRICS Consensus Cognitive Battery (MCCB) The MCCB measures seven separable cognitive domains: speed of processing; attention/vigilance; working memory (verbal and nonverbal); verbal learning; visual learning; reasoning and problem solving; and social cognition. Administration of the MCCB requires about 75–90 min. The following tests were administered in this standard order: • Trail making A • Brief Assessment of Cognition in Schizophrenia (BACS) Symbol Coding • Hopkins Verbal Learning Test – Revised (HVLT-R) • Wechsler Memory Scale III (WMS-III) Spatial Span
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
• • • • •
University of Maryland Letter Number Span Neuropsychological Assessment Battery (NAB) – Mazes Brief Visuospatial Memory Test – Revised (BVMT-R) Category Fluency Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT) Managing Emotions • Continuous Performance Test – Identical Pairs (CPT-IP) Since the overall study involved 5 assessments with the MCCB (including three assessments after treatment was initiated, whose results are not reported here), 2 alternate forms of the NAB Mazes and 4 alternate forms of the HVLT-R and BVMT-R were administered in a counterbalanced manner across patients. Thus all 50 combinations of these three tests were evenly distributed across the entire patient sample at each MCCB assessment point. All testers were trained on the administration and scoring of the MCCB using video and group training sessions and were individually certified by an expert on the MCCB. MCCB data were scored and sent to a central site, where they were rescored and double-entered data into the MCCB scoring program to ensure data accuracy. Training, data collection, and data quality assurance were implemented or supervised by an experienced psychologist as per the guidelines outlined in the MCCB manual (Nuechterlein and Green, 2006). The MCCB scoring program yields seven domain scores and a composite score which are standardized to the same T-score measurement scale with a mean of 50 and an SD of 10 based upon the normative data collected from a sample of 300 community participants as part of the MATRICS PASS and published in the MCCB manual and the MCCB scoring program (Nuechterlein and Green, 2006). 2.3. UCSD Performance-based Skills Assessment Brief version (UPSA-B) The UPSA-B was designed to evaluate the abilities of individuals to perform everyday tasks that are considered necessary for independent functioning in the community. The UPSA-B is a briefer version of the UPSA, which uses role-playing situations to evaluate skills in five areas: household chores, communication, finance, transportation, and planning recreational activities (Mausbach et al., 2007). The UPSA-B consists of two of the five original UPSA domains, finances and communication. Each of the two subscales contributes 50 points, and total scores range from 0 to 100 points; higher scores reflect better performance. The UPSA-B scores were found to correlate at a very high level, r = −.93, with scores on the five-subscale UPSA, and the UPSA-B has been shown to be equally sensitive to skills required in independent living when compared to the long form of the UPSA (Mausbach et al., 2007). The UPSA-B requires about 7–10 min to complete and, involves only props that are widely available (a telephone and money), commonly utilized and relevant at any site in the United States. Raters received specific training and education regarding this assessment prior to study initiation. 2.4. Cognitive Assessment Interview (CAI) The CAI is an interview-based measure of cognitive function that incorporates ratings from 3 primary sources – subject, informant, and interviewer – to produce a total score (Ventura
163
et al., 2010). The test consists of a set of 10 items, rated on a 7-point, anchor-based scale. Reasonable efforts were made to identify an informant (e.g., caregiver, family member, or friend) who was familiar with the subject's overall functioning and was willing to provide such information about the subject. An informant was not required for subjects' participation in the trial; only 123 of the patients had informants available to provide information for the CAI. The CAI raters were required to meet specific credential and educational criteria before they were certified to rate for this study.
2.5. Time course of assessments 2.5.1. Screening assessment Patients were assessed with the MCCB and PANSS at the screening visit while receiving their baseline antipsychotic medications, prior to the switch to risperidone or lurasidone. Patients who met all eligibility criteria at screening visit 1 were withdrawn from their current antipsychotic medication as tolerated and in accordance with labeling recommendations and conventional medical practice in a 1–7 day transition period. The baseline visit (day 0) was the final transition day and the day of randomization. Subjects could be hospitalized for up to 7 days during the transition period at the investigator's discretion.
2.5.2. Baseline assessment The MCCB and PANSS were administered for the second time along with the initial assessment with the UPSA-B and CAI. This baseline assessment was scheduled to be completed 7–21 days after screening. However, a small number of subjects received their baseline assessment outside the 7–21 day window.
2.6. Statistical analysis All data presented here were obtained from enrolled subjects prior to randomization to double-blind study treatment (at screening and/or baseline). Test–retest reliability for the individual cognitive domains and composite T-score were evaluated using Intra-Class Correlation (ICC) with the Shrout–Fleiss 2-way ANOVA for reliability of a single rating model. Practice effects for each cognitive domain and composite T-score were assessed by effect size (change score normalized by standard deviation of screening score). Measurement error and precision over time (at testing and retesting) were compared using a paired-sample test for two correlated variances. Missing data patterns were summarized for the MCCB individual domains and composite T-score. Predictive validity was assessed by correlations of MCCB composite T-score with the UPSA-B composite score and CAI total score at baseline. The primary CAI endpoint is Global Severity of Cognitive Impairment (Rater Composite Impression) based on information collected from patient and informant. Subjects with no informant will be included in the study, but excluded from the analysis of CAI composite score. Differences between patient and informant CAI ratings were examined using t-test.
164
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
3. Results 3.1. Completeness of study dataset Three hundred twenty-three patients were tested 2 times with the 10-test MCCB. Of these 6460 individual cognitive test assessments, 14 were invalid or missing (99.8%) (see Table 2). Three hundred eighteen of the 323 patients (98.5%) had complete MCCB assessments at screening and baseline. The published MCCB guidelines (Nuechterlein and Green, 2006) recommend that a composite score is valid if 5 of 7 domains are complete at pre-treatment assessment, and 4 of 7 are complete posttreatment assessment. All subjects met this criterion at baseline, and all but 1 subject met this criterion at the end of screening. 3.2. Sensitivity of MCCB to impairment with composite scores and domain scores
2.2 points. In calculating an effect size for this change, we utilized the standard deviation of the initial assessment in the denominator; the test–retest effect size using this formula (Cohen, 1977) was d = 0.18. The changes in domain scores from screening to baseline were also small, ranging from 2.4 points for speed of processing (d = 0.20) to 0.4 points for social cognition (d = 0.03). 3.5. Correlations between UPSA-B and MCCB The UPSA-B composite score had a slight negative skew (skewness = −0.70; normality test p b .05) The mean UPSA-B score at baseline was 70.2 (SD = 16.2). Despite the slight negative skew of the UPSA-B composite score, the correlation between UPSA-B composite score and MCCB composite score was substantial (r = 0.60, df = 314, p b .001; see Fig. 3). 3.6. MCCB correlations with CAI and PANSS
The distribution of the MATRICS battery composite scores at screening and baseline was normal (see Fig. 1). T-scores for all domains except Working Memory and Speed of Processing were significantly different from normal. Table 3 describes the mean performance of patients at screening and at baseline, which was a mean/median of 18 days later (SD 5 days). Mean composite T-scores (SD) at screening and baseline were 24.7 (12.1) and 26.9 (12.4), indicating that schizophrenia patients in this study performed at 2.5 SD below the healthy controls from the normative sample. The profile of impairments across the 7 domains was variable, but for all domains ranged between 1 and 2 SD below healthy controls.
Spearman rank correlation of the baseline MCCB composite score with the baseline CAI interviewer total score (for global severity of cognitive impairment based on patient and informant interviews) was r = −0.14 (df= 121; P = .066) in the subgroup of patients with an informant, suggesting a weak relationship between poor performance on the MCCB and interview-based reports of cognitive impairment. The mean PANSS total score at baseline was 67.5 (SD = 11.7). Correlation between MCCB composite score and baseline PANSS total score was r=−0.20, df= 319, p b .0010, indicating that patients who had less severe symptoms performed slightly better on the MCCB.
3.3. Test–retest reliability 4. Discussion The test–retest reliability of the composite score was excellent (ICC = 0.88, DFerror = 318, p b .001) (see Table 3 and Fig. 2). The test–retest reliability of the individual domains was variable ranging from a high of 0.79 for speed of processing, attention/vigilance and working memory, to 0.58 for verbal learning and 0.65 for visual learning. It is noteworthy that the 2 domains with the lowest test–retest reliability involve the use of alternate forms between sessions. 3.4. Practice effects Practice effects were small (see Table 3). The composite T-score change from screening to baseline improved a mean of
The MATRICS Neurocognition Committee determined that the most important characteristics required for cognitive measures in the context of clinical trials included: test–retest reliability; utility as a repeated measure, such as minimal practice effects; relationship to functioning or functional capacity; and practicality for clinical trials and tolerability for patients (Nuechterlein et al., 2008). In this 29-site clinical trial designed to assess the comparative effects of lurasidone or risperidone in patients with schizophrenia, 323 patients were assessed at screening and then again after an average of 18 days with the MATRICS Consensus Cognitive Battery (MCCB), providing an opportunity to assess the psychometric charac-
Table 2 MCCB test scores: quality assurance. All subjects (N = 323) Total number of invalid or missing MCCB tests of total tests (N = 323 patients tested twice on 10 tests = 6460 tests) Subjects with complete MCCB tests (excluding invalid or missing tests) Screening Baseline Screening and Baseline scores Subjects with missing N = 3 tests Screening Baseline Duration between screening and baseline tests
14 320/323 (99.1%) 321/323 (99.4%) 318/323 (98.5%) 1 (0.3%) 0 (0%) 18 days (SD 5) Range = 6–37
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
165
Fig. 1. Screening and baseline MCCB composite T-score: Normal density estimate and histogram (N = 323). Mean T-score at baseline = 26.9 (SD = 12.4); skewness = 0.13; normality test p N 0.05. Mean T-score at screening 24.7 (SD = 12.1). Area under the curve (histogram) = 1.
teristics of the MCCB in the context of the type of trial and the type of sites for which it was designed. In this study, the MCCB demonstrated clear sensitivity to the cognitive deficits of schizophrenia, excellent test–retest reliability, minimal practice effects, and significant correlations with measures of functional capacity. One of the early concerns with the MCCB was that the burden of cognitive assessments, especially in psychotic patients, would lead to a large number of missing variables. These concerns may be particularly strong for clinical trials with antipsychotic medications, since patients with psychosis are presumed to be more difficult to test. To address these concerns in this antipsychotic trial, strict tester training and certification procedures and data review processes were put in place. The result was that only 14 item scores (0.2%) were missing out of a total of 6460 assessments. Since composite scores can be calculated if 5 of 7 domains are validly collected, only one patient in 323 did not have sufficient data to calculate a composite score from the testing occasions at both screening and baseline. The completeness of this study data set suggests that missing data with the MCCB should be rare, although this
result may depend upon the utilization of intensive rater training and data review processes. The average MCCB composite score for the patients in this study was 24.7 at screening and 26.9 at baseline, suggesting an overall impairment of about 2.5 standard deviations compared to the healthy controls included in the MCCB normative study (Kern et al., 2008). This level of severity is consistent with preliminary data from other studies (Buchanan et al, in press; Javitt et al, submitted; Buchanan et al, submitted) It is noteworthy that the composite score from the MCCB is more diagnostically sensitive and more reliable than any of the individual domain scores, and more sensitive than any single test scores in previous meta-analyses (Heinrichs and Zakzanis, 1998). This increased sensitivity is due to the moderate correlation among the MCCB tests (Nuechterlein et al., 2008), which serves to reduce the variability of the composite score compared to the domain scores, and creates a measure that may be more sensitive to treatment effects than a single domain score. From a survey of 68 experts on schizophrenia cognition, the MATRICS project investigators concluded that test–retest
Table 3 MCCB composite T-score: screening versus baseline (N = 323).
Composite T-score Speed of processing Attention/vigilance Working memory Verbal learning Visual learning Reason/problem solving Social cognition a
Screening mean (SD)
Baseline mean (SD)
ICC
Test–retest effect size a
p-value (SDScreening = SDBaseline)
24.7 (12.1) 31.3 (12.0) 34.2 (11.8) 32.8 (11.1) 35.1 (7.0) 32.7 (11.8) 38.9 (8.1) 36. (13.0)
26.9 33.7 36.2 34.5 35.7 34.2 39.9 36.4
0.88 0.79 0.79 0.79 0.58 0.65 0.77 0.75
0.18 0.20 0.17 0.16 0.09 0.13 0.14 0.03
0.32 0.77 0.06 0.73 0.13 0.73 b0.001 0.44
(12.4) (12.1) (12.5) (11.2) (7.5) (11.7) (9.3) (12.6)
Effect size for paired sample = (Baseline − Screening)/SDScreening.
166
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
Fig. 2. MCCB composite T-score at screening and baseline (N = 318).
reliability was the most important feature of a test battery to be used in schizophrenia clinical trials (Green et al., 2004). In a separate survey conducted by the leaders of another NIMH Project, the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS), the optimal test–retest reliability was determined to be an ICC of 0.90
(Barch et al., 2008). In this study, the test–retest reliability of the MCCB composite score was ICC= 0.88, very closely matching the ideal cited by the CNTRICS experts. The test–retest reliability of the domain scores was lower, although acceptable, ranging between 0.75 and 0.79 in the tests without alternate forms. It is noteworthy that the test–retest reliability of the two
Fig. 3. Cross-sectional relationship between MCCB composite T-score and UPSA-B composite score.
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
memory tests that use alternate forms, visual learning and verbal learning, were particularly low at 0.65 and 0.58 respectively. While the design of the MCCB allows an emphasis on individual domains, and such a focus is encouraged in the published FDA-NIMH-MATRICS guidelines (Buchanan et al., 2005), these low reliabilities suggest that perhaps additional memory tests should be included if verbal or visual memory is to be the primary outcome of a trial so that domain-specific reliability can be enhanced. Previous studies using very small samples have reported that the practice effects of the MCCB are medium size (Freedman et al., 2008), raising concerns that the use of the MCCB in a repeated measures design may reduce sensitivity to real treatment effects (Lieberman et al., 2008). In the current study, the composite score improvement over an average period of 18 days was 2.2 points, equal to a Cohen's d of 0.18, which is less than “small” according to Cohen's definition (Cohen, 1977). This amount of practice effect raises the average level of the deficit in these patients from 2.5 to 2.3 standard deviations below the healthy control mean. Thus, the test battery allows substantial room for sensitivity to any real treatment effects. This amount of improvement with practice is entirely consistent with the amount of improvement reported in previous studies of practice-related cognitive changes in patients with psychotic disorders and healthy controls (Goldberg et al., 2007, 2010). The FDA-NIMH-MATRICS guidelines state that if FDA requires a co-primary measure in addition to the MCCB for approval of a cognitive-enhancing compound, the measure should be expected to change in close temporal proximity to changes on cognitive performance measures. It is difficult to determine if the MCCB and the set of currently available potential co-primary measures will change in parallel with one another in response to a cognitive-enhancing compound, since currently there are no effective treatments for cognitive impairment in schizophrenia. However, one source of evidence that a cognitive measure may change in conjunction with a co-primary measure is whether they are correlated with one another at baseline. There has been ample evidence recently from academic research sites that measures of functional capacity such as the UPSA and UPSA-B are strongly correlated with a variety of neurocognitive measures, with several studies reporting correlations in the range of 0.54 to 0.67 (Leifker et al., 2009). The data from the current study suggest that this correlation holds true even in large multisite studies, as the Spearman correlation between the MCCB composite score and the UPSA-B total score of 0.60 fits squarely within the range of previous findings from academic research centers. The lack of a significant correlation with the CAI, an interview-based assessment of cognition, suggests that this rating scale measures a different aspect of cognition from the performance-based elements of the MCCB. In sum, in a 29-site antipsychotic clinical trial assessing 323 schizophrenia patients at screening and then 2–3 weeks later at baseline, the MCCB was able to generate a very complete data set, with composite scores that demonstrated high sensitivity to cognitive impairments in schizophrenia, excellent test–retest reliability, small practice effects, and strong correlations with functional capacity. The individual domain scores demonstrated substantial sensitivity to impairment, although less than composite score, with similar or
167
smaller practice effects, and weaker reliability, especially for tests with alternate forms. Role of funding source This study was designed and supported by Dainippon Sumitomo Pharma America. NeuroCog Trials trained the cognitive testers and conducted data quality review. Analyses were completed by Dr. Cynthia Sui in consultation with the manuscript co-authors. The decision to submit the paper for publication was made by all co-authors. Following the role of the funding source text, authors are required to declare their individual contribution to the manuscript under Contributors. Contributors Dr. Keefe wrote the first draft of the paper. All other authors edited and commented on versions of the manuscript and approved the final version. This study was designed and supported by Drs. Loebel, Cucchiaro, and others at Dainippon Sumitomo Pharma America, now Sunovion. NeuroCog Trials trained the cognitive testers and conducted data quality review under the supervision of Drs Fox and Keefe. Analyses were completed by Dr. Cynthia Sui in consultation with the manuscript co-authors. Dr. Harvey consulted on the design of the study. Conflict of interest Dr. Keefe reports that he currently or in the past 3 years has received investigator-initiated research funding support from the Allon, Eli Lilly, GlaxoSmithKline, Novartis, Department of Veteran's Affairs, National Institute of Mental Health, and the Singapore National Medical Research Council, and an unrestricted educational grant from Astra-Zeneca and Organon. He currently or in the 3 years has received honoraria, served as a consultant, or advisory board member for Abbott, Acadia, BiolineRx, BrainCells, Bristol-Myers Squibb, CHDI, Cypress BioScience, Dainippon Sumitomo Pharma/Sepracor/Sunovion, Eli Lilly, EnVivo, Johnson & Johnson, Lundbeck, Memory Pharmaceuticals, Merck, Neurosearch, Orexigen, Orion, Otsuka, Pfizer, Prophase, Roche, Sanofi-Aventis, Shering-Plough, Shire, Solvay, Takeda, World Wide Clinical Trials, Wyeth, and Xenoport. Dr. Keefe receives royalties from the Brief Assessment of Cognition in Schizophrenia (BACS) testing battery and the MATRICS Battery (BACS Symbol Coding). He is also a shareholder in NeuroCog Trials, Inc. NeuroCog Trials conducted cognitive rater training and data quality review for this study. Dr. Fox is a full time employee of NeuroCog Trials, Inc. Dr. Harvey reports in the past 3 years he has served as a consultant to Abbott Labs, Cypress Bioscience, Dainippon Sumitomo Pharma America/ Sepracor/Sunovion, Eli Lilly, Merck and Company, Shire Pharma, Solvay Pharma, Teva Pharma., and Wyeth Pharma (now Pfizer). Also, he has a grant from Astra-Zeneca. Dr. Cucchiaro reports that she is a full time employee of Dainippon Sumitomo Pharma/Sepracor/Sunovion, Inc. Dr. Siu was a paid consultant to Dainippon Sumitomo Pharma/Sepracor/ Sunovion in connection with the statistical analysis and development of this manuscript and has served as a consultant to Pfizer, Memory Pharmaceutical/Roche Laboratories, and Wyeth/Pfizer over the past 3 years. Dr. Loebel reports that he is an employee of Dainippon Sumitomo Pharma/Sepracor/Sunovion. Acknowledgements None.
References Barch, D.M., Carter, C.S., CNTRICS Executive Committee, 2008. Measurement issues in the use of cognitive neuroscience tasks in drug development for impaired cognition in schizophrenia: a report of the second consensus building conference of the CNTRICS initiative. Schizophr. Bull. 34 (4), 613–618. Buchanan, R.W., Davis, M., Goff, D., Green, M.F., Keefe, R.S.E., Leon, A., Nuechterlein, K., Laughren, T., Levin, R., Stover, E., Fenton, W., Marder, S., 2005. A summary of the FDA-NIMH-MATRICS workshop on clinical trial design for neurocognitive drugs for schizophrenia. Schizophr. Bull. 31 (1), 5–19. Buchanan, R.W., Keefe, R.S.E., Umbricht, D., Green, M.F., Laughren, T., Marder, S.R., 2010. The FDA-NIMH-MATRICS guidelines for clinical trial design of cognitive-enhancing drugs: what do we know 5 years later? Schizophr. Bull. (epub ahead of print).
168
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
Cohen, J., 1977. Statistical power analysis for the behavioral sciences. Academic Press, Inc., Orlando, Florida. Freedman, R., Olincy, A., Buchanan, R.W., Harris, J.G., Gold, J.M., Johnson, L., Allensworth, D., Guzman-Bonilla, A., Clement, B., Ball, M.P., Kutnick, J., Pender, V., Martin, L.F., Stevens, K.E., Wagner, B.D., Zerbe, G.O., Soti, F., Kem, W.R., 2008. Initial phase 2 trial of a nicotinic agonist in schizophrenia. Am. J. Psychiatry 165 (8), 1040–1047. Goldberg, T.E., Goldman, R.S., Burdick, K.E., Malhotra, A.K., Lencz, T., Patel, R.C., Woerner, M.G., Schooler, N.R., Kane, J.M., Robinson, D.G., 2007. Cognitive improvement after treatment with second-generation antipsychotic medications in first-episode schizophrenia: is it a practice effect? Arch. Gen. Psychiatry 64 (10), 1115–1122. Goldberg, T.E., Keefe, R.S., Goldman, R.S., Robinson, D.G., Harvey, P.D., 2010. Circumstances under which practice does not make perfect: a review of the practice effect literature in schizophrenia and its relevance to clinical treatment studies. Neuropsychopharmacology 35 (5), 1053–1062. Green, M.F., Nuechterlein, K.H., Gold, J.M., Barch, D., Cohen, J., Essock, S., Fenton, W.S., Frese, F., Goldberg, T.E., Heaton, R.K., Keefe, R.S.E., Kern, R.S., Kraemer, H., Stover, E., Weinberger, D.R., Zalcman, S., Marder, S.R., 2004. Approaching a consensus cognitive battery for clinical trials in schizophrenia: the NIMH-MATRICS conference to select cognitive domains and test criteria. Biol. Psychiatry 56 (5), 301–307. Heinrichs, R.W., Zakzanis, K.K., 1998. Neurocognitive deficit in schizophrenia: a quantitative review of the evidence. Neuropsychology 12 (3), 426–444. Kern, R.S., Nuechterlein, K.H., Green, M.F., Baade, L.E., Fenton, W.S., Gold, J.M., Keefe, R.S.E., Mesholam-Gately, R., Mintz, J., Seidman, L.J., Stover, E.,
Marder, S.R., 2008. The MATRICS consensus cognitive battery: Part 2. Co-norming and standardization. Am. J. Psychiatry 165 (2), 214–220. Leifker, F.R., Bowie, C.R., Harvey, P.D., 2009. Determinants of everyday outcomes in schizophrenia: the influences of cognitive impairment, functional capacity, and symptoms. Schizophr. Res. 115 (1), 82–87. Lieberman, J.A., Javitch, J.A., Moore, H., 2008. Cholinergic agonists as novel treatments for schizophrenia: the promise of rational drug development for psychiatry. Am. J. Psychiatry 165 (8), 931–936. Mausbach, B.T., Harvey, P.D., Goldman, S.R., Jeste, D.V., Patterson, T.L., 2007. Development of a brief scale of everyday functioning in persons with serious mental illness. Schizophr. Bull. 33 (6), 1364–1372. Nuechterlein, K.H., Green, M.F., 2006. MATRICS Consensus Battery Manual. MATRICS Assessment Inc., Los Angeles. Nuechterlein, K.H., Green, M.F., Kern, R.S., Baade, L.E., Barch, D., Cohen, J., Essock, S., Fenton, W.S., Frese, F.J., Gold, J.M., Goldberg, T., Heaton, R., Keefe, R.S.E., Kraemer, H., Mesholam-Gately, R., Seidman, L.J., Stover, E., Weinberger, D., Young, A.S., Zalcman, S., Marder, S.R., 2008. The MATRICS consensus cognitive battery: Part 1. Test selection, reliability, and validity. Am. J. Psychiatry 165 (2), 203–213. Ventura, J., Reise, S.P., Keefe, R.S.E., Baade, L.E., Gold, J.M., Green, M.F., Kern, R.S., Mesholam-Gately, R., Nuechterlein, K.H., Seidman, L.J., Bilder, R.M., 2010. The Cognitive Assessment Interview (CAI): development and validation of an empirically derived, brief interview-based measure of cognition. Schizophr. Res. 121 (1-3), 24–31.
Contents lists available at ScienceDirect
Schizophrenia Research j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / s c h r e s
Characteristics of the MATRICS Consensus Cognitive Battery in a 29-site antipsychotic schizophrenia clinical trial Richard S.E. Keefe a,b,⁎, Kolleen Hurley Fox b, Philip D. Harvey c, Josephine Cucchiaro d, Cynthia Siu e, Antony Loebel d a b c d e
Department of Psychiatry, Box 3270, Duke University Medical Center, Durham, NC, 27710, USA NeuroCog Trials, Inc., 3211 Shannon Rd., Suite 300, Durham, NC, 27707, USA Department of Psychiatry and Behavioral Sciences, University of Miami Miller School of Medicine, 1120 NW 14th Street, Suite 1450, Miami, FL 33136, USA Dainippon Sumitomo Pharma USA/Sepracor/Sunovion, 1 Bridge Plaza N #510, Fort Lee, NJ, 07024, USA Data Power (DP), Inc., One Palomino Ct., Ringoes, NJ, 8551, USA
a r t i c l e
i n f o
Article history: Received 30 August 2010 Accepted 9 September 2010 Available online 31 December 2010
Keywords: Cognition Neuropsychology Reliability MCCB Schizophrenia Clinical trials
a b s t r a c t Objective: The Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Project produced a battery of tests, the MATRICS Consensus Cognitive Battery (MCCB), designed to assess cognitive treatment effects in clinical trials of patients with schizophrenia. In validation studies, the MCCB demonstrated excellent reliability, minimal practice effects and significant correlations with measures of functional capacity. This study addresses whether the MCCB demonstrates these favorable characteristics when administered in the context of the type of large multi-site industry trial for which it was designed. Methods: In a clinical trial comparing risperidone and lurasidone, 323 clinically-stable outpatients with schizophrenia at 29 sites were assessed with MCCB at screening and a median of 15 days later at baseline. A measure of functional capacity, the UCSD Performance-based Skills Assessment – Brief (UPSA-B) was administered at baseline. Results: All 323 (100%) patients had sufficient data for computing a composite score according to the MCCB criteria. The test–retest reliability of the MCCB composite score was excellent (ICC = 0.88). The severity of cognitive impairment was T = 24.7 (SD = 12.1) at screening and T = 26.7 (SD = 12.4) at baseline. The MCCB composite score demonstrated a large correlation with the UPSA-B composite score (r = .60, df= 304, p b .001). The practice effect on the composite score was small (z = 0.18). Discussion: In the context of a 29-site antipsychotic trial in stable outpatients with schizophrenia, the MCCB is sensitive to cognitive deficits in all domains, demonstrates excellent test–retest reliability and small practice effects, and is strongly correlated with a leading measure of functional capacity. © 2010 Elsevier B.V. All rights reserved.
1. Introduction ⁎ Corresponding author. Department of Psychiatry, Box 3270, Duke University Medical Center, Durham, NC, 27710, USA. Tel.: +1 919 684 4306; fax: +1 919 684 2632. E-mail addresses: [email protected] (R.S.E. Keefe), [email protected] (K.H. Fox), [email protected] (P.D. Harvey), [email protected] (J. Cucchiaro), [email protected] (C. Siu), [email protected] (A. Loebel). 0920-9964/$ – see front matter © 2010 Elsevier B.V. All rights reserved. doi:10.1016/j.schres.2010.09.015
One of the major objectives of the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Project was to develop a battery of tests that could be used to assess cognitive treatment effects in clinical trials of patients with schizophrenia (Green et al., 2004; Nuechterlein and Green, 2006). The MATRICS Neurocognition Committee
162
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
established criteria for the test battery emphasizing characteristics required for cognitive measures in the context of clinical trials: test–retest reliability; utility as a repeated measure; relationship to functional status; potential changeability in response to pharmacological agents; and practicality for clinical trials and tolerability for patients (Nuechterlein et al., 2008). The final battery of tests, the MATRICS Consensus Cognitive Battery (MCCB), was vetted through a detailed process involving leaders from academia, industry and NIMH, and validated through a 5-site academic consortium, the MATRICS Psychometric and Statistics Study (PASS). Normative data were collected on 300 healthy controls matched on key demographic factors to the United States population (Kern et al., 2008). The MCCB was approved by FDA for use in clinical trials for cognitive improvement in schizophrenia (Nuechterlein et al., 2008; Buchanan et al., 2005; Buchanan et al, 2010). In the MATRICS PASS validation study, conducted at 5 medical centers with schizophrenia cognition expertise, the MCCB tests demonstrated excellent reliability, minimal practice effects, and significant correlations with measures of functional capacity (Nuechterlein et al., 2008). However, it has been an empirical question whether the MCCB would demonstrate these favorable characteristics when administered in the context of the type of large multi-site industry trial for which it was designed. The current study is the first attempt to address this question; it describes cognitive data collected with the MCCB in a 29-site clinical trial assessing the comparative effects of antipsychotic treatment with lurasidone or risperidone in patients with schizophrenia. The design of this treatment trial involved a multiple-baseline assessment period during which patients were assessed twice with the MCCB, providing the opportunity to investigate the sensitivity, test–retest reliability, and predictive validity of the MCCB in a schizophrenia clinical trial setting. For the MCCB to be useful in clinical trials, it is expected that it will demonstrate high test–retest reliability, minimal practice effects, and substantial correlations with functional capacity co-primary measures such as the UCSD Performance-based Skills Assessment, Brief (UPSA-B) which has been identified as a leading co-primary measure for schizophrenia cognition trials. This study provides a test–retest convenience sample to assess these psychometric characteristics of the MCCB. 2. Methods 2.1. Patients Three hundred twenty-three clinically stable outpatients with schizophrenia in the United States were randomized 2:1 to lurasidone and risperidone flexible dose treatment
Table 1 Demographics and baseline characteristics. All subjects (N = 323) Age Gender Race
PANSS total score
43.1 (SD = 10.4) Male 231 (72%); female 92 (28%) White 99 (31%) African American 206 (64%) Other 18 (5%) 67.5 (SD = 11.7)
(clinicaltrials.gov ID: NCT00641745). Table 1 describes the demographic and baseline characteristics of this patient sample. The treatment protocol was approved by the Human Subjects Committees at the 29 sites. As part of the protocol screening process, patients received the full MCCB. Patients met the following inclusion criteria, which closely follow the MATRICS guidelines presented at a panel sponsored by FDA, NIMH and MATRICS (Buchanan et al., 2005): age 18–65; diagnosis of schizophrenia; CGI-S b 4 at both screening and baseline; no major change in antipsychotic medications for at least 6 weeks prior to screening; no hospitalization for psychiatric illness for at least 8 weeks prior to screening; and moderate or less (b4) severity rating on selected PANSS positive scale items at both screening and baseline. Patients were excluded from the study if they demonstrated any of the major exclusion criteria: resistance to antipsychotic treatment defined as failure to respond to 2 or more antipsychotic agents administered at an adequate dose for at least 6 weeks or a history of treatment intolerance to 3 or more antipsychotic agents; previous treatment with risperidone within 6 weeks prior to baseline; history of poor response to risperidone or intolerability to risperidone treatment; history of treatment with clozapine within 4 months of baseline visit or treatment with clozapine for refractory psychosis; history of antipsychotic withdrawal for N3 months without evidence of psychosis; unstable medical illness, or a history of hypersensitivity to risperidone or an allergic reaction to more than 2 chemical classes of drugs (prescription or non-prescription). Subjects were considered ineligible for participation in the cognitive component of the study described here if they met any of the following exclusion criteria relevant to the cognitive data analysis: treatment with N10 mg of haloperidol per day or equivalent dose of other conventional antipsychotics within 30 days prior to screening; history of moderate or greater head injury; clinically significant or history of alcohol abuse/ alcoholism or drug abuse/dependence within the last 6 months; history of previous psychosurgery; history of neuroleptic malignant syndrome; evidence of severe tardive dyskinesia, tardive dystonia, or other severe chronic movement disorder; clinically significant suicidal ideation or behavior or violent behavior in the last 6 months; electroconvulsive therapy within the past 3 months; treatment with mood stabilizers or antidepressants within 1 week, fluoxetine within 1 month, or MAO inhibitor within 3 weeks of baseline visit. Subjects voluntarily agreed to participate in the study by giving written informed consent. 2.2. MATRICS Consensus Cognitive Battery (MCCB) The MCCB measures seven separable cognitive domains: speed of processing; attention/vigilance; working memory (verbal and nonverbal); verbal learning; visual learning; reasoning and problem solving; and social cognition. Administration of the MCCB requires about 75–90 min. The following tests were administered in this standard order: • Trail making A • Brief Assessment of Cognition in Schizophrenia (BACS) Symbol Coding • Hopkins Verbal Learning Test – Revised (HVLT-R) • Wechsler Memory Scale III (WMS-III) Spatial Span
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
• • • • •
University of Maryland Letter Number Span Neuropsychological Assessment Battery (NAB) – Mazes Brief Visuospatial Memory Test – Revised (BVMT-R) Category Fluency Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT) Managing Emotions • Continuous Performance Test – Identical Pairs (CPT-IP) Since the overall study involved 5 assessments with the MCCB (including three assessments after treatment was initiated, whose results are not reported here), 2 alternate forms of the NAB Mazes and 4 alternate forms of the HVLT-R and BVMT-R were administered in a counterbalanced manner across patients. Thus all 50 combinations of these three tests were evenly distributed across the entire patient sample at each MCCB assessment point. All testers were trained on the administration and scoring of the MCCB using video and group training sessions and were individually certified by an expert on the MCCB. MCCB data were scored and sent to a central site, where they were rescored and double-entered data into the MCCB scoring program to ensure data accuracy. Training, data collection, and data quality assurance were implemented or supervised by an experienced psychologist as per the guidelines outlined in the MCCB manual (Nuechterlein and Green, 2006). The MCCB scoring program yields seven domain scores and a composite score which are standardized to the same T-score measurement scale with a mean of 50 and an SD of 10 based upon the normative data collected from a sample of 300 community participants as part of the MATRICS PASS and published in the MCCB manual and the MCCB scoring program (Nuechterlein and Green, 2006). 2.3. UCSD Performance-based Skills Assessment Brief version (UPSA-B) The UPSA-B was designed to evaluate the abilities of individuals to perform everyday tasks that are considered necessary for independent functioning in the community. The UPSA-B is a briefer version of the UPSA, which uses role-playing situations to evaluate skills in five areas: household chores, communication, finance, transportation, and planning recreational activities (Mausbach et al., 2007). The UPSA-B consists of two of the five original UPSA domains, finances and communication. Each of the two subscales contributes 50 points, and total scores range from 0 to 100 points; higher scores reflect better performance. The UPSA-B scores were found to correlate at a very high level, r = −.93, with scores on the five-subscale UPSA, and the UPSA-B has been shown to be equally sensitive to skills required in independent living when compared to the long form of the UPSA (Mausbach et al., 2007). The UPSA-B requires about 7–10 min to complete and, involves only props that are widely available (a telephone and money), commonly utilized and relevant at any site in the United States. Raters received specific training and education regarding this assessment prior to study initiation. 2.4. Cognitive Assessment Interview (CAI) The CAI is an interview-based measure of cognitive function that incorporates ratings from 3 primary sources – subject, informant, and interviewer – to produce a total score (Ventura
163
et al., 2010). The test consists of a set of 10 items, rated on a 7-point, anchor-based scale. Reasonable efforts were made to identify an informant (e.g., caregiver, family member, or friend) who was familiar with the subject's overall functioning and was willing to provide such information about the subject. An informant was not required for subjects' participation in the trial; only 123 of the patients had informants available to provide information for the CAI. The CAI raters were required to meet specific credential and educational criteria before they were certified to rate for this study.
2.5. Time course of assessments 2.5.1. Screening assessment Patients were assessed with the MCCB and PANSS at the screening visit while receiving their baseline antipsychotic medications, prior to the switch to risperidone or lurasidone. Patients who met all eligibility criteria at screening visit 1 were withdrawn from their current antipsychotic medication as tolerated and in accordance with labeling recommendations and conventional medical practice in a 1–7 day transition period. The baseline visit (day 0) was the final transition day and the day of randomization. Subjects could be hospitalized for up to 7 days during the transition period at the investigator's discretion.
2.5.2. Baseline assessment The MCCB and PANSS were administered for the second time along with the initial assessment with the UPSA-B and CAI. This baseline assessment was scheduled to be completed 7–21 days after screening. However, a small number of subjects received their baseline assessment outside the 7–21 day window.
2.6. Statistical analysis All data presented here were obtained from enrolled subjects prior to randomization to double-blind study treatment (at screening and/or baseline). Test–retest reliability for the individual cognitive domains and composite T-score were evaluated using Intra-Class Correlation (ICC) with the Shrout–Fleiss 2-way ANOVA for reliability of a single rating model. Practice effects for each cognitive domain and composite T-score were assessed by effect size (change score normalized by standard deviation of screening score). Measurement error and precision over time (at testing and retesting) were compared using a paired-sample test for two correlated variances. Missing data patterns were summarized for the MCCB individual domains and composite T-score. Predictive validity was assessed by correlations of MCCB composite T-score with the UPSA-B composite score and CAI total score at baseline. The primary CAI endpoint is Global Severity of Cognitive Impairment (Rater Composite Impression) based on information collected from patient and informant. Subjects with no informant will be included in the study, but excluded from the analysis of CAI composite score. Differences between patient and informant CAI ratings were examined using t-test.
164
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
3. Results 3.1. Completeness of study dataset Three hundred twenty-three patients were tested 2 times with the 10-test MCCB. Of these 6460 individual cognitive test assessments, 14 were invalid or missing (99.8%) (see Table 2). Three hundred eighteen of the 323 patients (98.5%) had complete MCCB assessments at screening and baseline. The published MCCB guidelines (Nuechterlein and Green, 2006) recommend that a composite score is valid if 5 of 7 domains are complete at pre-treatment assessment, and 4 of 7 are complete posttreatment assessment. All subjects met this criterion at baseline, and all but 1 subject met this criterion at the end of screening. 3.2. Sensitivity of MCCB to impairment with composite scores and domain scores
2.2 points. In calculating an effect size for this change, we utilized the standard deviation of the initial assessment in the denominator; the test–retest effect size using this formula (Cohen, 1977) was d = 0.18. The changes in domain scores from screening to baseline were also small, ranging from 2.4 points for speed of processing (d = 0.20) to 0.4 points for social cognition (d = 0.03). 3.5. Correlations between UPSA-B and MCCB The UPSA-B composite score had a slight negative skew (skewness = −0.70; normality test p b .05) The mean UPSA-B score at baseline was 70.2 (SD = 16.2). Despite the slight negative skew of the UPSA-B composite score, the correlation between UPSA-B composite score and MCCB composite score was substantial (r = 0.60, df = 314, p b .001; see Fig. 3). 3.6. MCCB correlations with CAI and PANSS
The distribution of the MATRICS battery composite scores at screening and baseline was normal (see Fig. 1). T-scores for all domains except Working Memory and Speed of Processing were significantly different from normal. Table 3 describes the mean performance of patients at screening and at baseline, which was a mean/median of 18 days later (SD 5 days). Mean composite T-scores (SD) at screening and baseline were 24.7 (12.1) and 26.9 (12.4), indicating that schizophrenia patients in this study performed at 2.5 SD below the healthy controls from the normative sample. The profile of impairments across the 7 domains was variable, but for all domains ranged between 1 and 2 SD below healthy controls.
Spearman rank correlation of the baseline MCCB composite score with the baseline CAI interviewer total score (for global severity of cognitive impairment based on patient and informant interviews) was r = −0.14 (df= 121; P = .066) in the subgroup of patients with an informant, suggesting a weak relationship between poor performance on the MCCB and interview-based reports of cognitive impairment. The mean PANSS total score at baseline was 67.5 (SD = 11.7). Correlation between MCCB composite score and baseline PANSS total score was r=−0.20, df= 319, p b .0010, indicating that patients who had less severe symptoms performed slightly better on the MCCB.
3.3. Test–retest reliability 4. Discussion The test–retest reliability of the composite score was excellent (ICC = 0.88, DFerror = 318, p b .001) (see Table 3 and Fig. 2). The test–retest reliability of the individual domains was variable ranging from a high of 0.79 for speed of processing, attention/vigilance and working memory, to 0.58 for verbal learning and 0.65 for visual learning. It is noteworthy that the 2 domains with the lowest test–retest reliability involve the use of alternate forms between sessions. 3.4. Practice effects Practice effects were small (see Table 3). The composite T-score change from screening to baseline improved a mean of
The MATRICS Neurocognition Committee determined that the most important characteristics required for cognitive measures in the context of clinical trials included: test–retest reliability; utility as a repeated measure, such as minimal practice effects; relationship to functioning or functional capacity; and practicality for clinical trials and tolerability for patients (Nuechterlein et al., 2008). In this 29-site clinical trial designed to assess the comparative effects of lurasidone or risperidone in patients with schizophrenia, 323 patients were assessed at screening and then again after an average of 18 days with the MATRICS Consensus Cognitive Battery (MCCB), providing an opportunity to assess the psychometric charac-
Table 2 MCCB test scores: quality assurance. All subjects (N = 323) Total number of invalid or missing MCCB tests of total tests (N = 323 patients tested twice on 10 tests = 6460 tests) Subjects with complete MCCB tests (excluding invalid or missing tests) Screening Baseline Screening and Baseline scores Subjects with missing N = 3 tests Screening Baseline Duration between screening and baseline tests
14 320/323 (99.1%) 321/323 (99.4%) 318/323 (98.5%) 1 (0.3%) 0 (0%) 18 days (SD 5) Range = 6–37
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
165
Fig. 1. Screening and baseline MCCB composite T-score: Normal density estimate and histogram (N = 323). Mean T-score at baseline = 26.9 (SD = 12.4); skewness = 0.13; normality test p N 0.05. Mean T-score at screening 24.7 (SD = 12.1). Area under the curve (histogram) = 1.
teristics of the MCCB in the context of the type of trial and the type of sites for which it was designed. In this study, the MCCB demonstrated clear sensitivity to the cognitive deficits of schizophrenia, excellent test–retest reliability, minimal practice effects, and significant correlations with measures of functional capacity. One of the early concerns with the MCCB was that the burden of cognitive assessments, especially in psychotic patients, would lead to a large number of missing variables. These concerns may be particularly strong for clinical trials with antipsychotic medications, since patients with psychosis are presumed to be more difficult to test. To address these concerns in this antipsychotic trial, strict tester training and certification procedures and data review processes were put in place. The result was that only 14 item scores (0.2%) were missing out of a total of 6460 assessments. Since composite scores can be calculated if 5 of 7 domains are validly collected, only one patient in 323 did not have sufficient data to calculate a composite score from the testing occasions at both screening and baseline. The completeness of this study data set suggests that missing data with the MCCB should be rare, although this
result may depend upon the utilization of intensive rater training and data review processes. The average MCCB composite score for the patients in this study was 24.7 at screening and 26.9 at baseline, suggesting an overall impairment of about 2.5 standard deviations compared to the healthy controls included in the MCCB normative study (Kern et al., 2008). This level of severity is consistent with preliminary data from other studies (Buchanan et al, in press; Javitt et al, submitted; Buchanan et al, submitted) It is noteworthy that the composite score from the MCCB is more diagnostically sensitive and more reliable than any of the individual domain scores, and more sensitive than any single test scores in previous meta-analyses (Heinrichs and Zakzanis, 1998). This increased sensitivity is due to the moderate correlation among the MCCB tests (Nuechterlein et al., 2008), which serves to reduce the variability of the composite score compared to the domain scores, and creates a measure that may be more sensitive to treatment effects than a single domain score. From a survey of 68 experts on schizophrenia cognition, the MATRICS project investigators concluded that test–retest
Table 3 MCCB composite T-score: screening versus baseline (N = 323).
Composite T-score Speed of processing Attention/vigilance Working memory Verbal learning Visual learning Reason/problem solving Social cognition a
Screening mean (SD)
Baseline mean (SD)
ICC
Test–retest effect size a
p-value (SDScreening = SDBaseline)
24.7 (12.1) 31.3 (12.0) 34.2 (11.8) 32.8 (11.1) 35.1 (7.0) 32.7 (11.8) 38.9 (8.1) 36. (13.0)
26.9 33.7 36.2 34.5 35.7 34.2 39.9 36.4
0.88 0.79 0.79 0.79 0.58 0.65 0.77 0.75
0.18 0.20 0.17 0.16 0.09 0.13 0.14 0.03
0.32 0.77 0.06 0.73 0.13 0.73 b0.001 0.44
(12.4) (12.1) (12.5) (11.2) (7.5) (11.7) (9.3) (12.6)
Effect size for paired sample = (Baseline − Screening)/SDScreening.
166
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
Fig. 2. MCCB composite T-score at screening and baseline (N = 318).
reliability was the most important feature of a test battery to be used in schizophrenia clinical trials (Green et al., 2004). In a separate survey conducted by the leaders of another NIMH Project, the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS), the optimal test–retest reliability was determined to be an ICC of 0.90
(Barch et al., 2008). In this study, the test–retest reliability of the MCCB composite score was ICC= 0.88, very closely matching the ideal cited by the CNTRICS experts. The test–retest reliability of the domain scores was lower, although acceptable, ranging between 0.75 and 0.79 in the tests without alternate forms. It is noteworthy that the test–retest reliability of the two
Fig. 3. Cross-sectional relationship between MCCB composite T-score and UPSA-B composite score.
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
memory tests that use alternate forms, visual learning and verbal learning, were particularly low at 0.65 and 0.58 respectively. While the design of the MCCB allows an emphasis on individual domains, and such a focus is encouraged in the published FDA-NIMH-MATRICS guidelines (Buchanan et al., 2005), these low reliabilities suggest that perhaps additional memory tests should be included if verbal or visual memory is to be the primary outcome of a trial so that domain-specific reliability can be enhanced. Previous studies using very small samples have reported that the practice effects of the MCCB are medium size (Freedman et al., 2008), raising concerns that the use of the MCCB in a repeated measures design may reduce sensitivity to real treatment effects (Lieberman et al., 2008). In the current study, the composite score improvement over an average period of 18 days was 2.2 points, equal to a Cohen's d of 0.18, which is less than “small” according to Cohen's definition (Cohen, 1977). This amount of practice effect raises the average level of the deficit in these patients from 2.5 to 2.3 standard deviations below the healthy control mean. Thus, the test battery allows substantial room for sensitivity to any real treatment effects. This amount of improvement with practice is entirely consistent with the amount of improvement reported in previous studies of practice-related cognitive changes in patients with psychotic disorders and healthy controls (Goldberg et al., 2007, 2010). The FDA-NIMH-MATRICS guidelines state that if FDA requires a co-primary measure in addition to the MCCB for approval of a cognitive-enhancing compound, the measure should be expected to change in close temporal proximity to changes on cognitive performance measures. It is difficult to determine if the MCCB and the set of currently available potential co-primary measures will change in parallel with one another in response to a cognitive-enhancing compound, since currently there are no effective treatments for cognitive impairment in schizophrenia. However, one source of evidence that a cognitive measure may change in conjunction with a co-primary measure is whether they are correlated with one another at baseline. There has been ample evidence recently from academic research sites that measures of functional capacity such as the UPSA and UPSA-B are strongly correlated with a variety of neurocognitive measures, with several studies reporting correlations in the range of 0.54 to 0.67 (Leifker et al., 2009). The data from the current study suggest that this correlation holds true even in large multisite studies, as the Spearman correlation between the MCCB composite score and the UPSA-B total score of 0.60 fits squarely within the range of previous findings from academic research centers. The lack of a significant correlation with the CAI, an interview-based assessment of cognition, suggests that this rating scale measures a different aspect of cognition from the performance-based elements of the MCCB. In sum, in a 29-site antipsychotic clinical trial assessing 323 schizophrenia patients at screening and then 2–3 weeks later at baseline, the MCCB was able to generate a very complete data set, with composite scores that demonstrated high sensitivity to cognitive impairments in schizophrenia, excellent test–retest reliability, small practice effects, and strong correlations with functional capacity. The individual domain scores demonstrated substantial sensitivity to impairment, although less than composite score, with similar or
167
smaller practice effects, and weaker reliability, especially for tests with alternate forms. Role of funding source This study was designed and supported by Dainippon Sumitomo Pharma America. NeuroCog Trials trained the cognitive testers and conducted data quality review. Analyses were completed by Dr. Cynthia Sui in consultation with the manuscript co-authors. The decision to submit the paper for publication was made by all co-authors. Following the role of the funding source text, authors are required to declare their individual contribution to the manuscript under Contributors. Contributors Dr. Keefe wrote the first draft of the paper. All other authors edited and commented on versions of the manuscript and approved the final version. This study was designed and supported by Drs. Loebel, Cucchiaro, and others at Dainippon Sumitomo Pharma America, now Sunovion. NeuroCog Trials trained the cognitive testers and conducted data quality review under the supervision of Drs Fox and Keefe. Analyses were completed by Dr. Cynthia Sui in consultation with the manuscript co-authors. Dr. Harvey consulted on the design of the study. Conflict of interest Dr. Keefe reports that he currently or in the past 3 years has received investigator-initiated research funding support from the Allon, Eli Lilly, GlaxoSmithKline, Novartis, Department of Veteran's Affairs, National Institute of Mental Health, and the Singapore National Medical Research Council, and an unrestricted educational grant from Astra-Zeneca and Organon. He currently or in the 3 years has received honoraria, served as a consultant, or advisory board member for Abbott, Acadia, BiolineRx, BrainCells, Bristol-Myers Squibb, CHDI, Cypress BioScience, Dainippon Sumitomo Pharma/Sepracor/Sunovion, Eli Lilly, EnVivo, Johnson & Johnson, Lundbeck, Memory Pharmaceuticals, Merck, Neurosearch, Orexigen, Orion, Otsuka, Pfizer, Prophase, Roche, Sanofi-Aventis, Shering-Plough, Shire, Solvay, Takeda, World Wide Clinical Trials, Wyeth, and Xenoport. Dr. Keefe receives royalties from the Brief Assessment of Cognition in Schizophrenia (BACS) testing battery and the MATRICS Battery (BACS Symbol Coding). He is also a shareholder in NeuroCog Trials, Inc. NeuroCog Trials conducted cognitive rater training and data quality review for this study. Dr. Fox is a full time employee of NeuroCog Trials, Inc. Dr. Harvey reports in the past 3 years he has served as a consultant to Abbott Labs, Cypress Bioscience, Dainippon Sumitomo Pharma America/ Sepracor/Sunovion, Eli Lilly, Merck and Company, Shire Pharma, Solvay Pharma, Teva Pharma., and Wyeth Pharma (now Pfizer). Also, he has a grant from Astra-Zeneca. Dr. Cucchiaro reports that she is a full time employee of Dainippon Sumitomo Pharma/Sepracor/Sunovion, Inc. Dr. Siu was a paid consultant to Dainippon Sumitomo Pharma/Sepracor/ Sunovion in connection with the statistical analysis and development of this manuscript and has served as a consultant to Pfizer, Memory Pharmaceutical/Roche Laboratories, and Wyeth/Pfizer over the past 3 years. Dr. Loebel reports that he is an employee of Dainippon Sumitomo Pharma/Sepracor/Sunovion. Acknowledgements None.
References Barch, D.M., Carter, C.S., CNTRICS Executive Committee, 2008. Measurement issues in the use of cognitive neuroscience tasks in drug development for impaired cognition in schizophrenia: a report of the second consensus building conference of the CNTRICS initiative. Schizophr. Bull. 34 (4), 613–618. Buchanan, R.W., Davis, M., Goff, D., Green, M.F., Keefe, R.S.E., Leon, A., Nuechterlein, K., Laughren, T., Levin, R., Stover, E., Fenton, W., Marder, S., 2005. A summary of the FDA-NIMH-MATRICS workshop on clinical trial design for neurocognitive drugs for schizophrenia. Schizophr. Bull. 31 (1), 5–19. Buchanan, R.W., Keefe, R.S.E., Umbricht, D., Green, M.F., Laughren, T., Marder, S.R., 2010. The FDA-NIMH-MATRICS guidelines for clinical trial design of cognitive-enhancing drugs: what do we know 5 years later? Schizophr. Bull. (epub ahead of print).
168
R.S.E. Keefe et al. / Schizophrenia Research 125 (2011) 161–168
Cohen, J., 1977. Statistical power analysis for the behavioral sciences. Academic Press, Inc., Orlando, Florida. Freedman, R., Olincy, A., Buchanan, R.W., Harris, J.G., Gold, J.M., Johnson, L., Allensworth, D., Guzman-Bonilla, A., Clement, B., Ball, M.P., Kutnick, J., Pender, V., Martin, L.F., Stevens, K.E., Wagner, B.D., Zerbe, G.O., Soti, F., Kem, W.R., 2008. Initial phase 2 trial of a nicotinic agonist in schizophrenia. Am. J. Psychiatry 165 (8), 1040–1047. Goldberg, T.E., Goldman, R.S., Burdick, K.E., Malhotra, A.K., Lencz, T., Patel, R.C., Woerner, M.G., Schooler, N.R., Kane, J.M., Robinson, D.G., 2007. Cognitive improvement after treatment with second-generation antipsychotic medications in first-episode schizophrenia: is it a practice effect? Arch. Gen. Psychiatry 64 (10), 1115–1122. Goldberg, T.E., Keefe, R.S., Goldman, R.S., Robinson, D.G., Harvey, P.D., 2010. Circumstances under which practice does not make perfect: a review of the practice effect literature in schizophrenia and its relevance to clinical treatment studies. Neuropsychopharmacology 35 (5), 1053–1062. Green, M.F., Nuechterlein, K.H., Gold, J.M., Barch, D., Cohen, J., Essock, S., Fenton, W.S., Frese, F., Goldberg, T.E., Heaton, R.K., Keefe, R.S.E., Kern, R.S., Kraemer, H., Stover, E., Weinberger, D.R., Zalcman, S., Marder, S.R., 2004. Approaching a consensus cognitive battery for clinical trials in schizophrenia: the NIMH-MATRICS conference to select cognitive domains and test criteria. Biol. Psychiatry 56 (5), 301–307. Heinrichs, R.W., Zakzanis, K.K., 1998. Neurocognitive deficit in schizophrenia: a quantitative review of the evidence. Neuropsychology 12 (3), 426–444. Kern, R.S., Nuechterlein, K.H., Green, M.F., Baade, L.E., Fenton, W.S., Gold, J.M., Keefe, R.S.E., Mesholam-Gately, R., Mintz, J., Seidman, L.J., Stover, E.,
Marder, S.R., 2008. The MATRICS consensus cognitive battery: Part 2. Co-norming and standardization. Am. J. Psychiatry 165 (2), 214–220. Leifker, F.R., Bowie, C.R., Harvey, P.D., 2009. Determinants of everyday outcomes in schizophrenia: the influences of cognitive impairment, functional capacity, and symptoms. Schizophr. Res. 115 (1), 82–87. Lieberman, J.A., Javitch, J.A., Moore, H., 2008. Cholinergic agonists as novel treatments for schizophrenia: the promise of rational drug development for psychiatry. Am. J. Psychiatry 165 (8), 931–936. Mausbach, B.T., Harvey, P.D., Goldman, S.R., Jeste, D.V., Patterson, T.L., 2007. Development of a brief scale of everyday functioning in persons with serious mental illness. Schizophr. Bull. 33 (6), 1364–1372. Nuechterlein, K.H., Green, M.F., 2006. MATRICS Consensus Battery Manual. MATRICS Assessment Inc., Los Angeles. Nuechterlein, K.H., Green, M.F., Kern, R.S., Baade, L.E., Barch, D., Cohen, J., Essock, S., Fenton, W.S., Frese, F.J., Gold, J.M., Goldberg, T., Heaton, R., Keefe, R.S.E., Kraemer, H., Mesholam-Gately, R., Seidman, L.J., Stover, E., Weinberger, D., Young, A.S., Zalcman, S., Marder, S.R., 2008. The MATRICS consensus cognitive battery: Part 1. Test selection, reliability, and validity. Am. J. Psychiatry 165 (2), 203–213. Ventura, J., Reise, S.P., Keefe, R.S.E., Baade, L.E., Gold, J.M., Green, M.F., Kern, R.S., Mesholam-Gately, R., Nuechterlein, K.H., Seidman, L.J., Bilder, R.M., 2010. The Cognitive Assessment Interview (CAI): development and validation of an empirically derived, brief interview-based measure of cognition. Schizophr. Res. 121 (1-3), 24–31.