- Email: [email protected]
Validity of a multi-domain computerized cognitive assessment battery for patients with multiple sclerosis
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
Contents lists available at ScienceDirect
Multiple Sclerosis and Related Disorders journal homepage: www.elsevier.com/locate/msard
Clinical trial
Validity of a multi-domain computerized cognitive assessment battery for patients with multiple sclerosis
T
Daniel Golana,b, Jeffrey Wilkenc,d, Glen M. Donigere,f, Timothy Frattoc, Robert Kanec, Jared Srinivasang, Myassar Zarifg, Barbara Bumsteadg, Marijean Buhseg, Lori Fafardg, ⁎ Ilir Topallig, Mark Gudesblattg, a
Department of Neurology & Multiple Sclerosis Center, Lady Davis Carmel Medical Center, Haifa, Israel Rappaport Faculty of Medicine, Technion-Israel Institute of Technology, Haifa, Israel c Washington Neuropsychology Research Group, Fairfax, VA, USA d Department of Neurology, Georgetown University, Washington DC, USA e Department of Clinical Research, NeuroTrax Corporation, Modiin, Israel f Joseph Sagol Neuroscience Center & Center of Advanced Technologies in Rehabilitation, Sheba Medical Center, Ramat Gan, Israel g South Shore Neurologic Associates, 77 Medford Avenue, Patchogue, NY 11772, USA b
A R T I C LE I N FO
A B S T R A C T
Keywords: Cognitive function Computerized cognitive assessment Multiple sclerosis Response time Validity
Background: Computerized cognitive batteries may facilitate the integration of neuropsychological assessments into routine clinical care of patients with multiple sclerosis (PwMS). Objective: To assess the construct and criterion validity of a computerized, multi-domain cognitive assessment battery (CAB, NeuroTrax) in MS. Methods: 81 PwMS and 15 healthy controls (HC) completed the CAB and a set of traditional neuropsychological tests recommended for MS on the same day. Principal component factor analysis was used to assess construct validity. For criterion validity, the gold standard definition of cognitive impairment was a score of ≥1.5SD below average on at least one cognitive domain, based upon traditional test normative data. Receiver operating characteristic (ROC) analysis was used to examine the ability of the CAB to discriminate cognitively impaired PwMS. Results: Traditional and computerized tests of memory, processing speed, visuospatial and executive function converged by factor analysis. Computerized tests detected cognitive impairment with 85% sensitivity and 70% specificity. PwMS classified as impaired on only the computerized battery had significantly prolonged response times and a higher rate of unemployment compared with PwMS classified as unimpaired on both batteries. Poor executive function was more likely to be revealed by the CAB. Conclusion: The specific computerized assessment battery evaluated is valid for cognitive screening of people with MS and may be more likely to detect prolonged response times and impaired executive function.
1. Introduction Cognitive dysfunction is a prevalent debilitating symptom of Multiple Sclerosis (MS). More than 50% of people with MS (PwMS) experience some form of cognitive impairment (Sumowski et al., 2018). MS-related cognitive dysfunction may prevent maintaining gainful employment (Rao et al., 1991a), adversely impact quality of life (Rao et al., 1991b) and may be a manifestation of disease activity (Benedict et al., 2014). Routine neurological assessment lacks sensitivity in identifying cognitive impairment. Neurologists' prediction of cognitive impairment based on a typical clinical visit is not significantly ⁎
different from chance (Romero et al., 2015). It is therefore important to include some form of objective cognitive assessment in the routine care and management of PwMS. In 2001 an expert panel of MS neuropsychologists recommended a 90-min battery, Minimal Assessment of Cognitive Function in MS (MACFIMS), covering 5 cognitive domains (processing speed/working memory, learning and memory, executive function, visual-spatial processing, and word retrieval) (Benedict et al., 2002). Unfortunately, this battery has several limitations which have resulted in its infrequent use in PwMS routine care and clinical trials. Among these limitations are: dependence upon neuropsychologists to supervise testing, labor-
Corresponding author. E-mail addresses: [email protected] (D. Golan), [email protected] (M. Gudesblatt).
https://doi.org/10.1016/j.msard.2019.01.051 Received 10 September 2018; Received in revised form 4 January 2019; Accepted 29 January 2019 2211-0348/ © 2019 Elsevier B.V. All rights reserved.
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
by the traditional neurocognitive tests. A 10-min break was given between the two testing batteries. Participants were instructed not to take psychotropic medications (benzodiazepines, opioids) in the 6 h prior to testing. Cognitive assessments were supervised by a neuropsychologist.
intensive manual computation of standardized scores, lack of alternate forms for some components, and lengthy administration times. One proposed solution is to decrease the number of domains tested (Langdon et al., 2012). This approach may lead to loss of critical information for identifying the spectrum and degree of cognitive impairment in the population of PwMS and for any individual PwMS. Computerized screening tools offer the opportunity to incorporate such objective cognitive screening measures into routine PwMS clinical care while retaining broad multi-domain assessment. Periodic computerized cognitive screening is already integrated into our routine for PwMS care, and data from ∼700 PwMS has provided further insight into the impact of fatigue and depression on MS-related cognitive impairment (Golan et al., 2018). Additional merits of computerized tools include: inherent administration standardization, instant result normalization, multiple alternate forms, and testing session supervision not requiring physician/neuropsychologist involvement. The computerized cognitive assessment battery (CAB) utilized in this study (NeuroTrax) has been validated and used to screen for cognitive impairment in other neurological diseases (Dwolatzky et al., 2003; Schweiger et al., 2003) (see the supplementary materials). Significant correlations have been shown between CAB domain scores and test results from Rao's neuropsychological screening battery among 58 PwMS (Achiron et al., 2007). In that study, most CAB index scores significantly discriminated PwMS from healthy individuals (Achiron et al., 2007). In this study we further evaluated the construct and criterion validity of this CAB among PwMS relative to a set of recommended, traditional neuropsychological tests, mainly from the Minimal Assessment of Cognitive Function in MS (MACFIMS) battery.
2.3. Traditional cognitive testing Traditional neuropsychological tests were mainly from the Minimal Assessment of Cognitive Function in MS (MACFIMS) battery, which was developed for use in PwMS by expert consensus (Benedict et al., 2002) with subsequent empirical validation (Benedict et al., 2006). The only difference was inclusion of the Selective Reminding Test (SRT) instead of the California Verbal Learning Test-II (CVLT-II). These two tests have shown comparable sensitivity in discriminating PwMS from demographically-matched controls (Strober et al., 2009). The SRT is also recommended for PwMS as part of the MS-COG battery (Sumowski et al., 2018; Erlanger et al., 2014). We opted for the SRT due to substantially reduced administration time than for the CVLT-II, and availability of multiple forms of SRT for longitudinal follow-up. The traditional cognitive assessment consisted of 7 tests which can be grouped into 5 domains (Benedict et al., 2002): information processing speed and working memory—Paced Auditory Serial Addition Test (PASAT) Trial 1: three second and Trial 2: two second (Rao et al., 1991b); SDMT (Sheridan et al., 2006). Memory—SRT for verbal memory (Ehrenreich, 1995); Brief Visuospatial Memory Test-Revised (BVMT-R) (Benedict, 1997) for visual memory. Verbal fluency—Controlled Oral Word Association Test (COWAT) (Heaton et al., 2004). Visuospatial processing—Judgment of Line Orientation (JLO) (Mitrushina et al., 2005). Executive function—Delis-Kaplan Executive Function System (D-KEFS) Sorting Test (Delis et al., 2001). For all tests, z-scores were calculated based on published normative data. To improve the robustness of cognitive classification, the z-scores of all tests within a particular cognitive domain were averaged to obtain summary cognitive domain z-scores.
2. Methods 2.1. Participants 81 PwMS, according to the revised McDonald criteria, under treatment at one MS Care Center (South Shore Neurologic Associates, New York, USA) were recruited for this study. Inclusion criteria were: age 25–55, vision corrected to at least 20/50, 12+ years of education, able to write and press buttons on a computer mouse. Exclusion criteria were: unable to understand and follow all test instructions, history of head injury, seizures, neurological conditions involving the central nervous system (other than MS), colorblindness, history of psychosis or other severe mental illness (other than mild-to-moderate depression), current alcohol/substance abuse and MS relapse < 30 days prior to enrollment. 15 healthy controls (HC) underwent equivalent study procedures. Inclusion and exclusion criteria for HC were similar to those of PwMS. All HC had to be free of neurological conditions known to impact cognitive function. For each participant, all evaluations (see below) were carried out on the same day. Institutional review board (IRB) approval was obtained, and all participants gave informed consent.
2.4. Gold standard definition of cognitive impairment PwMS with a cognitive domain z-score of −1.5 or lower on at least one cognitive domain were defined as cognitively impaired by ‘gold standard’ traditional tests (Sumowski et al., 2018). 2.5. Computerized assessment battery This CAB consists of 10 tests: Verbal Memory, Non-Verbal Memory, Go-NoGo, Stroop, Verbal Function (naming, rhyming), Problem Solving, Visuospatial Processing, Staged Information Processing Speed (IPS), Finger Tapping, and a psychomotor “Catch Game”. For tests measuring response time, participants respond by clicking one mouse button on a two-button mouse (to avoid confounding effects of weakness or poor coordination, no movement of the mouse is required). Further detailed description of these tests is available in the supplementary materials. To minimize differences in age and education and to permit averaging performance across different types of outcome parameters (e.g., accuracy, response times), each CAB outcome parameter is normalized and fit to an IQ-style scale (mean: 100, SD: 15) in an age- and education-specific fashion. Normative data is from individuals classified as cognitively healthy in controlled studies conducted at academic research centers. Normalized subsets of outcome parameters were averaged to produce nine summary scores, each indexing a different cognitive domain (Table 1). A Global Cognitive Score was computed as the average of the 9 domain index scores.
2.2. Study procedures To examine ability of the CAB to detect various degrees of cognitive impairment, PwMS participants were first screened for cognitive dysfunction using the Symbol Digit Modalities Test (SDMT). Based on SDMT results and normative data (Smith, 1982; Sheridan et al., 2006), PwMS participants were stratified into 4 subgroups: the first subgroup had performance better than 1.00 standard deviation below the age expected mean. The other 3 subgroups had SDMT scores 1.00 to 1.50, 1.51 to 2.00 and >2.00 standard deviations below the age expected mean. An equal number of participants were recruited to each impairment-based subgroup. All participants completed the computerized battery first, followed
3. Statistical analysis Data analysis was performed using SPSS, version 23.0 (IBM Corp., 155
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Table 1 Cognitive domain scores components of the computerized assessment battery. Domain
Averaged componentsa
Memory
Verbal Memory: Total Accuracy Delayed Verbal Memory: Accuracy Non-Verbal Memory: Total Accuracy Delayed Non-Verbal Memory: Accuracy Go-NoGo: Composite Score Verbal Memory: Accuracy, Repetition 1 Non-Verbal Memory: Accuracy, Repetition 1 Go-NoGo: Composite Score Stroop: Composite Score, Level 3 Catch Game: Total Score Go-NoGo: Response Time Go-NoGo: Response Time Standard Deviation Stroop Interference: Response Time, Level 2 Staged Information Processing Speed: Response Time, Level 1.2 Staged Information Processing Speed: Accuracy, Level 2.3 Staged Information Processing Speed: Composite Scores, Levels 1.1, 1.3, 2.1 and 2.2 Visual Spatial Processing: Accuracy Verbal Function: Rhyming, Accuracy Finger Tapping: Inter-Tap Interval Finger Tapping: Tap Interval Standard Deviation Catch Game: Time to Make First Move Problem Solving: Accuracy
Working Memory
Executive Function
Attention
Information Processing Speed Visual Spatial Verbal Function Motor Skills
Problem Solving
Table 2 Patient characteristicsa. N Age Female Race Education (years) EDSSb Disease durationc (years) Disease course
Disease modifying medication
a b c
81 45 ± 8.1 59 (73%) White: 77 (95%) African American: 4 (5%) 14.5 ± 2.4 4 ± 1.7 10.2 ± 7.3 Relapsing remitting: 74 (91%) Secondary progressive: 5 (6%) Primary progressive: 2 (3%) Natalizumab 48 (59%) Dimethyl Fumarate 11 (14%) Fingolimod 6 (7%) Ocrelizumab 4 (5%) Glatiramer acetate 3 (4%) Alemtuzumab 2 (3%) No medication 7 (8%)
Summary statistics are mean ± standard deviation. EDSS, Expanded Disability Status Scale. Time from first confirmed clinical episode suggestive of multiple sclerosis.
COWAT, which taps search and retrieval of verbal information). For the second factor (F2), tests loading on executive functions converged (Computerized: Problem Solving; traditional: D-KEFS sorting and COWAT). In the third factor (F3), computerized and traditional tests of visuospatial processing clustered. The fourth factor (F4) represents memory, wherein traditional and computerized tests of verbal and visual memory converged. The last factor (F5) contained only computerized tests of verbal function (naming and rhyming); this domain was not covered by recommended traditional tests.
a Details regarding the individual tests are provided in the supplementary materials.
Armonk, N.Y., USA). Principal components factor analysis with varimax rotation was conducted to assess how raw scores from traditional and computerized tests in PwMS clustered. Continuous variables were analyzed using between-groups t-test, with corresponding effect sizes (Cohen's d). Receiver operating characteristic (ROC) analysis was used to evaluate the ability of the CAB cognitive domain (index) scores to discriminate PwMS with cognitive impairment from unimpaired PwMS, as defined by traditional tests. Sensitivity, specificity and accuracy of the CAB domain scores for predicting the ‘gold standard’ classification of cognitive impairment based on traditional measures were calculated, using two common cutoffs: ≤1 (domain score ≤85) and ≤1.5 (domain score ≤77.5) standard deviations below the age and education expected mean. Finally, sensitivity, specificity and accuracy of various combinations of the computerized domain scores for predicting the ‘gold standard’ classification of cognitive impairment were tested. Domain scores were analyzed individually, using the cutoff that yielded the best classification accuracy. Domains were retained if they improved the ability of the CAB to discriminate among groups. Hence, the optimal combination of domain scores was ascertained. Demographics by subgroup of cognitive impairment classification were compared using analysis of variance (ANOVA) for continuous variables and Fisher's exact test for categorical variables, followed by Bonferroni-adjusted pairwise comparisons. Descriptive statistics are reported as mean ± standard deviation.
4.2. Criterion validity Computerized domain scores discriminated PwMS from healthy controls (Table 4). Age- and education-adjusted computerized domain scores were significantly lower among PwMS as compared to healthy controls, with large effect sizes for the computerized memory, information processing speed, working memory, attention, executive functions, visuospatial processing and motor domains. Verbal and problem solving domain scores were not significantly different between the groups. Notably, PwMS and healthy controls were comparable in years of education and gender (Table 2); however, controls were significantly younger than patients [33.3 ± 8.1 vs. 45.0 ± 8.1 years, P < 0.001]. Notably, age-adjusted scores were analyzed mitigating this difference. 58 PwMS were cognitively impaired and 23 were cognitively normal, by 'gold standard' traditional tests. All the computerized domain scores discriminated cognitively impaired from cognitively unimpaired PwMS (Table 5). For all computerized domain scores, a cutoff for impairment of ≥1 standard deviation below the age- and educationadjusted mean (domain score ≤85) was superior, defined as improved sensitivity and accuracy without substantially reduced specificity, as compared to the alternative ≥1.5 standard deviations below the normative mean (Table 5). Consequently, we used the domain score ≤85 cutoff for analysis of the discriminability of various domain score combinations (Table 6) and for describing frequency of impairment (Figs. 1 and 3a). The optimal computerized domain combination for predicting cognitive impairment among PwMS, as defined by traditional tests, included memory, information processing speed, attention, working memory and visuospatial indices. Computerized domain score ≤85 in at least one of these 5 domains predicted 'gold standard' cognitive impairment with 85% sensitivity and 70% specificity. Using the complete set of 9 computerized domains improved the sensitivity to 86% but decreased specificity to 57% (Table 6).
4. Results 4.1. Construct validity Patient characteristics are shown in Table 2. Traditional and computerized tests converged by factor analysis. Five components were rotated, together explaining 66.4% of the variance (Table 3). The first factor (F1) included tests related to attention, processing speed and response times (Computerized: IPS, Stroop, Go-NoGo, “Catch Game”; Traditional: SDMT). Tests from the first factor also rely on executive abilities (Computerized: Stroop and Go-NoGo, demanding response inhibition and the “Catch Game”, which requires planning; Traditional: 156
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Table 3 Construct validity of traditional and computerized batteries by principal components factor analysis. Testa
F1
C: Staged Information Processing Speed: Composite Score, Level 2.2 C: Stroop: Composite Score, Level 3 C: Go-NoGo: Composite Score C: Catch Game: Total Score C: Visual Spatial Processing: Accuracy C: Problem Solving: Accuracy C: Delayed Non-Verbal Memory: Accuracy C: Delayed Verbal Memory: Accuracy C: Verbal Function: Matching, Accuracy C: Verbal Function: Rhyming, Accuracy T: Symbol Digit Modalities Test (SDMT), Oral T: Controlled Oral Word Association Test (COWAT), Animals T: Controlled Oral Word Association Test (COWAT), F-A-S T: Delis-Kaplan Executive Function System (D-KEFS), Correct Sorts T: Paced Auditory Serial Addition Test (PASAT), 3 s T: Brief Visuospatial Memory Test-Revised (BVMT-R), Delayed Recall T: Judgment of Line Orientation (JLO), Number Correct T: Selective Reminding Test (SRT), Delayed Recall % variance explainedb
0.76 0.73 0.77 0.64 0.4
F2
F3
F4
0.4 0.4
0.77 0.56 0.44
F5
0.4 0.66 0.84 0.78
0.58 0.45
0.54 0.67 0.76 0.4 0.71 0.5
0.41 0.77
17
15.6
0.83 10.8
12.4
10.7
C, computerized; T, traditional. a Loadings <0.4 are omitted. b Percent of variance explained by all 18 variables for each factor. Table 4 Group data comparing age- and education-adjusted computerized cognitive scores of patients with multiple sclerosis and normal healthy controls.
Global Cognitive Score Memory Information Processing Speed Working Memory Attention Executive Function Visual Spatial Motor Skills Verbal Function Problem Solving a b c
MS Na
Score
81 81 76 81 81 81 79 77 80 79
89.1 87.7 85.8 89.4 86.6 90.0 96.5 89.9 87.6 91.8
± ± ± ± ± ± ± ± ± ±
13.9 18.4 17.5 13.4 18.3 14.9 16.8 18.5 26.6 17.7
Controls (N = 15) Score
Cohen's db
P valuec
101.2 ± 6.9 101.5 ± 9.2 98.0 ± 11.8 102.0 ± 123 100.5 ± 7.6 103.5 ± 10.2 109.0 ± 11.8 104.7 ± 4.3 95.5 ± 21.7 95.9 ± 15.4
1.1 0.93 0.82 0.97 0.96 1.1 0.86 1.1 0.33 0.25
<0.001 <0.001 0.002 0.001 <0.001 0.001 0.007 <0.001 0.28 0.4
Outcome parameter scores may be missing if patient responses are insufficient on relevant tests. By convention, Cohen’s d values of 0.8, 0.5, and 0.2 represented large, medium and small effect sizes, respectively (Fritz et al., 2012). Between-groups t-test.
Table 5 Validity of computerized domain scores in discriminating cognitively impaired from cognitively unimpaired participants with multiple sclerosis. Domain score
Area under curve
Standard error
Asymptotic significance
95% confidence interval
Sensitivity (Sn), Specificity (Sp) and Accuracy (Ac) of selected cutoffsa Score ≤ 85 Score ≤ 77.5 Sn Sp Ac Sn Sp Ac
Global Cognitive Score Memory Information Processing Speed Attention Working Memory Executive Function Motor Skills Problem Solving Verbal Function Visual Spatial
0.80 0.80 0.79
0.062 0.058 0.056
<0.001 <0.001 <0.001
0.68–0.92 0.68–0.91 0.68–0.90
41% 52% 52%
87% 87% 83%
54% 62% 60%
24% 33% 43%
91% 91% 87%
43% 49% 56%
0.78 0.75 0.72 0.69 0.68 0.67 0.66
0.061 0.064 0.069 0.071 0.071 0.068 0.077
<0.001 0.001 0.003 0.01 0.01 0.02 0.03
0.66–0.90 0.63–0.88 0.59–0.85 0.55–0.83 0.54–0.82 0.53–0.80 0.51–0.81
43% 36% 36% 35% 38% 35% 24%
87% 87% 83% 78% 83% 87% 83%
56% 51% 49% 47% 51% 49% 41%
29% 26% 24% 28% 21% 21% 14%
96% 96% 96% 87% 83% 91% 87%
48% 46% 44% 44% 38% 41% 40%
a Cutoffs for impairment were derived from research analyses in the present study, and no recommendation in this regard has been made by the developers of the computerized battery.
PwMS had impaired motor skills and attention, as well as verbal naming and rhyming (Fig. 1). The lower specificity associated with using all 9 CAB domain scores arises from 10 PwMS classified as: impaired by the CAB but unimpaired by traditional tests (Table 6). Interestingly, PwMS who were found to be
The frequency of cognitive impairment and count of affected cognitive domains on both batteries among PwMS is shown in Fig. 1. Information processing speed and memory were the most commonly impaired domains on both batteries. For some PwMS, poor performance on domains not covered by traditional tests was evident: 28–35% of
157
Multiple Sclerosis and Related Disorders 30 (2019) 154–162 77% 77% 78% 80% 78%
impaired by the CAB only, had significantly longer age- and educationadjusted response times on various tasks, such as time taken to decide whether to move left or right to catch an on-screen object or the time to add (or subtract) 2 numbers and decide if the sum is >4 (Fig. 2). Comparison of demographics by cognitive classification is shown in Table 7. EDSS of participants impaired on only one battery, whether traditional or computerized, was not different from EDSS of participants who were normal on both batteries. More participants unimpaired on both batteries were employed, as compared to participants impaired on both batteries. Employment status of those classified as impaired on only one battery, either computerized or traditional, was not significantly different from those classified as impaired on both batteries (Table 7). Taken together, PwMS classified as impaired on the CAB but not ‘gold standard' tests seem to be more disabled than participants classified as cognitively intact on both batteries. Details of the impaired cognitive domains for participants classified as impaired by only one battery is presented in Fig. 3. A classification of impairment on the CAB only was usually associated with impairment in multiple domains (Fig. 3a). Impaired executive function was common among those classified as impaired on the CAB alone. Impairment on traditional tests only was due to reduced information processing speed and working memory, verbal fluency or memory (Fig. 3b).
70% 70% 74% 70% 57%
72% 76% 78% 78%
This study substantiated the validity of a particular computerized cognitive screening battery for PwMS (Achiron et al., 2007). Construct validity is supported by the convergence of recommended traditional tests and computerized tests by factor analysis into 4 distinct domains, conceptually representing information processing speed, executive function, memory and visuospatial processing. Criterion validity was confirmed by the ability of CAB to detect cognitive impairment among PwMS, as defined by recommended traditional tests, with 85% sensitivity, as well as by significantly lower scores of PwMS compared to healthy controls, with generally large effect sizes. This CAB was not designed to mimic MACFIMS. It was meant to be comprehensive, with broad rather than disease-specific coverage of cognitive domains. This study offers the opportunity to explore how tests not traditionally included (e.g., tests of response inhibition, goaldirected planning, and verbal naming and rhyming) perform in detecting cognitive impairment among PwMS. Executive deficit as determined by the recommended D-KEFS sorting test was uncommon among our PwMS, despite the high overall frequency of cognitive impairment. Previous research with MACFIMS tests also identified low failure rates on this particular test, with at best moderate effect sizes compared to healthy controls (Kim et al., 2017; Migliore et al., 2017). The frequency of executive deficit on the CAB was 31%, arising from tests of response inhibition (Go-NoGo, Stroop) and from a 'Catch Game' that measures goal-directed planning. Traditional verbal fluency tests may rely on executive abilities, but as they also rely on information processing speed, they may not be specific enough for detecting executive decline. These findings may call for a reconsideration of the optimal approach for identifying executive deficit among PwMS. This CAB includes a ‘motor skills’ domain, derived from a combination of a finger tapping task and the time it takes to make the first move to catch a falling object on the screen. This domain score discriminated PwMS from healthy controls with a large effect size (Table 4) and also discriminated impaired PwMS from cognitively normal patients (Table 5). As primary motor pathways (e.g., cerebellar circuits, pyramidal tracts) may be damaged in MS, the degree to which this outcome is influenced by pure physical disability requires further exploration. It is well recognized that finger tapping may be impaired in neurological diseases without an overt motor disability until later stages, like Alzheimer's disease (Roalf et al., 2018), implying that this domain is at least partly independent from primary motor modalities.
79% 79% 79% 85% 86% 12 12 12 9 8
IPS, Information Processing Speed. CAB, computerized cognitive assessment battery. b
a
7 7 6 7 10 46 46 46 49 50
16 16 17 16 13
5 5 5 6 42 44 45 45
18 18 18 17
5. Discussion
IPSa, Memory IPS, Memory, Attention IPS, Memory, Attention, Working Memory IPS, Memory, Attention, Working Memory, Executive Function IPS, Memory, Attention, Working Memory, Motor Skills IPS, Memory, Attention, Working Memory, Problem Solving IPS, Memory, Attention, Working Memory, Verbal Function IPS, Memory, Attention, Working memory, Visual Spatial All 9 CABb Domains
Impaired on traditional battery only Unimpaired on both batteries Impaired on computerized battery only Impaired on both batteries Computerized domain combination (at least 1 domain score ≤85):
Table 6 Sensitivity (Sn), specificity (Sp) and accuracy (Ac) of various computerized domain combinations for classification of patients with multiple sclerosis as cognitively impaired.
Sn
16 14 13 13
Sp
78% 78% 78% 74%
Ac
74% 77% 78% 77%
D. Golan, et al.
158
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Fig. 1. (a) Percent of PwMS (N = 81) classified as impaired on traditional tests in each domain. Domain scores were the averaged z-scores of relevant traditional tests as described in the text. Cognitive domain z-score of −1.5 or lower was considered impaired. (b) Distribution of count of impaired domains on traditional tests per MS patient. 28% of PwMS (23 patients) had no impaired domains, thus classified as cognitively unimpaired on traditional tests. (c) Percent of PwMS (N = 81) classified as impaired on computerized tests in each domain. For all domains, a cutoff score of ≤85 (at least 1 SD below average) was superior in predicting traditional classification of impairment and was used to determine the frequencies shown. (d) Distribution of count of impaired domains on computerized tests per MS patient. 26% of PwMS (21 patients) had no impaired domains, thus classified as cognitively unimpaired on computerized tests. IPS, Information Processing Speed.
PwMS from healthy controls (Table 4) but did discriminate impaired PwMS from cognitively normal PwMS (Table 5). This test is intended to be a measure of non-verbal IQ and is conceptually and empirically correlated to Raven's standard progressive matrices (Raven, 2000; Doniger et al., 2008). This test imposes demands upon executive functions and visuospatial processing, which may decline in cognitively impaired PwMS. This decline is the likely explanation for decreased performance on the computerized problem solving test among cognitively impaired PwMS in this study; a similar decline has been reported for the Raven's standard progressive matrices (Zakzanis, 2000). The 57% specificity of the complete CAB (9 domains), arises from PwMS who were classified as impaired by the CAB but classified as unimpaired by traditional tests. This discordance may be due to ‘overdiagnosis’ by the CAB or to ‘under-diagnosis’ (e.g., under-recognition) of certain aspects of cognitive dysfunction by the recommended ‘gold standard’ tests. The high rate of unemployment among PwMS who were classified as impaired only on the CAB (Table 7) lends credence to the later explanation. Computerized tests are very sensitive to prolonged response times of patients (Fig. 2), which is not true of traditional pen and paper tests that lack the millisecond precision in response time measurement. Prolonged response times among PwMS were found by others using the same CAB (Achiron et al., 2007) and other computerized tests (Lapshin et al., 2013; Papathanasiou et al., 2014; Wilken
Furthermore, it has been shown that 73% of PwMS who failed the 9hole peg test, were still able to perform finger tapping (Tanigawa et al., 2017), implying that only a severe degree of physical disability actually interferes with finger tapping. Indeed 77/81 (95%) of patients in this study were able to complete the CAB motor skills tests, despite a rather high average EDSS (4.0 ± 1.7). Finally, functional MRI studies showed that among PwMS who perform finger tapping, activation of primary motor brain areas (e.g., cerebellum, motor cortex) occurs, but there is also activation of subcortical structures, such as the basal ganglia, thalamus and amygdala (Bonzano et al., 2017). These additional structures may be relevant to cognitive dysfunction (Batista et al., 2012). Taken together, the CAB ‘motor skills’ domain is likely valid for detection of cognitive impairment in PwMS, although in those with severe pyramidal or cerebellar deficits, this measure should be interpreted with caution. The computerized verbal function score represents ability to name an object and select a rhyming word. Although verbal fluency is conspicuously reduced in MS, confrontation naming, as assessed by such tests as the Boston Naming Test, is only modestly impaired (Zakzanis, 2000). Accordingly, this CAB domain did not discriminate PwMS from healthy controls (Table 4) but did discriminate impaired from cognitively normal PwMS (Table 5), with a modest effect size. Likewise, the computerized problem solving test did not discriminate 159
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Further, the difference in complex response times between older drivers who passed or failed an on-road driving test has been reported to have a large effect size (Cohen's d = 1.32) (Mathias and Lucas, 2009) similar in magnitude to the difference in response times between PwMS classified as impaired on the CAB only compared to those classified as unimpaired on both batteries (Fig. 2). Additional work is needed to define clinical thresholds for prolonged response times among PwMS. EDSS of participants classified as impaired on the CAB only was not different from EDSS of participants classified as normal on both batteries (Table 7). Therefore, neither classification as impaired on the CAB only nor prolonged response times among those classified as impaired on the CAB only were due to confounding by physical disability. Impaired executive function was also common among those classified as impaired on the CAB (Fig. 3a). Greater sensitivity of computerized tests to prolonged response times and impaired executive function may spuriously inflate the 'false positive' rate and decrease the specificity of computerized tests when compared with currently endorsed traditional batteries. This study has several limitations. Our control group was rather small and controls were younger than PwMS. However, the CAB domain scores are age-adjusted which mitigates this difference. Predictive validity (e.g., of vocational status or MRI parameters) and reliability are not addressed in this report and should be further explored. The 9-hole peg test (9HPT) was not part of this research protocol. Impaired manual dexterity, as measured by the 9HPT may affect finger tapping and thus reduce this CAB motor domain score. However, only one participant was classified as impaired on the CAB due to impairment in the motor domain alone (Fig. 3a), which reassures us that impaired manual dexterity is not the main reason for classification as impaired on the CAB only. Notwithstanding the merits of computerized batteries, their role is to screen for cognitive impairment, not to diagnose or manage such impairment. Ideally, whenever a computerized screening tool detects cognitive impairment, referral to a neuropsychologist or to a cognitive neurologist is important to critically confirm the result, address differential diagnosis issues, weigh the contribution of comorbidities and tailor potential intervention or rehabilitation programs (Rao, 2018). In conclusion, this study demonstrates acceptable construct and criterion validity for this particular multi-domain computerized cognitive screening battery, which should encourage its incorporation in the routine care of PwMS. This CAB also detects certain aspects of MS-related cognitive impairment that may evade traditional tests, such as prolonged response times and impaired executive function.
Fig. 2. PwMS who were classified as impaired on the computerized cognitive assessment battery only had significantly longer age- and education-adjusted response times on various tasks, including time taken to decide whether to move left or right to ‘catch’ an object on the screen (“Catch Game”) or the time to add/subtract 2 numbers and decide if the sum is greater than 4 (Staged Information Processing Speed [IPS] test). Smaller standardized scores reflect longer response times. P values are for a between-group t-test. By convention, Cohen's d ≥ 0.8 is considered a large effect size (Fritz et al., 2012).
et al., 2003). Consequently, PwMS with subtly prolonged response times may be classified as impaired on computerized batteries but as intact on traditional tests. Notably, prolonged response times have been shown to predict driving ability among the elderly (Mathias and Lucas, 2009), illustrating the functional relevance of these measures.
Acknowledgment This study was supported by a grant from Biogen Inc.
Table 7 Demographics by cognitive classificationa. Impaired on both batteries (N = 50)1 Age Education (years) EDSSc Disease durationd (years) Percent unemployed
Impaired on computerized battery only (N = 10)2
Impaired on traditional battery only (N = 8)3
Unimpaired on both batteries (N = 13)4
Group differences P valueb
46 ± 7.4 14.2 ± 2 4.5 ± 1.7(4) 11.3 ± 7.9
42.3 ± 9.1 15.3 ± 2.6 3.2 ± 1.5 7.2 ± 4.9
48.7 ± 7.8 15.9 ± 2.9 4.1 ± 1.7 8.5 ± 6.1
44.3 ± 9.9 14.6 ± 3 2.9 ± 1.4(1) 9.2 ± 6.6
0.37 0.23 0.006 0.32
35 (70%)(4)
5 (50%)
3 (38%)
3 (23%)(1)
0.01
a
1,2,3,4
Summary statistics are mean ± standard deviation or frequency (%). Superscripts indicate the group numbers for significant pairwise comparisons. Significant pairwise comparisons are indicated by bold with the relevant group number shown in parentheses. b P values are for differences across all groups (ANOVA for continuous variables, Fisher's exact tests for categorical variable). c EDSS, Expanded Disability Status Scale. d Time from first confirmed clinical episode suggestive of multiple sclerosis. 160
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Fig. 3. (a) Ten patients were classified as impaired on the computerized battery only. The impaired domains for each patient are shown as shaded boxes. (b) Eight patients were classified as impaired on the traditional battery only. The impaired domains for each patient are shown as shaded boxes. IPS, Information Processing Speed; CAB, computerized cognitive assessment battery; Working M, Working Memory.
Supplementary materials
co-normed estimate of premorbid intelligence: implementation using mindstreams computerized testing. Appl. Neuropsychol. 15 (4), 250–263. Dwolatzky, T., Whitehead, V., Doniger, G.M., et al., 2003. Validity of a novel computerized cognitive battery for mild cognitive impairment. BMC Geriatr. 3, 4. Ehrenreich, J.H., 1995. Normative data for adults on a short form of the Selective Reminding Test. Psychol. Rep. 76, 387–390. Erlanger, D.M., Kaushik, T., Caruso, L.S., et al., 2014. Reliability of a cognitive endpoint for use in a multiple sclerosis pharmaceutical trial. J. Neurol. Sci. 340 (1–2), 123–129. Fritz, C.O., Morris, P.E., Richler, J.J., 2012. Effect size estimates: current use, calculations, and interpretation. J. Exp. Psychol. Gen. 141 (1), 2–18. Golan, D., Doniger, G.M., Wissemann, K., et al., 2018. The impact of subjective cognitive fatigue and depression on cognitive function in patients with multiple sclerosis. Mult. Scler. J. 24 (2), 196–204. Heaton, R.K., Miller, S.W., Taylor, M.J., et al., 2004. Revised Comprehensive Norms for an Expanded Halstead–Reitan Battery: Demographically Adjusted Neuropsychological Norms for African American and Caucasian Adults Professional Manual. In: Lutz, F.L. (Ed.), Psychological Assessment Resources, Inc.. Kim, S., Zemon, V., Rath, J.F., et al., 2017. Screening instruments for the early detection of cognitive impairment in patients with multiple sclerosis. Int. J. MS Care 19 (1), 1–10. Langdon, D.W., Amato, M.P., Boringa, J., et al., 2012. Recommendations for a Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS). Mult. Scler. 18 (6), 891–898. Lapshin, H., Lanctôt, K.L., O'Connor, P., et al., 2013. Assessing the validity of a computergenerated cognitive screening instrument for patients with multiple sclerosis. Mult. Scler. J. 19 (14), 1905–1912. Mathias, J.L., Lucas, L.K., 2009. Cognitive predictors of unsafe driving in older drivers: a meta-analysis. Int. Psychogeriatr. 21 (4), 637–653. Migliore, S., Ghazaryan, A., Simonelli, I., et al., 2017. Cognitive impairment in relapsing-
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.msard.2019.01.051. References Achiron, A., Doniger, G.M., Harel, Y., et al., 2007. Prolonged response times characterize cognitive performance in multiple sclerosis. Eur. J. Neurol. 14 (10), 1102–1108. Batista, S., Zivadinov, R., Hoogs, M., et al., 2012. Basal ganglia, thalamus and neocortical atrophy predicting slowed cognitive processing in multiple sclerosis. J. Neurol. 259 (1), 139–146. Benedict, R., 1997. Brief Visuospatial Memory Test – Revised: Professional Manual. Psychological Assessment Resources, Inc., Lutz, FL. Benedict, R., Morrow, S., Rodgers, J., et al., 2014. Characterizing cognitive function during relapse in multiple sclerosis. Mult. Scler. 20 (13), 1745–1752. Benedict, R.H.B., Cookfair, D., Gavett, R., et al., 2006. Validity of the minimal assessment of cognitive function in multiple sclerosis (MACFIMS). J. Int. Neuropsychol. Soc. 12 (4), 549–558. Benedict, R.H.B., Fischer, J.S., Beatty, W.W., et al., 2002. Minimal neuropsychological assessment of MS patients: a consensus approach. Clin. Neuropsychol. 16 (3), 381–397. Bonzano, L., Pardini, M., Roccatagliata, L., et al., 2017. How people with multiple sclerosis cope with a sustained finger motor task: a behavioural and fMRI study. Behav. Brain Res. 325, 63–71. Delis, D.C., Kaplan, E., Kramer, J., 2001. Delis-Kaplan Executive Functioning System Examiner's Manual. NCS Pearson, Inc., San Antonio, TX. Doniger, G.M., Simon, E.S., Schweiger, A., 2008. Adjustment of cognitive scores with a
161
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
cognitive impairment in multiple sclerosis. Mult. Scler. Relat. Disord. 4 (4), 291–295. Schweiger, A., Doniger, G.M., Dwolatzky, T., et al., 2003. Reliability of a novel computerized neuropsychological battery for mild cognitive impairment. Acta Neuropsychol 1, 407–413. Sheridan, L.K., Fitzgerald, H.E., Adams, K.M., et al., 2006. Normative Symbol Digit Modalities Test performance in a community-based sample. Arch. Clin. Neuropsychol. 21 (1), 23–28. Smith, A., 1982. Symbol Digit Modalities Test: Manual. Western Psychological Services, Los Angeles. Strober, L., Englert, J., Munschauer, F., et al., 2009. Sensitivity of conventional memory tests in multiple neuropsychological battery and the minimal assessment. Mult. Scler. 15, 1077–1084. Sumowski, J.F., Benedict, R., Enzinger, C., et al., 2018. Cognition in multiple sclerosis. Neurology 0 10.1212/WNL.0000000000004977. Tanigawa, M., Stein, J., Park, J., et al., 2017. Finger and foot tapping as alternative outcomes of upper and lower extremity function in multiple sclerosis. Mult. Scler. J. Exp. Transl. Clin. 3 (1), 205521731668893. Wilken, J.A., Kane, R., Sullivan, C.L., et al., 2003. The utility of computerized neuropsychological assessment of cognitive dysfunction in patients with relapsing remitting multiple sclerosis. Mult. Scler. J. 9 (2), 119–128. Zakzanis, K.K., 2000. Distinct neurocognitive profiles in multiple sclerosis subtypes. Arch. Clin. Neuropsychol. 15 (2), 115–136.
remitting multiple sclerosis patients with very mild clinical disability. Behav. Neurol. 1–10. Mitrushina, M., Boone, B.K., Razani, J., et al., 2005. Locator and data tables for the Judgment of Line Orientation test. Handbook of Normative Data for Neuropsychological Assessment, second ed. Oxford university press, New York, pp. 798–804. Papathanasiou, A., Messinis, L., Georgiou, V.L., et al., 2014. Cognitive impairment in relapsing remitting and secondary progressive multiple sclerosis patients: efficacy of a computerized cognitive screening battery. ISRN Neurol. 2014, 1–7. Rao, S.M., 2018. Role of computerized screening in healthcare teams: why computerized testing is not the death of neuropsychology. Arch. Clin. Neuropsychol. 33 (3), 375–378. Rao, S.M., Leo, G.J., Bernardin, L., et al., 1991b. Cognitive dysfunction in multiple sclerosis. I. Frequency, patterns, and prediction. Neurology 41 (5), 685–691. Rao, S.M., Leo, G.J., Ellington, L., et al., 1991a. Cognitive dysfunction in multiple sclerosis. II. Impact on employment and social functioning. Neurology 41 (5), 692–696. Raven, J., 2000. The Raven's progressive matrices: change and stability over culture and time. Cogn. Psychol. 41 (1), 1–48. Roalf, D.R., Rupert, P., Mechanic-Hamilton, D., et al., 2018. Quantitative assessment of finger tapping characteristics in mild cognitive impairment, Alzheimer's disease, and Parkinson's disease. J. Neurol. 04 (0123456789), 1–4. Romero, K., Shammi, P., Feinstein, A., 2015. Neurologists accuracy in predicting
162
Contents lists available at ScienceDirect
Multiple Sclerosis and Related Disorders journal homepage: www.elsevier.com/locate/msard
Clinical trial
Validity of a multi-domain computerized cognitive assessment battery for patients with multiple sclerosis
T
Daniel Golana,b, Jeffrey Wilkenc,d, Glen M. Donigere,f, Timothy Frattoc, Robert Kanec, Jared Srinivasang, Myassar Zarifg, Barbara Bumsteadg, Marijean Buhseg, Lori Fafardg, ⁎ Ilir Topallig, Mark Gudesblattg, a
Department of Neurology & Multiple Sclerosis Center, Lady Davis Carmel Medical Center, Haifa, Israel Rappaport Faculty of Medicine, Technion-Israel Institute of Technology, Haifa, Israel c Washington Neuropsychology Research Group, Fairfax, VA, USA d Department of Neurology, Georgetown University, Washington DC, USA e Department of Clinical Research, NeuroTrax Corporation, Modiin, Israel f Joseph Sagol Neuroscience Center & Center of Advanced Technologies in Rehabilitation, Sheba Medical Center, Ramat Gan, Israel g South Shore Neurologic Associates, 77 Medford Avenue, Patchogue, NY 11772, USA b
A R T I C LE I N FO
A B S T R A C T
Keywords: Cognitive function Computerized cognitive assessment Multiple sclerosis Response time Validity
Background: Computerized cognitive batteries may facilitate the integration of neuropsychological assessments into routine clinical care of patients with multiple sclerosis (PwMS). Objective: To assess the construct and criterion validity of a computerized, multi-domain cognitive assessment battery (CAB, NeuroTrax) in MS. Methods: 81 PwMS and 15 healthy controls (HC) completed the CAB and a set of traditional neuropsychological tests recommended for MS on the same day. Principal component factor analysis was used to assess construct validity. For criterion validity, the gold standard definition of cognitive impairment was a score of ≥1.5SD below average on at least one cognitive domain, based upon traditional test normative data. Receiver operating characteristic (ROC) analysis was used to examine the ability of the CAB to discriminate cognitively impaired PwMS. Results: Traditional and computerized tests of memory, processing speed, visuospatial and executive function converged by factor analysis. Computerized tests detected cognitive impairment with 85% sensitivity and 70% specificity. PwMS classified as impaired on only the computerized battery had significantly prolonged response times and a higher rate of unemployment compared with PwMS classified as unimpaired on both batteries. Poor executive function was more likely to be revealed by the CAB. Conclusion: The specific computerized assessment battery evaluated is valid for cognitive screening of people with MS and may be more likely to detect prolonged response times and impaired executive function.
1. Introduction Cognitive dysfunction is a prevalent debilitating symptom of Multiple Sclerosis (MS). More than 50% of people with MS (PwMS) experience some form of cognitive impairment (Sumowski et al., 2018). MS-related cognitive dysfunction may prevent maintaining gainful employment (Rao et al., 1991a), adversely impact quality of life (Rao et al., 1991b) and may be a manifestation of disease activity (Benedict et al., 2014). Routine neurological assessment lacks sensitivity in identifying cognitive impairment. Neurologists' prediction of cognitive impairment based on a typical clinical visit is not significantly ⁎
different from chance (Romero et al., 2015). It is therefore important to include some form of objective cognitive assessment in the routine care and management of PwMS. In 2001 an expert panel of MS neuropsychologists recommended a 90-min battery, Minimal Assessment of Cognitive Function in MS (MACFIMS), covering 5 cognitive domains (processing speed/working memory, learning and memory, executive function, visual-spatial processing, and word retrieval) (Benedict et al., 2002). Unfortunately, this battery has several limitations which have resulted in its infrequent use in PwMS routine care and clinical trials. Among these limitations are: dependence upon neuropsychologists to supervise testing, labor-
Corresponding author. E-mail addresses: [email protected] (D. Golan), [email protected] (M. Gudesblatt).
https://doi.org/10.1016/j.msard.2019.01.051 Received 10 September 2018; Received in revised form 4 January 2019; Accepted 29 January 2019 2211-0348/ © 2019 Elsevier B.V. All rights reserved.
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
by the traditional neurocognitive tests. A 10-min break was given between the two testing batteries. Participants were instructed not to take psychotropic medications (benzodiazepines, opioids) in the 6 h prior to testing. Cognitive assessments were supervised by a neuropsychologist.
intensive manual computation of standardized scores, lack of alternate forms for some components, and lengthy administration times. One proposed solution is to decrease the number of domains tested (Langdon et al., 2012). This approach may lead to loss of critical information for identifying the spectrum and degree of cognitive impairment in the population of PwMS and for any individual PwMS. Computerized screening tools offer the opportunity to incorporate such objective cognitive screening measures into routine PwMS clinical care while retaining broad multi-domain assessment. Periodic computerized cognitive screening is already integrated into our routine for PwMS care, and data from ∼700 PwMS has provided further insight into the impact of fatigue and depression on MS-related cognitive impairment (Golan et al., 2018). Additional merits of computerized tools include: inherent administration standardization, instant result normalization, multiple alternate forms, and testing session supervision not requiring physician/neuropsychologist involvement. The computerized cognitive assessment battery (CAB) utilized in this study (NeuroTrax) has been validated and used to screen for cognitive impairment in other neurological diseases (Dwolatzky et al., 2003; Schweiger et al., 2003) (see the supplementary materials). Significant correlations have been shown between CAB domain scores and test results from Rao's neuropsychological screening battery among 58 PwMS (Achiron et al., 2007). In that study, most CAB index scores significantly discriminated PwMS from healthy individuals (Achiron et al., 2007). In this study we further evaluated the construct and criterion validity of this CAB among PwMS relative to a set of recommended, traditional neuropsychological tests, mainly from the Minimal Assessment of Cognitive Function in MS (MACFIMS) battery.
2.3. Traditional cognitive testing Traditional neuropsychological tests were mainly from the Minimal Assessment of Cognitive Function in MS (MACFIMS) battery, which was developed for use in PwMS by expert consensus (Benedict et al., 2002) with subsequent empirical validation (Benedict et al., 2006). The only difference was inclusion of the Selective Reminding Test (SRT) instead of the California Verbal Learning Test-II (CVLT-II). These two tests have shown comparable sensitivity in discriminating PwMS from demographically-matched controls (Strober et al., 2009). The SRT is also recommended for PwMS as part of the MS-COG battery (Sumowski et al., 2018; Erlanger et al., 2014). We opted for the SRT due to substantially reduced administration time than for the CVLT-II, and availability of multiple forms of SRT for longitudinal follow-up. The traditional cognitive assessment consisted of 7 tests which can be grouped into 5 domains (Benedict et al., 2002): information processing speed and working memory—Paced Auditory Serial Addition Test (PASAT) Trial 1: three second and Trial 2: two second (Rao et al., 1991b); SDMT (Sheridan et al., 2006). Memory—SRT for verbal memory (Ehrenreich, 1995); Brief Visuospatial Memory Test-Revised (BVMT-R) (Benedict, 1997) for visual memory. Verbal fluency—Controlled Oral Word Association Test (COWAT) (Heaton et al., 2004). Visuospatial processing—Judgment of Line Orientation (JLO) (Mitrushina et al., 2005). Executive function—Delis-Kaplan Executive Function System (D-KEFS) Sorting Test (Delis et al., 2001). For all tests, z-scores were calculated based on published normative data. To improve the robustness of cognitive classification, the z-scores of all tests within a particular cognitive domain were averaged to obtain summary cognitive domain z-scores.
2. Methods 2.1. Participants 81 PwMS, according to the revised McDonald criteria, under treatment at one MS Care Center (South Shore Neurologic Associates, New York, USA) were recruited for this study. Inclusion criteria were: age 25–55, vision corrected to at least 20/50, 12+ years of education, able to write and press buttons on a computer mouse. Exclusion criteria were: unable to understand and follow all test instructions, history of head injury, seizures, neurological conditions involving the central nervous system (other than MS), colorblindness, history of psychosis or other severe mental illness (other than mild-to-moderate depression), current alcohol/substance abuse and MS relapse < 30 days prior to enrollment. 15 healthy controls (HC) underwent equivalent study procedures. Inclusion and exclusion criteria for HC were similar to those of PwMS. All HC had to be free of neurological conditions known to impact cognitive function. For each participant, all evaluations (see below) were carried out on the same day. Institutional review board (IRB) approval was obtained, and all participants gave informed consent.
2.4. Gold standard definition of cognitive impairment PwMS with a cognitive domain z-score of −1.5 or lower on at least one cognitive domain were defined as cognitively impaired by ‘gold standard’ traditional tests (Sumowski et al., 2018). 2.5. Computerized assessment battery This CAB consists of 10 tests: Verbal Memory, Non-Verbal Memory, Go-NoGo, Stroop, Verbal Function (naming, rhyming), Problem Solving, Visuospatial Processing, Staged Information Processing Speed (IPS), Finger Tapping, and a psychomotor “Catch Game”. For tests measuring response time, participants respond by clicking one mouse button on a two-button mouse (to avoid confounding effects of weakness or poor coordination, no movement of the mouse is required). Further detailed description of these tests is available in the supplementary materials. To minimize differences in age and education and to permit averaging performance across different types of outcome parameters (e.g., accuracy, response times), each CAB outcome parameter is normalized and fit to an IQ-style scale (mean: 100, SD: 15) in an age- and education-specific fashion. Normative data is from individuals classified as cognitively healthy in controlled studies conducted at academic research centers. Normalized subsets of outcome parameters were averaged to produce nine summary scores, each indexing a different cognitive domain (Table 1). A Global Cognitive Score was computed as the average of the 9 domain index scores.
2.2. Study procedures To examine ability of the CAB to detect various degrees of cognitive impairment, PwMS participants were first screened for cognitive dysfunction using the Symbol Digit Modalities Test (SDMT). Based on SDMT results and normative data (Smith, 1982; Sheridan et al., 2006), PwMS participants were stratified into 4 subgroups: the first subgroup had performance better than 1.00 standard deviation below the age expected mean. The other 3 subgroups had SDMT scores 1.00 to 1.50, 1.51 to 2.00 and >2.00 standard deviations below the age expected mean. An equal number of participants were recruited to each impairment-based subgroup. All participants completed the computerized battery first, followed
3. Statistical analysis Data analysis was performed using SPSS, version 23.0 (IBM Corp., 155
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Table 1 Cognitive domain scores components of the computerized assessment battery. Domain
Averaged componentsa
Memory
Verbal Memory: Total Accuracy Delayed Verbal Memory: Accuracy Non-Verbal Memory: Total Accuracy Delayed Non-Verbal Memory: Accuracy Go-NoGo: Composite Score Verbal Memory: Accuracy, Repetition 1 Non-Verbal Memory: Accuracy, Repetition 1 Go-NoGo: Composite Score Stroop: Composite Score, Level 3 Catch Game: Total Score Go-NoGo: Response Time Go-NoGo: Response Time Standard Deviation Stroop Interference: Response Time, Level 2 Staged Information Processing Speed: Response Time, Level 1.2 Staged Information Processing Speed: Accuracy, Level 2.3 Staged Information Processing Speed: Composite Scores, Levels 1.1, 1.3, 2.1 and 2.2 Visual Spatial Processing: Accuracy Verbal Function: Rhyming, Accuracy Finger Tapping: Inter-Tap Interval Finger Tapping: Tap Interval Standard Deviation Catch Game: Time to Make First Move Problem Solving: Accuracy
Working Memory
Executive Function
Attention
Information Processing Speed Visual Spatial Verbal Function Motor Skills
Problem Solving
Table 2 Patient characteristicsa. N Age Female Race Education (years) EDSSb Disease durationc (years) Disease course
Disease modifying medication
a b c
81 45 ± 8.1 59 (73%) White: 77 (95%) African American: 4 (5%) 14.5 ± 2.4 4 ± 1.7 10.2 ± 7.3 Relapsing remitting: 74 (91%) Secondary progressive: 5 (6%) Primary progressive: 2 (3%) Natalizumab 48 (59%) Dimethyl Fumarate 11 (14%) Fingolimod 6 (7%) Ocrelizumab 4 (5%) Glatiramer acetate 3 (4%) Alemtuzumab 2 (3%) No medication 7 (8%)
Summary statistics are mean ± standard deviation. EDSS, Expanded Disability Status Scale. Time from first confirmed clinical episode suggestive of multiple sclerosis.
COWAT, which taps search and retrieval of verbal information). For the second factor (F2), tests loading on executive functions converged (Computerized: Problem Solving; traditional: D-KEFS sorting and COWAT). In the third factor (F3), computerized and traditional tests of visuospatial processing clustered. The fourth factor (F4) represents memory, wherein traditional and computerized tests of verbal and visual memory converged. The last factor (F5) contained only computerized tests of verbal function (naming and rhyming); this domain was not covered by recommended traditional tests.
a Details regarding the individual tests are provided in the supplementary materials.
Armonk, N.Y., USA). Principal components factor analysis with varimax rotation was conducted to assess how raw scores from traditional and computerized tests in PwMS clustered. Continuous variables were analyzed using between-groups t-test, with corresponding effect sizes (Cohen's d). Receiver operating characteristic (ROC) analysis was used to evaluate the ability of the CAB cognitive domain (index) scores to discriminate PwMS with cognitive impairment from unimpaired PwMS, as defined by traditional tests. Sensitivity, specificity and accuracy of the CAB domain scores for predicting the ‘gold standard’ classification of cognitive impairment based on traditional measures were calculated, using two common cutoffs: ≤1 (domain score ≤85) and ≤1.5 (domain score ≤77.5) standard deviations below the age and education expected mean. Finally, sensitivity, specificity and accuracy of various combinations of the computerized domain scores for predicting the ‘gold standard’ classification of cognitive impairment were tested. Domain scores were analyzed individually, using the cutoff that yielded the best classification accuracy. Domains were retained if they improved the ability of the CAB to discriminate among groups. Hence, the optimal combination of domain scores was ascertained. Demographics by subgroup of cognitive impairment classification were compared using analysis of variance (ANOVA) for continuous variables and Fisher's exact test for categorical variables, followed by Bonferroni-adjusted pairwise comparisons. Descriptive statistics are reported as mean ± standard deviation.
4.2. Criterion validity Computerized domain scores discriminated PwMS from healthy controls (Table 4). Age- and education-adjusted computerized domain scores were significantly lower among PwMS as compared to healthy controls, with large effect sizes for the computerized memory, information processing speed, working memory, attention, executive functions, visuospatial processing and motor domains. Verbal and problem solving domain scores were not significantly different between the groups. Notably, PwMS and healthy controls were comparable in years of education and gender (Table 2); however, controls were significantly younger than patients [33.3 ± 8.1 vs. 45.0 ± 8.1 years, P < 0.001]. Notably, age-adjusted scores were analyzed mitigating this difference. 58 PwMS were cognitively impaired and 23 were cognitively normal, by 'gold standard' traditional tests. All the computerized domain scores discriminated cognitively impaired from cognitively unimpaired PwMS (Table 5). For all computerized domain scores, a cutoff for impairment of ≥1 standard deviation below the age- and educationadjusted mean (domain score ≤85) was superior, defined as improved sensitivity and accuracy without substantially reduced specificity, as compared to the alternative ≥1.5 standard deviations below the normative mean (Table 5). Consequently, we used the domain score ≤85 cutoff for analysis of the discriminability of various domain score combinations (Table 6) and for describing frequency of impairment (Figs. 1 and 3a). The optimal computerized domain combination for predicting cognitive impairment among PwMS, as defined by traditional tests, included memory, information processing speed, attention, working memory and visuospatial indices. Computerized domain score ≤85 in at least one of these 5 domains predicted 'gold standard' cognitive impairment with 85% sensitivity and 70% specificity. Using the complete set of 9 computerized domains improved the sensitivity to 86% but decreased specificity to 57% (Table 6).
4. Results 4.1. Construct validity Patient characteristics are shown in Table 2. Traditional and computerized tests converged by factor analysis. Five components were rotated, together explaining 66.4% of the variance (Table 3). The first factor (F1) included tests related to attention, processing speed and response times (Computerized: IPS, Stroop, Go-NoGo, “Catch Game”; Traditional: SDMT). Tests from the first factor also rely on executive abilities (Computerized: Stroop and Go-NoGo, demanding response inhibition and the “Catch Game”, which requires planning; Traditional: 156
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Table 3 Construct validity of traditional and computerized batteries by principal components factor analysis. Testa
F1
C: Staged Information Processing Speed: Composite Score, Level 2.2 C: Stroop: Composite Score, Level 3 C: Go-NoGo: Composite Score C: Catch Game: Total Score C: Visual Spatial Processing: Accuracy C: Problem Solving: Accuracy C: Delayed Non-Verbal Memory: Accuracy C: Delayed Verbal Memory: Accuracy C: Verbal Function: Matching, Accuracy C: Verbal Function: Rhyming, Accuracy T: Symbol Digit Modalities Test (SDMT), Oral T: Controlled Oral Word Association Test (COWAT), Animals T: Controlled Oral Word Association Test (COWAT), F-A-S T: Delis-Kaplan Executive Function System (D-KEFS), Correct Sorts T: Paced Auditory Serial Addition Test (PASAT), 3 s T: Brief Visuospatial Memory Test-Revised (BVMT-R), Delayed Recall T: Judgment of Line Orientation (JLO), Number Correct T: Selective Reminding Test (SRT), Delayed Recall % variance explainedb
0.76 0.73 0.77 0.64 0.4
F2
F3
F4
0.4 0.4
0.77 0.56 0.44
F5
0.4 0.66 0.84 0.78
0.58 0.45
0.54 0.67 0.76 0.4 0.71 0.5
0.41 0.77
17
15.6
0.83 10.8
12.4
10.7
C, computerized; T, traditional. a Loadings <0.4 are omitted. b Percent of variance explained by all 18 variables for each factor. Table 4 Group data comparing age- and education-adjusted computerized cognitive scores of patients with multiple sclerosis and normal healthy controls.
Global Cognitive Score Memory Information Processing Speed Working Memory Attention Executive Function Visual Spatial Motor Skills Verbal Function Problem Solving a b c
MS Na
Score
81 81 76 81 81 81 79 77 80 79
89.1 87.7 85.8 89.4 86.6 90.0 96.5 89.9 87.6 91.8
± ± ± ± ± ± ± ± ± ±
13.9 18.4 17.5 13.4 18.3 14.9 16.8 18.5 26.6 17.7
Controls (N = 15) Score
Cohen's db
P valuec
101.2 ± 6.9 101.5 ± 9.2 98.0 ± 11.8 102.0 ± 123 100.5 ± 7.6 103.5 ± 10.2 109.0 ± 11.8 104.7 ± 4.3 95.5 ± 21.7 95.9 ± 15.4
1.1 0.93 0.82 0.97 0.96 1.1 0.86 1.1 0.33 0.25
<0.001 <0.001 0.002 0.001 <0.001 0.001 0.007 <0.001 0.28 0.4
Outcome parameter scores may be missing if patient responses are insufficient on relevant tests. By convention, Cohen’s d values of 0.8, 0.5, and 0.2 represented large, medium and small effect sizes, respectively (Fritz et al., 2012). Between-groups t-test.
Table 5 Validity of computerized domain scores in discriminating cognitively impaired from cognitively unimpaired participants with multiple sclerosis. Domain score
Area under curve
Standard error
Asymptotic significance
95% confidence interval
Sensitivity (Sn), Specificity (Sp) and Accuracy (Ac) of selected cutoffsa Score ≤ 85 Score ≤ 77.5 Sn Sp Ac Sn Sp Ac
Global Cognitive Score Memory Information Processing Speed Attention Working Memory Executive Function Motor Skills Problem Solving Verbal Function Visual Spatial
0.80 0.80 0.79
0.062 0.058 0.056
<0.001 <0.001 <0.001
0.68–0.92 0.68–0.91 0.68–0.90
41% 52% 52%
87% 87% 83%
54% 62% 60%
24% 33% 43%
91% 91% 87%
43% 49% 56%
0.78 0.75 0.72 0.69 0.68 0.67 0.66
0.061 0.064 0.069 0.071 0.071 0.068 0.077
<0.001 0.001 0.003 0.01 0.01 0.02 0.03
0.66–0.90 0.63–0.88 0.59–0.85 0.55–0.83 0.54–0.82 0.53–0.80 0.51–0.81
43% 36% 36% 35% 38% 35% 24%
87% 87% 83% 78% 83% 87% 83%
56% 51% 49% 47% 51% 49% 41%
29% 26% 24% 28% 21% 21% 14%
96% 96% 96% 87% 83% 91% 87%
48% 46% 44% 44% 38% 41% 40%
a Cutoffs for impairment were derived from research analyses in the present study, and no recommendation in this regard has been made by the developers of the computerized battery.
PwMS had impaired motor skills and attention, as well as verbal naming and rhyming (Fig. 1). The lower specificity associated with using all 9 CAB domain scores arises from 10 PwMS classified as: impaired by the CAB but unimpaired by traditional tests (Table 6). Interestingly, PwMS who were found to be
The frequency of cognitive impairment and count of affected cognitive domains on both batteries among PwMS is shown in Fig. 1. Information processing speed and memory were the most commonly impaired domains on both batteries. For some PwMS, poor performance on domains not covered by traditional tests was evident: 28–35% of
157
Multiple Sclerosis and Related Disorders 30 (2019) 154–162 77% 77% 78% 80% 78%
impaired by the CAB only, had significantly longer age- and educationadjusted response times on various tasks, such as time taken to decide whether to move left or right to catch an on-screen object or the time to add (or subtract) 2 numbers and decide if the sum is >4 (Fig. 2). Comparison of demographics by cognitive classification is shown in Table 7. EDSS of participants impaired on only one battery, whether traditional or computerized, was not different from EDSS of participants who were normal on both batteries. More participants unimpaired on both batteries were employed, as compared to participants impaired on both batteries. Employment status of those classified as impaired on only one battery, either computerized or traditional, was not significantly different from those classified as impaired on both batteries (Table 7). Taken together, PwMS classified as impaired on the CAB but not ‘gold standard' tests seem to be more disabled than participants classified as cognitively intact on both batteries. Details of the impaired cognitive domains for participants classified as impaired by only one battery is presented in Fig. 3. A classification of impairment on the CAB only was usually associated with impairment in multiple domains (Fig. 3a). Impaired executive function was common among those classified as impaired on the CAB alone. Impairment on traditional tests only was due to reduced information processing speed and working memory, verbal fluency or memory (Fig. 3b).
70% 70% 74% 70% 57%
72% 76% 78% 78%
This study substantiated the validity of a particular computerized cognitive screening battery for PwMS (Achiron et al., 2007). Construct validity is supported by the convergence of recommended traditional tests and computerized tests by factor analysis into 4 distinct domains, conceptually representing information processing speed, executive function, memory and visuospatial processing. Criterion validity was confirmed by the ability of CAB to detect cognitive impairment among PwMS, as defined by recommended traditional tests, with 85% sensitivity, as well as by significantly lower scores of PwMS compared to healthy controls, with generally large effect sizes. This CAB was not designed to mimic MACFIMS. It was meant to be comprehensive, with broad rather than disease-specific coverage of cognitive domains. This study offers the opportunity to explore how tests not traditionally included (e.g., tests of response inhibition, goaldirected planning, and verbal naming and rhyming) perform in detecting cognitive impairment among PwMS. Executive deficit as determined by the recommended D-KEFS sorting test was uncommon among our PwMS, despite the high overall frequency of cognitive impairment. Previous research with MACFIMS tests also identified low failure rates on this particular test, with at best moderate effect sizes compared to healthy controls (Kim et al., 2017; Migliore et al., 2017). The frequency of executive deficit on the CAB was 31%, arising from tests of response inhibition (Go-NoGo, Stroop) and from a 'Catch Game' that measures goal-directed planning. Traditional verbal fluency tests may rely on executive abilities, but as they also rely on information processing speed, they may not be specific enough for detecting executive decline. These findings may call for a reconsideration of the optimal approach for identifying executive deficit among PwMS. This CAB includes a ‘motor skills’ domain, derived from a combination of a finger tapping task and the time it takes to make the first move to catch a falling object on the screen. This domain score discriminated PwMS from healthy controls with a large effect size (Table 4) and also discriminated impaired PwMS from cognitively normal patients (Table 5). As primary motor pathways (e.g., cerebellar circuits, pyramidal tracts) may be damaged in MS, the degree to which this outcome is influenced by pure physical disability requires further exploration. It is well recognized that finger tapping may be impaired in neurological diseases without an overt motor disability until later stages, like Alzheimer's disease (Roalf et al., 2018), implying that this domain is at least partly independent from primary motor modalities.
79% 79% 79% 85% 86% 12 12 12 9 8
IPS, Information Processing Speed. CAB, computerized cognitive assessment battery. b
a
7 7 6 7 10 46 46 46 49 50
16 16 17 16 13
5 5 5 6 42 44 45 45
18 18 18 17
5. Discussion
IPSa, Memory IPS, Memory, Attention IPS, Memory, Attention, Working Memory IPS, Memory, Attention, Working Memory, Executive Function IPS, Memory, Attention, Working Memory, Motor Skills IPS, Memory, Attention, Working Memory, Problem Solving IPS, Memory, Attention, Working Memory, Verbal Function IPS, Memory, Attention, Working memory, Visual Spatial All 9 CABb Domains
Impaired on traditional battery only Unimpaired on both batteries Impaired on computerized battery only Impaired on both batteries Computerized domain combination (at least 1 domain score ≤85):
Table 6 Sensitivity (Sn), specificity (Sp) and accuracy (Ac) of various computerized domain combinations for classification of patients with multiple sclerosis as cognitively impaired.
Sn
16 14 13 13
Sp
78% 78% 78% 74%
Ac
74% 77% 78% 77%
D. Golan, et al.
158
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Fig. 1. (a) Percent of PwMS (N = 81) classified as impaired on traditional tests in each domain. Domain scores were the averaged z-scores of relevant traditional tests as described in the text. Cognitive domain z-score of −1.5 or lower was considered impaired. (b) Distribution of count of impaired domains on traditional tests per MS patient. 28% of PwMS (23 patients) had no impaired domains, thus classified as cognitively unimpaired on traditional tests. (c) Percent of PwMS (N = 81) classified as impaired on computerized tests in each domain. For all domains, a cutoff score of ≤85 (at least 1 SD below average) was superior in predicting traditional classification of impairment and was used to determine the frequencies shown. (d) Distribution of count of impaired domains on computerized tests per MS patient. 26% of PwMS (21 patients) had no impaired domains, thus classified as cognitively unimpaired on computerized tests. IPS, Information Processing Speed.
PwMS from healthy controls (Table 4) but did discriminate impaired PwMS from cognitively normal PwMS (Table 5). This test is intended to be a measure of non-verbal IQ and is conceptually and empirically correlated to Raven's standard progressive matrices (Raven, 2000; Doniger et al., 2008). This test imposes demands upon executive functions and visuospatial processing, which may decline in cognitively impaired PwMS. This decline is the likely explanation for decreased performance on the computerized problem solving test among cognitively impaired PwMS in this study; a similar decline has been reported for the Raven's standard progressive matrices (Zakzanis, 2000). The 57% specificity of the complete CAB (9 domains), arises from PwMS who were classified as impaired by the CAB but classified as unimpaired by traditional tests. This discordance may be due to ‘overdiagnosis’ by the CAB or to ‘under-diagnosis’ (e.g., under-recognition) of certain aspects of cognitive dysfunction by the recommended ‘gold standard’ tests. The high rate of unemployment among PwMS who were classified as impaired only on the CAB (Table 7) lends credence to the later explanation. Computerized tests are very sensitive to prolonged response times of patients (Fig. 2), which is not true of traditional pen and paper tests that lack the millisecond precision in response time measurement. Prolonged response times among PwMS were found by others using the same CAB (Achiron et al., 2007) and other computerized tests (Lapshin et al., 2013; Papathanasiou et al., 2014; Wilken
Furthermore, it has been shown that 73% of PwMS who failed the 9hole peg test, were still able to perform finger tapping (Tanigawa et al., 2017), implying that only a severe degree of physical disability actually interferes with finger tapping. Indeed 77/81 (95%) of patients in this study were able to complete the CAB motor skills tests, despite a rather high average EDSS (4.0 ± 1.7). Finally, functional MRI studies showed that among PwMS who perform finger tapping, activation of primary motor brain areas (e.g., cerebellum, motor cortex) occurs, but there is also activation of subcortical structures, such as the basal ganglia, thalamus and amygdala (Bonzano et al., 2017). These additional structures may be relevant to cognitive dysfunction (Batista et al., 2012). Taken together, the CAB ‘motor skills’ domain is likely valid for detection of cognitive impairment in PwMS, although in those with severe pyramidal or cerebellar deficits, this measure should be interpreted with caution. The computerized verbal function score represents ability to name an object and select a rhyming word. Although verbal fluency is conspicuously reduced in MS, confrontation naming, as assessed by such tests as the Boston Naming Test, is only modestly impaired (Zakzanis, 2000). Accordingly, this CAB domain did not discriminate PwMS from healthy controls (Table 4) but did discriminate impaired from cognitively normal PwMS (Table 5), with a modest effect size. Likewise, the computerized problem solving test did not discriminate 159
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Further, the difference in complex response times between older drivers who passed or failed an on-road driving test has been reported to have a large effect size (Cohen's d = 1.32) (Mathias and Lucas, 2009) similar in magnitude to the difference in response times between PwMS classified as impaired on the CAB only compared to those classified as unimpaired on both batteries (Fig. 2). Additional work is needed to define clinical thresholds for prolonged response times among PwMS. EDSS of participants classified as impaired on the CAB only was not different from EDSS of participants classified as normal on both batteries (Table 7). Therefore, neither classification as impaired on the CAB only nor prolonged response times among those classified as impaired on the CAB only were due to confounding by physical disability. Impaired executive function was also common among those classified as impaired on the CAB (Fig. 3a). Greater sensitivity of computerized tests to prolonged response times and impaired executive function may spuriously inflate the 'false positive' rate and decrease the specificity of computerized tests when compared with currently endorsed traditional batteries. This study has several limitations. Our control group was rather small and controls were younger than PwMS. However, the CAB domain scores are age-adjusted which mitigates this difference. Predictive validity (e.g., of vocational status or MRI parameters) and reliability are not addressed in this report and should be further explored. The 9-hole peg test (9HPT) was not part of this research protocol. Impaired manual dexterity, as measured by the 9HPT may affect finger tapping and thus reduce this CAB motor domain score. However, only one participant was classified as impaired on the CAB due to impairment in the motor domain alone (Fig. 3a), which reassures us that impaired manual dexterity is not the main reason for classification as impaired on the CAB only. Notwithstanding the merits of computerized batteries, their role is to screen for cognitive impairment, not to diagnose or manage such impairment. Ideally, whenever a computerized screening tool detects cognitive impairment, referral to a neuropsychologist or to a cognitive neurologist is important to critically confirm the result, address differential diagnosis issues, weigh the contribution of comorbidities and tailor potential intervention or rehabilitation programs (Rao, 2018). In conclusion, this study demonstrates acceptable construct and criterion validity for this particular multi-domain computerized cognitive screening battery, which should encourage its incorporation in the routine care of PwMS. This CAB also detects certain aspects of MS-related cognitive impairment that may evade traditional tests, such as prolonged response times and impaired executive function.
Fig. 2. PwMS who were classified as impaired on the computerized cognitive assessment battery only had significantly longer age- and education-adjusted response times on various tasks, including time taken to decide whether to move left or right to ‘catch’ an object on the screen (“Catch Game”) or the time to add/subtract 2 numbers and decide if the sum is greater than 4 (Staged Information Processing Speed [IPS] test). Smaller standardized scores reflect longer response times. P values are for a between-group t-test. By convention, Cohen's d ≥ 0.8 is considered a large effect size (Fritz et al., 2012).
et al., 2003). Consequently, PwMS with subtly prolonged response times may be classified as impaired on computerized batteries but as intact on traditional tests. Notably, prolonged response times have been shown to predict driving ability among the elderly (Mathias and Lucas, 2009), illustrating the functional relevance of these measures.
Acknowledgment This study was supported by a grant from Biogen Inc.
Table 7 Demographics by cognitive classificationa. Impaired on both batteries (N = 50)1 Age Education (years) EDSSc Disease durationd (years) Percent unemployed
Impaired on computerized battery only (N = 10)2
Impaired on traditional battery only (N = 8)3
Unimpaired on both batteries (N = 13)4
Group differences P valueb
46 ± 7.4 14.2 ± 2 4.5 ± 1.7(4) 11.3 ± 7.9
42.3 ± 9.1 15.3 ± 2.6 3.2 ± 1.5 7.2 ± 4.9
48.7 ± 7.8 15.9 ± 2.9 4.1 ± 1.7 8.5 ± 6.1
44.3 ± 9.9 14.6 ± 3 2.9 ± 1.4(1) 9.2 ± 6.6
0.37 0.23 0.006 0.32
35 (70%)(4)
5 (50%)
3 (38%)
3 (23%)(1)
0.01
a
1,2,3,4
Summary statistics are mean ± standard deviation or frequency (%). Superscripts indicate the group numbers for significant pairwise comparisons. Significant pairwise comparisons are indicated by bold with the relevant group number shown in parentheses. b P values are for differences across all groups (ANOVA for continuous variables, Fisher's exact tests for categorical variable). c EDSS, Expanded Disability Status Scale. d Time from first confirmed clinical episode suggestive of multiple sclerosis. 160
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
Fig. 3. (a) Ten patients were classified as impaired on the computerized battery only. The impaired domains for each patient are shown as shaded boxes. (b) Eight patients were classified as impaired on the traditional battery only. The impaired domains for each patient are shown as shaded boxes. IPS, Information Processing Speed; CAB, computerized cognitive assessment battery; Working M, Working Memory.
Supplementary materials
co-normed estimate of premorbid intelligence: implementation using mindstreams computerized testing. Appl. Neuropsychol. 15 (4), 250–263. Dwolatzky, T., Whitehead, V., Doniger, G.M., et al., 2003. Validity of a novel computerized cognitive battery for mild cognitive impairment. BMC Geriatr. 3, 4. Ehrenreich, J.H., 1995. Normative data for adults on a short form of the Selective Reminding Test. Psychol. Rep. 76, 387–390. Erlanger, D.M., Kaushik, T., Caruso, L.S., et al., 2014. Reliability of a cognitive endpoint for use in a multiple sclerosis pharmaceutical trial. J. Neurol. Sci. 340 (1–2), 123–129. Fritz, C.O., Morris, P.E., Richler, J.J., 2012. Effect size estimates: current use, calculations, and interpretation. J. Exp. Psychol. Gen. 141 (1), 2–18. Golan, D., Doniger, G.M., Wissemann, K., et al., 2018. The impact of subjective cognitive fatigue and depression on cognitive function in patients with multiple sclerosis. Mult. Scler. J. 24 (2), 196–204. Heaton, R.K., Miller, S.W., Taylor, M.J., et al., 2004. Revised Comprehensive Norms for an Expanded Halstead–Reitan Battery: Demographically Adjusted Neuropsychological Norms for African American and Caucasian Adults Professional Manual. In: Lutz, F.L. (Ed.), Psychological Assessment Resources, Inc.. Kim, S., Zemon, V., Rath, J.F., et al., 2017. Screening instruments for the early detection of cognitive impairment in patients with multiple sclerosis. Int. J. MS Care 19 (1), 1–10. Langdon, D.W., Amato, M.P., Boringa, J., et al., 2012. Recommendations for a Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS). Mult. Scler. 18 (6), 891–898. Lapshin, H., Lanctôt, K.L., O'Connor, P., et al., 2013. Assessing the validity of a computergenerated cognitive screening instrument for patients with multiple sclerosis. Mult. Scler. J. 19 (14), 1905–1912. Mathias, J.L., Lucas, L.K., 2009. Cognitive predictors of unsafe driving in older drivers: a meta-analysis. Int. Psychogeriatr. 21 (4), 637–653. Migliore, S., Ghazaryan, A., Simonelli, I., et al., 2017. Cognitive impairment in relapsing-
Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.msard.2019.01.051. References Achiron, A., Doniger, G.M., Harel, Y., et al., 2007. Prolonged response times characterize cognitive performance in multiple sclerosis. Eur. J. Neurol. 14 (10), 1102–1108. Batista, S., Zivadinov, R., Hoogs, M., et al., 2012. Basal ganglia, thalamus and neocortical atrophy predicting slowed cognitive processing in multiple sclerosis. J. Neurol. 259 (1), 139–146. Benedict, R., 1997. Brief Visuospatial Memory Test – Revised: Professional Manual. Psychological Assessment Resources, Inc., Lutz, FL. Benedict, R., Morrow, S., Rodgers, J., et al., 2014. Characterizing cognitive function during relapse in multiple sclerosis. Mult. Scler. 20 (13), 1745–1752. Benedict, R.H.B., Cookfair, D., Gavett, R., et al., 2006. Validity of the minimal assessment of cognitive function in multiple sclerosis (MACFIMS). J. Int. Neuropsychol. Soc. 12 (4), 549–558. Benedict, R.H.B., Fischer, J.S., Beatty, W.W., et al., 2002. Minimal neuropsychological assessment of MS patients: a consensus approach. Clin. Neuropsychol. 16 (3), 381–397. Bonzano, L., Pardini, M., Roccatagliata, L., et al., 2017. How people with multiple sclerosis cope with a sustained finger motor task: a behavioural and fMRI study. Behav. Brain Res. 325, 63–71. Delis, D.C., Kaplan, E., Kramer, J., 2001. Delis-Kaplan Executive Functioning System Examiner's Manual. NCS Pearson, Inc., San Antonio, TX. Doniger, G.M., Simon, E.S., Schweiger, A., 2008. Adjustment of cognitive scores with a
161
Multiple Sclerosis and Related Disorders 30 (2019) 154–162
D. Golan, et al.
cognitive impairment in multiple sclerosis. Mult. Scler. Relat. Disord. 4 (4), 291–295. Schweiger, A., Doniger, G.M., Dwolatzky, T., et al., 2003. Reliability of a novel computerized neuropsychological battery for mild cognitive impairment. Acta Neuropsychol 1, 407–413. Sheridan, L.K., Fitzgerald, H.E., Adams, K.M., et al., 2006. Normative Symbol Digit Modalities Test performance in a community-based sample. Arch. Clin. Neuropsychol. 21 (1), 23–28. Smith, A., 1982. Symbol Digit Modalities Test: Manual. Western Psychological Services, Los Angeles. Strober, L., Englert, J., Munschauer, F., et al., 2009. Sensitivity of conventional memory tests in multiple neuropsychological battery and the minimal assessment. Mult. Scler. 15, 1077–1084. Sumowski, J.F., Benedict, R., Enzinger, C., et al., 2018. Cognition in multiple sclerosis. Neurology 0 10.1212/WNL.0000000000004977. Tanigawa, M., Stein, J., Park, J., et al., 2017. Finger and foot tapping as alternative outcomes of upper and lower extremity function in multiple sclerosis. Mult. Scler. J. Exp. Transl. Clin. 3 (1), 205521731668893. Wilken, J.A., Kane, R., Sullivan, C.L., et al., 2003. The utility of computerized neuropsychological assessment of cognitive dysfunction in patients with relapsing remitting multiple sclerosis. Mult. Scler. J. 9 (2), 119–128. Zakzanis, K.K., 2000. Distinct neurocognitive profiles in multiple sclerosis subtypes. Arch. Clin. Neuropsychol. 15 (2), 115–136.
remitting multiple sclerosis patients with very mild clinical disability. Behav. Neurol. 1–10. Mitrushina, M., Boone, B.K., Razani, J., et al., 2005. Locator and data tables for the Judgment of Line Orientation test. Handbook of Normative Data for Neuropsychological Assessment, second ed. Oxford university press, New York, pp. 798–804. Papathanasiou, A., Messinis, L., Georgiou, V.L., et al., 2014. Cognitive impairment in relapsing remitting and secondary progressive multiple sclerosis patients: efficacy of a computerized cognitive screening battery. ISRN Neurol. 2014, 1–7. Rao, S.M., 2018. Role of computerized screening in healthcare teams: why computerized testing is not the death of neuropsychology. Arch. Clin. Neuropsychol. 33 (3), 375–378. Rao, S.M., Leo, G.J., Bernardin, L., et al., 1991b. Cognitive dysfunction in multiple sclerosis. I. Frequency, patterns, and prediction. Neurology 41 (5), 685–691. Rao, S.M., Leo, G.J., Ellington, L., et al., 1991a. Cognitive dysfunction in multiple sclerosis. II. Impact on employment and social functioning. Neurology 41 (5), 692–696. Raven, J., 2000. The Raven's progressive matrices: change and stability over culture and time. Cogn. Psychol. 41 (1), 1–48. Roalf, D.R., Rupert, P., Mechanic-Hamilton, D., et al., 2018. Quantitative assessment of finger tapping characteristics in mild cognitive impairment, Alzheimer's disease, and Parkinson's disease. J. Neurol. 04 (0123456789), 1–4. Romero, K., Shammi, P., Feinstein, A., 2015. Neurologists accuracy in predicting
162