Reliability and construct validity of the Automated Neuropsychological Assessment Metrics (ANAM) mood scale

Archives of Clinical Neuropsychology 23 (2008) 73–85

Reliability and construct validity of the Automated Neuropsychological Assessment Metrics (ANAM) mood scale Dan R. Johnson ∗ , Andrea S. Vincent, Ashley E. Johnson, Kirby Gilliland, Robert E. Schlegel Center for the Study of Human Operator Performance, University of Oklahoma, United States Accepted 11 October 2007

Abstract The reliability and construct validity of the Automated Neuropsychological Assessment Metrics (ANAM) mood scale (AMS) were examined using concurrent, well-validated measures of mood and confirmatory factor analysis (CFA) with a sample of 210 volunteer college students. The AMS was given in computerized format with multiple adjectives using a visual analog Likert scale yielding seven dimensions of mood including vigor, restlessness, depression, anger, fatigue, anxiety, and happiness. All seven mood dimensions of the AMS demonstrated excellent test–retest reliability and internal consistency. Also, the AMS anxiety dimension correlated strongly with the Spielberger’s State Anxiety Inventory (r = 0.67) and the AMS depression dimension correlated strongly with the Beck Depression Inventory-II (r = 0.71). CFA revealed that the AMS 7-factor mood model fit the data well and significantly better than an alternative, theoretically plausible model. When concurrent measures of mood were incorporated in the CFA model, the AMS demonstrated both convergent and discriminant validity. The AMS 7-factor model explained 55.12% of the total variance in the items. It was concluded that the AMS provides a brief yet reasonably complete and valid assessment of mood. © 2007 National Academy of Neuropsychology. Published by Elsevier Ltd. All rights reserved. Keywords: Mood; Construct validity; Neuropsychological assessment; Traumatic brain injury

Computerized assessment of neuropsychological function is becoming an invaluable tool for the screening and monitoring of neuropsychological impairment and disease. In conjunction with cognitive testing, mood assessment is proving to be an integral component of computerized neuropsychological screening and monitoring. The focus of this paper is on the mood scale (AMS) of the Automated Neuropsychological Assessment Metrics (ANAM1 ; see also Reeves, Winter, Kane, Elsmore, & Bleiberg, 2001). ANAM is a library of computer-based tests that are often organized into batteries designed for a wide range of clinical and research applications. Both researchers and

∗ Corresponding author at: Center for the Study of Human Operator Performance, University of Oklahoma, 455 W. Lindsey, DAHT Room 740, Norman, OK 73019-2007, United States. Tel.: +1 515 360 5600; fax: +1 405 325 4737. E-mail address: [email protected] (D.R. Johnson). 1 The Center for the Study of Human Operator Performance (C-SHOP) at the University of Oklahoma is now responsible for the development and distribution of the Automated Neuropsychological Assessment Metrics. The latest version of ANAM is now version 4.1 and is referred to as ANAM4. Information regarding ANAM or ANAM4 can be obtained by contacting C-SHOP at 3200 Marshall Ave., Suite 260, University of Oklahoma, Norman, OK 73072.

0887-6177/$ – see front matter © 2007 National Academy of Neuropsychology. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.acn.2007.10.001

74

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

clinicians have recognized ANAM’s clinical and non-clinical utility in a number of domains including multiple sclerosis, systemic lupus erythematosus, Parkinson’s disease, Alzheimer’s disease, acquired brain injury, migraine headaches, environmental stressors, and general human performance (see Kane, Roebuck-Spencer, Short, Kabat, & Wilken, 2007 for review). Although construct validation has been reported for many of ANAM’s cognitive tests (see Short, Cernich, Wilken, & Kane, 2007), little validation of the AMS has been reported. Therefore, the reliability and construct validity of the AMS were examined using concurrent, well-validated measures of mood and confirmatory factor analysis. Considering the diversity of domains to which ANAM is applied, the AMS was validated with mood scales from both clinical and non-clinical domains. As Kane et al. (2007) note, the application of computer-based neuropsychological testing was motivated by the need to develop a screen for neurocognitive deficits that is not feasible using traditional, comprehensive neuropsychological examination. In this extensive review, Kane et al. demonstrated the utility of ANAM’s cognitive tests serving such a purpose for a variety of neurological diseases. There is also evidence that mood should be considered an important factor in the screening for neurological disease. For example, Montgomery et al. (2000) performed a prospective study on probable idiopathic Parkinson’s patients and controls in an attempt to develop a neuropsychological battery with high sensitivity and specificity to Parkinson’s disease. They found that depressed mood was an important factor in differentiating a probable Parkinson’s group from a control group. Notably, the sensitivity and specificity attained using the mood test alone was comparable to using a psychomotor test alone. Mood has also shown importance in screening for mild traumatic brain injury (TBI). Some of the commonly reported symptoms of mild TBI are elevated mood states such as fatigue, frustration, depression, and anxiety (Alexander, 1995; Serio & Devens, 1993). A significant portion of mild TBI patients may even develop depressive disorders with the average frequency being about 40% (Brooks, McKinlay, Symington, Beattie, & Campsie, 1987; Jorge & Robinson, 2002; McCleary et al., 1998). Notably, the onset of depressive disorders may occur long after the TBI (Gaultieri & Cox, 1991). Consequently, following a TBI, mood should be monitored along with cognitive function to track recovery of neurocognitive function and screen for those who may be at risk for developing depression. The need for monitoring mood disturbances is evident in soldiers who have been deployed. For example, a subset of those deployed in the 1991 Persian Gulf War generally demonstrated poor mental outlook and emotional resilience (Clauw, 2003; Glass et al., 2004). Stortzbach et al. (2000) found that compared to a non-deployed sample, deployed soldiers exhibited differences in cognitive function but also large differences in self-reported mood disturbances. Using self-report measures of mood and cognitive states, they were able to achieve a classification sensitivity of 89.2% (percentage of clinical group classified as clinical group) and a classification specificity of 80.4% (percentage of control group classified as control group). In addition, soldiers may be exposed to extreme environmental stressors or neurotoxins. Mood changes are often the earliest and most consistent symptoms of neurotoxin exposure and environmental stressors such as hypoxia and neuroglycopenia (Anger, Otto, & Letz, 1996; Frier, 2001; Fulco & Cymerman, 1988). Frequently, cognitive deficits are apparent after neurotoxin exposure, but can present later and less consistently (Anger, 2003). Therefore, effective measures of mood assessed shortly after neurotoxin exposure or an extreme environmental stressor can be important predictors of future cognitive decline. Recently, Vasterling et al. (2006) collected neuropsychological data from 961 soldiers pre- and post-deployment to Iraq. At post-deployment soldiers exhibited cognitive deficits in sustained attention, verbal learning, and visual spatial memory, as well as increased negative affect. Specifically, deployed soldiers reported significantly higher confusion and tension. This provides the most recent evidence in a large sample that mood changes are present in post-deployed soldiers. As opposed to a comorbid factor, mood can also be considered a confounding factor in the screening for neuropsychological dysfunction. It is now fairly well established that mood can have a significant impact on cognitive functions such as memory and attention (Christensen, Griffiths, Mackinnon, & Jacomb, 1997; Massman, Delis, Butters, Dupont, & Gillin, 1992; Shackman et al., 2006; Tiersky, Johnson, Lange, Natelson, & DeLuca, 1997). Considering the anxietyprovoking nature of neuropsychological assessment, it is important to understand neurocognitive deficits in the context of decrements caused by anxiety or depression alone. Recently, Roebuck-Spencer et al. (2006) used ANAM to predict neuropsychological functioning and emotional distress in patients with systemic lupus erythematosus. Depression and sleepiness are commonly reported symptoms of patients with systemic lupus erythematosus. Roebuck-Spencer et al. viewed depression and sleepiness as potential

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

75

confounding factors in screening for systemic lupus erythematosus and therefore investigated whether decrements in cognitive function persisted after controlling for these factors. They found that mood factors accounted for just over 10% of global neuropsychological functioning. Importantly, they stated that “ANAM cognitive tests remained independent predictors of neuropsychological functioning, even when depression and sleepiness were covaried” (Roebuck-Spencer et al.). Relevant to the current study, Roebuck-Spencer et al. (2006) compared the AMS and Beck’s Depression Inventory-II (Beck, Steer, & Brown, 1996) and found that the AMS depression subscale correlated highly with BDI-II scores. This result provides preliminary evidence that the AMS is a valid indicator of depressed mood. 1. Brief description, history, and previous psychometric properties of AMS From the beginnings of ANAM, the AMS has been incorporated in various forms in the ANAM test library. The current version of the AMS included in the ANAM4 test library assesses seven dimensions of mood including vigor, restlessness, depression, anger, fatigue, anxiety, and happiness. Multiple adjectives representing each of these domains are serially presented on the computer screen. Test takers rate each adjective on a Likert scale to indicate how well the adjectives describe how they feel at that moment. The AMS origins can be traced to Johnson and Meyers’ (1967) primary affect scale. This scale was based on mood intensity ratings by a moderately sized Navy personnel group. Ryman, Biersner, and La Rocco (1974) created a shortened version of this mood scale and established initial predictive validity by demonstrating its sensitivity to stress and anxiety reports from divers (Biersner, McHugh, & Rahe, 1984). More recently, Eckert, Goenert, Harris, and Nelson (1997) provided evidence for an alternate version of this mood scale’s concurrent validity by demonstrating high correlations between mood scale responses and responses to the well-validated Profile of Mood States (POMS; McNair, Lorr, & Droppleman, 1971). 2. Current study The purpose of this study is to determine the test–retest reliability, internal consistency, factor structure, and construct validity of the current AMS. This study extends previous psychometric work on ANAM cognitive tests (Short et al., 2007). Regarding the construct validity of the AMS, confirmatory factor analysis is used to (1) determine if ANAM’s 7-factor mood model fits the data, (2) compare ANAM’s 7-factor model to an alternative theoretically plausible model of mood, specifically the 2-factor positive and negative affect model posited by Watson, Clark, and Tellegen (1988), and (3) test the convergent and discriminant validity of AMS utilizing well-validated, concurrent measures of mood. Concurrent mood measures from diverse mood domains included the Standard Profile of Mood States (McNair et al., 1971), Beck Depression Inventory-II (BDI-II; Beck et al., 1996), State/Trait Anxiety Inventory (STAI; Spielberger, Gorsuch, Lushene, Vagg, & Jacobs, 1983), and the Dundee Stress State Questionnaire (DSSQ; Matthews et al., 1999). 3. Methods 3.1. Participants The sample consisted of 210 undergraduate students at the University of Oklahoma. Participants were recruited through an online experiment management system and received class credit for voluntary participation. This experiment was approved by the Internal Review Board at the University of Oklahoma. 3.2. Measures 3.2.1. ANAM Mood Scale II v.4.1 (AMS) The AMS is designed to assess seven dimensions of mood including vigor, restlessness, depression, anger, fatigue, anxiety, and happiness. Each dimension consists of six adjectives for a total of 42 items that are rated on a 7-point Likert scale of mood intensity. Mean ratings are computed for each scale with higher values reflecting greater degree of endorsement of each of the mood states. The AMS takes, on average, just over 2 min to complete.

76

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

3.2.2. Dundee Stress State Questionnaire (DSSQ) The mood scale of the DSSQ (Matthews et al., 1999) consists of 29 items designed to measure transient states associated with anxiety, energetic arousal, positive feelings, and anger/frustration. This scale was selected because these dimensions more often have relevance in the context of human performance research. In a large-scale study, Matthews et al. (2002) demonstrated the sensitivity of the DSSQ to changes in mood in response to stressful working memory and vigilance tasks. 3.2.3. Profile of Mood States (POMS) The Standard POMS (McNair et al., 1971) consists of a set of 65 Likert-type items for rating the intensity of six mood state factors: tension/anxiety, depression/dejection, anger/hostility, fatigue, vigor, confusion/bewilderment, as well as an index of total mood disturbance. The POMS was selected because it is a popular scale designed for mood assessment in clinical and non-clinical domains. 3.2.4. Spielberger State Anxiety Inventory (STAI-S) Spielberger’s State Anxiety Inventory (Spielberger et al., 1983) consists of 20 items that ask how a person feels currently, and reflects situational factors that may influence anxiety levels. Scores range from 20 to 80 and the higher the score, the greater the level of anxiety. The STAI-S was selected because state anxiety has relevance in both clinical and non-clinical domains. 3.2.5. Beck Depression Inventory-II (BDI-II) The BDI-II (Beck et al., 1996) consists of 21 items assessing the intensity of symptoms of depression. Each item is a list of four statements arranged in increasing severity about a particular symptom of depression. This scale was selected because it has been validated as a clinical scale of depression. 3.3. Procedure Upon arrival, each participant reviewed and signed an informed consent and completed a printed demographic questionnaire. Participants began the study session by completing four mood questionnaires/scales: (1) ANAM4 AMS (computerized), (2) DSSQ (paper and pencil), (3) STAI-S (paper and pencil), and (4) POMS (paper and pencil). Presentation of the questionnaires was counterbalanced to reduce potential carryover effects. After completing the initial set of questionnaires, participants were administered four performance tests from the ANAM4 library of tests (Simple Reaction Time, Two-Choice Reaction Time, Code Substitution – Learning, and Memory Search). These tests served as a time-filler between the first and second administrations of the mood questionnaires. These tests were selected because they are of a relatively low level of difficulty and should have lower emotional reactivity in comparison to anxiety-provoking tests like math and extremely difficult tests. These filler tests take, on average, 12 min to complete. Participants then completed a second administration of the four mood questionnaires/scales. To reduce questionnaire reactivity, the BDI-II was administered following the second administration of the mood scales. 3.4. Data analysis Analyses included calculation of descriptive statistics (mean and standard deviation) for the AMS and concurrent mood measures, examination of AMS internal consistency, test–retest reliability, and construct validity. Internal consistency was assessed using Cronbach’s α. Test–retest reliability was assessed by Pearson’s correlations between scores from the two administrations. Additionally, convergent and discriminant validity were assessed by Pearson’s correlations between scores on AMS and validation measures (POMS, BDI-II, DSSQ, and STAI-S). Finally, confirmatory factor analysis was performed to assess construct validity of the scales of the AMS. All analyses were conducted using SAS version 9.1 (SAS Institute, Inc., Cary, NC). 4. Results Table 1 contains demographics information on the sample. The mean age of the sample was 19.2 years (S.D. = 2.7), ranging in age from 18 to 42 years (Table 1). The majority of participants were female (72%) with a freshman-level

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

77

Table 1 Sample characteristics (N = 210) Variable Age (years) M (S.D.) Range

Value 19.2 (2.7) 18–42

Sex (n, %) Female

151 (72)

Ethnicity (n, %) White, non-Hispanic

162 (77.0)

College standing (n, %) Freshman Sophomore Junior Senior Other

142 (67.6) 32 (15.2) 19 (9.1) 15 (7.1) 2 (<1)

college standing (68%). Seventy-seven percent of the sample reported their ethnicity as white, non-Hispanic. Table 2 presents descriptive statistics for the AMS and concurrent mood measures. 4.1. Internal consistency Table 3 presents the internal consistency for each of the AMS scales. Cronbach’s α coefficients ranged from 0.80 (restlessness) to 0.93 (depression). Table 2 Scale means and standard deviations Scale ANAM AMS (first administration) Vigor Restlessness Depression Anger Fatigue Anxiety Happiness

Mean (S.D.) 3.26 (1.07) 1.41 (0.96) 1.08 (1.21) 0.82 (0.99) 2.06 (1.18) 1.22 (1.05) 4.01 (1.14)

POMS Tension/anxiety Depression/dejection Anger/hostility Vigor Fatigue Confusion Total mood disturbance

12.7 (5.8) 13.1 (10.9) 10.1 (8.8) 16.6 (5.7) 10.7 (5.5) 10.6 (4.1) 40.7 (31.8)

DSSQ Energetic arousal Tense arousal Hedonic tone Anger/frustration

21.1 (4.3) 14.5 (4.4) 25.7 (5.1) 7.6 (3.0)

BDI-II

7.85 (6.9)

STAI-S

36.2 (10.4)

POMS, Profile of Mood States (n = 208); DSSQ, Dundee Stress State Questionnaire (n = 208); BDI-II, Beck Depression Inventory (n = 203); STAI-S, Spielberger State Anxiety Inventory (n = 208).

78

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

Table 3 Internal consistency of AMS scales (N = 210) Scale

Raw Cronbach’s α

Vigor Restlessness Depression Anger Fatigue Anxiety Happiness

0.83 0.80 0.93 0.91 0.84 0.85 0.92

4.2. Test–retest reliability Table 4 demonstrates the test–retest reliability with correlations of the AMS from Time 1 to Time 2 ranging from r = 0.75 to 0.91. This table also indicates some significant differences in mean scores from Time 1 to Time 2. However, the sample size is large and power is very high, so these significant differences may not necessarily be interpreted as important shifts in mood. 4.3. Validity Table 5 shows the correlations between the AMS (taken at Time 1) and the other concurrent measures of mood, including the POMS, STAI-S, DSSQ, and BDI-II. Most theoretically consistent correlations were strong (r ranging between 0.59 and 0.70). Specifically, AMS mood dimensions correlated strongly with the corresponding dimensions of the POMS, BDI-II, DSSQ, and STAI-S, suggesting convergent validity. For example, the AMS anxiety dimension strongly correlated with the STAI-S (r = 0.67) as well as the POMS tension/anxiety dimension (r = 0.56). The AMS depression dimension strongly correlated with the BDI-II (r = 0.71) as well as the POMS depression/dejection dimension (r = 0.71). AMS mood dimensions unrelated to POMS, BDI-II, and STAI-S had weak correlations, suggesting discriminant validity. For example, the AMS vigor dimension weakly correlated with the BDI-II (r = −0.31) and AMS vigor dimension weakly correlated with the POMS tension/anxiety dimension (r = −0.16). Table 6 presents the correlations between all mood measures used to validate the AMS. Correlations were strong between theoretically consistent variables. For example, the POMS depression/dejection dimension strongly correlated with the BDI-II (r = 0.60). 4.4. Confirmatory factor analysis Confirmatory factor analysis (CFA) was used to fit and compare four models of mood. Model estimation was performed with a maximum likelihood approach using the CALIS procedure in SAS version 9.1 (SAS Institute, Inc., Cary, NC). This estimation procedure was used because it tends to be more robust to violations of multivariate normality Table 4 Test–retest reliability correlations of AMS scales (N = 210) Scale

Vigor Restlessness Depression Anger Fatigue Anxiety Happiness

Time 1

T2 − T1

Time 2

T1, T2

M

S.D.

M

S.D.

M

S.D.

p

r

3.26 1.41 1.08 0.82 2.06 1.22 4.01

1.07 0.96 1.21 0.99 1.18 1.05 1.14

2.80 1.38 0.99 1.01 2.18 1.02 3.62

1.25 1.07 1.12 1.11 1.29 0.99 1.26

−0.46 −0.01 −0.09 0.19 0.13 −0.20 −0.40

0.76 0.72 0.49 0.70 0.83 0.65 0.75

<0.001 0.800 0.011 <0.001 0.028 <0.001 <0.001

0.80 0.75 0.91 0.78 0.78 0.80 0.81

Scale

Vigor Restlessness Depression Anger Fatigue Anxiety Happiness

POMS

DSSQ

Tension/ anxiety

Depression/ Anger/ dejection hostility

Vigor

Fatigue

Confusion

Total mood disturbance

Energetic arousal

Tense arousal

Hedonic tone

Anger/ frustration

−0.16 0.49 0.44 0.41 0.31 0.56 −0.40

−0.33 0.47 0.71 0.53 0.50 0.57 −0.55

0.64 −0.19 −0.43 −0.21 −0.47 −0.24 0.54

−0.31 0.42 0.49 0.39 0.58 0.44 −0.41

−0.20 0.51 0.50 0.45 0.47 0.55 −0.31

−0.39 0.55 0.69 0.60 0.58 0.61 −0.60

0.76 −0.28 −0.54 −0.35 −0.74 −0.33 0.61

−0.27 0.60 0.53 0.51 0.43 0.65 −0.48

0.58 −0.46 −0.78 −0.56 −0.64 −0.54 0.81

−0.28 0.54 0.62 0.74 0.48 0.53 −0.48

−0.20 0.46 0.52 0.65 0.40 0.45 −0.46

STAI-S

BDI-II

−0.44 0.63 0.72 0.61 0.63 0.67 −0.64

−0.31 0.48 0.71 0.47 0.55 0.61 −0.53

POMS, Profile of Mood States (n = 208); DSSQ, Dundee Stress State Questionnaire (n = 208); BDI-II, Beck Depression Inventory (n = 203); STAI-S, Spielberger State Anxiety Inventory (n = 208).

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

Table 5 Correlations of AMS scales with validation measures

79

80

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

Table 6 Correlations between mood validity measures Factor

1

2

3

4

5

6

7

8

9

10

11

12

P. Tension P. Depression P. Anger P. Vigor P. Fatigue P. Confusion D. Energetic D. Tense D. Hedonic D. Anger STAI-S BDI-II

– 0.65 0.63 −0.09 0.59 0.67 −0.19 0.48 −0.35 0.41 0.52 0.39

– 0.72 −0.36 0.60 0.66 −0.39 0.42 −0.60 0.50 0.59 0.60

– −0.17 0.51 0.58 −0.24 0.38 −0.48 0.55 0.47 0.38

– −0.38 −0.13 0.65 −0.20 0.52 −0.22 −0.41 −0.45

– 0.57 −0.64 0.30 −0.44 0.33 0.52 0.47

– −0.28 0.40 −0.34 0.44 0.49 0.46

– −0.35 0.65 −0.39 −0.52 −0.51

– −0.55 0.59 0.74 0.37

– −0.62 −0.72 −0.58

– 0.65 0.42

– 0.56

–

P., Profile of Mood States (n = 208); D., Dundee Stress State Questionnaire (n = 208); BDI-II, Beck Depression Inventory (n = 203); STAI-S, Spielberger State Anxiety Inventory (n = 208).

than estimation by generalized least squares (Browne, 1982; West, Finch, & Curran, 1995). The sample was suitable for factor analysis according the KMO (0.927) and Bartlett (χ2 = 6406.02, p < 0.0001) statistics. Fit indices included the conventional χ2 , but χ2 should not be interpreted without other indices because it is sensitive to sample size and tends to produce Type II errors with large sample sizes (Chou & Bentler, 1995; Loehlin, 2004). However, the χ2 can also be used to perform a χ2 difference test in which two nested models are compared. The test indicates whether a model that estimates more parameters fits the data significantly better than a simpler model. The root mean square error of approximation (RMSEA) and its 90% confidence intervals are reported. Values less than 0.10 are considered acceptable fits (Browne & Cudeck, 1993). The RMSEA adjusts for both the number of parameters in the model (i.e., parsimony) and the sample size (Kline, 2005; Loehlin, 2004). The root mean square residual (RMR) is reported as an additional measure of absolute fit that, like the χ2 , does not adjust for sample size (Maruyama, 1998; Tanaka, 1993). Acceptable fits according to the RMR are under 0.10 (Kline, 2005; Loehlin, 2004). The Consistent Akaike’s Information Criterion (CAIC) is also reported because it adjusts for parsimony and provides a metric for model comparisons for models that are not nested. Smaller values for CAIC indicate better fit. 4.5. Tests of univariate and multivariate normality SAS provides skewness and kurtosis estimates originally derived by Browne (1982), where values near 0 indicate univariate normal distributions. According to Tabachnick and Fidell (2001), a z-score based on skewness and kurtosis standard errors can be calculated to test whether a distribution is significantly skewed or kurtotic with a conventional alpha of 0.001. Using this guideline, adjectives with extreme emotional valence (e.g., depressed, furious) were both significantly positively skewed and kurtotic, whereas adjectives with less extreme emotional valence (e.g., alert, lazy) were not. This is perhaps unsurprising given that a non-clinical group of healthy young adults was sampled and mood was not induced. We would not expect, on average, a sample of college students to report high depression scores or to become infuriated with relatively benign cognitive tasks. However, it should be noted that although maximum likelihood is relatively robust to non-normality, estimates could be inflated and should not be over interpreted (West et al., 1995). 4.6. Model comparisons Table 7 presents the fit indices for four models used to test the convergent and discriminant validity of the AMS. ANAM’s 7-factor model exhibited good fit to the data according to all fit indices. Table 8 provides standardized estimates (i.e., factor loadings) and r2 values for all AMS adjectives for each of the dimensions of mood. The ANAM factors explained variance in adjective ratings quite well with most r2 being above 0.50. Table 9 presents the factor correlations between each of the AMS mood dimensions. Some correlations are quite high (e.g., restlessness and anxiety) suggesting a simpler model with fewer mood dimensions may more closely capture

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

81

Table 7 Model comparisons with fit indices Model

d.f.

χ2

RMSEA

RMSEA 90% CI

RMR

CAIC

ANAM 7-factor Pos./neg. affect, fatigue Convergent validity Discriminant validity

798 816 1304 1304

1776.00 2395.78 3354.11 3529.23

0.0759 0.0953 0.0859 0.0895

0.0712–0.0806 0.0909–0.0998 0.0823–0.0895 0.0860–0.0931

0.0727 0.0891 0.0785 0.1026

−3303.05 −2798.86 −4947.12 −4772.00

Table 8 AMS’s 7-factor model (N = 210) Factors

Indicator

Standardized estimates

r2

Vigor

Energetic Lively Alert Spirited Active Vigorous

0.81 0.80 0.54 0.81 0.66 0.48

0.65 0.64 0.29 0.66 0.43 0.23

Restlessness

Restless Agitated Jittery Fidgety On-edge Shaky

0.60 0.70 0.67 0.55 0.72 0.63

0.36 0.49 0.44 0.30 0.52 0.40

Depression

Unhappy Depressed Hopeless Discouraged Miserable Sad

0.85 0.88 0.78 0.85 0.76 0.84

0.72 0.77 0.60 0.72 0.57 0.71

Anger

Irritated Furious Annoyed Grouchy Angry Enraged

0.83 0.83 0.81 0.80 0.86 0.68

0.68 0.69 0.65 0.64 0.74 0.46

Fatigue

Lazy Inactive Tired Weary Sluggish Drowsy

0.64 0.59 0.73 0.76 0.71 0.69

0.41 0.34 0.54 0.58 0.51 0.48

Anxiety

Insecure Uneasy Nervous Afraid Anxious Alarmed

0.68 0.83 0.72 0.71 0.67 0.63

0.46 0.68 0.52 0.50 0.45 0.39

Happiness

Good Satisfied Pleased Happy Content Cheerful

0.77 0.79 0.86 0.89 0.71 0.82

0.59 0.63 0.74 0.79 0.51 0.67

82

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

Table 9 Factor correlations between ANAM mood factors (N = 210) Factor

1

2

3

4

5

6

7

Vigor Restlessness Depression Anger Fatigue Anxiety Happiness

– −0.22 −0.48 −0.24 −0.62 −0.24 0.83

– 0.70 0.79 0.66 0.86 −0.49

– 0.75 0.78 0.83 −0.71

– 0.64 0.76 −0.50

– 0.67 0.62

– −0.54

–

the data. Therefore, Watson et al.’s (1988) model of mood was compared to the AMS model. Watson et al.’s model assumes two relatively independent dimensions of mood including positive and negative affect. The vigor and happiness dimensions of the AMS were combined to form the positive affect factor and the restlessness, anxiety, depression, and anger dimensions were combined to form the negative affect factor. Also, the AMS fatigue factor was kept in the model, but as a separate factor. Table 7 gives the fit indices for the ANAM’s 7-factor model and the Positive/Negative Affect and Fatigue model. Note that all fit indices worsened for the simpler model. A χ2 difference test between the two nested models indicated that ANAM’s 7-factor model fits the data significantly better than the Positive/Negative Affect and Fatigue model (χ2 (18) = 619.78, p < 0.0001). A convergent validity model was constructed by incorporating all concurrent measures of mood including the POMS, BDI-II, STAI, and the DSSQ. The facets of each concurrent mood dimension were placed under the concordant AMS mood dimensions (e.g., POMS-anger under AMS-anger). Table 7 indicates good convergent validity for the AMS, as the fit indices for the convergent validity model were within acceptable ranges. Also, the ANAM 7-factor model explained considerable variance in the concurrent measures (e.g., BDI-II r2 = 0.53, POMS-tension r2 = 0.41). A discriminant validity model was constructed by placing the BDI-II and POMS-tension facets of the concurrent mood dimensions under a discordant AMS mood dimension (see Table 7). When the BDI-II and POMS-tension scores were placed under the AMS-vigor dimension, the fit indices all worsened and their r2 dropped considerably (BDIII r2 = 0.27, POMS-tension r2 = 0.05). This result indicates that the AMS mood dimensions are measuring different aspects of mood. 5. Discussion The purpose of this study was to examine the psychometric properties of the AMS, including its reliability and construct validity. The AMS exhibited excellent test–retest reliability and internal consistency. Confirmatory factor analysis demonstrated that ANAM’s 7-factor mood model fits the data quite well and significantly better than a more parsimonious Positive/Negative Affect model. When concurrent measures of mood were incorporated into the ANAM 7-factor model, the model fits the data well, thereby providing empirical support for convergent validity of the AMS mood dimensions. Discriminant validity for the AMS was established by showing that the variance in the concurrent measures of mood was not explained well when they were put under a discordant AMS dimension (e.g., BDI-II under AMS vigor). This construct validity study of the AMS extends the psychometric research on ANAM and provides evidence that it is a reliable and valid testing system of both cognitive function and mood. Short et al. (2007) employed confirmatory factor analysis and concurrent, well-validated measures of cognitive function to provide evidence that ANAM cognitive tests measure attention, processing efficiency, and working memory. The construct validity analysis in the current study indicated that the AMS measures seven separable, but not completely independent, dimensions of mood that measure dimensions similar to those assessed by well-validated measures of mood including the BDI-II, STAI, DSSQ, and POMS. It is particularly noteworthy that the AMS dimensions were highly related to all concurrent measures because these measures were selected to cover the diverse domains in which mood is relevant, including clinical, non-clinical, and human performance domains. This result provides initial evidence that the AMS can be used to assess mood in the variety of areas in which the ANAM neuropsychological assessment batteries are used.

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

83

There are limitations in this study that conditionalize its results and conclusions. One important caveat is that all CFAs were performed on variables in which a subset of them was significantly non-normal. This can cause the inflation of estimates (West et al., 1995). However, the large sample size and relative robustness of maximum likelihood helped protect the models from spurious estimates. In addition, most of the results and conclusions were not based on interpreting single estimates but on overall model fit and model comparisons. Regarding the AMS reliability, the test–retest interval was brief, as the second administration of the AMS was given after four filler ANAM tests. Future work could include test–retest reliabilities over different time periods. However, by definition, mood is a transient state and one should not expect that mood be identical across all testing sessions, regardless of the test–retest interval, unless contextual circumstances were identical. It should also be noted that there were some mood changes from test to retest that achieved statistical significance. However, these changes should not be over-interpreted because power was quite high, effect size was low, and therefore these differences in mood may not qualify as truly important or meaningful changes in mood state. Also, these differences in mood can be due simply to the passage of time, instead of the testing itself. An additional limitation is that the results are generalizable primarily to a non-clinical, young adult population. The goal of the study was to investigate the reliability and construct validity of the AMS in a population that is most representative of the diversity of domains in which the AMS can be applied. However, as a thoughtful reviewer noted, future work is needed to test whether the factor structure changes as function of different populations. Indeed, this may even be an important diagnostic index above and beyond simple changes in mood. In addition, in a clinical domain it is often important to investigate a test’s sensitivity and specificity in discriminating between a clinical and control group or between two different clinical groups (e.g., Montgomery et al., 2000; Roebuck-Spencer et al., 2006). Future work should investigate the classification accuracy of the AMS with various clinical criterion groups. Although this study provides validation of an additional mood scale in a large body of previously constructed mood scales, the AMS is, to our knowledge, one of the few validated mood scales contained in a computer-based neuropsychological assessment battery. This is an important step in developing a complete but brief computer-based neuropsychological screening battery. There is accumulating evidence to suggest that screening and monitoring mood is vital to the assessment of TBI, soldier deployment effects, and neurological disease (David et al., 2002; Jorge & Robinson, 2002; Montgomery et al., 2000; Vasterling et al., 2006). The AMS demonstrated sound reliability and construct validity. The current study advocates the application of the AMS in a variety of domains, but especially in the neuropsychological domain. It is important to note that like other computer-based neuropsychological tests in ANAM, the AMS takes only a brief time to administer. Given the increasing importance given to mood in the neuropsychological domain, it is recommended that the AMS be combined with other ANAM cognitive tests in an endeavor to create a brief but complete computer-based neuropsychological assessment battery. Acknowledgments A special acknowledgement to Dr. Dennis Reeves for his design and incorporation of the ANAM Mood Scale in the original ANAM test library. A special thanks to Brook Weber for her coordination of the data collection process. We would also like to thank Kristen Hudec, Whitney Riggs Haaskama, Lauren Kennedy, and Jessica Wilkin for their help with data collection. References Alexander, M. P. (1995). Mild traumatic brain injury: Pathophysiology, natural history, and clinical management. Neurology, 45(7), 1253–1260. Anger, W. K. (2003). Neurobehavioural tests and systems to assess neurotoxic exposures in the workplace and community. Occupational and Environmental Medicine, 60, 531–538. Anger, W. K., Otto, D. A., & Letz, R. (1996). Symposium on computerized behavioral testing of humans in neurotoxicology research. Neurotoxicology and Teratology, 18, 347–520. Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the Beck Depression Inventory-II. San Antonio, TX: Psychological Corporation. Biersner, R. J., McHugh, W. B., & Rahe, R. H. (1984). Biochemical and mood responses predictive of stressful diving performance. Journal of Human Stress, 10(1), 43–49. Brooks, N., McKinlay, W., Symington, C., Beattie, A., & Campsie, L. (1987). Return to work within the first seven years of severe head injury. Brain Injury, 1, 5–19.

84

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

Browne, M. W. (1982). Covariance structures. In D. M. Hawkins (Ed.), Topics in applied multivariate analysis (pp. 72–141). Cambridge, England: Cambridge University Press. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: Sage. Chou, C.-P., & Bentler, P. M. (1995). Estimates and tests in structural equation modeling. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage. Christensen, H., Griffiths, K., Mackinnon, A., & Jacomb, P. (1997). A quantitative review of cognitive deficits in depression and Alzheimer-type dementia. Journal of the International Neuropsychological Society, 3, 631–651. Clauw, D. (2003). The health consequences of the first Gulf War – the lessons are general (for many patients) rather than specific to that war. British Medical Journal, 327, 1357–1358. David, A. S., Farrin, L., Hull, L., Unwin, C., Wessley, S., & Wykes, T. (2002). Cognitive functioning and disturbances of mood in UK veterans of the Persian Gulf War: A comparative study. Psychological Medicine, 32, 1357–1370. Eckert, L. H., Goernert, P. N., Harris, W. L., & Nelson, K. (1997). Computer-assisted test administration: Establishing equivalency of two mood measures. Proceedings of the Human Factors and Ergonomics Society U.S.A., 2, 1408. Frier, B. M. (2001). Hypoglycaemia and cognitive function in diabetes. International Journal of Clinical Practice, 123(Suppl.), 30–37. Fulco, C. S., & Cymerman, A. (1988). Human performance and acute hypoxia. In K. B. Pandolf, M. N. Sawka, & R. R. Gonzalez (Eds.), Human performance physiology and environmental medicine at terrestrial extremes (pp. 467–495). Traverse City, MI: Cooper Publishing Group. Gaultieri, C., & Cox, D. (1991). The delayed neurobehavioral sequelae of traumatic brain injury. Brain Injury, 5, 219–232. Glass, J. M., Lyden, A. K., Petzske, F., Stein, P., Whalen, F., Ambrose, K., et al. (2004). The effect of brief exercise cessation on pain, fatigue, and mood symptom development in healthy, fit individuals. Journal of Psychosomatic Research, 57, 391–398. Johnson, E., & Myers, T. I. (1967). The development and use of the Primary Affect Scale (PAS). Report 67-1. Bethesda, MD: Naval Medical Research Institute. Jorge, R., & Robinson, R. G. (2002). Mood disorders following traumatic brain injury. Neurorehabilitation, 17(4), 311–324. Kane, R. L., Roebuck-Spencer, T., Short, P., Kabat, M., & Wilken, J. (2007). Identifying and monitoring cognitive deficits in clinical populations using Automated Neuropsychological Assessment Metrics (ANAM) tests. Archives of Clinical Neuropsychology, 22S, 115–126. Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed.). New York, NY: Guilford Press. Loehlin, J. C. (2004). Latent variable models: An introduction to factor path and structural equation analysis (4th ed.). Mahwah, NJ: Lawrence Erlbaum. Maruyama, G. M. (1998). Basics of structural equation modeling. Thousand Oaks, CA: Sage. Massman, P. J., Delis, D. C., Butters, N., Dupont, R. M., & Gillin, J. C. (1992). The subcortical dysfunction hypothesis of memory deficits in depression: Neuropsychological validation in a subgroup of patients. Journal of Clinical and Experimental Neuropsychology, 14, 687–706. Matthews, G., Campbell, S. E., Falconer, S., Joyner, L. A., Huggins, J., Gilliland, K., et al. (2002). Fundamental dimensions of subjective state in performance settings: Task engagement, distress, and worry. Emotion, 2(4), 315–340. Matthews, G., Joyner, L., Gilliland, K., Campbell, S. E., Huggins, J., & Falconer, S. (1999). Validation of a comprehensive stress state questionnaire: Towards a state ‘Big Three’? In I. Mervielde, I. J. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality psychology in Europe (pp. 335–350). Tilburg: Tilburg University Press. McCleary, C., Satz, P., Forney, D., Roger, L., Zaucha, F., Asarnow, R., et al. (1998). Depression after traumatic brain injury as a function of Glasgow outcome score. Journal of Clinical and Experimental Neuropsychology, 20(2), 270–279. McNair, D. M., Lorr, M., & Droppleman, L. F. (1971). Manual for the profile of mood states. San Diego, CA: Educational & Industrial Testing Service. Montgomery, E. B., Koller, W. C., LaMantia, T. J. K., Newman, M. C., Swanson-Hyland, E., Kaszniak, W., et al. (2000). Early detection of probably idiopathic Parkinson’s disease: I. Development of a diagnostic test battery. Movement Disorders, 15(3), 467–473. Reeves, D., Winter, K., Kane, R., Elsmore, T., & Bleiberg, J. (2001). Automated neuropsychological assessment metrics user’s manual: Clinical and research methods (Special Rep. No. NCRF-SR-2002-01). The Army Medical Research and Material Command, National Cognitive Recovery Foundation. Roebuck-Spencer, T. M., Yarboro, C., Nowak, M., Takada, K., Jacobs, G., Lapteva, L., et al. (2006). Use of computerized assessment to predict neuropsychological functioning and emotional distress in patients with systemic lupus erythematosus. Arthritis and Rheumatism, 55(3), 434– 441. Ryman, D. H., Biersner, R. J., & La Rocco, J. M. (1974). Reliabilities and validities of the mood questionnaire. Psychological Reports, 35, 479–484. Serio, C. D., & Devens, M. (1993). Employment problems following traumatic brain injury: Families assess the causes. Neurorehabilitation, 4, 5–19. Shackman, A. J., Sarinopoulos, I., Maxwell, J. S., Pizzagali, D. A., Lavric, A., & Davidson, R. J. (2006). Anxiety selectively disrupts visuospatial working memory. Emotion, 6(1), 41–60. Short, P., Cernich, A., Wilken, J. A., & Kane, R. L. (2007). Initial construct validation of frequently employed ANAM measures through structural equation modeling. Archives of Clinical Neuropsychology, 22S, 63–77. Spielberger, C. D., Gorsuch, R. L., Lushene, R., Vagg, P. R., & Jacobs, G. A. (1983). Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press. Stortzbach, D., Campbell, K. A., Binder, L. M., McCauley, L., Anger, K. W., Rohlman, D. S., et al. (2000). Psychological differences between veterans with and without Gulf War unexplained symptoms. Psychosomatic Medicine, 62(5), 726–735. Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics (4th ed.). Needham Heights, MA: Allyn & Bacon. Tanaka, J. S. (1993). Multifaceted conceptions of fit in structural equation models. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 10–39). Thousand Oaks, CA: Sage.

D.R. Johnson et al. / Archives of Clinical Neuropsychology 23 (2008) 73–85

85

Tiersky, L. A., Johnson, S. K., Lange, G., Natelson, B. H., & DeLuca, J. (1997). Neuropsychology of chronic fatigue syndrome: A critical review. Journal of Clinical and Experimental Neuropsychology, 19, 560–586. Vasterling, J. J., Proctor, S. P., Amoroso, P., Kane, R., Heeren, T., & White, R. (2006). Neuropsychological outcomes of army personnel following deployment to the Iraq War. Journal of the American Medical Association, 296(5), 519–529. Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063–1070. West, S. G., Finch, J. F., & Curran, P. J. (1995). Structural equation models with non-normal variables: Problems and remedies. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage.