Psychometric properties of the Mid-Range Expanded Trail Making Test An examination of learning-disabled and non-learning-disabled children

Archives of Clinical Neuropsychology 18 (2003) 107–120

Psychometric properties of the Mid-Range Expanded Trail Making Test An examination of learning-disabled and non-learning-disabled children夽 Daniel E. Stanczak a,∗ , George Triplett b a

Psychology Research Service, Wilford Hall Air Force Medical Center, 59th Medical Wing/MMCPR, 2200 Bergquist Drive, Suite 1, Lackland Air Force Base, TX 78236, USA b Counseling Associates of Central Florida, Lakeland, FL, USA Accepted 1 October 2001

Abstract By systematically varying the stimuli within the Mid-Range Trail Making Test (MTMT), it may eventually be possible to isolate the cognitive demands of this test. Toward this end, the Mid-Range Expanded TMT (METMT) was developed by adding five new forms to the original forms A and B. To insure its appropriateness for clinical and experimental use, the current study sought to quantify the psychometric properties of this new test. The results indicate that the METMT is reliable and has adequate construct, criterion, and factoral validity. The results also cross-validated the findings of Davis et al. [J. Clin. Psychol. 45 (1989) 423.], suggesting that brief neuropsychological evaluations can effectively differentiate normal learners from learning-disabled children. The present study suggests that the MTMT is robust to alterations of its stimulus dimensions. Preliminary METMT normative data are presented. © 2001 National Academy of Neuropsychology. Published by Elsevier Science Ltd. All rights reserved. Keywords: Neuropsychological assessment; Learning disabilities expanded trail making test

夽 The views contained herein are solely those of the authors. Endorsement by the Department of the Air Force, Department of Defense, or any other governmental agency should not be presumed. Specimen sets of the METMT are available from the first author. ∗ Corresponding author. Tel.: +1-210-671-3498; fax: +1-210-671-3737. E-mail address: [email protected] (D.E. Stanczak).

0887-6177/01/$ – see front matter © 2001 National Academy of Neuropsychology. PII: S 0 8 8 7 - 6 1 7 7 ( 0 1 ) 0 0 1 9 0 - 1

108

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

1. Introduction As with the adult version of the Trail Making Test (TMT), meaningful interpretation of the Mid-Range version of the TMT (MTMT; Reitan, 1971) is difficult, because the specific cognitive components of successful MTMT performance are unknown. Using the adult TMT, Fossum, Holmberg, and Reinvang (1992) varied the test stimuli by alternating the configural arrangement of the test stimuli, that is, placing the stimuli of form B in the spatial configuration of form A and visa versa. Using this paradigm, they were able to determine that at least a portion of TMT variance could be explained by two stimulus dimensions: symbolic complexity and spatial configuration. The also raised the question of “. . . whether an expanded version of the TMTshould be developed and standardized.” Such an expanded version of the TMT was introduced by Stanczak, Lynch, McNeil, and Brown (1998), and efforts continue to tease out the sources of variance on this expanded measure. A similar attempt to systematically vary to the stimulus properties of the MTMT in terms of stimulus complexity, type, and background has been undertaken. The result of this effort is the development of the Mid-Range Expanded TMT (METMT; Stanczak, unpublished test). The METMT includes forms A and B from the Mid-Range Halstead– Reitan Neuropsychological Test Battery and five experimental Trail Making forms (X, Y, Z, G, and B4). METMT forms X, Y, and Z are downward extensions of the respective adult forms presented in Stanczak et al. (1998) and Stanczak, Stanczak, and Awadalla (2001). METMT form G requires the subject to connect circles containing an increasing number of blocks, while form B4 is similar to form B, alternating numbers and letters, with the exception that there is a distracting background consisting of rows of uncircled letters and numbers. One previous study employing forms X and Y of the METMT was published by Davis, Adams, Gates, and Cheramie (1989). In this earlier study, the investigators found that— using a battery consisting of the Tactual Performance Test total time, memory, and localization scores and the METMT forms A, B, X, and Y scores—they could differentiate learning-disabled from non-learning-disabled children with 82.5% accuracy. While this earlier work was important in establishing the clinical utility of the METMT, the external validity of the study was compromised by several factors. First, the study appears to have been severely underpowered. Second, Davis et al. failed to control for the potentially confounding effects of concomitant attention-deficit disorders (ADD). Third, there was no attempt to screen participants vis-à-vis medications. To be effectively used as an investigational tool, the psychometric properties of the METMT must be determined and such is the purpose of the present study. In addition to attempting to cross-validate the earlier results of Davis et al. (1989), the present study (a) examines the reliability and validity of the METMT, (b) provides preliminary normative data for the Expanded TMT, (c) assesses the unique proportion of between-group variance (learning-disabled versus normal learners) explained by each METMT form, (d) examines the factor structure of the METMT, and (e) determines if the METMT factor structure is stable across clinical and control samples.

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

109

2. Method 2.1. Subjects Subjects were 188 students, ages 9–14, recruited from schools in California, Georgia, Alabama, and Florida. The learning-disabled group (n = 95) were selected on the basis of the following inclusion criteria: (a) participation in resource classes for learning-disabled students; (b) a California Achievement Test or Colorado Achievement Test score, for any content area, of one or more standard deviations below the student’s IQ as measured by Kaufman Brief Intelligence Test (K-BIT; Kaufman & Kaufman, 1990); (c) a Conner’s Continuous Performance Test (CCPT) index score (Connors, 1995) less than 8 in order to minimize the potentially confounding effect of concomitant ADD; and (d) a negative history for neurological problems. Normal learners (n = 93) had achievement test scores within one standard deviation of their K-BIT IQ, CCPT index scores less than 8, and a negative history for neurological problems. The learning-disabled subjects did not differ significantly from the normal learners in terms of age (t = −0.364, df = 186, P = .72), grade level (t = 0.481, df = 186, P = .63), or gender (χ 2 = 1.38, df = 1, P = .24). However, significant between-group differences were noted in measures of intelligence (K-BITVIQ: t = 7.25, df = 178, P < .001; K-BITPIQ: t = 4.86, df = 178, P < .001; K-BITFIQ: t = 7.27, df = 178, P < .001). Such differences in IQ were to be expected given the strong correlation between intelligence and achievement and the negative impact learning disabilities have upon achievement. Participant demographic information is summarized in Table 1. 2.2. Independent measures All subjects were given, in order, forms A, B, X, Y, Z, G, and B4 of the Mid-Range version of the METMT according to the procedures outlined by Reitan and Wolfson (1993), with modifications as necessary to reflect the different stimuli on forms X to B4. The time(s) to completion of each form served as the independent measure. All tests were administered by neuropsychology graduate students under the supervision of a licensed neuropsychologist. 2.3. Reliability studies 2.3.1. Analysis 1: interrater reliability To assess the interrater reliability of the METMT, 70 subjects were randomly assigned to one of two raters who were blind to the results obtained by the other and who administered all METMT forms to each child in his/her group. The two groups did not differ significantly in terms of age (t = −1.00, df = 34, P = .32), gender (χ 2 = 0.532, df = 1, P = .52), or grade level (t = 0.481, df = 186, P = .35). Independent sample t tests were calculated for each METMT form on the scores obtained by each examiner. The results, summarized in Table 2, indicated no significant group differences, suggesting adequate interrater reliability for the METMT.

110

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

Table 1 Group demographics Normal learners Age

Learning disabled

Mean Standard deviation

11.47 11.50 1.70 1.69 Gender M = 40, F = 53 M = 48, F = 45 Grade level 3 4 4 4 19 19 5 19 24 6 16 15 7 9 15 8 26 16 K-BITVIQ Mean 104.28 92.61 Standard deviation 10.96 10.61 K-BITPIQ Mean 107.01 98.50 Standard deviation 12.55 10.81 K-BITFIQ Mean 106.27 95.12 Standard deviation 11.05 9.41 Type of learning disability (determined by discrepancy between IQ and achievement test domain scores) Math 14 Reading 15 Language 15 Math and language 49

2.3.2. Analysis 2: test–retest reliability Thirty-five children, tested by Examiner 1, were retested 1 month later by the same examiner. A coefficient of stability was calculated, using initial and delayed test scores, for each METMT form. These results are summarized in Table 3. 2.3.3. Analysis 3: internal consistency To determine its internal consistency, the scores for all METMT forms were entered into a reliability analysis. Cronbach’s α and Guttman’s λ2 , λ5 , and λ6 were chosen as indices of reliability since they establish a lower bound for the true reliability of a measure and, particularly in the case of λ5 and λ6 , do not presume equal covariances or high interitem Table 2 METMT interrater reliability METMT form

t

P

A B X Y Z G B4

−0.317 −1.133 −1.389 0.656 −1.098 −0.715 −0.236

.753 .265 .174 .513 .260 .479 .815

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

111

Table 3 METMT test–retest reliability METMT form

Coefficient of stability

P

A B X Y Z G B4

0.55 0.72 0.62 0.75 0.54 0.71 0.79

<.001 <.001 <.001 <.001 <.001 <.001 <.001

Table 4 METMT internal consistency Measure

Value

Cronbach’s α Guttman’s λ2 Guttman’s λ5 Guttman’s λ6

0.86 0.86 0.84 0.86

correlations relative to item squared multiple correlations. The results, summarized in Table 4, indicate the lower reliability bounds to be approximately 0.84, a value that does not appreciably change if a single METMT form is deleted. 2.4. Validity studies 2.4.1. Analysis 4: construct validity Numerous studies have demonstrated the sensitivity of the TMT to cerebral dysfunction in adults (Alvarez, 1962; Armitage, 1946; Cosgrove & Newell, 1991; Reitan, 1955, 1958; Reitan & Wolfson, 1985) and children ages 9–14 (Reitan, 1971; Reitan & Herring, 1982). The Expanded TMT (Stanczak et al., 1998) has also been shown to be sensitive to cerebral dysfunction in adults. It is probable, therefore, that the METMT should also be sensitive to cerebral dysfunction such as that found in learning disabilities. To test this hypothesis, the logarithmically transformed performances of normal learners and learning-impaired children on the METMT were compared using analysis of variance (ANOVA). The reasons for the logarithmic transformations of all METMT scores prior to the analysis are presented below. Levene’s test indicated that homogeneity of variance could not be presumed on METMT forms X and G. Nevertheless, the robustness of the ANOVA technique was supported by the fact that (a) sample sizes were equal, (b) the ratios of within-cells variances were within acceptable limits, (c) transformed scores were used, and (d) multivariate outliers were eliminated (Tabachnick & Fidell, 1996, p. 328). Thus, ANOVA was employed in spite of questions regarding homogeneity of variance. Because of the multiple comparisons, a Bonferroni correction to the α level resulted in an α of .007. The results, which are summarized in Table 5, indicate that

112

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

Table 5 METMT construct validity METMT form

Normal learners

Learning disabled

df

M.S.E.

F

P

Alog Blog Xlog Ylog Zlog Glog B4log

2.77 (0.35) 3.43 (0.37) 4.60 (0.43) 3.41 (0.33) 3.43 (0.36) 4.79 (0.37) 3.46 (0.31)

3.08 (0.36) 3.89 (0.40) 4.96 (0.35) 3.82 (0.41) 3.81 (0.39) 5.06 (0.26) 3.82 (0.35)

1,184 1,184 1,184 1,184 1,184 1,184 1,184

4.50 9.86 6.13 7.63 6.47 3.45 5.88

34.93 66.86 39.99 53.51 45.65 33.30 54.01

<.001 <.001 <.001 <.001 <.001 <.001 <.001

Cell content = mean (standard deviation).

learning-disabled children performed significantly worse than normal learners on all METMT forms. 2.4.2. Analysis 5: factorial validity Prior to analysis, the score distributions of all variables (age, gender, K-BITVIQ, K-BITPIQ, and the various METMT forms) were examined vis-à-vis the assumptions of multivariate analysis. The distributions of all METMT forms were highly kurtotic, and the distributions for forms B, Y, Z, and B4 were positively skewed. Thus, to improve pairwise linearity and to more closely approximate normal distributions, all METMT variables were subjected to natural logarithmic transformation. A review of these transformed distributions revealed skewness within acceptable boundaries. However, the variable, METMT form Y, remained slightly kurtotic. It was nevertheless retained for subsequent analyses. Using Mahalanobis distance, two cases were identified as multivariate outliers, both from the learning-disabled group. In order to more accurately estimate population parameters, these cases were deleted from subsequent analyses. This procedure left 186 cases, 93 in each group, for the principle components analysis (PCA). To rule out possible multicollinearity or singularity, squared multiple correlations (SMC’s) were calculated using, in turn, each variable as the dependent variable and the other variables as independent variables. The SMC’s ranged from 0.15 to 0.60, with seven of the eleven SMC’s being less than 0.5. In addition, the determinant was 0.03. These results argued against the presence of collinearity, and all variables were thus retained for the PCA. The factorability of the correlation matrix was established by Bartlett’s Test of Sphericity (χ 2 = 672.46, df = 55, P < .001), Kaiser–Meyer–Olkin Measure of Sampling Adequacy (0.813), significant bivariate correlations (only 4 of 35 correlations were not significant at the 0.05 level), and by examination of the anti-image correlation matrix. Principle factors extraction with orthogonal (varimax) rotation was performed. Three factors were extracted which accounted for 62% of the variance. The SMC’s between the variables and their respective factors were significant (Factor 1: r 2 = .99; Factor 2: r 2 = .96; and Factor 3: r 2 = .73) indicating that the factors were internally consistent and well defined by the variables. Variables were, by and large, only moderately well defined by this factor solution as indicated by the communality values in Table 6.

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

113

Table 6 METMT factor structure Factor loadings Communalities Full sample

Variance explained (%) Normal learners

Variance explained (%) Learning disabled

Variance explained (%)

B4log Blog Zlog Ylog Alog Glog Xlog Age PIQ VIQ Gender

0.709 0.724 0.634 0.644 0.516 0.575 0.400 0.496 0.721 0.714 0.747

Zlog B4log Age Blog Xlog Ylog Glog Gender Alog PIQ VIQ

0.61 0.76 −0.72 0.58 0.41 0.53 0.64 0.55 0.45 0.77 0.67

B4log Alog Zlog Ylog Blog VIQ PIQ Xlog Glog Age Gender

0.74 0.61 0.59 0.70 0.71 0.60 0.50 0.34 0.53 0.59 0.56

Factor 1

Factor 2

Factor 3

0.82 0.81 0.80 0.73 0.69 0.60 0.58 −0.50 0.84 0.82 37.81 0.77 0.76 −0.72 0.70 0.64 0.56

23.2 0.81 0.77 0.77 0.70 0.64

29.39

14.04

−0.86 10.68

0.46 0.74 −0.68 0.58

14.7

0.87 0.81 11.03

0.46 −0.52 0.74 0.69 −0.54 −0.47

16.04

0.72 −0.70 13.41

With an arbitrarily established factor loading criterion of 0.45, each variable loaded on one and only one factor. Factor loadings, communalities, and percentage of variance explained are shown in Table 6. In order to facilitate interpretation, factor loadings below 0.45 are not shown. To determine if the obtained factor structure was stable across diagnostic categories, separate principle factors extractions with orthogonal (varimax) rotations were conducted for each

114

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

group. For normal learners, the resulting rotated factors were very similar to those obtained on the full sample. A three-factor solution emerged which accounted for approximately 59% of the variance. The SMC’s between the variables and their respective factors were significant (Factor 1: r 2 = .94; Factor 2: r 2 = .93; Factor 3: r 2 = .96), indicating that the factors were internally consistent and well defined by the variables. As indicated by the communalities, variables were only moderately well defined by this factor solution. With an arbitrarily established factor loading of 0.45, all variables, except METMT Form Y, loaded on one and only one factor. Form Y loaded on both factors I and II. Factor loadings, communalities, and percentage of variance explained are shown in Table 6. The PCA with orthogonal rotation of factors for the learning-disabled group also yielded a three-factor solution, which accounted for approximately 59% of the variance. In this solution, the SMCs between the variables and their respective factors were also significant (Factor 1: r 2 = .96; Factor 2: r 2 = .95; Factor 3: r 2 = .91), again indicating that the factors were internally consistent and well defined by the variables. As with the previous analyses, variables were only moderately well defined by this factor solution as indicated by the communality values. Using the same arbitrarily established factor loading criterion of 0.45, all variables except METMT form Y loaded on one and only one factor. METMT form Y loaded on Factors 1 and 3. Factor loadings, communalities, and percentage of variance explained are shown in Table 6. 2.4.3. Analysis 6: criterion validity A direct discriminant function analysis was performed using the natural logarithmically transformed METMT scores as predictors of group membership (normal learners versus learning disabled). As above, two multivariate outliers were identified and eliminated from subsequent analyses, yielding a total of 186 subjects, 93 in each group. Evaluation of assumptions of linearity, normality, multicollinearity or singularity, and homogeneity of variance–covariance matrices revealed no threat to multivariate analysis. One significant discriminant function (Wilks’ λ = 0.622; χ 2 = 85.61; df = 7, P < .0001) was calculated which maximally separated normal from disabled learners and which explained 37.8% of the between-group variance. The loading matrix between predictors and the discriminant function suggested that METMT form B was the best predictor, though all METMT forms had loadings in excess of 0.50. Using a jackknifed classification procedure, 67 (72%) of normal learners and 68 (73.1%) of learning-disabled students were correctly classified. Twenty-six normal learners (28%) were incorrectly predicted to be learning disabled, while 25 (26.9%) of learning-disabled students were incorrectly assigned to the normal learner group. The overall correct classification rate for the discriminant analysis was 72.6%. To determine if these results produced by the discriminant function differed significant from those obtained in the direct discriminant analysis reported by Davis et al. (1989, p. 426), a χ 2 analysis was performed using Davis et al.’s results as the expected frequencies. This analysis indicated that the results did, indeed, differ significantly (χ 2 = 40.50, df = 3, P < .01). Post hoc χ 2 tests revealed that the two functions differed significantly only in terms of the number of false-positive errors, with the function derived in the present study producing a higher rate than that obtained by Davis et al.

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

115

To determine which of the METMT forms explained the most between-group variance, a follow-up stepwise discriminant function analysis was performed, again using the logarithmically transformed METMT forms as predictors. The resulting discriminant function was significant (Wilks’ λ = 0.641, χ 2 = 81.1, df = 3, P < .001) and explained 35.9% of the between-group variance. Three variables emerged as significant predictors: METMT forms B (exact F = 66.86, df = 1, 184, P < .00001), Y(exact F = 44.5, df = 2, 183, P < .00001), and X (exact F = 33.95, df = 3, 182, P < .001). Again using a jackknifed procedure, the resulting discriminant function correctly identified 70 (75.3%) of normal learners and 69 (74.2%) of learning-disabled children. Twenty-three normal learners (24.7%) were incorrectly classified as learning disabled, and 24 (25.8) of learning-disabled children were incorrectly classified as normal learners. The overall correct classification or “hit” rate was 74.7%. The results of this analysis differed significantly from the results of the stepwise analysis performed by Davis et al. (χ 2 = 11.6, df = 3, P < .01). Post hoc χ 2 tests indicated that the two functions differed significantly in terms of false-negative rates, with the number of false negatives obtained in the present study being higher than that produced by Davis et al.’s discriminant function. To determine if the results of the direct and stepwise procedures of the present study differed significantly, a χ 2 analysis was performed using the results obtained in the direct discriminant analysis as the expected frequencies. The two frequency distributions did not differ significantly (χ 2 = 0.54, df = 3, P < .90). A review of the residuals of the current stepwise procedure revealed that roughly 55% of misclassified cases occurred within the 9- and 10-year age groups. To determine whether or not the various age groups differed in terms of the percentage of misclassified cases, the number of observed misclassifications for each age, divided by the n for that age group, was compared to the percentage of misclassifications that might be expected by chance alone. The resulting χ 2 analysis was significant (χ 2 = 440.06, df = 5, P < .01). Follow-up χ 2 analyses demonstrated that the percentage of diagnostic misclassifications among 9- and 10-year-olds was significantly higher than that for all other age groups (Table 7). To more closely examine the differences between the younger (9- and 10-year-olds) and older groups (11–14-year-olds), each age group was separately entered into a stepwise discriminant function analysis using the logarithmically transformed METMT scores as predictor variables. For the younger group, the resulting discriminant function was significant (Wilks’ λ = 0.717, χ 2 = 8.81, df = 1, P = .003) and explained 28.3% of the variance. The only variable to emerge as a significant predictor was METMT form B4. Using a jackknifed procedure, the discriminant function correctly identified 10 (71.4%) of learning-disabled children and 13 (86.7%) of normal learners. Four learning-disabled participants (28.6%) were incorrectly classified as normal learners, while two normal learners (13.3%) were incorrectly classified as learning disabled. The overall correct classification rate for this analysis was 79.3%. The discriminant function, which emerged from the analysis with the older children, was also significant (Wilks’ λ = 0.465, χ 2 = 105.42, df = 3, P < .001) and explained 53.5% of the variance. Variables emerging as significant predictors were METMT forms B, Z, X, and B4. Using a jack-knifed procedure, the discriminant function correctly identified 68 (86.1%) of leaning-disabled students and 56 (90.3%) of normal learners. Eleven (13.9%)

116

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

Table 7 Comparison of diagnostic statistics for the two studies Davis et al.

Current study

Measure

Direct

Stepwise

Direct

Stepwise

Younger group stepwise

Older group stepwise

Sensitivity Specificity Positive predictive power Negative predictive power False-positive rate False-negative rate Observed agreement “hit” rate Chance agreement κ Standard error of κ Z of κ p

0.75 0.90 0.88 0.78 0.10 0.25 0.82 0.50 0.65 0.16 4.16 <.00004

0.88 0.75 0.70 0.90 0.25 0.12 0.80 0.50 0.60 0.15 3.87 <.0002

0.73 0.72 0.72 0.73 0.28 0.27 0.72 0.50 0.45 0.07 6.16 <.00001

0.74 0.75 0.75 0.74 0.25 0.25 0.75 0.50 0.49 0.07 6.75 <.00001

0.71 0.87 0.83 0.76 0.13 0.28 0.79 0.50 0.58 0.18 3.17 <.002

0.86 0.90 0.92 0.84 0.10 0.14 0.88 0.56 0.76 0.08 9.02 <.00001

of learning-disabled children were misclassified as normal learners, and six (9.7%) of normal learners were misclassified as learning disabled. The observed agreement “hit” rate for this function was 87.9%. To further assess the criterion validity of the METMT, separate direct discriminant analyses were performed, each employing one of the METMT forms as the discriminating variable with group membership (normal learners vs. learning disabled) as the dependent variable.

Table 8 Discriminant validity of individual and combined METMT forms METMT form(s) Alog Sensitivity Specificity Positive predictive power Negative predictive power False-positive rate False-negative rate Observed agreement “hit” rate Chance agreement κ Standard error of κ Z of κ ∗

P < .00008. P < .00001.

∗∗

∗

Blog

∗∗

Xlog

∗∗

Ylog

∗∗

Zlog

∗∗

Glog

∗∗

B4log

∗∗

Alog + Blog ∗∗

Xlog ∗∗ − B4log ∗∗

0.66 0.63 0.64 0.65 0.36 0.34 0.64

0.75 0.77 0.77 0.76 0.22 0.25 0.76

0.76 0.58 0.64 0.71 0.42 0.24 0.67

0.71 0.78 0.77 0.73 0.22 0.29 0.75

0.67 0.77 0.75 0.70 0.22 0.33 0.72

0.80 0.60 0.67 0.75 0.40 0.20 0.70

0.73 0.73 0.73 0.73 0.27 0.27 0.73

0.75 0.74 0.74 0.75 0.26 0.25 0.75

0.75 0.73 0.72 0.76 0.27 0.25 0.74

0.50 0.29 0.07 3.96

0.50 0.53 0.07 7.19

0.50 0.34 0.07 4.77

0.50 0.49 0.07 6.76

0.50 0.44 0.07 6.05

0.50 0.40 0.07 5.53

0.50 0.46 0.07 6.30

0.50 0.49 0.07 6.75

0.50 0.48 0.07 6.60

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

117

Table 9 Percentage of unique between-group variance explained by the independent variables Variable

Unique percentage of betweengroup variance explained (%)

Age Gender VIQ PIQ METMT form Alog METMT form Blog METMT form Xlog METMT form Ylog METMT form Zlog METMT form Glog METMT form B4log All variables combined

2.7 0.9 6.3 0.0 0.1 0.9 1.2 0.1 0.2 0.6 0.7 50.3

To compare the relative criterion validity of the original versus newer forms of the METMT, two additional discriminant analyses were conducted: one employing forms A and B as the discriminating variables and the other employing forms X, Y, Z, G, and B4 as the discriminating variables. The results of the analyses are summarized in Table 8. 2.4.4. Analysis 7: decomposition of variance A decomposition of the between-group variance was undertaken to determine the unique percentage of variance explained by each METMTform, age, gender, verbal IQ, and performance IQ. In a series of multiple regression analyses, r 2 was obtained by directly regressing all variables onto group. In subsequent analyses, one variable at a time was, in turn, systematically deleted from the list of independent variables. This procedure produced 11 r 2 s, which were, in turn, subtracted from the r 2 obtained when all predictor variables were entered directly. The difference between the full model r 2 and the r 2 obtained with one variable deleted provided an estimate of the unique between-group variance explained by that particular deleted variable. The results of these analyses are summarized in Table 9.

3. Discussion The present study served three major purposes: (a) to attempt cross-validation of Davis et al.’s (1989) findings, (b) to document the psychometric properties of the METMT, and (c) to provide preliminary normative data for the METMT. In terms of the first goal, the findings of Davis et al. were indeed cross-validated even in spite of this earlier study’s methodological shortcomings. Such successful cross-validation is encouraging in that it reinforces the clinical utility of what had, up to now, been an experimental procedure. In contrast to Davis et al., the present study was able to achieve comparable results using only short, paper-and-pencil tests,

118

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

eliminating the need for the Tactual Performance Test, a bulky and somewhat inconvenient testing apparatus for a school setting. While Davis et al. (1989) did achieve somewhat higher diagnostic “hit” rates, it is unclear whether these higher rates are attributable to the use of the Tactual Performance Test or the failure to use jackknifed procedures. Additionally, because discriminant analysis can capitalize on specious correlations, particularly when sample sizes are small, some shrinkage of “hit” rates upon at-tempted cross-validation is to be expected and, indeed, seems to be the rule rather than the exception. Finally, sampling differences or methodological differences between the two studies may have contributed to the differences observed in the rate of correct diagnostic classifications. Thus, it is not surprising that Davis et al. achieved rates that differed from those obtained in the present study. However, though some of the rates vary significantly between the two studies, from a clinical perspective, the two studies are quite similar in demonstrating that learning-disabled students can effectively be differentiated from normal learners on the basis of their performance on paper-and-pencil neuropsychological instruments requiring a relatively brief administration time. This finding provides evidence of the criterion validity of the METMT. The results of the present study provide further evidence of METMT reliability and validity, paving the way for its use as a clinical and research tool. As was shown above, the METMT is internally consistent and has good interrater reliability and adequate to good test-retest reliability. However, the test-retest reliabilities for METMT forms A, X, and Z were somewhat lower than anticipated. The reason for the apparently attenuated test–retest reliability for these forms is unclear, and post hoc inspection of the raw data does not suggest a clear hypothesis. Indeed, explaining the marginally adequate coefficients of stability for these particular METMT forms would provide the basis for an entirely separate study. The present study also provides additional evidence supporting the validity of the METMT. In terms of construct validity, the performances of learning-disabled children and normal learners differed significantly on all METMT forms. Further evidence of construct validity is the fact that each of the newer METMT forms explained unique between-group variance above and beyond that explained by the original forms A and B. Moreover, the results of factor analysis, at least with the combined sample, seem to suggest that all METMT forms measure the same construct, whatever that particular construct might be. This construct also appears to be something other than verbal or performance intelligence. When factor analyses were conducted separately for the learning-disabled and normal learner groups, slightly different results were obtained. Among normal learners, the time taken to complete METMT forms A, Y, and G was positively associated with age and gender, with males completing these tasks more quickly than their female counterparts. In contrast, among learning-disabled children, faster METMT forms B, X, and G completion times were positively associated with higher IQ. The meaning of these differences is not readily apparent. Indeed, it would seem that further study regarding the differences in METMT factor structure between learning-disabled children and normal learners could potentially shed light on the differences in information processing between these groups. The finding that 9- and 10-year-olds tended to be misclassified at a higher rate than their older classmates was unanticipated. However, in retrospect, these results seem intuitive given what is known about developmental neurophysiology. Yakolev (1962) asserted that full functional

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

119

brain maturity, as defined by the completion of myelination, may not occur until the age of 10. It then follows that learning disabilities may not express themselves fully until the age of 10 or thereafter. It may also be that the school curriculum becomes more demanding after grade 3. As a consequence of increased information processing demands, learning disabilities become more apparent. Obviously, the data obtained in the present study do not address these two hypotheses, and they will remain as competing hypotheses pending further study. Although METMT form B4 was intended to test the subjects’ performances in the face of distracting stimuli, neither group differed significantly on their respective completion times of forms B and B4. This was also an unanticipated finding. However, given that evidence of ADD or attention-deficit hyperactivity disorder (ADHD) was an exclusion criterion for this study, the present design probably did not provide a fair test of susceptibility to distraction as form B4 was intended to do. Further study with ADD and ADHD populations are needed to determine whether or not form B4 possesses adequate construct and criterion validity. Forms B and B4 share 46% of their variance (Pearson product–moment correlation = .681), yet they were not found to be collinear in any of the various analyses conducted in the present study. It will be interesting to determine what other constructs contribute to the variance of form B4. The final goal of the present study was to provide preliminary normative data on the METMT.Those data are presented in Table 10. In conclusion, it seems fair to say that the METMT is a reliable and valid instrument for experimental and limited clinical use. The present study suggests that the MTMT is robust to variations in stimulus dimensions, in that such variations, while obviously affecting completion times, do not significantly alter the criterion validity of the test. Indeed, it was found that the criterion validity of the MTMT could actually be enhanced through the concomitant use of the newer forms contained in the METMT. Also, like its adult counterpart, the Expanded TMT (Stanczak et al., 1998), the METMT provides the researcher with an alternative methodology for identifying the cognitive components of successful MTMT performance. The process of systematically varying test stimuli has long been employed, with great success, in cognitive and behavioral research. It is probable that this same methodology can be imported to the realm of neuropsychology to assist investigators and clinicians in more accurately interpreting test results. Table 10 Preliminary METMT normsa METMT form (raw scores) A B X Y Z G B4

Normal learners All

Learning disabled 9–10

11–14

All

9–10

11–14

16.99 (6.14) 19.22 (6.08) 15.87 (5.91) 23.29 (8.65) 22.39 (9.42) 23.74 (8.28) 33.20 (14.87) 44.26 (19.48) 27.68 (7.32) 53.29 (25.86) 60.68 (36.16) 49.60 (18.00) 108.03 (43.85) 137.61 (38.91) 93.24 (38.58) 150.0 (41.09) 153.06 (36.37) 148.47 (43.46) 32.34 (12.70) 38.06 (15.67) 29.48 (9.86) 50.47 (28.40) 46.81 (18.75) 52.31 (32.14) 33.33 (15.43) 41.64 (15.60) 29.18 (13.66) 48.72 (21.34) 51.97 (25.72) 47.10 (18.81) 127.69 (42.77) 150.58 (38.07) 116.24 (40.58) 161.91 (33.06) 157.94 (40.12) 163.90 (29.06) 33.55 (11.11) 40.58 (13.29) 30.03 (7.85) 48.44 (17.82) 56.06 (24.23) 44.63 (12.09) a

Cell content = mean (standard deviation) time to completion (s).

120

D.E. Stanczak, G. Triplett / Archives of Clinical Neuropsychology 18 (2003) 107–120

References Alvarez, R. R. (1962). Comparison of depressive and brain-injured subjects on the Trail Making Test. Perceptual and Motor Skills, 14, 91–96. Armitage, S. G. (1946). An analysis of certain psychological tests used for the evaluation of brain injury. Psychological Monographs, 60, 11–48. Connors, C. K. (1995). Conners’ Continuous Performance Test (version 3). North Tonawanda, NY: Multi-Health System. Cosgrove, J., & Newell, T. G. (1991). Recovery of neuropsychological functions during reductions in use of phencyclidine. Journal of Clinical Psychology, 47, 159–169. Davis, R. D., Adams, R. R., Gates, D. O., & Cheramie, G. M. (1989). Screening for learning disabilities: A neuropsychological approach. Journal of Clinical Psychology, 45, 423–429. Fossum, B., Holmberg, H., & Reinvang, I. (1992). Spatial and symbolic factors in performance on the Trail Making Test. Neuropsychology, 6, 71–75. Kaufman, A. S., & Kaufman, N. L. (1990). Manual for the Kaufman Brief Intelligence Test. Circle Pines, MN: American Guidance Service. Reitan, R. M. (1955). The relation of the Trail Making Test to organic brain damage. Journal of Consulting Psychology, 19, 393–394. Reitan, R. M. (1958). Validity of the Trail Making Test as an indicator of organic brain damage. Perceptual and Motor Skills, 8, 271–276. Reitan, R. M. (1971). Trail Making Test results for normal and brain-damaged children. Perceptual and Motor Skills, 33, 575–581. Reitan, R. M., & Herring, S. (1982). A short screening device for identification of cerebral dysfunction in children. Journal of Clinical Psychology, 45, 643–650. Reitan, R. M., & Wolfson, D. (1985). The Halstead–Reitan neuropsychological test battery: theory and clinical interpretation. South Tucson: Neuropsychology Press. Reitan, R. M., & Wolfson, D. (1993). The Halstead–Reitan neuropsychological test battery: theory and clinical interpretation (2nd ed.). Tucson, AZ: Neuropsychology Press. Stanczak, D. E., Lynch, M. D., McNeil, C. K., & Brown, B. (1998). The Expanded Trail Making Test: Rationale, development, and psychometric properties. Archives of Clinical Neuropsychology, 13, 473–487. Stanczak, D. E., Stanczak, E. M., & Awadalla, A. W. (2001). Development and initial validation of an Arabic version of the Expanded Trail Making Test: Implications for cross-cultural assessment. Archives of Clinical Neuropsychology, 16, 141–149. Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics (3rd ed.). New York: Harper Collins. Yakolev, P. I. (1962). Morphological criteria of growth and maturation of the nervous system in man. Association for Research in Nervous and Mental Disease, 39, 3–46.