Investigating the factor structure of the Child Behavior Checklist in a large sample of children with autism spectrum disorder

Research in Autism Spectrum Disorders 40 (2017) 24–40

Contents lists available at ScienceDirect

Research in Autism Spectrum Disorders journal homepage: www.elsevier.com/locate/rasd

Investigating the factor structure of the Child Behavior Checklist in a large sample of children with autism spectrum disorder

MARK

⁎

Kristen Medeiros ,a, Micah O. Mazurekb, Stephen Kannea a

Department of Health Psychology, University of Missouri, Thompson Center for Autism & Neurodevelopmental Disorders, 205 Portland Street, Columbia, MO 65211, United States Curry School of Education, University of Virginia, 405 Emmet Street South, Charlottesville, VA 22903, United States

b

AR TI CLE I NF O

AB S T R A CT

Number of reviews completed is 2

Background: Autism spectrum disorder (ASD) is characterized by core impairments in social communication and restricted and repetitive behaviors, with high rates of co-occurring emotional and behavioral problems. The Child Behavior Checklist (CBCL) is one of the most widely accepted rating scales used to assess childhood emotional and behavioral problems, and it has been used in many large-scale studies of children with ASD. However, it is not known whether the previously established factor model sufficiently accounts for symptom patterns in children with ASD. Method: We conducted two Confirmatory Factor Analyses for each of the two versions of the CBCL (ages 1.5–5 and ages 6–18) in a large sample of children with ASD: one on the established measurement model and one on the structural model produced from an Exploratory Factor Analyses. We used several model fit indices to determine the best fitting model. Results: We found that the established CBCL factor structure was the best fitting model for young children with ASD, but not for older children with ASD. Conclusions: Models produced from Exploratory Factor Analyses provided evidence that the underlying behavioral constructs measured by the CBCL for ages 6–18 are different in children with ASD than among the typically developing sample. The results of this study have implications regarding how the CBCL should be interpreted in children with ASD.

Keywords: Child behavior checklist Autism spectrum disorder Construct validity Factor analysis

Autism spectrum disorder (ASD) is characterized by core impairments in social communication and interactions and by restricted and repetitive behaviors (American Psychiatric Association, 2013). In addition, rates of co-occurring emotional and behavioral problems are very high among children with ASD (Simonoff et al., 2008). The prevalence of comorbid psychiatric disorders has been estimated to range widely from 27% to 95% (Joshi et al., 2010; Rosenberg et al., 2011) among children with ASD and 54% to 80% among adults with ASD (Buck et al., 2014; Croen et al., 2015; Ghaziuddin & Zafar, 2008). Among children with ASD, anxiety disorders are the most common cooccurring problem (Caamaño et al., 2013; Gjevik et al., 2011; Salazar et al., 2015); whereas, among adolescents and adults with ASD, major depressive disorder is the most common co-occurring problem (Greenlee, Mosley, Shui, Veenstra-VanderWeele, & Gotham, 2016; Mayes, Calhoun, Murray, & Zahid, 2011). In addition, difficulties with behavior and attention regulation, such as attention-deficit/hyperactivity disorder, behavior problems, and oppositional or defiant behavior frequently co-occur in ASD (Brereton et al., 2006; Eisenhower et al., 2005; Frazier et al., 2001; Gadow et al., 2005; Mayes et al., 2012; Salazar et al., 2015). Thus, it is important that clinicians and researchers routinely screen for potential emotional and behavioral symptoms across a range of areas. Rating scales and behavioral checklists represent a useful and efficient way of gathering information about a

⁎

Corresponding author. E-mail addresses: [email protected] (K. Medeiros), [email protected] (M.O. Mazurek), [email protected] (S. Kanne).

http://dx.doi.org/10.1016/j.rasd.2017.06.001 Received 3 March 2017; Received in revised form 31 May 2017; Accepted 5 June 2017 1750-9467/ © 2017 Elsevier Ltd. All rights reserved.

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

child’s current symptoms, and are useful for both screening and to inform diagnosis (Lempp, de Lange, Radeloff, & Bachmann, 2012). The Child Behavior Checklist (CBCL) (Achenbach & Rescorla, 2001) is one of the most widely accepted rating scales used to assess childhood emotional and behavioral problems. The CBCL is a parent-report measure that assesses observed functioning across both internalizing and externalizing domains of symptomatology. The CBCL provides scores along both broad-band scales (i.e., Internalizing and Externalizing), and narrow-band syndrome scales, which were empirically derived and developed through factor analysis. The normative group included a very large general population sample of children and adolescents (Achenbach & Rescorla, 2001). The CBCL has been widely used an increasing number of research studies of children with ASD. Specifically, it has been used to assess types and correlates of behavioral and emotional problems in children with ASD (Gonzalez & Stern, 2016; Hirata et al., 2016; Ross & Cuskelly, 2006; Samson, Hardan, Lee, Phillips, & Gross, 2015; Son et al., 2015; Xu, Neece, & Parker, 2014; Wade, Cox, Reeve, & Hull, 2014). It has and has also been used as a measure of treatment outcome across a range of interventions. In psychometric research, the CBCL has been involved with verifying the convergent and divergent validity of several measures in ASD, including the Aberrant Behavior Checklist (Kaat, Lecavalier, & Aman, 2014), the Modified Checklist for Autism in Toddlers (Kim et al., 2016), the Social Responsiveness Scale (Cholemkery, Kitzerow, Rohrmann, & Freitag, 2014), the Autism Spectrum Disorders − Comorbidity for Children (Rieske et al., 2013), the Pediatric Anxiety Rating Scale (Storch et al., 2012), the Behavioral Assessment System for Children (Haas, Brown, Brady, & Johnson, 2012), and the Repetitive Behavior Scale-Revised (Mirenda et al., 2010). The developers of the CBCL have suggested that, in addition to measuring emotional, behavioral, and social problems in children, the CBCL can be used as a screening tool for ASD in clinical settings (Achenbach & Rescorla, 2013; p. 31–39). In practice, several studies have found that the usefulness of the CBCL as a sensitive and specific screener comes more from certain items or subscale scores than from clinically relevant internalizing, externalizing, or total scores. With the CBCL version for younger children (ages 1.5–5), the Withdrawn syndrome scale has been found to have high accuracy in differentiating preschoolers with ASD from preschoolers with other disorders (Muratori et al., 2011; Narzisi et al., 2013; Rescorla, Kim, & Oh, 2015). With the CBCL for older children (ages 6–18), the Withdrawn/Depressed, Social Problems, and Thought Problems syndrome scales have been successful in distinguishing school-age children with ASD from other disorders (Biederman et al., 2010; Ooi, Rescorla, Ang, Woo, & Fung, 2010). At an item-level, one study found ten items were most predictive of ASD: acts young, obsessions, daydreams, prefers to be alone, clumsy, repeats acts, speech problems, stares, behaves strangely, and withdrawn (So, Greaves-Lord, van der Ende, Verhulst, Rescorla, & de Nijs, 2013). Another study found nine items were most predictive: acts young, does not get along with other kids, fears specific animals and situations, prefers to be alone, nervous, repeats acts, speech problems, behaves strangely, and withdrawn (Ooi et al., 2010). Yet another study found just seven items distinguished children with ASD from children with other disorders: avoiding eye contact, not answering when people talk to him/her, not getting along with other kids, lack of guilt after misbehaving, little affection, little interest, and being uncooperative (Rescorla et al., 2015). By contrast, some research has shown lower accuracy of CBCL profiles in identifying children with ASD in the context of children with other clinical problems (Myers et al., 2014; Ooi et al., 2014; Rescorla et al., 2015; So et al., 2013). With the low discriminative accuracy, the use of CBCL profiles for ASD-specific screening may result in a large amount of misclassifications (Havdahl, von Tetzchner, Huerta, Lord, & Bishop, 2016). The CBCL is also very commonly used when studying comorbid psychopathologies in ASD, as seen in several literature reviews (e.g., Kaat & Lecavalier, 2013; Mannion & Leader, 2013; Matson & Cervantes, 2014; Mazzone, Ruta, & Reale, 2012). Although the CBCL has been widely used to examine co-occurring internalizing and externalizing problems among children with ASD (e.g., Hartley, Sikora, & McCoy, 2008; Mazurek & Kanne, 2010; Vasa et al., 2013), it is not known whether the established factor model that led to the published domain scores adequately accounts for symptom patterns among samples of children with ASD. There are quite a few reasons to suspect a different factor structure for this population. First, ASD involves different types of emotional and behavioral problems than are seen in neurotypical children (e.g., stereotyped or repetitive behaviors and self-injury). Second, emotional and behavioral problems might manifest differently in youth with ASD. For example, the expression of internalizing problems like anxiety and depression may be difficult to interpret, particularly for nonverbal children, and may resemble withdrawn behaviors or lack of social interaction that are also core features of ASD. Third, ASD symptoms may moderate the display of emotional and behavioral problems. For example, restricted interests may mask attention problems or social communication deficits may mask somatic complaints or intensify perceived social problems. By extension, it is not known whether the behavioral constructs thought to underlie the published domain scores are the same in individuals with ASD. If not, then the use and interpretation of the CBCL (and possibly similar questionnaires) may be significantly different with regard to internalizing and externalizing symptom patterns. Exploratory Factor Analysis (EFA) is useful in initially determining underlying patterns within scales; however, EFA methods are generally assumed to be exploratory and without pre-specified theoretical or empirical expectations. In contrast, confirmatory factor analysis (CFA) can be used to confirm a priori hypotheses regarding the structure of data. In developing new measures, CFA can be particularly useful in establishing construct validity. Specifically, models can be tested to examine the extent to which the underlying structure of the scale is consistent with theoretical expectations regarding number and nature of underlying constructs (Floyd & Widaman, 1995). The factor structure of the CBCL has been tested and validated in a range of samples (Dedrick, Greenbaum, Friedman, Wetherington, & Knoff, 1997; Greenbaum & Dedrick, 1998). Construct validity of the CBCL syndrome scales has been established using CFA among a wide range of cross-cultural samples (De Groot, Koot, & Verhulst, 1994; Dedrick, Tan, & Marfo, 2008; Ivanova et al., 2010). However, its psychometric properties have yet to be examined among large samples of children with ASD. To our knowledge, only two studies have specifically examined the factor structure of the CBCL among children with ASD. In the first study, conducted in 2009, Pandolfi and colleagues examined the factor structure of the preschool version of the CBCL among a sample of 128 children with ASD. Given the small sample size, the authors were not able to examine the entire hierarchical model. 25

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Instead, each of the seven syndrome scales were analyzed separately using robust diagonally weighted least squares estimation to determine whether there was evidence for a single latent factor using only those items. Root Mean Square Error of Approximation (RMSEA) was used to examine model fit in each case. The results supported the hypothesis that each scale (with the exception of Sleep Problems) was accounted for by a single factor (Pandolfi, Magyar, & Dill, 2009). A follow-up study using similar methods was conducted to examine the factor structure of the 6–18 version of the CBCL among a sample of 122 children with ASD (Pandolfi, Magyar, & Dill, 2011). The results of this study provided support for unidimensionality of most scales, with the exception of Thought Problems. However, the small sample sizes used in both studies prevented examination of the entire CBCL model; thus, further research with larger samples is needed to replicate these findings. The purpose of the current study is to examine the factor structure of the CBCL among a large, well-characterized sample of children with ASD, and to examine whether the existing factor model is appropriate for use in this population. We hope that the results of this study will inform the use of the CBCL among children with ASD and will help to guide interpretation of particular patterns of results in future research. 1. Methods 1.1. Participants Participants included 623 children with ASD ranging in age from 2 to 17 years old (M = 7.8, SD = 3.8). The sample was largely male (83.9%). Participants were recruited from two academic medical centers located in the Midwest and South Central United States and were participants in either the Simons Simplex Collection or Autism Treatment Network Registry study at each site. Diagnosis was confirmed for both studies using gold-standard diagnostic tools, including the Autism Diagnostic Interview − Revised (Lord, Rutter, & le Couteur, 1994) and/or the Autism Diagnostic Observation Schedule (Lord, DiLavorne, & Risi, 2002). Secondary analysis of item-level CBCL data collected from each site was conducted for the purpose of the current study. For the purposes of this study, raw score data (not norm-referenced T-score data) were used to make use of the total variation in the dataset. Given that the CBCL comprises two versions based on age, 36% of our sample (n = 225) completed the CBCL for Ages 1.5–5 (average age = 4.06, SD = 0.89), and 64% (n = 398) completed the CBCL for Ages 6–18, (average age = 9.92, SD = 3.10). A power analysis conducted using G*Power software revealed that a sample of 35 participants would be sufficient to detect a medium effect size (d = 0.485; derived from averaging effect sizes from Pandolfi et al., 2009, 2011) with a power of 0.85. 1.2. Measures 1.2.1. Intelligence Intelligence (IQ) was assessed using a range of measures across sites, including the Early Years Differential Ability Scales–Second Edition (20.4%) (Elliot, 2007), the School-Age Differential Ability Scales–Second Edition (19.7%) (Elliot, 2007), the Wechsler Abbreviated Scale of Intelligence − Second Edition (11.1%) (Wechsler, 2011), the Wechsler Intelligence Scale for Children − Fourth Edition (1.5%) (Wechsler, 2003), the Wechsler Preschool and Primary Scale of Intelligence − Third Edition (2.5%) (Wechsler, 2002), the Wechsler Preschool and Primary Scale of Intelligence − Fourth Edition (.2%) (Wechsler, 2012), the Stanford Binet Scales of Intelligence − 5th Edition (7.7%) (Roid, 2003), and the Mullen Scales of Early Learning (2.3%) (Mullen, 1995). Fourteen percent of the sample was administered a measure of nonverbal intelligence, the Leiter International Performance Scale − Third Edition, (Roid, Miller, Pomplun, & Koch, 2013); therefore, a Full Scale IQ score was not available for this subsample. IQ testing was not completed for 20.8% of the sample due to difficulties participating or understanding the demands of the task. Full Scale IQ was only used to provide a broad description of cognitive abilities across the sample, and it was not included in the analyses of the CBCL factor structure. Therefore, although IQ data was missing for 51.8% of the younger sample and 24.4% of the older sample, we did not substitute missing values for IQ. 1.2.2. Vineland adaptive behavior scales (VABS) Adaptive behavior was assessed using the VABS (Sparrow, Cicchetti, & Balla, 2005) for 88.5% of the sample. The VABS is a standardized, semi-structured parent-report interview assessing day-to-day adaptive functioning in Communication, Daily Living, and Socialization domains. Research has shown strong reliability and validity of the VABS (Sparrow et al., 2005), as well as clinical utility in individuals with ASD. Adaptive behavior assessment helps to identify areas of assistance and support, allowing interventions to target specific skills (Cicchetti, Carter, & Gray, 2013). Similarly to IQ, the Composite VABS score was only used to provide a broad description of adaptive functioning across the sample, and it was not included in the analyses of the CBCL factor structure. Therefore, although Composite VABS data was missing for 16.4% of the younger sample and 9.4% of the older sample, we did not substitute missing values for the VABS. 1.2.3. Child behavior checklist (CBCL) The CBCL (Achenbach & Rescorla, 2001) is a parent-report questionnaire assessing behavioral and emotional symptoms across a number of domains. Items are rated on a 3-point Likert-type scale (0 = not true, 1 = somewhat or sometimes true, and 2 = very true or often true). Two separate versions are available based on the child’s age, and each version comprises a different number of items and slightly different syndrome and DSM-oriented subscales. Both versions yield Total Scores as well as Internalizing and Externalizing Scale scores. 26

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

The younger version (CBCL for Ages 1.5–5) includes 100 items, and the previously established measurement model as described in the CBCL manual consists of the following Syndrome Scales: Emotionally Reactive, Anxious/Depressed, Somatic Complaints, Withdrawn, Sleep Problems, Attention Problems, and Aggressive Behavior. Item 100, “Please write in any problems the child has that were not listed above,” is an open-ended question that was not analyzed for the purposes of this study. The older version (CBCL for Ages 6–18) includes 113 items, and the previously established measurement model consists of the following Syndrome Scales: Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, RuleBreaking Behavior, and Aggressive Behavior. Two open-ended questions (Item 56 h, “Physical problems without known medical causes- Other”, and Item 113, “Please write in any problems your child has that were not listed above”) were not analyzed for the purposes of this study. Research has shown strong test-retest reliability, inter-rater agreement, and internal consistency of the CBCL (Achenbach, Dumenci, & Rescorla, 2003). The factor structure has been supported with a community sample of youths (Achenbach et al., 2003), and validity and reliability of the syndrome and DSM-focused scales have been supported (Achenbach & Rescorla, 2001). In addition, a large clinical sample showed scale reliability and convergent and discriminative validity (Hallerod et al., 2010; Nakamura, Ebesutani, Bernstein, & Chorpita, 2009). Research has also supported test-retest reliability over a one-year period (Frizzo, Pedrini, de Souza, Bandeira, & Borsa, 2014), and good internal consistency (Albores-Gallo et al., 2007). 1.3. Analysis In order for our factor analysis to produce meaningful results, it was necessary to substitute missing values. We replaced missing values with the mode for that particular CBCL item. This resulted in 136 substitutions (.61% of the data) in the younger sample and 181 substitutions (.39% of the data) in the older sample. 1.3.1. Unidimensional model replications In attempt to replicate the findings by Pandolfi et al. (2009), we performed seven item-level CFAs to determine if the items within each syndrome scale were explained by a single underlying latent factor. 1.3.2. Confirmatory factor analysis and exploratory factor analysis For each version of the CBCL, we analyzed two models using CFA: the previously established factor structure of the measurement model based on the empirically-derived CBCL syndrome scales, and the factor structure based on an EFA of the data. Each CFA included an arbitrary sequence of post hoc adjustments to improve model fit, which we performed in order, based on arbitrary levels chosen prior to beginning data analysis using recommendations from Kenny (2011) and Schermelleh-Engel, Moosbrugger, and Muller (2003). Specifically, we covaried error terms with modification indices over 20 on the same latent variable, we removed four variables that had 5 or more instances of standardized residual covariance values above an absolute value of 2.58, and we removed items with factor loadings below 0.30. Note that since we executed these adjustments in this order, and did not perform them more than once for any model, the final factor loadings for some items may be below 0.30. Step 1. We created a measurement model to test whether the previously established factor structure based on the empiricallyderived CBCL syndrome scales provided good fit to the data in our current ASD sample. The established model for the CBCL for ages 1.5–5 included 7 latent variables: Emotionally Reactive (9 items), Anxious/Depressed (8 items), Somatic Complaints (11 items), Withdrawn (8 items), Sleep Problems (7 items), Aggressive Behavior (19 items), and Attention Problems (5 items). The established model for the CBCL for ages 6–18 included eight latent variables: Aggressive Behavior (18 items), Rule-Breaking Behavior (8 items), Thought Problems (14 items), Anxious/Depressed (12 items), Somatic Complaints (10 items), Social Problems (11 items), Attention Problems (10 items), and Withdrawn/Depressed (8 items). We performed a Confirmatory Factor Analysis (Amos, version 24.0). In this measurement model, all of the latent variables were covaried, all of the error term weights were constrained to 1 as per default, and one regression path weight was constrained to 1 within each latent variable construct to adhere to Amos requirements. Using recommendations put forth by Schermelleh-Engel et al. (2003), we used maximum likelihood estimation and limited the iterations to 500. We performed the sequence of post hoc adjustments to this measurement model to improve model fit. Step 2. We began our exploratory statistics using bottom-up logic. We generated simple correlation matrices to eliminate any variables that were correlating poorly with a majority of the items, since this would result in poor model fit. We performed an EFA on the remaining items using maximum likelihood estimation with an oblique rotation to test the number of factors on which our sample would converge (SPSS version 24.0). We selected EFA because statistical theorists have argued that it is the most preferred method of finding underlying constructs (Williams, Onsman, & Brown, 2010). We selected an oblique non-orthogonal rotation so that our factors were allowed to be correlated (Osborne & Costello, 2009). To determine the number of factors to retain, we considered eigen values, the scree plot, and the percentage of variance explained, since using eigen values alone is the least accurate method for selecting the number of factors (Velicer & Jackson, 1990). To determine which items loaded onto which factors, we examined the factor loadings, wherein we noted complex items that cross-loaded on more than one factor. Any complex item was always placed under the factor to which it loaded highest. Step 4. Using the new set of factors produced by the EFA, we performed a second CFA to serve as our exploratory structural model. Identical to the measurement model, all of the latent variables were covaried, all of the error term weights were constrained to 1, one regression path weight was constrained to 1 within each latent variable construct, and we used maximum likelihood estimation with the iterations limited to 500. We performed the sequence of post hoc adjustments to this structural model to improve model fit. 27

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

1.3.3. Fit statistics Each model was first evaluated using three absolute fit indices: model chi-square (χ2) (wherein non-significant values indicate acceptable model fit), Root Mean Square Residual (RMR; wherein values less than 0.10 indicate acceptable model fit), and RMSEA (wherein values at or below 0.08 indicate acceptable model fit) (Little, 2013; Schermelleh-Engel et al., 2003). We expected model χ2 to show poor fit, since this statistic is sensitive to sample size and model complexity. Once a model was deemed to have good fit by reaching acceptable levels on the RMR and RMSEA, we used a relative fit index to determine the best fitting model: Comparative Fit Index (CFI; wherein larger values indicate better fit relative to other models). 1.3.4. Power analyses Using recommendations from MacCallum et al. (2006), we calculated the power estimates of each SEM model, given the sample size and the degrees of freedom. The power estimates included values for not-close fit, close fit, and exact fit. 2. Results 2.1. Sample characteristics Tables 1A and 1B display the comparison of age, gender, IQ, VABS, and missing CBCL data across the three sites. Given the unequal group sizes, any significant differences were not able to be confirmed. Thus, we felt justified in including all participants in two groups based on the version of the CBCL they completed. Participants ranged in full scale IQ from 20 to 141 (M = 84.01, SD = 23.94). The two age groups in our study did not differ significantly in IQ [t (405) = −1.724, p = 0.085]. In terms of adaptive skills, participants scored between one and two standard deviations below the mean (Communication: M = 73.51, SD = 15.096; Daily Living Skills: M = 75.59, SD = 14.49; Socialization: M = 69.40, SD = 13.13). The two age groups in our study did not differ significantly in Communication skills [t (547) = 1.038, p = 0.300] or Daily Living Skills [t (5470 = −0.583, p = 0.560]; however, younger children had significantly better Socialization skills than older children [t (547) = 3.059, p = 0.002]. 2.2. CBCL for ages 1.5–5 2.2.1. Unidimensional model replications − results for syndrome scales With the exception of Attention Problems, the RMSEA and CFI indicated that the syndrome scales were not unidimensional, meaning that the items were not explained by one underlying factor (see Table 2). 2.2.2. Established measurement model The previously established measurement model included 7 latent variables: Emotionally Reactive, Anxious/Depressed, Somatic Complaints, Withdrawn, Sleep Problems, Attention Problems, and Aggressive Behavior. We covaried error terms that had modification indices over 20 and were within the same latent variable construct (i.e., item 8 with 16; 15 with 20; 21 with 92; 53 with 35 Table 1A Comparison of group characteristics by site (ATN, Baylor, and Simons) for ages 1.5–5. ATN (n = 128) Age**†

M = 3.71 SD = 0.97 Gender 81.3% male Full IQ R: 47–128 M = 78.36 S = 19.3 Vineland Adaptive Behavior Scales Communication**† R: 42–118 M = 71.13 SD = 16.22 Daily Living R: 24–113 M = 74.94 SD = 14.71 Socialization R: 23–112 M = 72.12 SD = 12.44 Composite*† R: 44–107 M = 71.55 SD = 11.76 Missing CBCL values 116

Baylor (n = 57)

Simons (n = 42)

Test Statistic

p

M = 4.47 SD = 0.50 80.7% male R: 61–121 M = 87.11 S = 16.12

M = 4.64 SD = 0.53 88.1% male R: 22–130 M = 78.65 S = 26.93

F (2, 224) = 30.74., p < 0.001

ATN < Baylor & Simons

R: 40–100 M = 75.74 SD = 16.21 R: 48–93 M = 73.0 SD = 11.25 R: 49–92 M = 68.74 SD = 9.79 R: 51–88 M = 71.42 SD = 10.49 5

R: 42–114 M = 81.95 SD = 17.01 R: 51–107 M = 77.40 SD = 13.92 R: 49–114 M = 73.40 SD = 14.81 R: 48–107 M = 76.95 SD = 13.90 15

χ2(2) = 1.17, p = 0.558 F (2, 106) = 1.59, p = 0.209

F (2, 188) = 6.80, p = 0.001

F (2, 188) = 1.02, p = 0.364

F (2, 188) = 1.51, p = 0.224

F (2, 186) = 3.34, p = 0.038

M = mean; SD = standard deviation; R = range. *p < .05. **p < .001. † The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

28

ATN < Simons

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Table 1B Comparison of group characteristics by site (ATN, Baylor, and Simons) for ages 6–18.

Age*† Gender Full IQ**†

Vineland Adaptive Behavior Scales Communication*†

Daily Living

Socialization*†

Composite

Missing CBCL values

ATN (n = 135)

Baylor (n = 133)

Simons (n = 131)

Test Statistic

p

M = 10.05 SD = 3.14 85.9% male R: 47–136 M = 91.51 SD = 20.49

M = 9.29 SD = 2.86 82.0% male R: 40–139 M = 91.40 SD = 18.35

M = 10.39 SD = 3.23 86.3% male R: 20–141 M = 77.32 SD = 27.99

F (2, 396) = 4.4, p = 0.013

Baylor < Simons

χ (2) = 1.17, p = 0.55 F (2, 295) = 13.15, p < 0.001

Simons < ATN & Baylor

R: 28–129 M = 70.68 SD = 14.55 R: 28–121 M = 75.21 SD = 16.29 R: 38–107 M = 66.64 SD = 13.52 R: 30–111 M = 69.37 SD = 13.06 134

R: 33–108 M = 72.81 SD = 14.03 R: 30–134 M = 76.0 SD = 15.32 R: 38–112 M = 66.92 SD = 13.52 R: 31–103 M = 70.36 SD = 12.42 2

R: 47–125 M = 75.31 SD = 13.33 R: 40–109 M = 76.31 SD = 12.93 R: 40–98 M = 70.56 SD = 12.23 R: 41–99 M = 72.39 SD = 11.07 45

F (2, 355) = 3.43, p = 0.033

ATN < Baylor & Simons

2

F (2, 355) = 0.18, p = 0.838

F (2, 355) = 3.45, p = 0.033

F (2, 354) = 2.0, p = 0.137

M = mean; SD = standard deviation; R = range. * p < .05. ** p < .001. † The group sizes are unequal. The harmonic mean of the group sizes is used. Type I error levels are not guaranteed.

and 40; 79 with 82, 67 with 70). We removed four variables that had more than five instances of standardized residual covariances over an absolute value of 2.58 (i.e., items 7, 86, 95, and 96). We removed items with factor loadings below 0.30 (i.e., items 1, 2, 19, 24, 37, 45, 56, and 93). The final measurement model included 7 factors (see Table S1 for the items and their standardized regression weights). Although the χ2 was significant, this established 7-factor measurement model had acceptable model fit, with RMR under 0.10 and RMSEA under 0.08 (see Table 3).

2.2.3. Exploratory structural model A simple correlation matrix revealed 22 items with either low correlations (i.e., not significantly correlated with more than 50% of the variables: items 1, 12, 19, 39, 45, and 68) or poor distribution (i.e., one response category with less than 5% frequency: items Table 2 CFA Model Fit for Syndrome Scales CBCL for Ages 1.5–5 Model

χ2

df

p

RMR

RMSEA

LO 90

HI 90

CFI

Emotionally Reactive Anxious/Depressed Somatic Complaints Withdrawn Sleep Problems Attention Problems* Aggressive Behavior

132.19 84.33 178.70 77.37 81.81 4.21 475.23

27 20 44 20 14 5 152

0.000 0.000 0.000 0.000 0.000 0.519 0.000

0.037 0.039 0.031 0.031 0.035 0.015 0.035

0.132 0.120 0.117 0.113 0.147 0.000 0.097

0.110 0.094 0.099 0.087 0.117 0.000 0.088

0.155 0.147 0.135 0.140 0.179 0.085 0.107

0.811 0.772 0.646 0.836 0.834 1.0 0.830

CBCL for Ages 6–18 Model

χ2

df

p

RMR

RMSEA

LO 90

HI 90

CFI

Anxious/Depressed Withdrawn/Depressed Somatic Complaints* Social Problems Thought Problems Attention Problems Rule-Breaking Behavior Aggressive Behavior

341.6 79.82 125.54 170.40 467.08 175.67 170.69 882.65

65 20 44 44 89 34 20 135

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

0.032 0.025 0.014 0.031 0.034 0.037 0.038 0.033

0.104 0.087 0.068 0.085 0.103 0.102 0.138 0.118

0.093 0.067 0.054 0.072 0.094 0.088 0.119 0.111

0.114 0.107 0.083 0.099 0.113 0.118 0.157 0.126

0.798 0.847 0.901 0.843 0.506 0.839 0.773 0.716

*RMSEA and CFI indicate good model fit.

29

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Table 3 Model Fit for the CBCL for Ages 1.5-5. Model

χ2

df

Δχ2

Δdf

p

RMR

RMSEA

LO 90

HI 90

CFI

†Established 7-factor Exploratory 4-factor

2323.34* 1444.61*

1402 685

878.73

717

< 0.001

0.033 0.044

0.054 0.070

0.050 0.065

0.058 0.075

0.809 0.773

Established 7-factor: The previously established measurement model as described in the CBCL manual, with post hoc modifications. Exploratory 4-factor: The structural model produced from the exploratory factor analysis, with post-hoc modifications. †best fitting model. *p < .05. Indications of acceptable model fit: Root Mean Square Residual (RMR): values < .10. Root Mean Square Error of Approximation (RMSEA): values ≤ .08. Indication of best fitting model: Comparative Fit Index (CFI): highest value.

14, 26, 28, 41, 43, 48, 49, 50, 51, 52, 57, 78, 83, 90, 93, and 99) to be removed. We performed an EFA on the remaining 77 items. Retaining all eigen values over 1 would have resulted in 21 factors. Using the scree plot, a four-factor solution explained 34.55% of the variance (See Table 4 for the items and unstandardized factor loadings from the EFA and Fig. S1 for the scree plot). We performed a CFA using this structural model of four latent variables. We covaried error terms on the same latent variable with modification indices over 20 (i.e., item 7 with 21; 10 with 37; 34 with 56; 70 with 30, 67, 71, and 96; 30 with 96; 15 with 20; 17 with 18; and 72 with 95), removed items with more than 5 instances of standardized residual covariances over an absolute value of 2.58 (i.e., items 62, 73, and 77), and removed items with factor loadings below 0.30 (i.e., items 23, 33, 55, 65, 75, and 89). The final model included 4 factors, which we labeled as Accident-Prone, Social Difficulties, Challenging Behavior, and Emotion Regulation (See Table S1 for the items and standardized regression weights). Although the model χ2 was significant, this exploratory, 4-factor structural model had acceptable model fit, with RMR under 0.10 and RMSEA under 0.08 (see Table 3). Comparing the established 7-factor measurement model and the exploratory 4-factor structural model, the measurement model had a higher CFI, so we determined the established measurement model to be the better fitting model (see Table 5 for a comparison of the items and factors on the established measurement model and the exploratory structural model). 2.2.4. Power analyses Estimates of power for the established and exploratory models ranged from 0.87 to 0.96, given a sample size of 225 and degrees of freedom over 100 (MacCallum, Browne, & Cai, 2006). 2.3. CBCL for ages 6–18 2.3.1. Unidimensional model replications − results for syndrome scales With the exception of Somatic Complaints, the RMSEA and CFI indicated that the syndrome scales were not unidimensional, meaning that the items were not explained by one underlying factor (see Table 2). 2.3.2. Established measurement model We removed three items about alcohol, cigarettes, and drugs from the dataset because they did not have any variance (i.e., all parents answered Not True). We also removed nine items that had poor distribution (i.e., one category was endorsed less than 5 times in the whole sample: items 40, 51, 67, 72, 73, 91, 96, 101, and 106). The previously established measurement model included 8 latent variables: Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behavior, and Aggressive Behavior. We covaried 8 error terms that had a modification index over 20 and were within the same latent variable construct (i.e., item 3 with 86; 48 with 27 and 38; 33 with 35; 36 with 62; 45 with 50; 75 with 111; and 76 with 100). We removed 25 items that had more than 5 instances of standardized residual covariances above an absolute value of 2.58 (i.e., items 10, 11, 13, 16, 17, 19, 20, 21, 22, 23, 26, 28, 39, 43, 47, 57, 66, 68, 79, 80, 82, 87, 88, 89, and 94). We removed nine items with factor loadings below 0.30 (i.e., items 1, 18, 29, 42, 59, 60, 63, 65, and 70). During these modifications, the Rule-Breaking Behavior construct was reduced to only two items, and was therefore removed from the model (See Table S2 for the items and standardized regression weights). Despite the significant χ2 value, this previously established, measurement model had good model fit, with RMR below 0.10 and RMSEA below 0.08 (see Table 6). 2.3.3. Exploratory structural model We removed five items that did not significantly correlate with at least half of the other items (i.e., items 29, 32, 42, 75, and 111). We performed an EFA on the remaining 99 items. Retaining all components with Eigen values greater than 1 would have resulted in 27 factors. Using the scree plot, we retained 3 factors, which accounted for 26.4% of the variance (See Table 7 for the items and unstandardized factor loadings of the EFA, and see Fig. S2 for the scree plot). We did have complex variables, meaning that several variables had loadings on more than one factor. In all of these cases, we placed the item under a construct using the highest factor loading. Using these three latent variables, we performed a CFA. We covaried error terms on the same latent construct with 30

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Table 4 Unstandardized Factor Loadings for the Exploratory Factor Analysis on the CBCL Subscales for Ages 1.5–5 CBCL Subscales for Ages. Item

Clumsy Accident-prone Underactive Chews inedibles Nervous Demanding Little affect Unresponsive to affect Little interests Withdrawn Doesn’t answer Wants attention Feelings hurt Selfish No guilt Doesn’t get along Jealous Destroys own things Destroy others’ things Disobedient Picks at body Hurts others Get into things Stares Can’t be punished Hyperactive Smears BMs Hits Rapid emotions Frustrated High-strung Inactive Defiant Rocks body Upset by new Disturbed by change Upset by separation Shy Little fear Wanders Resists toileting Tantrums Wants things neat Mood swings Avoids eyes Stubborn Clingy Loud Plays with sex parts

Factor 1

2

1.052 0.611 0.371 0.175 0.164 0.134

−0.101

3

4

0.166 0.152

1.060 0.614 0.312 0.219 0.157 −0.155 −0.155 0.151 0.140 0.136 −0.114

0.112

0.107

0.140 0.151

0.125 0.116

−0.107 0.147

0.101

1.025 0.824 0.217 0.203 0.184 0.179 −0.170 0.168 0.164 0.148 .128 0.127 0.126 0.111 0.110 0.108 0.106 0.153

0.118 0.102 0.112

0.101

0.148 0.110

0.105 0.947 0.313 0.240 0.240 0.234 0.194 0.146 0.139 0.137 0.135 0.133 0.125 0.124 0.112 −0.102

modification indices over 20 (i.e., item 13 with 8, 90, and 97; 15 with 16; 20 with 21; 22 with 23 and 37; 31 with 52; 57 with 37; 69 with 81), removed items with more than 5 instances of standardized residual covariances over an absolute value of 2.58 (i.e., items 3, 12, 18, 35, 54, 68, 71, 79, 84, and 88), and we removed variables with factor loadings below 0.30 (i.e., items 1, 17, 47, 49, 56e, 65, 66, 70, and 80). The final model produced by the EFA included three factors, which we labeled Conduct Problems, Attention/ Behavior Regulation, and Social-Emotional Problems (See Table S2 for the items and standardized regression weights). This exploratory structural model of 3 latent variables, had good model fit, with RMR below 0.10 and RMSEA below 0.08 (see Table 6). In comparing the established 8-factor measurement model with the exploratory 3-factor structural model, the structural model had a higher CFI, so we determined the structural model to be the better fitting model (see Table 8 for a comparison of the items and factors on the established measurement model and the exploratory structural model). 2.3.4. Power analyses The power estimates for the established and exploratory models were 1.0, given a sample size of 398 and degrees of freedom over 100 (MacCallum et al., 2006). 31

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Table 5 Comparison of the Factors between the Established Measurement Model and the Exploratory Structural Model for the CBCL for ages 1.5–5. Item

Measurement Model Factor

Structural Model Factor

Accident-prone Acts young Afraid of new Angry Attacks Avoids eyes Behaves strangely Body aches Can’t be punished Can’t sleep alone Chews inedibles Clingy Clumsy Constipated Cries Cruel Defiant Demanding Destroy others’ things Destroys own things Diarrhea Disobedient Disturbed by change Disturbed sleep Doesn’t eat Doesn’t get along Doesn’t leave home Eats inedibles Eats poorly Eye problems Fearful Fears Feelings hurt Fights Frustrated Get into things Headaches Helpless High-strung Hits Holds breath Hurts others Hyperactive Impatient Inactive Jealous Little affect Little fear Little interests Loud Mood swings Nausea Nervous Nightmares No fun No guilt Nonresponsive Other problems Overeats Overtired Painful BMs Panicky Picks at body Poor concentration Plays with sex parts Quickly shifts Rapid emotions Resists bedtime

Other Problems Withdrawn Other Problems Aggressive Behavior Aggressive Behavior Withdrawn Other Problems Somatic Complaints Aggressive Behavior Sleep Problems Other Problems Anxious/Depressed Attention Problems Somatic Complaints Other Problems Other Problems Aggressive Behavior Aggressive Behavior Aggressive Behavior Other Problems Somatic Complaints Aggressive Behavior Emotionally Reactive Sleep Problems Other Problems Other Problems Other Problems Other Problems Somatic Complaints Other Problems Anxious/Depressed Other Problems Anxious/Depressed Aggressive Behavior Aggressive Behavior Other Problems Somatic Complaints Other Problems Anxious/Depressed Aggressive Behavior Other Problems Aggressive Behavior Attention Problems Aggressive Behavior Withdrawn Other Problems Withdrawn Other Problems Withdrawn Other Problems Emotionally Reactive Somatic Complaints Emotionally Reactive Sleep Problems Other Problems Aggressive Behavior Withdrawn Other Problems Other Problems Other Problems Somatic Complaints Emotionally Reactive Other Problems Attention Problems Other Problems Attention Problems Emotionally Reactive Sleep Problems

Accident-Prone – – – – – – – Challenging Behavior – Accident-Prone Emotion Regulation Accident-Prone – – – Challenging Behavior Accident-Prone Challenging Behavior Challenging Behavior – Challenging Behavior Emotion Regulation – – Social Difficulties – – – – – – – – Challenging Behavior Challenging Behavior – – – Challenging Behavior – Challenging Behavior Challenging Behavior – – Social Difficulties Social Difficulties Emotion Regulation Social Difficulties Emotion Regulation Emotion Regulation – Accident-Prone – – Social Difficulties – – – – – – Challenging Behavior – – – Challenging Behavior – (continued on next page)

32

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Table 5 (continued) Item

Measurement Model Factor

Structural Model Factor

Resists toileting Rocks body Sad Screams Self-conscious Selfish Shy Skin problems Sleep problems Sleeps less Smears BMs Speech problems Stares Stomach-aches Stubborn Sulks Talks in sleep Tantrums Uncooperative Underactive Unhappy Unresponsive to affect Upset by new Upset by separation Vomiting Wakes often Wanders Wants attention Wants things neat Whines Withdrawn Worries

Other Problems Other Problems Anxious/Depressed Aggressive Behavior Anxious/Depressed Aggressive Behavior Other Problems Other Problems Sleep Problems Sleep Problems Other Problems Other Problems Other Problems Somatic Complaints Aggressive Behavior Emotionally Reactive Sleep Problems Aggressive Behavior Aggressive Behavior Other Problems Anxious/Depressed Withdrawn Emotionally Reactive Anxious/Depressed Somatic Complaints Sleep Problems Attention Problems Aggressive Behavior Somatic Complaints Emotionally Reactive Withdrawn Emotionally Reactive

– Challenging Behavior – – – Social Difficulties – – – – – – – – Emotion Regulation – – Emotion Regulation – – – Social Difficulties Emotion Regulation Emotion Regulation – – Emotion Regulation Social Difficulties Emotion Regulation – Social Difficulties –

3. Discussion This study aimed to test the factor structure of the CBCL in a sample of children and adolescents with ASD. The CBCL was designed to measure general problem behavior and internalizing and externalizing difficulties in children. Each age-based version of the measure is posited to consist of several empirically-derived syndrome scales. However, the extent to which these scales are appropriate for use in children with ASD has not been previously examined in large-scale studies. Therefore, we tested measurement models using CFA to examine the extent to which the underlying structure of each scale was consistent with theoretical expectations regarding number and nature of underlying constructs in a population of children with ASD. We also tested structural models produced from EFA to examine the appropriateness of a new number and arrangement of factors in this clinical population. Our analyses of individual unidimensional syndrome scale models did not replicate the findings of Pandolfi et al. (2009, 2011). Whereas Pandolfi et al. (2009) found that all but Sleep Problems were accounted for by a single underlying construct in younger children; we found that almost none of the syndrome scales were explained by a single factor, with the exception of Attention Problems. Similarly, Pandolfi et al. (2011) found that all but Thought Problems and Aggressive Behavior were accounted for by a singly underlying construct in older children; but we found that almost none of the syndrome scales were explained by a single factor, Table 6 Model Fit for CBCL for Ages 6–18. Model

χ2

df

Δχ2

Δdf

p

RMR

RMSEA

LO 90

HI 90

CFI

Established 8-factor †Exploratory 3-factor

2622.96* 1240.70*

1401 514

1,382.26

887

< 0.001

0.028 0.027

0.047 0.060

0.044 0.055

0.050 0.064

0.800 0.824

Established 8-factor: The previously established measurement model as described in the CBCL manual, with post hoc modifications. Exploratory 3-factor: The structural model produced from the exploratory factor analysis, with post-hoc modifications. †best fitting model. *p < .05. Indications of acceptable model fit: Root Mean Square Residual (RMR): values < .10. Root Mean Square Error of Approximation (RMSEA): values ≤ .08. Indication of best fitting model: Comparative Fit Index (CFI): highest value.

33

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Table 7 Unstandardized Factor Loadings for the Exploratory Factor Analysis on the CBCL Subscales for Ages 6–18. Item

Factor 1

Destroys own things Destroys others’ things Bullies people Disobeys at home Repeats acts Disobeys at school Cruel Breaks rules Doesn't feel guilty Secretive Physically attacks people Tantrums Poor school work Fights people Needs attention Nightmares Steals at home Confused Daydreams Stares blankly Strange ideas Constipated Poor concentration Swears Refuses to talk Threatens people Clingy Clumsy Sees things that aren't there Acts too young for his/her age Stubborn Lonely Feels unloved Feels worthless Others are out to get them Feels guilty Sad Jealous Sulks Self-conscious Afraid of school Argues Overtired without good reason Aches or pains Bragging Hangs around trouble Fears doing wrong Screams a lot Speech problem Behaves strangely Rashes or skin problems Teased Deliberately harms self

2

1.032 0.760 0.196 0.157 0.153 0.142 0.191 0.185 0.178 0.136 0.131 0.113 0.113 0.111 0.110 0.109 0.104

3

0.113

0.104

−1.032 −0.287 −0.264 −0.151 −0.140 −0.137 .129 −0.119 0.117 −0.114 −0.107 −0.106 −0.105 −0.102

0.122 −0.105

−0.123 0.105

0.127 −0.111

.624 0.579 0.556 0.391 0.291 .267 0.229 0.236 0.219 0.199 0.152 0.148 −0.144 0.121 0.136 0.104 −0.104 −0.101 −0.105 −0.124 0.179 0.122

with the exception of Somatic Complaints. These very different findings may bring to light the influence of data-driven results in these analytical approaches. Findings from the current study provide evidence that the factor structure of the CBCL is multidimensional, but that the interrelated domains of problem behavior show a different pattern in older children with ASD than in the general population. The results suggest that data from the sample of younger children with ASD did adequately fit the previously established CBCL 7-factor structure (i.e., Emotionally Reactive, Anxious/Depressed, Somatic Complaints, Withdrawn, Sleep Problems, Attention Problems, and Aggressive Behavior). However, data from the sample of older children with ASD fit the exploratory structural model better than the previously established CBCL 8-factor structure (i.e., Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-Breaking Behavior, and Aggressive Behavior). 34

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Table 8 Comparison of the Factors between the Established Measurement Model and the Exploratory Structural Model for the CBCL for ages 6–18. Item

Measurement Model Factor

Structural Model Factor

Accident-prone Acts young Afraid of school Alcohol Anxious Argues Attacks people Behaves strangely Bites nails Body aches BMs Brags Breaks rules Bullies people Clingy Clumsy Confused Constipated Cries Cruel Daydreams Demanding Destroys others’ things Destroys own things Destruction of property Disobeys at home Disobeys at school Dizzy Doesn't feel guilty Doesn’t finish tasks Doesn’t get along Doesn’t sleep enough Doesn’t talk Drugs Eats poorly Eye problems Fears Fears doing wrong Feels guilty Feels unloved Feels worthless Fights people Gender confusion Hangs around trouble Harms self Has strange ideas Has to be perfect Headaches Hears things High-strung Hoards things Hyperactive Impulsive Inattentive Jealous Lacks energy Lies Little enjoyment Lonely Loud Mood swings Nausea Nervous Nightmares Not liked by others Obsessions Other physical problems Other problems

Social Problems Attention Problems Anxious/Depressed Rule-Breaking Behavior Anxious/Depressed Aggressive Behavior Aggressive Behavior Thought Problems Other Problems Somatic Complaints Other Problems Other Problems Rule Breaking Behavior Aggressive Behavior Social Problems Social Problems Attention Problems Somatic Complaints Anxious/Depressed Other Problems Attention Problems Aggressive Behavior Aggressive Behavior Aggressive Behavior Rule-Breaking Behavior Aggressive Behavior Aggressive Behavior Somatic Complaints Rule-Breaking Behavior Attention Problems Social Problems Thought Problems Withdrawn/Depressed Rule-Breaking Behavior Other Problems Somatic Complaints Anxious/Depressed Anxious/Depressed Anxious/Depressed Anxious/Depressed Anxious/Depressed Aggressive Behavior Other Problems Rule-Breaking Behavior Thought Problems Thought Problems Anxious/Depressed Somatic Complaints Thought Problems Anxious/Depressed Thought Problems Attention Problems Attention Problems Attention Problems Social Problems Withdrawn/Depressed Rule-Breaking Behavior Withdrawn/Depressed Social Problems Aggressive Behavior Aggressive Behavior Somatic Complaints Thought Problems Somatic Complaints Social Problems Thought Problems Other Problems Other Problems

– – Social-Emotional Problems – –

35

Conduct Problems – – Social-Emotional Problems – Social-Emotional Problems Conduct Problems Conduct Problems Attention/Behavior Regulation Attention/Behavior Regulation Attention/Behavior Regulation – – Conduct Problems – Conduct Problems Conduct Problems Conduct Problems – Conduct Problems Conduct Problems – Conduct Problems – – – – – – – – Social-Emotional Problems Social-Emotional Problems Social-Emotional Problems – Conduct Problems – Social-Emotional Problems – – – – – – – – – – Social-Emotional Problems – – – – – – – – – – – – – (continued on next page)

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Table 8 (continued) Item

Measurement Model Factor

Structural Model Factor

Others are out to get Overeats Overtired Overweight Picks at body Poor concentration Poor school work Prefers alone Prefers older kids Prefers younger kids Runs away Sad Screams Secretive Sees things Self-conscious Sets fires Sex parts in public Sex parts too much Sex problems Shows off Shy Skin problems Sleep problems Sleeps too much Speech problems Stares Steals at home Steals outside Stomach aches Strange ideas Stubborn Sucks thumb Suicide Sulks Suspicious Swears Talkative Tantrums Teased Teases Thinks about sex Threatens Tobacco Trouble sleeping Truancy Vomits Wets bed Wets self Whines Withdrawn Worries

Social Problems Other Problems Somatic Complaints Other Problems Thought Problems Attention Problems Attention Problems Withdrawn/Depressed Rule-Breaking Behavior Social Problems Rule-Breaking Behavior Withdrawn/Depressed Aggressive Behavior Withdrawn/Depressed Thought Problems Anxious/Depressed Rule-Breaking Behavior Thought Problems Thought Problems Rule-Breaking Behavior Other Problems Withdrawn/Depressed Somatic Complaints Thought Problems Other Problems Social Problems Attention Problems Rule-Breaking Behavior Rule-Breaking Behavior Somatic Complaints Thought Problems Aggressive Behavior Other Problems Anxious/Depressed Aggressive Behavior Aggressive Behavior Rule-Breaking Behavior Other Problems Aggressive Behavior Social Problems Aggressive Behavior Rule-Breaking Behavior Aggressive Behavior Rule-Breaking Behavior Thought Problems Rule-Breaking Behavior Somatic Complaints Other Problems Other Problems Other Problems Withdrawn/Depressed Anxious/Depressed

Social-Emotional Problems – – – – Attention/Behavior Regulation Conduct Problems – – – – Social-Emotional Problems – Conduct Problems – – – – – – – – – – – – Conduct Problems – – Attention/Behavior Regulation Attention/Behavior Regulation – – – – Attention/Behavior Regulation – Conduct Problems Social-Emotional Problems – – Attention/Behavior Regulation – – – – – – – – –

Our EFAs produced structural models with relatively independent factors that were methodologically substantial and conceptually meaningful, and we determined both structural models to have good fit. It is important to acknowledge that alternative models may exist that fit the data equally well or possibly better. Therefore, the current study does not provide a conclusive answer about the nature of the underlying factor structure of the CBCL for children with ASD. 3.1. Findings for younger children with ASD The results from the current study provide interesting implications for understanding behavioral and emotional symptom presentation in children with ASD. In young children with ASD, our study found evidence for four underlying constructs of behavior (i.e., Challenging Behavior, Emotion Regulation problems, Social Difficulties, and Accident-Prone). Elements of the previously established Aggressive Behavior subscale were actually distributed across all four different factors in children with ASD, with the majority falling under the Challenging Behavior subscale. The Challenging Behavior factor included many items from the typically-developing Aggressive Behavior syndrome scale, but also included self-injurious and stereotyped behaviors, such as picking at their body or 36

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

rocking their head and body. Some of these Challenging Behavior items were originally not included on any CBCL subscale, and instead are listed in the “Other Problems” category for typically developing children. In contrast, several items from the previously established Aggressive Behavior syndrome scale were not retained on any factor (e.g., angry, uncooperative) and several moved to other factors (e.g., temper tantrums, selfish/not sharing). This pattern of results indicates that challenging behaviors in young children with ASD may encompass both overt aggressive/destructive behaviors as well as self-harm and idiosyncratic repetitive behavior. By contrast, behaviors that may be associated with aggression in typically developing children (e.g., temper tantrums, not sharing) appear to be more closely associated with emotion regulation and/or social difficulties in children with ASD. Regarding emotional problems in our younger sample, there were notable item-level discrepancies between the established model and the exploratory model. Some elements of the previously established Emotionally Reactive subscale were spread across several different factors in the current sample. Interestingly, items indicative of external displays of emotion (panic, whining, worrying, and sulking), were not retained in any factor in the ASD sample. Similarly, only two items (i.e., clingy and upset by separation) included in the established Anxious/Depressed syndrome scale were retained in the ASD sample, and they both loaded onto the Emotion Regulation problems factor. The Emotion Regulation problems factor from the exploratory model also included items that seemed more indicative of irritability (e.g., mood swings, temper tantrums, disturbed by changes) rather than exclusively anxiety or depression. Regarding socially-relevant symptom domains, the current results indicate that many of the items represented in the previously established Withdrawn syndrome scale were also represented in a single factor (labeled Social Difficulties). However, this factor in the ASD sample included additional socially-relevant behaviors, such as trouble getting along with others, not sharing, not feeling guilty after punishment, and seeming jealous. This suggests that many behaviors that are either associated with aggression or are extraneous “Other” problem behaviors in typically developing children are more closely related to social impairments in children with ASD. Interestingly, the exploratory factor structure did not support separate domains for Somatic Complaints, Sleep Problems, or Attention Problems in children with ASD. In fact, almost none of the items from these previously established syndrome scales were retained in the structural model. This is somewhat surprising, given the high rates of co-occurring sleep, attention, and gastrointestinal problems in children with ASD (Buie et al., 2010; Frazier et al., 2001; Richdale & Schreck, 2009). However, this does replicate the findings from Pandolfi et al. (2009), wherein the Sleep Problems subscale was not unidimensional in young children with ASD. It may be the case that because these symptoms are so prevalent across the population, they did not emerge as separate factors or constructs. Alternatively, it is possible that language and communication deficits in children with ASD might make it difficult for caregivers to rate some of these symptoms using the current CBCL descriptions. For example, several items on the Somatic Complaints and Sleep Problems subscales expect language functionality, such as complaining about aches and pains, headaches, or stomach-aches and questions about talking during sleep and having nightmares. These challenging social and behavioral problems may exhibit differently in children with ASD, leaving respondents unable to accurately denote their child’s symptoms using the CBCL. 3.2. Findings for older children with ASD For older children with ASD, the current results provided evidence for three underlying constructs of behavior (i.e., SocialEmotional problems, Conduct Problems, and Attention/Behavior Regulation problems). The first factor contained a combination of items from the previously established Anxious/Depressed and Social Problems syndrome scales. Most of these items reflected negative or anxious feelings about perceived slights, failures, or social difficulties, as well as deficits in interpreting social situations. In contrast, more classic symptoms of anxiety or depression (e.g., worrying, feeling worthless, crying) were not represented in the exploratory factor structure. It is possible that these symptoms do not manifest in the same manner among children with ASD. Alternatively, perceiving internally experienced symptoms may be difficult for caregivers of youth with ASD. Regarding behavioral difficulties, there were also notable differences when comparing the established measurement model to the exploratory model. Some items from the previously established Aggressive Behavior syndrome scale were exhibited as either Conduct Problems or Attention/Behavior Regulation problems in youth with ASD. Additionally, the Conduct Problems factor in youth with ASD included many more overt problem behaviors than were accounted for in the established model, such as disobedience and destruction of property, stealing, breaking rules, bullying, and being cruel to animals or other people. This may be consistent with prior evidence of a high risk of aggression among adolescents with ASD (Kanne & Mazurek, 2011). It may also be the case that these types of aggressive behaviors among pre-teens and adolescents are less easily seen as driven by social deficits as they were in younger children with ASD, and more quickly interpreted as potentially serious conduct problems as the child grows bigger and stronger. The Attention/Behavior Regulation construct also included several elements that were not represented in the previously established syndrome scales, such as being confused or in a daze, swearing or saying inappropriate things, being overly clingy and dependent on adults, and having poor concentration. It is possible that attention problems and behavior regulation difficulties manifest differently in youth with ASD than among typically developing youth, due to their underlying social and cognitive differences. Interestingly, items from three established syndrome scales did not appear in the model with children with ASD: Thought Problems, Attention Problems, and Somatic Complaints. This replicates the findings by Pandolfi et al. (2011), wherein the Thought Problems subscale was not unidimensional in older children with ASD. Again, these factors may be expressed differently in children with ASD, may be harder to interpret by caregivers, or may not be perceived as problematic by caregivers who are raising a child with ASD. In addition, the previously established Rule-Breaking Behavior subscale could not be examined due to lack of endorsement of many items among parents of youth with ASD (e.g., vandalism, truancy, drugs, alcohol, or sexual problems). This provides clear evidence that some constructs of problem behavior for typically developing children are not applicable to older children with ASD. 37

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

3.3. Limitations Several items were not included in the EFA or CFA models because there was little variation in the data for each of these items, and they were therefore dropped from the analyses. This may reflect the varying set of behaviors exhibited by children with ASD, as Pandolfi et al. (2009, 2011) experienced the same issue with their samples of children with ASD. Our samples were relatively small, which present potential issues with generalization. However, we were able to perform a CFA on the entire CBCL model in addition to evaluating each syndrome scale separately (building on the research by Pandolfi et al., 2009). One limitation to this study is our analytical approach of performing CFAs to examine structural models derived using EFAs on same data. This method may potentially inflate fit statistics; however, with our small sample sizes for each age group, we were unable to split the data and still have confidence in our results. Future research should endeavor to use an even larger sample that would enable splitting of the data, deriving underlying factors from one-half, and testing the fit of that model on the reserved half. Given our limited IQ and adaptive functioning data, it is still unclear whether these results apply only to particular ASD subgroups. For example, children with ASD with more severe limitations or concurrent intellectual disability may exhibit a different set of behaviors. Future research with larger samples should perform multi-group comparisons of these structural models to test for differential patterns across subgroups of children. 3.4. Implications Because the CBCL is so widely used in research with ASD, it is critical to understand its applicability for this population. The only previous work on the factor structure of the CBCL among children with ASD examined the syndrome scales separately, and used only the RMSEA to measure goodness of fit (Pandolfi et al., 2009, 2011). Our sample sizes allowed for examining the entire theoretical model for both the younger and older versions of the CBCL, with sufficient power. In addition, we used several different fit statistics to determine the validity of the models. These psychometrically sound methods allowed us to be confident in our conclusion that the existing factor models may be more appropriate for use among younger children with ASD than among older children with ASD. The results of this study suggest that the previously established CBCL syndrome scales should be interpreted with caution among children with ASD. Emotional and behavioral problems may manifest differently in children with ASD than among typically developing children, and pre-existing syndrome scales may not yield useful information about the underlying construct of interest in this population (e.g., anxiety, depression, etc.). For example, if the CBCL from a child with ASD indicated significant anxiety based on the domain scores, that child may not truly be experiencing anxiety, but the anxiety scale could be elevated for other reasons associated with ASD (namely, emotion-regulation problems in young children and social-emotional deficits in older children). This would then have clear diagnostic and treatment implications. Similarly, our exploratory models suggest that observable behaviors in children with ASD may reflect different underlying social/emotional constructs compared to typically developing children, and the previously deemed extraneous, or odd “other” problem behaviors in typically developing children may be more closely associated with specific challenging behavior subscales in children with ASD. Thus, it may be more valuable for practitioners working with children or youth with ASD to examine the individual item responses rather than syndrome or subscale scores until a more conclusive factor structure is established for this population. Conflict of interest The authors declare no conflicts of interest. Acknowledgements This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.rasd.2017.06. 001. References Achenbach, T. M., & Rescorla, L. (2001). Manual for the ASEBA school-Age forms & profiles: An integrated system of multi-informant assessment. Burlington, VT: University of Vermont, Research Center for Children, Youth & Families. Achenbach, T. M., & Rescorla, L. A. (2013). Achenbach system of empirically based assessment. In F. Volkmar (Ed.), Encyclopedia of autism spectrum disorders. New York, NY: Springer. Achenbach, T. M., Dumenci, L., & Rescorla, L. A. (2003). DSM-Oriented and empirically based approaches to constructing scales from the same item pools. Journal of Clinical Child and Adolescent Psychology, 32, 328–340. http://dx.doi.org/10.1207/S15374424JCCP3203_02. Albores-Gallo, L., Lara-Muñoz, C., Esperón-Vargas, C., Zetina, J. A. C., Soriano, A. M. P., & Colin, G. V. (2007). Validity and reability of the CBCL/6-18. Includes DSM Scales. Actas Espanolas De Psiquiatria, 35(6), 393–399. American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders (5th ed. (dsm-5)). Washington, DC: merican Psychiatric Association. Biederman, J., Petty, C. R., Fried, R., Wozniak, J., Micco, J. A., Henin, A., & Faraone, S. V. (2010). Child Behavior Checklist clinical scales discriminate referred youth with autism spectrum disorder: A preliminary study. Journal of Developmental & Behavioral Pediatrics, 31(6), 485–490.

38

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Buck, T. R., Viskochil, J., Farley, M., Coon, H., McMahon, W. M., Morgan, J., & Bilder, D. A. (2014). Psychiatric comorbidity and medication use in adults withautism spectrum disorder. Journal of Autism and Developmental Disorders, 44(12), 3063–3071. Buie, T., Campbell, D. B., Fuchs, G. J., Furuta, G. T., Levy, J., VandeWater, J., ... Winter, H. (2010). Evaluation, diagnosis, and treatment of gastrointestinal disorders in individuals with ASDs: A consensus report. Pediatrics, 125(Supplement 1), S1–S18. Caamaño, M., Boada, L., Merchán-Naranjo, J., Moreno, C., Llorente, C., Moreno, D., Arango, C., & Parellada, M. (2013). Psychopathology in children andadolescents with ASD without mental retardation. Journal of Autism and Developmental Disorders, 43, 2442–2449. Cholemkery, H., Kitzerow, J., Rohrmann, S., & Freitag, C. M. (2014). Validity of the social responsiveness scale to differentiate between autism spectrum disorders and disruptive behaviour disorders. European Child & Adolescent Psychiatry, 23, 81–93. Cicchetti, D. V., Carter, A. S., & Gray, S. A. O. (2013). Vineland adaptive behavior scales. In F. Volkmar (Ed.), Encyclopedia of autism spectrum disorders. New York, NY: Springer. Croen, L. A., Zerbo, O., Qian, Y., Massolo, M. L., Rich, S., Sidney, S., & Kripke, C. (2015). The health status of adults on the autism spectrum. Autism, 19, 814–823. De Groot, A., Koot, H. M., & Verhulst, F. C. (1994). Cross-cultural generalizability of the Child Behavior Checklist cross-informant syndromes. Psychological Assessment, 6(3), 225–230. Dedrick, R. F., Greenbaum, P. E., Friedman, R. M., Wetherington, C. M., & Knoff, H. M. (1997). Testing the structure of the Child Behavior Checklist/4-18 using confirmatory factor analysis. Educational and Psychological Measurement, 57(2), 306–313. http://dx.doi.org/10.1177/0013164497057002009. Dedrick, R. F., Tan, T. X., & Marfo, K. (2008). Factor structure of the Child Behavior Checklist/6-18 in a sample of girls adopted from China. Psychological Assessment, 20(1), 70–75. Elliot, C. D. (2007). Differntial ability scales (2nd ed.). San Antonio, TX: PsychCorp Harcourt Assessment Inc. Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286–299. Frazier, J. A., Biederman, J., Bellordre, C. A., Garfield, S. B., Geller, D. A., Coffey, B. J., & Faraone, S. V. (2001). Should the diagnosis of Attention-Deficit/Hyperactivity disorder be considered in children with Pervasive Developmental Disorder? Journal of Attention Disorders, 4(4), 203–211. Frizzo, G. B., Pedrini, J. R., de Souza, D. S., Bandeira, D. R., & Borsa, J. C. (2014). Reliability of child behavior checklist and teacher's report form in a sample of brazilian children. Universitas Psychologica, 14(1), 149–156. http://dx.doi.org/10.11144/Javeriana.upsy14-1.rcbc. Ghaziuddin, M., & Zafar, S. (2008). Psychiatric comorbidity of adults with autism spectrum disorders. Clinical Neurophsychiatry, 5, 9–12. Gjevik, E., Eldevik, S., Fjæran-Granum, T., & Sponheim, E. (2011). Kiddie-SADS reveals high rates of DSM-IV disorders in children and adolescents with autism spectrum disorders. Journal of Autism and Developmental Disorders, 41, 761–769. Gonzalez, M., & Stern, K. (2016). Co-occurring behavioral difficulties in children with severe feeding problems: A descriptive study. Research in Developmental Disabilities, 58, 45–54. Greenbaum, P. E., & Dedrick, R. F. (1998). Hierarchical confirmatory factor analysis of the Child Behavior Checklist/4-18. Psychological Assessment, 10(2), 149–155. Greenlee, J. L., Mosley, A. S., Shui, A. M., Veenstra-VanderWeele, J., & Gotham, K. O. (2016). Medical and behavioral correlates of depression history in children and adolescents with autism spectrum disorder. Pediatrics, 137(Supplement 2), S105–S114. Haas, M. R., Brown, R. S., Brady, J., & Johnson, D. B. (2012). Validating the BASC-TRS for use with children and adolescents with an educational diagnosis of autism. Remedial and Special Education, 33, 173–183. Hallerod, S. L., Larson, T., Ståhlberg, O., Carlström, E., Gillberg, C., Anckarsäter, H., ... Gillberg, C. (2010). The autism- tics, AD/HD and other comorbidities (A-TAC) telephone interview: Convergence with the child behavior checklist (CBCL). Nordic Journal of Psychoatry, 64(3), 218–224. Hartley, S. L., Sikora, D. M., & McCoy, R. (2008). Prevalence and risk factors of maladaptive behaviour in young children with Autistic Disorder. Journal of Intellectual Disability Research, 52(10), 819–829. Havdahl, K. A., von Tetzchner, S., Huerta, M., Lord, C., & Bishop, S. L. (2016). Utility of the child behavior checklist as a screener for autism spectrum disorder. Autism Research, 9, 33–42. Hirata, I., Mohri, I., Kato-Nishimura, K., Tachibana, M., Kuwada, A., Kagitani-Shimono, K., ... Taniike, M. (2016). Sleep problems are more frequent and associated with problematic behaviors in preschoolers with autism spectrum disorder. Research in Developmental Disabilities, 49, 86–99. Ivanova, M. Y., Achenbach, T. M., Rescorla, L. A., Harder, V. S., Ang, R. P., Bilenberg, N., ... Verhulst, F. C. (2010). Preschool psychopathology reported by parents in 23 societies: Testing the seven-syndrome model of the child behavior checklist for ages 1.5–5. Journal of the American Academy of Child & Adolescent Psychiatry, 49(12), 1215–1224. Joshi, G., Petty, C., Wozniak, J., Henin, A., Fried, R., Galdo, M., Kotarski, M., Walls, S., & Biederman, J. (2010). The heavy burden of psychiatric comorbidity in youth with autism spectrum disorders: A large comparative study of a psychiatrically referred population. Journal of Autism and Developmental Disorders, 40(11), 1361–1370. Kaat, A. J., & Lecavalier, L. (2013). Disruptive behavior disorders in children and adolescents with autism spectrum disorders: A review of the prevalence, presentation, and treatment. Research in Autism Spectrum Disorders, 7(12), 1579–1594. Kaat, A. J., Lecavalier, L., & Aman, M. G. (2014). Validity of the Aberrant Behavior Checklist in children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 44, 1103–1116. Kanne, S., & Mazurek, M. (2011). Aggression in children and adolescents with ASD: Prevalence and risk factors. Journal of Autism and Developmental Disorders, 41(7), 926–937. Kenny, D. A. (2011). Respecification of latent variable models. [Retrieved November 18, 2016.]. Kim, S. H., Joseph, R. M., Frazier, J. A., O’Shea, T. M., Chawarska, K., Allred, E. N., ... Kuban, K. K. (2016). Predictive validity of the Modified Checklist for Autism in Toddlers (M-CHAT) born very preterm. The Journal of Pediatrics, 178, 101–107. Lempp, T., de Lange, D., Radeloff, D., & Bachmann, C. (2012). The clinical examination of children, adolescents and their families. In J. M. Rey (Ed.), IACAPAP eTextbook of child and adolescent mental health. Geneva: International Association for Child and Adolescent Psychiatry and Allied Professions. Little, P. T. D. (2013). Longitudinal structural equation modeling. Guilford Press. Lord, C., Rutter, M., & le Couteur, A. (1994). Autism diagnostic interview-revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism & Developmental Disorders, 24(5), 659–685. Lord, C., DiLavorne, P. C., & Risi, S. (2002). Autism diagnostic observation schedule. Los Angeles, CA: Western Psychological Services. MacCallum, R. C., Browne, M. W., & Cai, L. (2006). Testing differences between covariance structure models: Power analysis and null hypothesis. Psychological Methods, 11(1), 19–35. Mannion, A., & Leader, G. (2013). Comorbidity in autism spectrum disorder: A literature review. Research in Autism Spectrum Disorders, 7(12), 1595–1616. Matson, J. L., & Cervantes, P. E. (2014). Commonly studied comorbid psychopathologies among persons with autism spectrum disorder. Research in Developmental Disabilities, 35(5), 952–962. Mayes, S. D., Calhoun, S. L., Murray, M. J., & Zahid, J. (2011). Variables associated with anxiety and depression in children with autism. Journal of Developmental and Physical Disabilities, 23(4), 325–337. Mazurek, M. O., & Kanne, S. (2010). Friendship and internalizing symptoms among children and adolescents with ASD. Journal of Autism and Developmental Disorders, 40(12), 1512–1520. Mazzone, L., Ruta, L., & Reale, L. (2012). Psychiatric comorbidities in Asperger syndrome and high functioning autism: Diagnostic challenges. Directions in Psychiatry, 32, 237–251. Mirenda, P., Smith, I. M., Vaillancourt, T., Georgiades, S., Duku, E., Szatmari, P., ... Zwaigenbaum, L. (2010). Validating the Repetitive Behavior Scale-Revised in young children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 40, 1521–1530. Mullen, E. M. (1995). Mullen scales of early learning (ags ed.). Circle Pines, MN: American Guidance Service Inc. Muratori, F., Narzisi, A., Tancredi, R., Cosenza, A., Calugi, S., Saviozzi, I., ... Calderoni, S. (2011). The CBCL 1. 5-5 and the identification of preschoolers with autism in Italy. Epidemiology and Psychiatric Sciences, 20(4), 329–338. Myers, C. L., Gross, A. D., & McReynolds, B. M. (2014). Broadband behavior rating scales as screeners for autism? Journal of Autism and Developmental Disorders, 44(6), 1403–1413.

39

Research in Autism Spectrum Disorders 40 (2017) 24–40

K. Medeiros et al.

Nakamura, B. J., Ebesutani, C., Bernstein, A., & Chorpita, B. F. (2009). A psychometric analysis of the child behavior checklist DSM-oriented scales. Journal of Psychopathology and Behavioral Assessment, 31, 178–189. http://dx.doi.org/10.1007/s10862-008-9119-8. Narzisi, A., Calderoni, S., Maestro, S., Calugi, S., Mottes, E., & Muratori, F. (2013). Child behavior check list 1½?5 as a tool to identify toddlers with autism spectrum disorders: A case-control study. Research in Developmental Disabilities, 34(4), 1179–1189. Ooi, Y. P., Rescorla, L., Ang, R. P., Woo, B., & Fung, D. S. S. (2010). Identification of autism spectrum disorders using the Child Behavior Checklist in Singapore. Journal of Autism and Developmental Disorders, 41, 1147–1156. Ooi, Y. P., Rescorla, L., Sung, M., Fung, D. S., Woo, B., & Ang, R. P. (2014). Comparisons between autism spectrum disorders and anxiety disorders: Findings from a clinic sample in Singapore. Asia-Pacific Psychiatry, 6(1), 46–53. Osborne, J. W., & Costello, A. B. (2009). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Pan-Pacific Management Review, 12, 131–146. Pandolfi, V., Magyar, C., & Dill, C. (2009). Confirmatory factor analysis of the Child Behavior Checklist 1.5–5 in a sample of children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 39(7), 986–995. Pandolfi, V., Magyar, C. I., & Dill, C. A. (2011). An initial psychometric evaluation of the CBCL 6–18 in a sample of youth with autism spectrum disorders. Research in Autism Spectrum Disorders, 6, 96–108. http://dx.doi.org/10.1016/j.rasd.2011.03.009. Rescorla, L., Kim, Y. A., & Oh, K. J. (2015). Screening for ASD with the korean CBCL/1½ −5. Journal of Autism and Developmental Disorders, 45, 4039–4050. Richdale, A. L., & Schreck, K. A. (2009). Sleep problems in autism spectrum disorders: Prevalence, nature, possible biopsychosocial aetiologies. Sleep Medicine Reviews, 13(6), 403–411. Rieske, R. D., Matson, J. L., Davis, T. E., Konst, M. J., Williams, L. W., & Whiting, S. E. (2013). Examination and validation of a measure of anxiety specific to children with autism spectrum disorders. Developmental Neurorehabilitiation, 16, 9–16. Roid, G. H., Miller, L. J., Pomplun, M., & Koch, C. (2013). Leiter international performance (3rd ed (leiter-3)). Wood Dale. IL: Stoelting. Roid, G. H. (2003). Stanford-Binet intelligence scales (5th ed: technical manual). Itasca, IL: Riverside Publishing. Rosenberg, R. E., Kaufmann, W. E., Law, J. K., & Law, P. A. (2011). Parent report of community psychiatric comorbid diagnoses in autism spectrum disorders. Autism Research and Treatment, 405–849. Ross, P., & Cuskelly, M. (2006). Adjustment: Sibling problems and coping strategies of brothers and sisters of children with autistic spectrum disorder. Journal of Intellectual and Developmental Disability, 31, 77–86. Salazar, F., Baird, G., Chandler, S., Tseng, E., O’Sullivan, T., Howlin, P., Pickels, A., & Simonhoff, E. (2015). Comorbid psychiatric disorders in preschool and elementary school-aged children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 45, 2283–2294. Samson, A. C., Hardan, A. Y., Lee, I. A., Phillips, J. M., & Gross, J. J. (2015). Maladaptive behavior in Autism Spectrum Disorder: The role of emotion experience and emotion regulation. Journal of Autism and Developmental Disorders, 45, 3424–3432. Schermelleh-Engel, K., Moosbrugger, H., & Muller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Onlines, 8, 23–74. Simonoff, E. M., Pickles, A. P., Charman, T. P., Chandler, S. P., Loucas, T. O. M. P., & Baird, G. F. (2008). Psychiatric disorders in children with autism spectrum disorders: Prevalence, comorbidity, and associated factors in a population-derived sample. Journal of the American Academy of Child & Adolescent Psychiatry, 47(8), 921–929. So, P., Greaves-Lord, K., van der Ende, J., Verhulst, F. C., Rescorla, L., & de Nijs, P. F. A. (2013). Using the Child Behavior Checklist and the Teacher’s Report Form for identification of children with autism spectrum disorders. Autism, 17(5), 595–607. Son, J. S., Zheng, L. J., Rowehl, L. M., Tian, X., Zhang, Y., Zhu, W., ... Li, E. (2015). Comparison of fecal microbiota in children with Autism Spectrum Disorders and neurotypical siblings in the Simons Simplex Collection. Public Library Of Science, 10(10), e0137725. http://dx.doi.org/10.1371/journal.pone.0137725. Sparrow, S. S., Cicchetti, D. V., & Balla, D. A. (2005). Vineland adaptive behavior scales-II. Circle Pines, MN: American Guidance Service. Storch, E. A., Wood, J. J., Ehrenreich-May, J., Jones, A. M., Park, J. M., Lewin, A. B., & Murphy, T. K. (2012). Convergent and discriminant validity and reliability of the pediatric anxiety rating scale in youth with autism spectrum disorders. Journal of Autism and Developmental Disorders, 42, 2374–2382. Vasa, R. A., Kalb, L., Mazurek, M., Kanne, S., Freedman, B., Keefer, A., ... Murray, D. (2013). Age-related differences in the prevalence and correlates of anxiety in youth with autism spectrum disorders. Research in Autism Spectrum Disorders, 7(11), 1358–1369. Velicer, W. F., & Jackson, D. N. (1990). Component analysis versus common factor-analysis: Some further observations. Multivariate Behavioral Research, 25, 89–95. Wade, J. L., Cox, N. B., Reeve, R. E., & Hull, M. (2014). Brief report: Impact of child problem behaviors and parental broad autism phenotype traits on substance use among parents of children with ASD. Journal of Autism and Developmental Disorders, 44, 2621–2627. Wechsler, D. (2002). The wechsler preschool and primary scale of intelligence (3rd ed.). San Antonio, TX: The Psychological Corporation. Wechsler, D. (2003). Wechsler intelligence scale for children (4th ed (wisc-iv)). San Antonio, TX: NCS Pearson. Wechsler, D. (2011). Wechsler abbreviated scale of intelligence (2nd ed (wasi-ii)). San Antonio, TX: NCS Pearson. Wechsler, D. (2012). Wechsler preschool and primary scale of intelligence (4th ed.). Bloomington, MN: Pearson. Williams, B., Onsman, A., & Brown, T. (2010). Exploratory factor analysis: A five-step guide for novices. Journal of Emergency Primary Health Care, 8 [Article 990399]. Xu, Y., Neece, C. L., & Parker, K. H. (2014). Parental depression and child behavior problems: A pilot study examining pathways of influence. Journal of Mental Health Research in Intellectual Disabilities, 7, 126–142.

40