Screening accuracy for risk of autism spectrum disorder using the Brief Infant-Toddler Social and Emotional Assessment (BITSEA)

Research in Autism Spectrum Disorders 7 (2013) 591–600

Contents lists available at SciVerse ScienceDirect

Research in Autism Spectrum Disorders Journal homepage: http://ees.elsevier.com/RASD/default.asp

Screening accuracy for risk of autism spectrum disorder using the Brief Infant-Toddler Social and Emotional Assessment (BITSEA) Lauren M. Gardner a, Laura Murphy a,*, Jonathan M. Campbell b, Frances Tylavsky c, Frederick B. Palmer a, J. Carolyn Graff d a

Boling Center for Developmental Disabilities, University of Tennessee Health Science Center, 711 Jefferson Avenue, Memphis, TN, USA Department of Educational, School, and Counseling Psychology, University of Kentucky, 236 Dickey Hall, Lexington, KY, USA Preventive Medicine, University of Tennessee Health Science Center, Memphis, TN, USA d Associate Professor, College of Nursing and Chief of Nursing, Boling Center for Developmental Disabilities, University of Tennessee Health Science Center, 711 Jefferson Avenue, Memphis, TN, USA b c

A R T I C L E I N F O

A B S T R A C T

Article history: Received 25 October 2012 Received in revised form 17 January 2013 Accepted 17 January 2013

Early identification of autism spectrum disorders (ASDs) is facilitated by the use of standardized screening scales that assess the social emotional behaviors associated with ASD. Authors examined accuracy of Brief Infant-Toddler Social and Emotional Assessment (BITSEA) subscales in detecting Modified Checklist for Autism in Toddlers (M-CHAT) risk for 456 toddlers. An ASD-specific screener, Total ASD, was developed from BITSEA subscales incorporating both behavioral and social communicative features of ASD. Results indicated that Total ASD is more accurate in detecting autism risk than scales measuring general behavior problems or social competence. Results provide support for the BITSEA as a Level I screener for social emotional problems and a Level II screener for ASD symptoms at community sites such as well child clinics. ß 2013 Published by Elsevier Ltd.

Keywords: Autism spectrum disorders Developmental screening BITSEA M-CHAT

1. Introduction Autism spectrum disorders (ASDs) are complex neurodevelopmental disorders that affect individuals across the lifespan. The behavioral symptoms of ASDs are often present early in life and include (a) impairments in social interaction, (b) communication delays, and (c) restricted and stereotypical behaviors. The behavioral symptoms characteristic of ASD appear early in development, typically between ages 12 and 24 months (American Psychiatric Association, 2000). Researchers have demonstrated the stability of restricted and/or repetitive behaviors in very young children with ASD (Kim & Lord, 2010). Research suggests that impairments in social behaviors are most likely to be present at early ages (Stone, Coonrod, & Ousley, 2000) and can be identified during the second year of life, if not sooner (McConnell, 2002; Pierce, Carter, Weinfield, & Desmond, 2011; Webster, Feiler, & Webster, 2003; Woods & Wetherby, 2003). Specifically, deficits in non-verbal social communication, lack of social or emotional reciprocity, and speech/language delays are the most prevalent diagnostic characteristics for children under 3 years of age (Stone et al., 1999). From an early age children with ASD also often exhibit developmental delays in the areas of orienting to social stimuli, play, motor imitation, and joint attention skills (McConnell, 2002; Stone et al., 2000; Webster et al., 2003; Woods & Wetherby, 2003). Also, parents of children with autism often identify language delays and social–emotional concerns within the first two years of their children’s development, with up to 30% of parents identifying these concerns by the first birthday (De Giacomo & Fombonne, 1998). As such, parents are thought to

* Corresponding author. Tel.: +1 901 448 6567; fax: +1 901 448 3844. E-mail address: [email protected] (L. Murphy). 1750-9467/$ – see front matter ß 2013 Published by Elsevier Ltd. http://dx.doi.org/10.1016/j.rasd.2013.01.004

592

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

serve as useful respondents when detecting autism risk in young children (Kim & Lord, 2012). Further, as these developmental areas represent social and communicative behaviors that are predictive of later language outcomes, early detection of deficits in these areas is imperative (Carpenter & Tomasello, 2000; Mundy & Sigman, 2006; Woods & Wetherby, 2003). The current prevalence rates from the Center for Disease Control and Prevention estimate that 1 in every 88 8-year-old children is diagnosed with an autism spectrum disorder (Autism and Developmental Disabilities Monitoring Network Surveillance Year 2008 Principal Investigators, 2012). There is no known universal cause of ASD, but accurate early diagnosis is crucial as early intensive treatment is associated with improved outcomes for children with ASD (Harris & Handleman, 2000). Research indicates that few pediatricians utilize standard ASD specific screening instruments (Dosreis, Weiner, Johnson, & Newschaffer, 2006), which influences the age at which children with ASD are first diagnosed and receive intervention services. Therefore, ongoing efforts are needed to develop and evaluate screening instruments that detect ASD specific symptomatology, are psychometrically sound (Matson, Wilkins, & Gonzalez, 2008), and can be easily administered in clinical practice. Early intervention necessitates early detection and diagnosis; therefore, several recently developed screening tools have incorporated ASD-specific items to detect the early symptoms of ASD and identify those children who are at-risk for ASD and would benefit from further follow-up (Kogan et al., 2009). An exhaustive list of autism-specific screening instruments is beyond the scope of the current investigation; however, there are a variety of screening instruments available to pediatric primary care providers that provide valuable information regarding the presence of early symptoms of ASD. The reader is encouraged to pursue more comprehensive reviews on this topic (Barbaro & Dissanayake, 2009). Examples of such screening instruments include The Modified Checklist for Autism in Toddlers (M-CHAT; Robins, Fein, Barton, & Green, 2001), Screening Tool for Autism in Two-year-olds (STAT; Stone et al., 2000), Baby and Infant Screen for Children with aUtIsm Traits-Part 1 (BISCUIT-Part 1; Matson, Boisjoli, & Wilkins, 2007), and the Social Communication Questionnaire (SCQ; Berument, Rutter, Lord, Pickles, & Bailey, 1999). The present investigation utilized two screening instruments, the Brief Infant Toddler Social Emotional Assessment (BITSEA; Briggs-Gowan & Carter, 2006) and the Modified-Checklist for Autism in Toddlers (M-CHAT; Robins et al., 2001). While the BITSEA was developed to identify a broad range of behavior problems, it also incorporates autism-specific items. Screening instruments differ in their purpose and breadth of screening. A Level I screening instrument is used to identify children who are at-risk in the general population whereas Level II screening instruments are used to identify risk among children who are already considered to be at increased risk. Level I screening instruments are generally designed to be brief as they are administered to large samples. Level II screening instruments are used with children who have been identified at-risk for a specific disorder or class of disorders and typically take longer to administer. Some screening tools, like the BITSEA and the MCHAT, can be used at both Level I and Level II screenings (Johnson, Myers, & the Council on Children with Disabilities, 2007; Robins & Dumont-Mathieu, 2006). As a Level II screening instrument is used to identify risk among children who are already considered to be at increased risk, the BITSEA has 9 ASD-specific behavioral problem items (ASD Problem Behavior) and 8 ASD-specific competency items (ASD Social Competence); however, norms for the BITSEA ASD Problem Behavior and ASD Social Competence scales are not available. Although there are a number of instruments that screen for autism-specific behaviors, relatively few represent a single instrument that can be used at both Level I and Level II screenings. As such, the BITSEA is a screening instrument that provides a broader screening for a range of behavior problems and a more specific screening for behaviors related to ASD. Another advantage the BITSEA offers as a screening instrument is the early age at which it can be administered, and the brevity of time required of caregivers to complete the measure. Pediatricians and primary care providers would benefit from screening instruments for young children that assess for both ASD and broader developmental delays. The purpose of this study was to examine 12- and 24-month BITSEA behavior problem scores and competency delays, as well as BITSEA ASD-specific competency and behavior problem items, as related to M-CHAT scores at 24 months using simple correlations and receiver operating characteristic (ROC) analyses. BITSEA ASD-specific scales were expected to yield greater accuracy for ASD-risk measured by the M-CHAT when compared to general BITSEA scales. 2. Method 2.1. Participants and procedure The participants for this study were enrolled in a longitudinal cohort study, Conditions Affecting Neurocognitive Development and Learning in Early Childhood (CANDLE), which is funded by The Urban Child Institute in Shelby County, Tennessee. The CANDLE study is designed to examine a wide range of maternal and infant characteristics associated with early childhood development. CANDLE staff recruit healthy women in Memphis and Shelby County, Tennessee during their second trimester of pregnancy and follow mother–infant dyads until the child reaches four years of age. For the present study, participants included 456 (65.4% African American, 32.5% Caucasian, and 2.2% other) infant/mother dyads who participated in CANDLE data collection at clinic visits coinciding with infants’ first and second birthdays. At the time of infants’ first clinic visit, a majority of mothers (58%) were either married or living with a partner in a committed relationship, almost half of the mother’s had a high school degree or General Education Diploma (GED), and over half of the families reported receiving Medicaid insurance. Table 1 provides a detailed description of participant characteristics at the clinic visit coinciding with infant’s first birthday. Infant/mother dyads who participated in CANDLE data collection were administered

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

593

Table 1 Participant characteristics. Variable Race of mother African-American White/Caucasian Other Marital status Married Never married Living with partner Divorced Separated Widowed Missing Educational status High school diploma or GED College degree Graduate or professional degree Technical school Less than high school Missing Insurance status Medicaid Other None Child’s gender Male Female Child’s Risk for Autism Spectrum Disorder No Yes Participant characteristics (N = 456)

N

%

298 148 10

65.4 32.5 2.2

198 172 67 10 7 1 1

43.4 37.7 14.7 2.2 1.5 .2 .2

217 110 63 40 25 1

47.6 24.1 13.8 8.8 5.5 .2

239 209 8

52.4 45.8 1.8

240 216

52.6 47.4

415 41

91.0 9.0

M

SD

27.9 28.9

5.5 5.5

12.4 24.4

1.4 1.4

Variable Mothers’ age (yrs) Time 1 (12-month visit) Time 2 (24-month visit) Child’s age (months) Time 1 (12-month visit) Time 2 (24-month visit) Note. (N = 456); M: mean; SD: standard deviation.

the BITSEA at both clinic visits coinciding with infant’s first and second birthdays. The M-CHAT was administered to participants at the clinic visit coinciding with the child’s second birthday. The BITSEA and MCHAT were administered in interview format by a licensed psychologist or advanced graduate student during the clinic visits. The Institutional Review Board at the University of Tennessee, Memphis approved all procedures (UTHSC IRB 06-08495-FB). 2.2. Measures 2.2.1. Brief Infant-Toddler Social and Emotional Assessment (BITSEA; Briggs-Gowan & Carter, 2006) The BITSEA, a parent-report measure of socio-emotional problems and competence delays, is a screening tool which includes behavioral and competency items for infants and toddlers. The BITSEA is appropriate for children ages 12–36 months and includes 42 items with three options for responding: 0 = Not True/Rarely; 1 = Somewhat True/Sometimes; or 2 = Very True/Often. The BITSEA consists of a Problem Behavior scale (31 items, which include 9 ASD-specific behavioral problem items), and a Competence scale (11 items, which include 8 ASD-specific competency items). Scores from the BITSEA competency scale have been found to be significantly lower in children with autism (Karabekiroglua, Briggs-Gowan, Carter, Rodopman-Armand, & Akbasa, 2010). Previous research has demonstrated the acceptability, reliability, and validity of the BITSEA as a measure of broad social– emotional/behavioral problems in early intervention settings (Briggs-Gowan & Carter, 2007). According to the BITSEA manual, test–retest reliability was evaluated for the Problem and Competence Total scores for 84 children (40 boys and 44 girls) with an average of 6 days between the two ratings. For the total sample, Pearson correlations were .92 Total Problem score was .82 for the Total Competence score. Inter-rater reliability for the BITSEA was evaluated by ratings provided by 94 pairs of parents (46 daughters and 48 sons were rated by parents) with an average of 4 days between ratings. Interclass coefficients were used to evaluate inter-rater reliability. Inter-rater reliability coefficients for the Problem Total score ranged

594

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

from .70 for boys to .78 for girls which is considered to be in the good to excellent range. Inter-rater reliability coefficients for the Competence Total score ranged from .58 for girls to .67 for boys which is considered to be in the adequate to good range. The scale can be completed independently by a caregiver or as part of a structured interview with an administration time of 5–7 min. Once completed, the BITSEA yields a Total Problem score and a Total Competence score. A Total Problem score on the BITSEA is derived by summing ratings on 31 Problem items that measure behavioral difficulties in externalizing, internalizing, and dysregulation. If the child’s Total Problem score meets or exceeds the recommended cut-score value (a score greater than or equal to the highest 25th percentile), then the score falls within the Possible Problem range which may indicate clinically significant behavior problems. The Total Competence Score on the BITSEA is derived by summing ratings on the 11 Competence items that measure social–emotional skills and development. If the child’s Total Competence score is less than or equal to the cut-score value indicated for the lowest 15th percentile, then the score falls within the Possible Deficit/Delay range and indicates the child may not have acquired the same level of social and emotional competencies expected for same-aged peers. There are 14 BITSEA Red Flag items that may indicate low base rate, clinically significant impairments even when the child does not earn scores in the Possible Problem or Possible Deficit/Delays score range. Followup is recommended when a parent indicates the presence of any of the Red Flag items. Investigators calculated BITSEA Total Problem, Total Competence, and Red Flag scores following standardized procedures (Briggs-Gowan & Carter, 2006). The BITSEA also includes items that describe behaviors and difficulties that are frequently exhibited by children with ASD. The test authors identified a set of 17 items that ‘‘may be useful as a screener for identifying children with [autism spectrum] disorders’’ (Briggs-Gowan & Carter, 2006, p. 19). The 17 ASD items sample social (Social Competence) and problem behaviors (Problem Behavior); however, responses for these items are meant to be interpreted with clinical judgment when ASD is suspected and norms for the ASD screening items are not available (Briggs-Gowan, Carter, Irwin, Wachtel, & Cicchetti, 2004). There are eight BITSEA ASD Social Competence items (1, 10, 13, 15, 22, 25, 29, and 31), that sample various social communicative behaviors, such as imitative play, protodeclarative pointing, and appropriate eye gaze. There are nine BITSEA ASD Problem Behavior items (9, 14, 21, 35, 36, 37, 38, 39, and 40), that sample various problem behaviors, such as difficulty adjusting to change, repetitive behavior, and avoiding physical contact. For the present investigation, the authors calculated ASD Social Competence Scores, ASD Problem Behavior Scores, and a Total ASD Screening Score, which consisted of subtracting ASD Social Competence Scores from ASD Problem Behavior Scores. The Total ASD screening scores were calculated as a combination of ASD Social Competence and ASD Problem Behavior Scores as children with ASD typically demonstrate deficits in both areas. Higher Total ASD Screening Scores were expected to indicate greater risk for the presence of an ASD. 2.2.2. Modified-Checklist for Autism in Toddlers (M-CHAT; Robins et al., 2001) The MCHAT is an ASD-specific parent report measure that includes 23 yes/no questions and is typically used with 16–30 month old toddlers who are at-risk for ASD. Of the 23 items, six items are considered to be critical to predicting an ASD diagnosis and include protodeclarative pointing, response to name, interest in peers, bringing things to show parents, following a point, and imitation. Children who fail any 3 of the 23 total items or 2 of the 6 critical items are considered to be at-risk for ASD. Robins and Dumont-Mathieu (2006) reviewed research on the MCHAT as a Level I and Level II screening instrument. The preliminary results for the MCHAT as a screening instrument were derived from parent reports from two groups of toddlers. The first group of toddlers (n = 1122, Level I) were those administered the MCHAT by the primary care physician. The second group of toddlers (n = 171, Level II) were those who had been referred for early intervention due to developmental delay (Robins et al., 2001). Reportedly, the psychometrics of the measure include good internal consistency for the entire checklist at .85, and for the critical items at .83. Furthermore, initial research reported MCHAT sensitivity of .87 and specificity of .99 (Robins et al., 2001). More recent reports of the sensitivity and specificity of the MCHAT are lower than these initial estimates with sensitivity ranging from .70 to .97 and specificity ranging from .38 to .99 (Matson et al., 2009; Snow & Lecavalier, 2008). Validation of the MCHAT is ongoing; however, findings thus far suggest it may result in a higher false positive rate when the follow-up interview is not completed subsequent to initially failing the instrument. Research conducted by Matson, Kozlowski, Fitzgerald, and Sipes (2013) found significant differences between true and false positive groups using the MCHAT, with the true positive group scoring significantly higher on the overall number of items failed. Furthermore, the true positive group’s average critical items score was above the MCHAT’s cut-off of two items, whereas the false positive group’s average critical items score was not, suggesting that the critical items may differentiate between true and false negatives. Longitudinal data continue to be collected to address the issue of psychometric properties as well as sensitivity and specificity for the MCHAT as a Level I and Level II screening instrument (Robins & Dumont-Mathieu, 2006). The M-CHAT as a Level I screening instrument served as the criterion variable in the investigation. Consistent with prior research using the M-CHAT, two M-CHAT scores were calculated: (a) Total M-CHAT scores and (b) Critical Item M-CHAT scores. 2.3. Data analyses Differences for six BITSEA subtests at both 12 months and 24 months (i.e., a total of 12 contrasts) were calculated between groups identified at risk by the M-CHAT (n = 41) and those not at risk (n = 415). Authors utilized a one-way multivariate

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

595

analysis of variance (MANOVA) to examine differences across risk groups. Pearson’s product–moment correlations were calculated between six BITSEA subtests and two M-CHAT scores. BITSEA subtests included in the correlational analyses were the: (a) Red Flag Total Raw Score, (b) Total Problem Raw Score, (c) Total Competence Raw Score, (d) ASD Social Competence Raw Score, (e) ASD Problem Behavior Raw Score, and (f) Total ASD Screening Score. Total and Critical Item M-CHAT scores were also included in the correlational analysis. 2.3.1. Receiver operating characteristic (ROC) analyses BITSEA ASD Screener accuracy for detecting risk of ASD was assessed via ROC curve analysis and area under the curve (AUC) calculation. The ROC analysis was completed using SPSS version 19. ROC analysis represents paired true-positive (i.e., sensitivity) and false-positive ratios across a test’s full range of decision thresholds (i.e., all cutoff points for the screener). A measure with random accuracy produces an AUC value of .50. AUCs falling between .5 and .7 denote low test accuracy, .7 and .9 denote moderate test accuracy, and .9 and 1.0 indicate high test accuracy (Swets, 1998). Investigators were interested in determining if the BITSEA Total ASD Screening score resulted in improved accuracy for detecting ASD risk in the sample when compared to general BITSEA screening results. Therefore, we compared AUCs between the BITSEA ASD scales and the Red Flag, Total Problem, and Total Competence Scores. AUC comparisons were conducted via z tests using the formula (Hanley & McNeil, 1983): z¼

AUC 1  AUC 2 ½ðSE1 Þ2 þ ðSE2 Þ2  2rSE1 SE2 

1=2

where AUC1 and SE1 refer to the observed area and estimated standard error for test 1; AUC2 and SE2 refer to the observed area and estimated standard error for test 2, and r references the correlation between AUC1 and AUC2. 3. Results As shown in Table 1, data collection visits typically occurred close to children’s first and second birthdays. Mother MCHAT ratings at 24 months resulted in a total of 41 (9.0%) toddlers identified as demonstrating behavior that indicate risk for ASD. Descriptive statistics for both the BITSEA and M-CHAT rating scales are presented in Table 2. 3.1. Group differences on the BITSEA scales Children identified as at risk (n = 41) scored significantly higher on all BITSEA scales when compared to children identified as not at risk (n = 415). See Table 2 for descriptive statistics for all BITSEA scales. The MANOVA resulted in a significant effect of risk group, Wilks’ L = .79, F(10, 445) = 11.73, p < .001; h2 = .21. Follow-up univariate tests revealed significant differences between groups for all scales, including the BITSEA Total ASD screening score at both 12 and 24 months (see Table 2 for results).

Table 2 Brief Infant-Toddler Social and Emotional Assessment (BITSEA) scores for children passing and failing the 24-month Modified Checklist for Autism in Toddlers (M-CHAT). BITSEA scale

Failed (n = 41) M

12-month ratings Red Flag Problem Competence ASD Social ASD Behavior Total ASDa 24-month ratings Red Flag Problem Competence ASD Social ASD Behavior Total ASDa

Passed (n = 415) SD

M

SD

F

h2

4.54 14.05 13.56 10.44 3.07 7.37

2.70 6.18 3.05 2.64 2.04 2.94

3.00 9.87 15.04 11.53 1.75 9.77

2.46 5.92 3.24 2.62 1.74 3.30

14.43 18.47 7.84 6.44 20.67 20.24

.03 .04 .02 .01 .04 .04

6.68 19.05 15.46 12.37 4.41 7.95

5.39 12.41 2.98 2.32 3.66 4.34

2.58 9.77 18.16 14.20 1.70 12.51

2.17 5.41 2.55 1.82 1.69 2.76

91.84 79.75 40.26 36.10 72.96 89.81

.17 .15 .08 .07 .14 .17

Note. (N = 456); ASD Risk is measured by M-CHAT; Brief Infant-Toddler Social and Emotional Assessment (BITSEA); Red Flag: BITSEA Red Flag Total Raw Score items (2, 14, 18, 24, 32, 34, 35, 36, 37, 38, 39, 40, 41, and 42); Problem: BITSEA Total Problem Raw Score; Competence: BITSEA Total Competence Raw Score; ASD Social: BITSEA ASD Social Competence Raw Score items (1, 10, 13, 15, 22, 25, 29, and 31); ASD Behavior: ASD Problem Behavior Raw Score items (9, 14, 21, 35, 36, 37, 38, 39, and 40). Total ASD: BITSEA ASD Screening Score. M: mean; SD: standard deviation; h2 = partial eta squared. a Total ASD Score calculated as: ASD Problem Behavior  ASD Social Competence; higher score indicates greater problem.

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

596

Table 3 Correlations among mother’s 12-month BITSEA and 24-month M-CHAT ratings. Red flag

Problem

BITSEA subtests Red Flag – Problem .80** – Competence .09* .07 ASD Social .06 .01 ASD Behavior .79** .70** Total ASDa .47** .37** Modified Checklist for Autism in Toddlers (M-CHAT) Total .25** .29** Critical .08 .06

Competence

ASD Social

ASD Behavior

– .93** .12* .79**

– .10* .84**

–

.21** .16**

.19** .19**

Total ASD

.62** .29** .14**

.31** .22*

Note. (N = 456); Brief Infant-Toddler Social and Emotional Assessment (BITSEA); Red Flag: BITSEA Red Flag Total Raw Score; Problem: BITSEA Total Problem Raw Score; Competence: BITSEA Total Competence Raw Score; Social: BITSEA ASD Social Competence Raw Score; Behavior: BITSEA ASD Problem Behavior Raw Score; Total ASD: BITSEA ASD Screening Total Score. Total: M-CHAT Rating at 24 months; Critical: M-CHAT Rating on critical items at 24 months. a Total ASD Score calculated as: ASD Problem Behavior Items  ASD Social Competence Items; higher score indicates greater problem. * p < .05. ** p < .01.

3.2. Correlational findings 3.2.1. Relationships between BITSEA scales Pearson product–moment correlations amongst BITSEA scales at 12 months (Table 3) and 24 months (Table 4) revealed expected patterns of positive relationship between similar scales (e.g., BITSEA Total Competence and BITSEA ASD Social Competence) and negative relationship between dissimilar scales (e.g., BITSEA Total Competence and Total Problem Behavior). Correlations were statistically significant with the exception of the relationships between (a) the BITSEA Total Problem Score and the Total Competence, (b) the BITSEA Total Problem Score and the ASD Social Competence scores, and (c) the BITSEA ASD Social Competence and Red Flag scores at 12 months. At 12 months, the BITSEA ASD Screening correlated moderately and positively with BITSEA Red Flag, Total Problem, and ASD Problem Behavior scores (rs ranged from .37 to .62) and strongly and negatively with the BITSEA Total Competence and ASD Social Competence scores (rs ranged from .79 to .84). At 24 months, the BITSEA ASD Screening also correlated moderately and positively with BITSEA Red Flag, Total Problem, and ASD Problem Behavior scores (rs ranged from .65 to .81) and strongly and negatively with the BITSEA Total Competence and ASD Social Competence scores (rs ranged from .75 to .78). 3.2.2. Relationships between BITSEA scales and the M-CHAT Pearson product–moment correlations between the 12 month BITSEA scales and the M-CHAT (Table 3) and the 24 month BITSEA scales and the M-CHAT (Table 4) revealed small to moderate relationships between scales in expected directions. At 12 months, four BITSEA problem scales correlated positively with the total M-CHAT score (rs ranged from .25 to .31); however, the BITSEA Red Flag and Total Problem scores were not associated with critical M-CHAT scores while the BITSEA ASD Competence and ASD Screening scales did. At 12 months, BITSEA competence scales correlated negatively with both MCHAT total and critical scores. Table 4 Correlations among mother’s 24-month BITSEA and 24-month M-CHAT ratings. Red flag

Problem

BITSEA subtests Red Flag – Problem .86** – Competence .32** .34** ** ASD Social .25 .24** ASD Behavior .84** .78** Total ASDa .70** .65** Modified Checklist for Autism in Toddlers (M-CHAT) Total .43** .44** Critical .17** .18**

Competence

ASD Social

ASD Behavior

– .91** .31** .75**

– .27** .78**

–

.37** .34*

.33** .32**

Total ASD

.81**

–

.43** .18**

.48** .31*

Note. (N = 456); Brief Infant-Toddler Social and Emotional Assessment (BITSEA); Red Flag: BITSEA Red Flag Total Raw Score; Problem: BITSEA Total Problem Raw Score; Competence: BITSEA Total Competence Raw Score; ASD Social: BITSEA ASD Social Competence Raw Score; ASD Behavior: BITSEA ASD Problem Behavior Raw Score; Total ASD: BITSEA ASD Screening Total Score. Total: M-CHAT Rating at 24 months; Critical: M-CHAT Rating on critical items at 24 months. a Total ASD Score calculated as: ASD Problem Behavior Items  ASD Social Competence Items; higher score indicates greater problem. * p < .05. ** p < .01.

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

597

At 24 months, four BITSEA problem scales correlated positively with total M-CHAT scores (rs ranged from .43 to .48) and M-CHAT critical scores (rs ranged from .17 to .31). At 24 months, BITSEA competence scales correlated negatively with both M-CHAT total and critical scores (rs ranged from .32 to .37). As expected, the magnitude of relationship between the 24month BITSEA ratings and the M-CHAT was stronger than that obtained at 12 months due to ratings being completed during the same visit. Even so, significant, yet small, relationships were found between 12-month BITSEA ratings and 24-month MCHAT ratings. 3.3. ROC analyses 3.3.1. BITSEA scales predicting M-CHAT risk status BITSEA 12-month ratings yielded AUC values that ranged from .632 (BITSEA ASD Social Competence) to .712 (BITSEA Total ASD Screening; see Table 5). The BITSEA ASD scales produced AUCs that fell within the range of low test accuracy as a screener for ASD risk behaviors as defined by MCHAT status. At 24 months, all BITSEA AUC values fell within the range of moderate test accuracy for ASD risk behaviors; the BITSEA ASD Social Competence score produced the lowest AUC (.740), the BITSEA Total ASD Screening score produced the highest AUC (.833). 3.3.2. Comparing BITSEA scale accuracy At 12 months, all BITSEA scales yielded comparable accuracy for detecting ASD risk behaviors, with the exception of the BITSEA Total ASD Screening score yielding significantly better detection of ASD risk behaviors when compared to the BITSEA Total Competence score (see Table 6). At 24 months, the BITSEA Total ASD Screening score yielded accuracy that exceeded Table 5 Receiver operating characteristic (ROC) analyses using BITSEA scales to predict ASD risk. BITSEA scale 12-Month ratings Red Flag Problem Competence ASD Social ASD Behavior Total ASDa 24-Month ratings Red Flag Problem Competence ASD Social ASD Behavior Total ASDa

M

SD

AUC

SE

3.13 10.24 14.91 11.43 1.87 9.56

2.52 6.06 3.25 2.64 1.81 3.34

.678 .708 .649 .632 .696 .712

.043 .046 .044 .046 .044 .037

2.95 10.61 17.92 14.04 1.94 12.10

2.86 6.86 2.71 1.94 2.09 3.21

.784 .754 .762 .740 .768 .833

.038 .041 .036 .040 .037 .031

Note. (N = 456); ASD Risk is measured by M-CHAT; Brief Infant-Toddler Social and Emotional Assessment (BITSEA); Red Flag: BITSEA Red Flag Total Raw Score items (2, 14, 18, 24, 32, 34, 35, 36, 37, 38, 39, 40, 41, and 42); Problem: BITSEA Total Problem Raw Score; Competence: BITSEA Total Competence Raw Score; ASD Social: BITSEA ASD Social Competence Raw Score items (1, 10, 13, 15, 22, 25, 29, and 31); ASD Behavior: ASD Problem Behavior Raw Score items (9, 14, 21, 35, 36, 37, 38, 39, and 40). Total ASD: BITSEA ASD Screening Score. AUC: area under the curve. SE: standard error of the AUC. All BITSEA scores are raw scores. a Total ASD Score calculated as: ASD Problem Behavior  ASD Social Competence; higher score indicates greater problem. Table 6 Results from AUC comparisons for BITSEA scales predicting 24-month M-CHAT risk status. Comparison 12-Month ratings Red Flag – Problem Red Flag – Competence Red Flag – Total ASDa Problem – Competence Problem – Total ASDa Competence – Total ASDa 24-Month ratings Red Flag – Problem Red Flag – Competence Red Flag – Total ASDa Problem – Competence Problem – Total ASDa Competence – Total ASDa

AUC1

(SE1)

AUC2

(SE2)

r

z

p

.678 .678 .678 .708 .708 .649

(.043) (.043) (.043) (.046) (.046) (.044)

.708 .649 .712 .649 .712 .712

(.046) (.044) (.037) (.044) (.037) (.037)

.73 .02 .45 .02 .27 .67

.91 .48 .80 .94 .08 1.88

ns ns ns ns ns .03

.784 .784 .784 .754 .754 .762

(.038) (.038) (.038) (.041) (.041) (.036)

.754 .762 .833 .762 .833 .833

(.041) (.036) (.031) (.036) (.031) (.031)

.83 .19 .63 .21 .59 .65

1.42 .47 1.61 .16 2.34 2.50

ns ns .054 ns .009 .006

Note. (N = 456); BITSEA: Brief Infant-Toddler Social and Emotional Assessment; Red Flag: BITSEA Red Flag Total Score; Problem: BITSEA Total Problem Score; Competence: BITSEA Total Competence Score; Total ASD: BITSEA ASD Screening Score. AUC: area under the curve. SE: standard error of the AUC. All BITSEA scores are raw scores. r = Correlation coefficient between two AUC curves based upon Hanley and McNeil (1983). a Total ASD Score calculated as: ASD Problem Behavior Items  ASD Social Competence Items; higher score indicates greater problem.

598

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

both the general BITSEA Total Problem and Total Competence scores (see Table 6). The difference between the BITSEA Total ASD Screening and Red Flag scales accuracy approached statistical significance (p = .054). The findings indicate that the BITSEA Total ASD Screening performs better than either the general problem or competency scores as a specific screener for ASD risk behaviors. At 12 and 24 months, no BITSEA Total ASD Screening scores yielded recommended sensitivity and specificity values (i.e., .80 or greater). At 12 months, no BITSEA Total ASD Screening score yielded sensitivity and specificity above .70. At 24 months, two BITSEA Total ASD Screening scores yielded sensitivity and specificity values exceeding .70. A Total ASD Screening cutoff score of 12.00 yielded a sensitivity of .76 and specificity of .71; a Total ASD Screening cutoff score of 11.00 yielded a sensitivity of .73 and specificity of .80. 4. Discussion Early diagnosis of ASD is important for children to begin early, effective interventions (Harris & Handleman, 2000). Parents serve as critical informants regarding the development of their children as they are able to observe their children under various contexts for lengthy periods of time. The results from this study provide modest support for the premise that parent ratings of BITSEA ASD-specific behaviors at the time of their child’s first birthday are early correlates of later ASD diagnostic characteristics as measured by the M-CHAT and are significantly better predictors of ASD risk behaviors than overall social–emotional competence. Furthermore, BITSEA ASD-specific behaviors, particularly at 24 months, are more useful in predicting M-CHAT scores when compared to overall problem behaviors and delays in social–emotional competency. These findings support that higher Total ASD Screening scores on the BITSEA at 12 and 24 months indicates greater risk for the presence of behaviors that demonstrate risk for ASD at 24 months. Overall, the results from this study also suggest that practitioners’ ASD diagnostic accuracy may be aided by incorporating parent report of behaviors that are specific to the social deficits and problem behaviors specifically associated with ASD. Screening measures that assess social deficits in infant orientation to social stimuli, play, motor imitation, and joint attention skills from an early age (e.g., points, hugs stuffed animals, affectionate, ‘‘looks’’ after name is called) in combination with the presence of problem behaviors (e.g., difficulty with adjusting to change, repetitive behavior, avoiding physical contact) will also be useful in the diagnostic process. Incorporating information obtained from the BITSEA ASD items may aid practitioners in diagnostic accuracy when evaluating young children for ASD. Although the BITSEA ASD screening items appear to capture ASD symptomatology above and beyond general developmental delays or behavior problems, selection of a single cut-off score proved difficult in our analysis. No cut score achieved recommended sensitivity and specificity values at either 12 months or 24 months. At 24 months, several reasonable raw cut scores are suggested (12 and 11 or higher) for detecting ASD risk, with greater scores indicating greater risk for ASD behaviors. 4.1. Implications for clinical practice The results of the present study indicate that the BITSEA may prove useful as a Level I screener for social emotional problems and a Level II screener for ASD symptoms. Due to the continuing high prevalence rates of children diagnosed with ASD there is a continuing need for measures that accurately identify the presence of behaviors associated with ASD. The BITSEA is a parent rating form that can be easily administered to parents at community sites and may be a useful tool for identifying behaviors early in development to facilitate a child’s diagnosis of ASD and access to early intervention services. Furthermore, the BITSEA may be incorporated into well child visits to screen for general and specific concerns related to ASD. Previous research has indicated a need for standard ASD specific screening instruments (Dosreis et al., 2006). The BITSEA represents a screening instrument that can act as both screener for social emotional problems in the general population as well as provide important developmental information regarding the presence of behavioral concerns specific to ASD. 4.2. Limitations of the study There are several limitations of this study that warrant consideration when interpreting findings. An inherent design limitation of this study is the non-representativeness of the sample. Participants in the study sample were predominantly African-American mother/child dyads which may limit generalizability of the current findings. Differences in ratings by racial and cultural groups for the BITSEA and M-CHAT are largely unexamined. Further research is needed to examine possible differences in scores obtained for these screening measures by race, culture, and socioeconomic status of respondents. Utilizing the M-CHAT screening measure as the criterion for determining presence of autism is problematic for several reasons. In the present study, the M-CHAT follow-up interview was not administered for failed items. The interview items are scored in the same manner as the M-CHAT and are used to identify children who are at increased risk for ASD (Robins, Fein, & Barton, 1999). Using the M-CHAT screening and follow-up interview in combination has been found to reduce false positives and increase its positive predictive value (Kleinman et al., 2008). Another limitation of the current study is the use of the M-CHAT as a criterion variable which does not allow for diagnostic validation of the findings that a ‘‘gold standard’’

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

599

autism assessment or a formal diagnosis of autism would provide. Furthermore, both instruments involve parent report and, therefore, share method variance which may have inflated correlations found between the two screeners. 4.3. Recommendations for future research To further validate the use of the BITSEA ASD screener, future research should examine the sensitivity and specificity of the BITSEA ASD screener with a formal diagnostic evaluation conducted to serve as the appropriate criterion. Secondly, additional long-term longitudinal design studies are needed to determine the predictive validity of ASD screening measures compared to ASD status by 5–6 years of age. To determine the extent to which the current study’s findings can be generalized to other populations, research is needed to examine the validity of the BITSEA ASD screener with other groups of children and parents. Finally, further field testing of specific cut-offs scores identified in the current study are needed for children at 24 months. 4.4. Summary As deficits in nonverbal social communication, social and emotional reciprocity and delayed speech/language skills are the diagnostic symptoms more often displayed by children 3 years and younger, it is critical to identify screening measures that accurately identify deficits in these areas that are core features of children with ASD. Overall, the results from this study provide initial support for practitioners’ use of the BITSEA ASD items to aid in assessing for behavior that demonstrates risk for ASD in young children. Further research with the BITSEA scales in detecting ASD risk is warranted. Acknowledgements The participants for this study were enrolled in a longitudinal cohort study, Conditions Affecting Neurocognitive Development and Learning in Early Childhood (CANDLE), which is funded by The Urban Child Institute in Shelby County, Tennessee. The authors would like to thank Bruce Keisling, Sacha Bliss, Christina Warner-Metzger, Colby Reed, Jessica Myszak, Kristin Hoffman, Pam Najera, Maureen Sorrels, Amy Scheck, & Wonsuk Yoo for their contributions to this research project. Without their dedication and that of the participants, this research would not be possible. References American Psychiatric Association. (2000). Diagnostic and statistical manual of mental of mental disorders (fourth edition, text revision). Washington, DC: Author. Autism and Developmental Disabilities Monitoring Network Surveillance Year 2008 Principal Investigators. (2008). Prevalence of Autism Spectrum Disorders— Autism and Developmental Disabilities Monitoring Network, 14 Sites, United States. Morbidity and Mortality Weekly Report, 61(SS-03), 1–19, http:// www.cdc.gov/mmwr/preview/mmwrhtml/ss6103a1.htm. Barbaro, J., & Dissanayake, C. (2009). Autism spectrum disorders in infancy and toddlerhood: A review of the evidence on early signs, early identification tools, and early diagnosis. Journal of Developmental and Behavioral Pediatrics, 30, 447–459. Berument, S. K., Rutter, M., Lord, C., Pickles, A., & Bailey, A. (1999). Autism screening questionnaire: Diagnostic validity. British Journal of Psychiatry, 175, 444–451. Briggs-Gowan, M. J., & Carter, A. S. (2007). Applying the Infant Toddler Social Emotional Assessment: ITSEA & Brief-ITSEA in early intervention. Infant Mental Health Journal, 28, 564–583. Briggs-Gowan, M. J., & Carter, A. S. (2006). BITSEA: Brief Infant Toddler Social and Emotional Assessment Examiner’s Manual. San Antonio: Pearson. Briggs-Gowan, M. J., Carter, A. S., Irwin, J. R., Wachtel, K., & Cicchetti, D. V. (2004). The Brief Infant-Toddler Social and Emotional Assessment: Screening for social– emotional problems and delays in competence. Journal of Pediatric Psychology, 29(2), 143–155. Carpenter, M., & Tomasello, M. (2000). Joint attention, cultural learning, and language acquisition: Implications for children with autism. In A. M. Wetherby & B. M. Prizant (Eds.), Autism spectrum disorders: A transactional developmental perspective. Communication and language intervention series (pp. 21–54). Baltimore, MD: Brookes Publishing. De Giacomo, A., & Fombonne, E. (1998). Parental recognition of developmental abnormalities in autism. European Child and Adolescent Psychiatry, 7, 131–136. Dosreis, S., Weiner, C. L., Johnson, L., & Newschaffer, C. J. (2006). Autism spectrum disorder screening and management practices among general pediatric providers. Developmental and Behavioral Pediatrics, 27, 88–94. Hanley, J. A., & McNeil, B. J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148, 839–843. Harris, S., & Handleman, J. (2000). Age and IQ at intake as predictors of placement for young children with autism: A four- to six-year follow-up. Journal of Autism and Developmental Disorders, 30, 137–142. Johnson, C. P., Myers, S. M., & the Council on Children with Disabilities, (2007). Identification and evaluation of children with autism spectrum disorders. Pediatrics, 120, 1183–1215. Karabekiroglua, K., Briggs-Gowan, M. J., Carter, A. S., Rodopman-Armand, A., & Akbasa, S. (2010). The clinical validity and reliability of the Brief Infant-Toddler Social and Emotional Assessment (BITSEA). Infant Behavior and Development, 33, 503–509. Kim, S. H., & Lord, C. (2010). Restricted and repetitive behaviors in toddlers and preschoolers with autism spectrum disorders Based on the Autism Diagnostic Observation Schedule (ADOS). Autism Research, 3, 162–173. Kim, S. H., & Lord, C. (2012). New autism diagnostic interview-revised algorithms for toddlers and young preschoolers from 12 to 47 months of age. Journal of Autism and Developmental Disorders, 42(1), 82–93. Kleinman, J. M., Robins, D. L., Ventola, P. E., Pandey, J., Boorstein, H. C., Esser, E. L., et al. (2008). The Modified Checklist for Autism in Toddlers: A follow-up study investigating the early detection of Autism Spectrum Disorders. Journal of Autism and Developmental Disorders, 38, 827–839. Kogan, M. D., Blumberg, S. J., Schieve, L. A., Boyle, C. A., Perrin, J. M., Ghandour, R. M., et al. (2009). Prevalence of parent-reported diagnosis of Autism Spectrum Disorder among children in the US, 2007. Pediatrics, 124, 1395–1403. McConnell, S. R. (2002). Interventions to facilitate social interaction for young children with autism: Review of available research and recommendations for educational intervention and future research. Journal of Autism and Developmental Disorders, 32, 351–372. Matson, J. K., Boisjoli, J. A., & Wilkins, J. (2007). The Baby and Infant Screen for Children with aUtIsm Traits (BISCUIT). Baton Rouge, LA: Disability Consultants, LLC. Matson, J. L., Kozlowski, A. M., Fitzgerald, M. E., & Sipes, M. (2013). True versus false positives and negatives on the Modified Checklist for Autism in Toddlers. Research in Autism Spectrum Disorders, 7, 17–22.

600

L.M. Gardner et al. / Research in Autism Spectrum Disorders 7 (2013) 591–600

Matson, J. L., Wilkins, J., & Gonzalez, M. (2008). Early identification and diagnosis in autism spectrum disorders in young children and infants: How early is too early. Research in Autism Spectrum Disorders, 2, 75–84. Matson, J. L., Wilkins, J., Sharp, B., Knight, C., Sevin, J. A., & Boisjoli, J. A. (2009). Sensitivity and specificity of the Baby and Infant Screen for Children with aUtIsm Traits. Validity and cutoff scores for autism and PDD-NOS in toddlers. Research in Autism Spectrum Disorders, 3, 924–930. Mundy, P., & Sigman, M. (2006). Joint attention social competence and developmental psychopathology. In D. Cicchetti & D. J. Cohen (Eds.), Developmental psychopathology: Vol. 1: Theory and method (pp. 293–332). Hoboken, NJ: John Wiley and Sons. Pierce, K., Carter, C., Weinfield, M., & Desmond, J. (2011). Detecting, studying, and treating autism early: The 1 year well baby check-up. Pediatrics, 159, 458–465. Robins, D. L., & Dumont-Mathieu, T. M. (2006). Early screening for Autism Spectrum Disorders: Update on the Modified Checklist for Autism in Toddlers and other measures. Developmental and Behavioral Pediatrics, 27, 111–119. Robins, D. L., Fein, D., & Barton, M. L. (1999). Follow-up Interview for the Modified Checklist for Autism in Toddlers (M-CHAT FUI). Self-published. Robins, D. L., Fein, D., Barton, M., & Green, J. (2001). The Modified-Checklist for Autism in Toddlers (M-CHAT): An initial investigation in the early detection of Autism and Pervasive Developmental Disorders. Journal of Autism and Developmental Disorders, 31, 131–144. Snow, A., & Lecavalier, L. (2008). Sensitivity and specificity of the Modified Checklist of Autism in Toddlers and the Social Communication Questionnaire in preschoolers suspected of having pervasive developmental disorders. Autism, 12, 627–644. Stone, W. L., Coonrod, E. E., & Ousley, O. Y. (2000). Brief report: Screening tool for Autism in Two-Tear-Olds (STAT): Development and preliminary data. Journal of Autism and Developmental Disorders, 30, 607–612. Stone, W. L., Lee, E. B., Ashford, L., Brissie, J., Hepburn, S. L., Coonrod, E. E., et al. (1999). Can autism be diagnosed accurately in children under three years? Journal of Child Psychology and Psychiatry, 40, 219–226. Swets, J. A. (1998). Measuring the accuracy of diagnostic systems. Science, 240, 1285–1293. Webster, A., Feiler, A., & Webster, V. (2003). Early intensive family intervention and evidence of effectiveness: Lessons from the south west autism programme. Early Child Development and Care, 173, 383–398. Woods, J. J., & Wetherby, A. M. (2003). Early identification of and intervention for infants and toddlers who are at risk for autism spectrum disorder. Language, Speech and Hearing Services in Schools, 34, 180–193.