- Email: [email protected]
The factor structure of the Social Interaction Anxiety Scale and the Social Phobia Scale
Journal of Anxiety Disorders 25 (2011) 579–583
Contents lists available at ScienceDirect
Journal of Anxiety Disorders
The factor structure of the Social Interaction Anxiety Scale and the Social Phobia Scale Thomas Heidenreich a,∗ , Karin Schermelleh-Engel b , Elisabeth Schramm c , Stefan G. Hofmann d , Ulrich Stangier e a
Faculty of Social Work, Health Care and Nursing Sciences, University of Applied Sciences Esslingen, Flandernstrasse 101, 73732 Esslingen am Neckar, Germany Department of Psychology, Unit of Psychological Research Methods and Evaluation, Johann Wolfgang Goethe-University of Frankfurt/Main, Mertonstrasse 17, 60054 Frankfurt/Main, Germany c Department of Psychiatry and Psychotherapy, University of Freiburg Medical Center, Hauptstr. 5, 79104 Freiburg, Germany d Department of Psychology, Boston University, 648 Beacon Street, Boston, MA 02215, USA e Department of Psychology, Unit of Clinical Psychology and Psychotherapy, Johann Wolfgang Goethe-University of Frankfurt/Main, Varrentrappstr. 40-42, 60486 Frankfurt/Main, Germany b
a r t i c l e
i n f o
Article history: Received 5 January 2010 Received in revised form 22 January 2011 Accepted 24 January 2011 Keywords: Social anxiety disorder Social phobia Confirmatory factor analysis Multi-sample analysis Interaction anxiety Performance anxiety
a b s t r a c t The Social Interaction Anxiety Scale (SIAS) and the Social Phobia Scale (SPS) are two compendium measures that have become some of the most popular self-report scales of social anxiety. Despite their popularity, it remains unclear whether it is necessary to maintain two separate scales of social anxiety. The primary objective of the present study was to examine the factor analytic structure of both measures to determine the factorial validity of each scale. For this purpose, we administered both scales to 577 patients at the beginning of outpatient treatment. Analyzing both scales simultaneously, a CFA with two correlated factors showed a better fit to the data than a single factor model. An additional EFA with an oblique rotation on all 40 items using the WLSMV estimator further supported the two factor solution. These results suggest that the SIAS and SPS measure similar, but not identical facets of social anxiety. Thus, our findings provide support to retain the SIAS and SPS as two separate scales. © 2011 Elsevier Ltd. All rights reserved.
Although social anxiety disorder (SAD) is considered a diagnostic entity, social anxiety is a multifaceted and multidimensional construct (for a review, see Hofmann, Heinrichs, & Moscovitch, 2004). Patients with this disorder fear and avoid a range of different social situations to different degrees, and self-report instruments have to account for the heterogeneity of individuals who receive this diagnostic label. Numerous attempts have been made to classify the feared situations into distinctive domains. Most of the classifications comprised two or three types of situations, including performance and public speaking, interaction, and being observed while performing acts such as writing or eating. (Hofmann et al., 2004). Such categorization might be a useful basis for the specification of different subtypes of SAD. Currently, the DSM-V Work Group on Anxiety found strong support for the definition of a subtype “predominantly performance” and some evidence for another subtype “fear of showing anxiety symptoms” (e.g., blushing), whereas most patients suffer from fears across domains, including interaction anxiety (Bögels et al., 2010). As a consequence, to investigate the diagnostic and etiological relevance of
∗ Corresponding author. Tel.: +49 711 397 4575; fax: +49 711 397 4595. E-mail address: [email protected] (T. Heidenreich). 0887-6185/$ – see front matter © 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.janxdis.2011.01.006
situational domains, assessment instruments are needed which discriminate between social fears in performance and interaction situations. Two of the most commonly used self-report instruments for measuring social anxiety are the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) and the Social Phobia Scale (SPS; Mattick & Clarke, 1998). The purpose of the latter one is to measure fears of being scrutinized during activities and performance tasks, whereas the SIAS was created to assess fears of more general social interactions (Mattick & Clarke, 1998). While the SIAS was not specifically designed for individuals with a clinical diagnosis of social anxiety disorder, the SPS was specifically developed for this population. However, studies that have used the SPS in social phobic samples and non-social phobic samples yielded with very similar psychometric properties and factor structures (e.g., Carleton et al., 2009). Research has yielded broad support for the reliability and validity of both the SIAS and the SPS. The two scales are highly internally consistent and show good retest-reliability (Heimberg, Mueller, Holt, Hope, & Liebowitz, 1992; Mattick & Clarke, 1998). In addition, the SIAS and SPS are able to discriminate between individuals with SAD and healthy controls (Heimberg et al., 1992) as well as between patients with SAD and those with other forms of anxiety disorders (Mattick & Clarke, 1998; Stangier, Heidenreich,
580
T. Heidenreich et al. / Journal of Anxiety Disorders 25 (2011) 579–583
Berardi, Golbs, & Hoyer, 1999). No significant correlation was found between self-reported SAD as measured by SIAS and SPS and social desirability (Heimberg et al., 1992; Mattick & Clarke, 1998). Furthermore, Heimberg et al. (1992) found that the SIAS correlated more highly with a measure of interaction anxiety than with a measure of performance anxiety, whereas the SPS was only correlated with a measure of performance anxiety. In a similar fashion and based on behavioral assessment tests, Ries et al. (1998) observed that the SIAS was correlated with verbal reports regarding positive and negative thoughts in speech and conversation situations while the SPS was only related to verbal reports in the speech task (negative correlation with speech duration). Brown et al. (1997) have argued that the SIAS and SPS constitute subscales of a higher-order construct social anxiety. However, despite these positive psychometric characteristics, the assumption of two underlying dimensions for the SIAS and SPS (Social Interaction Anxiety and Social Performance Anxiety, respectively) has yet to be adequately addressed. All published reports have found high to very high intercorrelations between the SIAS and SPS (Brown et al., 1997: r = .72; Heimberg et al., 1992: r = .41 social phobic group, r = .89 community sample; Peters, 2000: r = .73; Ries et al., 1998: r = .66; Stangier et al., 1999: r = .78 social phobic group, r = .69 clinical control group). While the amount of explained variance evidently varies according to the respective sample, there is clear evidence of a large amount of shared variance. Furthermore, there is little evidence that the SIAS and SPS differ significantly with respect to important clinical domains, such as treatment sensitivity (Stangier, Schramm, Heidenreich, Berger, & Clark, in press). So far, only one study has investigated the joint factor structure of the SIAS and the SPS (Safren, Turk, & Heimberg, 1998). In this study, a confirmatory factor analysis (CFA) failed to adequately support the hypothesis of two distinct constructs. However, in light of the very large number of parameters (81) to be estimated in their CFA comprising all 40 SIAS and SPS items, the sample size of N = 167 might have simply been too small for these tests (cf. Muthén & Muthén, 2002). While a large number of degrees of freedom may compensate for a small sample size (MacCallum, Browne, & Sugawara, 1996), power may have been too low to obtain precise parameter estimates. In the same trial (Safren et al., 1998), a three-factor solution was obtained using exploratory factor analysis (EFA) with varimax rotation. The first factor consisted of 17 SIAS items (“interaction anxiety”), the second factor of 11 SPS items (“being observed by others”), and the third factor of 5 SPS Items (“fear that others will notice anxiety symptoms”); 3 SIAS and 4 SPS items were eliminated due to high cross-loadings. More recently, two reports (Rodebaugh, Woods, & Heimberg, 2007; Rodebaugh, Woods, Heimberg, Liebowitz, & Schneier, 2006) questioned the factorial structure of the SIAS. Results of CFAs from both studies provide strong evidence that there are systematic differences between responses to negatively worded items in comparison to positively worded items of the scale. In addition, the negatively worded items showed consistently weaker relationships with a variety of comparison measures. The authors concluded that only the 17 positively worded items of the SIAS should be used. Given the high intercorrelations found in all studies employing the SIAS and SPS and the problem that previous studies predominantly applied EFA for construct validation, evidence to date does not allow firm conclusions regarding the factorial structures of SIAS and SPS. It also remains unclear whether both scales assess the same or different underlying constructs. Hence, the primary objective of the present study was to examine whether the existence of two separate scales is justified by investigating the latent factor structures of the SIAS and the SPS using CFA. We further examined possible response pattern biases due to similar wording.
1. Materials and methods 1.1. Participants and instruments The sample consisted of 577 patients (318 women, 256 men, 3 individuals with missing data) with a mean age of 38.73 years (SD = 10.94) who sought treatment at the behavior therapy outpatient clinic of the Goethe University in Frankfurt, Germany. Fifty-five percent of participants (N = 315) received a principal diagnosis of SAD according to DSM-IV (American Psychiatric & Association, 1994), while 45% (N = 262) were diagnosed with other non-psychotic disorders (anxiety disorders other than SAD, depression, or somatoform disorders) with no comorbid SAD. Diagnoses were made using the Structured Clinical Interview for DSM-IV (SCID-IV, First, Williams, & Spitzer, 1997). Comorbid diagnoses were permitted, with the exception of bipolar disorders, psychotic disorders, and drug or alcohol dependence. All clinical interviews and measures were administered by Ph.D. clinical psychologists or students in postgraduate clinical training. Raters received an intense training in administering clinical interviews (2-day training and follow-up) and interviews were closely supervised by T.H. and U.S. Self-rated measures comprised German translations of the SIAS (Stangier et al., 1999) and the SPS; the German versions of both scales showed very similar psychometric characteristics as their original English versions (Stangier et al., 1999). 1.2. Statistical analyses 1.2.1. Missing data Our sample included 18 cases (3.12% of N = 577) for which missing values were imputed using the estimation maximization (EM) algorithm of the PRELIS 2.54 program (Jöreskog & Sörbom, 1996; see also Du Toit & Du Toit, 2001). As we employed different analyses on the same data (i.e., confirmatory factors analysis, exploratory factor analyses, and logistic regression), we were not able to apply a full information maximum likelihood approach, since this method simultaneously estimates missing data and model parameters (cf. Schafer, 1997; Schafer & Graham, 2002). 1.2.2. Factor analyses Using CFA to assess the factorial structure of both scales separately, we tested whether (1) all SIAS items loaded onto one common factor (Model 1a) and (2) whether all SPS items loaded onto one common factor (Model 2a). In view of the fact that both scales measure quite heterogeneous constructs, we tested several hypothesized multidimensional structures of both scales: Regarding the SIAS, we further tested whether the negatively worded items of this scale also load onto a method (residual) factor (Model 1b) or whether negatively worded and positively worded items measure two method (residual) factors in addition to the common social interaction construct (Model 1c; as suggested by Rodebaugh et al., 2007). Additionally, we also investigated the one-factor solution (Model 1d) suggested by Safren et al. (1998). Regarding the SPS, we further tested Safren et al.’s (1998) twodimensional structure (Model 2b) with 11 items measuring “being observed by others” and 5 Items measuring “fear that others will notice anxiety symptoms”; 4 items that had cross-loadings on the SIAS or the SPS scales had been eliminated according to Safren et al.’s (1998) recommendation. In order to examine the factorial structure of both scales simultaneously, we tested whether all SIAS and SPS items loaded onto one single common factor (Model 3a) or onto two correlated factors (Model 3b). We further tested Safren et al.’s (1998) three factor model by combining Model 1a and Model 2b (Model 3c). Finally, we employed EFA with an oblique rotation on all 40 items. All CFA and EFA models were fitted to polychoric correlations using the robust weighted least squares mean and
T. Heidenreich et al. / Journal of Anxiety Disorders 25 (2011) 579–583
variance adjusted estimator (WLSMV) appropriate for categorical level data of the Mplus program (version 6.1, Muthén & Muthén, 1998–2010). Several goodness of fit statistics representing different classes of fit indices are reported as recommended by Bollen and Long (1993): The likelihood-ratio 2 test and its associated p-value, the rootmean square error of approximation (RMSEA, Steiger, 1990), the comparative fit index (CFI, Bentler, 1990), the Tucker–Lewis index, also known as non-normed fit index (TLI, Bentler & Bonett, 1980; Tucker & Lewis, 1973), and the weighted root mean-square residual (WRMR, Muthén, 1998–2004). Good model fit was indicated by a nonsignificant 2 value, RMSEA ≤ .06 (Hu & Bentler, 1999), CFI and TLI ≥ .97 (Schermelleh-Engel, Moosbrugger, & Müller, 2003), and WRMR ≤ .95 (Yu, 2002). 1.2.3. Logistic regression A logistic regression analysis was performed to examine the relative contribution of both scales to the prediction of group membership. For this analysis, the sample was divided into two subsamples, the first group (SAD) consisting of patients with principal diagnoses of SAD (N = 315) and the second group (control) consisting of patients who were diagnosed with anxiety disorders other than SAD, depression, or somatoform disorders (N = 262). The dependent variable was coded as 0 for controls and 1 for SAD. 2. Results 2.1. Confirmatory factor analyses The dimensionality of the SIAS and SPS was initially tested separately using the WLSMV estimator of the Mplus program. All negatively worded items were recoded before conducting the analyses. Fit indices are listed in Table 1. First, we tested dimensionality of the SIAS items. In the first CFA model, all 20 items of the SIAS loaded onto one common construct (Model 1a). Results indicated that all items including the reverse-coded negatively worded items correlated positively with the respective factor. In accordance with Rodebaugh et al.’s findings (2007), our results also revealed that the negatively worded items had the lowest factor loadings of all 20 items with parameter estimates of .31 (SIAS 5), .53 (SIAS 9), and .53 (SIAS 11). As the data did not fit the model, we tested in Model 1b whether in addition to the loadings on the common factor the three negatively worded items loaded onto an uncorrelated method (residual) factor (cf. Rodebaugh et al., 2007). The model fit improved, but CFI < .97 and WRMR > .95 still indicated that there was evidence of model misfit. As suggested by Rodebaugh et al. (2007), we also tested a model (Model 1c) with one substantive factor and two uncorrelated method (residual) factors (the first method factor consisting of three negatively worded items and the second method factor consisting of 17 positively worded items). The model fit now improved significantly compared with the common factor model (Model 1a) and the model with one method factor (Model 1b). Although statistical comparisons via chi-square difference tests were not possible because (a) the chi-square value for WLSMV cannot be used for chi-square difference testing in the regular way (Muthén & Muthén, 1998–2010) and (b) these models were not hierarchically nested, the model employing one substantive factor and two method factors (Model 1c) fitted the data best (CFI = .97, TLI = .96, WRMR = 1.22). However, the RMSEA (.085) gave evidence that some model misfit remained which may be due to error covariances not included in the model. In Model 1c, five items had relatively high loadings on the positively worded method factor: SIAS12 (“I worry about expressing myself in case I appear awkward”), SIAS13 (“I find it difficult to disagree with another’s point of view”), SIAS15 (“I find myself wor-
581
rying that I won’t know what to say in social situations”), SIAS17 (“I feel I’ll say something embarrassing when talking”), and SIAS18 (“When mixing in a group, I find myself worrying I will be ignored”) with standardized loadings on the method factor of .62, .40, .47, .66, and .42, respectively. These loadings may be interpreted as a method bias due to similar item wording, because these items all include either the word “worry” or the words “I find”/”I feel.” Additionally, we tested another common factor model as suggested by Safren et al. (1998). In this model (Model 1d), 17 items loaded onto a common factor, while items 1, 8, and 12 were eliminated. As the fit criteria show, the model fit was not acceptable. Internal consistencies for SIAS-20 and SIAS-17 were very similar (.94 vs. .93) and, as can be expected, both scales were very highly intercorrelated (r = .97). Furthermore, the results showed that the negatively worded items, again, had the lowest factor loadings. Second, we examined dimensionality of the SPS items. In the first CFA model, all 20 items of the SPS loaded onto one common construct (Model 2a). Results indicated that all items correlated positively with the respective factor (see Table 1); only two SPS items had factor loadings lower than .50 (SPS 2 and SPS 9). As suggested by Safren et al. (1998), we also tested a model (Model 2b) with two correlated factors (factor 1: items 3, 5, 7, 18, 19; factor 2: 11 items; items 6, 14, 15, and 20 were deleted). Results showed that data fitted the model slightly better than Model 2a, but again these could not be compared because they were not nested and they used a different number of indicators. In Model 2b, both factors correlated very high (r = .84) raising doubts that they both measure different constructs. Third, we examined dimensionality of all SIAS and SPS items simultaneously. In Model 3a, all 40 items of SIAS and SPS loaded onto a single factor. As the fit criteria show, the model fit was not acceptable. With Model 3b we tested a model with two correlated factors. In this model, all 20 items of the SIAS loaded onto a single factor and all 20 items of the SPS loaded onto a second factor. Results indicated that the items correlated positively with the respective factor; only two SPS items had factor loadings lower than .50 (SPS 2 and SPS 9). Both factors were substantially correlated (r = .84). A replication of Safren et al.’s 3-factor solution in Model 3c (a combination of Model 1a and Model 2b) did not substantially improve the model fit. All three factors were highly correlated with coefficients of .80 or above. Overall, results of our CFAs indicate that the SIAS does not have a straightforward unidimensional structure. The model employing two method factors and one common factor appears to provide a much better fit than the other models. We also did not find a clear indication of a unidimensional structure for the SPS items, but loadings on the common factor were quite high indicating that all items measure the same construct. Given the failure of the CFA to provide a good fit to the data, we additionally employed EFA with an oblique rotation on all 40 items using the WLSMV estimator of the Mplus program. Results suggested retaining either two factors (Kaiser criterion) or four factors (parallel analysis). In the 2-factor solution (see Table 2) the first factor consisted only of SIAS items and the second only of SPS items. Three SPS items (3, 5, 18) had cross loadings on the SIAS (of about the same size as on the SPS factor): “I can suddenly become aware of my own voice of others listening to me” (SPS3), “I fear I may blush when I am with others” (SPS5), and “I can get tense when I speak in front of other people” (SPS18). Both factors of the 2-factor solution were substantially correlated with r = .69. The cross-loadings on the SIAS factor indicate an overlap of item contents with regard to fear that others will notice anxiety symptoms. The 4-factor solution was not interpretable because of too many high cross loadings. In sum, results from the CFA and EFA suggested that SIAS and SPS each measure a single construct but that two method factors of the SIAS exist, which EFA could not detect.
582
T. Heidenreich et al. / Journal of Anxiety Disorders 25 (2011) 579–583
Table 1 Goodness-of-fit indices of SIAS and SPS confirmatory factor models.
SIAS 1a 1b
1c
1d SPS 2a 2b
SIAS and SPS 3a 3b 3c
Model
2
df
p
RMSEA
CFI
TLI
WRMR
1 common factor (20 items) 1 common factor (20 items), 1 method factor (3 reverse-coded items; cf. Rodebaugh et al., 2007) 1 common factor (20 items), 2 method factors (3 reverse-coded items, 17 positively coded items; cf. Rodebaugh et al., 2007) 1 common factor (17 items; items 1, 8, 12 deleted; cf. Safren et al., 1998)
1428.865 1313.219
170 167
.00 .00
0.113 0.109
0.937 0.943
0.930 0.935
1.887 1.797
775.389
150
.00
0.085
0.969
0.960
1.219
988.125
119
.00
0.113
0.944
0.936
1.713
1 common factor (20 items) 2 correlated factors (factor 1: 5 items – “fear that others will notice anxiety symptoms”; factor 2: 11 items – “being observed by others”; items 6, 14, 15, 20 deleted; cf. Safren et al., 1998)
984.637 579.386
170 103
.00 .00
0.091 0.090
0.945 0.947
0.939 0.939
1.579 1.448
1 common factor (40 items) 2 correlated factors (a SIAS and a SPS factor) 3 factors (Safren et al., 1998), combination of Model 1a and Model 2b
3892.887 2778.910
740 739
.00 .00
0.086 0.069
0.904 0.938
0.899 0.935
2.031 1.655
2314.353
591
.00
0.071
0.938
0.933
1.634
Note. RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker–Lewis incremental fit index; WRMR = weighted root mean square residual.
Table 2 Factor loadings of SIAS and SPS items (exploratory factor analysis with oblique rotation).
SIAS1 SIAS2 SIAS3 SIAS4 SIAS5 SIAS6 SIAS7 SIAS8 SIAS9 SIAS10 SIAS11 SIAS12 SIAS13 SIAS14 SIAS15 SIAS16 SIAS17 SIAS18 SIAS19 SIAS20 SPS1 SPS2 SPS3 SPS4 SPS5a SPS6 SPS7 SPS8 SPS9a SPS10 SPS11a SPS12 SPS13 SPS14 SPS15 SPS16 SPS17 SPS18 SPS19 SPS20 a
Factor 1 Social Interaction Anxiety
Factor 2 Social Performance Anxiety
0.628 0.633 0.514 0.601 0.465 0.608 0.724 0.696 0.586 0.972 0.766 0.830 0.634 0.746 0.856 0.729 0.772 0.712 0.768 0.548 0.188 0.019 0.328 0.143 0.326 0.385 0.013 0.065 −0.333 −0.085 0.104 0.276 0.064 −0.070 0.262 0.119 −0.002 0.473 −0.085 0.355
0.118 0.143 0.128 0.079 −0.200 0.195 0.156 0.119 −0.077 −0.184 −0.316 −0.004 0.069 −0.002 −0.024 0.119 0.088 −0.013 0.145 0.066 0.408 0.480 0.251 0.679 0.367 0.520 0.729 0.796 0.789 0.815 0.676 0.556 0.717 0.776 0.601 0.728 0.824 0.408 0.714 0.568
Reverse-coded items.
2.2. Logistic regression To determine whether both SIAS and SPS scales contribute to the prediction of group membership, a logistic regression analysis was performed. A test of the model with both scales against a constantonly model was statistically significant (Model 2 (2df) = 167.00, p < .01) indicating that the scales, as a set, reliably distinguish the SAD group from the control group. Logistic regression also allows for the assessment of each of the scales to the prediction of group membership. The coefficients (ˇ) give the change in the logit for a one standard deviation change in the predictors. The coefficients are “semi-standardized,” only standardized on the predictor side. The results indicated that both scales were significant predictors of group membership (SIAS: ˇ = .83, SE = .15, p < .01; SPS: ˇ = .56, SE = .16, p < .01). Nagelkerke’s R2 was .34, and overall the model classified 74.2% of the patients correctly (70.2% of the patients in the control group and 77.5% of the patients in the SAD group). Logistic regression shows that both scales contribute independently to the prediction of group membership. 3. Discussion In order to examine whether it is useful to retain the SIAS and the SPS as two separate, but highly correlated instruments of social anxiety, we employed CFAs to study the factor structure of these two scales. Our analyses revealed that variance of the SIAS and variance of the SPS can each be largely accounted for by a single underlying factor. However, good model fit of the SIAS could only be achieved by including two method factors. These findings support recent research of Rodebaugh et al. (2006) as well as Rodebaugh et al. (2007) who also found that the loadings for the negatively worded items were substantially lower than the loadings for all other items. Negatively worded items are often used to reduce the potential effects of response pattern biases. Research has shown that they may produce artifactual response factors consisting exclusively of negatively worded items (cf. Podsakoff, MacKenzie, Lee, & Podsakoff, 2003). Our results indicate that future research should eliminate the negatively worded items because they do not measure the common construct as reliable as the other items. In addition to the negatively worded items, similarly worded
T. Heidenreich et al. / Journal of Anxiety Disorders 25 (2011) 579–583
items may also produce method bias. Some of these items of the SIAS include “I worry” or “I find myself worrying.” However, a separate factor consisting of these items loaded highly onto the common factor and, therefore, did not support our assumption that these items measure mainly method effects. Analyzing both scales simultaneously, a CFA with two correlated factors showed a better fit with the data than a single factor model. As model fit was not quite satisfactory, we additionally employed an EFA with an oblique rotation on all 40 items. A 2-factor solution seemed to explain the data best, indicating that both SIAS and SPS do not measure the same construct but rather, two distinct constructs. The high correlation of r = .69 may also suggest that both scales represent facets of a higher-order construct. In sum, our findings provide support to retain the SIAS and SPS as two separate scales. A logistic regression analysis supported this finding as both scales contributed independently to the prediction of group membership. Our results are also in line with studies reporting differential correlations of the SIAS and SPS with other constructs. The work by Heimberg et al. (1992), for example, showed the SIAS to be significantly more strongly related to a measure of interaction anxiety than a measure of performance anxiety. However, they are in contrast to the psychometric results reported by Safren et al. (1998). Possible explanations for this discrepancy may be that (1) our analyses were based on a larger sample, and (2) we used CFA together with the WLSMV estimator of the Mplus program appropriate for the analysis of categorical data. The following limitations of the present examination should be noted. First, due to the hypothesis-testing nature of CFA, we were only able to compare one and two-factor models. However, as noted by Safren et al. (1998), it is possible that the variance of situations relevant to the measurement of SAD is explained by more than 2 factors. Second, further studies should investigate whether the number of items of the two scales can be reduced without a loss of reliability or validity. A smaller number of items may represent the constructs equally well as the current 20 items and at the same time may reduce possible method effects. Third, due to different methods of factor analyses, criteria for factor selection, rotation methods and samples in previous studies, it is unclear whether measurement invariance of the SIAS and SPS exist between American and German samples. Fourth, due to the fact that this study stresses factorial validity of SIAS/SPS, no further conclusions can be drawn with regard to correlations with other measures (convergent and discriminant) and therapy sensitivity. Further studies should address these important questions. Despite these limitations, results of this study add to the current literature on the measurement of social anxiety. We conclude that the SIAS and SPS are two correlated instruments, each measuring related, but separate, facets of social anxiety in patients with SAD. Author note Research was performed at the Department of Psychology, Department of Clinical Psychology and Psychotherapy, University of Frankfurt/Main, Varrentrappstr. 40-42, 60486 Frankfurt am Main, Germany. Acknowledgements Dr. Hofmann is a paid consultant by Schering-Plough and supported by NIMH grant MH078308. References American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.
583
Bentler, P. M. (1990). Comparative fix indexes in structural models. Psychological Bulletin, 107, 238–246. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606. Bögels, S. M., Alden, L., Beidel, D. C., Clark, L. A., Pine, D. S., Stein, M. B., & Voncken, M. (2010). Social anxiety disorder: questions and answers. Depression and Anxiety, 27, 168–189. Bollen, K. A., & Long, J. S. (Eds.). (1993). Testing structural equation models. Newbury Park, CA: Sage. Brown, E. J., Turovsky, J., Heimberg, R. G., Juster, H. R., Brown, T. A., & Barlow, D. H. (1997). Validation of the Social Interaction Anxiety Scale and the Social Phobia Scale across the anxiety disorders. Psychological Assessment, 9, 21–27. Carleton, R. N., Collimore, K. C., Asmundson, G. J., McCabe, R. E., Rowa, K., & Antony, M. M. (2009). Refining and validating the Social Interaction Anxiety Scale and the Social Phobia Scale. Depression and Anxiety, 26, E71–81. Du Toit, M., & Du Toit, S. (2001). Interactive LISREL: user’s guide (with a contribution by D. Hawkins). Chicago, IL: Scientific Software International. First, M. B., Williams, J. B., & Spitzer, R. L. (1997). Structured clinical interview for DSM-IV axis I disorders (SCID-I). Washington: American Psychiatric Association. Heimberg, R. G., Mueller, G. P., Holt, C. S., Hope, D. A., & Liebowitz, M. R. (1992). Assessment of anxiety in social interaction and being observed by others: the Social Interaction Anxiety Scale and the Social Phobia Scale. Behavior Therapy, 23, 53–75. Hofmann, S. G., Heinrichs, N., & Moscovitch, D. A. (2004). The nature and expression of social phobia: toward a new classification. Clinical Psychology Review, 24, 769–797. Hu, L., & Bentler, P. M. (1999). Cutoff criterion for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. Jöreskog, K. G., & Sörbom, D. (1996). PRELIS 2: user’s reference guide. Chicago, IL: Scientific Software International. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149. Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455–470. Muthén, B. O. (1998–2004). Mplus technical appendices. Los Angeles, CA: Muthén & Muthén. Muthén, L. K., & Muthén, B. O. (1998–2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén. Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo Study to decide on sample size and determine power. Structural Equation Modeling, 9, 599–620. Peters, L. (2000). Discriminant validity of the Social Phobia and Anxiety Inventory (SPAI), the Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS). Behaviour Research and Therapy, 38, 943–950. Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903. Ries, B. J., McNeil, D. W., Boone, M. L., Turk, C. L., Carter, L. E., & Heimberg, R. G. (1998). Assessment of contemporary social phobia verbal report instruments. Behaviour Research and Therapy, 36, 983–994. Rodebaugh, T. L., Woods, C. M., & Heimberg, R. G. (2007). The reverse of social anxiety is not always the opposite: the negatively worded items of the social interaction anxiety scale do not belong. Behavior Therapy, 38, 192–206. Rodebaugh, T. L., Woods, C. M., Heimberg, R. G., Liebowitz, M. R., & Schneier, F. R. (2006). The factor structure and screening utility of the Social Interaction Anxiety Scale. Psychological Assessment, 18, 231–237. Safren, S. A., Turk, C. L., & Heimberg, R. G. (1998). Factor Structure of the Social Interaction Anxiety Scale and the Social Phobia Scale. Behaviour Research and Therapy, 36, 443–453. Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall. Schafer, J. L., & Graham, J. W. (2002). Missing data: our view of the state of the art. Psychological Methods, 7, 147–177. Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8, 23–74. Stangier, U., Heidenreich, T., Berardi, A., Golbs, U., & Hoyer, J. (1999). Die Erfassung sozialer Phobie durch die Social Interaction Anxiety Scale (SIAS) und die Social Phobia Scale (SPS) [Assessment of social phobia with the Social Interaction Anxiety Scale (SIAS) and Social Phobia Scale (SPS)]. Zeitschrift für Klinische Psychologie, 28, 28–36. Stangier, U., Schramm, Heidenreich, T., Berger, M. & Clark, D. M. Cognitive Therapy versus Interpersonal Psychotherapy in Social Phobia: a randomized controlled trial. Archives of General Psychiatry, in press. Steiger, J. H. (1990). Structural model evaluation and modification: an interval estimation approach. Multivariate Behavioral Research, 25, 173–180. Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. Yu, C. Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Unpublished doctoral dissertation, University of California, Los Angeles.
Contents lists available at ScienceDirect
Journal of Anxiety Disorders
The factor structure of the Social Interaction Anxiety Scale and the Social Phobia Scale Thomas Heidenreich a,∗ , Karin Schermelleh-Engel b , Elisabeth Schramm c , Stefan G. Hofmann d , Ulrich Stangier e a
Faculty of Social Work, Health Care and Nursing Sciences, University of Applied Sciences Esslingen, Flandernstrasse 101, 73732 Esslingen am Neckar, Germany Department of Psychology, Unit of Psychological Research Methods and Evaluation, Johann Wolfgang Goethe-University of Frankfurt/Main, Mertonstrasse 17, 60054 Frankfurt/Main, Germany c Department of Psychiatry and Psychotherapy, University of Freiburg Medical Center, Hauptstr. 5, 79104 Freiburg, Germany d Department of Psychology, Boston University, 648 Beacon Street, Boston, MA 02215, USA e Department of Psychology, Unit of Clinical Psychology and Psychotherapy, Johann Wolfgang Goethe-University of Frankfurt/Main, Varrentrappstr. 40-42, 60486 Frankfurt/Main, Germany b
a r t i c l e
i n f o
Article history: Received 5 January 2010 Received in revised form 22 January 2011 Accepted 24 January 2011 Keywords: Social anxiety disorder Social phobia Confirmatory factor analysis Multi-sample analysis Interaction anxiety Performance anxiety
a b s t r a c t The Social Interaction Anxiety Scale (SIAS) and the Social Phobia Scale (SPS) are two compendium measures that have become some of the most popular self-report scales of social anxiety. Despite their popularity, it remains unclear whether it is necessary to maintain two separate scales of social anxiety. The primary objective of the present study was to examine the factor analytic structure of both measures to determine the factorial validity of each scale. For this purpose, we administered both scales to 577 patients at the beginning of outpatient treatment. Analyzing both scales simultaneously, a CFA with two correlated factors showed a better fit to the data than a single factor model. An additional EFA with an oblique rotation on all 40 items using the WLSMV estimator further supported the two factor solution. These results suggest that the SIAS and SPS measure similar, but not identical facets of social anxiety. Thus, our findings provide support to retain the SIAS and SPS as two separate scales. © 2011 Elsevier Ltd. All rights reserved.
Although social anxiety disorder (SAD) is considered a diagnostic entity, social anxiety is a multifaceted and multidimensional construct (for a review, see Hofmann, Heinrichs, & Moscovitch, 2004). Patients with this disorder fear and avoid a range of different social situations to different degrees, and self-report instruments have to account for the heterogeneity of individuals who receive this diagnostic label. Numerous attempts have been made to classify the feared situations into distinctive domains. Most of the classifications comprised two or three types of situations, including performance and public speaking, interaction, and being observed while performing acts such as writing or eating. (Hofmann et al., 2004). Such categorization might be a useful basis for the specification of different subtypes of SAD. Currently, the DSM-V Work Group on Anxiety found strong support for the definition of a subtype “predominantly performance” and some evidence for another subtype “fear of showing anxiety symptoms” (e.g., blushing), whereas most patients suffer from fears across domains, including interaction anxiety (Bögels et al., 2010). As a consequence, to investigate the diagnostic and etiological relevance of
∗ Corresponding author. Tel.: +49 711 397 4575; fax: +49 711 397 4595. E-mail address: [email protected] (T. Heidenreich). 0887-6185/$ – see front matter © 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.janxdis.2011.01.006
situational domains, assessment instruments are needed which discriminate between social fears in performance and interaction situations. Two of the most commonly used self-report instruments for measuring social anxiety are the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) and the Social Phobia Scale (SPS; Mattick & Clarke, 1998). The purpose of the latter one is to measure fears of being scrutinized during activities and performance tasks, whereas the SIAS was created to assess fears of more general social interactions (Mattick & Clarke, 1998). While the SIAS was not specifically designed for individuals with a clinical diagnosis of social anxiety disorder, the SPS was specifically developed for this population. However, studies that have used the SPS in social phobic samples and non-social phobic samples yielded with very similar psychometric properties and factor structures (e.g., Carleton et al., 2009). Research has yielded broad support for the reliability and validity of both the SIAS and the SPS. The two scales are highly internally consistent and show good retest-reliability (Heimberg, Mueller, Holt, Hope, & Liebowitz, 1992; Mattick & Clarke, 1998). In addition, the SIAS and SPS are able to discriminate between individuals with SAD and healthy controls (Heimberg et al., 1992) as well as between patients with SAD and those with other forms of anxiety disorders (Mattick & Clarke, 1998; Stangier, Heidenreich,
580
T. Heidenreich et al. / Journal of Anxiety Disorders 25 (2011) 579–583
Berardi, Golbs, & Hoyer, 1999). No significant correlation was found between self-reported SAD as measured by SIAS and SPS and social desirability (Heimberg et al., 1992; Mattick & Clarke, 1998). Furthermore, Heimberg et al. (1992) found that the SIAS correlated more highly with a measure of interaction anxiety than with a measure of performance anxiety, whereas the SPS was only correlated with a measure of performance anxiety. In a similar fashion and based on behavioral assessment tests, Ries et al. (1998) observed that the SIAS was correlated with verbal reports regarding positive and negative thoughts in speech and conversation situations while the SPS was only related to verbal reports in the speech task (negative correlation with speech duration). Brown et al. (1997) have argued that the SIAS and SPS constitute subscales of a higher-order construct social anxiety. However, despite these positive psychometric characteristics, the assumption of two underlying dimensions for the SIAS and SPS (Social Interaction Anxiety and Social Performance Anxiety, respectively) has yet to be adequately addressed. All published reports have found high to very high intercorrelations between the SIAS and SPS (Brown et al., 1997: r = .72; Heimberg et al., 1992: r = .41 social phobic group, r = .89 community sample; Peters, 2000: r = .73; Ries et al., 1998: r = .66; Stangier et al., 1999: r = .78 social phobic group, r = .69 clinical control group). While the amount of explained variance evidently varies according to the respective sample, there is clear evidence of a large amount of shared variance. Furthermore, there is little evidence that the SIAS and SPS differ significantly with respect to important clinical domains, such as treatment sensitivity (Stangier, Schramm, Heidenreich, Berger, & Clark, in press). So far, only one study has investigated the joint factor structure of the SIAS and the SPS (Safren, Turk, & Heimberg, 1998). In this study, a confirmatory factor analysis (CFA) failed to adequately support the hypothesis of two distinct constructs. However, in light of the very large number of parameters (81) to be estimated in their CFA comprising all 40 SIAS and SPS items, the sample size of N = 167 might have simply been too small for these tests (cf. Muthén & Muthén, 2002). While a large number of degrees of freedom may compensate for a small sample size (MacCallum, Browne, & Sugawara, 1996), power may have been too low to obtain precise parameter estimates. In the same trial (Safren et al., 1998), a three-factor solution was obtained using exploratory factor analysis (EFA) with varimax rotation. The first factor consisted of 17 SIAS items (“interaction anxiety”), the second factor of 11 SPS items (“being observed by others”), and the third factor of 5 SPS Items (“fear that others will notice anxiety symptoms”); 3 SIAS and 4 SPS items were eliminated due to high cross-loadings. More recently, two reports (Rodebaugh, Woods, & Heimberg, 2007; Rodebaugh, Woods, Heimberg, Liebowitz, & Schneier, 2006) questioned the factorial structure of the SIAS. Results of CFAs from both studies provide strong evidence that there are systematic differences between responses to negatively worded items in comparison to positively worded items of the scale. In addition, the negatively worded items showed consistently weaker relationships with a variety of comparison measures. The authors concluded that only the 17 positively worded items of the SIAS should be used. Given the high intercorrelations found in all studies employing the SIAS and SPS and the problem that previous studies predominantly applied EFA for construct validation, evidence to date does not allow firm conclusions regarding the factorial structures of SIAS and SPS. It also remains unclear whether both scales assess the same or different underlying constructs. Hence, the primary objective of the present study was to examine whether the existence of two separate scales is justified by investigating the latent factor structures of the SIAS and the SPS using CFA. We further examined possible response pattern biases due to similar wording.
1. Materials and methods 1.1. Participants and instruments The sample consisted of 577 patients (318 women, 256 men, 3 individuals with missing data) with a mean age of 38.73 years (SD = 10.94) who sought treatment at the behavior therapy outpatient clinic of the Goethe University in Frankfurt, Germany. Fifty-five percent of participants (N = 315) received a principal diagnosis of SAD according to DSM-IV (American Psychiatric & Association, 1994), while 45% (N = 262) were diagnosed with other non-psychotic disorders (anxiety disorders other than SAD, depression, or somatoform disorders) with no comorbid SAD. Diagnoses were made using the Structured Clinical Interview for DSM-IV (SCID-IV, First, Williams, & Spitzer, 1997). Comorbid diagnoses were permitted, with the exception of bipolar disorders, psychotic disorders, and drug or alcohol dependence. All clinical interviews and measures were administered by Ph.D. clinical psychologists or students in postgraduate clinical training. Raters received an intense training in administering clinical interviews (2-day training and follow-up) and interviews were closely supervised by T.H. and U.S. Self-rated measures comprised German translations of the SIAS (Stangier et al., 1999) and the SPS; the German versions of both scales showed very similar psychometric characteristics as their original English versions (Stangier et al., 1999). 1.2. Statistical analyses 1.2.1. Missing data Our sample included 18 cases (3.12% of N = 577) for which missing values were imputed using the estimation maximization (EM) algorithm of the PRELIS 2.54 program (Jöreskog & Sörbom, 1996; see also Du Toit & Du Toit, 2001). As we employed different analyses on the same data (i.e., confirmatory factors analysis, exploratory factor analyses, and logistic regression), we were not able to apply a full information maximum likelihood approach, since this method simultaneously estimates missing data and model parameters (cf. Schafer, 1997; Schafer & Graham, 2002). 1.2.2. Factor analyses Using CFA to assess the factorial structure of both scales separately, we tested whether (1) all SIAS items loaded onto one common factor (Model 1a) and (2) whether all SPS items loaded onto one common factor (Model 2a). In view of the fact that both scales measure quite heterogeneous constructs, we tested several hypothesized multidimensional structures of both scales: Regarding the SIAS, we further tested whether the negatively worded items of this scale also load onto a method (residual) factor (Model 1b) or whether negatively worded and positively worded items measure two method (residual) factors in addition to the common social interaction construct (Model 1c; as suggested by Rodebaugh et al., 2007). Additionally, we also investigated the one-factor solution (Model 1d) suggested by Safren et al. (1998). Regarding the SPS, we further tested Safren et al.’s (1998) twodimensional structure (Model 2b) with 11 items measuring “being observed by others” and 5 Items measuring “fear that others will notice anxiety symptoms”; 4 items that had cross-loadings on the SIAS or the SPS scales had been eliminated according to Safren et al.’s (1998) recommendation. In order to examine the factorial structure of both scales simultaneously, we tested whether all SIAS and SPS items loaded onto one single common factor (Model 3a) or onto two correlated factors (Model 3b). We further tested Safren et al.’s (1998) three factor model by combining Model 1a and Model 2b (Model 3c). Finally, we employed EFA with an oblique rotation on all 40 items. All CFA and EFA models were fitted to polychoric correlations using the robust weighted least squares mean and
T. Heidenreich et al. / Journal of Anxiety Disorders 25 (2011) 579–583
variance adjusted estimator (WLSMV) appropriate for categorical level data of the Mplus program (version 6.1, Muthén & Muthén, 1998–2010). Several goodness of fit statistics representing different classes of fit indices are reported as recommended by Bollen and Long (1993): The likelihood-ratio 2 test and its associated p-value, the rootmean square error of approximation (RMSEA, Steiger, 1990), the comparative fit index (CFI, Bentler, 1990), the Tucker–Lewis index, also known as non-normed fit index (TLI, Bentler & Bonett, 1980; Tucker & Lewis, 1973), and the weighted root mean-square residual (WRMR, Muthén, 1998–2004). Good model fit was indicated by a nonsignificant 2 value, RMSEA ≤ .06 (Hu & Bentler, 1999), CFI and TLI ≥ .97 (Schermelleh-Engel, Moosbrugger, & Müller, 2003), and WRMR ≤ .95 (Yu, 2002). 1.2.3. Logistic regression A logistic regression analysis was performed to examine the relative contribution of both scales to the prediction of group membership. For this analysis, the sample was divided into two subsamples, the first group (SAD) consisting of patients with principal diagnoses of SAD (N = 315) and the second group (control) consisting of patients who were diagnosed with anxiety disorders other than SAD, depression, or somatoform disorders (N = 262). The dependent variable was coded as 0 for controls and 1 for SAD. 2. Results 2.1. Confirmatory factor analyses The dimensionality of the SIAS and SPS was initially tested separately using the WLSMV estimator of the Mplus program. All negatively worded items were recoded before conducting the analyses. Fit indices are listed in Table 1. First, we tested dimensionality of the SIAS items. In the first CFA model, all 20 items of the SIAS loaded onto one common construct (Model 1a). Results indicated that all items including the reverse-coded negatively worded items correlated positively with the respective factor. In accordance with Rodebaugh et al.’s findings (2007), our results also revealed that the negatively worded items had the lowest factor loadings of all 20 items with parameter estimates of .31 (SIAS 5), .53 (SIAS 9), and .53 (SIAS 11). As the data did not fit the model, we tested in Model 1b whether in addition to the loadings on the common factor the three negatively worded items loaded onto an uncorrelated method (residual) factor (cf. Rodebaugh et al., 2007). The model fit improved, but CFI < .97 and WRMR > .95 still indicated that there was evidence of model misfit. As suggested by Rodebaugh et al. (2007), we also tested a model (Model 1c) with one substantive factor and two uncorrelated method (residual) factors (the first method factor consisting of three negatively worded items and the second method factor consisting of 17 positively worded items). The model fit now improved significantly compared with the common factor model (Model 1a) and the model with one method factor (Model 1b). Although statistical comparisons via chi-square difference tests were not possible because (a) the chi-square value for WLSMV cannot be used for chi-square difference testing in the regular way (Muthén & Muthén, 1998–2010) and (b) these models were not hierarchically nested, the model employing one substantive factor and two method factors (Model 1c) fitted the data best (CFI = .97, TLI = .96, WRMR = 1.22). However, the RMSEA (.085) gave evidence that some model misfit remained which may be due to error covariances not included in the model. In Model 1c, five items had relatively high loadings on the positively worded method factor: SIAS12 (“I worry about expressing myself in case I appear awkward”), SIAS13 (“I find it difficult to disagree with another’s point of view”), SIAS15 (“I find myself wor-
581
rying that I won’t know what to say in social situations”), SIAS17 (“I feel I’ll say something embarrassing when talking”), and SIAS18 (“When mixing in a group, I find myself worrying I will be ignored”) with standardized loadings on the method factor of .62, .40, .47, .66, and .42, respectively. These loadings may be interpreted as a method bias due to similar item wording, because these items all include either the word “worry” or the words “I find”/”I feel.” Additionally, we tested another common factor model as suggested by Safren et al. (1998). In this model (Model 1d), 17 items loaded onto a common factor, while items 1, 8, and 12 were eliminated. As the fit criteria show, the model fit was not acceptable. Internal consistencies for SIAS-20 and SIAS-17 were very similar (.94 vs. .93) and, as can be expected, both scales were very highly intercorrelated (r = .97). Furthermore, the results showed that the negatively worded items, again, had the lowest factor loadings. Second, we examined dimensionality of the SPS items. In the first CFA model, all 20 items of the SPS loaded onto one common construct (Model 2a). Results indicated that all items correlated positively with the respective factor (see Table 1); only two SPS items had factor loadings lower than .50 (SPS 2 and SPS 9). As suggested by Safren et al. (1998), we also tested a model (Model 2b) with two correlated factors (factor 1: items 3, 5, 7, 18, 19; factor 2: 11 items; items 6, 14, 15, and 20 were deleted). Results showed that data fitted the model slightly better than Model 2a, but again these could not be compared because they were not nested and they used a different number of indicators. In Model 2b, both factors correlated very high (r = .84) raising doubts that they both measure different constructs. Third, we examined dimensionality of all SIAS and SPS items simultaneously. In Model 3a, all 40 items of SIAS and SPS loaded onto a single factor. As the fit criteria show, the model fit was not acceptable. With Model 3b we tested a model with two correlated factors. In this model, all 20 items of the SIAS loaded onto a single factor and all 20 items of the SPS loaded onto a second factor. Results indicated that the items correlated positively with the respective factor; only two SPS items had factor loadings lower than .50 (SPS 2 and SPS 9). Both factors were substantially correlated (r = .84). A replication of Safren et al.’s 3-factor solution in Model 3c (a combination of Model 1a and Model 2b) did not substantially improve the model fit. All three factors were highly correlated with coefficients of .80 or above. Overall, results of our CFAs indicate that the SIAS does not have a straightforward unidimensional structure. The model employing two method factors and one common factor appears to provide a much better fit than the other models. We also did not find a clear indication of a unidimensional structure for the SPS items, but loadings on the common factor were quite high indicating that all items measure the same construct. Given the failure of the CFA to provide a good fit to the data, we additionally employed EFA with an oblique rotation on all 40 items using the WLSMV estimator of the Mplus program. Results suggested retaining either two factors (Kaiser criterion) or four factors (parallel analysis). In the 2-factor solution (see Table 2) the first factor consisted only of SIAS items and the second only of SPS items. Three SPS items (3, 5, 18) had cross loadings on the SIAS (of about the same size as on the SPS factor): “I can suddenly become aware of my own voice of others listening to me” (SPS3), “I fear I may blush when I am with others” (SPS5), and “I can get tense when I speak in front of other people” (SPS18). Both factors of the 2-factor solution were substantially correlated with r = .69. The cross-loadings on the SIAS factor indicate an overlap of item contents with regard to fear that others will notice anxiety symptoms. The 4-factor solution was not interpretable because of too many high cross loadings. In sum, results from the CFA and EFA suggested that SIAS and SPS each measure a single construct but that two method factors of the SIAS exist, which EFA could not detect.
582
T. Heidenreich et al. / Journal of Anxiety Disorders 25 (2011) 579–583
Table 1 Goodness-of-fit indices of SIAS and SPS confirmatory factor models.
SIAS 1a 1b
1c
1d SPS 2a 2b
SIAS and SPS 3a 3b 3c
Model
2
df
p
RMSEA
CFI
TLI
WRMR
1 common factor (20 items) 1 common factor (20 items), 1 method factor (3 reverse-coded items; cf. Rodebaugh et al., 2007) 1 common factor (20 items), 2 method factors (3 reverse-coded items, 17 positively coded items; cf. Rodebaugh et al., 2007) 1 common factor (17 items; items 1, 8, 12 deleted; cf. Safren et al., 1998)
1428.865 1313.219
170 167
.00 .00
0.113 0.109
0.937 0.943
0.930 0.935
1.887 1.797
775.389
150
.00
0.085
0.969
0.960
1.219
988.125
119
.00
0.113
0.944
0.936
1.713
1 common factor (20 items) 2 correlated factors (factor 1: 5 items – “fear that others will notice anxiety symptoms”; factor 2: 11 items – “being observed by others”; items 6, 14, 15, 20 deleted; cf. Safren et al., 1998)
984.637 579.386
170 103
.00 .00
0.091 0.090
0.945 0.947
0.939 0.939
1.579 1.448
1 common factor (40 items) 2 correlated factors (a SIAS and a SPS factor) 3 factors (Safren et al., 1998), combination of Model 1a and Model 2b
3892.887 2778.910
740 739
.00 .00
0.086 0.069
0.904 0.938
0.899 0.935
2.031 1.655
2314.353
591
.00
0.071
0.938
0.933
1.634
Note. RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker–Lewis incremental fit index; WRMR = weighted root mean square residual.
Table 2 Factor loadings of SIAS and SPS items (exploratory factor analysis with oblique rotation).
SIAS1 SIAS2 SIAS3 SIAS4 SIAS5 SIAS6 SIAS7 SIAS8 SIAS9 SIAS10 SIAS11 SIAS12 SIAS13 SIAS14 SIAS15 SIAS16 SIAS17 SIAS18 SIAS19 SIAS20 SPS1 SPS2 SPS3 SPS4 SPS5a SPS6 SPS7 SPS8 SPS9a SPS10 SPS11a SPS12 SPS13 SPS14 SPS15 SPS16 SPS17 SPS18 SPS19 SPS20 a
Factor 1 Social Interaction Anxiety
Factor 2 Social Performance Anxiety
0.628 0.633 0.514 0.601 0.465 0.608 0.724 0.696 0.586 0.972 0.766 0.830 0.634 0.746 0.856 0.729 0.772 0.712 0.768 0.548 0.188 0.019 0.328 0.143 0.326 0.385 0.013 0.065 −0.333 −0.085 0.104 0.276 0.064 −0.070 0.262 0.119 −0.002 0.473 −0.085 0.355
0.118 0.143 0.128 0.079 −0.200 0.195 0.156 0.119 −0.077 −0.184 −0.316 −0.004 0.069 −0.002 −0.024 0.119 0.088 −0.013 0.145 0.066 0.408 0.480 0.251 0.679 0.367 0.520 0.729 0.796 0.789 0.815 0.676 0.556 0.717 0.776 0.601 0.728 0.824 0.408 0.714 0.568
Reverse-coded items.
2.2. Logistic regression To determine whether both SIAS and SPS scales contribute to the prediction of group membership, a logistic regression analysis was performed. A test of the model with both scales against a constantonly model was statistically significant (Model 2 (2df) = 167.00, p < .01) indicating that the scales, as a set, reliably distinguish the SAD group from the control group. Logistic regression also allows for the assessment of each of the scales to the prediction of group membership. The coefficients (ˇ) give the change in the logit for a one standard deviation change in the predictors. The coefficients are “semi-standardized,” only standardized on the predictor side. The results indicated that both scales were significant predictors of group membership (SIAS: ˇ = .83, SE = .15, p < .01; SPS: ˇ = .56, SE = .16, p < .01). Nagelkerke’s R2 was .34, and overall the model classified 74.2% of the patients correctly (70.2% of the patients in the control group and 77.5% of the patients in the SAD group). Logistic regression shows that both scales contribute independently to the prediction of group membership. 3. Discussion In order to examine whether it is useful to retain the SIAS and the SPS as two separate, but highly correlated instruments of social anxiety, we employed CFAs to study the factor structure of these two scales. Our analyses revealed that variance of the SIAS and variance of the SPS can each be largely accounted for by a single underlying factor. However, good model fit of the SIAS could only be achieved by including two method factors. These findings support recent research of Rodebaugh et al. (2006) as well as Rodebaugh et al. (2007) who also found that the loadings for the negatively worded items were substantially lower than the loadings for all other items. Negatively worded items are often used to reduce the potential effects of response pattern biases. Research has shown that they may produce artifactual response factors consisting exclusively of negatively worded items (cf. Podsakoff, MacKenzie, Lee, & Podsakoff, 2003). Our results indicate that future research should eliminate the negatively worded items because they do not measure the common construct as reliable as the other items. In addition to the negatively worded items, similarly worded
T. Heidenreich et al. / Journal of Anxiety Disorders 25 (2011) 579–583
items may also produce method bias. Some of these items of the SIAS include “I worry” or “I find myself worrying.” However, a separate factor consisting of these items loaded highly onto the common factor and, therefore, did not support our assumption that these items measure mainly method effects. Analyzing both scales simultaneously, a CFA with two correlated factors showed a better fit with the data than a single factor model. As model fit was not quite satisfactory, we additionally employed an EFA with an oblique rotation on all 40 items. A 2-factor solution seemed to explain the data best, indicating that both SIAS and SPS do not measure the same construct but rather, two distinct constructs. The high correlation of r = .69 may also suggest that both scales represent facets of a higher-order construct. In sum, our findings provide support to retain the SIAS and SPS as two separate scales. A logistic regression analysis supported this finding as both scales contributed independently to the prediction of group membership. Our results are also in line with studies reporting differential correlations of the SIAS and SPS with other constructs. The work by Heimberg et al. (1992), for example, showed the SIAS to be significantly more strongly related to a measure of interaction anxiety than a measure of performance anxiety. However, they are in contrast to the psychometric results reported by Safren et al. (1998). Possible explanations for this discrepancy may be that (1) our analyses were based on a larger sample, and (2) we used CFA together with the WLSMV estimator of the Mplus program appropriate for the analysis of categorical data. The following limitations of the present examination should be noted. First, due to the hypothesis-testing nature of CFA, we were only able to compare one and two-factor models. However, as noted by Safren et al. (1998), it is possible that the variance of situations relevant to the measurement of SAD is explained by more than 2 factors. Second, further studies should investigate whether the number of items of the two scales can be reduced without a loss of reliability or validity. A smaller number of items may represent the constructs equally well as the current 20 items and at the same time may reduce possible method effects. Third, due to different methods of factor analyses, criteria for factor selection, rotation methods and samples in previous studies, it is unclear whether measurement invariance of the SIAS and SPS exist between American and German samples. Fourth, due to the fact that this study stresses factorial validity of SIAS/SPS, no further conclusions can be drawn with regard to correlations with other measures (convergent and discriminant) and therapy sensitivity. Further studies should address these important questions. Despite these limitations, results of this study add to the current literature on the measurement of social anxiety. We conclude that the SIAS and SPS are two correlated instruments, each measuring related, but separate, facets of social anxiety in patients with SAD. Author note Research was performed at the Department of Psychology, Department of Clinical Psychology and Psychotherapy, University of Frankfurt/Main, Varrentrappstr. 40-42, 60486 Frankfurt am Main, Germany. Acknowledgements Dr. Hofmann is a paid consultant by Schering-Plough and supported by NIMH grant MH078308. References American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.
583
Bentler, P. M. (1990). Comparative fix indexes in structural models. Psychological Bulletin, 107, 238–246. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606. Bögels, S. M., Alden, L., Beidel, D. C., Clark, L. A., Pine, D. S., Stein, M. B., & Voncken, M. (2010). Social anxiety disorder: questions and answers. Depression and Anxiety, 27, 168–189. Bollen, K. A., & Long, J. S. (Eds.). (1993). Testing structural equation models. Newbury Park, CA: Sage. Brown, E. J., Turovsky, J., Heimberg, R. G., Juster, H. R., Brown, T. A., & Barlow, D. H. (1997). Validation of the Social Interaction Anxiety Scale and the Social Phobia Scale across the anxiety disorders. Psychological Assessment, 9, 21–27. Carleton, R. N., Collimore, K. C., Asmundson, G. J., McCabe, R. E., Rowa, K., & Antony, M. M. (2009). Refining and validating the Social Interaction Anxiety Scale and the Social Phobia Scale. Depression and Anxiety, 26, E71–81. Du Toit, M., & Du Toit, S. (2001). Interactive LISREL: user’s guide (with a contribution by D. Hawkins). Chicago, IL: Scientific Software International. First, M. B., Williams, J. B., & Spitzer, R. L. (1997). Structured clinical interview for DSM-IV axis I disorders (SCID-I). Washington: American Psychiatric Association. Heimberg, R. G., Mueller, G. P., Holt, C. S., Hope, D. A., & Liebowitz, M. R. (1992). Assessment of anxiety in social interaction and being observed by others: the Social Interaction Anxiety Scale and the Social Phobia Scale. Behavior Therapy, 23, 53–75. Hofmann, S. G., Heinrichs, N., & Moscovitch, D. A. (2004). The nature and expression of social phobia: toward a new classification. Clinical Psychology Review, 24, 769–797. Hu, L., & Bentler, P. M. (1999). Cutoff criterion for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. Jöreskog, K. G., & Sörbom, D. (1996). PRELIS 2: user’s reference guide. Chicago, IL: Scientific Software International. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149. Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455–470. Muthén, B. O. (1998–2004). Mplus technical appendices. Los Angeles, CA: Muthén & Muthén. Muthén, L. K., & Muthén, B. O. (1998–2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén. Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo Study to decide on sample size and determine power. Structural Equation Modeling, 9, 599–620. Peters, L. (2000). Discriminant validity of the Social Phobia and Anxiety Inventory (SPAI), the Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS). Behaviour Research and Therapy, 38, 943–950. Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903. Ries, B. J., McNeil, D. W., Boone, M. L., Turk, C. L., Carter, L. E., & Heimberg, R. G. (1998). Assessment of contemporary social phobia verbal report instruments. Behaviour Research and Therapy, 36, 983–994. Rodebaugh, T. L., Woods, C. M., & Heimberg, R. G. (2007). The reverse of social anxiety is not always the opposite: the negatively worded items of the social interaction anxiety scale do not belong. Behavior Therapy, 38, 192–206. Rodebaugh, T. L., Woods, C. M., Heimberg, R. G., Liebowitz, M. R., & Schneier, F. R. (2006). The factor structure and screening utility of the Social Interaction Anxiety Scale. Psychological Assessment, 18, 231–237. Safren, S. A., Turk, C. L., & Heimberg, R. G. (1998). Factor Structure of the Social Interaction Anxiety Scale and the Social Phobia Scale. Behaviour Research and Therapy, 36, 443–453. Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall. Schafer, J. L., & Graham, J. W. (2002). Missing data: our view of the state of the art. Psychological Methods, 7, 147–177. Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8, 23–74. Stangier, U., Heidenreich, T., Berardi, A., Golbs, U., & Hoyer, J. (1999). Die Erfassung sozialer Phobie durch die Social Interaction Anxiety Scale (SIAS) und die Social Phobia Scale (SPS) [Assessment of social phobia with the Social Interaction Anxiety Scale (SIAS) and Social Phobia Scale (SPS)]. Zeitschrift für Klinische Psychologie, 28, 28–36. Stangier, U., Schramm, Heidenreich, T., Berger, M. & Clark, D. M. Cognitive Therapy versus Interpersonal Psychotherapy in Social Phobia: a randomized controlled trial. Archives of General Psychiatry, in press. Steiger, J. H. (1990). Structural model evaluation and modification: an interval estimation approach. Multivariate Behavioral Research, 25, 173–180. Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. Yu, C. Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Unpublished doctoral dissertation, University of California, Los Angeles.