- Email: [email protected]
Discriminant validity of the Social Phobia and Anxiety Inventory (SPAI), the Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS)
Behaviour Research and Therapy 38 (2000) 943±950
www.elsevier.com/locate/brat
Discriminant validity of the Social Phobia and Anxiety Inventory (SPAI), the Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS) Lorna Peters a,b,* a
Clinical Research Unit for Anxiety Disorders, St Vincent's Hospital, 299 Forbes St, Darlinghurst, NSW 2010, Australia. b School of Psychiatry, University of New South Wales, Sydney, NSW, Australia. Received 28 June 1999
Abstract Three measures commonly used in assessment of social phobia, the Social Phobia and Anxiety Inventory (SPAI [Turner, S. M., Beidel, D. C. & Dancu, C. V. (1996). Social phobia and anxiety inventory: manual. Toronto, Ont.: Multi-Health Systems Inc.), the Social Phobia Scale (SPS [Mattick, R. P. & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455±470] and the Social Interaction Anxiety Scale (SIAS [Mattick, R. P. & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455±470], were compared for their ability to discriminate between social phobia and other anxiety disorders (panic disorder with or without agoraphobia). Participants were 117 patients attending a specialized anxiety disorders unit for treatment. While all three measures were able to detect dierences between social phobic patients and patients with panic disorder with or without agoraphobia, a logistic regression analysis showed that the SPAI, but not the SPS and SIAS, was a signi®cant predictor of membership of the social phobia group. Receiver operating characteristic (ROC) analysis also showed that the SPAI was the better measure for discriminating between social phobia and panic disorder with and without agoraphobia. Analysis of the sensitivity, speci®city and positive and negative predictive power of the measures at the optimum cuto scores produced by the ROC analysis are presented. 7 2000 Elsevier Science Ltd. All rights reserved. Keywords: Measurement of social phobia; Discriminant validity
* Tel.: +61-2-9332-1013; fax: +61-2-9332-4316. E-mail address: [email protected] (L. Peters). 0005-7967/00/$ - see front matter 7 2000 Elsevier Science Ltd. All rights reserved. PII: S 0 0 0 5 - 7 9 6 7 ( 9 9 ) 0 0 1 3 1 - X
944
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
There are a number of reliable and valid self-report measures that can be used to measure aspects of social phobia (see Cox & Swinson, 1995, for a review). The focus of the present paper is the discriminant validity of social phobia measures Ð in particular, the ability of measures to discriminate social phobia from other anxiety disorders. One of the most widely used measures in studies of social phobia is the Fear of Negative Evaluation (FNE) scale developed by Watson and Friend (1969) along with its companion measure the Social Avoidance and Distress (SAD) scale. Although the development of these measures preceded the inclusion of social phobia in the diagnostic classi®cation scheme, they are widely used in studies of social phobia perhaps because it seems that the features tapped by the measures are at the core of the disorder. The evidence does not, however, suggest that the FNE and SAD discriminate between anxiety disorder groups. Turner, McCanna and Beidel (1987) found that scores on the SAD and FNE were not signi®cantly dierent for participants with social phobia, agoraphobia with and without panic disorder, panic disorder, generalized anxiety disorder and obsessive compulsive disorder. Only participants with simple phobia were signi®cantly dierent on their FNE and SAD scores from social phobic participants. Oei, Kenna and Evans (1991) found a similar pattern of results with no signi®cant dierences on the SAD and FNE between participants with social phobia, panic disorder with agoraphobia and generalized anxiety disorder. While the SAD showed signi®cant dierences between participants with social phobia and those with panic disorder or simple phobia, the FNE only showed signi®cant dierences between participants with social phobia and participants with simple phobia. Three alternative measures which were designed to assess social phobia speci®cally are the pair of measures called the Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) and the Social Phobia and Anxiety Inventory (SPAI; Turner, Beidel & Dancu, 1996; Turner, Beidel, Dancu & Stanley, 1989). The SPS was designed to measure social phobia de®ned as ``anxiety and fear at the prospect of being observed or watched by other people, and in particular, where the individual expresses distress when undertaking certain activities in the presence of others'' (Mattick & Clarke, 1998, p. 457). The SIAS scale was designed to measure social interaction anxiety de®ned as ``distress when meeting and talking with other people'' (Mattick & Clarke, 1998, p. 457). Thus the two companion measures were designed to distinguish between scrutiny fears and concerns about interaction. In the original study of the measures, Mattick and Clarke (1998) showed that both scales were able to discriminate between social phobia, agoraphobia and simple phobia samples and between social phobia and normal samples. Heimberg, Mueller, Holt, Hope and Liebowitz (1992) found that the scales discriminated between social phobic and community samples. Brown et al. (1997) found that the scales discriminated between social phobics and other anxiety disorder groups and control participants, with the exception of agoraphobic participants who could not be discriminated from social phobic participants on the basis of the SPS. The SPAI was developed on an empirical basis to assess social phobia and in particular to assess ``speci®c somatic symptoms, cognitions and behavior across a range of potentially fearproducing situations'' (Turner et al., 1989, p. 35). The SPAI score (scored as the dierence between the social phobia subscale and the agoraphobia subscale) has been shown to discriminate between socially anxious and nonsocially anxious participants (Beidel, Turner,
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
945
Stanley & Dancu, 1989) and between participants with social phobia and those with panic disorder with or without agoraphobia or those with obsessive±compulsive disorder (Turner et al., 1989). In a direct comparison of the SPS, the SIAS and the SPAI in patients with social phobia, Ries et al. (1998) found that the three measures were highly correlated. Of signi®cance to the present study was the apparent dierential ability of the measures to discriminate between subtypes of social phobia. While all three measures dierentiated a circumscribed speech phobia group from a group of patients with generalised social phobia and avoidant personality disorder, the SPS was less able than the other measures to discriminate between the circumscribed speech phobia group and a group with generalised social phobia without avoidant personality disorder. The Ries et al. (1998) study did not address the question of the ability of the scales to discriminate between social phobia and other anxiety disorders. The present study aims to compare the three speci®c measures of social phobia in terms of their relative ability to discriminate between social phobia and other anxiety disorders, in particular, panic disorder with or without agoraphobia. 1. Method Participants were 117 (67 female; 50 male) patients who were accepted for treatment at the Anxiety Disorders Unit at St Vincent's Hospital, Sydney. The patients were treated for social phobia (n = 82) or panic disorder with (n = 18) or without (n = 17) agoraphobia. Their primary DSM-III-R diagnosis was made by a clinician at an initial interview. Only patients whose diagnosis was con®rmed using a structured diagnostic interview (the Composite International Diagnostic Interview, computerized version (WHO, 1993)) were included in the analysis. Diagnosis was con®rmed for 105 patients (73 with social phobia, 17 with panic disorder with agoraphobia and 15 with panic disorder without agoraphobia). The average age of the sample was 32.18 years (S.D.=8.58; range=18±63). For the purposes of analysis, the subjects were divided into 2 groups: social phobia and other anxiety disorder. The demographic characteristics of the social phobia and other anxiety disorder groups are described in Table 1. There were no signi®cant dierences between the groups in terms of age (F(1,103)=2.55; p = 0.11). The sex distributions of the two groups were signi®cantly dierent (w 2 (1)=6.62; p = 0.01): the male to female ratio was approximately equal for the social
Table 1 Demographic characteristics for the patients by group Social phobia (n = 73)
Other anxiety disorder (n = 32)
Age
mean S.D.
31.30 7.57
34.19 10.39
Sex
males (n ) females (n)
38 35
8 24
946
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
phobia group, whereas there were more females than males in the other anxiety disorder group. Patients with comorbid diagnoses (as de®ned by the CIDI) were included in the study: 66 patients in the social phobia group had one or more other anxiety disorders (59 had panic disorder with or without agoraphobia, 27 had a simple phobia, 10 had generalised anxiety disorder and 5 had obsessive compulsive disorder) and 28 patients in the other anxiety disorder group had one or more additional anxiety disorders (10 had social phobia, 23 had a simple phobia, 1 had generalised anxiety disorder and 3 had OCD). The three measures were administered prior to treatment. The Social Phobia and Anxiety Inventory (Turner et al., 1996) is a 45 item self-report questionnaire which measures the frequency (on a 7 point scale) with which the respondent experiences the ``somatic, cognitive and behavioral aspects of social phobia across a wide range of social situations and settings'' (Turner et al., 1996, p. 2). It is an empirically derived measure made up of 2 subscales, Social Phobia and Agoraphobia, with the total SPAI score being the dierence between the Social Phobia and Agoraphobia subscales. Since the present study assessed the ability of the scales to distinguish between social phobia and panic disorder with or without agoraphobia, the data reported here are for the total SPAI score (i.e. the dierence score). The Social Phobia Scale and the Social Interaction Anxiety Scale (Mattick & Clarke, 1998) were developed as a companion set of measures. The scales were designed to assess scrutiny fears (SPS) and fears of general social interaction (SIAS). Each scale has 20 items rated on a scale from 0 to 4 which indicates how true or characteristic the statements are of the respondent (`not at all' to `extremely'). 2. Results and discussion Similar to previous data on the construct validity of the three measures (Ries et al., 1998), they were found to be highly and signi®cantly correlated (SPS-SIAS: r = 0.73, p < 0.001; SPSSPAI: r = 0.72, p < 0.001; SIAS-SPAI: r = 0.85, p < 0.001) suggesting that the measures tap a similar construct. Table 2 presents the mean scores on the three measures for the social phobia and other anxiety disorders groups. The social phobia and other anxiety disorder groups were compared on the three dependent variables using a one-way between subjects' multivariate analysis of variance (MANOVA). Table 2 Scores on the measures for the social phobia and other anxiety disorder groups. p < 0.05a
SIAS SPS SPAI a
Social phobia (n = 74)
Other anxiety disorder (n = 34)
M
(s)
M
(s)
55.24 44.41 110.61
(12.97) (14.05) (25.10)
29.46 23.75 42.90
(17.29) (16.84) (38.29)
Multivariate F(3,101)=38.06; p < 0.05.
Univariate F
df
71.27 42.54 115.70
1,103 1,103 1,103
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
947
SPSS MANOVA was used for the analysis. With the use of Pillais' criterion, the combined DVs were signi®cantly dierent between the two groups (F(3,101)=38.06, p < 0.05). The scores on the SPAI, the SIAS and the SPS were higher for the social phobia group than for the other anxiety disorder group (see Table 2). All of the univariate F values were signi®cant (Table 2), con®rming previous research on these three measures which has shown that they are all able to make distinctions between social phobic patients and patients with other anxiety disorder diagnoses (Brown et al., 1997; Turner et al., 1989). Simply comparing the measures at a univariate level, however, does not provide evidence that one of the measures is better than the others for discriminating between social phobia and other anxiety disorders. To determine if one of the measures was better than the others, a logistic regression was performed. Logistic regression allows prediction of group membership from a set of variables (Tabachnick & Fidell, 1996)1. A test of the model with all three measures against a constant-only model was statistically reliable (w 2 (3, N = 105)=65.63; p < 0.001) indicating that the measures, as a set, reliably distinguish the social phobia group from the other anxiety disorder group. Logistic regression also allows for the assessment of contribution of individual variables to the prediction of group membership. Given that only the SPAI reliably predicted group membership (z = 12.02, p < 0.001), a second model without the SPAI was tested. The model without the SPAI was reliably dierent from the model with all three measures (w 2(1)=32.90, p < 0.05), suggesting that the ability of a model without the SPAI to predict membership of the social phobia versus the other anxiety disorders group is not as good as a model with the SPAI. Thus, it appears that the SPAI is a better predictor of social phobia group membership than the SPS and the SIAS. To support this conclusion and to provide guidelines as to the cuto scores on the measures, which de®ne social phobia as opposed to panic disorder with or without agoraphobia, a receiver operating characteristic (ROC) analysis was carried out. ROC analysis allows the ability of a test to discriminate diseased cases from normal cases (or in this case, social phobic cases from nonsocial phobic cases) to be evaluated (Hsiao, Bartko & Potter, 1989). The area under the ROC curve (AUC) summarizes the ability of the instrument to discriminate between those with the disorder of interest and those who do not have the disorder (Hanley & McNeil, 1982). The AUC value can also be used to compare the diagnostic performance of two or more tests (Griner, Mayewski, Mushlin & Greenland, 1981). In addition to the ROC analysis, four indicators of test performance were calculated for each of the three tests to allow recommendations for the best cuto scores to be made: sensitivity (proportion of true positives), speci®city (proportion of false negatives), positive predictive value (PPV; probability that the disease is present when the test is positive) and negative predictive value (NPV; probability that the disease is not present when the test is negative). The MedCalc program was used to calculate and compare the AUCs for the three tests and a commercial computer program (CUTOFF; McKenzie & Clarke, 1992) was used to calculate the test performance indicators of sensitivity, speci®city, PPV and NPV. The AUC for the SPAI was 0.93 (95% CI=0.87±0.97), for the SIAS was 0.86 (95%
1 Although the measures are intercorrelated, multicollinearity is unlikely to be a problem in the logistic regression since the parameter estimates in the model are relatively low (b < 3) (Tabachnick & Fidell, 1996).
948
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
CI=0.78±0.92) and for the SPS was 0.80 (95% CI=0.72±0.87). None of the 95% con®dence intervals around the AUC values include a value of 0.5 indicating that all of the tests perform better than chance in discriminating between those who have social phobia and those who have panic disorder with or without agoraphobia. Of more interest here, however, is the comparison between the three tests. The dierence between the AUC for the SPAI and the AUC for the SIAS was signi®cantly dierent (dierence=0.07; 95% CI=0.02±0.13; p = 0.008), as was the dierence between the SPAI and the SPS (dierence=0.13; 95% CI=0.06±0.20; p < 0.001). The dierence between the AUC for the SIAS and for the SPS was not dierent (dierence=0.05; 95% CI=ÿ0.020±0.13; p = 0.124). These results con®rm the multivariate analysis which showed that the SPAI was a better measure than the SPS and the SIAS for discriminating between social phobia and panic disorder with and without agoraphobia. Table 3 presents the test performance indicators for the three tests. Since these indicators can be computed for each possible cuto score on the test, only the values for the cuto which was determined by the ROC analysis to be the best cuto score are presented (the full range of values is available on request from the author). It should be noted that the scores found in this study to be the optimum cutos for social phobia versus panic disorder with or without agoraphobia are similar to the average scores for social phobics reported in other studies. For example, Turner et al. (1996; p. 13) report that a SPAI dierence score of 80 is the best for discriminating between agoraphobics and social phobics, compared to a score of 88 in the current study. Ries et al. (1998) report a mean score of 27.2 for generalised social phobics on the SPS compared to a score of 27 in the current study, although the mean score for generalised social phobics is higher in the Ries et al. (1998) study than in the current study (48.5 vs. 36). That the SPS and the SIAS did not appear to be the best measures for discriminating between social phobia and other anxiety disorders does not, however, mean that the measures are not valid. Used alone, the measures would make signi®cant discriminations, as shown by the signi®cant univariate F-tests, the AUC values and the indicators of test performance. The results of this study suggest that if the researcher is making a choice between instruments, the SPAI may be preferable if the purpose of the assessment is to discriminate between features of social phobia and of other anxiety disorders, particularly panic disorder with or without agoraphobia. Further research will need to be conducted to determine which, if any, of the three measures are more appropriate for other assessment purposes, for example, detecting Table 3 Indicators of test performance for the SPAI, SPS and SIAS at the optimum cuto scorea,b
SPAI SIAS SPS a
Cuto score
Sensitivity
Speci®city
PPV
NPV
> 88 > 36 > 26
0.87 0.93 0.89
0.91 0.60 0.66
0.96 0.84 0.86
0.74 0.78 0.72
The optimum cuto score is the one calculated by the ROC analysis to produce the best trade o between sensitivity and speci®city of the test. b PPV: positive predictive value; NPV=negative predictive value.
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
949
treatment eects (see Taylor, Woody, McLean & Koch, 1997, for a discussion of measuring sensitivity to change). In addition, the other anxiety disorder group in this study was made up of panic disorder with and without agoraphobia. An examination of the ability of the measures to discriminate between social phobia and dierent anxiety disorders is warranted. The ability of all three measures to discriminate between social phobia and panic disorder with or without agoraphobia is particularly impressive given the high levels of comorbidity between social phobia and panic disorder with and without agoraphobia in this study: 50.4% of patients with a primary diagnosis of social phobia also had panic disorder with or without agoraphobia as a secondary disorder, and 31.3% of patients with a primary diagnosis of panic disorder with or without agoraphobia group had secondary social phobia. Although the numbers are small, a post hoc analysis of the ability of the measures to discriminate between the social phobic patients who did not have comorbid panic disorder with or without agoraphobia and the other anxiety disorder patients who did not have comorbid social phobia yielded a similar pattern of results. All three measures signi®cantly discriminated between the two groups at the univariate level, but when compared in a logistic regression, only the SPAI signi®cantly predicted group membership. While this study suggests that the all three measures discriminate between social phobia and panic disorder with or without agoraphobia, the measures are limited in that they are measures of the sorts of situations which social phobics ®nd distressing, but they do not necessarily measure the reasons for the distress. Given the view embodied in the diagnostic classi®cation that the role of being judged critically is central to social phobia, measures that tap cognitions relating to this concern are necessary. While the FNE was once thought of as a promising measure for de®ning the cause of distress in social phobia Ð a fear of being evaluated negatively Ð that promise has not been kept. Fear of negative evaluation may well be a cognition that crosses the diagnostic boundaries. In a review of cognitive assessments of social phobia, Heimberg (1994, p. 270) lamented that ``a limited number of devices have actually been used in studies of treatment of social phobia''. One of the reasons for this may be that there are no standardized self-report measures of cognitions in social phobia available. Future work may go towards development of such a measure.
References Beidel, D. C., Turner, S. M., Stanley, M. A., & Dancu, C. V. (1989). The social phobia and anxiety inventory: concurrent and external validity. Behavior Therapy, 20, 417±427. Brown, E. J., Turovsky, J., Heimberg, R. G., Juster, H. R., Brown, T. A., & Barlow, D. H. (1997). Validation of the social interaction anxiety scale and the social phobia scale across the anxiety disorders. Psychological Assessment, 9, 21±27. Cox, B. J., & Swinson, R. P. (1995). Assessment and measurement. In M. B. Stein, Social phobia: clinical and research perspectives. Washington, DC: American Psychiatric Press, Inc. Griner, P. F., Mayewski, R. J., Mushlin, A. I., & Greenland, P. (1981). Selection and interpretation of diagnostic tests and procedures. Annals of Internal Medicine, 94, 555±600. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29±36. Hsiao, J. K., Bartko, J. J., & Potter, W. Z. (1989). Diagnosing diagnoses: receiver operating characteristic methods and psychiatry. Archives of General Psychiatry, 46, 664±667.
950
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
Heimberg, R. G. (1994). Cognitive assessment strategies and the measurement of outcome of treatment for social phobia. Behaviour Research and Therapy, 32, 269±280. Heimberg, R. G., Mueller, G. P., Holt, C. S., Hope, D. A., & Liebowitz, M. R. (1992). Assessment of anxiety in social interaction and being observed by others: the social interaction anxiety scale and the social phobia scale. Behavior Therapy, 23, 53±73. Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455±470. McKenzie, D. P., & Clarke, D. M. (1992). CUTOFF: a Fortran program for establishing thresholds for screening indices. Educational and Psychological Measurement, 52, 891±893. Oei, T. P. S., Kenna, D., & Evans, L. (1991). The reliability, validity and utility of the SAD and FNE scales for anxiety disorder patients. Personality and Individual Dierences, 12, 111±116. Ries, B. J., McNeil, D. W., Boone, M. L., Turk, C. L., Carter, L. E., & Heimberg, R. G. (1998). Assessment of contemporary social phobia verbal report instruments. Behaviour Research and Therapy, 36, 983±994. Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics. New York, NY: Harper Collins Publishers. Taylor, S., Woody, S., McLean, P. D., & Koch, W. J. (1997). Sensitivity of outcome measures for treatment of generalized social phobia. Assessment, 4, 181±191. Turner, S. M., Beidel, D. C., & Dancu, C. V. (1996). Social phobia and anxiety inventory: manual. Toronto, Ont: Multi-Health Systems Inc. Turner, S. M., Beidel, D. C., Dancu, C. V., & Stanley, M. A. (1989). An empirically derived inventory to measure social fears and anxiety: the social phobia and anxiety inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 1, 35±40. Turner, S. M., McCanna, M., & Beidel, D. C. (1987). Validity of the social avoidance and distress and fear of negative evaluation scales. Behaviour Research and Therapy, 25, 113±115. Watson, D., & Friend, R. (1969). Measurement of social-evaluative anxiety. Journal of Consulting and Clinical Psychology, 33, 448±457. World Health Organization (1993). CIDI-Auto version 1.0: Administrator's guide and reference. Sydney: Training and Reference Center for WHO CIDI.
www.elsevier.com/locate/brat
Discriminant validity of the Social Phobia and Anxiety Inventory (SPAI), the Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS) Lorna Peters a,b,* a
Clinical Research Unit for Anxiety Disorders, St Vincent's Hospital, 299 Forbes St, Darlinghurst, NSW 2010, Australia. b School of Psychiatry, University of New South Wales, Sydney, NSW, Australia. Received 28 June 1999
Abstract Three measures commonly used in assessment of social phobia, the Social Phobia and Anxiety Inventory (SPAI [Turner, S. M., Beidel, D. C. & Dancu, C. V. (1996). Social phobia and anxiety inventory: manual. Toronto, Ont.: Multi-Health Systems Inc.), the Social Phobia Scale (SPS [Mattick, R. P. & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455±470] and the Social Interaction Anxiety Scale (SIAS [Mattick, R. P. & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455±470], were compared for their ability to discriminate between social phobia and other anxiety disorders (panic disorder with or without agoraphobia). Participants were 117 patients attending a specialized anxiety disorders unit for treatment. While all three measures were able to detect dierences between social phobic patients and patients with panic disorder with or without agoraphobia, a logistic regression analysis showed that the SPAI, but not the SPS and SIAS, was a signi®cant predictor of membership of the social phobia group. Receiver operating characteristic (ROC) analysis also showed that the SPAI was the better measure for discriminating between social phobia and panic disorder with and without agoraphobia. Analysis of the sensitivity, speci®city and positive and negative predictive power of the measures at the optimum cuto scores produced by the ROC analysis are presented. 7 2000 Elsevier Science Ltd. All rights reserved. Keywords: Measurement of social phobia; Discriminant validity
* Tel.: +61-2-9332-1013; fax: +61-2-9332-4316. E-mail address: [email protected] (L. Peters). 0005-7967/00/$ - see front matter 7 2000 Elsevier Science Ltd. All rights reserved. PII: S 0 0 0 5 - 7 9 6 7 ( 9 9 ) 0 0 1 3 1 - X
944
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
There are a number of reliable and valid self-report measures that can be used to measure aspects of social phobia (see Cox & Swinson, 1995, for a review). The focus of the present paper is the discriminant validity of social phobia measures Ð in particular, the ability of measures to discriminate social phobia from other anxiety disorders. One of the most widely used measures in studies of social phobia is the Fear of Negative Evaluation (FNE) scale developed by Watson and Friend (1969) along with its companion measure the Social Avoidance and Distress (SAD) scale. Although the development of these measures preceded the inclusion of social phobia in the diagnostic classi®cation scheme, they are widely used in studies of social phobia perhaps because it seems that the features tapped by the measures are at the core of the disorder. The evidence does not, however, suggest that the FNE and SAD discriminate between anxiety disorder groups. Turner, McCanna and Beidel (1987) found that scores on the SAD and FNE were not signi®cantly dierent for participants with social phobia, agoraphobia with and without panic disorder, panic disorder, generalized anxiety disorder and obsessive compulsive disorder. Only participants with simple phobia were signi®cantly dierent on their FNE and SAD scores from social phobic participants. Oei, Kenna and Evans (1991) found a similar pattern of results with no signi®cant dierences on the SAD and FNE between participants with social phobia, panic disorder with agoraphobia and generalized anxiety disorder. While the SAD showed signi®cant dierences between participants with social phobia and those with panic disorder or simple phobia, the FNE only showed signi®cant dierences between participants with social phobia and participants with simple phobia. Three alternative measures which were designed to assess social phobia speci®cally are the pair of measures called the Social Phobia Scale (SPS) and the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) and the Social Phobia and Anxiety Inventory (SPAI; Turner, Beidel & Dancu, 1996; Turner, Beidel, Dancu & Stanley, 1989). The SPS was designed to measure social phobia de®ned as ``anxiety and fear at the prospect of being observed or watched by other people, and in particular, where the individual expresses distress when undertaking certain activities in the presence of others'' (Mattick & Clarke, 1998, p. 457). The SIAS scale was designed to measure social interaction anxiety de®ned as ``distress when meeting and talking with other people'' (Mattick & Clarke, 1998, p. 457). Thus the two companion measures were designed to distinguish between scrutiny fears and concerns about interaction. In the original study of the measures, Mattick and Clarke (1998) showed that both scales were able to discriminate between social phobia, agoraphobia and simple phobia samples and between social phobia and normal samples. Heimberg, Mueller, Holt, Hope and Liebowitz (1992) found that the scales discriminated between social phobic and community samples. Brown et al. (1997) found that the scales discriminated between social phobics and other anxiety disorder groups and control participants, with the exception of agoraphobic participants who could not be discriminated from social phobic participants on the basis of the SPS. The SPAI was developed on an empirical basis to assess social phobia and in particular to assess ``speci®c somatic symptoms, cognitions and behavior across a range of potentially fearproducing situations'' (Turner et al., 1989, p. 35). The SPAI score (scored as the dierence between the social phobia subscale and the agoraphobia subscale) has been shown to discriminate between socially anxious and nonsocially anxious participants (Beidel, Turner,
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
945
Stanley & Dancu, 1989) and between participants with social phobia and those with panic disorder with or without agoraphobia or those with obsessive±compulsive disorder (Turner et al., 1989). In a direct comparison of the SPS, the SIAS and the SPAI in patients with social phobia, Ries et al. (1998) found that the three measures were highly correlated. Of signi®cance to the present study was the apparent dierential ability of the measures to discriminate between subtypes of social phobia. While all three measures dierentiated a circumscribed speech phobia group from a group of patients with generalised social phobia and avoidant personality disorder, the SPS was less able than the other measures to discriminate between the circumscribed speech phobia group and a group with generalised social phobia without avoidant personality disorder. The Ries et al. (1998) study did not address the question of the ability of the scales to discriminate between social phobia and other anxiety disorders. The present study aims to compare the three speci®c measures of social phobia in terms of their relative ability to discriminate between social phobia and other anxiety disorders, in particular, panic disorder with or without agoraphobia. 1. Method Participants were 117 (67 female; 50 male) patients who were accepted for treatment at the Anxiety Disorders Unit at St Vincent's Hospital, Sydney. The patients were treated for social phobia (n = 82) or panic disorder with (n = 18) or without (n = 17) agoraphobia. Their primary DSM-III-R diagnosis was made by a clinician at an initial interview. Only patients whose diagnosis was con®rmed using a structured diagnostic interview (the Composite International Diagnostic Interview, computerized version (WHO, 1993)) were included in the analysis. Diagnosis was con®rmed for 105 patients (73 with social phobia, 17 with panic disorder with agoraphobia and 15 with panic disorder without agoraphobia). The average age of the sample was 32.18 years (S.D.=8.58; range=18±63). For the purposes of analysis, the subjects were divided into 2 groups: social phobia and other anxiety disorder. The demographic characteristics of the social phobia and other anxiety disorder groups are described in Table 1. There were no signi®cant dierences between the groups in terms of age (F(1,103)=2.55; p = 0.11). The sex distributions of the two groups were signi®cantly dierent (w 2 (1)=6.62; p = 0.01): the male to female ratio was approximately equal for the social
Table 1 Demographic characteristics for the patients by group Social phobia (n = 73)
Other anxiety disorder (n = 32)
Age
mean S.D.
31.30 7.57
34.19 10.39
Sex
males (n ) females (n)
38 35
8 24
946
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
phobia group, whereas there were more females than males in the other anxiety disorder group. Patients with comorbid diagnoses (as de®ned by the CIDI) were included in the study: 66 patients in the social phobia group had one or more other anxiety disorders (59 had panic disorder with or without agoraphobia, 27 had a simple phobia, 10 had generalised anxiety disorder and 5 had obsessive compulsive disorder) and 28 patients in the other anxiety disorder group had one or more additional anxiety disorders (10 had social phobia, 23 had a simple phobia, 1 had generalised anxiety disorder and 3 had OCD). The three measures were administered prior to treatment. The Social Phobia and Anxiety Inventory (Turner et al., 1996) is a 45 item self-report questionnaire which measures the frequency (on a 7 point scale) with which the respondent experiences the ``somatic, cognitive and behavioral aspects of social phobia across a wide range of social situations and settings'' (Turner et al., 1996, p. 2). It is an empirically derived measure made up of 2 subscales, Social Phobia and Agoraphobia, with the total SPAI score being the dierence between the Social Phobia and Agoraphobia subscales. Since the present study assessed the ability of the scales to distinguish between social phobia and panic disorder with or without agoraphobia, the data reported here are for the total SPAI score (i.e. the dierence score). The Social Phobia Scale and the Social Interaction Anxiety Scale (Mattick & Clarke, 1998) were developed as a companion set of measures. The scales were designed to assess scrutiny fears (SPS) and fears of general social interaction (SIAS). Each scale has 20 items rated on a scale from 0 to 4 which indicates how true or characteristic the statements are of the respondent (`not at all' to `extremely'). 2. Results and discussion Similar to previous data on the construct validity of the three measures (Ries et al., 1998), they were found to be highly and signi®cantly correlated (SPS-SIAS: r = 0.73, p < 0.001; SPSSPAI: r = 0.72, p < 0.001; SIAS-SPAI: r = 0.85, p < 0.001) suggesting that the measures tap a similar construct. Table 2 presents the mean scores on the three measures for the social phobia and other anxiety disorders groups. The social phobia and other anxiety disorder groups were compared on the three dependent variables using a one-way between subjects' multivariate analysis of variance (MANOVA). Table 2 Scores on the measures for the social phobia and other anxiety disorder groups. p < 0.05a
SIAS SPS SPAI a
Social phobia (n = 74)
Other anxiety disorder (n = 34)
M
(s)
M
(s)
55.24 44.41 110.61
(12.97) (14.05) (25.10)
29.46 23.75 42.90
(17.29) (16.84) (38.29)
Multivariate F(3,101)=38.06; p < 0.05.
Univariate F
df
71.27 42.54 115.70
1,103 1,103 1,103
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
947
SPSS MANOVA was used for the analysis. With the use of Pillais' criterion, the combined DVs were signi®cantly dierent between the two groups (F(3,101)=38.06, p < 0.05). The scores on the SPAI, the SIAS and the SPS were higher for the social phobia group than for the other anxiety disorder group (see Table 2). All of the univariate F values were signi®cant (Table 2), con®rming previous research on these three measures which has shown that they are all able to make distinctions between social phobic patients and patients with other anxiety disorder diagnoses (Brown et al., 1997; Turner et al., 1989). Simply comparing the measures at a univariate level, however, does not provide evidence that one of the measures is better than the others for discriminating between social phobia and other anxiety disorders. To determine if one of the measures was better than the others, a logistic regression was performed. Logistic regression allows prediction of group membership from a set of variables (Tabachnick & Fidell, 1996)1. A test of the model with all three measures against a constant-only model was statistically reliable (w 2 (3, N = 105)=65.63; p < 0.001) indicating that the measures, as a set, reliably distinguish the social phobia group from the other anxiety disorder group. Logistic regression also allows for the assessment of contribution of individual variables to the prediction of group membership. Given that only the SPAI reliably predicted group membership (z = 12.02, p < 0.001), a second model without the SPAI was tested. The model without the SPAI was reliably dierent from the model with all three measures (w 2(1)=32.90, p < 0.05), suggesting that the ability of a model without the SPAI to predict membership of the social phobia versus the other anxiety disorders group is not as good as a model with the SPAI. Thus, it appears that the SPAI is a better predictor of social phobia group membership than the SPS and the SIAS. To support this conclusion and to provide guidelines as to the cuto scores on the measures, which de®ne social phobia as opposed to panic disorder with or without agoraphobia, a receiver operating characteristic (ROC) analysis was carried out. ROC analysis allows the ability of a test to discriminate diseased cases from normal cases (or in this case, social phobic cases from nonsocial phobic cases) to be evaluated (Hsiao, Bartko & Potter, 1989). The area under the ROC curve (AUC) summarizes the ability of the instrument to discriminate between those with the disorder of interest and those who do not have the disorder (Hanley & McNeil, 1982). The AUC value can also be used to compare the diagnostic performance of two or more tests (Griner, Mayewski, Mushlin & Greenland, 1981). In addition to the ROC analysis, four indicators of test performance were calculated for each of the three tests to allow recommendations for the best cuto scores to be made: sensitivity (proportion of true positives), speci®city (proportion of false negatives), positive predictive value (PPV; probability that the disease is present when the test is positive) and negative predictive value (NPV; probability that the disease is not present when the test is negative). The MedCalc program was used to calculate and compare the AUCs for the three tests and a commercial computer program (CUTOFF; McKenzie & Clarke, 1992) was used to calculate the test performance indicators of sensitivity, speci®city, PPV and NPV. The AUC for the SPAI was 0.93 (95% CI=0.87±0.97), for the SIAS was 0.86 (95%
1 Although the measures are intercorrelated, multicollinearity is unlikely to be a problem in the logistic regression since the parameter estimates in the model are relatively low (b < 3) (Tabachnick & Fidell, 1996).
948
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
CI=0.78±0.92) and for the SPS was 0.80 (95% CI=0.72±0.87). None of the 95% con®dence intervals around the AUC values include a value of 0.5 indicating that all of the tests perform better than chance in discriminating between those who have social phobia and those who have panic disorder with or without agoraphobia. Of more interest here, however, is the comparison between the three tests. The dierence between the AUC for the SPAI and the AUC for the SIAS was signi®cantly dierent (dierence=0.07; 95% CI=0.02±0.13; p = 0.008), as was the dierence between the SPAI and the SPS (dierence=0.13; 95% CI=0.06±0.20; p < 0.001). The dierence between the AUC for the SIAS and for the SPS was not dierent (dierence=0.05; 95% CI=ÿ0.020±0.13; p = 0.124). These results con®rm the multivariate analysis which showed that the SPAI was a better measure than the SPS and the SIAS for discriminating between social phobia and panic disorder with and without agoraphobia. Table 3 presents the test performance indicators for the three tests. Since these indicators can be computed for each possible cuto score on the test, only the values for the cuto which was determined by the ROC analysis to be the best cuto score are presented (the full range of values is available on request from the author). It should be noted that the scores found in this study to be the optimum cutos for social phobia versus panic disorder with or without agoraphobia are similar to the average scores for social phobics reported in other studies. For example, Turner et al. (1996; p. 13) report that a SPAI dierence score of 80 is the best for discriminating between agoraphobics and social phobics, compared to a score of 88 in the current study. Ries et al. (1998) report a mean score of 27.2 for generalised social phobics on the SPS compared to a score of 27 in the current study, although the mean score for generalised social phobics is higher in the Ries et al. (1998) study than in the current study (48.5 vs. 36). That the SPS and the SIAS did not appear to be the best measures for discriminating between social phobia and other anxiety disorders does not, however, mean that the measures are not valid. Used alone, the measures would make signi®cant discriminations, as shown by the signi®cant univariate F-tests, the AUC values and the indicators of test performance. The results of this study suggest that if the researcher is making a choice between instruments, the SPAI may be preferable if the purpose of the assessment is to discriminate between features of social phobia and of other anxiety disorders, particularly panic disorder with or without agoraphobia. Further research will need to be conducted to determine which, if any, of the three measures are more appropriate for other assessment purposes, for example, detecting Table 3 Indicators of test performance for the SPAI, SPS and SIAS at the optimum cuto scorea,b
SPAI SIAS SPS a
Cuto score
Sensitivity
Speci®city
PPV
NPV
> 88 > 36 > 26
0.87 0.93 0.89
0.91 0.60 0.66
0.96 0.84 0.86
0.74 0.78 0.72
The optimum cuto score is the one calculated by the ROC analysis to produce the best trade o between sensitivity and speci®city of the test. b PPV: positive predictive value; NPV=negative predictive value.
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
949
treatment eects (see Taylor, Woody, McLean & Koch, 1997, for a discussion of measuring sensitivity to change). In addition, the other anxiety disorder group in this study was made up of panic disorder with and without agoraphobia. An examination of the ability of the measures to discriminate between social phobia and dierent anxiety disorders is warranted. The ability of all three measures to discriminate between social phobia and panic disorder with or without agoraphobia is particularly impressive given the high levels of comorbidity between social phobia and panic disorder with and without agoraphobia in this study: 50.4% of patients with a primary diagnosis of social phobia also had panic disorder with or without agoraphobia as a secondary disorder, and 31.3% of patients with a primary diagnosis of panic disorder with or without agoraphobia group had secondary social phobia. Although the numbers are small, a post hoc analysis of the ability of the measures to discriminate between the social phobic patients who did not have comorbid panic disorder with or without agoraphobia and the other anxiety disorder patients who did not have comorbid social phobia yielded a similar pattern of results. All three measures signi®cantly discriminated between the two groups at the univariate level, but when compared in a logistic regression, only the SPAI signi®cantly predicted group membership. While this study suggests that the all three measures discriminate between social phobia and panic disorder with or without agoraphobia, the measures are limited in that they are measures of the sorts of situations which social phobics ®nd distressing, but they do not necessarily measure the reasons for the distress. Given the view embodied in the diagnostic classi®cation that the role of being judged critically is central to social phobia, measures that tap cognitions relating to this concern are necessary. While the FNE was once thought of as a promising measure for de®ning the cause of distress in social phobia Ð a fear of being evaluated negatively Ð that promise has not been kept. Fear of negative evaluation may well be a cognition that crosses the diagnostic boundaries. In a review of cognitive assessments of social phobia, Heimberg (1994, p. 270) lamented that ``a limited number of devices have actually been used in studies of treatment of social phobia''. One of the reasons for this may be that there are no standardized self-report measures of cognitions in social phobia available. Future work may go towards development of such a measure.
References Beidel, D. C., Turner, S. M., Stanley, M. A., & Dancu, C. V. (1989). The social phobia and anxiety inventory: concurrent and external validity. Behavior Therapy, 20, 417±427. Brown, E. J., Turovsky, J., Heimberg, R. G., Juster, H. R., Brown, T. A., & Barlow, D. H. (1997). Validation of the social interaction anxiety scale and the social phobia scale across the anxiety disorders. Psychological Assessment, 9, 21±27. Cox, B. J., & Swinson, R. P. (1995). Assessment and measurement. In M. B. Stein, Social phobia: clinical and research perspectives. Washington, DC: American Psychiatric Press, Inc. Griner, P. F., Mayewski, R. J., Mushlin, A. I., & Greenland, P. (1981). Selection and interpretation of diagnostic tests and procedures. Annals of Internal Medicine, 94, 555±600. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29±36. Hsiao, J. K., Bartko, J. J., & Potter, W. Z. (1989). Diagnosing diagnoses: receiver operating characteristic methods and psychiatry. Archives of General Psychiatry, 46, 664±667.
950
L. Peters / Behaviour Research and Therapy 38 (2000) 943±950
Heimberg, R. G. (1994). Cognitive assessment strategies and the measurement of outcome of treatment for social phobia. Behaviour Research and Therapy, 32, 269±280. Heimberg, R. G., Mueller, G. P., Holt, C. S., Hope, D. A., & Liebowitz, M. R. (1992). Assessment of anxiety in social interaction and being observed by others: the social interaction anxiety scale and the social phobia scale. Behavior Therapy, 23, 53±73. Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455±470. McKenzie, D. P., & Clarke, D. M. (1992). CUTOFF: a Fortran program for establishing thresholds for screening indices. Educational and Psychological Measurement, 52, 891±893. Oei, T. P. S., Kenna, D., & Evans, L. (1991). The reliability, validity and utility of the SAD and FNE scales for anxiety disorder patients. Personality and Individual Dierences, 12, 111±116. Ries, B. J., McNeil, D. W., Boone, M. L., Turk, C. L., Carter, L. E., & Heimberg, R. G. (1998). Assessment of contemporary social phobia verbal report instruments. Behaviour Research and Therapy, 36, 983±994. Tabachnick, B. G., & Fidell, L. S. (1996). Using multivariate statistics. New York, NY: Harper Collins Publishers. Taylor, S., Woody, S., McLean, P. D., & Koch, W. J. (1997). Sensitivity of outcome measures for treatment of generalized social phobia. Assessment, 4, 181±191. Turner, S. M., Beidel, D. C., & Dancu, C. V. (1996). Social phobia and anxiety inventory: manual. Toronto, Ont: Multi-Health Systems Inc. Turner, S. M., Beidel, D. C., Dancu, C. V., & Stanley, M. A. (1989). An empirically derived inventory to measure social fears and anxiety: the social phobia and anxiety inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 1, 35±40. Turner, S. M., McCanna, M., & Beidel, D. C. (1987). Validity of the social avoidance and distress and fear of negative evaluation scales. Behaviour Research and Therapy, 25, 113±115. Watson, D., & Friend, R. (1969). Measurement of social-evaluative anxiety. Journal of Consulting and Clinical Psychology, 33, 448±457. World Health Organization (1993). CIDI-Auto version 1.0: Administrator's guide and reference. Sydney: Training and Reference Center for WHO CIDI.