Reliability and validity of DSM-IV generalized anxiety disorder features

Journal of Anxiety Disorders 25 (2011) 813–821

Contents lists available at ScienceDirect

Journal of Anxiety Disorders

Reliability and validity of DSM-IV generalized anxiety disorder features夽 Dina Gordon, Richard G. Heimberg ∗ Adult Anxiety Clinic, Department of Psychology, Temple University, United States

a r t i c l e

i n f o

Article history: Received 26 October 2010 Received in revised form 30 March 2011 Accepted 10 April 2011 Keywords: Diagnostic criteria GAD Reliability Semi-structured interview Validity

a b s t r a c t The reliability of generalized anxiety disorder (GAD) features has been shown to be moderate, based on research utilizing the Anxiety Disorders Interview Schedule for DSM-IV (ADIS-IV), a semi-structured diagnostic interview. This may be a function of the criteria for the diagnosis of GAD, which have undergone much revision since its first inclusion in the Diagnostic and Statistical Manual of Mental Disorders. The reliability and validity of disorder feature ratings were examined in a diverse sample of patients who presented for assessment and treatment of excessive worry, generalized anxiety, or tension at an anxiety specialty clinic and who met criteria for a principal diagnosis of GAD (N = 129). Internal consistency of the ratings of excessiveness of worry, uncontrollability of worry, and the associated symptom cluster was moderate to low and varied by disorder feature. Inter-rater reliability for all features of GAD and severity of the disorder varied between good and poor. Additional findings showed that the GAD features, as measured using the ADIS-IV module, have modest to strong convergent validity, varying by feature, and poor discriminant validity when tested against measures of social anxiety. Potential reasons for rater disagreement are discussed. Results are also considered in terms of how they may inform the evolving criteria for GAD in DSM-V. © 2011 Elsevier Ltd. All rights reserved.

The diagnostic criteria for generalized anxiety disorder (GAD) have been constantly evolving (Barlow & Wincze, 1998), and controversy has historically surrounded this diagnosis. The Diagnostic and Statistical Manual of Mental Disorders – 4th edition (DSM-IV; American Psychiatric Association [APA], 1994) defines GAD as anxiety and worry about multiple events or activities in which the worry is excessive, difficult to control, and associated with at least 3 of 6 somatic symptoms, and the full syndrome is chronic (lasting at least 6 months) and disabling. When it first appeared in DSM-III (APA, 1980), GAD was a residual category, diagnosed only if a patient did not meet the criteria for any other anxiety or mood disorder. This operationalization of GAD was associated with low inter-rater reliability ( = .47; Di Nardo, O’Brien, Barlow, Waddell, & Blanchard, 1983). The reformulation of GAD in DSM-III-R (APA, 1987) failed to substantially improve reliability of the disorder, with kappa coefficients in the poor to fair range (’s for current GAD were .27, .53, and .56 in Mannuzza et al., 1989; Di Nardo, Moras, Barlow, Rapee, & Brown, 1993; Williams et al., 1992, respectively). A major change implemented in DSM-IV was the reduction of the number of associated symptoms from 18 to 6, as there was a very low rate

夽 Portions of this work were presented at the November 2010 meeting of the Association for Behavioral and Cognitive Therapies, San Francisco, CA. ∗ Corresponding author at: 1701 North 13 Street, Temple University, Philadelphia, PA 19122, United States. Tel.: +1 215 204 1575; fax: +1 215 204 5184. E-mail address: [email protected] (R.G. Heimberg). 0887-6185/$ – see front matter © 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.janxdis.2011.04.001

of endorsement of the autonomic symptom cluster (Marten et al., 1993), and this change was associated with increased diagnostic reliability ( = .67; Brown, Di Nardo, Lehman, & Campbell, 2001). Among the revisions made to the criteria for GAD in DSM-IV, a requirement was added that the worry must be perceived by the person as uncontrollable, a criterion thought to distinguish worry among persons with GAD from those with normal or normative levels of worry (Abel & Borkovec, 1995; Borkovec, 1994). Changes to the criteria for GAD are once again under consideration as the fifth edition of the diagnostic manual (DSM-V) approaches publication, and preliminary suggestions have been made to amend the diagnosis with the intent of further improving its reliability and validity (Andrews et al., 2010). Some of the changes being considered include removing the difficult-to-control criterion and reducing the number of associated symptoms from six to two (restlessness and muscle tension). Based on epidemiological studies, it has been reasoned that excessiveness should remain in the criteria because removing it would greatly increase the prevalence of GAD (Ruscio et al., 2005). Conversely, removing the uncontrollability criterion would not affect the prevalence of the disorder; therefore, it was asserted that this feature may not be essential to the diagnosis of GAD and its removal might be justified (Andrews et al., 2010). However, changes in prevalence do not necessarily inform validity as prevalence will certainly change with the amendment of the criteria, and these findings may not be sufficient to inform the decision of whether to keep one or both features of worry in the definition of GAD. An alternate possibility

814

D. Gordon, R.G. Heimberg / Journal of Anxiety Disorders 25 (2011) 813–821

is that the difficult-to-control criterion and the excessiveness criterion tap the same underlying construct and are to some extent redundant. The two features have been shown to be highly correlated in previous research (r = .91; Brown et al., 2001), potentially rendering the inclusion of both superfluous. The associated symptoms that remain in the DSM-IV were those most frequently endorsed from the larger DSM-III-R criteria set (Brawman-Mintzer et al., 1994; Turvey, Stevens, & Merikangas, 1999). However, this method of editing the associated symptom list may have been flawed in that the symptoms endorsed more rarely may simply be indicative of more severe GAD (Kubarych, Aggen, Hettema, Kendlerm, & Neale, 2005). Another way in which the associated symptoms have been scrutinized is on the basis of epidemiological studies, and a reduction in the number of associated symptoms does not appear to affect the prevalence of GAD (Ruscio et al., 2007). However, as with excessiveness and controllability, epidemiological findings regarding prevalence do not determine validity, so this finding may not be sufficient to inform the decision of whether or not to retain all associated symptoms as part of the GAD criteria set. The specificity of the associated symptoms is open to further consideration; for example, the symptoms of restlessness and muscle tension seem to be most specific to GAD and might therefore be retained in DSM-V, whereas the remaining symptoms, which are also characteristic of mood disorders and posttraumatic stress disorder, might be eliminated (Andrews et al., 2010). Research demonstrating that some associated symptoms are more strongly related to major depressive disorder (MDD) than to GAD (Joormann & Stöber, 1999) and that GAD shares many somatic symptoms with MDD and dysthymia (Mennin, Heimberg, Fresco, & Ritter, 2008) implies a lack of specificity of the associated symptoms to GAD. Joormann and Stöber (1999) found that only muscle tension was uniquely associated with pathological worry, whereas difficulty concentrating related specifically to depression. This led us to hypothesize that the associated symptoms would not form a cohesive symptom group. Semi-structured interviews are frequently used in the diagnostic assessment of clinical populations, and information about the reliability and validity of resultant diagnoses is of great importance. The Anxiety Disorders Interview Schedule for DSM-IV (Current version: ADIS-IV, Brown, Di Nardo, & Barlow, 1994; Lifetime version: ADIS-IV-L, Di Nardo, Brown, & Barlow, 1994) is a commonly used interview in studies of the assessment and treatment of anxiety disorders. Thus far, the ADIS-IV has been shown to have good or excellent diagnostic reliability for most anxiety disorders, with  = .67 for current GAD as a principal diagnosis and  = .65 for lifetime GAD (Brown et al., 2001) in a mixed sample of anxiety disorder patients; this is a significant improvement over the  of .57 derived using the Anxiety Disorders Interview Schedule-Revised (Di Nardo & Barlow, 1988) and DSM-III-R (APA, 1987) criteria, and it may be attributable to the revision of the criteria for GAD in DSM-IV (Brown et al., 2001). Inter-rater reliability for dimensional ratings of features of GAD was adequate, as represented by correlations for excessiveness of worry, uncontrollability of worry, associated symptoms, and the ADIS clinical severity rating at r = .73, .78, .83, and .72, respectively (Brown et al., 2001). Potential sources of unreliability were attributed to the high comorbidity of GAD with other disorders and the subjective features of GAD (e.g., severity and quantity of worry domains and somatic symptoms) in relation to other conditions (e.g., mood disorders), which may be suggestive of the ambiguity of these diagnostic features and patients’ difficulty differentiating them from the features of other disorders. Although promising, few other studies of the reliability of ratings of GAD features using the ADIS-IV and DSM-IV criteria have been conducted. Only one other study examined the reliability of all GAD features using the ADIS-IV (Brown et al., 2001). One other study reported on the reliability of GAD features, but it utilized

DSM-III-R criteria and only examined the associated symptom criterion (Marten et al., 1993). Two other studies which utilized DSM-IV were also limited by their examination of just the associated symptom criterion, and they examined only the validity of this criterion, not its reliability (Brown, Marten, & Barlow, 1995; Turvey et al., 1999). The primary aim of the present study was to examine the internal consistency and inter-rater agreement of feature ratings made within the GAD module of the ADIS-IV in a diverse urban treatment-seeking sample of individuals with GAD. The focus on features of GAD in the current study is prompted by the need to examine symptom-level reliability, as this contributes to (and may ultimately limit) the subsequent establishment of diagnostic reliability. We also sought to further explore the associated symptoms of GAD in terms of their internal consistency and association with other features of GAD. Another purpose of the present study was to evaluate the convergent and discriminant validity of the various symptom ratings that make up the GAD module by examining the extent to which these ratings correlated with other measures related to GAD or its features (e.g., excessive worry) and were less related to measures of another anxiety disorder (i.e., social anxiety disorder). Findings from this study should help elucidate the reliability and validity of GAD features and may also inform decisions regarding revision of the GAD criteria for DSM-V. 1. Method 1.1. Participants Participants were 129 patients presenting for assessment and treatment of excessive worry, generalized anxiety, or tension at the Adult Anxiety Clinic at Temple between January of 1999 and January of 2010. Women constituted the larger portion of the sample (65%); average age was 33.21 (SD = 12.32, range = 18–81). Of the 126 patients providing information on ethnicity, the breakdown was 98 (78%) Caucasian, 17 (13%) African American, 2 (2%) Native American, 2 (2%) Pacific Islander, and 7 (5%) other. All individuals had a principal diagnosis of GAD. Diagnoses were based on DSM-IV (APA, 1994) criteria and assessed by the Anxiety Disorders Interview Schedule for DSM-IV-Lifetime Version (ADIS-IV-L; Di Nardo et al., 1994). Data from 96 patients were drawn from two independent assessments. The first was conducted upon intake and included administration of the entire ADIS-IV-L, and the second was completed before initiation of treatment or placement on the waiting list as part of a randomized controlled trial (median = 27 days; range = 3–260 days1 ) and included only the ADIS-IV-L GAD module (plus other measures specifically related to GAD). Thirtythree patients received the initial interview but did not complete the second interview because they were no longer interested in treatment or were referred to other venues. Self-report questionnaires were completed by participants at intake or at home during the time between the two assessments. 1.2. Measures 1.2.1. Clinician-administered The ADIS-IV-L (Di Nardo et al., 1994) is a semi-structured interview designed to establish reliable diagnoses of the DSM-IV anxiety, mood, somatoform, and substance use disorders. Diagnosticians were advanced graduate students or clinical psychologists, trained

1 A total of 12 patients had more than 3 months between assessments, four of whom had 200 or more days between assessments. All patients were included in the statistical analyses. However, reananlyses excluding these 12 patients yielded identical results to those reported herein.

D. Gordon, R.G. Heimberg / Journal of Anxiety Disorders 25 (2011) 813–821

to strict reliability standards (for details on training procedures, see Brown et al., 2001). For each diagnosis, clinicians assigned a dimensional clinical severity rating (CSR), ranging from 0 = none to 8 = very severely disturbing/disabling, with scores of 4 or greater indicative of the severity of distress and impairment that would meet criteria for diagnosis. Interviewers have demonstrated good interrater reliability for the principal diagnosis of GAD in a mixed sample of anxiety disorder patients ( = .67; Brown et al., 2001). Ratings (0–8 scale) in the GAD section examined in the present study include (a) excessiveness and uncontrollability of worry in eight domains, (b) frequency and severity of the six associated symptoms, (c) interference and distress due to worry and associated symptoms, and (d) the CSR for GAD. We did not examine agreement on diagnosis per se because there was insufficient independence between the interviewers at this level (second interviewers were aware that patients had received a principal diagnosis of GAD in some instances but unaware of the specific ratings assigned); therefore, the focus was on the features of GAD. 1.2.2. Self-report Patients completed several self-report scales that were utilized in the present study. These measures assessed pathological worry, social anxiety (employed here to examine discriminant validity of the ratings in the ADIS GAD module), and measures of functional impairment and life satisfaction. Social anxiety was chosen as an index of discriminant validity because the sample available in our clinic was limited to clients with principal diagnoses of GAD or social anxiety disorder (SAD). Previous research has successfully utilized SAD as a comparison group in the evaluation of the psychometric characteristics of the Penn State Worry Questionnaire (Fresco, Mennin, Heimberg, & Turk, 2003) and GAD as a comparison group in the evaluation of a social anxiety measure (the Liebowitz Social Anxiety Scale; Heimberg & Holloway, 2007). The Penn State Worry Questionnaire (PSWQ; Meyer, Miller, Metzger, & Borkovec, 1990) consists of 16 items rated on a 1–5 Likert-type scale (not at all typical of me to very typical of me) and assesses the extent to which worry is excessive, uncontrollable, and pervasive (e.g., “I am always worrying about something”). A total score is calculated by summing the items (with 5 items reversescored). The PSWQ has excellent internal consistency (˛ = .91; Meyer et al., 1990) and has been shown to discriminate individuals with GAD, other anxiety disorders, and community controls (Brown, Antony, & Barlow, 1992). Internal consistency has also been demonstrated in both undergraduate and clinical samples, with alphas ranging from .86 to .93 (Molina & Borkovec, 1994). In addition, correlations between the PSWQ and measures of anxiety, depression, and emotional control have supported the convergent and discriminant validity of the measure (Brown et al., 1992). Good test–retest reliability (r = .74 to .92) across time frames of 2–10 weeks in undergraduate samples (Meyer et al., 1990; Molina & Borkovec, 1994; Stöber, 1998) has also been demonstrated. The PSWQ is significantly correlated with other measures of anxiety (r = .40 to .74) and depression (r = .36, Molina & Borkovec, 1994), along with worry (e.g., Davey, 1993), and is sensitive to change across 6-week and 12-week treatment interventions for GAD (Borkovec & Costello, 1993). Cronbach’s alpha was .86 in the current sample (n = 99). The Generalized Anxiety Disorder Questionnaire for DSM-IV (GADQ-IV; Newman et al., 2002) is a 9-item self-report screening questionnaire that samples the criteria for GAD as delineated in the DSM-IV (APA, 1994). Items are phrased according to DSM-IV criteria. Most items are dichotomous and measure the excessive and uncontrollable nature of worry as experienced by persons with GAD (e.g., “During the last six months, have you been bothered by excessive worries more days than not?”) and associated somatic symptoms, including restlessness, sleep disturbance, difficulty con-

815

centrating, irritability, fatigue, and muscle tension. One item is open-ended and asks for a listing of the most frequent worry topics. Two items are rated on a scale of zero to eight (none to very severe) and measure functional impairment and subjective distress. The GAD-Q-IV’s dimensional scoring system provides an overall index of the severity of GAD. For each of the first four questions, a participant can receive a score of one for a “yes” response and a zero for a “no” response. For item five, participants can list up to six areas of worry and the total score is divided by three. Similarly, for item seven, each physical symptom that is endorsed earns one point and the total score is divided by three. Finally, for items eight and nine, the number chosen (ranging from 0 to 8) is divided by four. All the numbers are added together for a score ranging from 0 to 13. A total score of 5.7 or above suggests a diagnosis of GAD with the best balance of sensitivity and specificity (Newman et al., 2002). The factor structure of these items appears to be unitary (Rodebaugh, Holoway, & Heimberg, 2008). The GAD-Q-IV has demonstrated good internal consistency (˛ = .84) and good 2-week test–retest reliability (r = .81) in a college sample. The measure correlates well with measures of excessive worry (r = .63) and trait anxiety (r = .58). It also shows a modest correlation with depression (r = .23) and is uncorrelated with conceptually unrelated measures (Newman et al., 2002). Finally, validity of the GAD-Q-IV is supported by agreement with diagnoses derived on the basis of the ADIS-IV ( = .70; Newman et al., 2002). The Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) is a 20-item measure employing a 0–4 Likert-type scale (0 = not at all characteristic of me, 4 = extremely characteristic of me). The items describe anxiety-related reactions to a variety of social interaction situations involving dyads and groups. Example items include “I find it easy to make friends of my own age” and “When mixing in a group, I find myself worrying I will be ignored.” The SIAS has been shown to be internally consistent (˛ = 0.88) and stable over time (re-test coefficient >.90; Mattick & Clarke, 1998). Rodebaugh, Woods, and Heimberg (2007) have reported that the 17 straightforwardly worded items of the SIAS are more valid indicators of social interaction anxiety than the reverse-scored items in both undergraduate and clinical samples (these items appear to be better indicators of extraversion). Consequently, Rodebaugh et al. (2007) suggested the scoring strategy of utilizing only the straightforward SIAS items to calculate the total score, thereby yielding a 17-item score, hereafter referred to as the SIAS-Straightforward (SIAS-S) score. The SIAS-S has demonstrated excellent internal consistency (˛ = .93) and factorial validity in undergraduate samples and has demonstrated strong construct validity in both undergraduate and clinical samples (Rodebaugh et al., 2007). The 20-item SIAS was administered; however, only the straightforward items (SIAS-S) were utilized in the present analyses. Cronbach’s alpha was .96 in the current sample (n = 64). The Mini-Social Phobia Inventory (Mini-SPIN; Connor, Kobak, Churchill, Katzelnick, & Davidson, 2001) is a 3-item self-report screening instrument used to assess social anxiety disorder. The Mini-SPIN is an abbreviated version of the Social Phobia Inventory (SPIN; Connor et al., 2000), a 17-item scale that has demonstrated good reliability, significant correlations with related measures, and the ability to discriminate between individuals with SAD and those with other disorders. The Mini-SPIN items are rated on a 5point Likert-type scale, ranging from 0 (Not at all) to 4 (Extremely), and summed. A cutoff score of 6 yielded strong sensitivity (88.7% of actual cases of SAD correctly identified), specificity (90.0% of patients without SAD correctly identified) and diagnostic efficiency (89.9% overall hit rate) in a sample of treatment seekers (Connor et al., 2001). Additionally, Seeley-Wait, Abbott, and Rapee (2009) found that the Mini-SPIN demonstrated good test-retest reliability (r = .70) and excellent internal consistency (˛ = .91). The Mini-SPIN also correlated moderately with other measures of SAD and fear of

816

D. Gordon, R.G. Heimberg / Journal of Anxiety Disorders 25 (2011) 813–821

Table 1 Internal consistency of GAD module ratings. Excessiveness of worry domains

Uncontrollability of worry domains

Severity of associated symptoms

˛

No. items

˛

No. items

˛

No. items

.60

8

.60

8

.37

6

Note: Excessiveness n = 129; uncontrollability n = 128; associated symptoms n = 128. GAD = generalized anxiety disorder. Differences in sample size are due to missing data.

negative evaluation but does not correlate with measures of generalized anxiety disorder (Seeley-Wait et al., 2009; Weeks, Spokas, & Heimberg, 2007). It has shown utility as a screening instrument in primary care (Connor et al., 2001) and in an anxiety specialty clinic (Weeks et al., 2007). Cronbach’s alpha was .80 in the current sample (n = 68). The Liebowitz Self-Rated Disability Scale (LSRDS; Schneier et al., 1994) is a 22-item measure that examines current and lifetime impairment resulting from the individual’s emotional problem in 11 domains (e.g., education, career, romantic relationships). Possible responses on a 4-point scale range from 0 to 3 (problem does not limit me at all to problem limits me severely). The LSRDS scales have demonstrated high internal consistency (current ˛ = .75; lifetime ˛ = .82; Hambrick, Turk, Heimberg, Schneier, & Liebowitz, 2004) and moderate to strong relationships with measures of symptoms and role functioning among individuals with SAD (Schneier et al., 1994; Wittchen, Fuetsch, Sonntag, Muller, & Liebowitz, 1999). In the present study, Cronbach’s alpha for current impairment was .71 (n = 53). The Sheehan Disability Scale (SDS; Sheehan, 1983) is a self-report measure of current disability due to emotional symptoms. It is comprised of three items assessing impairment at work, in social relationships, and in responsibilities at home and with family. Each of these items is rated on an 11-point Likert-type scale from 0 to 10 (not at all to very severe). The SDS also includes an additional global rating of work and social disability. The sum of the first three items of this four-item scale was used in this study, indicating the patient’s current level of disability in work, social life and leisure activities, and family life. The SDS is internally consistent, ˛ = .89 (Leon, Olfson, Portera, Farber, & Sheehan, 1997), and is a sensitive measure of impairment for a broad constellation of disorders (Olfson et al., 1997). However, the three-item SDS Total Score showed relatively low internal consistency (˛ = .55) in a clinical sample of patients with SAD (Hambrick et al., 2004), a finding that is common for brief scales. Cronbach’s alpha was .71 in the current sample (n = 106). 2. Results 2.1. Internal consistency Internal consistency (Cronbach’s alpha) of the dimensional ratings of excessiveness and uncontrollability for each of the eight worry domains, along with that of the six associated symptoms, was calculated to examine whether each of these item sets should be treated as scales and to explore any meaningful relationships

among the six associated symptoms. Given the heterogeneity of concerns in GAD, we were uncertain of whether the severity of excessiveness and uncontrollability among worry domains would be internally consistent. However, we expected that the associated symptoms would fail to exhibit satisfactory internal consistency due to their overlap with other anxiety and mood disorders. Alpha coefficients are presented in Table 1 for ratings made by the original diagnostic interviewer. These coefficients for excessiveness and uncontrollability ratings, each of which are based on eight items, were moderate. Consistent with expectation, alpha for the six associated symptoms was low. The frequency with which each associated symptom was endorsed at a clinical level (rating of 4 or greater) was as follows: 94% of the sample endorsed experiencing restlessness, 84% endorsed difficulty concentrating, 84% endorsed sleep difficulties, 73% fatigue, 73% irritability, and 68% muscle tension. 2.2. Inter-rater reliability Inter-rater reliability analyses were conducted for GAD module ratings using intraclass correlation coefficients (ICCs). Because ICCs are interpreted in a manner similar to kappa coefficients (Fleiss & Cohen, 1973), interpretation of ICCs was based on guidelines used in previous studies using kappa coefficients to assess the reliability of anxiety and mood disorders (e.g., Brown et al., 2001; Di Nardo et al., 1993). According to these standards, excellent agreement is indicated by r ≥ .75, good agreement is indicated by .60 ≤ r ≤ .74, fair agreement is indicated by .40 ≤ r ≤ .59, and poor agreement is indicated by r < .40. As demonstrated in Table 2, total excessiveness, total uncontrollability, and interference due to worry showed good agreement. Agreement was fair for the severity of diagnosis (CSR) and poor for distress due to worry. The agreement for severity of six associated symptoms varied by symptom, from good to poor, with good agreement for fatigue, fair agreement for irritability, muscle tension, sleep disturbance, and concentration difficulties, and poor agreement for restlessness (Table 3). Additionally, percent agreement for the CSR, interference due to worry, and distress due to worry was calculated by dividing the total number of ratings made by both raters that were within one point of each other by the total number of ratings made by both raters. The criterion to which assessors are trained is to be reliable within 1 CSR point, which is why we report these results in this manner. Inter-rater agreement for the CSR was 97.6% (82/84), for distress due to worry, 83.5% (66/79), and for interference due to worry, 70.5% (55/78). Percent agreement does not take into account

Table 2 Inter-rater reliability of GAD module ratings. Total excessiveness of worry domains

Total uncontrollability of worry domains

Interference due to worry

Distress due to worry

r

r

r

r

n ***

.60

96

n ***

.59

96

n ***

.62

78

Clinical severity rating for GAD n

*

.30

79

r

n ***

.49

Note: Agreement was assessed using intraclass correlation coefficients. GAD = generalized anxiety disorder. Differences in sample size are due to missing data. * p < .05. *** p < .001.

84

D. Gordon, R.G. Heimberg / Journal of Anxiety Disorders 25 (2011) 813–821

817

Table 3 Inter-rater reliability of GAD module associated symptom severity ratings. Restlessness

Difficulty concentrating

Irritability

Muscle tension

Difficulty sleeping

r

n

Fatigue r

n

r

n

r

n

r

n

r

n

.22*

96

.65***

96

.50***

96

.56***

96

.52***

96

.43***

96

Note: Agreement was assessed using intraclass correlation coefficients. GAD = generalized anxiety disorder. * p < .05. *** p < .001.

agreement due solely to chance; therefore, these results should be considered with some caution. Additionally, although the dimensional ratings have a possible range from 0 to 8, the actual range was smaller and more accurately characterized as 4–8 because 4 is the clinical cut-off and the sample consisted of patients with a principal diagnosis of GAD. Therefore, given the combination of restricted range and the allowance of agreement within one point, these percentages of agreement are considered moderate to fair, which is more or less consistent with the more conservative ICC coefficients reported above. 2.3. Correlations among GAD symptoms The Pearson correlation between total excessiveness and total uncontrollability was very high (r = .91, p < .001), matching the magnitude of the correlation previously reported by Brown et al. (2001). However, correlations between pairs of associated symptoms were quite modest. Restlessness was significantly correlated with difficulty concentrating (r = .28, p < .001) and irritability (r = .20, p < .01). Fatigue and sleep disturbance were also significantly correlated (r = .18, p < .01). No other correlations between pairs of associated symptoms were significant. Correlations were also calculated to examine whether associated symptoms were related to other features of GAD, including total excessiveness, total uncontrollability, distress due to worry, interference due to worry, and the CSR, in an attempt to clarify how integral the associated symptoms are to the diagnosis of GAD. Results, presented in Table 4, show that concentration difficulties were significantly related to all other associated features and to the severity of diagnosis, and fatigue was related to three of the other features. The remaining symptoms (irritability, muscle tension, sleep disturbance, and restlessness) were correlated with no more than two other features.

sures of pathological worry were taken to demonstrate support for the convergent validity of these GAD module ratings, whereas relatively lower correlations with the measures of social anxiety would be taken to indicate discriminant validity. Correlations with measures of functional impairment were examined as constructs generally related to the diagnosis of GAD. Total excessiveness and uncontrollability for the eight worry domains, degree of interference and distress due to worry, and the CSR for GAD were each correlated with these instruments. Correlations were conducted separately for the total sample (N = 129), for a subsample consisting of patients who did not receive a comorbid diagnosis of SAD (n = 74; 57% of total sample), and for a subsample consisting of patients who received diagnoses of both GAD and SAD (n = 55; 43% of total sample). Results are presented in Tables 5–7, respectively. In the full sample, total excessiveness of worry was significantly correlated with both the PSWQ and the GAD-Q-IV to a modest degree. Total uncontrollability of worry, the CSR, and distress due to worry were also correlated with the PSWQ but not the GAD-Q-IV. Correlations between feature ratings and measures of functional impairment (i.e., LSRDS-Current, SDS), were modest to strong, as demonstrated by 10/10 significant correlations ranging from .31 to .67. However, the GAD feature ratings failed to show good discriminant validity in the total sample. The magnitude of correlations with the discriminant measures (i.e., Mini-SPIN and SIAS) was higher than for some convergent measures. This unexpected finding prompted post hoc analyses utilizing a subsample of patients without comorbid SAD, as it was hypothesized that the poor discriminant validity may be accounted for by an additional diagnosis of SAD, considering the discriminant measures assess social anxiety. Discriminant validity was somewhat stronger in the subsample of patients without comorbid SAD, in which only total excessiveness of worry was significantly correlated with the Mini-SPIN and total excessiveness and total uncontrollability were significantly correlated with the SIAS.

2.4. Convergent and discriminant validity 3. Discussion Self-report instruments measuring constructs related to pathological worry (i.e., PSWQ; GAD-Q-IV), functional impairment (LSRDS-Current; SDS), and social anxiety (Mini-SPIN; SIAS) were correlated with the GAD feature ratings to examine convergent and discriminant validity of these ratings. Correlations with mea-

The current study utilized the ADIS-IV interview to explore features of GAD by examining internal consistency, inter-rater reliability, and associations among symptoms. We were also interested in convergent and discriminant validity of GAD features as

Table 4 Correlations between severity of associated symptoms and other features of generalized anxiety disorder. Symptom

Restlessness Fatigue Concentration Irritability Muscle tension Sleep disturbance

Total excessiveness of worry domains

Total uncontrollability of worry domains

Interference due to worry

Distress due to worry

Clinical severity rating for GAD

r

n

r

n

r

n

r

n

r

n

.16 .32** .25** .36** .20* .07

128 128 128 128 128 128

.10 .31** .19* .37** .22* .17

128 128 128 128 128 128

.08 .37** .22* .18 .05 .10

119 119 119 119 119 119

.14 .18 .21* .14 .13 .20*

119 119 119 119 119 119

.11 .17 .19* .14 .08 −.03

125 125 125 125 125 125

Note. Correlations calculated on the basis of the original diagnostic interview for total sample. Sample sizes differ due to missing data. * p < .05. ** p < .01.

818

D. Gordon, R.G. Heimberg / Journal of Anxiety Disorders 25 (2011) 813–821

Table 5 Convergent and discriminant validity of GAD module ratings in total sample (N = 129). Measure

PSWQ GAD-Q-IV LSRDS-Current SDS Mini-SPIN SIAS

Total excessiveness of worry domains

Total uncontrollability of worry domains

Interference due to worry

Distress due to worry

Clinical severity rating for GAD

r

r

r

r

r

n **

.27 .25* .31* .38*** .39** .46***

n **

98 67 60 105 79 65

.26 .20 .34** .36*** .42*** .44***

98 67 60 105 67 65

n

.16 .12 .67*** .48*** .29* .23

n **

76 61 40 83 61 50

.32 .13 .45** .37** .18 .09

n *

76 61 40 83 61 50

.20 .19 .49*** .45*** .40** .22

97 65 58 103 65 65

Note: Validity analyses performed on the basis of the original diagnostic interview. Sample sizes differ due to missing data. GAD = generalized anxiety disorder; PSWQ = Penn State Worry Questionnaire; GAD-Q-IV = Generalized Anxiety Disorder Questionnaire for DSM-IV; Mini-SPIN = Mini Social Phobia Inventory; SIAS = Social Interaction Anxiety Scale; LSRDS = Leibowitz Self-Rated Disability Scale (current = past two weeks); SDS = Sheehan Disability Scale. * p < .05. ** p < .01. *** p < .001. Table 6 Convergent and discriminant validity of GAD module ratings in the subsample without comorbid social anxiety disorder (N = 74). Measure

PSWQ GAD-Q-IV LSRDS-current SDS Mini-SPIN SIAS

Total excessiveness of worry domains

Interference due to worry

Interference due to worry

Distress due to worry

Clinical severity rating for GAD

r

n

r

n

r

n

r

n

r

n

.29* .18 .33* .40** .44* .37*

60 32 40 65 32 40

.26* .18 .34* .43*** .34 .41**

60 32 40 65 32 40

.16 .08 .69*** .51*** .22 .23

47 32 27 53 32 29

.38** .21 .54** .39** .14 .02

47 32 27 53 32 29

.23 .19 .60*** .52*** .24 .24

59 30 38 63 30 40

Note: Validity analyses performed on the basis of the original diagnostic interview. Sample sizes differ due to missing data. GAD = generalized anxiety disorder; PSWQ = Penn State Worry Questionnaire; GAD-Q-IV = Generalized Anxiety Disorder Questionnaire for DSM-IV; Mini-SPIN = Mini Social Phobia Inventory; SIAS = Social Interaction Anxiety Scale; LSRDS = Leibowitz Self-Rated Disability Scale (current = past two weeks); SDS = Sheehan Disability Scale. * p < .05. ** p < .01. *** p < .001.

demonstrated by their correlation with scores on measures of varying relation to the disorder. Features of GAD were emphasized in present analyses, as ratings of features of the disorder are the basis on which diagnoses are ultimately determined. Data from the present study indicate that GAD features can be fairly reliably rated by pairs of assessors who conduct a semistructured interview at different times. Inter-rater reliability varied for different ratings within the GAD module. Excessiveness and uncontrollability of worries, interference due to worries, and the severity of fatigue showed good inter-rater reliability. Distress due to worry exhibited the lowest inter-rater reliability. Internal consistency of excessiveness and uncontrollability of worry was moderate, but, for the associated symptoms, it was low, a finding supported by examination of the correlations among the six symptoms, which resulted in only 3 of 15 significant associations. Two of the six symptoms, fatigue and difficulty concentrating, were corre-

lated with three to five GAD module ratings, whereas the remaining symptoms – restlessness, muscle tension, irritability, and sleep disturbance – were only correlated with no more than two other features of GAD. The five features of the GAD module examined here (excessiveness and uncontrollability of worry, interference and distress due to worry, and the CSR) were highly related to measures of functional impairment. Convergent validity of the GAD module ratings was modest in relation to the PSWQ and almost negligible in relation to the GAD-Q-IV. Discriminant validity was also poor in the full sample, based on the many significant correlations with the Mini-SPIN and SIAS, which were often as large as or larger than the correlations with measures of worry and GAD features. However, discriminant validity improved when individuals with comorbid SAD were removed from analyses, likely due to the fact that discriminant measures assessed social anxiety and were more likely

Table 7 Convergent and discriminant validity of GAD module ratings in the subsample with comorbid social anxiety disorder (N = 55). Measure

PSWQ GAD-Q-IV LSRDS-current SDS Mini-SPIN SIAS

Total excessiveness of worry domains

Total uncontrollability of worry domains

Interference due to worry

Distress due to worry

r

n

r

n

r

n

r

n

r

n

.05 .28 .07 .13 .15 .30

39 35 21 41 35 26

.15 .18 .19 .06 .30 .27

39 35 21 41 35 26

.05 .15 .52 .24 .24 .19

30 29 14 32 29 22

.18 .05 .11 .28 .22 .22

30 29 14 32 29 22

−.04 .14 .03 .07 .43* .09

39 35 21 41 35 26

Clinical severity rating for GAD

Note: Validity analyses performed on the basis of the original diagnostic interview. Sample sizes differ due to missing data. GAD = generalized anxiety disorder; PSWQ = Penn State Worry Questionnaire; GAD-Q-IV = Generalized Anxiety Disorder Questionnaire for DSM-IV; Mini-SPIN = Mini Social Phobia Inventory; SIAS = Social Interaction Anxiety Scale; LSRDS = Leibowitz Self-Rated Disability Scale (current = past two weeks); SDS = Sheehan Disability Scale. * p < .05.

D. Gordon, R.G. Heimberg / Journal of Anxiety Disorders 25 (2011) 813–821

endorsed by those with both GAD and SAD diagnoses. A possible explanation for why so few correlations were significant in the subsample of patients with comorbid SAD is the smaller sample size, and thus reduced power, in this group. Ratings of excessiveness and uncontrollability were very highly correlated, replicating the finding reported by Brown et al. (2001). Inter-rater reliability reported in this study was lower than that previously reported by Brown et al. (2001). However, this discrepancy may be reconciled in several ways. For instance, several of the present analyses utilized ICCs, more conservative statistics than the Pearson rs used by Brown et al. (2001). Additionally, the current sample included only patients with a principal diagnosis of GAD, whereas the Brown et al. (2001) used a mixed anxiety sample (with both principal and additional diagnoses of GAD and more variability in symptom severity); therefore, the current sample had a more restricted range of symptom severity scores, which may have contributed to the lower inter-rater reliability reported here. Furthermore, Brown et al.’s sample included 362 participants, whereas the current study had a sample of only 129. In contrast to the ample research on diagnostic reliability of GAD, reliability of GAD features has been rarely examined. It is not surprising that feature reliability is lower than diagnostic reliability, especially as interviewers are trained to be reliable on the presence and severity of the diagnosis, not on features of the diagnosis. Excessiveness, uncontrollability, severity of fatigue, and interference due to worry, which exhibited good inter-rater reliability, may perhaps be easier to rate reliably because patients are able to provide specific, concrete examples of these features. The low inter-rater reliability of distress due to worry may be a function of its more subjective nature and the difficulties that patients may have quantifying the degree of distress they experience or providing concrete examples of their worry-induced distress, consistent with findings of elevated alexithymia (Abel, 1994; Yamas, HazlettStevens, & Borkovec, 1997) and difficulties in emotion regulation (Mennin, Heimberg, Turk, & Fresco, 2005) among patients with GAD. The low internal consistency of the associated symptom cluster and the general lack of intercorrelation among these six symptoms indicate that they are not highly related and should not be considered a cohesive symptom group. Two of the symptoms were related to the majority of other GAD features, suggesting that these may be more essential to the criteria set than the others. This finding cannot speak to Andrews et al.’s (2010) position that restlessness and muscle tension are the most specific to GAD, as opposed to other disorders, such as major depressive disorder or post-traumatic stress disorder, because the current study included only participants with a principal diagnosis of GAD. There exists continued debate regarding what should be done about overlapping symptoms in regard to assignment of diagnoses (Mennin et al., 2008), but because the present sample did not include patients with a principal diagnosis other than GAD, these results cannot substantially inform that debate. It is interesting that restlessness was the symptom most commonly endorsed in patients with GAD but was also the symptom least associated with other features of GAD and least reliably rated by the two independent interviewers. The fact that fatigue and concentration correlated with many other features of GAD is important to keep in mind when deciding which symptom criteria to retain in DSM-V, rather than relying solely on epidemiological studies that may not speak very strongly to validity of these symptoms in the clinical presentation of GAD.2

2 In order to further examine the potential reasons for the low alpha obtained for the six associated symptoms, symptom ratings were subjected to exploratory factor analysis with principal axis factoring and promax rotation. Four symptoms loaded onto a three-factor solution, with the first factor including restlessness and

819

The five indices from the GAD module correlated highly with measures of functional impairment, as expected. However, it is unclear why these indices correlated poorly with the PSWQ and the GAD-Q-IV, as these self-report measures were included here to assess convergent validity. Poor discriminant validity may be due, in part, to the fact that measures of SAD were used for this purpose, and GAD and SAD are highly related. A substantial area of conceptual overlap between the two disorders involves the nature of social worries or concerns about others’ evaluations or interpersonal concerns in GAD (Eng & Heimberg, 2006; Erickson & Newman, 2007). Future investigation into discriminant validity should include instruments that measure symptoms or disorders that are more distinct. The high correlation between excessiveness and uncontrollability of worry may be explained in various ways. First, it is possible that the format of the ADIS GAD module artifactually increased the similarity of ratings, as interviewers often ask about the excessiveness and uncontrollability of each domain concurrently, which may allow the rating for one to be affected by the rating of the other. Another possibility is that these two features of GAD may be tapping essentially the same aspect of the disorder. That is, excessive worry may be worry that is difficult to control. If this explanation is to be considered, it may be prudent to ask patients to report on just excessiveness or uncontrollability, rather than both. Andrews et al. (2010) came to the conclusion, based on epidemiological research, that the difficult-to-control criterion should be removed from the GAD criterion set because its absence would not affect prevalence rates. However, the present study employed a clinical sample in which excessiveness and uncontrollability are very highly correlated, thereby suggesting that it does not matter which feature is kept and which is removed. Although epidemiological studies may reflect the reality of how symptoms affect prevalence and shed light on public health concerns, they do not determine the validity of the criteria. There are a variety of potential reasons for the relatively modest inter-rater reliability findings in the present study. Despite extensive training, the skills of interviewers may vary. Similarly, because the ADIS-IV is a semi-structured interview, it allows the assessor flexibility in clarifying patients’ responses; although the interview is designed in this way to improve validity of diagnoses, this may come at the cost of assessor disagreement. Here, the traditional standard that validity is limited by reliability is less relevant, as the flexibility that is built in to the interview for enhancing the quality of information received may increase the chances that discrepant information may be collected. Additionally, during administration of the interview, patients are asked to think about their symptoms in a way that is different (i.e., more specific) from how many individuals typically think about their anxiety, and by the time they come in for the second interview, they may be better able to elaborate subjective reports of their difficulties, presenting a clearer picture for the second assessor. Furthermore, one may speculate that the time interval between the two assessments may lead to interviewer disagreement, but our findings, which support the stability of ratings, do not support this interpretation, as results did not change significantly when analyzing various subsamples of patients (i.e., those assessed twice within a 3-month period versus

difficulty concentrating, the second factor including fatigue, and the third factor including muscle tension; sleep disturbance and irritability did not load. Four of six symptoms loaded onto a two-factor solution, with the first factor including sleep and the second factor including restlessness, difficulty concentrating, and irritability; fatigue and muscle tension did not load. Restlessness, difficulty concentrating, and irritability loaded onto a one-factor solution. These analyses further support that conclusion that the six associated symptoms are not an item set and that it may not be appropriate to consider them as a group. Further research, potentially in an epidemiological sample, is warranted.

820

D. Gordon, R.G. Heimberg / Journal of Anxiety Disorders 25 (2011) 813–821

those assessed twice over a longer period of time). It is possible, however, that this did affect reliability in individual cases. The present study is limited by the difference in the scope of the interview between the two administrations. The second interviewer administered only the GAD and SAD modules of the ADIS interview, whereas the first interviewer administered the entire ADIS, thereby obtaining a fuller clinical picture of the patients’ concerns, which may inform more global ratings (i.e., CSR) and allow for a more fine grained understanding of the features of GAD. Specifically, the source of interference may be clearer at the first interview because the rater may have a better understanding of how the interference originated (i.e., from GAD vs. MDD) because they cover all disorder modules of the ADIS. Similarly, overlapping associated symptoms like concentration may be more properly rated as being a feature of one disorder than another (Joormann & Stöber, 1999). In other words, raters may differ in the weight they give to different features of GAD when additional modules are administered. Future studies examining GAD features may choose to conduct two full diagnostic interviews regardless of the outcome of the first interview in order to circumvent this problem. The lack of other principal disorder groups is another limitation of the present study, although the current sample is a good starting point for examination of GAD features, and future studies should include patients with both principal diagnoses of GAD and additional diagnoses of GAD in the context of a range of other disorders. Despite these limitations, the present study addresses the reliability of the diagnosis of GAD by investigating its features, as well as the GAD module of the ADIS, while also leaving room for proposals for further study. This information may be useful in refining the diagnostic process as well as the evolving GAD criteria. Moreover, as we approach DSM-V it will be necessary to examine the reliability and validity of the new criteria set. It may be necessary to further consider the type and number of associated symptoms included in the diagnostic criteria and as well as the decision to retain one or both of the excessiveness and difficult-to-control criteria. Current results suggest that it is unnecessary to retain both excessiveness and uncontrollability criteria and removing the uncontrollability criterion, as suggested by Andrews et al. (2010), is a move toward a less redundant definition of GAD. More research is necessary to support this suggestion, as redundancy and prevalence considerations are insufficient in finalizing such a decision. Regarding the associated symptoms, it seems that fatigue and concentration are key symptoms; however, further research regarding whether to reduce the number of symptoms is warranted.

References Abel, J. L. (1994, November). Alexithymia in an analogue sample of generalized anxiety disorder and non-anxious matched controls. Paper presented at the annual meeting of the Association for the Advancement of Behavior Therapy, San Diego, CA. Abel, J. L., & Borkovec, T. D. (1995). Generalizability of DSM-III-R generalized anxiety disorder to proposed DSM-IV criteria and cross-validation of proposed changes. Journal of Anxiety Disorders, 9, 303–315. doi:10.1016/0887-6185(95)00011-C American Psychiatric Association. (1980). Diagnostic and statistical manual of mental disorders (3rd ed.). Washington, DC: Author. American Psychiatric Association. (1987). Diagnostic and statistical manual of mental disorders (3rd rev.). Washington, DC: Author. American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author. Andrews, G., Hobbs, M. J., Borkovec, T. D., Beesdo, K., Craske, M. G., Heimberg, R. G., et al. (2010). Generalized worry disorder: a review of DSM-IV generalized anxiety disorder and options for DSM-V. Depression and Anxiety, 27, 134–147. doi:10.1002/da.20658 Barlow, D. H., & Wincze, J. (1998). DSM-IV and beyond: what is generalized anxiety disorder? Acta Psychiatrica Scandinavica, 98, 23–29. doi:10.1111/j.16000447.1998.tb05962.x Borkovec, T. D. (1994). The nature, functions, and origins of worry. In: G. Davey, & F. Tallis (Eds.), Worrying: perspectives on theory, assessment, and treatment (pp. 5–33). New York: Wiley.

Borkovec, T. D., & Costello, E. (1993). Efficacy of applied relaxation and cognitive behavioral therapy in the treatment of generalized anxiety disorder. Journal of Consulting and Clinical Psychology, 61, 611–619. Brawman-Mintzer, O., Lydiard, R. B., Crawford, M. M., Emmanuel, N., Payeur, R., Johnson, M., et al. (1994). Somatic symptoms in generalized anxiety disorder with and without comorbid psychiatric disorders. American Journal of Psychiatry, 151, 930–932. Brown, T. A., Antony, M. M., & Barlow, D. H. (1992). Psychometric properties of the Penn State Worry Questionnaire in a clinical anxiety disorder sample. Behaviour Research and Therapy, 30, 33–37. doi:10.1016/0005-7967(92)90093-V Brown, T. A., Di Nardo, P. A., & Barlow, D. H. (1994). Anxiety disorders interview schedule for DSM-IV. New York: Oxford University Press. Brown, T. A., Di Nardo, P. A., Lehman, C. L., & Campbell, L. A. (2001). Reliabilty of DSMIV anxiety and mood disorders: implications for the classification of emotional disorders. Journal of Abnormal Psychology, 110, 49–58. doi:10.1037//0021843X.110.1.49 Brown, T. A., Marten, P. A., & Barlow, D. H. (1995). Discriminant validity of the symptoms constituting the DSM-III-R and DSM-IV associated symptom criterion of generalized anxiety disorder. Journal of Anxiety Disorders, 9, 317–328. doi:10.1016/0887-6185(95)00012-D Connor, K. M., Davidson, J. R. T., Churchill, E., Sherwood, A., Foa, E. B., & Weisler, R. H. (2000). Psychometric properties of the Social Phobia Inventory (SPIN): a new self-rating scale. British Journal of Psychiatry, 176, 379–386. Connor, K. M., Kobak, K. A., Churchill, E., Katzelnick, D., & Davidson, J. R. T. (2001). Mini-SPIN: a brief screening assessment for generalized social anxiety disorder. Depression and Anxiety, 14, 137–140. doi:10.1002/da.1055 Davey, G. C. L. (1993). A comparison of three worry questionnaires. Behaviour Research and Therapy, 31, 51–56. doi:10.1016/0005-7967(93)90042-S Di Nardo, P. A., & Barlow, D. H. (1988). Anxiety disorders interview schedule-revised (ADIS-R). Albany, NY: Graywind. Di Nardo, P. A., Brown, T. A., & Barlow, D. H. (1994). Anxiety disorders interview schedule for DSM-IV: lifetime version (ADIS-IV-L). New York: Oxford University Press. Di Nardo, P. A., Moras, K., Barlow, D. H., Rapee, R. M., & Brown, T. A. (1993). Reliability of DSM-III-R anxiety disorder categories: using the Anxiety Disorders Interview Schedule-Revised (ADIS-R). Archives of General Psychiatry, 50, 251– 256. Di Nardo, P. A., O’Brien, G. T., Barlow, D. H., Waddell, M. T., & Blanchard, E. B. (1983). Reliability of DSM-III anxiety disorder categories using a new structured interview. Archives of General Psychiatry, 40, 1070–1074. Eng, W., & Heimberg, R. G. (2006). Interpersonal correlates of generalized anxiety disorder: self versus other perception. Journal of Anxiety Disorders, 20, 380–387. doi:10.1016/j.janxdis.2005.02.005 Erickson, T. M., & Newman, M. G. (2007). Interpersonal and emotional processes in generalized anxiety disorder analogues during social interaction tasks. Behavior Therapy, 38, 364–377. doi:10.1016/j.beth.2006.10.005 Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613–619. doi:10.1177/001316447303300309 Fresco, D. M., Mennin, D. S., Heimberg, R. G., & Turk, C. L. (2003). Using the Penn State Worry Questionnaire to identify individuals with generalized anxiety disorder: a receiver operating characteristic analysis. Journal of Behavior Therapy and Experimental Psychiatry, 34, 283–291. doi:10.1016/j.jbtep.2003.09.001 Hambrick, J. P., Turk, C. L., Heimberg, R. G., Schneier, F. R., & Liebowitz, M. R. (2004). Psychometric properties of disability measures among patients with social anxiety disorder. Journal of Anxiety Disorders, 18, 825–839. doi:10.1016/j.janxdis.2003.10.004 Heimberg, R. G., & Holloway, R. M. (2007). Examination of the known-groups validity of the Liebowitz Social Anxiety Scale. Depression and Anxiety, 24, 447–454. doi:10.1002/da.20277 Joormann, J., & Stöber, J. (1999). Somatic symptoms of generalized anxiety disorder from the DSM-IV: associations with pathological worry and depression symptoms in a nonclinical sample. Journal of Anxiety Disorders, 13, 491–503. doi:10.1016/S0887-6185(99) 00017-1 Kubarych, T. S., Aggen, S. H., Hettema, J. M., Kendler, K. S., & Neale, M. C. (2005). Endorsement frequencies and factor structure of DSM-III-R and DSM-IV generalized anxiety disorders symptoms in women: implications for future research, classification, clinical practice and comorbidity. International Journal of Methods in Psychiatric Research, 14, 69–81. doi:10.1002/mpr.18 Leon, A. C., Olfson, M., Portera, L., Farber, L., & Sheehan, D. V. (1997). Assessing psychiatric impairment in primary care with the Sheehan Disability Scale. International Journal of Psychiatry in Medicine, 27, 93–105. Mannuzza, S., Fyer, A. J., Martin, L. Y., Gallops, M. S., Endicott, J., Gorman, J., et al. (1989). Reliability of anxiety assessment: I. Diagnostic agreement. Archives of General Psychiatry, 46, 1093–1101. Marten, P. A., Brown, T. A., Barlow, D. H., Borkovec, T. D., Shear, M. K., & Lydiard, B. (1993). Evaluation of the ratings comprising the associated symptom criterion of DSM-III-R generalized anxiety disorder. Journal of Nervous and Mental Disease, 181, 676–682. Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny and social interaction anxiety. Behaviour Research and Therapy, 36, 455–470. doi:10.1016/S0005-7967(97)10031-6 Mennin, D. S., Heimberg, R. G., Fresco, D. M., & Ritter, M. R. (2008). Is generalized anxiety an anxiety or mood disorder? Considering multiple factors as we ponder the fate of GAD. Depression and Anxiety, 25, 289–299. doi:10.1002/da.20493

D. Gordon, R.G. Heimberg / Journal of Anxiety Disorders 25 (2011) 813–821 Mennin, D. S., Heimberg, R. G., Turk, C. L., & Fresco, D. M. (2005). Preliminary evidence for an emotion dysregulation model of generalized anxiety disorder. Behaviour Research and Therapy, 43, 1281–1310. doi:10.1016/j.brat.2004. 08.008 Meyer, T. J., Miller, M. L., Metzger, R. L., & Borkovec, T. D. (1990). Development and validation of the Penn State Worry Questionnaire. Behaviour Research and Therapy, 28, 487–495. doi:10.1016/0005-7967(90)90135-6 Molina, S., & Borkovec, T. D. (1994). The Penn State Worry Questionnaire: psychometric properties and associated characteristics. In: G. C. L. Davey, & F. Tallis (Eds.), Worrying: perspectives on theory, assessment, and treatment (pp. 265–283). New York: Wiley. Newman, M. G., Zuellig, A. R., Kachin, K. E., Constantino, M. J., Przeworski, A., Erickson, T., et al. (2002). Preliminary reliability and validity of the Generalized Anxiety Disorder Questionnaire-IV: a revised self-report diagnostic measure of generalized anxiety disorder. Behavior Therapy, 33, 215–233. doi:10.1016/S0005-7894(02)80026-0 Olfson, M., Fireman, B., Weissman, M. M., Leon, A. C., Sheehan, D. V., Kathol, R. G., et al. (1997). Mental disorders and disability among patients in a primary care group practice. American Journal of Psychiatry, 154, 1734–1740. Rodebaugh, T. L., Holaway, R. M., & Heimberg, R. G. (2008). The factor structure and dimensional scoring of the Generalized Anxiety Disorder Questionnaire for DSM-IV. Assessment, 15, 343–350. doi:10.1177/1073191107312547 Rodebaugh, T. L., Woods, C. M., & Heimberg, R. G. (2007). The reverse of social anxiety is not always the opposite: the reverse-scored items of the Social Interaction Anxiety Scale do not belong. Behavior Therapy, 38, 192–206. doi:10.1016/j.beth.2006.08.001 Ruscio, A. M., Chiu, W. T., Roy-Byrne, P., Stang, P. E., Stein, D. J., Wittchen, H., et al. (2007). Broadening the definition of generalized anxiety disorder: effects on prevalence and associations with other disorders in the National Comorbidity Survey Replication. Journal of Anxiety Disorders, 21, 662–676. doi:10.1016/j.janxdis.2006. 10.004

821

Ruscio, A. M., Lane, M., Roy-Byrne, P., Stang, P. E., Stein, D. J., Wittchen, H., et al. (2005). Should excessive worry be required for a diagnosis of generalized anxiety disorder? Results from the US National Comorbidity Survey Replication. Psychological Medicine, 35, 1761–1772. doi:10.1017/S0033291705005908 Schneier, F. R., Heckelman, L. R., Garfinkel, R., Campeas, R., Fallon, B. A., Gitow, A., et al. (1994). Functional impairment in social phobia. Journal of Clinical Psychiatry, 55, 322–331. Seeley-Wait, E., Abbott, M. J., & Rapee, R. M. (2009). Psychometric properties of the Mini-Social Phobia Inventory. Primary Care Companion to the Journal of Clinical Psychiatry, 11, 231–236. doi:10.4088/PCC.07m00576 Sheehan, D. V. (1983). The anxiety disease. New York: Scribner. Stöber, J. (1998). Reliability and validity of two widely used worry questionnaires: self-report and self-peer convergence. Personality and Individual Differences, 24, 887–890. doi:10.1016/S0191-8869(97)00232-8 Turvey, C. L., Stevens, D. E., & Merikangas, K. R. (1999). The validity of the associated symptom criteria for DSM-IV generalized anxiety disorder. International Journal of Methods in Psychiatric Research, 8, 129–137. doi:10.1002/mpr.63 Weeks, J. W., Spokas, M. E., & Heimberg, R. G. (2007). Psychometric evaluation of the Mini-Social Phobia Inventory (Mini-SPIN) in a treatment-seeking sample. Depression and Anxiety, 24, 382–391. doi:10.1002/da.20250 Williams, J. B. W., Gibbon, M., First, M. B., Spitzer, R. L., Davies, M., Borus, J., et al. (1992). The Structured Clinical Interview for DSM-III-R (SCID): II. Multisite test–retest reliability. Archives of General Psychiatry, 49, 630– 636. Wittchen, H., Fuetsch, M., Sonntag, H., Muller, N., & Leibowitz, M. (1999). Disability and quality of life in pure and comorbid social phobia: findings from a controlled study. European Psychiatry, 14, 118–131. doi:10.1016/S0924-9338(99) 80729-9 Yamas, K., Hazlett-Stevens, H., & Borkovec, M. (1997, November). Alexithymia in generalized anxiety disorder. Paper presented at the Annual Meeting of the Association for the Advancement of Behavior Therapy, Miami, FL.