Reliability of self-reported Antisocial Personality Disorder Symptoms among substance abusers

Drug and Alcohol Dependence 49 (1998) 189 – 199

Reliability of self-reported Antisocial Personality Disorder Symptoms among substance abusers Linda B. Cottler a,*, Wilson M. Compton a, T. Andrew Ridenour a, Arbi Ben Abdallah a, Tim Gallagher b a

Washington Uni6ersity School of Medicine, Department of Psychiatry, Parc Frontenac, 40 N Kingshighway, Suite 4, St. Louis, MO 63108, USA b Kent State Uni6ersity, Department of Sociology, P.O. Box 5190, Kent, OH 44242 -001, USA Received 22 May 1997; accepted 15 December 1997

Abstract It is estimated that from 20 to 60% of substance abusers meet criteria for Antisocial Personality Disorder (APD). An accurate and reliable diagnosis is important because persons meeting criteria for APD, by the nature of their disorder, are less likely to change behaviors and more likely to relapse to both substance abuse and high risk behaviors. To understand more about the reliability of the disorder and symptoms of APD, the Diagnostic Interview Schedule Version III-R (DIS) was administered to 453 substance abusers ascertained from treatment programs and from the general population (St Louis Epidemiological Catchment Area (ECA) follow-up study). Estimates of the 1 week, test – retest reliability for the childhood conduct disorder criterion, the adult antisocial behavior criterion, and APD diagnosis fell in the good agreement range, as measured by k. The internal consistency of these DIS symptoms was adequate to acceptable. Individual DIS criteria designed to measure childhood conduct disorder ranged from fair to good for most items; reliability was slightly higher for the adult antisocial behavior symptom items. Finally, self-reported ‘liars’ were no more unreliable in their reports of their behaviors than ‘non-liars’. © 1998 Elsevier Science Ireland Ltd. All rights reserved. Keywords: Antisocial personality disorder; Substance abusers; Diagnosis

1. Introduction The reliability of psychiatric diagnosis has been identified as a central concern for psychiatry (Spitzer et al., 1967; Spitzer and Fleiss, 1974; Perry, 1992). The reaction of researchers to low interrater and test –retest agreement of the psychiatric status of patients has been to systematize both the criteria employed in estimating the level of agreement over time and between raters; and to standardize the assessment of psychiatric status. This has been accomplished primarily through the use of standardized nosology and decision criteria. The Diagnostic Interview Schedule (DIS) (Robins et al.,

* Corresponding author.

1989), a fully structured interview, is one of the most prominent examples of efforts to improve nosology, while the widespread adoption of Cohen’s k statistic (Cohen, 1960), has made uniform the estimation of agreement of two measurements. The outcome of these efforts to reduce various forms of error in measurement and estimation has been a general increase in the reliability of psychiatric diagnosis. Whereas early studies on the reliability of diagnosis provided less than optimal results (Beck, 1962; Zubin, 1967; Spitzer and Fleiss, 1974), reliability of diagnoses was improved quite soon after the adoption of standardized assessment protocols and k (Helzer et al., 1977; Spitzer et al., 1978, 1979; Helzer et al., 1981). With the development and adoption of DSM-III’s multiaxial system (American Psychiatric Association,

0376-8716/98/$19.00 © 1998 Elsevier Science Ireland Ltd. All rights reserved. PII S 0 3 7 6 - 8 7 1 6 ( 9 8 ) 0 0 0 1 3 - 1

190

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

1980), interest has grown in the specification and identification of the Axis II Personality Disorders and in the trait-state problem (Loranger et al., 1991). The reliability of personality disorders has, as a group, been lower than that for Axis I disorders (Spitzer et al., 1979; Mellsop et al., 1982). Since the early 1980s, however, the growing interest in the personality disorders has generated much research in the development of personality disorder-specific structured and semi-structured interviews (Stangl et al., 1985; Loranger et al., 1987; Zanarini et al., 1987). These efforts have attained some success in increasing reliability of diagnoses with results from studies showing reliability estimates ranging between 0.30 and 0.96, with an average across studies of 0.67 (Pfohl et al., 1986; Loranger et al., 1987; Zanarini et al., 1987). Assessing the reliability of Antisocial Personality Disorder (APD) may pose some unique problems because, by definition, the antisocial personality is characterized by a tendency to deceive, lie and in general act in opportunistic ways without regard for the feelings or desires of others (American Psychiatric Association, 1987, 1994). The diagnosis of APD in the DSM-III-R and DSM-IV systems is also unique in that the symptom criteria focuses exclusively, with one exception, on behavioral patterns. Other personality disorders in the DSM system rely mostly on underlying personality dynamics (American Psychiatric Association, 1987, 1994). These features of APD are especially salient for research on substance abuse because it is well established that substance abusers have a high comorbidity with APD (range of 20 – 60%; Rounsaville et al., 1982; Hesselbrock et al., 1985; Khantzian and Treece, 1985; Woody et al., 1985; Compton et al., 1995; Myers et al., 1995; Kessler et al., 1996; Newman et al., 1996; Brooner et al., 1997) and it is rare for individuals with APD not to exhibit comorbid substance abuse (Regier et al., 1990; Windle, 1990). In fact, recent research provides compelling evidence for at least some degree of overlap in the genetic liability to APD and alcohol abuse (McGue et al., 1992; Lynskey et al., 1994; Pickens et al., 1995; Clark et al., 1997). As well, comorbidity of this type is problematic for treatment of substance abusers with regard to outcome and relapse (Woody et al., 1985; Rounsaville et al., 1987; Helzer and Pryzbeck, 1988; Myers et al., 1995). In this paper, we assess the test – retest reliability of symptom criteria for Conduct Disorder (CD) and APD among drug abusers, and include a unique comparison of the stability of self-reported behavior between persons retrospectively reporting a high degree of lying in adulthood versus those not reporting a high degree of lying. Angold and Costello (1995) examined test –retest reliability of self-reported CD as a function of self-reported lying among adolescents and found that the reliability of CD symptoms was poorer for those who

admitted to lying during the first interview. No paper has presented APD results with this stratification. The results of the present study will allow us to estimate the test–retest stability of DIS-III-R CD and APD criteria as reported by adults. Also, to assess if the test–retest reliability of these measures varies as a function of an individuals’ admitted propensity to lie. Moreover, inclusion of substance abusing respondents at high risk for APD from both treatment and general population samples increases the generalizability of our findings.

2. Methods

2.1. Sample Data for these analyses were obtained from an ongoing NIDA-funded study of the reliability of substance use disorder diagnoses. The sample of 453 respondents were enrolled from several sources: 199 (45%) subjects were randomly drawn from the St. Louis Epidemiological Catchment Area (ECA) sample and reinterviewed from 1992 to 1994 (Eaton and Kessler, 1985; Robins and Regier, 1992; Demallie et al., 1995). The selected ECA participants had reported using marijuana, cocaine, opiates, pills or hallucinogens at least five times in their lifetime during the first wave of ECA data collection in 1981. The remaining sample was obtained from a residential drug treatment program (24%) and outpatient drug treatment program (32%). The mortality rate for the ECA sample was 2.6%; the refusal rate was 16.2% and 11% of the original sample had moved beyond this study’s geographic boundaries. The intensive ascertainment and refusal conversion techniques used by our team in other studies were not used in the present study, in order not to upset ECA respondents for future follow up studies (Cottler et al., 1996). Males (50.3%), females (49.7%), African Americans (54.3%) and whites (45.7%) were equally represented in the population. Moreover, all levels of educational attainment were represented, with 53% reporting high school only and 43% reporting college experience or a college degree.

2.2. Inter6iews Written informed consent was obtained from all subjects. In addition to an assessment of substance use disorders, all subjects were administered the antisocial personality criteria section of the Diagnostic Interview Schedule, Version III-R (DIS-III-R; Robins et al., 1989). Subjects were informed that the retest interviewer, a different person than the test interviewer, would be unaware of the data collected at Time 1. Respondent histories were protected by a NIDA-issued Certificate of Confidentiality. Extensively trained and

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

191

Table 1 Reliability of DSM-III-R conduct disorder criteria, antisocial personality criteria and antisocial personality disorder diagnosis Criterion

Time 1

Time 2 Met

% Agreement

k

95% CI

Not met

Conduct disorder Criterion

Met Not met

86 26

60 281

81

0.537

0.452 – 0.622

Adult behavior Criterion

Met Not met

163 30

58 202

81

0.610

0.538 – 0.683

APD

Met Not met

61 3

53 319

84

0.527

0.433 – 0.620

n=453.

experienced interviewers conducted the interviews faceto-face and in a private setting. Interviewers were instructed to keep all respondents’ answers confidential. A project supervisor monitored the interviewing, reviewed all protocols and discussed all errors with each interviewer on a weekly basis. The interval between the first and second interview was 10 days on average; the maximum was 14 days.

2.3. APD symptoms Two sets of age-specific criteria must be met to fulfill DSM-III-R criteria for antisocial personality (American Psychiatric Association, 1987). At least three of the twelve ‘childhood’ or Conduct Disorder symptoms (often truant, ran away from home, initiated physical fights, used a weapon in fights, forced another person into sexual activity, physically cruel to animals, physically cruel to other people, destroyed other people’s property, set fires, often lied, stole without confronting the victim, stole while confronting the victim) must be present before the age of 15 years. Of the ten adult symptoms listed, four must be exhibited since the age of 15 years. These Adult Antisocial Behavior criteria include nine behavioral symptoms (inconsistent work behavior, failure to conform to social norms, aggressive behavior, failure to honor financial obligations, impulsive behavior, disregard for the truth, reckless behavior, being unable to function as an adult and failure to maintain a monogamous relationship) and one non-behavioral symptom (lacks remorse in regard to the other criteria).

2.4. Analysis The focus of our analysis was to assess the reliability of the symptom criteria for Conduct Disorder (CD) and APD for the entire sample together, and then separately for subgroups of self- reported ‘liars’ versus ‘non-liars.’ Comparing the reliability estimates of symptom criteria for the entire group, will inform us on the

contribution of specific symptoms to the overall reliability of the given disorder, and this information may be useful for decision making regarding the inclusion/ exclusion of specific symptoms in the nosology of disorders. Comparing the reliability estimates of symptom criteria by self-reported lying in adulthood will reveal whether admitting to, or denial of a chief characteristic of APD (deception) has an effect on the stability of selfreported symptoms. The 1-week test–retest reliability for symptoms, criteria and APD diagnosis were examined using the k statistic, which tests the proportion of agreement obtained for nominal data after correcting for chance alone. A k value of zero indicates that chance alone can account for the agreement observed between raters (Cohen, 1960). Excellent agreement is indicated by k values of ] 0.75; from 0.65 to 0.74 indicates good agreement; 0.40–0.64 indicates fair agreement; and kB 0.4 indicate poor agreement. The 95% confidence limits are also reported and when differences between k were tested for significance, the Z statistic was used at the PB 0.05 level. Each set of criteria (childhood and adult criteria) was also examined separately for its internal consistency as symptoms of the same APD criteria. Then the internal consistency of two sets of criteria was examined as one scale for the APD diagnosis. In as much as the DIS is a test of psychiatric disorders, its diagnosisspecific psychometric properties indicate how well the DIS measures psychiatric illness and its internal consistency can be used as another indicator of the reliability of these items among these subjects. Coefficient a, which is a special case of the intraclass correlation and estimates between-item reliability (Hays, 1988; Winer et al., 1991), was used to assess the internal consistency of each set of criteria as a whole. Coefficient a can be used for dichotomously coded criteria and provides an estimate of the proportion of variability in a set of criteria that taps the construct that it is purported to measure (Crocker and Algina, 1986). In other words, coefficient a is a converse estimate to error

192

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

Table 2 Reliability of DSM-III-R adult antisocial behavior criteria Criterion

Time 1

Time 2 Met

% Agreement

k

95% CI

Not met

Bootstrapped k differencesa Mean k difference

Z

Often truant

Met Not met

69 41

35 308

83

0.535

0.443 – 0.627

−0.01

0.21

Ran away from home

Met Not met

33 13

23 384

92

0.603

0.485 – 0.720

0.06

0.92

Started fights

Met Not met

28 18

32 375

89

0.467

0.342 – 0.592

−0.07

1.16

Used a weapon in fights

Met Not met

19 24

21 389

90

0.403

0.261 – 0.545

−0.14

1.89

Forced someone to have sex

Met Not met

2 1

2 448

99

0.568

0.128 – 1.01

0.00

0.01

Physically cruel to animals

Met Not met

40 10

18 385

94

0.706

0.603 – 0.808

0.16

2.34*

Physically cruel to other people

Met Not met

49 29

54 321

82

0.430

0.328 – 0.531

−0.11

1.98*

Destroyed other’s property

Met Not met

29 13

30 381

91

0.523

0.398 – 0.647

−0.08

0.24

Intentionally set fires

Met Not met

21 7

25 400

93

0.532

0.390 – 0.673

−0.02

0.292

Often lied

Met Not met

85 37

67 264

77

0.449

0.371 – 0.546

−0.08

1.19

Stolen without confrontation

Met Not met

120 34

75 224

76

0.496

0.416 – 0.577

−0.04

0.74

Stolen with confrontation

Met Not met

7 7

9 430

96

0.448

0.223 – 0.674

−0.09

0.83

n=453. a Several item k values were larger than the symptom count k value. To test the significance of these differences, bootstrap comparisons between item k values and the symptom count k were conducted for each item. * PB0.05.

in measurement. The relation between each criterion and the rest of the criteria set to which it belonged was examined using two statistics: the corrected item-total correlation and the coefficient a if the item was deleted from the scale. Although all analyses were conducted separately for the general population and the treatment sample, the results were so similar that the sample was pooled to make the description of the subgroup analyses (liars and non-liars) easier to follow.

3. Results

3.1. General results From the information reported at Time 1, 32% of the sample met criteria for Conduct Disorder, 49% met criteria for Adult Antisocial Behavior and 25% met criteria for APD. Three reliability estimates based on

two measurement occasions are presented in Table 1 using contingency tables. These three estimates are for: (1) Three or more Childhood CD criteria (‘met’ criteria) versus 0–2 criteria (criteria ‘not met’); (2) four or more Adult Antisocial Behavior criteria (‘met’ criteria) versus 0–3 criteria (criteria ‘not met’); and (3) Antisocial Personality Disorder (‘met’ criteria versus criteria ‘not met’). Percent agreement, chance corrected agreement (k), and confidence intervals for k are presented. As shown, agreement for all three k coefficients are significantly different from zero. The adult criteria were found to have fair agreement, and to be more reliable than the childhood criteria (0.610 vs. 0.537). While this difference is not significant beyond chance, it raises the possibility that the reporting of more temporally distant childhood behaviors is less reliable than reporting more recent adult behaviors. In Tables 2 and 3 the number of persons reporting each individual symptom criterion at Time 1 and Time 2 for CD and APD are presented. Each table also gives

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

193

Table 3 Reliability of DSM-III-R adult antisocial behavior criteria Criterion

Time 1

Time 2

Met

Not met

% Agreement k

95% CI

Bootstrapped k differencesa Mean k difference

Z

2.50*

Inconsistent work behavior

Met Not met

191 50

65 147

75

0.488

0.407 – 0.569

−0.11

Not socially conforming

Met Not met

274 33

37 109

85

0.644

0.568 – 0.720

0.04

0.91

Aggressive behaviour

Met Not met

208 43

60 142

77

0.536

0.458 – 0.614

−0.06

1.36

Financial obligations disregarded

Met Not met

91 22

64 276

81

0.549

0.467 – 0.631

−0.06

1.14

Impulsive behavior

Met Not met

79 21

42 311

86

0.624

0.540 – 0.708

0.01

0.29

No regard for the truth

Met Not met

109 28

55 261

82

0.589

0.510 – 0.667

−0.03

0.63

Reckless behavior

Met Not met

132 35

36 250

84

0.624

0.592 – 0.735

0.06

1.18

Dysfunctional as a parent

Met Not met

50 18

33 352

89

0.596

0.496 – 0.695

−0.01

0.23

Unable to sustain monogamous relationship

Met Not met

24 21

20 388

91

0.489

0.355 – 0.624

−0.13

1.58

Lacks remorse

Met Not met

178 29

56 130

80

0.626

0.555 – 0.697

−0.14

2.19*

n= 453. Several item k values were larger than the symptom count k value. To test the significance of these differences, bootstrap comparisons between item k values and the symptom count k were conducted for each item. * PB0.05. a

the percent agreement, k, and 95% CI for the k estimate. As shown in Table 2, using the parameters described earlier, the reliability of 11 of the 12 Conduct Disorder items was fair; good agreement in reporting was found for ‘being physically cruel to animals’. For the ten adult antisocial behavior criteria (shown in Table 3), the range of estimates was also in the fair range. Because some of the individual criterion items seemed to have better agreement than the composite criteria made up of these individual items (i.e. k in Tables 2 and 3 compared with k in Table 1), a bootstrap technique was used to test the differences between k for each criterion and the corresponding criteria. The bootstrap procedure produced a mean difference in k values as well as a Z-score for that difference. In only two instances was a CD item k found to be significantly higher than the age-cluster criteria (‘physically cruel to animals’ and ‘physically cruel to other people’). There were also two instances where APD criteria were found to be more reliable than the disorder: inconsistent work behavior and lacking remorse. The probability that any

of the 22 comparisons made in Tables 2 and 3 would be significant just due to chance is: 1− (0.9522) =0.68, which suggests that it is highly likely that these ‘statistically significant’ comparisons are simply due to a chance and that the diagnoses are as reliable as the criterion items. The supposition that these differences are due to chance is further supported by the P values of each—they barely met the 0.05 cutoff. To further understand the reasons for inconsistent reporting of conduct disorder symptoms and adult antisocial behaviors, we evaluated the 2×2 contingency tables with the McNemar’s test to understand if respondents were more likely to report symptoms at Time 1 or Time 2. Analyses indicate that both Conduct Disorder and Adult Antisocial Behaviors criteria were more likely to be met at Time 1 than at Time 2. Of the 12 CD symptoms, half were more likely to be reported at Time 1. ‘Drop off’ of reporting was found for truancy, running away from home overnight, use of a weapon, forcing sex, being physically cruel to animals, and stealing with confrontation. Similarly, five of the ten symptoms of adult antisocial behavior were more often

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

194

Table 4 Internal consistency of DSM-III-R conduct disorder criteria Item-total statistics

Often truant Ran away Starts fights Used weapons Forced sex Cruel to animals Cruel to people Destroyed property Set fires Often lied Stolen without confrontation Stolen with confrontation

Scale mean if item deleted

Scale variance if item deleted

Corrected item−total correlation

a if item deleted

1.7417 1.8477 1.8389 1.8830 1.9625 1.8433 1.7439 1.8411 1.8698 1.6358 1.5408

3.4973 3.7223 3.6311 3.8381 4.2220 3.7387 3.3679 3.7313 3.8834 3.5595 3.5276

0.4155 0.3916 0.4505 0.3685 0.2546 0.3697 0.5096 0.3718 0.2970 0.3060 0.2983

0.6882 0.6928 0.6845 0.6970 0.7150 0.6955 0.6719 0.6952 0.7048 0.7096 0.7130

1.9360

4.0556

0.3212

0.7060

n=453, Coefficient a=0.7163. Statistics for criteria sum: mean, 3.9713; variance, 4.3288; S.D., 2.0806.

reported at the first interview compared with the second. Those that were reported more at the first than second were: disregard for financial obligations, impulsive behaviors, having no regard for the truth and being dysfunctional as a parent.

3.2. Internal consistency Internal consistency statistics provided estimates of the between-item reliability for CD, Adult Antisocial Behaviors and APD. These results are presented in Tables 4–6. Coefficient a estimates were less than ideal. However, given the relatively few criteria used to assess childhood conduct disorder and adulthood antisocial behavior, compared to psychological tests with dozens of items, these estimates of internal consistency are deemed adequate for their intended use in studies consisting of large samples. The between-item reliability for CD could not be improved by deleting any specific symptom as shown in Table 4. The ‘cruel to people’ item appeared to be characteristic of the childhood criterion because of its 0.51 corrected item-total correlation with the criteria sum. Moreover, ‘starts fights’ showed the second highest item-total correlation. On the other hand, by deleting ‘reckless behavior’, the adulthood and APD criterion between-item reliabilities would improve considerably. The correlation between reckless behavior and the adulthood criteria sum equals zero. This pattern was also observed in the general population and the treatment samples above.

3.3. Liars 6ersus non-liars Finally, the sample was divided into two groups based on their Time 1 or Time 2 response to the DIS question: ‘‘Have you thought that you lied pretty often since you have been an adult’. Persons who responded ‘yes’ to that question at either interview were coded as ‘liars’ (n= 162); those who said ‘no’ were coded as ‘non-liars’ (n= 291). At both Time 1 and Time 2, 83 persons reported lying pretty often since an adult; 291 persons denied lying at both Time 1 and Time 2. Liars were 1.9 times as likely to meet criteria for APD1 at both Time 1 and Time 2 (95% CI = 1.46–2.51). Non-liars were also more likely to meet criteria for APD at the first interview (OR= 2.08; 95% CI=1.54–2.82). In Tables 7 and 8, we report reliability of items for the CD and APD criteria, respectively and a test of the significance of the agreement between liars and nonliars. Liars and non-liars were, in general, ‘equally’ inconsistent in reporting their childhood Conduct Disorder symptoms. Only two behaviors discriminated liars from non-liars—running away from home overnight and setting fires. In both cases, liars were less reliable than non-liars. Reports of ‘forcing sex’ were rare; thus, the variance of this behavior among respondents is too small for generating an accurate estimate of reliability. On reports of more recent behaviors of Antisocial Personality Disorder (APD), liars and non-liars were found to differ on two of the ten comparisons. As 1

In this case, the DIS question about lying, ‘‘Have you thought that you lied pretty often since you have been an adult?’’, was omitted from the algorithm for calculating APD.

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

195

Table 5 Internal consistency of DSM-III-R adult antisocial behavior criteria Item-total statistics

Inconsistent work behavior Not socially conforming Aggressive behavior Financial obligations disregarded Impulsive behavior No regard for the truth Reckless behavior Dysfunctional as a parent Unable to sustain monogamous relationship Lacks remorse

Scale mean if item detected

Scale variance if item detected

Corrected item−total correlation

a If item deleted

3.1545 3.0331 3.1280 3.3775 3.4525 3.3576 3.3488 3.5364 3.6225

4.3035 4.2179 4.1960 4.2090 4.4607 4.3364 5.1790 4.6873 5.0452

0.4135 0.5073 0.4778 0.4961 0.3979 0.4159 −0.0052 0.3364 0.2014

0.6616 0.6440 0.6486 0.6457 0.6653 0.6613 0.7372 0.6764 0.6944

3.4658

4.5945

0.3303

0.6771

n =453, Cronbach’s a= 0.6958. Statistics for criteria sum: mean, 3.7196; variance, 5.4013; S.D., 2.3241. Table 6 Internal consistency of DSM-III-R antisocial personality disorder behavior criteria Item-total statistics

Inconsistent work behavior Not socially conforming Aggressive behavior Financial obligations disregarded Impulsive behavior No regard for the truth Reckless behavior Dysfunctional as a parent Unable to sustain monogamous relationship Lacks remorse Often truant Ran away Starts fights Used weapons Forced sex Cruel to animals Cruel to people Destroyed property Set fires Often lied Stolen without confrontation Stolen with confrontation

Scale mean if item deleted

Scale variance if item deleted

Corrected item−total correlation

a If item deleted

5.1258 5.0044 5.0993 5.3488 5.4238 5.3289 5.3201 5.5077 5.5938

12.6678 12.3451 12.3684 12.5639 12.7447 12.6770 13.9084 13.3655 13.6975

0.3893 0.5296 0.4850 0.4454 0.4263 0.4030 0.0408 0.2732 0.2296

0.7798 0.7702 0.7730 0.7759 0.7773 0.7788 0.8030 0.7864 0.7880

5.4371 5.4614 5.5673 5.5585 5.6026 5.6821 5.5629 5.4636 5.5607 5.5894 5.3554 5.2605 5.6556

13.2112 12.7137 13.2947 13.1498 13.4657 14.1244 13.4501 12.7359 13.3708 13.6231 12.8845 13.0382 13.8900

0.2804 0.4657 0.3687 0.4162 0.3564 0.2217 0.2964 0.4599 0.3267 0.2575 0.3481 0.2808 0.2651

0.7865 0.7751 0.7817 0.7793 0.7829 0.7900 0.7851 0.7755 0.7836 0.7869 0.7825 0.7875 0.7876

n= 453, Cronbach’s a= 0.7906. Statistics for criteria sum: mean, 5.6909; variance, 14.2892; S.D., 3.7801.

shown in Table 8, non-liars were more reliable than liars on the inconsistent work behavior and not being socially conforming criteria. Also, the diagnosis of APD was less reliable among persons who ‘admit’ that they lie pretty often since they have been an adult compared to those who do not admit to lying (data not shown).

4. Discussion The diagnosis of APD may be one of the hardest to assess from self-report because the behaviors are often illegal, and because the essential feature of the disorder is a pervasive pattern of disregard for the rights of others, lying and deception. Thus, although we can

196

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

Table 7 Reliability of DSM-III-R conduct disorder criteria by self-reported lyinga Liars (N =162)

Non-liars (N= 291)

k Differences

k 95% CI

k 95% CI

t

P

DSM-III-R Conduct disorder

0.431

0.295 – 0.568

0.589

0.476 – 0.701

3.044

0.081

Criteria Often truant Ran away Starts fights Used weapons Forced sex Cruel to animals Cruel to people Destroyed property Set fires Often lied Stolen without confrontation Stolen with confrontation

0.528 0.424 0.495 0.398 — 0.628 0.357 0.549 0.306 0.391 0.406 0.279

0.390 – 0.667 0.227 – 0.622 0.324 – 0.666 0.193 – 0.603 — 0.457 – 0.799 0.198 – 0.515 0.362 – 0.735 0.052 – 0.560 0.250 – 0.532 0.268 – 0.543 0.056 – 0.613

0.516 0.732 0.402 0.397 — 0.762 0.478 0.503 0.636 0.456 0.546 0.576

0.388 – 0.643 0.598 – 0.866 0.215 – 0.590 0.197 – 0.596 — 0.638 – 0.886 0.346 – 0.610 0.337 – 0.668 0.477 – 0.796 0.335 – 0.576 0.447 – 0.645 0.292 – 0.860

0.017 6.388 0.517 0.000 — 1.541 1.322 0.313 4.64 0.464 2.628 1.761

0.90 0.011 0.472 0.990 — 0.214 0.250 0.718 0.031 0.496 0.105 0.185

a

k were calculated separately for respondents who reported lying often after the age of 15 years.

Table 8 Reliability of DSM-III-R adult antisocial behavior criteria by self-reported lyinga Liars (N =162)

Non-liars (N= 291)

k Differences

k 95% CI

k 95% CI

t

P

Adult antisocial behavior

0.424

0.268 – 0.580

0.629

0.536 – 0.722

4.873

0.027

Criteria Inconsistent work behavior Not socially conforming Aggressive behavior Financial obligations disregarded Impulsive behavior No regard for the truth Reckless behavior Dysfunctional as a parent Unable to sustain monogamous relationship Lacks remorse

0.256 0.422 0.482 0.533 0.557 0.717 0.646 0.548 0.493 0.497

0.101 – 0.411 0.221 – 0.622 0.337 – 0.626 0.408 – 0.659 0.423 – 0.692 0.566 – 0.862 0.525 – 0.767 0.405 – 0.691 0.302 – 0.685 0.353 – 0.661

0.573 0.661 0.540 0.506 0.660 0.607 0.674 0.614 0.473 0.414

0.479 – 0.667 0.574 – 0.749 0.443 – 0.636 0387 – 0.636 0.551 – 0.770 0.431 – 0.783 0.585 – 0.762 0.467 – 0.761 0.281 – 0.665 0.286 – 0.542

11.785 4.626 0.430 0.093 1.351 0.864 0.128 0.395 0.023 0.699

0.001 0.031 0.512 0.761 0.245 0.353 0.721 0.529 0.881 0.403

a

k Were calculated separately for respondents who reported lying often after the age of 15 years.

evaluate the reliability or consistency of self-reports, the validity of the report is lacking. With this in mind, the present study has evaluated the ability of respondents to tell us at two periods, on average 10 days apart, about these behaviors. ADP was just one of several types of behaviors asked about, so the focus was not primarily on APD. The reliability of items designed to assess DSM system criteria were found to be significantly higher than would be expected by chance; however, the fair reliability found in general is not as high as we would like it. Adult Antisocial Behaviors were reported more consistently than the childhood CD behaviors. This non-significant finding could possibly be due to the format of the questions or could indicate that more temporally distant symptoms are recalled less reliably.

When item level reliability was evaluated, we found that 11 of the 12 CD and eight of the ten adult items were in the fair range. Cruelty to animals before age 15, not socially conforming and reckless behavior after age 15 were more reliable than any others. This is an interesting finding for those conducting studies on high risk behaviors, since the ‘not socially conforming’ item included risky sexual behaviors. Internal consistency estimates indicated that the two sets of criteria adequately measured the constructs they were purported to measure (childhood conduct disorder and adult antisocial behavior). These findings support not deleting any CD item, since the a for each item did not increase when a particular item was removed from the ‘scale’. However, there was one exception for adulthood criteria, which was for ‘reckless behavior’, where

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

the a jumped to 0.7372 when that item was removed. Further discussion of internal consistency regarding as it relates to APD assessment is offered below. Other aspects of our analyses are encouraging for the assessment of APD among drug users. According to our results, persons reporting that they have lied often in adulthood were at least as reliable as ‘non-liars’ in the reporting of DIS derived conduct disorder and antisocial personality disorder symptoms. Unfortunately, while we found that the self-reported tendency to deceive was not related to lower reliability of self-reported symptoms, we were not able to determine whether there are differences in the veracity of self-reports. In essence, we did not assess whether the validity of self-reports was different between ‘liars’ and ‘nonliars’. The complexity involved in discriminating individuals who respond honestly from those who distort their symptomology is evidenced by the growing literature investigating malingering (Pope et al., 1993; Nies and Sweet, 1994; Zielinski, 1994; Ridenour et al., 1996). Although this type of research has been conducted for decades, no single method consistently identifies response distorters well enough to be used (Nies and Sweet, 1994). To address the issue of improving the self-report of APD, several changes should be considered. To minimize poor reliability that is attributed to the respondent him/herself, perhaps investigators should introduce the assessment with the type of ‘commitment probe’ used in the National Comorbidity Survey. In this study, Dr Ron Kessler included a question at the beginning of the interview that asks respondents if they are willing to think hard about the questions and give the interviewer the truth. If they cannot commit to this, they are not included in the sample. Secondly, the assessment itself should be reviewed with a ‘fine tooth comb’. Two of the authors of this paper (LBC and WMC) are co-developers of the new DIS Version 4.0 (for DSM-IV). In this new version, the number of DIS questions has increased and the old questions from the DIS-III-R and III versions have been modified and updated in their language. This will hopefully improve the internal consistency as well as reliability of those items. For example, reckless behavior was scored in the DSM-III-R version from two DIS questions regarding only driving behavior. The DIS-IV ‘reckless behavior’ criterion now includes risky sexual behavior, leaving youngsters at home alone, risk-taking regarding firearms and driving behavior. It is hoped that these improvements in operationalization of criteria will lead to improvements in reliability; no data are yet available on this new instrument. Since drop off of symptoms was common at Time 2, additional improvements for a test – retest study of criteria might include expanding the Time 2 interview to an ‘informed’ format. In this format, the Time 1 data

197

would be available and loaded on computer so that after the APD questions were asked, the Time 1 interview responses would be checked for differences and if any were noted, they could be ‘resolved’ immediately. A paper and pencil version of this is used in studies now, with the Discrepancy Interview Protocol (DIP) (Cottler et al., 1994) where the Time 2 interviewer resolves disagreements between the two interviews at the end of the interview and without knowing ahead what was said at Time 1. Numerous studies of substance abusers have been conducted using the DIS (Griffin et al., 1987; Kokker and Stefanis, 1995; Yates et al., 1996) and other standardized instruments for assessing psychiatric disorder. The results of the present study indicate that the data regarding APD in substance abusers are fairly reliable. In fact, the data are as reliable as some medical conditions, such as knowledge of breast removal, heart conditions and other psychiatric diagnoses (Helzer, 1983; Semler et al., 1987; Wittchen et al., 1989). With creative solutions (including commitment probes) and outside validators (such as collateral informants and police records) we should be able to adequately increase the consistency of reports of these behaviors.

Acknowledgements The authors wish to thank Darren Nix and Douglas Mager for their technical support in this manuscript. Also, the following grants are acknowledged: the NIDA-funded data collection grant DA05585 (Dr Linda Cottler, P.I.), MH17104 (Dr Cottler, P.I., Ty Andrew Ridenour, Post-Doctoral Fellow), and DA00209 (Dr Wilson Compton, M.D., P.I.).

References American Psychiatric Association 1980. Diagnostic and Statistical Manual of Mental Disorders (III). Washington, DC. American Psychiatric Association 1987. Diagnostic and Statistical Manual of Mental Disorders (III-R). Washington, DC. American Psychiatric Association 1994. Diagnostic and Statistical Manual of Mental Disorders (IV). Washington, DC. Angold, A., Costello, E.J., 1995. A test – retest reliability study of child-reported psychiatric symptoms and diagnoses using the child and adolescent psychiatric assessment (CAPA-C). Psychol. Med. 25, 755 – 762. Beck, A, 1962. Reliability of psychiatric diagnoses: I. A critique of systematic studies. Am. J. Psychiatry 119, 210 – 216. Brooner, RK, King, VL, Kidorff, M, Schmidt, CW, Bigelow, G., 1997. Psychiatric and substance use comorbidity among treatment seeking opioid abusers. Arch. Gen. Psychiatry 45, 71 – 80. Clark, D.B., Moss, H.B., Kirisci, L., Mezzich, A.C., Miles, R., Ott, P., 1997. Psychopathology in preadolescent sons of fathers with substance use disorders. J. Am. Acad. Child Adolesc. Psychiatry 36, 495 – 502.

198

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199

Cohen, J., 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46. Compton, W.M., Cottler, L.B., Shillington, A.M., Price, R.K., 1995. Is antisocial personality disorder associated with increased HIV risk behaviors in cocaine users? Drug Alcohol Depend. 33, 37 – 43. Cottler, L.B., Compton, W.M., Brown, L., Shell, A., Keating, S., Shillington, A., Hummel, R., 1994. The Discrepancy Interview Protocol: A method for evaluating and interpreting discordant survey responses. Int J Methods Psychiatr. Res. 4, 173–182. Cottler, L.B., Compton, W.M., Ben-Abdallah, A., Horne, M., Claverie, D., 1996. Achieving a 96.6% follow-up rate in a longitudinal study of drug abusers. Drug Alcohol Depend. 41, 209 – 217. Crocker, L., Algina, J., 1986. Introduction to Classical and Modern Test Theory. Rinehart and Winston, Orlando, FL. Demallie, D., Cottler, L.B., Compton, W.M., 1995. Alcohol abuse and dependence: Consistency in reporting of symptoms over ten years. Addiction 90, 615–625. Eaton, W.W., Kessler, L.G., 1985. Epidemiologic Field Methods in Psychiatry: The NIMH Epidemiologic Catchment Area Program. Academic Press Inc., Orlando, FL. Griffin, M.L., Weiss, R.D., Mirin, S.M., Wilson, H., BouchardVoelk, B., 1987. The use of the Diagnostic Interview Schedule in drug-dependent patients. Am. J. Drug Alcohol Abuse 13, 281 – 291. Hays, W.L., 1988. Statistics. Holt, Rinehart and Winston, New York. Helzer, J.E., 1983. Standardized interviews in psychiatry. Psychiatr. Dev. 1, 161 – 178. Helzer, J.E., Pryzbeck, T.R., 1988. The co-occurrence of alcoholism with other psychiatric disorders in the general population and its impact on treatment. J. Stud. Alcohol 49, 219–224. Helzer, J.E., Clayton, P.J., Pambakian, R., Reich, T., Woodruff, R.A., Reveley, M.A., 1977. Reliability of psychiatric diagnosis. Arch. Gen. Psychiatry 34, 136–141. Helzer, J.E., Robins, L.N., Croughan, J.L., Welner, A., 1981. Renard diagnostic interview. Arch. Gen. Psychiatry 38, 393–398. Hesselbrock, M.N., Meyer, R.E., Keener, J.J., 1985. Psychopathology in hospitalized alcoholics. Arch. Gen. Psychiatry 42, 1050 – 1055. Kessler, R.C., Nelson, C.B., McGonagle, K.A., Edlund, M.J., Frank, R.G., Leaf, P.J., 1996. The epidemiology of co-occurring addictive and metal disorders: Implications for prevention and service utilization. Am. J. Orthopsychiatr. 66, 17–31. Khantzian, E.J., Treece, C., 1985. DSM-III psychiatric diagnosis of narcotic addicts: Recent findings. Arch. Gen. Psychiatry 42, 1067 – 1071. Kokker, A., Stefanis, C., 1995. Drug abuse and psychiatric comorbidity. Compr. Psychiatry 36, 329–337. Loranger, A.W., Susman, V.L., Oldham, J.M., Russakoff, L.M., 1987. The personality disorder examination: A preliminary report. J. Pers. Disord. 1, 1 –13. Loranger, A.W., Lenzenweger, M.F., Gartner, A.F., Susman, V.L., Herzig, J., Zammit, G.K., Gartner, J.D., Abrams, R.C., Young, R.C., 1991. Trait-state artefacts and the diagnosis of personality disorders. Arch. Gen. Psychiatry 48, 720–728. Lynskey, M.T., Fergusson, D.M., Horwood, L.J., 1994. The effect of parental alcohol problems on rates of adolescent psychiatric disorders. Addiction 89, 1277–1286. McGue, M., Pickens, R.W., Svikis, D.S., 1992. Sex and age effects on the inheritance of alcohol problems: A twin study. J. Abnorm. Psychol. 101, 3 – 17. Mellsop, G., Varghese, F., Joshua, S, Hicks, A., 1982. The reliability of Axis II of DSM-II. Am. J. Psychiatry 139, 1360–1361. Myers, M.G., Brown, S.A., Mott, M.A., 1995. Preadolescent conduct disorder behaviors predict relapse and progression of addition for adolescent alcohol and drug abusers. Alcohol. Clin. Exp. Res. 19, 1528 – 1536.

Newman, D.L., Moffitt, T.E., Caspi, A., Magdol, L., Silva, P.A., Stanton, W.R., 1996. Psychiatric disorder in a birth cohort of young adults: Prevalence, comorbidity, clinical significance, and new case incidence from ages 11 to 21. J. Consult. Clin. Psychol. 64, 552 – 562. Nies, K.J., Sweet, J.J., 1994. Neuropsychological assessment and malingering: A critical review of past and present strategies. Arch. Clin. Neuropsychol. 7, 501 – 552. Perry, J.C., 1992. Problems and considerations in the valid assessment of personality disorders. Am. J. Psychiatry 149, 1645 – 1653. Pfohl, B., Coryell, W., Zimmerman, M., Stangl, D., 1986. DSM-III personality disorders: Diagnostic overlap and internal consistency of individual DSM-III Criteria. Compr. Psychiatry 27, 21–34. Pickens, R.W., Svikis, D.S., McGue, M., LaBuda, M.C., 1995. Common genetic mechanisms in alcohol, drug, and metal disorder comorbidity. Drug Alcohol Depend. 39, 129 – 138. Pope, H.S., Butcher, J.N., Seelen, J., 1993. The MMPI, MMPI-2 & MMPI-A in court. American Psychological Association, Washington, D.C. Regier, D.A., Farmer, M.E., Rae, D.S., Locke, B.Z., Keith, S.J., Judd, L.L., Goodwin, F.K., 1990. Comorbidity of mental disorders with alcohol and other drug abuse. J. Am. Med. Assoc. 264, 2511 – 2518. Ridenour, T.A., McCoy, K.D., Dean, R.S., 1996. An exploratory stepwise discriminant function analysis of malingered and nondistorted responses to the neuropsychological symptom inventory. Int. J. Neurosci. 87, 91 – 95. Robins, L.N., Regier, D.A., 1991. Psychiatric Disorders in America: The Epidemiologic Catchment Area Study. The Free Press, New York, NY. Robins, L.N., Helzer, J., Cottler, L.B., Goldring, E., 1989. The Diagnostic Interview Schedule. Version III-R St. Louis. Rounsaville, B.J., Weissman, M.M., Kleber, H.D., Wilber, C.H., 1982. Heterogeneity of psychiatric diagnosis in treated opiate addicts. Arch. Gen. Psychiatry 39, 161 – 166. Rounsaville, B.J., Dolinsky, Z.S., Babor, T.F., Meyer, R.E., 1987. Psychopathology as a predictor of treatment outcome in alcoholics. Arch. Gen. Psychiatry 44, 505 – 513. Semler, G., Wittchen, H.U., Joschke, K., Zaudig, M., von Geiso, T., Kaiser, S., von Cranach, M., Pfister, H., 1987. Test-retest reliability of a standardized psychiatric interview (DIS/CIDI). Eur. Arch. Psychiatry Neurol. Sci. 236, 214 – 222. Spitzer, R.L., Fleiss, J.L., 1974. A re-analysis of the reliability of psychiatric diagnosis. Br. J. Psychiatry 125, 341 – 347. Spitzer, R.L., Cohen, J., Fleiss, J.L., Endicott, J., 1967. Quantification of agreement in psychiatric diagnosis: A new approach. Arch. Gen. Psychiatry 17, 83 – 87. Spitzer, R.L., Endicott, J., Robins, E., 1978. Research diagnostic criteria. Arch. Gen. Psychiatry 35, 773 – 782. Spitzer, R.L., Forman, J.B., Nee, J., 1979. DSM-III Field Trials: I. Initial interrater diagnostic reliability. Am. J. Psychiatry 136, 815 – 817. Stangl, D., Pfohl, B., Zimmerman, M., Bowers, W., Corenthal, R., 1985. A structured interview for the DSM-III personality disorders. A preliminary report. Arch. Gen. Psychiatry 42, 591–596. Windle, M., 1990. A longitudinal study of antisocial behaviors in early adolescence as predictors of late adolescent substance use: Gender and ethic group differences. J. Abnorm. Psychol. 99, 86 – 91. Winer, B.J., Brown, D.R., Michels, K.M., 1991. Statistical Principles in Experimental Design. McGraw-Hill, New York. Wittchen, H.U., Burke, J.D., Semler, G., Pfister, H., von Cranach, M., Zaudig, M., 1989. Recall and dating of psychiatric symptoms. Arch. Gen. Psychiatry 46, 437 – 443. Woody, G.E., McLellan, A., Luborsky, L., O’Brien, C.P., 1985. Sociopathy and psychotherapy outcome. Arch. Gen. Psychiatry 42, 1081 – 1086.

L.B. Cottler et al. / Drug and Alcohol Dependence 49 (1998) 189–199 Yates, W.R., Cadoret, R.J., Troughton, E., Stewart, M.A., 1996. An adoption study of DSM-IIIR alcohol and drug dependence severity. Drug Alcohol Depend. 41, 9–15. Zanarini, M.C., Frankenburg, F.R., Chauncey, D.L., Gunderson, J.G., 1987. The diagnostic interview for personality disorders: Interrater and test – retest reliability. Compr. Psychiatry 28, 467–480.

.

199

Zielinski, J.J., 1994. Malingering and defensiveness in the neuropsychological assessment of mild traumatic brain injury. Clin. Psychol. Sci. Pract. 1, 169 – 184. Zubin, J., 1967. Classification of the behavior disorders. In: Farnsworth, P.R., McNemar, O. (Eds.), Annual Review of Psychology. Annual Reviews, Palo Alto, CA, pp. 373 – 406.