- Email: [email protected]
Validity of self-reported data, scientific methods and drug policy
Drug and Alcohol Dependence 51 (1998) 265 – 266
Commentary
Validity of self-reported data, scientific methods and drug policy Don C. Des Jarlais * Beth Israel Medical Center, 16th Street and 1st A6enue, New York, NY 1003, USA Accepted 20 January 1998
Darke’s review of self-reported drug use, HIV risk behaviors and criminal activity among identified drug users shows consistent evidence for the reliability and concurrent validity of these measures. This is reassuring, as in the absence of such consistency, research on drug use would be immeasurably more difficult. Research in some fields, such as HIV risk behavior among drug users would be almost impossible if the self-report data were not reliable and valid. There is also a new method for data collection that offers possible improvement in the validity of self-report data on stigmatized behaviors. In audio computerassisted structured interviewing (Audio-CASI) subjects read the interview’s questions on a computer screen at the same time as the questions are read to the subject through headphones. The subject answers by typing in the appropriate responses on the computer keyboard (usually a single number or letter). This provides much greater privacy than face-to-face interviews and avoids the possible literacy problems with self-administered questionnaires. (A researcher is also present to assist with possible problems in using the computer.) The early findings from Audio-CASI suggest that the extra privacy does lead to higher rates of self-reporting for highly stigmatized behavior (Turner et al., 1992, 1998). Audio-CASI may be particularly helpful in obtaining data from persons who have not already been identified as engaging in the stigmatized behavior (note that the studies in Darke’s review were of persons who were identified as drug users prior to the self-report data collection.). Darke’s review of self-reported HIV risk behavior underestimates the evidence for the concurrent validity of this particular type of behavior. By my estimate, * Tel.: +1 212 4202000.
there are now literally thousands of studies that show relationships between self-reported risk behavior and HIV sero-status in the theoretically expected direction. Several hundred such studies are often presented at each of the big AIDS conferences. Indeed, failure to find relationships is typically seen as evidence that the researchers—not the subjects—are doing something wrong, e.g. not asking the right questions or not wording the questions properly. It should also be noted that there are many different types of HIV risk behavior, different types of ‘multi-person use of injection equipment’ and different types of sexual behavior, and that reliability and validity may not be the same for self-reports on these different types of risk behavior. Research on HIV risk behavior among drug users has gone beyond examining ‘concurrent’ validity of self-reported risk behavior to examine the ‘construct’ validity of AIDS-related behavior change. Reporting on AIDS-related behavior change involves estimating risk behavior at two or more points in time (possible change), as well as assessing whether the change should be attributed to concerns about AIDS versus other possible causes of behavior change. This AIDS-related behavior change or AIDS risk reduction should be considered as a theoretical construct (for which construct validity is applicable) versus the ‘simple’ overt HIV-risk behaviors (for which concurrent validity is applicable). The evidence for the construct validity of AIDS-related behavior change among injecting drug users suggest a high degree of construct validity. In a study of HIV seroconversion among injecting drug users in Bangkok, we found an odds ratio of 0.25 for seroconversion among those who reported AIDS-related behavior change (specifically stopping sharing of needles and syringes) compared to those who did not report this AIDS-related behavior change (Des Jarlais
0376-8716/98/$19.00 © 1998 Elsevier Science Ireland Ltd. All rights reserved. PII S0376-8716(98)00053-2
266
D.C. Des Jarlais / Drug and Alcohol Dependence 51 (1998) 265–266
et al., 1994). In a larger study of HIV seroprevalence among 4419 injecting drugs users in 12 cities on four continents, we found an odds ratio of 0.50 for being HIV seropositive among subjects who reported any AIDS-related behavior change compared to subjects who reported no AIDS-related behavior change (Des Jarlais et al. 1996). These studies indicate that drug injectors are capable of accurately assessing changes in AIDS-related behavior, validly reporting on the changes, and that the changes are of sufficient magnitude to substantially effect the likelihood of becoming infected with HIV (Chitwood, 1994; Osborn, 1996). The available evidence suggests very good reliability for self-reported behavior among persons who are known to be using psychoactive drugs (with the important qualifications of good question construction, lack of punishment for reporting the stigmatized behaviors, etc.). Given this research, then why do so many community and political leaders seem to believe that all drug users are, in Darke’s phrase, ‘inveterate liars’1? The ‘truthfulness’ of drug users self-reports has important policy implications. If drug use makes a person into an inveterate liar, then this is a very negative consequence of drug use and would serve to justify extreme actions to reduce the use of the drugs. Additionally, if the drug users are inveterate liars, one should ignore whatever the drug users have to say about how ‘anti-drug’ policies should be implemented. If, on the other hand, drug users can and will tell the ‘truth,’ then the reliable and valid statements of current
(not only former) drug users should be an important source of information for designing drug treatment programs, AIDS prevention programs and societal policies on psychoactive drug use. The question of the reliability and validity of self-reported behavior among drug users has implications beyond conducting research. It directly relates to how drug policies should be formulated and to the content of those policies. Continued belief that all drug users are ‘inveterate liars’ may reflect resistance to policy change rather than the state of the scientific evidence.
References Chitwood, D., 1994. Annotation: HIV risk and injection drug users — evidence for behavioral change. Am. J. Pub. Health 84, 350. Des Jarlais, D.C., Choopanya, K., et al., 1994. AIDS risk reduction and reduced HIV seroconversion among injection drug users in Bangkok. Am. J. Pub. Health 84 (3), 452 – 455. Des Jarlais, D.C., Friedmann, P., et al., 1996. The protective effect of AIDS-related behavioral change among injection drug users: a cross-national study. Am. J. Pub. Health 86 (12), 1780 –1785. Osborn, J., 1996. Editorial: drug use and behavior change. Am. J. Pub. Health 86 (12), 1697 – 1698. Turner, C.F., Ku, L., et al., 1998. Impact of audio-CASI on bias in reporting of male – male sexual contacts. In: Warnicke, R.M. (Ed.), NCHS Proceedings of the 6th Conference on Health Survey Methods. National Center on Health Statistics, Hyattsville, MD. Turner, C.F., Lessler, J., et al., 1992. Effects of mode of administration and wording on reporting of drug use. In: Turner, C.F., Lessler, J.T., Gfroerer, J.D. (Eds.), Survey Measurement of Drug Use: Methodological Issues. Government Printing Office, Washington, DC, DHHS Publication No. AMD 92-1929.
1
As a researcher, I must note that there is a lack of ‘hard data’ on what political leaders believe about drug users. There is also the possibility that ‘official’ statements about drug users do not reflect the private beliefs of the persons making the official statements. The statement that many political leaders act as if drug users were ‘inveterate liars’ is based on many years of personal experience with political leaders in many different countries, often in discussions of AIDS prevention and AIDS policies. This includes 4 years on the US National Commission on AIDS, and meetings with political leaders in a large number of countries. From discussions with other researchers involved in policy formulation, I trust that my experiences of many political leaders acting as if drug users were inveterate liars are typical.
.
Commentary
Validity of self-reported data, scientific methods and drug policy Don C. Des Jarlais * Beth Israel Medical Center, 16th Street and 1st A6enue, New York, NY 1003, USA Accepted 20 January 1998
Darke’s review of self-reported drug use, HIV risk behaviors and criminal activity among identified drug users shows consistent evidence for the reliability and concurrent validity of these measures. This is reassuring, as in the absence of such consistency, research on drug use would be immeasurably more difficult. Research in some fields, such as HIV risk behavior among drug users would be almost impossible if the self-report data were not reliable and valid. There is also a new method for data collection that offers possible improvement in the validity of self-report data on stigmatized behaviors. In audio computerassisted structured interviewing (Audio-CASI) subjects read the interview’s questions on a computer screen at the same time as the questions are read to the subject through headphones. The subject answers by typing in the appropriate responses on the computer keyboard (usually a single number or letter). This provides much greater privacy than face-to-face interviews and avoids the possible literacy problems with self-administered questionnaires. (A researcher is also present to assist with possible problems in using the computer.) The early findings from Audio-CASI suggest that the extra privacy does lead to higher rates of self-reporting for highly stigmatized behavior (Turner et al., 1992, 1998). Audio-CASI may be particularly helpful in obtaining data from persons who have not already been identified as engaging in the stigmatized behavior (note that the studies in Darke’s review were of persons who were identified as drug users prior to the self-report data collection.). Darke’s review of self-reported HIV risk behavior underestimates the evidence for the concurrent validity of this particular type of behavior. By my estimate, * Tel.: +1 212 4202000.
there are now literally thousands of studies that show relationships between self-reported risk behavior and HIV sero-status in the theoretically expected direction. Several hundred such studies are often presented at each of the big AIDS conferences. Indeed, failure to find relationships is typically seen as evidence that the researchers—not the subjects—are doing something wrong, e.g. not asking the right questions or not wording the questions properly. It should also be noted that there are many different types of HIV risk behavior, different types of ‘multi-person use of injection equipment’ and different types of sexual behavior, and that reliability and validity may not be the same for self-reports on these different types of risk behavior. Research on HIV risk behavior among drug users has gone beyond examining ‘concurrent’ validity of self-reported risk behavior to examine the ‘construct’ validity of AIDS-related behavior change. Reporting on AIDS-related behavior change involves estimating risk behavior at two or more points in time (possible change), as well as assessing whether the change should be attributed to concerns about AIDS versus other possible causes of behavior change. This AIDS-related behavior change or AIDS risk reduction should be considered as a theoretical construct (for which construct validity is applicable) versus the ‘simple’ overt HIV-risk behaviors (for which concurrent validity is applicable). The evidence for the construct validity of AIDS-related behavior change among injecting drug users suggest a high degree of construct validity. In a study of HIV seroconversion among injecting drug users in Bangkok, we found an odds ratio of 0.25 for seroconversion among those who reported AIDS-related behavior change (specifically stopping sharing of needles and syringes) compared to those who did not report this AIDS-related behavior change (Des Jarlais
0376-8716/98/$19.00 © 1998 Elsevier Science Ireland Ltd. All rights reserved. PII S0376-8716(98)00053-2
266
D.C. Des Jarlais / Drug and Alcohol Dependence 51 (1998) 265–266
et al., 1994). In a larger study of HIV seroprevalence among 4419 injecting drugs users in 12 cities on four continents, we found an odds ratio of 0.50 for being HIV seropositive among subjects who reported any AIDS-related behavior change compared to subjects who reported no AIDS-related behavior change (Des Jarlais et al. 1996). These studies indicate that drug injectors are capable of accurately assessing changes in AIDS-related behavior, validly reporting on the changes, and that the changes are of sufficient magnitude to substantially effect the likelihood of becoming infected with HIV (Chitwood, 1994; Osborn, 1996). The available evidence suggests very good reliability for self-reported behavior among persons who are known to be using psychoactive drugs (with the important qualifications of good question construction, lack of punishment for reporting the stigmatized behaviors, etc.). Given this research, then why do so many community and political leaders seem to believe that all drug users are, in Darke’s phrase, ‘inveterate liars’1? The ‘truthfulness’ of drug users self-reports has important policy implications. If drug use makes a person into an inveterate liar, then this is a very negative consequence of drug use and would serve to justify extreme actions to reduce the use of the drugs. Additionally, if the drug users are inveterate liars, one should ignore whatever the drug users have to say about how ‘anti-drug’ policies should be implemented. If, on the other hand, drug users can and will tell the ‘truth,’ then the reliable and valid statements of current
(not only former) drug users should be an important source of information for designing drug treatment programs, AIDS prevention programs and societal policies on psychoactive drug use. The question of the reliability and validity of self-reported behavior among drug users has implications beyond conducting research. It directly relates to how drug policies should be formulated and to the content of those policies. Continued belief that all drug users are ‘inveterate liars’ may reflect resistance to policy change rather than the state of the scientific evidence.
References Chitwood, D., 1994. Annotation: HIV risk and injection drug users — evidence for behavioral change. Am. J. Pub. Health 84, 350. Des Jarlais, D.C., Choopanya, K., et al., 1994. AIDS risk reduction and reduced HIV seroconversion among injection drug users in Bangkok. Am. J. Pub. Health 84 (3), 452 – 455. Des Jarlais, D.C., Friedmann, P., et al., 1996. The protective effect of AIDS-related behavioral change among injection drug users: a cross-national study. Am. J. Pub. Health 86 (12), 1780 –1785. Osborn, J., 1996. Editorial: drug use and behavior change. Am. J. Pub. Health 86 (12), 1697 – 1698. Turner, C.F., Ku, L., et al., 1998. Impact of audio-CASI on bias in reporting of male – male sexual contacts. In: Warnicke, R.M. (Ed.), NCHS Proceedings of the 6th Conference on Health Survey Methods. National Center on Health Statistics, Hyattsville, MD. Turner, C.F., Lessler, J., et al., 1992. Effects of mode of administration and wording on reporting of drug use. In: Turner, C.F., Lessler, J.T., Gfroerer, J.D. (Eds.), Survey Measurement of Drug Use: Methodological Issues. Government Printing Office, Washington, DC, DHHS Publication No. AMD 92-1929.
1
As a researcher, I must note that there is a lack of ‘hard data’ on what political leaders believe about drug users. There is also the possibility that ‘official’ statements about drug users do not reflect the private beliefs of the persons making the official statements. The statement that many political leaders act as if drug users were ‘inveterate liars’ is based on many years of personal experience with political leaders in many different countries, often in discussions of AIDS prevention and AIDS policies. This includes 4 years on the US National Commission on AIDS, and meetings with political leaders in a large number of countries. From discussions with other researchers involved in policy formulation, I trust that my experiences of many political leaders acting as if drug users were inveterate liars are typical.
.