- Email: [email protected]
Reliability of the problem oriented screening instrument for teenagers (POSIT) in adolescent medical practice1
JOURNAL OF ADOLESCENT HEALTH 2001;29:125–130
ORIGINAL ARTICLE
Reliability of the Problem Oriented Screening Instrument for Teenagers (POSIT) in Adolescent Medical Practice JOHN R. KNIGHT, M.D., ELIZABETH GOODMAN, M.D., TODD PULERWITZ, M.D., AND ROBERT H. DuRANT, PhD
Purpose: To determine the internal consistency and 1-week test–retest reliability of the Problem Oriented Screening Instrument for Teenagers (POSIT) among adolescent medical patients. Methods: A research assistant administered the POSIT to a consecutive sample of 15- to 18-year-old patients arriving for routine medical care at a hospital-based adolescent medical practice. Each subject returned for a retest 1 week later. Internal consistency for each scale on test and retest was calculated using Cronbach alpha, and 1-week test–retest reliability by the intraclass correlation coefficient (r) and the kappa coefficient (). Results: The Substance Use/Abuse, Mental Health Status, Educational Status, and Aggressive Behavior/Delinquency scales had favorable alpha scores (> .70). Others, including Physical Health Status, had lower alpha scores. High intraclass correlation coefficients were found for all 10 POSIT scales (r ⴝ .72 to .88), although (r) was lower for males on two of the scales. Kappa coefficients for all scales indicated good reproducibility beyond chance ( ⴝ .42 to .73). Conclusions: This study provided supportive evidence for the reliability of the POSIT in primary care medical settings, although some POSIT scales could likely be improved. The 20- to 30-min administration time is most
From the Department of Pediatrics, Harvard Medical School, Boston, Massachusetts (J.R.K., E.G., T.P., R.H.D.), the Division of General Pediatrics, Children’s Hospital, Boston, Massachusetts (J.R.K.); and the Division of Adolescsent/Young Adult Medicine, Children’s Hospital, Boston, Massachusetts (E.G., R.H.D.). Address correspondence to: Elizabeth Goodman, M.D., John R. Knight, M.D., Children’s Hospital, 300 Longwood Avenue, Boston, MA 02115. Funded in part by the Maternal and Child Health Bureau, Project MCJ-MA259195, HRSA, DHHS. Manuscript accepted July 30, 2000.
practical in settings that are dedicated to adolescent medicine, and computerized administration and scoring are needed. © Society for Adolescent Medicine, 2001 KEY WORDS: POSIT Reliability Substance abuse Screening High-risk behavior Adolescents
Substance abuse among adolescents is a major national problem. Almost 80% of high school students in the United States have begun to drink alcohol, 50% are current drinkers, and 33% are binge-drinkers [1]. Nearly 17% of students drive a car after drinking, and more than 36% ride in cars with an intoxicated driver [1]. Motor vehicle crashes are the leading cause of death among adolescents, and approximately 42% of these fatalities are related to the use of alcohol [2]. Use of illicit drugs among our nation’s youth also continues to be a problem. By the time they reach 12th grade, 54% of high school students have used a drug other than alcohol, and 29% have used an illicit drug other than marijuana [3]. The American Medical Association’s Guidelines for Adolescent Preventive Services (GAPS) recommend yearly screening of adolescents being seen for routine medical care for substance abuse and other biomedical, behavioral, and emotional conditions [4]. Clinicians need an efficient and reliable means for accomplishing this screening, and paper-and-pencil
© Society for Adolescent Medicine, 2001 Published by Elsevier Science Inc., 655 Avenue of the Americas, New York, NY 10010
1054-139X/01/$–see front matter PII S1054-139X(00)00206-8
KNIGHT ET AL.
126
JOURNAL OF ADOLESCENT HEALTH Vol. 29, No. 2
Table 1. Means, Standard Deviations, and Internal Consistency (␣) of POSIT Scales Test
Retest
POSIT Scale
No. of Items
n
Mean
SD
␣
n
Mean
SD
␣
Substance Use and Abuse Physical Health Status Mental Health Status Family Relations Peer Relations Educational Status Vocational Status Social Skills Leisure/Recreation Aggressive Behavior/Delinquency
17 10 22 11 10 23 18 11 12 16
145 153 143 146 152 136 148 147 151 149
1.0 2.8 6.5 3.4 3.2 6.8 4.9 3.6 4.5 5.0
1.8 1.7 3.8 2.2 2.2 3.6 2.5 1.9 2.0 3.3
.77 .45 .74 .63 .67 .71 .55 .46 .40 .79
82 87 78 83 85 79 81 82 86 78
0.8 2.4 5.8 3.6 3.2 6.3 4.8 3.3 4.7 4.4
1.9 1.9 4.6 2.2 2.5 4.1 2.5 2.1 2.1 3.6
.87 .51 .85 .60 .74 .79 .62 .58 .45 .85
SD ⫽ standard deviation.
questionnaires present one alternative to face-to-face interviews for busy physicians. Adolescents may prefer to answer personal questions on paper or a computer rather than during an interview with an adult. Numerous screening instruments are available for substance abuse, as well as other health risk and problem behaviors in adolescents [5]. However, relatively little research has been conducted to determine validity and reliability of these instruments in general adolescent populations or in health care settings that provide routine medical care. The Problem Oriented Screening Instrument for Teenagers (POSIT) is one such instrument that has received considerable attention in recent years. Designed for adolescents aged 12 through 19 years, POSIT is a self-report multi-problem screening instrument composed of 139 “yes/no” questions [6]. Its 10 scales are designed to screen for potential problems in the following functional domains: (a) Substance Use and Abuse, (b) Physical Health Status, (c) Mental Health Status, (d) Family Relations, (e) Peer Relations, (f) Educational Status, (g) Vocational Status, (h) Social Skills, (i) Leisure and Recreation, and (j) Aggressive Behavior and Delinquency. The number of items in each POSIT scale is shown in Table 1. Individual questions in the POSIT are of three types: generalpurpose items, age-related items, and red-flag items. Any positive score, whether a red-flag item or a scale total score, indicates that a potential problem exists and that further assessment in that area is recommended. Red-flag items are intended to identify problem areas in need of further assessment when they alone are scored as positive (e.g., “Do you get into trouble because you use alcohol or drugs at school?”). The psychometric properties of the POSIT have
been previously measured in a number of adolescent populations. In general, these previous studies have measured internal consistency, test–retest reliability, and validity of the 10 scales. Internal consistency indicates that items within a scale are measuring the same construct and that a higher total score is likely to indicate higher total risk. This is a particularly important measure for clinicians who may base treatment or referral decisions for their patients on scores that exceed a recommended cut-point. Test– retest reliability measures the temporal stability of a scale, and high reliability indicates that the measurement error of the test is relatively small over brief intervals of time during which behavior itself is unlikely to have changed. Concurrent validity indicates that the scale is indeed measuring the construct that it purports to measure (e.g., substance abuse vs. some other health-risk behavior) and is assessed by comparing the scale score in question with a criterion standard. Melchior et al. [7] studied the validity and internal reliability of POSIT scales in three groups of adolescents classified as low risk (high school students), moderate risk (in outpatient drug counseling clinics), or high risk (in drug inpatient treatment and juvenile detention). All 10 POSIT scales demonstrated good concurrent differential validity with measures of similar constructs (i.e., all scales successfully differentiated youth who were heavy substance users from those who were nonusers). Internal consistency, however, varied among scales. Alpha scores ranged from .47 to .93, with Substance Use/Abuse (.93), Mental Health Status (.86), Aggressive Behavior/ Delinquency (.85), and Educational Status (.82) having the highest alphas, and with Leisure/Recreation (.47), Vocational Status (.47), Social Skills (.51), and
August 2001
Physical Health Status (.56) having lower alpha scores. McLaney et al. [8] studied the psychometric properties of the POSIT in a drug abuse clinic population. They found lower internal consistency coefficients on most of the scales than those reported by Melchior et al. but found good evidence of scale validity. The correlation between the POSIT’s Substance Use/ Abuse scale and the 10 substance use scales of the Personal Experience Inventory (PEI) [9] ranged from .51 to .69, with a mean of .60. Similar correlations were found between other POSIT scales and their counterparts on the PEI. Dembo et al. [10] studied test–retest reliability of the POSIT over varying time intervals in a population of adolescent juvenile offenders. The highest test–retest reliability coefficients were found for the 0- to 2-week time interval. These ranged from .50 for the Vocational Status scale to .73 for the Peer Relations scale. The reliability coefficients decreased as the time interval between the first and second testing increased, a finding more likely indicative of changing life circumstances than actual scale psychometric properties. Prior studies have thus found that the POSIT has promising psychometric properties in certain settings. What is unknown is the reliability of the POSIT in other more general settings. The purpose of this study was to determine the internal consistency and 1-week test–retest reliability of the POSIT in an adolescent general medical clinic population. The results of this study will assist clinicians in determining the utility of the POSIT as a screening instrument for their adolescent medical patients.
Methods Subjects Subjects were drawn from 15- to 18-year-old patients receiving care in the Children’s Hospital, Adolescent/Young Adult Medical Practice, Boston, a primary care and referral clinic serving inner city and suburban youth from the full range of the social strata. This clinic has more than 4000 patients and more than 10,000 visits annually for well care and acute and nonacute medical problems. Twenty-four percent of patients have no health insurance, 33% have Medicaid, and 41% have private insurance. Sixty-nine percent of the study sample (n ⫽ 193) was female, 42% were African-American, 20% Latino, 16% white, and 1% Asian (21% other or no response), reflecting the demographic distribution of the clinic’s population.
RELIABILITY OF THE POSIT
127
Procedures A research assistant who was not involved in providing medical care invited a convenience sample of patients requesting routine care to participate in the study in June through August 1996. The purpose of the study was explained to each patient to be assessment of the consistency, reliability, and the overall “quality of the test,” and confidentiality was assured. Patients who agreed to participate gave informed consent. Based on the Guidelines for Adolescent Health Research, the Children’s Hospital Committee on Clinical Investigations (IRB-equivalent) waived the requirement for parental consent [11]. However, all subjects were encouraged to discuss their participation in the study with their parents or another responsible adult and were offered confidential referral if they wished to discuss further any issues that were raised by their participation in the study. We excluded from the study three patients with acute anorexia nervosa, one who came to be examined for alleged sexual assault, and three others who came to discuss results of a positive pregnancy test, because their providers judged they might be unduly stressed. Data were not kept on the number of patients who were invited to participate but refused. However, the research associate estimated that the percentage of refusers was very low after the first 2 weeks of the study; most cited lack of time as the reason. Potential subjects were offered a gift certificate to a local merchant (for a hamburger, slice of pizza, etc.) for participating. They were given these coupons after completion of the retest. The POSIT was administered in a private area at the end of the medical visit. No names were placed on the questionnaire. Subjects were assigned unique numerical identifiers to link test with retest while maintaining confidentiality. After the completion of the POSIT, subjects were given the opportunity to ask questions and were given an appointment for the retest 1 week later. They were asked for permission to have the research assistant telephone them the day prior to this appointment as a reminder. When subjects arrived for the retest, they were given the same instructions, and the same procedure was followed in administering the POSIT. If the retest appointment was missed, the research assistant telephoned and tried to reschedule the retest within 8 days of the initial test. Ninety-three of 173 subjects (53.8%) completed the retest. Data were entered and analyzed in SPSS statistical software [12]. Each of the 10 POSIT scales was
128
KNIGHT ET AL.
JOURNAL OF ADOLESCENT HEALTH Vol. 29, No. 2
Table 2. Test–Retest Reliabilities (r), Scale Cut-Points, Scale Positivity Rates, and Kappa Coefficients () of POSIT Scales POSIT Scale
Test–Retest r
Suggested CutPoint
% subjects ⱖ Cut-Point
Substance Use and Abuse Physical Health Status Mental Health Status Family Relations Peer Relations Educational Status Vocational Status Social Skills Leisure/Recreation Aggressive Behavior/Delinquency
.77 .78 .88 .78 .83 .82 .77 .73 .72 .83
1 3 4 4 1 6 5 3 5 6
37.6 50 74.7 39.9 95.3 60 56 60.6 47.3 31.7
.46 .68 .68 .73 .42 .69 .60 .52 .60 .65
computed for subjects who answered at least 75% of the questions on the scale. Subjects who answered less than 75% of the items in a scale were excluded from the analysis for that scale only. Means and standard deviations were computed for each scale. Internal consistency was measured using Cronbach alpha [13] and test–retest reliability using the single measure intraclass correlation coefficient (r). Kappa coefficients were computed for all items and for the positive/negative status of each scale according to suggested cut-points [6].
Results The means of each POSIT scale at test and retest are shown in Table 1. Retest completers and noncompleters did not differ significantly on any POSIT scale mean score (p ⫽ .08 to .96) or demographic variable (p ⫽ .20 to .76) other than gender (completers ⫽ 76.3% female, noncompleters ⫽ 62.5%, p ⫽ .049). The percentage of subjects who screened positive in each scale at initial test according to suggested scale cut-points [6] is shown in Table 2. Internal consistency of the scales as measured by Cronbach alpha varied at both test (␣ ⫽ .40 to .79) and retest (␣ ⫽ .45 to .87) (Table 1). Alpha coefficients of .70 and above are generally considered favorable. The Physical Health, Vocational Status, Social Skill, and Leisure/Recreation scales had alpha coefficients that fell below this level at the initial test. It should be noted, however, that alpha is partly a function of scale length and these scales consist of relatively fewer items [14]. Scale alphas did not increase substantially with deletion of any items, and scale alphas did not differ substantially (⌬␣ ⱖ .10) by gender with the exception of the Physical Health scale (␣ ⫽ .37 for males vs. .47 for females). With the exception of the Family Relations scale, all alphas increased substan-
tially at retest. The Substance Use/Abuse, Mental Health, and Aggressive Behavior scales had the highest scale alphas at both test and retest. High intraclass correlation coefficients were found for all 10 POSIT scales (Table 2). The lowest intraclass correlation was found for the Leisure/Recreation scale (r ⫽ .72) and the highest for the Mental Health scale (r ⫽ .88). Intraclass correlation coefficients for males and females did not differ substantially (⌬r ⱖ .10) for most scales. However, (r) was lower for males than females on the Family Relations scale (.58 vs. .83) and the Leisure/Recreation (.56 vs. .83) scale. We computed kappa coefficients on all items and found that 103/139 (74.1%) items demonstrated good reproducibility beyond chance ( ⫽ .40 to .75) and 15/139 (10.8%) additional items demonstrated excellent reproducibility beyond chance ( ⬎ .75) [15]. The kappa coefficient is a more important measure, however, for the positive/negative status of each scale, and for those items that are designated as “red-flags” because each is designed to indicate a need for further assessment when it alone is scored as positive. Kappa coefficients indicated good reproducibility beyond chance for positive/negative status in all 10 scales ( ⫽ .42 to .73) (Table 2). Of 44 red-flag items, 30 (68.2%) had kappa coefficients that indicated good reproducibility beyond chance, and five (11.4%) additional items had kappa coefficients that indicated excellent reproducibility beyond chance.
Discussion The POSIT was developed with a variety of aims: (a) to estimate the service needs in city/state political jurisdictions for multiple problem adolescents and their families, (b) to refer youths within the juvenile justice system to appropriate treatment services, and
August 2001
(c) to assist in treatment matching for troubled teenagers within clinical practice settings [6]. Our study addressed this third aim, examining the utility of the POSIT for purposes of risk identification in a health care setting. The findings of this study provide good evidence for the reliability of certain POSIT scales among adolescent medical patients, and these scales (i.e., Substance Use/Abuse, Mental Health, and Aggressive Behavior/Delinquency) are of great importance to medical care providers. Clinicians can be assured that the POSIT is a generally reliable tool for screening adolescent medical patients. For the scales that have high internal consistency, scores can also provide an indication of problem severity and assist the primary care provider to decide what management strategy is most appropriate (e.g., brief office intervention vs. referral to a specialist). This study has limitations. Data were not collected on those adolescent patients who refused participation, and it is unknown to what extent this group may have differed from the study sample as to risk severity. This study tested reliability of the POSIT, not validity, sensitivity, or specificity of individual scales. However, other studies have provided good evidence for the validity of the Substance Use/ Abuse, Mental Health Status, and Aggressive Behavior/Delinquency scales, and we believe that medical care providers can be assured that these POSIT scales are indeed measuring the intended problem areas [7,8] [Latimer WW, personal communication, 1996]. The POSIT should be studied further, but we believe that the current study provides evidence of reliability that supports its use in medical settings. It is not, however, a perfect instrument for medical office screening of adolescents. During the course of this study, we found that the administration time was approximately 20 to 30 min and the scoring time another 10 min (for a trained assistant). Although this may be practical in mental health or specialty settings dedicated to adolescents, it is not very practical for family practice and general pediatric offices. Further studies should determine whether the POSIT can be reduced in length. Item analysis in this study did not indicate that reliability would be improved by deleting any items, but it may be that scales could be shortened without sacrificing validity. Alternatively, medical care providers may wish to administer only those scales that have high clinical relevance and internal consistency (see above). One promising development is that the POSIT has been adapted for computer administration and scoring, with a CDROM version currently available [16]. Our experience with the program has been positive.
RELIABILITY OF THE POSIT
129
The suggested scale cut-points raise issues of practicality for medical care providers because more than 95% of our subjects screened “positive” on at least one scale. Given the fact that a positive screening score suggests nothing more than that further assessment is required, the POSIT may be a useful instrument for screening adolescent medical patients before they are seen by the provider. In this fashion, providers could show each patient his or her own results and use a high scale score to open further conversation regarding risks and problems in a particular area. As “expert clinician researchers” suggested the original scale cut-points, however, more empirical studies are needed [6]. These studies should be aimed at determining multiple cut-points that can guide clinical decision-making (e.g., no treatment vs. brief intervention vs. referral to a specialist). This limitation has been addressed in the POSIT software program. The computerized scoring algorithm computes each patient’s total scale score and displays all scores in a bar graph with indications of low-risk, moderate-risk, and high-risk categories. This program’s “administrator” section (available only by clinician password) also lists each item that contributed to the patient’s positive score. A printout of this graph and item list could serve as a valuable clinician-patient feedback device and would likely enhance rather than hinder clinician efficiency.
Conclusion The POSIT is a useful instrument for screening adolescents for a number of potential problems. Further refinement is needed to improve the internal consistency of several scales. However, this study presents strong evidence of internal consistency reliability of the Substance Use/Abuse, Mental Health, Educational Status, and Aggressive Behavior/Delinquency scales in a general adolescent medical clinic population. These are problem areas for which every adolescent patient should be screened. In addition, we found good evidence supporting the test–retest reliability of all POSIT scales. A new computerized administration and scoring version of the POSIT will enhance its practicality. The POSIT shows promise as a clinical adjunct to history taking in the medical office setting. We thank Lon Sherritt, M.P.H., and Ja’Nean Palacios, B.S., for their assistance with statistical analysis; and the entire staff of the Adolescent/Young Adult Medical Practice at Children’s Hospital in Boston for assisting in study implementation. Elizabeth Rahdert
130
KNIGHT ET AL.
JOURNAL OF ADOLESCENT HEALTH Vol. 29, No. 2
Ph.D., of the National Institute on Drug Abuse, provided information on POSIT development, administration, and scoring.
7. Melchior LA, Rahdert E, Huba GJ. Reliability and Validity Evidence for the Problem Oriented Screening Instruments for Teenagers (POSIT). Washington, DC: American Public Health Association, 1994.
References
8. McLaney MA, Del Boca F, Babor T. A validation study of the Problem Oriented Screening Instrument for Teenagers (POSIT). J Mental Health 1994;3:363–76.
1. Centers for Disease Control and Prevention. Youth risk behavior surveillance—United States, 1997. MMWR CDC Surveill Summ 1998;47:1– 89. 2. Centers for Disease Control and Prevention. Mortality trends and leading causes of death among adolescents and young adults—United States, 1979 –1988. MMWR Morb Mort Wkly Rep 1993;42:459 – 64. 3. Johnston L, O’Malley P, Bachman J. Drug use by young people begins to turn downward. Available at: http://www.isr.umich.edu/src/mtf/. Accessed 1998. 4. AMA. Guidelines for Adolescent Preventive Services (GAPS). Chicago: Department of Adolescent Health, American Medical Association, 1995. 5. Winters K. Screening and Assessing Adolescents for Substance Use Disorders. Treatment Improvement Protocol (TIP) Series. Vol. 31. DHHS Pub. No. (SMA) 99-3282. Rockville, MD: U.S. Department of Health and Human Services, 1999. 6. Rahdert ER. The Adolescent Assessment/Referral System Manual. Rockville, MD: U.S. Department of Health and Human Services, Alcohol, Drug Abuse, and Mental Health Administration, 1991.
9. Winters K, Stinchfield R, Henly G. The Personal Experience Inventory Test and Manual. Los Angeles: Western Psychological Services, 1988. 10. Dembo R, Schmeidler J, Brodern P, et al. Examination of the reliability of the Problem Oriented Screening Instrument for Teenagers (POSIT) among arrested youth entering a juvenile assessment center. Subst Use Misuse 1996;31:785– 824. 11. Society for Adolescent Medicine. Guidelines for adolescent health research. J Adol Health 1995;17:264 – 69. 12. SPSS version 9.0. Chicago, IL: SPSS, Inc., 1998. 13. Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297–334. 14. Nunnally J, Bernstein I. Psychometric Theory. 3rd ed. New York: McGraw-Hill, 1994:491–541. 15. Rosner B. Fundamentals of Biostatistics. Belmont, CA: Wadsworth Publishing, 1994:426. 16. Problem Oriented Screening Instrument for Teenagers (POSIT Version 2). Landover, MD: PowerTrain, Inc., 1998.
ORIGINAL ARTICLE
Reliability of the Problem Oriented Screening Instrument for Teenagers (POSIT) in Adolescent Medical Practice JOHN R. KNIGHT, M.D., ELIZABETH GOODMAN, M.D., TODD PULERWITZ, M.D., AND ROBERT H. DuRANT, PhD
Purpose: To determine the internal consistency and 1-week test–retest reliability of the Problem Oriented Screening Instrument for Teenagers (POSIT) among adolescent medical patients. Methods: A research assistant administered the POSIT to a consecutive sample of 15- to 18-year-old patients arriving for routine medical care at a hospital-based adolescent medical practice. Each subject returned for a retest 1 week later. Internal consistency for each scale on test and retest was calculated using Cronbach alpha, and 1-week test–retest reliability by the intraclass correlation coefficient (r) and the kappa coefficient (). Results: The Substance Use/Abuse, Mental Health Status, Educational Status, and Aggressive Behavior/Delinquency scales had favorable alpha scores (> .70). Others, including Physical Health Status, had lower alpha scores. High intraclass correlation coefficients were found for all 10 POSIT scales (r ⴝ .72 to .88), although (r) was lower for males on two of the scales. Kappa coefficients for all scales indicated good reproducibility beyond chance ( ⴝ .42 to .73). Conclusions: This study provided supportive evidence for the reliability of the POSIT in primary care medical settings, although some POSIT scales could likely be improved. The 20- to 30-min administration time is most
From the Department of Pediatrics, Harvard Medical School, Boston, Massachusetts (J.R.K., E.G., T.P., R.H.D.), the Division of General Pediatrics, Children’s Hospital, Boston, Massachusetts (J.R.K.); and the Division of Adolescsent/Young Adult Medicine, Children’s Hospital, Boston, Massachusetts (E.G., R.H.D.). Address correspondence to: Elizabeth Goodman, M.D., John R. Knight, M.D., Children’s Hospital, 300 Longwood Avenue, Boston, MA 02115. Funded in part by the Maternal and Child Health Bureau, Project MCJ-MA259195, HRSA, DHHS. Manuscript accepted July 30, 2000.
practical in settings that are dedicated to adolescent medicine, and computerized administration and scoring are needed. © Society for Adolescent Medicine, 2001 KEY WORDS: POSIT Reliability Substance abuse Screening High-risk behavior Adolescents
Substance abuse among adolescents is a major national problem. Almost 80% of high school students in the United States have begun to drink alcohol, 50% are current drinkers, and 33% are binge-drinkers [1]. Nearly 17% of students drive a car after drinking, and more than 36% ride in cars with an intoxicated driver [1]. Motor vehicle crashes are the leading cause of death among adolescents, and approximately 42% of these fatalities are related to the use of alcohol [2]. Use of illicit drugs among our nation’s youth also continues to be a problem. By the time they reach 12th grade, 54% of high school students have used a drug other than alcohol, and 29% have used an illicit drug other than marijuana [3]. The American Medical Association’s Guidelines for Adolescent Preventive Services (GAPS) recommend yearly screening of adolescents being seen for routine medical care for substance abuse and other biomedical, behavioral, and emotional conditions [4]. Clinicians need an efficient and reliable means for accomplishing this screening, and paper-and-pencil
© Society for Adolescent Medicine, 2001 Published by Elsevier Science Inc., 655 Avenue of the Americas, New York, NY 10010
1054-139X/01/$–see front matter PII S1054-139X(00)00206-8
KNIGHT ET AL.
126
JOURNAL OF ADOLESCENT HEALTH Vol. 29, No. 2
Table 1. Means, Standard Deviations, and Internal Consistency (␣) of POSIT Scales Test
Retest
POSIT Scale
No. of Items
n
Mean
SD
␣
n
Mean
SD
␣
Substance Use and Abuse Physical Health Status Mental Health Status Family Relations Peer Relations Educational Status Vocational Status Social Skills Leisure/Recreation Aggressive Behavior/Delinquency
17 10 22 11 10 23 18 11 12 16
145 153 143 146 152 136 148 147 151 149
1.0 2.8 6.5 3.4 3.2 6.8 4.9 3.6 4.5 5.0
1.8 1.7 3.8 2.2 2.2 3.6 2.5 1.9 2.0 3.3
.77 .45 .74 .63 .67 .71 .55 .46 .40 .79
82 87 78 83 85 79 81 82 86 78
0.8 2.4 5.8 3.6 3.2 6.3 4.8 3.3 4.7 4.4
1.9 1.9 4.6 2.2 2.5 4.1 2.5 2.1 2.1 3.6
.87 .51 .85 .60 .74 .79 .62 .58 .45 .85
SD ⫽ standard deviation.
questionnaires present one alternative to face-to-face interviews for busy physicians. Adolescents may prefer to answer personal questions on paper or a computer rather than during an interview with an adult. Numerous screening instruments are available for substance abuse, as well as other health risk and problem behaviors in adolescents [5]. However, relatively little research has been conducted to determine validity and reliability of these instruments in general adolescent populations or in health care settings that provide routine medical care. The Problem Oriented Screening Instrument for Teenagers (POSIT) is one such instrument that has received considerable attention in recent years. Designed for adolescents aged 12 through 19 years, POSIT is a self-report multi-problem screening instrument composed of 139 “yes/no” questions [6]. Its 10 scales are designed to screen for potential problems in the following functional domains: (a) Substance Use and Abuse, (b) Physical Health Status, (c) Mental Health Status, (d) Family Relations, (e) Peer Relations, (f) Educational Status, (g) Vocational Status, (h) Social Skills, (i) Leisure and Recreation, and (j) Aggressive Behavior and Delinquency. The number of items in each POSIT scale is shown in Table 1. Individual questions in the POSIT are of three types: generalpurpose items, age-related items, and red-flag items. Any positive score, whether a red-flag item or a scale total score, indicates that a potential problem exists and that further assessment in that area is recommended. Red-flag items are intended to identify problem areas in need of further assessment when they alone are scored as positive (e.g., “Do you get into trouble because you use alcohol or drugs at school?”). The psychometric properties of the POSIT have
been previously measured in a number of adolescent populations. In general, these previous studies have measured internal consistency, test–retest reliability, and validity of the 10 scales. Internal consistency indicates that items within a scale are measuring the same construct and that a higher total score is likely to indicate higher total risk. This is a particularly important measure for clinicians who may base treatment or referral decisions for their patients on scores that exceed a recommended cut-point. Test– retest reliability measures the temporal stability of a scale, and high reliability indicates that the measurement error of the test is relatively small over brief intervals of time during which behavior itself is unlikely to have changed. Concurrent validity indicates that the scale is indeed measuring the construct that it purports to measure (e.g., substance abuse vs. some other health-risk behavior) and is assessed by comparing the scale score in question with a criterion standard. Melchior et al. [7] studied the validity and internal reliability of POSIT scales in three groups of adolescents classified as low risk (high school students), moderate risk (in outpatient drug counseling clinics), or high risk (in drug inpatient treatment and juvenile detention). All 10 POSIT scales demonstrated good concurrent differential validity with measures of similar constructs (i.e., all scales successfully differentiated youth who were heavy substance users from those who were nonusers). Internal consistency, however, varied among scales. Alpha scores ranged from .47 to .93, with Substance Use/Abuse (.93), Mental Health Status (.86), Aggressive Behavior/ Delinquency (.85), and Educational Status (.82) having the highest alphas, and with Leisure/Recreation (.47), Vocational Status (.47), Social Skills (.51), and
August 2001
Physical Health Status (.56) having lower alpha scores. McLaney et al. [8] studied the psychometric properties of the POSIT in a drug abuse clinic population. They found lower internal consistency coefficients on most of the scales than those reported by Melchior et al. but found good evidence of scale validity. The correlation between the POSIT’s Substance Use/ Abuse scale and the 10 substance use scales of the Personal Experience Inventory (PEI) [9] ranged from .51 to .69, with a mean of .60. Similar correlations were found between other POSIT scales and their counterparts on the PEI. Dembo et al. [10] studied test–retest reliability of the POSIT over varying time intervals in a population of adolescent juvenile offenders. The highest test–retest reliability coefficients were found for the 0- to 2-week time interval. These ranged from .50 for the Vocational Status scale to .73 for the Peer Relations scale. The reliability coefficients decreased as the time interval between the first and second testing increased, a finding more likely indicative of changing life circumstances than actual scale psychometric properties. Prior studies have thus found that the POSIT has promising psychometric properties in certain settings. What is unknown is the reliability of the POSIT in other more general settings. The purpose of this study was to determine the internal consistency and 1-week test–retest reliability of the POSIT in an adolescent general medical clinic population. The results of this study will assist clinicians in determining the utility of the POSIT as a screening instrument for their adolescent medical patients.
Methods Subjects Subjects were drawn from 15- to 18-year-old patients receiving care in the Children’s Hospital, Adolescent/Young Adult Medical Practice, Boston, a primary care and referral clinic serving inner city and suburban youth from the full range of the social strata. This clinic has more than 4000 patients and more than 10,000 visits annually for well care and acute and nonacute medical problems. Twenty-four percent of patients have no health insurance, 33% have Medicaid, and 41% have private insurance. Sixty-nine percent of the study sample (n ⫽ 193) was female, 42% were African-American, 20% Latino, 16% white, and 1% Asian (21% other or no response), reflecting the demographic distribution of the clinic’s population.
RELIABILITY OF THE POSIT
127
Procedures A research assistant who was not involved in providing medical care invited a convenience sample of patients requesting routine care to participate in the study in June through August 1996. The purpose of the study was explained to each patient to be assessment of the consistency, reliability, and the overall “quality of the test,” and confidentiality was assured. Patients who agreed to participate gave informed consent. Based on the Guidelines for Adolescent Health Research, the Children’s Hospital Committee on Clinical Investigations (IRB-equivalent) waived the requirement for parental consent [11]. However, all subjects were encouraged to discuss their participation in the study with their parents or another responsible adult and were offered confidential referral if they wished to discuss further any issues that were raised by their participation in the study. We excluded from the study three patients with acute anorexia nervosa, one who came to be examined for alleged sexual assault, and three others who came to discuss results of a positive pregnancy test, because their providers judged they might be unduly stressed. Data were not kept on the number of patients who were invited to participate but refused. However, the research associate estimated that the percentage of refusers was very low after the first 2 weeks of the study; most cited lack of time as the reason. Potential subjects were offered a gift certificate to a local merchant (for a hamburger, slice of pizza, etc.) for participating. They were given these coupons after completion of the retest. The POSIT was administered in a private area at the end of the medical visit. No names were placed on the questionnaire. Subjects were assigned unique numerical identifiers to link test with retest while maintaining confidentiality. After the completion of the POSIT, subjects were given the opportunity to ask questions and were given an appointment for the retest 1 week later. They were asked for permission to have the research assistant telephone them the day prior to this appointment as a reminder. When subjects arrived for the retest, they were given the same instructions, and the same procedure was followed in administering the POSIT. If the retest appointment was missed, the research assistant telephoned and tried to reschedule the retest within 8 days of the initial test. Ninety-three of 173 subjects (53.8%) completed the retest. Data were entered and analyzed in SPSS statistical software [12]. Each of the 10 POSIT scales was
128
KNIGHT ET AL.
JOURNAL OF ADOLESCENT HEALTH Vol. 29, No. 2
Table 2. Test–Retest Reliabilities (r), Scale Cut-Points, Scale Positivity Rates, and Kappa Coefficients () of POSIT Scales POSIT Scale
Test–Retest r
Suggested CutPoint
% subjects ⱖ Cut-Point
Substance Use and Abuse Physical Health Status Mental Health Status Family Relations Peer Relations Educational Status Vocational Status Social Skills Leisure/Recreation Aggressive Behavior/Delinquency
.77 .78 .88 .78 .83 .82 .77 .73 .72 .83
1 3 4 4 1 6 5 3 5 6
37.6 50 74.7 39.9 95.3 60 56 60.6 47.3 31.7
.46 .68 .68 .73 .42 .69 .60 .52 .60 .65
computed for subjects who answered at least 75% of the questions on the scale. Subjects who answered less than 75% of the items in a scale were excluded from the analysis for that scale only. Means and standard deviations were computed for each scale. Internal consistency was measured using Cronbach alpha [13] and test–retest reliability using the single measure intraclass correlation coefficient (r). Kappa coefficients were computed for all items and for the positive/negative status of each scale according to suggested cut-points [6].
Results The means of each POSIT scale at test and retest are shown in Table 1. Retest completers and noncompleters did not differ significantly on any POSIT scale mean score (p ⫽ .08 to .96) or demographic variable (p ⫽ .20 to .76) other than gender (completers ⫽ 76.3% female, noncompleters ⫽ 62.5%, p ⫽ .049). The percentage of subjects who screened positive in each scale at initial test according to suggested scale cut-points [6] is shown in Table 2. Internal consistency of the scales as measured by Cronbach alpha varied at both test (␣ ⫽ .40 to .79) and retest (␣ ⫽ .45 to .87) (Table 1). Alpha coefficients of .70 and above are generally considered favorable. The Physical Health, Vocational Status, Social Skill, and Leisure/Recreation scales had alpha coefficients that fell below this level at the initial test. It should be noted, however, that alpha is partly a function of scale length and these scales consist of relatively fewer items [14]. Scale alphas did not increase substantially with deletion of any items, and scale alphas did not differ substantially (⌬␣ ⱖ .10) by gender with the exception of the Physical Health scale (␣ ⫽ .37 for males vs. .47 for females). With the exception of the Family Relations scale, all alphas increased substan-
tially at retest. The Substance Use/Abuse, Mental Health, and Aggressive Behavior scales had the highest scale alphas at both test and retest. High intraclass correlation coefficients were found for all 10 POSIT scales (Table 2). The lowest intraclass correlation was found for the Leisure/Recreation scale (r ⫽ .72) and the highest for the Mental Health scale (r ⫽ .88). Intraclass correlation coefficients for males and females did not differ substantially (⌬r ⱖ .10) for most scales. However, (r) was lower for males than females on the Family Relations scale (.58 vs. .83) and the Leisure/Recreation (.56 vs. .83) scale. We computed kappa coefficients on all items and found that 103/139 (74.1%) items demonstrated good reproducibility beyond chance ( ⫽ .40 to .75) and 15/139 (10.8%) additional items demonstrated excellent reproducibility beyond chance ( ⬎ .75) [15]. The kappa coefficient is a more important measure, however, for the positive/negative status of each scale, and for those items that are designated as “red-flags” because each is designed to indicate a need for further assessment when it alone is scored as positive. Kappa coefficients indicated good reproducibility beyond chance for positive/negative status in all 10 scales ( ⫽ .42 to .73) (Table 2). Of 44 red-flag items, 30 (68.2%) had kappa coefficients that indicated good reproducibility beyond chance, and five (11.4%) additional items had kappa coefficients that indicated excellent reproducibility beyond chance.
Discussion The POSIT was developed with a variety of aims: (a) to estimate the service needs in city/state political jurisdictions for multiple problem adolescents and their families, (b) to refer youths within the juvenile justice system to appropriate treatment services, and
August 2001
(c) to assist in treatment matching for troubled teenagers within clinical practice settings [6]. Our study addressed this third aim, examining the utility of the POSIT for purposes of risk identification in a health care setting. The findings of this study provide good evidence for the reliability of certain POSIT scales among adolescent medical patients, and these scales (i.e., Substance Use/Abuse, Mental Health, and Aggressive Behavior/Delinquency) are of great importance to medical care providers. Clinicians can be assured that the POSIT is a generally reliable tool for screening adolescent medical patients. For the scales that have high internal consistency, scores can also provide an indication of problem severity and assist the primary care provider to decide what management strategy is most appropriate (e.g., brief office intervention vs. referral to a specialist). This study has limitations. Data were not collected on those adolescent patients who refused participation, and it is unknown to what extent this group may have differed from the study sample as to risk severity. This study tested reliability of the POSIT, not validity, sensitivity, or specificity of individual scales. However, other studies have provided good evidence for the validity of the Substance Use/ Abuse, Mental Health Status, and Aggressive Behavior/Delinquency scales, and we believe that medical care providers can be assured that these POSIT scales are indeed measuring the intended problem areas [7,8] [Latimer WW, personal communication, 1996]. The POSIT should be studied further, but we believe that the current study provides evidence of reliability that supports its use in medical settings. It is not, however, a perfect instrument for medical office screening of adolescents. During the course of this study, we found that the administration time was approximately 20 to 30 min and the scoring time another 10 min (for a trained assistant). Although this may be practical in mental health or specialty settings dedicated to adolescents, it is not very practical for family practice and general pediatric offices. Further studies should determine whether the POSIT can be reduced in length. Item analysis in this study did not indicate that reliability would be improved by deleting any items, but it may be that scales could be shortened without sacrificing validity. Alternatively, medical care providers may wish to administer only those scales that have high clinical relevance and internal consistency (see above). One promising development is that the POSIT has been adapted for computer administration and scoring, with a CDROM version currently available [16]. Our experience with the program has been positive.
RELIABILITY OF THE POSIT
129
The suggested scale cut-points raise issues of practicality for medical care providers because more than 95% of our subjects screened “positive” on at least one scale. Given the fact that a positive screening score suggests nothing more than that further assessment is required, the POSIT may be a useful instrument for screening adolescent medical patients before they are seen by the provider. In this fashion, providers could show each patient his or her own results and use a high scale score to open further conversation regarding risks and problems in a particular area. As “expert clinician researchers” suggested the original scale cut-points, however, more empirical studies are needed [6]. These studies should be aimed at determining multiple cut-points that can guide clinical decision-making (e.g., no treatment vs. brief intervention vs. referral to a specialist). This limitation has been addressed in the POSIT software program. The computerized scoring algorithm computes each patient’s total scale score and displays all scores in a bar graph with indications of low-risk, moderate-risk, and high-risk categories. This program’s “administrator” section (available only by clinician password) also lists each item that contributed to the patient’s positive score. A printout of this graph and item list could serve as a valuable clinician-patient feedback device and would likely enhance rather than hinder clinician efficiency.
Conclusion The POSIT is a useful instrument for screening adolescents for a number of potential problems. Further refinement is needed to improve the internal consistency of several scales. However, this study presents strong evidence of internal consistency reliability of the Substance Use/Abuse, Mental Health, Educational Status, and Aggressive Behavior/Delinquency scales in a general adolescent medical clinic population. These are problem areas for which every adolescent patient should be screened. In addition, we found good evidence supporting the test–retest reliability of all POSIT scales. A new computerized administration and scoring version of the POSIT will enhance its practicality. The POSIT shows promise as a clinical adjunct to history taking in the medical office setting. We thank Lon Sherritt, M.P.H., and Ja’Nean Palacios, B.S., for their assistance with statistical analysis; and the entire staff of the Adolescent/Young Adult Medical Practice at Children’s Hospital in Boston for assisting in study implementation. Elizabeth Rahdert
130
KNIGHT ET AL.
JOURNAL OF ADOLESCENT HEALTH Vol. 29, No. 2
Ph.D., of the National Institute on Drug Abuse, provided information on POSIT development, administration, and scoring.
7. Melchior LA, Rahdert E, Huba GJ. Reliability and Validity Evidence for the Problem Oriented Screening Instruments for Teenagers (POSIT). Washington, DC: American Public Health Association, 1994.
References
8. McLaney MA, Del Boca F, Babor T. A validation study of the Problem Oriented Screening Instrument for Teenagers (POSIT). J Mental Health 1994;3:363–76.
1. Centers for Disease Control and Prevention. Youth risk behavior surveillance—United States, 1997. MMWR CDC Surveill Summ 1998;47:1– 89. 2. Centers for Disease Control and Prevention. Mortality trends and leading causes of death among adolescents and young adults—United States, 1979 –1988. MMWR Morb Mort Wkly Rep 1993;42:459 – 64. 3. Johnston L, O’Malley P, Bachman J. Drug use by young people begins to turn downward. Available at: http://www.isr.umich.edu/src/mtf/. Accessed 1998. 4. AMA. Guidelines for Adolescent Preventive Services (GAPS). Chicago: Department of Adolescent Health, American Medical Association, 1995. 5. Winters K. Screening and Assessing Adolescents for Substance Use Disorders. Treatment Improvement Protocol (TIP) Series. Vol. 31. DHHS Pub. No. (SMA) 99-3282. Rockville, MD: U.S. Department of Health and Human Services, 1999. 6. Rahdert ER. The Adolescent Assessment/Referral System Manual. Rockville, MD: U.S. Department of Health and Human Services, Alcohol, Drug Abuse, and Mental Health Administration, 1991.
9. Winters K, Stinchfield R, Henly G. The Personal Experience Inventory Test and Manual. Los Angeles: Western Psychological Services, 1988. 10. Dembo R, Schmeidler J, Brodern P, et al. Examination of the reliability of the Problem Oriented Screening Instrument for Teenagers (POSIT) among arrested youth entering a juvenile assessment center. Subst Use Misuse 1996;31:785– 824. 11. Society for Adolescent Medicine. Guidelines for adolescent health research. J Adol Health 1995;17:264 – 69. 12. SPSS version 9.0. Chicago, IL: SPSS, Inc., 1998. 13. Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297–334. 14. Nunnally J, Bernstein I. Psychometric Theory. 3rd ed. New York: McGraw-Hill, 1994:491–541. 15. Rosner B. Fundamentals of Biostatistics. Belmont, CA: Wadsworth Publishing, 1994:426. 16. Problem Oriented Screening Instrument for Teenagers (POSIT Version 2). Landover, MD: PowerTrain, Inc., 1998.