- Email: [email protected]
Development and Psychometric Properties of a Violence Screening Tool for Primary Care
Journal of Adolescent Health 48 (2011) 358 –365
www.jahonline.org Original article
Development and Psychometric Properties of a Violence Screening Tool for Primary Care Eric Sigel, M.D.*, Jan Hart, M.S.P.H., Analice Hoffenberg, M.D., M.S.P.H., and Melinda Dodge University of Colorado Denver School of Medicine, Aurora, Colorado
Article history: Received April 17, 2010; Accepted July 27, 2010 Keywords: Adolescents; Youth violence; Screening; Primary care
A B S T R A C T
Purpose: The aim of this study was to develop and validate a screening tool to detect youth at risk for future violence perpetration for primary care. Methods: Youth (n ⫽ 165) aged 11–17 years enrolled during a primary care appointment. Two clinics served as study sites. Youth filled out questionnaires confidentially at baseline and at 1-year follow-up. Primary outcome was violent behavior during the preceding year. At baseline, youth answered 18 risk and protective factor questions that predicted future violence involvement. Additional violence scales were asked for a total of 47 questions. Item analysis determined which combination best predicted future violence involvement. Psychometric properties, including internal consistency, test–retest reliability, convergent validity, and predictive validity, were analyzed. Results: A total of 101 youth (61%) completed 1-year follow-up: 16% reported violent behavior with no difference between gender or race/ethnicity. Twenty-five baseline questions correlated with violence involvement 1 year later. After item analysis, 14 questions demonstrated the strongest psychometric functioning with Cronbach’s ␣ ⫽ .77. External validity was strong, with the 14 item violence injury protection and risk screen correlating with the aggression (.74) and victimization (.54) scales, the Strength and Difficulties Questionnaire (.39), and current violence involvement (.78). For youth aged 14 –17 years, predictive validity was strongly correlated (.78) with future violence perpetration. A score of 5.0 for males and 6.0 for females revealed a sensitivity of 77%, a specificity of 98%, and a positive predictive value of 91%. Seventeen percent of youth aged 14 –17 screened positive using these cutpoints. Conclusion: A brief, 14-item questionnaire demonstrated strong psychometric functioning and performed well as a screening tool to predict future violence perpetration for youth aged 14 –17. 䉷 2011 Society for Adolescent Health and Medicine. All rights reserved.
The problem of youth violence continues to be widespread. Approximately 25%– 45% of youth have been reported to be in a fight in the past year [1,2] and homicide remains the second leading cause of death in adolescents and young adults [1]. Despite the significant morbidity and mortality related to youth violence, primary care providers (PCPs) do not adequately address youth violence [3]. Diagnosis of either current violence involvement (perpetration or victimization), or risk for future
* Address correspondence to: Eric Sigel, M.D., University of Colorado Denver School of Medicine, 13123 E 16th Ave B-025, Aurora, CO 80045. E-mail address: [email protected]
violence perpetration or injury is rarely made. Pediatricians do not feel comfortable dealing with violence issues with their patients [3,4]. The rate of diagnoses of assault injury and violence perpetration in 10 –21 year olds in the primary care setting was reported to be only .3% and .84%, respectively, much lower than expected compared with national youth self report [5]. A screening tool to assess an adolescent’s risk for future violence perpetration or injury in the primary care setting has not been developed. Despite youth self-reports indicating that violence involvement and injuries are common, PCPs infrequently screen youth about their violence involvement. Borowsky and Ireland [6] showed that 67% of practitioners never or rarely asked adoles-
1054-139X/$ - see front matter 䉷 2011 Society for Adolescent Health and Medicine. All rights reserved. doi:10.1016/j.jadohealth.2010.07.024
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
cents about their involvement in physical fighting. Chaffee et al [3] found that pediatricians only screen their patients 31% of the time for fighting, and 29% of the time for carrying a weapon. Both the American Medical Association and the American Academy of Pediatrics (AAP) have strongly encouraged PCPs to screen youth for violence-related behavior [7,8]. The AAP’s Task Force on Youth Violence published recommendations in 1999, with an update in 2009, describing the role of the pediatrician in youth violence prevention [9]. The AAP argued that because violence and injuries are a serious threat to the health of children, “pediatricians should incorporate preventive education, screening for risk, and linkages to necessary intervention and follow-up services.” Subsequently, the AAP developed a comprehensive program, Connected Kids [10], that guides health care practitioners to address youth violence throughout the life cycle. The program recommends screening youth for violence risk, but does not provide a validated tool to do so. Although there are no validated specific youth violence screening tools for use in primary care, several general mental health screening tools, such as the Pediatric Symptom Checklist (PSC) [11], the Achenbach [12], and the Strength and Difficulties Questionnaire (SDQ) [13,14], have been successful at detecting problem behavior [15]. The effectiveness of general mental health screening suggests that routine screening for violence could help PCPs identify and treat more youth at risk for violence perpetration. The purpose of this study was to develop and validate a screening tool that can help health care providers predict which adolescents are at risk for future violence perpetration and injury. We hypothesized that a screening tool could be developed on the basis of risk and protective factors that would predict future violence perpetration and injury. After a violence risk screen is established, it can assist PCPs in identifying youth for further assessment. Patients and methods Development of the violence injury protection and risk screen The violence injury protection and risk screen (VIPRS) is theoretically based on significant risk and protective factors identified from multiple longitudinal studies that are strongly associated with future violence-related injury, perpetration, and victimization in adolescents [16 –20]. Studies show that each factor by itself is associated with future violence involvement. The goal of combining these factors into a screening tool was to determine whether adding them together might create a stronger method to detect youth at risk for future violence perpetration than any one factor alone. The logic for including certain questions follows. Resnick et al [16] found that risk factors for violence perpetration 1 year from baseline included violence involvement at baseline, violence victimization, repeating a grade, carrying a weapon to school, marijuana and alcohol use, and treatment for emotional problems. Protective factors included parental expectations for school performance, family connectedness, grade point average, religiosity, and connectedness to adults outside the family. Borowsky and Ireland [17] found that witnessing or being a victim of violence, a history of violence-related injury, and history of physical fighting were strong predictors of future violence-related injury. Illicit drug use in male adolescents and high levels of depressive symptoms in female adolescents were
359
also predictors of future violence-related injury. Protective factors included excellent perceived general health and a high grade point average. Herrenkohl et al [18] evaluated youth for risk factors at age 10 and determined violence perpetration at ages 14, 16, and 18. Risk factors predictive of perpetrating violence included being male, being hyperactive, drug selling, early violence initiation, parental criminality, low academic performance, peer delinquency, and gang membership. Sege et al [19] found a correlation between having been in a fight, poor school performance, and using substances with future violence-related injury. Molnar et al [20] determined that girls who were victimized were more likely to report violence perpetration (24.5%) than those who had never been victimized (11.3%). In addition to this group of risk and protective factors that were hypothesized to predict future violence perpetration, youth filled out Aggression and Victimization scales [21], and the Violence Perpetration and Injury scale. In total, 47 violence risk and protective factors were asked at baseline to determine the final VIPRS scale. Subjects and study design A longitudinal design was used to evaluate the best combination of questions that predicted future violence perpetration and then psychometric properties of the selected items were assessed in a stepwise manner. Youth aged 11–17 years attending their primary care clinic for a routine or sick visit, together with their parents, were invited to participate in the study. Patients with significant developmental disabilities were excluded. Written informed consent was obtained from parents, and written assent from youth. The study took place in two primary care clinics located in a diverse urban setting: one academic adolescent medicine practice, and one private pediatric practice. The clinic sites were selected as they represented a wide range of socioeconomic, racial, and ethnic diversity, and a similar geographic area, located .5 mile apart. Each site contributed approximately half the sample. The study was approved by the Colorado Multiple Institutional Review Board. A quasi-randomized approach was used to identify eligible participants. Every other youth on the schedule who met inclusion criteria were invited to participate. Participants completed a self-administered, paper and pencil survey confidentially after enrollment in the waiting room. Youth were aware that if any suicidal or homicidal intent, or abuse, was revealed, confidentiality would be broken and appropriate personnel notified. Youth began the survey before their clinic visit. Some youth completed surveys before their visit, whereas others finished after the visit. Results were not shared with the PCP. The surveys were stored in a locked cabinet to protect privacy. Data were entered into a database, and only the Principal Investigator and Research Assistant had password-protected access to it. The RA contacted the youth by phone 3– 6 weeks after enrollment and got answers to a subset of questions that were asked to examine test–retest reliability. The subjects were contacted by phone again 1 year after enrollment to assess their degree of violence involvement during the preceding year. Study instruments The primary outcome, violence perpetration and injury, was determined by a 7-item scale developed and validated in the National Longitudinal Study on Adolescent Health [22,23], that measures violence perpetration and injuries related to violence
360
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
Table 1 Violence perpetration and injury scale: 1-year follow-up In the past 12 months how often did you:
1. Use or threaten to use a weapon to get something from someone? Total Males Females 11–13 year olds 14–17 year olds 2. Take part in a group fight? Total Males Females 11–13 year olds 14–17 year olds 3. Pull a knife/gun on someone? Total Males Females 11–13 year olds 14–17 year olds 4. Shoot/stab someone? Total Males Females 11–13 year olds 14–17 year olds 5. Get into a serious physical fight? Total Males Females 11–13 year olds 14–17 year olds 6. Get in a fight where you were injured and had to be treated by a doctor or nurse? Total Males Females 11–13 year olds 14–17 year olds 7. Hurt someone badly enough to need bandages or care from a doctor or nurse? Total Males Females 11–13 year olds 14–17 year olds Total positive on VPIS Males Females 11–13 year olds 14–17 year olds
1-Year follow-up: % (95% CI) n ⫽ 101
3 (0–6.3) 7.9 (⫺1 to 16.9) 0 0
p valuea
.02* .2
4.6 (0–9.7) 6.9 (1.9–12) 10.5 (.3–21) 4.8 (0–10.2) 2.9 (⫺2.9 to 8.7) 9.1 (2–16.2) 2 (0–4.7) 5.3 (⫺2 to 12.7) 0 0
.27 .24
.07 .3
3 (⫺1 to 7.3) 0 0 0 0 0 11.9 (5.5–18.3) 7.9 (⫺1 to 16.9) 14.3 (5.4–23.2) 5.7 (⫺2 to 13.8) 15.2 (6.3–24)
2 (0–4.7) 2.6 (⫺2.7 to 8) 1.6 (⫺1.6 to 4.8) 0
.52 .16
.72 .19
3 (⫺1 to 7.3)
5 (1–9.2) 7.9 (⫺1 to 16.9) 3.2 (⫺1.2 to 7.6) 0 7.6 (1–14) 15.8 (8.6–23) 18.4 (5.5–33) 14.3 (5.4–23) 8.6 (⫺1.1 to 18.3) 19.7 (9.9–30)
.3 .1
.58 .15
VPIS: violence perpetration and injuries related to violence. a p values are between group comparisons (gender, age). * p ⬍ .05.
(VPIS). A positive response to any one of the questions on the scale was considered positive (Table 1) for perpetrating violence, as established in the ADD Health studies (Cronbach’s ␣ for the VPIS ⫽ .83). Youth filled out Aggression and Victimization surveys, validated scales recommended by the Centers for Disease Control and Prevention [21]. The Aggressive Behavior Survey evaluates 12 areas of aggressive behavior [24,25]. The Violence Victimization scale evaluates self-reported victimization by peers, and has been used in several studies, with an internal consistency of .85 [26,27]. Additional scales used to validate the
VIPRS include the SDQ, a 25-item self-administered mental health screen validated in adolescents aged 11–15 years (Cronbach’s ␣ ⫽ .73) [13,14,28]. Parents filled out the VPIS pertaining to their children, and completed the PSC [29,30], a 35-item parent report tool that assesses general mental health status of their children. Analytical plan Descriptive statistics were computed to describe baseline characteristics of youth. Rates (with 95% CI) of positive responses to the initial 18 hypothesized risk and protective factors, and the additional 29 questions were determined. Item discrimination Only the responses of the youth who had completed follow-up (n ⫽ 101) were used to determine the final VIPRS scale. Each factor was evaluated to determine which individual questions predicted future violence involvement. Correlation coefficients between each item, and the patient-reported VPIS at 1-year follow-up were determined. Youth were classified as perpetrating violence if they had any positive response on the 7-item VPIS. Determining final VIPRS questions Items were initially discarded if they did not correlate to future violence perpetration. The remaining items were then analyzed, and items were discarded if they improved Cronbach’s ␣ by .02 if removed. Multiple combinations were then scored to determine which best predicted future violence perpetration, while keeping the number of questions reasonable for a screening tool. Once the final questions were determined, the VIPRS was scored using a sum of all items. Risk factors were assigned a positive score (1 for yes, 0 for no). A lack of the presence of a protective factor was scored as 1, and the presence of a protective factor was scored as 0. Scores could range from 0 (no risk) to 14 (highest risk). Instrument validity and reliability Psychometric properties of the VIPRS were assessed using standard approaches described by Nunnally et al [31]. Pearson’s correlation coefficients were used to assess the bivariate relationships between two variables, using two-sided tests (␣ ⫽ .05). A sample size of 165 subjects provided 80% power to detect correlations of .20 or greater. Internal consistency was measured using Cronbach’s ␣. Convergent validity was assessed by computing the Pearson correlation coefficient between the VIPRS, the baseline VPIS, the aggression and victimization scales, the subscales of the SDQ, and the parent-reported VPIS and PSC. Predictive validity of the VIPRS was examined using bivariate correlations between the VIPRS and the patient-reported VPIS at 1-year follow-up. Cutpoints in the VIPRS scores were evaluated to determine sensitivity, specificity, and predictive value of the VIPRS as a screen to detect future violence involvement (Figure 1). Results Baseline A total of 165 youth participated in the study. The average age of the participants was 14.5 years (SD: 1.7); 43% were male and 46% were white adolescents (Table 2). Baseline violence involvement was substantial (Table 3): 23.2% reported being involved in
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
361
Table 3 Baseline violence involvement characteristics
Fight in the past year Injured in a fight within the past year Injured in a fight needing medical attention Report perpetrating physical aggression Report physical victimization Serious violence involvement
Males % (n ⫽ 71)
Females % (n ⫽ 94)
p
27.1 16.9 4.2 38.6 22.1 25.4
18.1 16.5 4.3 32.2 18.1 16.1
.17 .94 .98 .4 .55 .14
One-year follow-up
Figure 1. Study flow. Baseline data collection (N ⫽ 165): Reliability, convergent validity done on final VIPRS questions. Test–retest reliability: 3– 6 weeks after baseline enrollment (n ⫽ 110). One-year follow-up (n ⫽ 102): predictive validity determined. Item analysis done to determine final VIPRS.
a fight in the past year, 13.6% were injured in a fight, and 3% were injured in a fight needing medical attention. In all, 19% scored positive on the VPIS (Table 1). All male and female adolescents responded similarly on all but 4 of the 47 questions: females reported more drug use (30% vs. 14%, p ⫽ .02), more emotional problems (33% vs. 13%, p ⫽ .003), and having had their feelings hurt (50% vs. 34%, p ⫽ .03) compared with males. Male adolescents watched a fight more frequently (59% vs. 42%, p ⫽ .03). Younger adolescents [11–13] answered similarly to older youth [14 –17] on all but 4 of the 47 questions. Older youth were more likely to have watched a fight (55% vs. 36%, p ⫽ .04), used drugs (30% vs. 6%, p ⫽ .001), have trouble with the law (46% vs. 19%, p ⫽ .001), and have lower academic performance (27% vs. 11%, p ⫽ .03).
Table 2 Baseline and follow-up demographic characteristics
Age (years) Age group 11–13 14–17 Gender Male Female Race/ethnicity White Black Hispanic Asian Other Visit type Routine physical exam Sick visit Contraceptive Follow-up/other Insurance (n ⫽ 164) Public Commercial None
Baseline % (N ⫽ 165)
Follow-up % (N ⫽ 101)
14.5 (SD: 1.7)
14.4 (SD: 1.8)a
29 71
34.7 65.3
43 57
38 62
46 22 27 2.4 2.6
49 19 27 3 2
47.2 16.6 19.0 17.2
47 15 21 17
43.4 49.4 6.0
43.6 52.6 3.8
There was no statistically significant difference between baseline and follow-up groups. a Baseline age.
A total of 101 youth (61%) completed 1-year follow-up, of which 38% were male. On the basis of gender, race/ethnicity, or visit type, no differences at baseline were found between those who were followed up as compared with those who were not. Youth who were unavailable for follow-up were those more likely to have hurt someone else in a fight, asked to fight, and beaten up. Sixteen percent reported perpetrating violence in the past year, with no differences between gender or race/ethnicity. However, age affected future violent behavior: 9% of 11–13 year olds reported violence perpetration at 1-year follow-up, whereas 24% of 14 –16 year olds reported violence perpetration (p ⫽ .04). More youth from the academic practice (27%) than youth from private practice (7%) reported violence perpetration or injury in the past year (p ⬍ .001). Logistic regression showed that site did not influence violence perpetration, and the VIPRS performed similarly between the sites (data not shown). Determining the final VIPRS scale A total of 47 baseline questions were analyzed to assess correlation with future violence involvement. Of these, 25 baseline questions correlated with future violence perpetration and injury: 11 of 18 from the theorized original VIPRS, 4 of 7 from the VPIS, 8 of 12 from the aggression scale, and 2 of 10 from the victimization scale. After item analysis and assessing prediction of future violence perpetration, 14 questions provided the strongest psychometric functioning and made up the final VIPRS questionnaire (Table 4). Items not associated with future violence perpetration included having repeated a grade, carrying a weapon to school, having an emotional diagnosis, and being injured in a fight that required medical attention. Items that did not protect against future violence involvement included feeling connected to adults in one’s family or outside one’s family, and religious involvement. Psychometric properties of the VIPRS The VIPRS showed strong psychometric functioning. Internal consistency revealed a Cronbach’s ␣ of .77. Test–retest reliability was evaluated 3– 6 weeks after initial enrollment (n ⫽ 106). The original 18 baseline questions had an r ⫽ .920. Additionally, 9 of 14 final VIPRS questions had an r ⫽ .910. The VIPRS performed well compared with several measures to establish convergent validity. Strong correlations were observed (p ⬍ .01) between VIPRS scores and the baseline VPIS (r ⫽ .57), SDQ (r ⫽ .39), the aggression scale (r ⫽ .74), and the victimization scale (r ⫽ .54). There was also strong association between the baseline VIPRS scores and the parental PSC (r ⫽ .43). The predictive validity of
362
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
Table 4 Violence injury and perpetration risk screen (final VIPRS scale): percent and odds of future violence perpetration by question type Protective factors
1. Do your parents expect you to do well at school? Most of the time Sometimes Rarely/never 2. Are your grades mostly A/B average C average D/F average Risk factors
3. Have you been suspended from school in the last year? Yes 4. How many fights have you been in during the last year? 0 1–5 5. Have you ever smoked marijuana or used other drugs? Yes 6. Have you or your friends ever been in trouble with the law?a Yes Males Females 7. Are you or your friends involved with a gang or tagging crew? Yes 8. Do you feel you are hyperactive, or have you ever been diagnosed with ADHDa Yes Males Females 9. Have you had any friends that have committed suicide? Yes 10. Have you ever been injured in a fight? Yes 11. When was the last time you hurt someone else in a fight? In the past month Between 1–6 months ago Between 6–12 months ago Over 1 year ago Never Positive if ⱕ12 months ago 12. When was the last time you watched a fight? In the past month Between 1–6 months ago Between 6–2 months ago Over 1 year ago Never Positive if ⱕ12 months ago 13. How many times has someone beat you up in the last 6 months 0 1–6 14. How many times has someone asked you to fight in the last 6 months? 0 1–6 a
n ⫽ 101 %
Odds of association with violence perpetration if protective factor not present OR (95% CI)
p value
6.1 (1.1–33.83)
.02
5.8 (1.7–19.2)
.002
93 6.1 .6 77.9 16.6 5.5 % Positive
17
Odds of association with future violence perpetration if risk factor present (OR 95% CI)
47 (11.1–201)
.000
82 18
10.7 (3.2–35.6)
.000
23
4.6 (1.5–14.2)
.005
38.2 40.8 36.2
2.1 (.72–6.2) 8.6 (1.4–54.2) .85 (.19–3.8)
.13 .02 .56
7.0
4.6 (.93–23.0)
.04
14.8 15.9 14.0
3.0 (.88–10.3) .83 (.08–8.5) 6.4 (1.3–30.7)
.07 .69 .03
11
3.7 (.93–14.4)
10
4.3 (1.1–17.6)
.03
24.6 (4.3–138.7)
.000
4.3 6.7 3.7 11.6 73.8 14.6 25 13.4 11.6 22.6 27.4 50
6.1 (1.6–22)
.004
91.2 9.8
6.2 (1.1–34.2)
.02
75 25
3.6 (1.1–11.7)
.03
Questions 6 and 8 display gender differences, since these two questions reveal statistically significant differences between genders.
the VIPRS was also strong, with an r ⫽ .78 with future violence perpetration. The performance of the VIPRS for predicting future violence involvement was not dependent on gender or race/ ethnicity, as demonstrated by a nonsignificant Breslow–Day test of homogeneity. VIPRS scores The higher the VIPRS score, the higher the likelihood of detecting violence involvement. The mean VIPRS score was 2.73 (SD: 2.67), with a range of 0 to 10 (Figure 2), with no differences between genders. After the final 14-item scale was determined,
the effect of age was determined. Younger adolescents aged 11–13 scored lower on the VIPRS (mean score, 1.8;SD: 2.2), compared with 14 –17 year olds (mean score, 3.1; SD: 2.8), (p ⫽ .004). Sensitivity, specificity, and predictive value of the VIPRS Table 5 displays the performance of the VIPRS for predicting future violence perpetration and injury. Specific cutpoints that predicted future violence perpetration and injury were explored-in total, by age and gender. Overall, a cutpoint of 6 had a sensitivity of 63%, a specificity of 99%, and a positive predictive value (PPV) of 91%. When examining test performance by age
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
363
Figure 2. VIPRS scoring distribution.
and gender, characteristics improve for older youth. For 14 –17 year olds, a cutpoint of 6 had a sensitivity of 77%, a specificity of 98%, and a PPV of 91%. Using this cutpoint, 17% screened positive. Examining male adolescents in this age group, a lower score seemed to perform well: a cutpoint of 5 had a sensitivity of 83%,
a specificity of 94%, and a PPV of 83%, with 25% screening positive. For female adolescents, a score of 6 carried a sensitivity of 86, a specificity of 97, and a PPV of 86, with 17% screening positive. The screen did not capture any young adolescents aged 11–13 who perpetrated violence 1 year later, and so sensitivity could not be
Table 5 Sensitivity, specificity, positive and negative predictive value, and varying VIPRS cutpoints, by total sample, by gender and by age group (n ⫽ 101) VIPRS Total Cut point 4 Cut point 5 Cut point 6 Male Cut point 4 Cut point 5 Cut point 6 Female Cut point 4 Cut point 5 Cut point 6 Age 11–13 Cut point 4 Cut point 5 Cut point 6 Age 14–17 Cut point 4 Cut point 5 Cut point 6 a
Sensitivity
Specificity
PPV
NPV
% Screen positive
OR
75 69 63
85 92 99
48 61 91
95 94 93
25 17.8 10.9
16.6 (4.6–59.6) 24.2 (6.6–90.8) 140 (15–1,284)
86 71 57
90 94 100
67 71 100
97 94 91
23.7 18.4 10.5
56 (4.9–635) 36 (4.1–319)
67 67 67
82 91 98
38 55 86
94 94 95
25.4 17.5 11.1
33 0 0
94 94 100
25 0 0
91 90 94
11.4 5.7
85 85 77
81 91 98
52 69 91
96 96 95
32 24.2 16.7
OR not calculated because at least 1 cell had 0.
a
8.8 (1.9–41.3) 19.6 (3.7–103) 106 (9.5–1,186) 4.8 (.3–70.4) .91 (.8–1.01) a
23.6 (4.5–123) 3.1 (1.5–6.4) 173 (16.3–1,839)
364
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
determined. However, the test was 94% specific, with a 90% negative predictive value in the younger age group. Violence perpetration at baseline to predict future violence perpetration Examining violence perpetration alone at baseline revealed an overall sensitivity of 44%, a specificity of 90%, and a PPV of 46%. For the older age group, aged 14 –17 years, the VPIS had sensitivity of 54%, a specificity of 85%, and a PPV of 70%. Discussion The VIPRS is the first screening tool that has been validated in a clinic population to predict future risk for violence perpetration and injury. The VIPRS demonstrated strong psychometric properties in terms of internal consistency, test–retest reliability, construct validity, and convergent validity. The VIPRS also exhibits excellent predictive validity for both male and female adolescents aged 14 –17 years. Although there was no statistical difference between genders in actual violence perpetration at follow-up, females needed a higher score on the VIPRS-6 versus VIPRS-5 to actually predict future violence perpetration. This may suggest that females need to experience more risk or have less protection at baseline to actually lead to future violence perpetration. The VIPRS did not demonstrate predictive validity for younger adolescents aged 11–13 years. The failure to work in the younger age group is likely related to the fact that few youth actually perpetrated violence at 1-year follow-up. The smaller sample size (n ⫽ 31) and the general lower reporting of risk behavior at baseline in the younger age group may explain why the screening tool did not work for the younger adolescents. Interestingly, the VIPRS, representing a group of risk and protective factors, performed much better than just using baseline violence perpetration to predict future violence perpetration in terms of sensitivity (83% compared with 54%), specificity (98% compared with 85%), and PPV (91% compared with 70%). The VIPRS takes into account other behaviors in addition to violencerelated behaviors, which makes it more robust. Risk behavior certainly exists that correlates with future violence perpetration and can be grouped by category. Physically aggressive behaviors, such as getting into fights, hurting others in fights, and being victims of physical aggression, group together. Other predictors are related to being bystanders around violent behavior such as watching a fight and being asked to fight. Several general adolescent risk behaviors are also associated with future violence perpetration, such as using drugs, being gang involved or having friends that are gang involved, suspension, having trouble with the law, and considering oneself to be hyperactive. Having a friend who has committed suicide is also considered a risk factor. Two factors that protect against future violence perpetration include academic performance and perceiving that parents have high academic expectations. These risk and protective factors are not newly identified, as the theoretic framework for this study is based on previous longitudinal assessment for violence risk and protection. What is novel is the attempt to combine these factors into a concrete scale that predicts future violence perpetration. Although baseline violence perpetration is a strong predictor of future violence perpetration, the VIPRS takes into account other risk and protective factors, making the scale broader and more effective at detecting risk for future violence perpetration.
Screening for violence involvement has been strongly recommended by both the AAP and American Medical Association. The AAPs recent update from the Committee on Violence [9] again called upon pediatricians to incorporate violence screening as part of the Connected Kids program. Connected Kids recommends using the acronym FISTS [32] (fighting, injury, suicide, threatening others, sexual violence), a construct proposed by Alpert, though it has never been validated. The VIPRS includes all the FISTS categories, except sexual violence. The VIPRS complements the AAP’s call to action for violence screening by providing a validated method for 14 –17 year olds, the group at the highest to perpetrate violence. The VIPRS could be added to the several mental health and risk screening tools that currently exist, or used alone to address violence specifically. By being able to detect which adolescents are at risk for perpetrating violence in the future, health care providers will be in position to further assess those youth at risk, and provide intervention that will decrease their future violence perpetration. The strengths of this study are that it used a longitudinal design and had a moderate size study population, representative of adolescents attending their primary care clinic in an urban setting. Demographic characteristics show that youth were from a diverse socio-economic and racial/ethnic population. The study included both risk and protective factors to help determine future violence perpetration and injury. Parents participated at baseline, and the screening tool correlated strongly with parental concern for general behavioral issues. Limitations were that the study relied on youth self report, and did not use school or legal data to corroborate violence involvement. Additional limitations were that although the study was performed in a clinic setting and was done confidentially, youth may not self-report as much risk if they know that a clinician will be evaluating their responses. Test–retest reliability, as well as the primary outcome measure, was obtained from phone responses compared with the baseline paper survey. The differing methods could potentially influence how an adolescent answered the survey questions. Conclusion The VIPRS, a brief, 14-item questionnaire demonstrates strong psychometric functioning and performs well as a screening tool that predicts future violence perpetration for youth aged 14 –17 years. The next step will be to implement this screening tool clinically by having youth complete the VIPRS either online, before their appointment, or electronically in the office setting. The 14 items can be added to existing screening questionnaires that are used frequently with teenagers. When identified as high risk, the PCP would have the opportunity to explore violence risk in depth, as well as link youth at risk to the variety of evidencebased services that exist in the community that decrease future violence perpetration. By so doing, the morbidity youth experience from violence involvement may begin to decrease. Acknowledgments This research project has been funded in part by the Colorado Injury Control Research Center, and the CDCP’s National Center for Injury Prevention and Control, grant Number 5K01CE001332-03. All those who have significantly contributed to this work have been listed.
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
References [1] Centers for Disease Control and Prevention. Web-Based Injury Statistics Query and Reporting System (WISQARS) National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, 2009. Available at: http://webappa.cdc.gov/cgi-bin/broker.exe. [2] Centers for Disease Control and Prevention. Youth risk behavioral surveillance—United States, 2007. Morb Mortal Wkly Rep 2007;57(SS04):1–131. [3] Chaffee TA, Bridges M, Boyer CB. Adolescent violence prevention practices among California pediatricians. Arch Pediatr Adolesc Med 2000; 154:1034 – 41. [4] Finch SA, Weiley V, Ip EH, Barkin S. Impact of pediatricians’ perceived self-efficacy and confidence on violence prevention counseling: A National Study. Matern Child Health J 2008;12:75– 82. [5] Sigel ES, Bublitz C, Kutner J. Examining primary Care practitioners detection of youth at risk for violence injury and aggression. Pediatric Academic Society Poster Symposium, April 30, 2006, San Francisco, CA. [6] Borowsky IW, Ireland M. National survey of pediatricians’ violence prevention counseling. Arch Pediatr Adolesc Med 1999;153:1170 – 6. [7] Youth and violence, medicine, health and nursing, connecting the dots to prevent violence commission for the prevention of youth violence, AMA publication, December 2000. Available at: http://www.ama-assn.org/ama/ upload/mm/386/fullreport.pdf. Accessed January 15, 2005. [8] American Academy of Pediatrics. The role of the pediatrician in youth violence prevention in clinical practice and at the community level. American Academy of Pediatrics Task Force on Violence. Pediatrics 1999;103: 173– 81. [9] American Academy of Pediatrics. Policy statement—Role of the pediatrician in youth violence prevention. AAP Committee on Injury, Violence and Poison Prevention. Pediatrics 2009;124:393– 402. [10] American Academy of Pediatrics. Available at: http://www.aap.org/ ConnectedKids/. Accessed July 20, 2009. [11] Murphy JM, Ichinose C, Hicks RC, et al. Utility of the pediatric symptom checklist as a psychosocial screen in EPSDT. J Pediatr 1996;129:864 –9. [12] Achenbach TM. The child behavior profile: I. Boys aged 6 –11. J Consult Clin Psychol 1978;46:478 – 88. [13] Goodman R. Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatry 2001;40:1337– 45. [14] Goodman R, Scott S. Comparing the strengths and difficulties questionnaire and the child behavior checklist: Is small beautiful? J Abnorm Psychol 1999;27:17–24. [15] Murphy JM, Reede J, Jellinek MS, Bishop SJ. Screening for psychosocial dysfunction in inner-city children: Further validation of the Pediatric Symptom checklist. J Am Acad Child Adolesc Psychiatry 1992;31:1105–11.
365
[16] Resnick MD, Ireland M, Borowsky I. Youth violence perpetration: What protects? What predicts? Findings from the National Longitudinal Study of Adolescent health. J Adolesc Health 2004;35:e1–10. [17] Borowsky IW, Ireland M. Predictors of future fight-related injury among adolescents. Pediatrics 2004;113:530 – 6. [18] Herrenkohl TI, Maguin E, Hill KG, et al. Developmental risk factors for youth violence. J Adolesc Health 2000;26:176 – 86. [19] Sege R, Stringham P, Short S, Griffith J. Ten years after: Examination of adolescent screening questions that predict future violence-related injury. J Adolesc Health 1999;24:395– 402. [20] Molnar BE, Browne A, Cerda M, Buka SL. Violent behavior by girls reporting violent victimization: A prospective study. Arch Pediatr Adolesc Med 2005; 159:731–9. [21] National Center for Injury Prevention, CDC. Measuring violence-related attitudes, beliefs, and behaviors among youths: A compendium of assessment tools. Available at: http://www.cdc.gov/ncipc/pub-res/measure.htm. Accessed July 1, 2005. [22] Resnick MD, Bearman PS, Blum RW, et al. Protecting adolescents from harm: Findings from the National Longitudinal Study on Adolescent Health. JAMA 1997;278:823–32. [23] Sieving RE, Beuhring T, Resnick MD, et al. Development of adolescent self-report measures from the National Longitudinal Study of Adolescent health. J Adolesc Health 2001;28:73– 81. [24] Paschall MJ, Flewelling RL. Measuring intermediate outcomes of violence prevention programs targeting African-American male youth: An exploratory assessment of the psychometricproperties of six psychosocial measures. Health Educ Res 1997;12:117–28. [25] Flewelling RL, Paschall MJ, Ringwalt CL. Sage Baseline Survey. Research Triangle Park, NC: Research Triangle Institute, 1993. [26] Orpinas P, Kelder S, Frankowski R, et al. Outcome evaluation of a multicomponent violence-prevention program for middle schools: The students for peace project. Health Educ Res 2000;15:45–58. [27] Orpinas P, Murray N, Kelder S. Parental influences on students’ aggressive behaviors and weapon carrying. Health Educ Behav 1999;26:774 – 87. [28] Goodman R, Meltzer H, Bailey V. The Strengths and Difficulties Questionnaire: A pilot study on the validity of the self-report version. Int Rev Psychiatry 2003;15:173–7. [29] Pediatric symptom checklist. Available at: http://psc.partners.org/psc_ basic.htm. Accessed January 15, 2005. [30] Jellinek MS, Murphy JM. The recognition of psychosocial disorders in pediatric office practice: The current status of the pediatric symptom checklist. J Dev Behav Pediatr 1990;11:273– 8. [31] Nunnally JC, Bernstein IH. Psychometric theory, 3rd edition. New York, NY: McGraw-Hill, 1994. [32] Alpert EJ, Sege RD, Bradshaw YS. Interpersonal violence and the education of physicians. Acad Med 1997;72(Suppl 1):S41–50.
www.jahonline.org Original article
Development and Psychometric Properties of a Violence Screening Tool for Primary Care Eric Sigel, M.D.*, Jan Hart, M.S.P.H., Analice Hoffenberg, M.D., M.S.P.H., and Melinda Dodge University of Colorado Denver School of Medicine, Aurora, Colorado
Article history: Received April 17, 2010; Accepted July 27, 2010 Keywords: Adolescents; Youth violence; Screening; Primary care
A B S T R A C T
Purpose: The aim of this study was to develop and validate a screening tool to detect youth at risk for future violence perpetration for primary care. Methods: Youth (n ⫽ 165) aged 11–17 years enrolled during a primary care appointment. Two clinics served as study sites. Youth filled out questionnaires confidentially at baseline and at 1-year follow-up. Primary outcome was violent behavior during the preceding year. At baseline, youth answered 18 risk and protective factor questions that predicted future violence involvement. Additional violence scales were asked for a total of 47 questions. Item analysis determined which combination best predicted future violence involvement. Psychometric properties, including internal consistency, test–retest reliability, convergent validity, and predictive validity, were analyzed. Results: A total of 101 youth (61%) completed 1-year follow-up: 16% reported violent behavior with no difference between gender or race/ethnicity. Twenty-five baseline questions correlated with violence involvement 1 year later. After item analysis, 14 questions demonstrated the strongest psychometric functioning with Cronbach’s ␣ ⫽ .77. External validity was strong, with the 14 item violence injury protection and risk screen correlating with the aggression (.74) and victimization (.54) scales, the Strength and Difficulties Questionnaire (.39), and current violence involvement (.78). For youth aged 14 –17 years, predictive validity was strongly correlated (.78) with future violence perpetration. A score of 5.0 for males and 6.0 for females revealed a sensitivity of 77%, a specificity of 98%, and a positive predictive value of 91%. Seventeen percent of youth aged 14 –17 screened positive using these cutpoints. Conclusion: A brief, 14-item questionnaire demonstrated strong psychometric functioning and performed well as a screening tool to predict future violence perpetration for youth aged 14 –17. 䉷 2011 Society for Adolescent Health and Medicine. All rights reserved.
The problem of youth violence continues to be widespread. Approximately 25%– 45% of youth have been reported to be in a fight in the past year [1,2] and homicide remains the second leading cause of death in adolescents and young adults [1]. Despite the significant morbidity and mortality related to youth violence, primary care providers (PCPs) do not adequately address youth violence [3]. Diagnosis of either current violence involvement (perpetration or victimization), or risk for future
* Address correspondence to: Eric Sigel, M.D., University of Colorado Denver School of Medicine, 13123 E 16th Ave B-025, Aurora, CO 80045. E-mail address: [email protected]
violence perpetration or injury is rarely made. Pediatricians do not feel comfortable dealing with violence issues with their patients [3,4]. The rate of diagnoses of assault injury and violence perpetration in 10 –21 year olds in the primary care setting was reported to be only .3% and .84%, respectively, much lower than expected compared with national youth self report [5]. A screening tool to assess an adolescent’s risk for future violence perpetration or injury in the primary care setting has not been developed. Despite youth self-reports indicating that violence involvement and injuries are common, PCPs infrequently screen youth about their violence involvement. Borowsky and Ireland [6] showed that 67% of practitioners never or rarely asked adoles-
1054-139X/$ - see front matter 䉷 2011 Society for Adolescent Health and Medicine. All rights reserved. doi:10.1016/j.jadohealth.2010.07.024
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
cents about their involvement in physical fighting. Chaffee et al [3] found that pediatricians only screen their patients 31% of the time for fighting, and 29% of the time for carrying a weapon. Both the American Medical Association and the American Academy of Pediatrics (AAP) have strongly encouraged PCPs to screen youth for violence-related behavior [7,8]. The AAP’s Task Force on Youth Violence published recommendations in 1999, with an update in 2009, describing the role of the pediatrician in youth violence prevention [9]. The AAP argued that because violence and injuries are a serious threat to the health of children, “pediatricians should incorporate preventive education, screening for risk, and linkages to necessary intervention and follow-up services.” Subsequently, the AAP developed a comprehensive program, Connected Kids [10], that guides health care practitioners to address youth violence throughout the life cycle. The program recommends screening youth for violence risk, but does not provide a validated tool to do so. Although there are no validated specific youth violence screening tools for use in primary care, several general mental health screening tools, such as the Pediatric Symptom Checklist (PSC) [11], the Achenbach [12], and the Strength and Difficulties Questionnaire (SDQ) [13,14], have been successful at detecting problem behavior [15]. The effectiveness of general mental health screening suggests that routine screening for violence could help PCPs identify and treat more youth at risk for violence perpetration. The purpose of this study was to develop and validate a screening tool that can help health care providers predict which adolescents are at risk for future violence perpetration and injury. We hypothesized that a screening tool could be developed on the basis of risk and protective factors that would predict future violence perpetration and injury. After a violence risk screen is established, it can assist PCPs in identifying youth for further assessment. Patients and methods Development of the violence injury protection and risk screen The violence injury protection and risk screen (VIPRS) is theoretically based on significant risk and protective factors identified from multiple longitudinal studies that are strongly associated with future violence-related injury, perpetration, and victimization in adolescents [16 –20]. Studies show that each factor by itself is associated with future violence involvement. The goal of combining these factors into a screening tool was to determine whether adding them together might create a stronger method to detect youth at risk for future violence perpetration than any one factor alone. The logic for including certain questions follows. Resnick et al [16] found that risk factors for violence perpetration 1 year from baseline included violence involvement at baseline, violence victimization, repeating a grade, carrying a weapon to school, marijuana and alcohol use, and treatment for emotional problems. Protective factors included parental expectations for school performance, family connectedness, grade point average, religiosity, and connectedness to adults outside the family. Borowsky and Ireland [17] found that witnessing or being a victim of violence, a history of violence-related injury, and history of physical fighting were strong predictors of future violence-related injury. Illicit drug use in male adolescents and high levels of depressive symptoms in female adolescents were
359
also predictors of future violence-related injury. Protective factors included excellent perceived general health and a high grade point average. Herrenkohl et al [18] evaluated youth for risk factors at age 10 and determined violence perpetration at ages 14, 16, and 18. Risk factors predictive of perpetrating violence included being male, being hyperactive, drug selling, early violence initiation, parental criminality, low academic performance, peer delinquency, and gang membership. Sege et al [19] found a correlation between having been in a fight, poor school performance, and using substances with future violence-related injury. Molnar et al [20] determined that girls who were victimized were more likely to report violence perpetration (24.5%) than those who had never been victimized (11.3%). In addition to this group of risk and protective factors that were hypothesized to predict future violence perpetration, youth filled out Aggression and Victimization scales [21], and the Violence Perpetration and Injury scale. In total, 47 violence risk and protective factors were asked at baseline to determine the final VIPRS scale. Subjects and study design A longitudinal design was used to evaluate the best combination of questions that predicted future violence perpetration and then psychometric properties of the selected items were assessed in a stepwise manner. Youth aged 11–17 years attending their primary care clinic for a routine or sick visit, together with their parents, were invited to participate in the study. Patients with significant developmental disabilities were excluded. Written informed consent was obtained from parents, and written assent from youth. The study took place in two primary care clinics located in a diverse urban setting: one academic adolescent medicine practice, and one private pediatric practice. The clinic sites were selected as they represented a wide range of socioeconomic, racial, and ethnic diversity, and a similar geographic area, located .5 mile apart. Each site contributed approximately half the sample. The study was approved by the Colorado Multiple Institutional Review Board. A quasi-randomized approach was used to identify eligible participants. Every other youth on the schedule who met inclusion criteria were invited to participate. Participants completed a self-administered, paper and pencil survey confidentially after enrollment in the waiting room. Youth were aware that if any suicidal or homicidal intent, or abuse, was revealed, confidentiality would be broken and appropriate personnel notified. Youth began the survey before their clinic visit. Some youth completed surveys before their visit, whereas others finished after the visit. Results were not shared with the PCP. The surveys were stored in a locked cabinet to protect privacy. Data were entered into a database, and only the Principal Investigator and Research Assistant had password-protected access to it. The RA contacted the youth by phone 3– 6 weeks after enrollment and got answers to a subset of questions that were asked to examine test–retest reliability. The subjects were contacted by phone again 1 year after enrollment to assess their degree of violence involvement during the preceding year. Study instruments The primary outcome, violence perpetration and injury, was determined by a 7-item scale developed and validated in the National Longitudinal Study on Adolescent Health [22,23], that measures violence perpetration and injuries related to violence
360
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
Table 1 Violence perpetration and injury scale: 1-year follow-up In the past 12 months how often did you:
1. Use or threaten to use a weapon to get something from someone? Total Males Females 11–13 year olds 14–17 year olds 2. Take part in a group fight? Total Males Females 11–13 year olds 14–17 year olds 3. Pull a knife/gun on someone? Total Males Females 11–13 year olds 14–17 year olds 4. Shoot/stab someone? Total Males Females 11–13 year olds 14–17 year olds 5. Get into a serious physical fight? Total Males Females 11–13 year olds 14–17 year olds 6. Get in a fight where you were injured and had to be treated by a doctor or nurse? Total Males Females 11–13 year olds 14–17 year olds 7. Hurt someone badly enough to need bandages or care from a doctor or nurse? Total Males Females 11–13 year olds 14–17 year olds Total positive on VPIS Males Females 11–13 year olds 14–17 year olds
1-Year follow-up: % (95% CI) n ⫽ 101
3 (0–6.3) 7.9 (⫺1 to 16.9) 0 0
p valuea
.02* .2
4.6 (0–9.7) 6.9 (1.9–12) 10.5 (.3–21) 4.8 (0–10.2) 2.9 (⫺2.9 to 8.7) 9.1 (2–16.2) 2 (0–4.7) 5.3 (⫺2 to 12.7) 0 0
.27 .24
.07 .3
3 (⫺1 to 7.3) 0 0 0 0 0 11.9 (5.5–18.3) 7.9 (⫺1 to 16.9) 14.3 (5.4–23.2) 5.7 (⫺2 to 13.8) 15.2 (6.3–24)
2 (0–4.7) 2.6 (⫺2.7 to 8) 1.6 (⫺1.6 to 4.8) 0
.52 .16
.72 .19
3 (⫺1 to 7.3)
5 (1–9.2) 7.9 (⫺1 to 16.9) 3.2 (⫺1.2 to 7.6) 0 7.6 (1–14) 15.8 (8.6–23) 18.4 (5.5–33) 14.3 (5.4–23) 8.6 (⫺1.1 to 18.3) 19.7 (9.9–30)
.3 .1
.58 .15
VPIS: violence perpetration and injuries related to violence. a p values are between group comparisons (gender, age). * p ⬍ .05.
(VPIS). A positive response to any one of the questions on the scale was considered positive (Table 1) for perpetrating violence, as established in the ADD Health studies (Cronbach’s ␣ for the VPIS ⫽ .83). Youth filled out Aggression and Victimization surveys, validated scales recommended by the Centers for Disease Control and Prevention [21]. The Aggressive Behavior Survey evaluates 12 areas of aggressive behavior [24,25]. The Violence Victimization scale evaluates self-reported victimization by peers, and has been used in several studies, with an internal consistency of .85 [26,27]. Additional scales used to validate the
VIPRS include the SDQ, a 25-item self-administered mental health screen validated in adolescents aged 11–15 years (Cronbach’s ␣ ⫽ .73) [13,14,28]. Parents filled out the VPIS pertaining to their children, and completed the PSC [29,30], a 35-item parent report tool that assesses general mental health status of their children. Analytical plan Descriptive statistics were computed to describe baseline characteristics of youth. Rates (with 95% CI) of positive responses to the initial 18 hypothesized risk and protective factors, and the additional 29 questions were determined. Item discrimination Only the responses of the youth who had completed follow-up (n ⫽ 101) were used to determine the final VIPRS scale. Each factor was evaluated to determine which individual questions predicted future violence involvement. Correlation coefficients between each item, and the patient-reported VPIS at 1-year follow-up were determined. Youth were classified as perpetrating violence if they had any positive response on the 7-item VPIS. Determining final VIPRS questions Items were initially discarded if they did not correlate to future violence perpetration. The remaining items were then analyzed, and items were discarded if they improved Cronbach’s ␣ by .02 if removed. Multiple combinations were then scored to determine which best predicted future violence perpetration, while keeping the number of questions reasonable for a screening tool. Once the final questions were determined, the VIPRS was scored using a sum of all items. Risk factors were assigned a positive score (1 for yes, 0 for no). A lack of the presence of a protective factor was scored as 1, and the presence of a protective factor was scored as 0. Scores could range from 0 (no risk) to 14 (highest risk). Instrument validity and reliability Psychometric properties of the VIPRS were assessed using standard approaches described by Nunnally et al [31]. Pearson’s correlation coefficients were used to assess the bivariate relationships between two variables, using two-sided tests (␣ ⫽ .05). A sample size of 165 subjects provided 80% power to detect correlations of .20 or greater. Internal consistency was measured using Cronbach’s ␣. Convergent validity was assessed by computing the Pearson correlation coefficient between the VIPRS, the baseline VPIS, the aggression and victimization scales, the subscales of the SDQ, and the parent-reported VPIS and PSC. Predictive validity of the VIPRS was examined using bivariate correlations between the VIPRS and the patient-reported VPIS at 1-year follow-up. Cutpoints in the VIPRS scores were evaluated to determine sensitivity, specificity, and predictive value of the VIPRS as a screen to detect future violence involvement (Figure 1). Results Baseline A total of 165 youth participated in the study. The average age of the participants was 14.5 years (SD: 1.7); 43% were male and 46% were white adolescents (Table 2). Baseline violence involvement was substantial (Table 3): 23.2% reported being involved in
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
361
Table 3 Baseline violence involvement characteristics
Fight in the past year Injured in a fight within the past year Injured in a fight needing medical attention Report perpetrating physical aggression Report physical victimization Serious violence involvement
Males % (n ⫽ 71)
Females % (n ⫽ 94)
p
27.1 16.9 4.2 38.6 22.1 25.4
18.1 16.5 4.3 32.2 18.1 16.1
.17 .94 .98 .4 .55 .14
One-year follow-up
Figure 1. Study flow. Baseline data collection (N ⫽ 165): Reliability, convergent validity done on final VIPRS questions. Test–retest reliability: 3– 6 weeks after baseline enrollment (n ⫽ 110). One-year follow-up (n ⫽ 102): predictive validity determined. Item analysis done to determine final VIPRS.
a fight in the past year, 13.6% were injured in a fight, and 3% were injured in a fight needing medical attention. In all, 19% scored positive on the VPIS (Table 1). All male and female adolescents responded similarly on all but 4 of the 47 questions: females reported more drug use (30% vs. 14%, p ⫽ .02), more emotional problems (33% vs. 13%, p ⫽ .003), and having had their feelings hurt (50% vs. 34%, p ⫽ .03) compared with males. Male adolescents watched a fight more frequently (59% vs. 42%, p ⫽ .03). Younger adolescents [11–13] answered similarly to older youth [14 –17] on all but 4 of the 47 questions. Older youth were more likely to have watched a fight (55% vs. 36%, p ⫽ .04), used drugs (30% vs. 6%, p ⫽ .001), have trouble with the law (46% vs. 19%, p ⫽ .001), and have lower academic performance (27% vs. 11%, p ⫽ .03).
Table 2 Baseline and follow-up demographic characteristics
Age (years) Age group 11–13 14–17 Gender Male Female Race/ethnicity White Black Hispanic Asian Other Visit type Routine physical exam Sick visit Contraceptive Follow-up/other Insurance (n ⫽ 164) Public Commercial None
Baseline % (N ⫽ 165)
Follow-up % (N ⫽ 101)
14.5 (SD: 1.7)
14.4 (SD: 1.8)a
29 71
34.7 65.3
43 57
38 62
46 22 27 2.4 2.6
49 19 27 3 2
47.2 16.6 19.0 17.2
47 15 21 17
43.4 49.4 6.0
43.6 52.6 3.8
There was no statistically significant difference between baseline and follow-up groups. a Baseline age.
A total of 101 youth (61%) completed 1-year follow-up, of which 38% were male. On the basis of gender, race/ethnicity, or visit type, no differences at baseline were found between those who were followed up as compared with those who were not. Youth who were unavailable for follow-up were those more likely to have hurt someone else in a fight, asked to fight, and beaten up. Sixteen percent reported perpetrating violence in the past year, with no differences between gender or race/ethnicity. However, age affected future violent behavior: 9% of 11–13 year olds reported violence perpetration at 1-year follow-up, whereas 24% of 14 –16 year olds reported violence perpetration (p ⫽ .04). More youth from the academic practice (27%) than youth from private practice (7%) reported violence perpetration or injury in the past year (p ⬍ .001). Logistic regression showed that site did not influence violence perpetration, and the VIPRS performed similarly between the sites (data not shown). Determining the final VIPRS scale A total of 47 baseline questions were analyzed to assess correlation with future violence involvement. Of these, 25 baseline questions correlated with future violence perpetration and injury: 11 of 18 from the theorized original VIPRS, 4 of 7 from the VPIS, 8 of 12 from the aggression scale, and 2 of 10 from the victimization scale. After item analysis and assessing prediction of future violence perpetration, 14 questions provided the strongest psychometric functioning and made up the final VIPRS questionnaire (Table 4). Items not associated with future violence perpetration included having repeated a grade, carrying a weapon to school, having an emotional diagnosis, and being injured in a fight that required medical attention. Items that did not protect against future violence involvement included feeling connected to adults in one’s family or outside one’s family, and religious involvement. Psychometric properties of the VIPRS The VIPRS showed strong psychometric functioning. Internal consistency revealed a Cronbach’s ␣ of .77. Test–retest reliability was evaluated 3– 6 weeks after initial enrollment (n ⫽ 106). The original 18 baseline questions had an r ⫽ .920. Additionally, 9 of 14 final VIPRS questions had an r ⫽ .910. The VIPRS performed well compared with several measures to establish convergent validity. Strong correlations were observed (p ⬍ .01) between VIPRS scores and the baseline VPIS (r ⫽ .57), SDQ (r ⫽ .39), the aggression scale (r ⫽ .74), and the victimization scale (r ⫽ .54). There was also strong association between the baseline VIPRS scores and the parental PSC (r ⫽ .43). The predictive validity of
362
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
Table 4 Violence injury and perpetration risk screen (final VIPRS scale): percent and odds of future violence perpetration by question type Protective factors
1. Do your parents expect you to do well at school? Most of the time Sometimes Rarely/never 2. Are your grades mostly A/B average C average D/F average Risk factors
3. Have you been suspended from school in the last year? Yes 4. How many fights have you been in during the last year? 0 1–5 5. Have you ever smoked marijuana or used other drugs? Yes 6. Have you or your friends ever been in trouble with the law?a Yes Males Females 7. Are you or your friends involved with a gang or tagging crew? Yes 8. Do you feel you are hyperactive, or have you ever been diagnosed with ADHDa Yes Males Females 9. Have you had any friends that have committed suicide? Yes 10. Have you ever been injured in a fight? Yes 11. When was the last time you hurt someone else in a fight? In the past month Between 1–6 months ago Between 6–12 months ago Over 1 year ago Never Positive if ⱕ12 months ago 12. When was the last time you watched a fight? In the past month Between 1–6 months ago Between 6–2 months ago Over 1 year ago Never Positive if ⱕ12 months ago 13. How many times has someone beat you up in the last 6 months 0 1–6 14. How many times has someone asked you to fight in the last 6 months? 0 1–6 a
n ⫽ 101 %
Odds of association with violence perpetration if protective factor not present OR (95% CI)
p value
6.1 (1.1–33.83)
.02
5.8 (1.7–19.2)
.002
93 6.1 .6 77.9 16.6 5.5 % Positive
17
Odds of association with future violence perpetration if risk factor present (OR 95% CI)
47 (11.1–201)
.000
82 18
10.7 (3.2–35.6)
.000
23
4.6 (1.5–14.2)
.005
38.2 40.8 36.2
2.1 (.72–6.2) 8.6 (1.4–54.2) .85 (.19–3.8)
.13 .02 .56
7.0
4.6 (.93–23.0)
.04
14.8 15.9 14.0
3.0 (.88–10.3) .83 (.08–8.5) 6.4 (1.3–30.7)
.07 .69 .03
11
3.7 (.93–14.4)
10
4.3 (1.1–17.6)
.03
24.6 (4.3–138.7)
.000
4.3 6.7 3.7 11.6 73.8 14.6 25 13.4 11.6 22.6 27.4 50
6.1 (1.6–22)
.004
91.2 9.8
6.2 (1.1–34.2)
.02
75 25
3.6 (1.1–11.7)
.03
Questions 6 and 8 display gender differences, since these two questions reveal statistically significant differences between genders.
the VIPRS was also strong, with an r ⫽ .78 with future violence perpetration. The performance of the VIPRS for predicting future violence involvement was not dependent on gender or race/ ethnicity, as demonstrated by a nonsignificant Breslow–Day test of homogeneity. VIPRS scores The higher the VIPRS score, the higher the likelihood of detecting violence involvement. The mean VIPRS score was 2.73 (SD: 2.67), with a range of 0 to 10 (Figure 2), with no differences between genders. After the final 14-item scale was determined,
the effect of age was determined. Younger adolescents aged 11–13 scored lower on the VIPRS (mean score, 1.8;SD: 2.2), compared with 14 –17 year olds (mean score, 3.1; SD: 2.8), (p ⫽ .004). Sensitivity, specificity, and predictive value of the VIPRS Table 5 displays the performance of the VIPRS for predicting future violence perpetration and injury. Specific cutpoints that predicted future violence perpetration and injury were explored-in total, by age and gender. Overall, a cutpoint of 6 had a sensitivity of 63%, a specificity of 99%, and a positive predictive value (PPV) of 91%. When examining test performance by age
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
363
Figure 2. VIPRS scoring distribution.
and gender, characteristics improve for older youth. For 14 –17 year olds, a cutpoint of 6 had a sensitivity of 77%, a specificity of 98%, and a PPV of 91%. Using this cutpoint, 17% screened positive. Examining male adolescents in this age group, a lower score seemed to perform well: a cutpoint of 5 had a sensitivity of 83%,
a specificity of 94%, and a PPV of 83%, with 25% screening positive. For female adolescents, a score of 6 carried a sensitivity of 86, a specificity of 97, and a PPV of 86, with 17% screening positive. The screen did not capture any young adolescents aged 11–13 who perpetrated violence 1 year later, and so sensitivity could not be
Table 5 Sensitivity, specificity, positive and negative predictive value, and varying VIPRS cutpoints, by total sample, by gender and by age group (n ⫽ 101) VIPRS Total Cut point 4 Cut point 5 Cut point 6 Male Cut point 4 Cut point 5 Cut point 6 Female Cut point 4 Cut point 5 Cut point 6 Age 11–13 Cut point 4 Cut point 5 Cut point 6 Age 14–17 Cut point 4 Cut point 5 Cut point 6 a
Sensitivity
Specificity
PPV
NPV
% Screen positive
OR
75 69 63
85 92 99
48 61 91
95 94 93
25 17.8 10.9
16.6 (4.6–59.6) 24.2 (6.6–90.8) 140 (15–1,284)
86 71 57
90 94 100
67 71 100
97 94 91
23.7 18.4 10.5
56 (4.9–635) 36 (4.1–319)
67 67 67
82 91 98
38 55 86
94 94 95
25.4 17.5 11.1
33 0 0
94 94 100
25 0 0
91 90 94
11.4 5.7
85 85 77
81 91 98
52 69 91
96 96 95
32 24.2 16.7
OR not calculated because at least 1 cell had 0.
a
8.8 (1.9–41.3) 19.6 (3.7–103) 106 (9.5–1,186) 4.8 (.3–70.4) .91 (.8–1.01) a
23.6 (4.5–123) 3.1 (1.5–6.4) 173 (16.3–1,839)
364
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
determined. However, the test was 94% specific, with a 90% negative predictive value in the younger age group. Violence perpetration at baseline to predict future violence perpetration Examining violence perpetration alone at baseline revealed an overall sensitivity of 44%, a specificity of 90%, and a PPV of 46%. For the older age group, aged 14 –17 years, the VPIS had sensitivity of 54%, a specificity of 85%, and a PPV of 70%. Discussion The VIPRS is the first screening tool that has been validated in a clinic population to predict future risk for violence perpetration and injury. The VIPRS demonstrated strong psychometric properties in terms of internal consistency, test–retest reliability, construct validity, and convergent validity. The VIPRS also exhibits excellent predictive validity for both male and female adolescents aged 14 –17 years. Although there was no statistical difference between genders in actual violence perpetration at follow-up, females needed a higher score on the VIPRS-6 versus VIPRS-5 to actually predict future violence perpetration. This may suggest that females need to experience more risk or have less protection at baseline to actually lead to future violence perpetration. The VIPRS did not demonstrate predictive validity for younger adolescents aged 11–13 years. The failure to work in the younger age group is likely related to the fact that few youth actually perpetrated violence at 1-year follow-up. The smaller sample size (n ⫽ 31) and the general lower reporting of risk behavior at baseline in the younger age group may explain why the screening tool did not work for the younger adolescents. Interestingly, the VIPRS, representing a group of risk and protective factors, performed much better than just using baseline violence perpetration to predict future violence perpetration in terms of sensitivity (83% compared with 54%), specificity (98% compared with 85%), and PPV (91% compared with 70%). The VIPRS takes into account other behaviors in addition to violencerelated behaviors, which makes it more robust. Risk behavior certainly exists that correlates with future violence perpetration and can be grouped by category. Physically aggressive behaviors, such as getting into fights, hurting others in fights, and being victims of physical aggression, group together. Other predictors are related to being bystanders around violent behavior such as watching a fight and being asked to fight. Several general adolescent risk behaviors are also associated with future violence perpetration, such as using drugs, being gang involved or having friends that are gang involved, suspension, having trouble with the law, and considering oneself to be hyperactive. Having a friend who has committed suicide is also considered a risk factor. Two factors that protect against future violence perpetration include academic performance and perceiving that parents have high academic expectations. These risk and protective factors are not newly identified, as the theoretic framework for this study is based on previous longitudinal assessment for violence risk and protection. What is novel is the attempt to combine these factors into a concrete scale that predicts future violence perpetration. Although baseline violence perpetration is a strong predictor of future violence perpetration, the VIPRS takes into account other risk and protective factors, making the scale broader and more effective at detecting risk for future violence perpetration.
Screening for violence involvement has been strongly recommended by both the AAP and American Medical Association. The AAPs recent update from the Committee on Violence [9] again called upon pediatricians to incorporate violence screening as part of the Connected Kids program. Connected Kids recommends using the acronym FISTS [32] (fighting, injury, suicide, threatening others, sexual violence), a construct proposed by Alpert, though it has never been validated. The VIPRS includes all the FISTS categories, except sexual violence. The VIPRS complements the AAP’s call to action for violence screening by providing a validated method for 14 –17 year olds, the group at the highest to perpetrate violence. The VIPRS could be added to the several mental health and risk screening tools that currently exist, or used alone to address violence specifically. By being able to detect which adolescents are at risk for perpetrating violence in the future, health care providers will be in position to further assess those youth at risk, and provide intervention that will decrease their future violence perpetration. The strengths of this study are that it used a longitudinal design and had a moderate size study population, representative of adolescents attending their primary care clinic in an urban setting. Demographic characteristics show that youth were from a diverse socio-economic and racial/ethnic population. The study included both risk and protective factors to help determine future violence perpetration and injury. Parents participated at baseline, and the screening tool correlated strongly with parental concern for general behavioral issues. Limitations were that the study relied on youth self report, and did not use school or legal data to corroborate violence involvement. Additional limitations were that although the study was performed in a clinic setting and was done confidentially, youth may not self-report as much risk if they know that a clinician will be evaluating their responses. Test–retest reliability, as well as the primary outcome measure, was obtained from phone responses compared with the baseline paper survey. The differing methods could potentially influence how an adolescent answered the survey questions. Conclusion The VIPRS, a brief, 14-item questionnaire demonstrates strong psychometric functioning and performs well as a screening tool that predicts future violence perpetration for youth aged 14 –17 years. The next step will be to implement this screening tool clinically by having youth complete the VIPRS either online, before their appointment, or electronically in the office setting. The 14 items can be added to existing screening questionnaires that are used frequently with teenagers. When identified as high risk, the PCP would have the opportunity to explore violence risk in depth, as well as link youth at risk to the variety of evidencebased services that exist in the community that decrease future violence perpetration. By so doing, the morbidity youth experience from violence involvement may begin to decrease. Acknowledgments This research project has been funded in part by the Colorado Injury Control Research Center, and the CDCP’s National Center for Injury Prevention and Control, grant Number 5K01CE001332-03. All those who have significantly contributed to this work have been listed.
E.J. Sigel et al. / Journal of Adolescent Health 48 (2011) 358 –365
References [1] Centers for Disease Control and Prevention. Web-Based Injury Statistics Query and Reporting System (WISQARS) National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, 2009. Available at: http://webappa.cdc.gov/cgi-bin/broker.exe. [2] Centers for Disease Control and Prevention. Youth risk behavioral surveillance—United States, 2007. Morb Mortal Wkly Rep 2007;57(SS04):1–131. [3] Chaffee TA, Bridges M, Boyer CB. Adolescent violence prevention practices among California pediatricians. Arch Pediatr Adolesc Med 2000; 154:1034 – 41. [4] Finch SA, Weiley V, Ip EH, Barkin S. Impact of pediatricians’ perceived self-efficacy and confidence on violence prevention counseling: A National Study. Matern Child Health J 2008;12:75– 82. [5] Sigel ES, Bublitz C, Kutner J. Examining primary Care practitioners detection of youth at risk for violence injury and aggression. Pediatric Academic Society Poster Symposium, April 30, 2006, San Francisco, CA. [6] Borowsky IW, Ireland M. National survey of pediatricians’ violence prevention counseling. Arch Pediatr Adolesc Med 1999;153:1170 – 6. [7] Youth and violence, medicine, health and nursing, connecting the dots to prevent violence commission for the prevention of youth violence, AMA publication, December 2000. Available at: http://www.ama-assn.org/ama/ upload/mm/386/fullreport.pdf. Accessed January 15, 2005. [8] American Academy of Pediatrics. The role of the pediatrician in youth violence prevention in clinical practice and at the community level. American Academy of Pediatrics Task Force on Violence. Pediatrics 1999;103: 173– 81. [9] American Academy of Pediatrics. Policy statement—Role of the pediatrician in youth violence prevention. AAP Committee on Injury, Violence and Poison Prevention. Pediatrics 2009;124:393– 402. [10] American Academy of Pediatrics. Available at: http://www.aap.org/ ConnectedKids/. Accessed July 20, 2009. [11] Murphy JM, Ichinose C, Hicks RC, et al. Utility of the pediatric symptom checklist as a psychosocial screen in EPSDT. J Pediatr 1996;129:864 –9. [12] Achenbach TM. The child behavior profile: I. Boys aged 6 –11. J Consult Clin Psychol 1978;46:478 – 88. [13] Goodman R. Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatry 2001;40:1337– 45. [14] Goodman R, Scott S. Comparing the strengths and difficulties questionnaire and the child behavior checklist: Is small beautiful? J Abnorm Psychol 1999;27:17–24. [15] Murphy JM, Reede J, Jellinek MS, Bishop SJ. Screening for psychosocial dysfunction in inner-city children: Further validation of the Pediatric Symptom checklist. J Am Acad Child Adolesc Psychiatry 1992;31:1105–11.
365
[16] Resnick MD, Ireland M, Borowsky I. Youth violence perpetration: What protects? What predicts? Findings from the National Longitudinal Study of Adolescent health. J Adolesc Health 2004;35:e1–10. [17] Borowsky IW, Ireland M. Predictors of future fight-related injury among adolescents. Pediatrics 2004;113:530 – 6. [18] Herrenkohl TI, Maguin E, Hill KG, et al. Developmental risk factors for youth violence. J Adolesc Health 2000;26:176 – 86. [19] Sege R, Stringham P, Short S, Griffith J. Ten years after: Examination of adolescent screening questions that predict future violence-related injury. J Adolesc Health 1999;24:395– 402. [20] Molnar BE, Browne A, Cerda M, Buka SL. Violent behavior by girls reporting violent victimization: A prospective study. Arch Pediatr Adolesc Med 2005; 159:731–9. [21] National Center for Injury Prevention, CDC. Measuring violence-related attitudes, beliefs, and behaviors among youths: A compendium of assessment tools. Available at: http://www.cdc.gov/ncipc/pub-res/measure.htm. Accessed July 1, 2005. [22] Resnick MD, Bearman PS, Blum RW, et al. Protecting adolescents from harm: Findings from the National Longitudinal Study on Adolescent Health. JAMA 1997;278:823–32. [23] Sieving RE, Beuhring T, Resnick MD, et al. Development of adolescent self-report measures from the National Longitudinal Study of Adolescent health. J Adolesc Health 2001;28:73– 81. [24] Paschall MJ, Flewelling RL. Measuring intermediate outcomes of violence prevention programs targeting African-American male youth: An exploratory assessment of the psychometricproperties of six psychosocial measures. Health Educ Res 1997;12:117–28. [25] Flewelling RL, Paschall MJ, Ringwalt CL. Sage Baseline Survey. Research Triangle Park, NC: Research Triangle Institute, 1993. [26] Orpinas P, Kelder S, Frankowski R, et al. Outcome evaluation of a multicomponent violence-prevention program for middle schools: The students for peace project. Health Educ Res 2000;15:45–58. [27] Orpinas P, Murray N, Kelder S. Parental influences on students’ aggressive behaviors and weapon carrying. Health Educ Behav 1999;26:774 – 87. [28] Goodman R, Meltzer H, Bailey V. The Strengths and Difficulties Questionnaire: A pilot study on the validity of the self-report version. Int Rev Psychiatry 2003;15:173–7. [29] Pediatric symptom checklist. Available at: http://psc.partners.org/psc_ basic.htm. Accessed January 15, 2005. [30] Jellinek MS, Murphy JM. The recognition of psychosocial disorders in pediatric office practice: The current status of the pediatric symptom checklist. J Dev Behav Pediatr 1990;11:273– 8. [31] Nunnally JC, Bernstein IH. Psychometric theory, 3rd edition. New York, NY: McGraw-Hill, 1994. [32] Alpert EJ, Sege RD, Bradshaw YS. Interpersonal violence and the education of physicians. Acad Med 1997;72(Suppl 1):S41–50.