Risk Score for Predicting Adolescent Mental Health Problems Among Children Using Parental Report Only: The TRAILS Study

Risk Score for Predicting Adolescent Mental Health Problems Among Children Using Parental Report Only: The TRAILS Study Huibert Burger, MD, PhD; Marco P. Boks, MD, PhD; Catharina A. Hartman, PhD; Maartje F. Aukes, PhD; Frank C. Verhulst, MD, PhD; Johan Ormel, PhD; Sijmen A. Reijneveld, MD, PhD From the Department of General Practice, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands (Dr Burger); Interdisciplinary Center for Psychopathology and Emotion Regulation, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands (Drs Burger, Hartman, and Ormel); Brain Center Rudolf Magnus, Department of Psychiatry, University Medical Center Utrecht, Utrecht, The Netherlands (Drs Boks and Aukes); Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands (Dr Aukes); Department of Child and Adolescent Psychiatry, Erasmus Medical Center, Rotterdam, The Netherlands (Dr Verhulst); and Department of Health Sciences, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands (Dr Reijneveld) The authors declare that they have no conflict of interest. Address correspondence to Huibert Burger, MD, PhD, Department of General Practice, University Medical Center Groningen, Antonius Deusinglaan 1, 9713 AV Groningen, The Netherlands (e-mail: [email protected]). Received for publication February 26, 2014; accepted July 19, 2014.

ABSTRACT OBJECTIVE: To construct a risk score for adolescent mental

showed good discriminatory power (area under the curve 0.75; 95% confidence interval 0.72–0.78), and validated well. The risk score stratified children in classes of risk ranging from 6.6% to 52.2%. CONCLUSIONS: A risk score based on parent-reported data only and without mental health items accurately estimated the 5-year risk of adolescent mental health problems among children from the general population. Children with high risk may benefit from further monitoring or intervention. The risk score may be particularly suitable when parents want to circumvent an explicit discussion on possible mental health problems of their child.

health problems among children, using parental data only and without potentially stigmatizing mental health items. METHODS: We prospectively derived a prediction model for mental health problems at age 16 using data from parent report on 1676 children aged 11 from the general population. Mental health problems were considered present in the top 15% scores on the combined Achenbach ratings. The model was validated in a separate cohort (n ¼ 336) children. A risk score was constructed for practical application. RESULTS: In the derivation cohort, 248 (14.8%) had mental health problems at follow-up. Predictors in the final model were gender, maternal educational level, family history of psychopathology, math achievement at school, frequently moving house, severe disease or death in the family, parental divorce, and child frustration level. The model was well calibrated,

KEYWORDS: adolescents; mental health; prediction; prevention; risk assessment ACADEMIC PEDIATRICS 2014;14:589–596

WHAT’S NEW

health services,3 and even among those, there is considerable treatment delay.4 Consequently, there is a global interest in the identification of high-risk individuals and the prevention of adolescent psychopathology.1,3,5 Timely intervention can substantially improve outcomes in terms of preventing psychopathology, suffering, criminal behavior, and substance abuse;3,6 further, these interventions are cost effective.7 To support targeted monitoring and possible early intervention, identification of high-risk children is needed. Risk stratification is probably most useful around the age of 11 to 12 years, the beginning of a vulnerable life phase,2 and early interventions are maximally effective around this age.8 Mild symptoms of mental health problems have shown to be predictive of more severe problems and could, next to other risk indicators, be used for risk stratification.9 However, parents may occasionally be reluctant to provide explicit information on potential mental health problems of

Fear of stigma may hamper the assessment of a child’s future mental health. We developed a risk score for children aged 11 indicating the probability of adolescent mental health problems. It requires parental report only and omits mental health items.

APPROXIMATELY 20% OF adolescents at age 16 experience psychopathology accompanied by substantial distress or social impairment.1 At that age, 60% of the anxiety disorders and 20% of the mood and substance-use disorders have commenced.2 These are alarming numbers, given that adolescent psychopathology is associated with poor outcomes, such as school dropout, antisocial behavior, poor sexual health, suicide, and adult psychopathology.3 Nevertheless, only a minority of adolescents with psychopathology—about 20%—is in contact with mental ACADEMIC PEDIATRICS Copyright ª 2014 by Academic Pediatric Association

589

Volume 14, Number 6 November–December 2014

590

BURGER ET AL

their child or to have their child questioned about these problems despite having concerns. Their reluctance may further increase if parents think that symptoms are not overt or not yet even present. Fear of child stigma may play a role here,10 which may partially explain cultural differences in the quality of early identification of psychosocial problems at well-child visits.11,12 Therefore, a tool for risk estimation that omits direct questions about the child’s mental symptoms and that requires parental report only could meet a need. As far as we know, no such instrument has yet been developed. In the present study, we thus aimed to develop a risk score that can aid a parent and professional in estimating the individual 5-year risk of mental health problems at the age of 16 from measures taken at the age of 11, and that can be used in routine, low-risk settings like schools. We aimed to construct a risk score that would not include items on the child’s mental health, would require parental report only, and would be concise. We used data obtained in a large cohort study among preadolescents followed into adolescence from the general population. The risk score was externally validated in a separate but similar cohort. As a benchmark for predictive performance, we used the predictive value of mental health problems at baseline according to the Child Behavior Checklist (CBCL).13 Our analyses therefore included a head-to-head comparison of the predictive performance of our risk score with that of the baseline CBCL.

METHODS POPULATIONS DERIVATION COHORT We derived the risk score using data from the children participating in the TRacking Adolescents’ Individual Lives Survey (TRAILS). TRAILS is a prospective cohort study of Dutch adolescents on adolescent mental health 14 . The Dutch national Central Committee on Research Involving Human Subjects approved the TRAILS study. Sample selection involved all inhabitants born between October 1, 1989, and September 30, 1990, in 5 municipalities in the north of the Netherlands, including urban and rural areas. Of all children approached (n ¼ 3145), 6.7% (n ¼ 211) were excluded because they were incapable of participating as a result of mental retardation or a serious physical illness or handicap, or because no Dutchspeaking parent or parent surrogate was available and the assessments could not be administered in the parents’ language. This resulted in a total of 2934. Finally, 76.0% participated in the baseline assessment (T1; n ¼ 2230; mean age 11.1 years, range 10.0–12.0 years); 10.3% had at least 1 parent born in a non-Western country, of whom 96.4% (n ¼ 2149) were reassessed at the first follow-up (T2) and 1816 (81.4%) at the second follow-up (T3; mean age 16.3 years, range 14.7–18.5 years). The T1 assessments were conducted from March 2001 through July 2002. T3 assessments were conducted on average

ACADEMIC PEDIATRICS

5.2 years (range 3.5–7.2 years) later, from September 2005 through December 2007. At T1, there was no evidence of differences in teacher-rated psychopathology between participants and nonparticipants.14 The present analysis included 1676 adolescents (75.2%) who had complete data on the CBCL at T1 and T3. The mean (standard deviation) CBCL total problems scores at baseline in this sample and in the total TRAILS were nearly identical, ie, 0.24 (0.16) and 0.25 (0.17), respectively (P for difference ¼ .77). We analyzed data from T1 as predictors for the presence of mental health problems at T3. VALIDATION COHORT Validation of our prediction model and risk score was done within a separate clinic-referred cohort of TRAILS. This cohort underwent the same assessments at the same time intervals as the population cohort, but its members had a history of being referred at least once to the child psychiatric outpatient clinic of the University Medical Center Groningen.14 It consisted of 543 children (T1 mean age 11.1 years, range 10.1–12.4 years). Baseline assessments ran from 2004 to 2005; the T3 wave was completed in 2011. Of the 1264 children approached, 543 children (43.0%) responded. Comparisons between the total response group and nonresponse group showed no significant differences in age, age at referral to the clinic, sex distribution, and education.12 The validation cohort included 336 adolescents (61.9%) who participated in the T3 assessment (T3 mean age 15.9 years, range 14.4–17.5 years) and had complete data on the T3 CBCL and on all predictors included in the final model. MEASURES OUTCOME Mental health outcome at follow-up (T3) was assessed using the Achenbach scales: the parent-rated CBCL, the Youth Self-Report (YSR), and an adaptation of the Teacher Checklist of Psychopathology (TCP),13,15 with a time frame of the past 6 months (CBCL and YSR) or past 2 months (TCP). Reports from these 3 sources were combined to obtain a most complete assessment of mental health problems.16 This was accomplished by taking the equally weighted average of the Z-transformed total problems score of the CBCL, YSR, and the TCP in the total population as previously reported.17 The outcome in the present study was defined as a score exceeding the 85th percentile of the combined score. These scores correspond to the borderline clinical or clinical range of mental health problems.18 PREDICTORS Potential predictors were those measures that could be determined using parental report only that were easy and reliable to assess and had a high prior probability to be linked to mental health problems. As already stated, psychopathology scales were not included as predictors in order to make the risk score shorter, easier to apply in school

ACADEMIC PEDIATRICS

RISK SCORE FOR PREDICTING ADOLESCENT MENTAL HEALTH PROBLEMS

settings, and free of stigma. Potential predictors were gender, maternal and paternal educational level, family history of psychopathology, language achievement, math achievement, frequently moving house, divorce, severe disease or death in family (first degree relatives), hospitalization, and a 5-item parental questionnaire on child frustration. Parental educational level was registered as elementary, lower-track secondary, higher-track secondary, senior vocational, or university and coded as 1 to 5 in that order. Family history of psychopathology was assessed with the TRAILS Family History Interview (FHI).19 A family history of internalizing and externalizing psychopathology was defined as a lifetime occurrence in one or both biological parents of depression or anxiety and antisocial behavior or substance abuse, respectively. Language and math achievement was assessed using a single question about the latest school grade and rated as 1 (fail), 2 (fair), 3 (moderate), 4 (good), and 5 (excellent). Frequently moving house was defined as more than once. Child frustration from parent report was assessed using the frustration scale of the Early Adolescent Temperament Questionnaire.20 It comprises 5 items on negative affect and is rated on a 5-point scale ranging from 1 (hardly ever true) to 5 (almost always true). The sum is the total score and was used for the analyses. A detailed list of the questions and their rating is provided in Online Appendix 1. ANALYSIS DATA IMPUTATION Rates of missing values on 4 predictors ranged between 7.6% and 13% in the derivation cohort. To include participants with missing data in the analyses, we performed multiple imputations using an iterative Markov chain Monte Carlo method based on multivariate normal regression under the assumption of data being missing at random (MAR) or missing completely at random (MCAR). The imputation model included all predictor variables as well as the outcome variable. This technique produces more valid results than complete case analysis, overall mean imputation or the missing-indicator method when data are MAR or MCAR.21 We created 10 imputed data sets, and results were combined using Rubin’s rules.22,23 DERIVATION We aimed to create a well-calibrated and discriminating prediction model.24 Because univariable preselection of predictors may yield unstable models,25 the initial logistic regression model included all potential predictors as independent variables. A dichotomous variable indicating the presence of mental health problems at follow-up ($85th percentile of the combined Achenbach scales) was the dependent variable. Odds ratios as measures of association were calculated with 95% confidence intervals (CIs). Interactions with gender were tested for maternal and paternal education, divorce, severe disease or death in family, and child frustration. To retain the strongest independent predictors only, the model was reduced by eliminating predic-

591

tors with P values of greater than .25. Thresholds of significance (alpha) greater than 0.05 are commonly used in prediction modeling to limit bias in the predictor coefficients.26,27 The performance of the reduced model was assessed in terms of calibration and discrimination. Calibration (ie, the agreement between predicted and observed risk of mental health problems) was assessed graphically by plotting the observed risk against deciles of predicted risk. Discriminatory power of the model (ie, its potential to separate children with and without mental health problems at follow-up), was quantified by constructing receiver operating characteristic (ROC) curves and calculating the area under the ROC curve (AUC) as a summary measure. Its value ranges from 0.5 (no discrimination) to 1.0 (perfect discrimination).28 VALIDATION External validation of our risk score was done in the clinic-referred cohort.29 First, we assessed its calibration (ie, the agreement between predicted and observed risks). This was done graphically and by using the HosmerLemeshow test. Second, we assessed its discriminative ability by calculating the relative difference in AUC between the derivation and the validation cohort. Validation was considered inadequate when calibration was poor as assessed graphically, when the Hosmer-Lemeshow goodness of fit test was statistically significant or when the AUC decreased by more than 10%.30 As a way of benchmarking the performance of our model, we compared its predictive performance with that of the total score of the CBCL at T1 in terms of the AUC. CONSTRUCTION OF RISK SCORE A practical risk score based on the logistic regression coefficients was constructed. We first calculated predictor loads by dividing each coefficient by the absolute smallest coefficient in the model, and then rounding it to the nearest integer. Next, we computed a total score for each participant by summing the predictor loads multiplied by the individual predictor values. An offset constant was added to prevent negative values. A score chart was created to facilitate the calculation of the total risk score. Total risk scores were related to the predicted probabilities graphically. To facilitate risk stratification, the total score was divided into 4 categories chosen such that they represented clinically meaningful classes of risk and were still of reasonable size. Predicted and observed risks were compared in these classes for both the derivation and validation cohort. Analyses were conducted using PASW Statistics 18.0 and Stata 11.0.

RESULTS The risk of mental health problems was 1.95 times higher in the clinic-referred validation cohort than in the derivation cohort (Table 1). In both cohorts, potential predictors were associated with mental health problems at follow-up in the expected direction (ie, protective or risk

592

BURGER ET AL

ACADEMIC PEDIATRICS

Table 1. General Characteristics of Study Populations at First Assessment Wave Derivation Cohort (n ¼ 1676)

Characteristic Age, y, mean (range) Female gender, n (%) Maternal education level, n (%) 1—Elementary 2—Lower track secondary 3—Higher track secondary 4—Senior vocational 5—University Paternal education level, n (%) 1—Elementary 2—Lower track secondary 3—Higher track secondary 4—Senior vocational 5—University Family history of internalizing psychopathology, n (%) Family history of externalizing psychopathology, n (%) Language achievement grade, mean (SD) Math achievement grade, mean (SD) Moved house more than once, n (%) Divorce, n (%) Severe disease or death in family, n (%)* Ever hospitalized, n (%) Child frustration from parent report, mean (SD)

Validation Cohort (n ¼ 336)

No Mental Health Mental Health No Mental Health Mental Health Problems at Follow-up Problems at Follow-up Problems at Follow-up Problems at Follow-up (n ¼ 1428) (n ¼ 248; 14.8%) (n ¼ 270) (n ¼ 97; 28.9%) 11.1 (10.0–12.6) 747 (52.3)

11.1 (10.1–12.5) 140 (56.5)

11.1 (10.1–12.4) 71 (29.7)

11.1 (10.2–12.3) 35 (36.1)

63 (4.4) 384 (26.9) 517 (36.2) 336 (23.5) 128 (9.0)

21 (8.5) 90 (36.3) 101 (40.7) 26 (10.5) 10 (4.0)

5 (2.1) 56 (23.4) 102 (42.7) 57 (23.8) 19 (7.9)

0 (0.0) 32 (33.0) 45 (46.4) 17 (17.5) 3 (3.1)

50 (3.9) 307 (24.2) 395 (31.1) 313 (24.7) 204 (16.1) 571 (40.0)

12 (6.2) 62 (31.8) 75 (38.5) 27 (13.8) 19 (9.7) 135 (54.4)

4 (1.7) 48 (20.1) 73 (30.5) 54 (22.6) 28 (11.7) 137 (57.3)

3 (3.1) 25 (25.8) 37 (38.1) 7 (7.2) 5 (5.2) 68 (70.1)

166 (11.6)

56 (22.6)

48 (20.1)

28 (28.9)

3.4 (1.0) 3.3 (1.0) 412 (28.9) 153 (10.7) 516 (36.1) 525 (36.8) 13.6 (3.2)

3.1 (1.1) 2.9 (1.1) 95 (38.3) 45 (18.1) 125 (50.4) 111 (44.8) 15.7 (3.1)

3.1 (1.0) 3.0 (1.1) 92 (38.5) 31 (13.0) 108 (45.2) 122 (51.0) 16.0 (3.5)

2.7 (1.0) 2.6 (1.0) 39 (40.2) 20 (20.6) 57 (58.8) 49 (50.5) 18.9 (2.7)

*First-degree relatives.

increasing) on the basis of scientific literature, with the exception of gender in both cohorts, and hospitalization in the validation cohort only. Table 2 displays the final multivariable prediction model after elimination of nonsignificant predictors: hospitalization, language achievement, and paternal education. None of the interaction terms was retained. The model was well calibrated as shown in Figure 1. The AUC of this model was 0.75 (95% CI 0.72–0.78), which also implies good discriminatory power. Inclusion of duration of follow-up as an independent variable had no effect on

either the regression coefficients or AUC. The discriminatory power for prediction of mental problems from CBCL total problems scores at baseline was identical to that of our prediction model with an AUC of 0.75 (95% CI 0.72–0.78) and similarly shaped ROCs (Fig. 2). The exact predicted probability of future mental health problems using the beta coefficients from the model can be calculated using formula 1, presented below Table 2. More easily, formula 2, using the predictor loads as provided in the last column of Table 2, can be used to calculate the total score for an individual: 3  female sex 

Table 2. Reduced Final Multivariable Model to Predict Mental Health Problems* Predictor

Odds Ratio (95% CI)

P

Regression Coefficient

Predictor Load

Female gender (yes/no) Maternal education (level 1–5) Family history of internalizing psychopathology (yes/no) Family history of externalizing psychopathology (yes/no) Math achievement (grade 1–5) Moved house more than once (yes/no) Severe disease or death in family (yes/no) Divorce (yes/no) Child frustration from parent report (total score)

0.78 (0.58–1.04) 0.73 (0.62–0.85) 1.26 (0.93–1.71) 1.37 (0.92–2.04) 0.78 (0.67–0.91) 1.09 (0.96–1.23) 1.51 (1.12–2.03) 1.28 (0.85–1.94) 1.22 (1.16–1.27)

.09 <.01 .13 .12 <.01 .18 <.01 .24 <.01

0.254 0.316 0.234 0.317 0.246 0.084 0.410 0.249 0.196

3 4 3 4 3 1 5 3 2

CI indicates confidence interval. *Predicted probability ¼ 1/(1 þ exp ((3.318  0.254  female sex  0.316  maternal education level þ 0.234  family history of internalizing psychopathology þ 0.317  family history of externalizing psychopathology  0.246  math achievement grade þ 0.084  moved house more than once þ 0.410  severe disease or death in family  þ 0.249  divorce þ 0.196  child frustration total score))). Risk score ¼ 3  female sex  4  maternal education level þ 3  family history of internalizing psychopathology þ 4  family history of externalizing psychopathology  3  math achievement grade þ moved house twice or more þ 5  severe disease or death in family (first degree relatives) þ 3  divorce þ 2  child frustration total score þ 25.

ACADEMIC PEDIATRICS

RISK SCORE FOR PREDICTING ADOLESCENT MENTAL HEALTH PROBLEMS

Observed risk

0.4 0.3 0.2 0.1 0 0

0.1

0.2

0.3

0.4

Predicted probability Figure 1. Calibration plot of the prediction model in the derivation cohort. Dots represent observed risk by deciles of predicted probability.

4  maternal education level þ 3  family history of internalizing psychopathology þ 4  family history of externalizing psychopathology  3  math achievement grade þ moved house more than once þ 5  severe disease or death in family (first degree relatives)  þ3  divorce þ 2  child frustration total score þ 25. Table 3 shows the chart that may aid in calculating the total risk score. The predicted probability for a given total score can be read from Figure 3. For example, a girl with a mother who followed the lower track in secondary school, who had a family history of internalizing psychopathology, who had a good math achievement (grade 4), and who had a child frustration score of 22 has a total risk score of 49 (3 8 þ 312 þ 44 þ 25) corresponding to an ample 30% risk of mental health problems. Observed risks in categories of the risk score ranged from 6.2% to 48.2% and corresponded well with the model based predictions ranging from 6.6 to 52.2% (Table 4). The girl with the total score of 49 is in the third (“increased risk”) category and therefore is in the 25.2% risk class. Using a cutoff of increased risk or more, one-third (487 þ 54/ 1676) of all children scored positive, and the sensitivity and

Figure 2. Receiver operating characteristic (ROC) curves for comparison of the discriminatory power of the prediction model with the Child Behavior Checklist (CBCL) total problems score at baseline. Areas under the ROC curves were identical for both the prediction model and the CBCL total problems score: 0.75 (95% CI 0.72–0.78).

593

specificity of our risk score at this threshold was 63%: [(130 þ 26)/248] and 73% [(833 þ 302  52  40)/ (1676248)], respectively. External validation in the clinic-referred cohort showed that the overall risk of 28.9% in this population agreed reasonably with the model prediction of 25.1% (Table 4). Also in the 4 risk categories, agreement between predicted and observed risk was good; Figure 4 shows good calibration over the total range of risk. This was confirmed by a nonsignificant Hosmer-Lemeshow test (P ¼ .29). The discriminatory power as defined by the AUC was somewhat higher (0.78; 95% CI 0.72–0.83) in the validation cohort than in the derivation cohort.

DISCUSSION This study yielded a concise yet accurate risk score for children aged 11 indicating the probability of mental health problems by the age of 16. It comprised gender, maternal education, family history of psychopathology, math achievement at school, frequently moving house, severe disease or death in the family, divorce, and child frustration level. Each of these predictors can be determined using parental report only and do not require potentially stigmatizing questions on mental health. The risk score validated well in a clinic-referred cohort. The equal predictive performance of the baseline CBCL suggests that our score is not less accurate than an assessment of existing mental health assessments. Our study has several limitations and strengths. A first limitation may be that defining the top 15% as experiencing mental health problems may be too lenient, as may be our exclusion of children with serious mental retardation or serious physical illness or handicap, or with no Dutch-speaking parent or parent surrogate. Therefore, further studies with full clinical diagnosis are welcomed. Second, the response rate in the validation cohort was rather low, which may have caused some selection bias. However, we assume that such bias is unlikely to explain the good validation of our model in this population. In this respect, it is also reassuring that the response group and the nonresponse group showed no significant differences in age, age at referral to the clinic, sex distribution, and education. The good validation of our model in a separate cohort is a strength, as many risk scores replicate poorly.31 As in this cohort children had been referred to psychiatric services at least once in their life, the validation results suggest that the score may be used not only in the general population but also in higher-risk settings. However, ethnic minorities living in the Netherlands are underrepresented in the TRAILS study, and we were unable to validate our risk score cross-culturally. Although the predictive performance of our risk score was good, it was not perfect. If a threshold of increased risk or high risk is deemed to test positive, still 37% (1  sensitivity) of all cases of mental health problems at age 16 are left undetected and 27% (1  specificity) classify as falsely positive. The consequences hereof are

594

BURGER ET AL

ACADEMIC PEDIATRICS

Table 3. Score Chart for Calculating Risk Score* Predictor Offset Gender Maternal education level

Family history of internalizing psychopathology Family history of externalizing psychopathology Math achievement Moved house more than once Severe disease or death in family† Divorce Child frustration from parent report Total score

Value 25 Male 0 Elementary 4 No 0 No 0 Grade 1 3 No 0 No 0 No 0 Score (total) *2

Female 3 Lower track secondary 8 Yes 3 Yes 4 Grade 2 6 Yes 1 Yes 5 Yes 3

Score

Higher track secondary 12

Senior Vocational

University

16

20

Grade 3 9

Grade 4 12

Grade 5 15

*Risk score equals sum of scores in rightmost column (total score) and can be related to predicted 5-year risk of mental health problems using Figure 3. †First-degree relatives.

clearly dependent on many factors and are beyond the scope of this study. Nevertheless, our concise score without mental items was equally predictive as a comprehensive baseline mental assessment using the CBCL. Notably, it could stratify children in classes of diverging risk ranging from 6.6% to 52.2%, which is, in our view, clinically useful. It may have consequences in terms of further monitoring or perhaps even intervention. The current study builds on previous similar research, as follows. To assess the probability of current, ie, prevalent, mental health problems, many well-validated screening tools are available to date, such as the SDQ32 and the PSC.33 As to forecasting mental problems, we previously created a model for psychopathology in children aged 11,34 and we studied the predictive value of sociodemographic characteristics, family characteristics, and recent life events using cross-sectional data.35

As far as we know, this is the first risk score to be applied to children aged around 11 that yields personalized risks of future adolescent mental health problems. The application of the present score allows for risk predictions without the need to question the child or to explicitly query the parents for mental problems in their offspring. Several important previous studies addressed the prediction of specific psychopathology, such as antisocial personality disorder,36 disruptive behavior,37 or depression.38 We may contribute to the field by creating a risk score that can aid in predicting an increased risk of any mental health problem, the nature of which must be assessed using further assessments. The predictors in our risk score should not be interpreted as causal factors for mental health problems per se. The data analytic strategy was aimed at identifying a set of variables that jointly had high predictive performance rather than at unraveling the etiology of mental problems. Yet

0.7

Observed risk

0.6 0.5 0.4 0.3 0.2 0.1 0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Predicted probability Figure 3. Relationship between total risk score and predicted 5year risk of mental health problems.

Figure 4. Calibration plot of the prediction model in the validation cohort. Dots represent observed risk by deciles of predicted probability.

ACADEMIC PEDIATRICS

RISK SCORE FOR PREDICTING ADOLESCENT MENTAL HEALTH PROBLEMS

595

Table 4. Predicted and Observed 5-Year Risk of Mental Health Problems According to Risk Score Categories in the Derivation and Validation Cohort Psychopathology Risk Score Category (Score) 1. Low risk (<35) 2. Medium risk (35–39) 3. Increased risk (40–54) 4. High risk ($55) Total

Derivation Population

Validation Population

Mean Predicted Risk

Observed Risk

Mean Predicted Risk

Observed Risk

6.6% 14.0% 25.2% 52.2% 14.8%

52/833 (6.2%) 40/302 (13.2%) 130/487 (26.7%) 26/54 (48.2%) 248/1676 (14.8%)

7.6% 14.7% 28.5% 56.1% 25.1%

4/81 (4.9%) 9/50 (18.0%) 55/160 (34.4%) 29/45 (64.4%) 97/336 (28.9%)

our findings are in line with previous research. For example, severe disease or death in the near family and low educational attainment are established risk factors for psychopathology.39–41 The information needed for our score can be easily obtained from parents in many low-risk settings, eg, school, preventive child health care, or general practice. In the latter 2 settings, we expect that most data to be fed into the risk score such as on parental mental health will already be in the child’s file. The lack of the necessity of obtaining data from the child directly by interview or questionnaire, or of explicitly asking parents about their child’s mental health problems, makes our score particularly appropriate for situations in which fear of stigma or other personal considerations could hamper proper risk profiling.11,12 Nevertheless, if the child’s file does not yet contain parental mental health data and parents need to be directly questioned for this in order to complete the risk score, they may be reluctant to be forthcoming as a result of fear of stigma. Although we constructed a risk score without potentially stigmatizing mental health items, we have no proof that our risk score is less stigmatizing than an any other instrument to assess the risk of developing mental health problems. This issue may be the subject of future research. The assignment of a certain risk or risk category to a child may have various consequences. Depending on the magnitude of the predicted risk and other factors, such as personal preferences, these consequences may be reassurance, further monitoring or testing, increased support, an offer of resources, or other interventions. Although the risk score is brief and easy to use, it can only be used as a general indicator of increased risk for mental health problems. The risk estimate could serve as a rough guideline only and by no means should be interpreted in any strict or final sense. Targeted further evaluations including teacher ratings, more comprehensive parental interview, and assessment of the child himself or herself will be needed to identify specific problems. Future research needs to demonstrate the practical feasibility of our risk score in daily practice, the impact of using the risk score on the decision making of its users and on health outcomes of those assessed, as well as the costeffectiveness of implementation of the score. In addition, cross-cultural validation studies are needed. This is pivotal because our score was developed in a sample of the general population without a substantial proportion of children of non-Western origin, and the known cultural differences

in the identification of psychosocial differences at wellchild visits.11,12

CONCLUSIONS Our prospective study shows that a feasible risk score for children of age 11 can accurately estimate the 5-year risk of future adolescent mental health problems. The score requires parental information only and does not include potentially stigmatizing mental health items. It may help in targeting monitoring or intervention strategies to highrisk children. ACKNOWLEDGMENTS This research is part of the TRacking Adolescents’ Individual Lives Survey (TRAILS). Participating centers of TRAILS include various departments of the University Medical Center and University of Groningen, the Erasmus University Medical Center Rotterdam, the University of Utrecht, the Radboud Medical Center Nijmegen, and the Parnassia Bavo group, all in the Netherlands. TRAILS has been financially supported by various grants from the Netherlands Organization for Scientific Research NWO (Medical Research Council program grant GB-MW 94038-011; ZonMW Brainpower grant 100-001-004; ZonMw Risk Behavior and Dependence grants 60-60600-98-018 and 60-60600-97-118; ZonMw Culture and Health grant 261-98-710; Social Sciences Council mediumsized investment grants GB-MaGW 480-01-006 and GB-MaGW 48007-001; Social Sciences Council project grants GB-MaGW 457-03-018, GB-MaGW 452-04-314, and GB-MaGW 452-06-004; NWO large-sized investment grant 175.010.2003.005); the Sophia Foundation for Medical Research (projects 301 and 393), the Dutch Ministry of Justice (WODC), the European Science Foundation (EuroSTRESS project FP006), and the participating universities. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the article.

SUPPLEMENTARY DATA Supplementary data related to this article can be found online at http://dx.doi.org/10.1016/j.acap.2014.07.006.

REFERENCES 1. Merikangas KR, He JP, Burstein M, et al. Lifetime prevalence of mental disorders in US adolescents: results from the National Comorbidity Survey Replication—Adolescent Supplement (NCS-A). J Am Acad Child Adolesc Psychiatry. 2010;49:980–989. 2. Kessler RC, Berglund P, Demler O, et al. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62: 593–602. 3. Patel V, Flisher AJ, Hetrick S, McGorry P. Mental health of young people: a global public-health challenge. Lancet. 2007;369: 1302–1313. 4. Wang PS, Berglund P, Olfson M, et al. Failure and delay in initial treatment contact after first onset of mental disorders in the National

596

5.

6.

7.

8.

9.

10.

11.

12.

13. 14.

15. 16.

17.

18.

19.

20.

21.

BURGER ET AL Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62: 603–613. Jensen PS, Goldman E, Offord D, et al. Overlooked and underserved: “action signs” for identifying children with unmet mental health needs. Pediatrics. 2011;128:970–979. Toumbourou JW, Stockwell T, Neighbors C, et al. Interventions to reduce harm associated with adolescent substance use. Lancet. 2007;369:1391–1401. Lynch FL, Hornbrook M, Clarke GN, et al. Cost-effectiveness of an intervention to prevent depression in at-risk teens. Arch Gen Psychiatry. 2005;62:1241–1248. Durlak JA, Wells AM. Evaluation of indicated preventive intervention (secondary prevention) mental health programs for children and adolescents. Am J Community Psychol. 1998;26:775–802. Cuijpers P, Smit F. Subthreshold depression as a risk indicator for major depressive disorder: a systematic review of prospective studies. Acta Psychiatr Scand. 2004;109:325–331. Heflinger CA, Hinshaw SP. Stigma in child and adolescent mental health services research: understanding professional and institutional stigmatization of youth with mental health problems and their families. Adm Policy Ment Health. 2010;37:61–70. Crone MR, Bekkema N, Wiefferink CH, Reijneveld SA. Professional identification of psychosocial problems among children from ethnic minority groups: room for improvement. J Pediatr. 2010;156: 277–284. Theunissen MH, Vogels AG, Reijneveld SA. Work experience and style explain variation among pediatricians in the detection of children with psychosocial problems. Acad Pediatr. 2012;12:495–501. Achenbach TM. Manual for the Child Behavior Checklist/4-18 and 1991 Profile. Burlington, Vt: University of Vermont; 1991. Huisman M, Oldehinkel AJ, de Winter A, et al. Cohort profile: the Dutch “TRacking Adolescents’ Individual Lives” Survey—TRAILS. Int J Epidemiol. 2008;37:1227–1235. Achenbach TM. Manual for the Youth Self-Report and 1991 Profile. Burlington, Vt: University of Vermont; 1991. Van der Ende J, Verhulst FC. Informant, gender and age differences in ratings of adolescent problem behaviour. Eur Child Adolesc Psychiatry. 2005;14:117–126. Amone-P’Olak K, Ormel J, Oldehinkel AJ, et al. Socioeconomic position predicts specialty mental health service use independent of clinical severity: the TRAILS study. J Am Acad Child Adolesc Psychiatry. 2010;49:647–655. Vogels AG, Crone MR, Hoekstra F, Reijneveld SA. Comparing three short questionnaires to detect psychosocial dysfunction among primary school children: a randomized method. BMC Public Health. 2009;9:489. Ormel J, Oldehinkel AJ, Ferdinand RF, et al. Internalizing and externalizing problems in adolescence: general and dimension-specific effects of familial loadings and preadolescent temperament traits. Psychol Med. 2005;35:1825–1835. Oldehinkel AJ, Hartman CA, De Winter AF, et al. Temperament profiles associated with internalizing and externalizing problems in preadolescence. Dev Psychopathol. 2004;16:421–440. Donders AR, van der Heijden GJ, Stijnen T, Moons KG. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol. 2006;59:1087–1091.

ACADEMIC PEDIATRICS 22. Schafer JL. Multiple imputation: a primer. Stat Methods Med Res. 1999;8:3–15. 23. Marshall A, Altman DG, Holder RL, Royston P. Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Med Res Methodol. 2009;9:57. 24. Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338:b604. 25. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15:361–387. 26. Steyerberg EW, Eijkemans MJ, Habbema JD. Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol. 1999;52:935–942. 27. Chatfield C. Model uncertainty, data mining, and statistical inference. R Stat Soc Ser A. 1995;158:419–466. 28. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143: 29–36. 29. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605. 30. Steyerberg EW, Borsboom GJ, van Houwelingen HC, et al. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med. 2004;23:2567–2586. 31. Maguire JL, Kulik DM, Laupacis A, et al. Clinical prediction rules for children: a systematic review. Pediatrics. 2011;128:e666–e677. 32. Goodman R. The Strengths and Difficulties Questionnaire: a research note. J Child Psychol Psychiatry. 1997;38:581–586. 33. Jellinek MS, Murphy JM, Little M, et al. Use of the Pediatric Symptom Checklist to screen for psychosocial problems in pediatric primary care: a national feasibility study. Arch Pediatr Adolesc Med. 1999;153:254–260. 34. Jaspers M, De Winter AF, de Meer G, et al. Early findings of preventive child healthcare professionals predict psychosocial problems in preadolescence: the TRAILS study. J Pediatr. 2010;157:316–321. 35. Harland P, Reijneveld SA, Brugman E, et al. Family factors and life events as risk factors for behavioural and emotional problems in children. Eur Child Adolesc Psychiatry. 2002;11:176–184. 36. Hill J. Early identification of individuals at risk for antisocial personality disorder. Br J Psychiatry. 2003;44(suppl):S11–S14. 37. Burt KB, Hay DF, Pawlby S, et al. The prediction of disruptive behaviour disorders in an urban community sample: the contribution of person-centred analyses. J Child Psychol Psychiatry. 2004;45: 1159–1170. 38. Pitzer M, Jennen-Steinmetz C, Esser G, et al. Prediction of preadolescent depressive symptoms from child temperament, maternal distress, and gender: results of a prospective, longitudinal study. J Dev Behav Pediatr. 2011;32:18–26. 39. Ragan PV, McGlashan TH. Childhood parental death and adult psychopathology. Am J Psychiatry. 1986;143:153–157. 40. Luecken LJ. Attachment and loss experiences during childhood are associated with adult hostility, depression, and social support. J Psychosom Res. 2000;49:85–91. 41. Maccabe JH. Population-based cohort studies on premorbid cognitive function in schizophrenia. Epidemiol Rev. 2008;30:77–83.