- Email: [email protected]
Routine outcome measurement in youth mental health: A comparison of two clinician rated measures, HoNOSCA and HoNOS
Psychiatry Research 200 (2012) 884–889
Contents lists available at SciVerse ScienceDirect
Psychiatry Research journal homepage: www.elsevier.com/locate/psychres
Routine outcome measurement in youth mental health: A comparison of two clinician rated measures, HoNOSCA and HoNOS Peter Brann a,d,n, Monique Alexander b, Tim Coombs c a
Eastern Health Child and Youth Mental Health Service, Australia Eastern Health Adult Mental Health Service, Australia c Australian Mental Health Outcomes and Classification Network, Australia d Monash University School of Psychology and Psychiatry, Australia b
a r t i c l e i n f o
a b s t r a c t
Article history: Received 14 March 2012 Received in revised form 29 June 2012 Accepted 12 July 2012
Mental health services engagement in routine outcome measurement has created a demand for a coherent infrastructure. Two clinician instruments for adolescents (HoNOSCA) and adults (HONOS) are used across many countries. However, the increasing emphasis on youth suggests protocols with historically driven age divisions may obscure outcomes. The current study examines these instruments’ congruence with regard to youth mental health. Members of national mental health expert panels rated four vignettes before discussing perceived strengths and weaknesses. The instruments were strongly correlated and HoNOSCA resulted in more severe symptom scores. Most subscales and scales correlated as predicted with some important exceptions. ‘Problems with family relationships’, tracked by HoNOSCA, did not correlate with its HoNOS counterpart. Qualitative feedback indicated using the HoNOSCA scale ‘School attendance’ for vocational attendance would improve its applicability to young people. The instruments have a strong relationship. While either could be used, HoNOS will underestimate symptom severity in youth. The importance of family relationships for young people suggests that HoNOSCA is preferable. While sited in the Australian context, these findings should be applicable to other countries using these instruments, and should interest services considering the continuity of youth presentations and their outcomes. Crown Copyright & 2012 Published by Elsevier Ireland Ltd. All rights reserved.
Keywords: Young adult Outcome measures Mental health services HoNOSCA HONOS
1. Introduction With a substantial proportion of the mental health burden falling on those aged between 12 and 25, there have been calls for services that specifically target this population (McGorry et al., 2007). It has long been argued that neither child nor adult services adequately meet the needs of youth (McGorry, 1998) while others have argued this is not an inherent limitation of this service configuration (Birleson and Vance, 2008). Irrespective of configuration concerns, development does not cease at 18 and there is great continuity in the presentation of consumers over the adolescent to young adult age range. Adult services have focussed on a narrower range of disorders than affect the majority of young people and the newer youth models can be expected to see a broader range of presentations and hence more domains of symptoms and functioning may be implicated (Birleson, 2009).
n Corresponding author: Eastern Health & Monash University, Child Youth Mental Health Service & School of Psychology and Psychiatry, Wundeela Centre, 21 Ware Cres, East Ringwood, Victoria 3135, Australia. Tel.: þ61 03 9870 9788; fax: þ 61 03 9870 7973. E-mail address: [email protected] (P. Brann).
Within Australia, the latest national mental health plan clearly calls for the expansion of youth mental health services (Commonwealth of Australia, 2009). Many countries are engaged in embedding outcome measurement in routine practice (Trauer, 2010). Monitoring and evaluating outcomes for individuals with psychiatric disorders are crucial aspects of service delivery. Effective services help people in a range of areas including quality of life, recovery, symptoms and functioning (Bickman et al., 1998). Standardised outcome measures provide a valuable aid in assessing and evaluating treatment progress as well as providing a common language that is transferrable across mental health services (Slade et al., 1999). The national outcomes and casemix collection (NOCC) was introduced in Australia to provide a suite of measures that support clinical practice and comparisons across services and different consumer populations. The health of the national outcome scales (HoNOS) are a family of instruments first developed in the United Kingdom (Wing et al., 1998) to allow clinicians to routinely map common problems for consumers with a mental illness. Originally designed to assess adults with a severe mental illness, variants were developed for children and adolescents (HoNOSCA) (Gowers et al., 1999b) and for older persons (HoNOS 65þ) (Burns et al., 1999). This family form the key clinician measures
0165-1781/$ - see front matter Crown Copyright & 2012 Published by Elsevier Ireland Ltd. All rights reserved. http://dx.doi.org/10.1016/j.psychres.2012.07.010
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
of problem severity in the NOCC suite (Department of Health and Ageing, 2003). All three instruments have been found to possess sufficient psychometric properties to warrant routine use in clinical practice (Brann, 2010; Pirkis et al., 2005a; Trauer, 2010), including inter-rater reliability (Hanssen-Bauer et al., 2007a; Pirkis et al., 2005b), and have been implemented in a range of countries (Adamis et al., 2011; Hanssen-Bauer et al., 2007b; Trauer, 2010). The development of youth mental health services is occurring across Australia (Callaly et al., 2011; Scott et al., 2009), however, the national protocol has remained silent on which of the HoNOS family of measures should be used in these types of services (Department of Health and Ageing, 2009). Currently, if a young person crosses the age boundary between child and adolescent mental health (CAMHS) and adult services, the national protocol requires the closing of a CAMHS episode and the opening of a new adult episode of care. Although this artificial closure and opening of treatment episodes can be coded, it is cumbersome and unless acknowledged will erroneously underestimate the effectiveness of treatment for those continuing across the age boundary. The arbitrary truncation of CAMHS service delivery at 18 years has been criticised as a process which unnecessarily fragments the continuing mental health needs of teenagers and young adults (Patel et al., 2007) and the cessation of its outcomes protocol mirrors this discontinuity. Despite the increased emphasis in youth-oriented services, there is no consensus on whether HoNOSCA or HoNOS measures should be used by clinicians in these youth services. As these two instruments have some differences in content, it may be problematic to compare one measure with the other. There has been little evaluation of how the different measures may perform in describing the presentation of consumers of youth services. The correlation between HoNOSCA and other adolescent instruments has been investigated (Yates et al., 2006). There has also been work to determine the correlation between HoNOS and the version for older persons, HoNOSCA65þ (Turner, 2004). However, a literature search conducted using Ovid Medline and Ovid PsycInfo failed to find any publication examining the relationships between HoNOS and HoNOSCA. Understanding the relationship between HoNOSCA and HoNOS is an important step in evaluating the applicability of these outcome measures across the lifespan and the suitability of these measures for use with the youth age group. However, as correlation is independent of an instrument’s metric (Tabachnick and Fidell, 1996), two instruments may correlate perfectly while indicating differing levels of severity. As the severity of symptoms may be important in the
885
future allocation of resources, differences between the extent of severity indicated by the two instruments will also be examined. HoNOS contain 12 scales across four subscales of behaviour, impairment, symptoms and social functioning. HoNOSCA are conceptually similar with 13 clinical scales and similarly named subscales, however, some HoNOSCA items are more specific to the adolescent developmental period, including an assessment of peer relationships, family relationships and school attendance (Garralda and Yates, 2000). Two additional HoNOSCA scales relate to information about the disorder and its treatment, have no HONOS counterpart and will be excluded from further consideration. Through the analysis of case study vignettes, the present study will evaluate the strength of association between HoNOSCA and HoNOS, and the benefits and limitations of each measure in capturing relevant outcomes for young people.
2. Method 2.1. Participants The Australian government created national expert advisory panels to provide advice on the implementation and the development of mental health information including routine outcome measurement. Panel members are experienced practitioners, academics, managers and the panels include carers and consumers. 2.2. Procedure Members of both the child and adolescent and the adult national mental health information development expert advisory panels were invited to participate. All members present at regular meetings consented to participate and focussed refresher training was provided by one author (T.C.). HoNOSCA and HoNOS glossaries were provided (Gowers et al., 1999a; Wing et al., 1999) including the HoNOSCA implementation guideline noting that if school was not the key vocational setting for that child or adolescent (e.g., child care and employed), then the key vocational setting for that young person should be used as the target to rate attendance (Brann et al., 2003). Four written case vignettes were selected from an international reliability study (HanssenBauer et al., 2007b) and the ages modified from adolescence to young adulthood (18, 19, 21 and 22 years). The original vignettes can be found at http://amhocn.org/ training-service-development. Members were asked to rate these four vignettes about young people using both HoNOSCA and HoNOS with counterbalancing of instrument and vignette order both within and between participants respectively. Following ratings, five questions were posed to the group: important mental health symptoms for this age group, symptom areas that HONOS/HoNOSCA allowed to be identified, and the symptom areas difficult to be identified with HONOS/HoNOSCA. The discussion was facilitated by one of the authors (P.B.) to ensure all participants were able to contribute. Themes from this discussion were noted by the panel secretariat. All results were de-identified by the secretariat before forwarding to the authors (P.B. and M.A.) for data analysis. The results of those with purely consumer, carer or policy development roles on the panel were excluded from quantitative analyses of the scales
Table 1 HoNOS and HoNOSCA Scales. Scale HoNOSCA
HoNOS
1
Problems with disruptive, antisocial or aggressive behaviour
2 3 4 5 6 7 8 9 10 11 12 13 14 15
Problems with overactivity, attention or concentration Non-accidental self-injury Problems with alcohol, substance or solvent misuse Problems with scholastic or language skills Physical illness or disability problems Problems associated with hallucinations, delusions or abnormal perceptions Problems with non-organic somatic symptoms Problems with emotional and related symptoms Problems with peer relationships Problems with self-care and independence Problems with family life and relationships Poor school attendance Problems with knowledge or understanding about the nature of the child or adolescent’s difficulties Problems with lack of information about services or management of the child or adolescent’s difficulties
Overactive, aggressive, disruptive or agitated behaviour Non-accidental self-injury Problem drinking or drug-taking Cognitive problems Physical illness or disability problems Problems associated with hallucinations and delusions Problems with depressed mood Other mental and behavioural problems Problems with relationships Problems with activities of daily living Problems with living conditions Problems with occupation and activities
886
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
although their views are included in the qualitative findings. HoNOS and HoNOSCA scales are each scored from 0 (No Problem) to 4 (Severe Problem) (Table 1). The sum of all HoNOS scales and the first 13 HoNOSCA scales are used to calculate the total score with the maximum being 48 and 52 respectively. There are subscales within both instruments; nominally titled ‘Behaviour’, ‘Impairment’, ‘Symptoms’ and ‘Social’. Higher scores indicate more severe problems. Quantitative analysis was conducted with SPSS (version 17) supplemented with syntax for establishing weighted kappa (http:// www-304.ibm.com/support/docview.wss?uid=swg21477357, last accessed 6/6/2012). 2.3. Statistical analyses Previous studies with these instruments have used both parametric and nonparametric analyses. In line with Kisely et al. (2007), the instruments’ total scores as well as the subscales, were treated as parametric and individual scales as nonparametric. Pearson correlations were used to examine the relationship between HoNOSCA and HoNOS scales as rated across the four vignettes. Responses from the non-clinical raters (n¼ 5) were excluded as were HoNOSCA scales 14 and 15 which focus on knowledge and information. Strength of association between the instruments was assessed with Pearson and Spearman correlations accordingly. Correlations were deemed significant at a ¼0.01 (two-tailed). Differences between mean scores for the totals and subscales were examined with t-tests. Where scales between the instruments have highly similar content, correlations were supplemented with descriptions of cross-tabulated dichotomous and absolute agreement, and weighted kappa (Agresti, 2003). Reported weighted kappa uses the squared distance between categories method. From the CAMHS panel, eight members were included in the quantitative analysis with a total of 32 ratings with HoNOSCA and 32 ratings with HONOS. The adult panel had five members with 18 HoNOSCA and 17 HONOS ratings included in the analysis. Any difference between vignettes or raters was not relevant to the purposes of this study and consequently between-vignette or between-rater comparisons were not conducted. To adjust for the difference in the instruments’ ranges, scores were converted to percentages of the maximum possible before t-tests were used to examine differences. It was hypothesised that the two instruments would correlate significantly and that the extent of this would vary with the degree of overlap between scales. It was also hypothesised that the instruments would indicate similar levels of severity.
3. Results 3.1. Total Scores and subscales The mean total scores for HoNOSCA and HoNOS were 22.1 (S.D. ¼8.5) and 16.5 (S.D. ¼4.9) respectively. Scores ranged from 8 to 40 using HoNOSCA and from 8 to 28 using HoNOS. As a percentage of the maximum possible score, the mean total was 43% (S.D.¼16%) and 35% (S.D. ¼10%) for HoNOSCA and HoNOS respectively with the ranges being from 15% to 77% for HoNOSCA and from 17% to 58% using HoNOS. HoNOSCA produced significantly higher total scores as a percentage of the maximum possible (t¼5.91, d.f. ¼1,48, p o0.001) indicating a higher estimate of severity than HoNOS. Correlations of total and subscale scores are Pearson. The total scores on HoNOSCA and HoNOS were found to have a significant and strong correlation of 0.85 (p o0.001). Young people found to be elevated with one instrument will be elevated on the other. The concept of symptom severity does appear to have substantial commonality with the two instruments. With one exception, the subscales on each instrument were only significantly correlated with their counterpart and no other subscale. HoNOSCA ‘Behaviour’ was significantly correlated with HoNOS ‘Behaviour’ (r ¼0.95, p o0.01), ‘Impairment’ with
‘Impairment’ (r ¼0.54, p o0.01) and ‘Symptoms’ with ‘Symptoms’ (r ¼0.43, p o0.01). While HoNOSCA ‘Social’ was significantly correlated with HoNOS ‘Social’ (r¼0.65, po0.01), it was also significantly correlated with the HoNOS ‘Behaviour’ subscale (r ¼0.64, p o0.01). All HoNOSCA subscale scores were significantly greater than the HoNOS counterparts with the exception of ‘Symptoms’. The mean scores for HoNOSCA and HoNOS respectively on each subscale, expressed as a percentage of the maximum possible score were: Behaviour (34% S.D. ¼28%; 27% S.D. ¼26%; t¼ 6.09, d.f.¼1,49, po0.001), Impairment (40% S.D. ¼ 19%; 25% S.D. ¼17%; t¼5.98, d.f.¼1,49, p o0.001), Symptoms (46% S.D. ¼ 19%; 47% S.D. ¼15%; t ¼ 0.39, ns), and Social (50% S.D. ¼23%; 36% S.D. ¼13%; t ¼5.47, d.f. ¼1,49, p o0.001). 3.2. Scales Scales were grouped into those with similar content, substantial content overlap, some overlap and those with no apparent counterpart. As individual scales could be expected to have a stronger relationship than totals or subscales, only those Spearman correlations significant at 0.001 are reported. As this is a stringent criteria, correlations of those scales with no apparent counterpart where the associated probability was less than 0.01 will also be reported. Overall, strong correlations were found for all scales with highly similar content overlap (Table 2). With scales with highly similar content, it is sensible to also examine both the absolute level of agreement (i.e., is a ‘3’ on HoNOSCA accompanied by a ‘3’ on HONOS) and the clinical agreement (i.e., are both scales rated as clinically significant [2, 3, or 4]). With ‘Non-accidental selfinjury’, there was complete agreement on the clinical significance of the ratings between instruments and only 2% of ratings differed on the absolute rating. ‘Substance misuse’ also had complete agreement on clinical significance and only 14% disagreement on the absolute rating. With Physical illness, 4% of ratings disagreed on clinical significance, while there was 16% disagreement on the absolute rating. With peer relationships and relationships, 8.1% differed on clinical significance while 40.7% of ratings differed on the absolute rating. With ‘Self-care’ and ‘Daily living’, 8.1% of ratings differed on clinical significance while 34.6% differed on the absolute rating. Weighted kappa (Table 2) echoed this finding of both strong association and agreement. Scales with substantial or some content overlap also had significant correlations (Table 3, Table 4). HoNOS scale 1 (‘Overactivity, aggression and agitation’) overlaps two HoNOSCA scales (scale 1: ‘Disruptive, antisocial, aggressive’; and scale 2: ‘Overactivity, attention, concentration’) and a significant correlation was found between each of these scales with the correlation being smaller with ‘Overactivity’. HoNOSCA scale 9 (emotional and related symptoms) focuses on both depressive and anxiety symptoms while HoNOS scale 7 (depressed mood) is far more circumscribed and a smaller correlation was obtained compared with scales with greater overlap. Generally, scales with no apparent counterpart had small correlations (Table 5). Associated probabilities are also reported
Table 2 Scales with highly similar content. HoNOSCA scale
HoNOS scale
Spearman’s rho
Weighted kappa
3. Non-accidental self-injury. 4. Alcohol, substance, solvent misuse 6. Physical illness or disability problems. 10. Problems with peer relationships. 11. Problems with self-care and independence.
2. Non-accidental self-injury 3. Problem drinking or drug-taking 5. Physical illness or disability problems 9. Problems with relationships. 10. Problems with activities of daily living.
0.99 0.99 0.87 0.80 0.60
0.99 0.92 0.84 0.75 0.65
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
887
Table 3 Scales with substantial content overlap. HoNOSCA scale
HoNOS scale
Spearman’s rho
1. Disruptive, antisocial, or aggressive behaviour. 2. Problems with overactivity, attention or concentration. 7. Problems associated with hallucinations, delusions or abnormal perceptions.
1. Overactive, aggressive, disruptive, agitated 1. Overactive, aggressive, disruptive, agitated 6. Problems associated with hallucinations and delusions
0.92 0.72 0.84
Previously, carers felt they had not focussed so clearly on the clinicians’ perspective as conveyed in these instruments.
Table 4 Scales with some content overlap. HoNOSCA scale
HoNOS scale
Spearman’s rho
9. Problems with emotional and related symptoms.
7. Problems with depressed mood.
0.58
4. Discussion
for these correlations. The key exception to this was ‘Poor school attendance’ which correlated strongly with ‘Overactive, aggressive’, ‘Non-accidental self-injury’ and ‘Problem drinking or drugtaking’ on HoNOS. 3.3. Qualitative findings The panels indicated that the important mental health symptoms for young people included self-harm, substance use, agitation, aggression, hallucinations, delusions, depression, anxiety, relationship difficulties and role functioning. HoNOS were seen as particularly able to identify hallucinations without the confounding of abnormal perceptions that occurs in HoNOSCA. The panels believed HoNOS were better able to identify issues with living conditions. The adult panel thought that HoNOS may be more useful with mental illnesses for those at the upper end of the youth age bracket. Conversely that panel thought HoNOSCA were more useful at the lower end of the youth bracket. Both groups perceived that HoNOSCA enabled easier identification of both family and peer relationships and that these are distinct and important areas of young peoples wellbeing. The CAMHS group also perceived HoNOSCA as more useful in symptoms of eating, emotional, attention and concentration difficulties. The adult panel agreed eating symptoms were not adequately rated with HoNOS. While HoNOSCA scales 14 and 15 have not been analysed, both groups were concerned about the diminishing involvement of parents with older youth and the range of barriers to their understanding of their offspring’s problems and treatment. The adult panel were clear that HoNOSCA scale 13 needed vocational as well as ‘School attendance’ to be explicitly included in the glossary to render this scale broadly useful for young people. HoNOS were seen as limited in its attention to family, problematic in its confounding of different symptoms (e.g., ‘Aggression and overactivity’) and requiring greater clarification of scale 8 (‘Other mental health problems’). On the instruments themselves, the adult panel emphasised the importance of not dissecting the instruments to create a youth version. They suggested that a choice be made available so clinicians could use either instrument depending on the salient issues of the young person. It was also suggested that issues of functioning be examined for this age group with existing or new measures. A valuable, though tangential to the study, discussion focussed on the utility of the information contained in these clinician instruments for carers and policy makers. The panels’ carers and policy makers discussed how management could both respond to ongoing service difficulties in treating these symptomatic areas, and share that information with consumers and carers.
The routine outcome collection protocol needs to be modified as a result of the emerging emphasis on youth-oriented mental health services. The current study explored how the key clinician rated measures functioned when directed at youth mental health vignettes. With regard to the two overarching hypotheses, there was support for a strong relationship between the two instruments with the obtained correlation strength varying as the scales overlap varied. However, the instruments did not indicate equivalent levels of severity. Overall, the strong relationship between the total scores is encouraging. Both appear to be tapping a similar construct of symptom severity even if greater severity will be indicated by HoNOSCA. This strong relationship was also apparent with the subscales where there was significant commonality between the constructs but again with HoNOSCA indicating greater severity than HoNOS. The symptom subscale was the exception indicating similar levels of severity with both instruments and even though HoNOSCA has only one extra item, the other three subscales all had significantly higher scores as a percentage of the maximum. This suggests that the HoNOSCA scales and their glossaries may be more inclusive of a broader range of mental health difficulties. For example, HoNOS blend overactivity with aggression while HoNOSCA allow for disruptive and aggressive behaviours to be rated separately to overactivity or attention problems. The narrative discussion reinforced the view that HoNOSCA applied to a broader range of symptoms. The difference between the instruments in severity may be relevant if services targeted a certain level of symptom severity. Depending on the ratings of older adults with HoNOS, it seems possible that HoNOS will present the broader range of problems, intended to be treated in a youth mental health service, as less severe than the target disorders HoNOS were oriented towards (e.g., schizophrenia). This could lead to a relative downplaying of the non-psychotic disorders. If activity based (Department of Health and Ageing, 2011) or insurance funding incorporates severity considerations, then the rated level of severity indicated by a scale may be an important influence in determining which disorders, and who, are treated. It is worth noting that the HoNOSCA ‘Social’ subscale also correlated with HoNOS ‘Behaviour’. This appeared to be due to ‘School attendance’ and ‘Family relationships’ from HoNOSCA having small to moderate correlations with ‘Overactive, aggressive’, ‘Self-injury’ and ‘Drinking/drug use’ on HoNOS. In addition, two of the HoNOS social scales did not significantly correlate with any HoNOSCA scales. ‘Living conditions’ and ‘Occupation/activities’ found no significant counterpart with any HoNOSCA scales. These scales have been previously described as problematic (Pirkis et al., 2005b). The glossary of these two scales are concerned with the quality of the environment and available opportunities unlike the HoNOSCA social items which focus on the person. For example, ‘School attendance’ focuses on the
888
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
Table 5 Scales with no apparent counterpart. HoNOSCA scale Unpaired HoNOSCA scales 5. Problems with scholastic or language skills. 8. Problems with non-organic somatic symptoms. 12. Problems with family life and relationships. 13. Poor school attendance.
Unpaired HoNOS scales 3. Non-accidental self-injury 4. Alcohol, substance, solvent misuse No scales correlated p o0.01 No scales correlated p o0.01
HoNOS scale
Spearman’s rho
4. Cognitive problems 9. Problems with relationships No scales correlated po 0.01 2. Non-accidental self-injury 3. Problem drinking or drug-taking 1. Overactive, aggressive, agitated 2. Non-accidental self-injury 3. Problem drinking or drug-taking 7. Depressed mood
0.42 (p ¼0.003) 0.41 (p ¼0.003)
8. Other mental and behavioural problems
0.40 (p¼ 0.005) 0.41 (p o0.003)
0.38 0.39 0.69 0.72 0.71 0.41
(p ¼0.007) (p ¼0.005) (p o 0.001) (p o 0.001) (p o 0.001) (p ¼0.004)
11. Problems with living conditions 12. Problems with occupation and activities
person’s participation in their prime vocational activity rather than the quality of the available activity. ‘Family relationships’ is focussed on the impact on the young person rather than any ‘objective’ assessment of the quality of the family relationships. However, objective concerns about the quality of accommodation and activities are important and may require separate attention through measures of recovery. With individual scales, those with similar or substantial content overlap correlated strongly with their counterparts. Clinicians can expect those scales to generally be operating in the same way. However, while the HoNOS scale ‘Overactivity, aggression, agitation’ correlated strongly with the HoNOSCA ‘Aggression’ scale counterpart, it was smaller with the HoNOSCA ‘Overactivity’ counterpart indicating that ratings of overactivity in HoNOS are overshadowed by aggression. This may prove problematic if youth mental health services assess presentations where ‘Overactivity, attention or concentration’ occurs independently of aggressive behaviours. Young people live with their families longer than in the previous generations (Cobb-Clark, 2008). While there is little published on the impact of youth mental illness and the process of individuation from families, it has been argued that additional challenges are faced by families in helping their offspring in the transition to adulthood when mental health difficulties are present (Jivanjee et al., 2009). Young people with mental illness have cited their family as an important supportive factor in enabling them to achieve their goals (Jivanjee et al., 2008). The failure of the HoNOS ‘Problems with relationships’ scale to correlate with HoNOSCA’s ‘Problems with family relationships’ may be a major limitation for HoNOS use in youth mental health settings. Participants in the current study were clear that family and peer relationships were different areas and that family relationships were important to young people. There was a clear view that HoNOSCA ‘School attendance’ needed to explicitly include vocational attendance to make the instrument useful with youth mental health. This has been part of the supporting documentation on HoNOSCA since its Australian inception (Brann et al., 2003, last accessed 20/12/2011) but clearly would need to be re-emphasised in any implementation by youth mental health services. A number of authors have taken a dichotomous approach to outcome measures preferring to champion client over clinical measures. While not the focus of the current study, it is interesting that the carer consultant in the narrative discussion was very interested in having clinician views made transparent and in using clinician measures to investigate service responses. For example, they thought that knowledge of the clinician instruments placed them in a strong position to question services’ systematic responses to identified problematic areas.
This study has a number of limitations. The results of any study will be influenced by the targets to be rated. This applies to both real world and vignette based studies. It is unlikely that issues around, for example, eating disorders, may have arisen in a service that never had contact with those presentations. It may be that the vignettes chosen with their emphasis on suicidality and substance use, eating, developmental disability and anxiety, delusion and elevated mood are not representative of the presentations in youth mental health services. It is possible that vignettes could be manipulated to produce higher or lower correlations depending on how overtly they are written to the instrument. However, these four were chosen from a larger set of vignettes developed for examining inter-rater reliability. Similarly though, an increase in complexity or ambiguity in real world cases could affect the obtained correlations. While HoNOSCA provided more severe ratings than HoNOS, there is no independent gold standard and it cannot be determined whether this is an over or under estimation of symptom severity. This study focuses on vignettes and expert panels. Vignette studies are important as they can ensure that a number of raters receive exactly the same information. It will also be important to see if these results hold in clinical services with actual clients and another study will address this question. There is little point increasing access to ineffective youth mental health services. It is important that we have clarity around the approach to routine outcome measurement for this age group. The current study has identified a strong relationship between the two key clinical outcome instruments in Australia. Furthermore, it has indicated that the current adult measure will rate youth presentations as less severe, has a blind spot when it comes to the impact of family relationships and is more circumscribed in the range of mental health difficulties considered. Ratings of the quality of the young person’s environmental opportunities may require a separate approach. In conclusion, there appears to be little support for the existing practice of changing outcome instruments mid treatment episode with young people. References Adamis, D., Giannakopoulou, D.F., Konstantopoulou, A., Michailides, M., 2011. Translation and standardization of the honosca (health of the nation outcome scales for children and adolescents) scale in a Greek sample. Clinical Child Psychology and Psychiatry 16, 567–573. Agresti, A., 2003. Categorical Data Analysis. John Wiley & Sons Inc, Hoboken, New Jersey. Bickman, L., Nurcombe, B., Townsend, C., Belle, M., Schut, J.,Karver, M., 1998. Consumer measurement systems for child and adolescent mental health (A report to the Commonwealth mental health branch). Department of Health and Family Services, Canberra, ACT.
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
Birleson, P., 2009. Should youth mental health become a specialty in its own right? No. British Medical Journal, 339. Birleson, P., Vance, A., 2008. Developing the ‘youth model’ in mental health services. Australasian Psychiatry 16, 22–26. Brann, P., 2010. Routine outcome measurement in child and adolescent mental health. In: Trauer, T. (Ed.), Outcome measurement in mental health: Theory and practice. Cambridge University Press, Cambridge, pp. 105–115. Brann, P., Carling, M., Chipps, J., Coombs, T., 2003. Frequently asked questions, rarely asked questions and never asked questions. The clinicians faq, raq and naq guide to honosca in Australia. Retrieved October, 2005, from /www. amhocn.org/home/faqsS. Burns, A., Beevor, A., Lelliott, P., Wing, J., Blakey, A., Orrell, M., Mulinga, J., Hadden, S., 1999. Health of the nation outcome scales for elderly people (honos 65þ ). British Journal of Psychiatry 174, 424–427. Callaly, T., von Treuer, K., van Hamond, T., Windle, K., 2011. Forming and sustaining partnerships to provide integrated services for young people: An overview based on the headspace Geelong experience. Early Intervention in Psychiatry 5 (Suppl. 1), 28–33. Cobb-Clark, D.A., 2008. Leaving home: What economics has to say about the living arrangements of young Australians. Australian Economic Review 41, 160. Commonwealth of Australia., 2009. Fourth national mental health plan—an agenda for collaborative government action in mental health 2009–2014. Commonwealth of Australia, Canberra. Department of Health and Ageing. (2003). Mental health national outcomes and casemix collection: Overview of clinician-rated and consumer self-report measures. Version 1.50. Department of Health and Ageing, Canberra. Department of Health and Ageing. (2009). Mental health national outcomes and casemix collection: Technical specification of state and territory reporting requirements. Version 1.60. Commonwealth of Australia, Canberra. Department of Health and Ageing. (2011). National health reform: Progress and delivery. Commonwealth of Australia, Canberra. Garralda, E., Yates, P., 2000. Honosca: Uses and limitations. Child Psychology and Psychiatry Review 5, 131–132. Gowers, S.G., Harrington, R.C., Whitton, A., Beevor, A., Lelliott, P., Jezzard, R., Wing, J., 1999a. Health of the nation outcome scales for children and adolescents (honosca). Glossary for honosca score sheet. British Journal of Psychiatry 174, 428–431. Gowers, S.G., Harrington, R.C., Whitton, A., Lelliott, P., Beevor, A., Wing, J., Jezzard, R., 1999b. Brief scale for measuring the outcomes of emotional and behavioural disorders in children. Health of the nation outcome scales for children and adolescents (honosca). British Journal of Psychiatry 174, 413–416. Hanssen-Bauer, K., Aalen, O.O., Ruud, T., Heyerdahl, S., 2007a. Inter-rater reliability of clinician-rated outcome measures in child and adolescent mental health services. Administration & Policy in Mental Health 34, 504–512. Hanssen-Bauer, K., Gowers, S., Aalen, O.O., Bilenberg, N., Brann, P., Garralda, E., Merry, S., Heyerdahl, S., 2007b. Cross-national reliability of clinician-rated
889
outcome measures in child and adolescent mental health services. Administration & Policy in Mental Health 34, 513–518. Jivanjee, P., Kruzich, J., Gordon, L., 2008. Community integration of transition-age individuals: Views of young with mental health disorders. Journal of Behavioral Health Services and Research 35, 402–418. Jivanjee, P., Kruzich, J., Gordon, L., 2009. The age of uncertainty: Parent perspectives on the transitions of young people with mental health difficulties to adulthood. Journal of Child and Family Studies 18, 435–446. Kisely, S., Campbell, L.A., Crossman, D., Gleich, S., Campbell, J., 2007. Are the health of the nation outcome scales a valid and practical instrument to measure outcomes in North America? A three-site evaluation across Nova Scotia. Community Mental Health Journal 43, 91–107. McGorry, P., 1998. Beyond adolescent psychiatry: The logic of a youth mental health model. The Australian and New Zealand Journal of Psychiatry 32, 138–140. McGorry, P., Purcell, R., Hickie, I.B., Jorm, A.F., 2007. Investing in youth mental health is a best buy. Medical Journal of Australia 187 (Suppl. 7), S5–7. Patel, V., Flisher, A.J., Hetrick, S., McGorry, P., 2007. Mental health of young people: A global public-health challenge. Lancet 369, 1302–1313. Pirkis, J., Burgess, P., Kirk, P., Dodson, S., Coombs, T., 2005a. Review of standardised measures used in the national outcomes and casemix collection (NOCC). Version 1.2. Australian Mental Health Outcomes and Classification Network, Sydney. Pirkis, J., Burgess, P.M., Kirk, P.K., Dodson, S., Coombs, T.J., Williamson, M.K., 2005b. A review of the psychometric properties of the health of the nation outcome scales (honos) family of measures. Health & Quality of Life Outcomes 3, 76. Scott, E., Naismith, S., Whitwell, B., Hamilton, B., Chudleigh, C., Hickie, I., 2009. Delivering youth-specific mental health services: The advantages of a collaborative, multi-disciplinary system. Australasian Psychiatry 17, 189–194. Slade, M., Thornicroft, G., Glover, G., 1999. The feasibility of routine outcome measures in mental health. Social Psychiatry and Psychiatric Epidemiology 34, 243–249. Tabachnick, B.G., Fidell, L.S., 1996. Using Multivariate Statistics, 3rd ed. Harper Collins College Publishers, New York. Trauer, T. (Ed.), 2010. Outcome Measurement in Mental Health: Theory and Practice. Cambridge University Press, Cambridge. Turner, S., 2004. Are the health of the nation outcome scales (honos) useful for measuring outcomes in older people’s mental health services? Aging & Mental Health 8, 387–396. Wing, J.K., Beevor, A.S., Curtis, R.H., Park, S.B.G., Hadden, S., Burns, A., 1998. Health of the nation outcome scales (honos). British Journal of Psychiatry 172, 11–18. Wing, J.K., Curtis, R.H., Beevor, A.S., 1999. Health of the nation outcome scales (honos). Glossary for honos score sheet. British Journal of Psychiatry 174, 432–434. Yates, P., Kramer, T., Garralda, M.E., 2006. Use of a routine mental health measure in an adolescent secure unit. British Journal of Psychiatry 188, 583–584.
Contents lists available at SciVerse ScienceDirect
Psychiatry Research journal homepage: www.elsevier.com/locate/psychres
Routine outcome measurement in youth mental health: A comparison of two clinician rated measures, HoNOSCA and HoNOS Peter Brann a,d,n, Monique Alexander b, Tim Coombs c a
Eastern Health Child and Youth Mental Health Service, Australia Eastern Health Adult Mental Health Service, Australia c Australian Mental Health Outcomes and Classification Network, Australia d Monash University School of Psychology and Psychiatry, Australia b
a r t i c l e i n f o
a b s t r a c t
Article history: Received 14 March 2012 Received in revised form 29 June 2012 Accepted 12 July 2012
Mental health services engagement in routine outcome measurement has created a demand for a coherent infrastructure. Two clinician instruments for adolescents (HoNOSCA) and adults (HONOS) are used across many countries. However, the increasing emphasis on youth suggests protocols with historically driven age divisions may obscure outcomes. The current study examines these instruments’ congruence with regard to youth mental health. Members of national mental health expert panels rated four vignettes before discussing perceived strengths and weaknesses. The instruments were strongly correlated and HoNOSCA resulted in more severe symptom scores. Most subscales and scales correlated as predicted with some important exceptions. ‘Problems with family relationships’, tracked by HoNOSCA, did not correlate with its HoNOS counterpart. Qualitative feedback indicated using the HoNOSCA scale ‘School attendance’ for vocational attendance would improve its applicability to young people. The instruments have a strong relationship. While either could be used, HoNOS will underestimate symptom severity in youth. The importance of family relationships for young people suggests that HoNOSCA is preferable. While sited in the Australian context, these findings should be applicable to other countries using these instruments, and should interest services considering the continuity of youth presentations and their outcomes. Crown Copyright & 2012 Published by Elsevier Ireland Ltd. All rights reserved.
Keywords: Young adult Outcome measures Mental health services HoNOSCA HONOS
1. Introduction With a substantial proportion of the mental health burden falling on those aged between 12 and 25, there have been calls for services that specifically target this population (McGorry et al., 2007). It has long been argued that neither child nor adult services adequately meet the needs of youth (McGorry, 1998) while others have argued this is not an inherent limitation of this service configuration (Birleson and Vance, 2008). Irrespective of configuration concerns, development does not cease at 18 and there is great continuity in the presentation of consumers over the adolescent to young adult age range. Adult services have focussed on a narrower range of disorders than affect the majority of young people and the newer youth models can be expected to see a broader range of presentations and hence more domains of symptoms and functioning may be implicated (Birleson, 2009).
n Corresponding author: Eastern Health & Monash University, Child Youth Mental Health Service & School of Psychology and Psychiatry, Wundeela Centre, 21 Ware Cres, East Ringwood, Victoria 3135, Australia. Tel.: þ61 03 9870 9788; fax: þ 61 03 9870 7973. E-mail address: [email protected] (P. Brann).
Within Australia, the latest national mental health plan clearly calls for the expansion of youth mental health services (Commonwealth of Australia, 2009). Many countries are engaged in embedding outcome measurement in routine practice (Trauer, 2010). Monitoring and evaluating outcomes for individuals with psychiatric disorders are crucial aspects of service delivery. Effective services help people in a range of areas including quality of life, recovery, symptoms and functioning (Bickman et al., 1998). Standardised outcome measures provide a valuable aid in assessing and evaluating treatment progress as well as providing a common language that is transferrable across mental health services (Slade et al., 1999). The national outcomes and casemix collection (NOCC) was introduced in Australia to provide a suite of measures that support clinical practice and comparisons across services and different consumer populations. The health of the national outcome scales (HoNOS) are a family of instruments first developed in the United Kingdom (Wing et al., 1998) to allow clinicians to routinely map common problems for consumers with a mental illness. Originally designed to assess adults with a severe mental illness, variants were developed for children and adolescents (HoNOSCA) (Gowers et al., 1999b) and for older persons (HoNOS 65þ) (Burns et al., 1999). This family form the key clinician measures
0165-1781/$ - see front matter Crown Copyright & 2012 Published by Elsevier Ireland Ltd. All rights reserved. http://dx.doi.org/10.1016/j.psychres.2012.07.010
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
of problem severity in the NOCC suite (Department of Health and Ageing, 2003). All three instruments have been found to possess sufficient psychometric properties to warrant routine use in clinical practice (Brann, 2010; Pirkis et al., 2005a; Trauer, 2010), including inter-rater reliability (Hanssen-Bauer et al., 2007a; Pirkis et al., 2005b), and have been implemented in a range of countries (Adamis et al., 2011; Hanssen-Bauer et al., 2007b; Trauer, 2010). The development of youth mental health services is occurring across Australia (Callaly et al., 2011; Scott et al., 2009), however, the national protocol has remained silent on which of the HoNOS family of measures should be used in these types of services (Department of Health and Ageing, 2009). Currently, if a young person crosses the age boundary between child and adolescent mental health (CAMHS) and adult services, the national protocol requires the closing of a CAMHS episode and the opening of a new adult episode of care. Although this artificial closure and opening of treatment episodes can be coded, it is cumbersome and unless acknowledged will erroneously underestimate the effectiveness of treatment for those continuing across the age boundary. The arbitrary truncation of CAMHS service delivery at 18 years has been criticised as a process which unnecessarily fragments the continuing mental health needs of teenagers and young adults (Patel et al., 2007) and the cessation of its outcomes protocol mirrors this discontinuity. Despite the increased emphasis in youth-oriented services, there is no consensus on whether HoNOSCA or HoNOS measures should be used by clinicians in these youth services. As these two instruments have some differences in content, it may be problematic to compare one measure with the other. There has been little evaluation of how the different measures may perform in describing the presentation of consumers of youth services. The correlation between HoNOSCA and other adolescent instruments has been investigated (Yates et al., 2006). There has also been work to determine the correlation between HoNOS and the version for older persons, HoNOSCA65þ (Turner, 2004). However, a literature search conducted using Ovid Medline and Ovid PsycInfo failed to find any publication examining the relationships between HoNOS and HoNOSCA. Understanding the relationship between HoNOSCA and HoNOS is an important step in evaluating the applicability of these outcome measures across the lifespan and the suitability of these measures for use with the youth age group. However, as correlation is independent of an instrument’s metric (Tabachnick and Fidell, 1996), two instruments may correlate perfectly while indicating differing levels of severity. As the severity of symptoms may be important in the
885
future allocation of resources, differences between the extent of severity indicated by the two instruments will also be examined. HoNOS contain 12 scales across four subscales of behaviour, impairment, symptoms and social functioning. HoNOSCA are conceptually similar with 13 clinical scales and similarly named subscales, however, some HoNOSCA items are more specific to the adolescent developmental period, including an assessment of peer relationships, family relationships and school attendance (Garralda and Yates, 2000). Two additional HoNOSCA scales relate to information about the disorder and its treatment, have no HONOS counterpart and will be excluded from further consideration. Through the analysis of case study vignettes, the present study will evaluate the strength of association between HoNOSCA and HoNOS, and the benefits and limitations of each measure in capturing relevant outcomes for young people.
2. Method 2.1. Participants The Australian government created national expert advisory panels to provide advice on the implementation and the development of mental health information including routine outcome measurement. Panel members are experienced practitioners, academics, managers and the panels include carers and consumers. 2.2. Procedure Members of both the child and adolescent and the adult national mental health information development expert advisory panels were invited to participate. All members present at regular meetings consented to participate and focussed refresher training was provided by one author (T.C.). HoNOSCA and HoNOS glossaries were provided (Gowers et al., 1999a; Wing et al., 1999) including the HoNOSCA implementation guideline noting that if school was not the key vocational setting for that child or adolescent (e.g., child care and employed), then the key vocational setting for that young person should be used as the target to rate attendance (Brann et al., 2003). Four written case vignettes were selected from an international reliability study (HanssenBauer et al., 2007b) and the ages modified from adolescence to young adulthood (18, 19, 21 and 22 years). The original vignettes can be found at http://amhocn.org/ training-service-development. Members were asked to rate these four vignettes about young people using both HoNOSCA and HoNOS with counterbalancing of instrument and vignette order both within and between participants respectively. Following ratings, five questions were posed to the group: important mental health symptoms for this age group, symptom areas that HONOS/HoNOSCA allowed to be identified, and the symptom areas difficult to be identified with HONOS/HoNOSCA. The discussion was facilitated by one of the authors (P.B.) to ensure all participants were able to contribute. Themes from this discussion were noted by the panel secretariat. All results were de-identified by the secretariat before forwarding to the authors (P.B. and M.A.) for data analysis. The results of those with purely consumer, carer or policy development roles on the panel were excluded from quantitative analyses of the scales
Table 1 HoNOS and HoNOSCA Scales. Scale HoNOSCA
HoNOS
1
Problems with disruptive, antisocial or aggressive behaviour
2 3 4 5 6 7 8 9 10 11 12 13 14 15
Problems with overactivity, attention or concentration Non-accidental self-injury Problems with alcohol, substance or solvent misuse Problems with scholastic or language skills Physical illness or disability problems Problems associated with hallucinations, delusions or abnormal perceptions Problems with non-organic somatic symptoms Problems with emotional and related symptoms Problems with peer relationships Problems with self-care and independence Problems with family life and relationships Poor school attendance Problems with knowledge or understanding about the nature of the child or adolescent’s difficulties Problems with lack of information about services or management of the child or adolescent’s difficulties
Overactive, aggressive, disruptive or agitated behaviour Non-accidental self-injury Problem drinking or drug-taking Cognitive problems Physical illness or disability problems Problems associated with hallucinations and delusions Problems with depressed mood Other mental and behavioural problems Problems with relationships Problems with activities of daily living Problems with living conditions Problems with occupation and activities
886
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
although their views are included in the qualitative findings. HoNOS and HoNOSCA scales are each scored from 0 (No Problem) to 4 (Severe Problem) (Table 1). The sum of all HoNOS scales and the first 13 HoNOSCA scales are used to calculate the total score with the maximum being 48 and 52 respectively. There are subscales within both instruments; nominally titled ‘Behaviour’, ‘Impairment’, ‘Symptoms’ and ‘Social’. Higher scores indicate more severe problems. Quantitative analysis was conducted with SPSS (version 17) supplemented with syntax for establishing weighted kappa (http:// www-304.ibm.com/support/docview.wss?uid=swg21477357, last accessed 6/6/2012). 2.3. Statistical analyses Previous studies with these instruments have used both parametric and nonparametric analyses. In line with Kisely et al. (2007), the instruments’ total scores as well as the subscales, were treated as parametric and individual scales as nonparametric. Pearson correlations were used to examine the relationship between HoNOSCA and HoNOS scales as rated across the four vignettes. Responses from the non-clinical raters (n¼ 5) were excluded as were HoNOSCA scales 14 and 15 which focus on knowledge and information. Strength of association between the instruments was assessed with Pearson and Spearman correlations accordingly. Correlations were deemed significant at a ¼0.01 (two-tailed). Differences between mean scores for the totals and subscales were examined with t-tests. Where scales between the instruments have highly similar content, correlations were supplemented with descriptions of cross-tabulated dichotomous and absolute agreement, and weighted kappa (Agresti, 2003). Reported weighted kappa uses the squared distance between categories method. From the CAMHS panel, eight members were included in the quantitative analysis with a total of 32 ratings with HoNOSCA and 32 ratings with HONOS. The adult panel had five members with 18 HoNOSCA and 17 HONOS ratings included in the analysis. Any difference between vignettes or raters was not relevant to the purposes of this study and consequently between-vignette or between-rater comparisons were not conducted. To adjust for the difference in the instruments’ ranges, scores were converted to percentages of the maximum possible before t-tests were used to examine differences. It was hypothesised that the two instruments would correlate significantly and that the extent of this would vary with the degree of overlap between scales. It was also hypothesised that the instruments would indicate similar levels of severity.
3. Results 3.1. Total Scores and subscales The mean total scores for HoNOSCA and HoNOS were 22.1 (S.D. ¼8.5) and 16.5 (S.D. ¼4.9) respectively. Scores ranged from 8 to 40 using HoNOSCA and from 8 to 28 using HoNOS. As a percentage of the maximum possible score, the mean total was 43% (S.D.¼16%) and 35% (S.D. ¼10%) for HoNOSCA and HoNOS respectively with the ranges being from 15% to 77% for HoNOSCA and from 17% to 58% using HoNOS. HoNOSCA produced significantly higher total scores as a percentage of the maximum possible (t¼5.91, d.f. ¼1,48, p o0.001) indicating a higher estimate of severity than HoNOS. Correlations of total and subscale scores are Pearson. The total scores on HoNOSCA and HoNOS were found to have a significant and strong correlation of 0.85 (p o0.001). Young people found to be elevated with one instrument will be elevated on the other. The concept of symptom severity does appear to have substantial commonality with the two instruments. With one exception, the subscales on each instrument were only significantly correlated with their counterpart and no other subscale. HoNOSCA ‘Behaviour’ was significantly correlated with HoNOS ‘Behaviour’ (r ¼0.95, p o0.01), ‘Impairment’ with
‘Impairment’ (r ¼0.54, p o0.01) and ‘Symptoms’ with ‘Symptoms’ (r ¼0.43, p o0.01). While HoNOSCA ‘Social’ was significantly correlated with HoNOS ‘Social’ (r¼0.65, po0.01), it was also significantly correlated with the HoNOS ‘Behaviour’ subscale (r ¼0.64, p o0.01). All HoNOSCA subscale scores were significantly greater than the HoNOS counterparts with the exception of ‘Symptoms’. The mean scores for HoNOSCA and HoNOS respectively on each subscale, expressed as a percentage of the maximum possible score were: Behaviour (34% S.D. ¼28%; 27% S.D. ¼26%; t¼ 6.09, d.f.¼1,49, po0.001), Impairment (40% S.D. ¼ 19%; 25% S.D. ¼17%; t¼5.98, d.f.¼1,49, p o0.001), Symptoms (46% S.D. ¼ 19%; 47% S.D. ¼15%; t ¼ 0.39, ns), and Social (50% S.D. ¼23%; 36% S.D. ¼13%; t ¼5.47, d.f. ¼1,49, p o0.001). 3.2. Scales Scales were grouped into those with similar content, substantial content overlap, some overlap and those with no apparent counterpart. As individual scales could be expected to have a stronger relationship than totals or subscales, only those Spearman correlations significant at 0.001 are reported. As this is a stringent criteria, correlations of those scales with no apparent counterpart where the associated probability was less than 0.01 will also be reported. Overall, strong correlations were found for all scales with highly similar content overlap (Table 2). With scales with highly similar content, it is sensible to also examine both the absolute level of agreement (i.e., is a ‘3’ on HoNOSCA accompanied by a ‘3’ on HONOS) and the clinical agreement (i.e., are both scales rated as clinically significant [2, 3, or 4]). With ‘Non-accidental selfinjury’, there was complete agreement on the clinical significance of the ratings between instruments and only 2% of ratings differed on the absolute rating. ‘Substance misuse’ also had complete agreement on clinical significance and only 14% disagreement on the absolute rating. With Physical illness, 4% of ratings disagreed on clinical significance, while there was 16% disagreement on the absolute rating. With peer relationships and relationships, 8.1% differed on clinical significance while 40.7% of ratings differed on the absolute rating. With ‘Self-care’ and ‘Daily living’, 8.1% of ratings differed on clinical significance while 34.6% differed on the absolute rating. Weighted kappa (Table 2) echoed this finding of both strong association and agreement. Scales with substantial or some content overlap also had significant correlations (Table 3, Table 4). HoNOS scale 1 (‘Overactivity, aggression and agitation’) overlaps two HoNOSCA scales (scale 1: ‘Disruptive, antisocial, aggressive’; and scale 2: ‘Overactivity, attention, concentration’) and a significant correlation was found between each of these scales with the correlation being smaller with ‘Overactivity’. HoNOSCA scale 9 (emotional and related symptoms) focuses on both depressive and anxiety symptoms while HoNOS scale 7 (depressed mood) is far more circumscribed and a smaller correlation was obtained compared with scales with greater overlap. Generally, scales with no apparent counterpart had small correlations (Table 5). Associated probabilities are also reported
Table 2 Scales with highly similar content. HoNOSCA scale
HoNOS scale
Spearman’s rho
Weighted kappa
3. Non-accidental self-injury. 4. Alcohol, substance, solvent misuse 6. Physical illness or disability problems. 10. Problems with peer relationships. 11. Problems with self-care and independence.
2. Non-accidental self-injury 3. Problem drinking or drug-taking 5. Physical illness or disability problems 9. Problems with relationships. 10. Problems with activities of daily living.
0.99 0.99 0.87 0.80 0.60
0.99 0.92 0.84 0.75 0.65
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
887
Table 3 Scales with substantial content overlap. HoNOSCA scale
HoNOS scale
Spearman’s rho
1. Disruptive, antisocial, or aggressive behaviour. 2. Problems with overactivity, attention or concentration. 7. Problems associated with hallucinations, delusions or abnormal perceptions.
1. Overactive, aggressive, disruptive, agitated 1. Overactive, aggressive, disruptive, agitated 6. Problems associated with hallucinations and delusions
0.92 0.72 0.84
Previously, carers felt they had not focussed so clearly on the clinicians’ perspective as conveyed in these instruments.
Table 4 Scales with some content overlap. HoNOSCA scale
HoNOS scale
Spearman’s rho
9. Problems with emotional and related symptoms.
7. Problems with depressed mood.
0.58
4. Discussion
for these correlations. The key exception to this was ‘Poor school attendance’ which correlated strongly with ‘Overactive, aggressive’, ‘Non-accidental self-injury’ and ‘Problem drinking or drugtaking’ on HoNOS. 3.3. Qualitative findings The panels indicated that the important mental health symptoms for young people included self-harm, substance use, agitation, aggression, hallucinations, delusions, depression, anxiety, relationship difficulties and role functioning. HoNOS were seen as particularly able to identify hallucinations without the confounding of abnormal perceptions that occurs in HoNOSCA. The panels believed HoNOS were better able to identify issues with living conditions. The adult panel thought that HoNOS may be more useful with mental illnesses for those at the upper end of the youth age bracket. Conversely that panel thought HoNOSCA were more useful at the lower end of the youth bracket. Both groups perceived that HoNOSCA enabled easier identification of both family and peer relationships and that these are distinct and important areas of young peoples wellbeing. The CAMHS group also perceived HoNOSCA as more useful in symptoms of eating, emotional, attention and concentration difficulties. The adult panel agreed eating symptoms were not adequately rated with HoNOS. While HoNOSCA scales 14 and 15 have not been analysed, both groups were concerned about the diminishing involvement of parents with older youth and the range of barriers to their understanding of their offspring’s problems and treatment. The adult panel were clear that HoNOSCA scale 13 needed vocational as well as ‘School attendance’ to be explicitly included in the glossary to render this scale broadly useful for young people. HoNOS were seen as limited in its attention to family, problematic in its confounding of different symptoms (e.g., ‘Aggression and overactivity’) and requiring greater clarification of scale 8 (‘Other mental health problems’). On the instruments themselves, the adult panel emphasised the importance of not dissecting the instruments to create a youth version. They suggested that a choice be made available so clinicians could use either instrument depending on the salient issues of the young person. It was also suggested that issues of functioning be examined for this age group with existing or new measures. A valuable, though tangential to the study, discussion focussed on the utility of the information contained in these clinician instruments for carers and policy makers. The panels’ carers and policy makers discussed how management could both respond to ongoing service difficulties in treating these symptomatic areas, and share that information with consumers and carers.
The routine outcome collection protocol needs to be modified as a result of the emerging emphasis on youth-oriented mental health services. The current study explored how the key clinician rated measures functioned when directed at youth mental health vignettes. With regard to the two overarching hypotheses, there was support for a strong relationship between the two instruments with the obtained correlation strength varying as the scales overlap varied. However, the instruments did not indicate equivalent levels of severity. Overall, the strong relationship between the total scores is encouraging. Both appear to be tapping a similar construct of symptom severity even if greater severity will be indicated by HoNOSCA. This strong relationship was also apparent with the subscales where there was significant commonality between the constructs but again with HoNOSCA indicating greater severity than HoNOS. The symptom subscale was the exception indicating similar levels of severity with both instruments and even though HoNOSCA has only one extra item, the other three subscales all had significantly higher scores as a percentage of the maximum. This suggests that the HoNOSCA scales and their glossaries may be more inclusive of a broader range of mental health difficulties. For example, HoNOS blend overactivity with aggression while HoNOSCA allow for disruptive and aggressive behaviours to be rated separately to overactivity or attention problems. The narrative discussion reinforced the view that HoNOSCA applied to a broader range of symptoms. The difference between the instruments in severity may be relevant if services targeted a certain level of symptom severity. Depending on the ratings of older adults with HoNOS, it seems possible that HoNOS will present the broader range of problems, intended to be treated in a youth mental health service, as less severe than the target disorders HoNOS were oriented towards (e.g., schizophrenia). This could lead to a relative downplaying of the non-psychotic disorders. If activity based (Department of Health and Ageing, 2011) or insurance funding incorporates severity considerations, then the rated level of severity indicated by a scale may be an important influence in determining which disorders, and who, are treated. It is worth noting that the HoNOSCA ‘Social’ subscale also correlated with HoNOS ‘Behaviour’. This appeared to be due to ‘School attendance’ and ‘Family relationships’ from HoNOSCA having small to moderate correlations with ‘Overactive, aggressive’, ‘Self-injury’ and ‘Drinking/drug use’ on HoNOS. In addition, two of the HoNOS social scales did not significantly correlate with any HoNOSCA scales. ‘Living conditions’ and ‘Occupation/activities’ found no significant counterpart with any HoNOSCA scales. These scales have been previously described as problematic (Pirkis et al., 2005b). The glossary of these two scales are concerned with the quality of the environment and available opportunities unlike the HoNOSCA social items which focus on the person. For example, ‘School attendance’ focuses on the
888
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
Table 5 Scales with no apparent counterpart. HoNOSCA scale Unpaired HoNOSCA scales 5. Problems with scholastic or language skills. 8. Problems with non-organic somatic symptoms. 12. Problems with family life and relationships. 13. Poor school attendance.
Unpaired HoNOS scales 3. Non-accidental self-injury 4. Alcohol, substance, solvent misuse No scales correlated p o0.01 No scales correlated p o0.01
HoNOS scale
Spearman’s rho
4. Cognitive problems 9. Problems with relationships No scales correlated po 0.01 2. Non-accidental self-injury 3. Problem drinking or drug-taking 1. Overactive, aggressive, agitated 2. Non-accidental self-injury 3. Problem drinking or drug-taking 7. Depressed mood
0.42 (p ¼0.003) 0.41 (p ¼0.003)
8. Other mental and behavioural problems
0.40 (p¼ 0.005) 0.41 (p o0.003)
0.38 0.39 0.69 0.72 0.71 0.41
(p ¼0.007) (p ¼0.005) (p o 0.001) (p o 0.001) (p o 0.001) (p ¼0.004)
11. Problems with living conditions 12. Problems with occupation and activities
person’s participation in their prime vocational activity rather than the quality of the available activity. ‘Family relationships’ is focussed on the impact on the young person rather than any ‘objective’ assessment of the quality of the family relationships. However, objective concerns about the quality of accommodation and activities are important and may require separate attention through measures of recovery. With individual scales, those with similar or substantial content overlap correlated strongly with their counterparts. Clinicians can expect those scales to generally be operating in the same way. However, while the HoNOS scale ‘Overactivity, aggression, agitation’ correlated strongly with the HoNOSCA ‘Aggression’ scale counterpart, it was smaller with the HoNOSCA ‘Overactivity’ counterpart indicating that ratings of overactivity in HoNOS are overshadowed by aggression. This may prove problematic if youth mental health services assess presentations where ‘Overactivity, attention or concentration’ occurs independently of aggressive behaviours. Young people live with their families longer than in the previous generations (Cobb-Clark, 2008). While there is little published on the impact of youth mental illness and the process of individuation from families, it has been argued that additional challenges are faced by families in helping their offspring in the transition to adulthood when mental health difficulties are present (Jivanjee et al., 2009). Young people with mental illness have cited their family as an important supportive factor in enabling them to achieve their goals (Jivanjee et al., 2008). The failure of the HoNOS ‘Problems with relationships’ scale to correlate with HoNOSCA’s ‘Problems with family relationships’ may be a major limitation for HoNOS use in youth mental health settings. Participants in the current study were clear that family and peer relationships were different areas and that family relationships were important to young people. There was a clear view that HoNOSCA ‘School attendance’ needed to explicitly include vocational attendance to make the instrument useful with youth mental health. This has been part of the supporting documentation on HoNOSCA since its Australian inception (Brann et al., 2003, last accessed 20/12/2011) but clearly would need to be re-emphasised in any implementation by youth mental health services. A number of authors have taken a dichotomous approach to outcome measures preferring to champion client over clinical measures. While not the focus of the current study, it is interesting that the carer consultant in the narrative discussion was very interested in having clinician views made transparent and in using clinician measures to investigate service responses. For example, they thought that knowledge of the clinician instruments placed them in a strong position to question services’ systematic responses to identified problematic areas.
This study has a number of limitations. The results of any study will be influenced by the targets to be rated. This applies to both real world and vignette based studies. It is unlikely that issues around, for example, eating disorders, may have arisen in a service that never had contact with those presentations. It may be that the vignettes chosen with their emphasis on suicidality and substance use, eating, developmental disability and anxiety, delusion and elevated mood are not representative of the presentations in youth mental health services. It is possible that vignettes could be manipulated to produce higher or lower correlations depending on how overtly they are written to the instrument. However, these four were chosen from a larger set of vignettes developed for examining inter-rater reliability. Similarly though, an increase in complexity or ambiguity in real world cases could affect the obtained correlations. While HoNOSCA provided more severe ratings than HoNOS, there is no independent gold standard and it cannot be determined whether this is an over or under estimation of symptom severity. This study focuses on vignettes and expert panels. Vignette studies are important as they can ensure that a number of raters receive exactly the same information. It will also be important to see if these results hold in clinical services with actual clients and another study will address this question. There is little point increasing access to ineffective youth mental health services. It is important that we have clarity around the approach to routine outcome measurement for this age group. The current study has identified a strong relationship between the two key clinical outcome instruments in Australia. Furthermore, it has indicated that the current adult measure will rate youth presentations as less severe, has a blind spot when it comes to the impact of family relationships and is more circumscribed in the range of mental health difficulties considered. Ratings of the quality of the young person’s environmental opportunities may require a separate approach. In conclusion, there appears to be little support for the existing practice of changing outcome instruments mid treatment episode with young people. References Adamis, D., Giannakopoulou, D.F., Konstantopoulou, A., Michailides, M., 2011. Translation and standardization of the honosca (health of the nation outcome scales for children and adolescents) scale in a Greek sample. Clinical Child Psychology and Psychiatry 16, 567–573. Agresti, A., 2003. Categorical Data Analysis. John Wiley & Sons Inc, Hoboken, New Jersey. Bickman, L., Nurcombe, B., Townsend, C., Belle, M., Schut, J.,Karver, M., 1998. Consumer measurement systems for child and adolescent mental health (A report to the Commonwealth mental health branch). Department of Health and Family Services, Canberra, ACT.
P. Brann et al. / Psychiatry Research 200 (2012) 884–889
Birleson, P., 2009. Should youth mental health become a specialty in its own right? No. British Medical Journal, 339. Birleson, P., Vance, A., 2008. Developing the ‘youth model’ in mental health services. Australasian Psychiatry 16, 22–26. Brann, P., 2010. Routine outcome measurement in child and adolescent mental health. In: Trauer, T. (Ed.), Outcome measurement in mental health: Theory and practice. Cambridge University Press, Cambridge, pp. 105–115. Brann, P., Carling, M., Chipps, J., Coombs, T., 2003. Frequently asked questions, rarely asked questions and never asked questions. The clinicians faq, raq and naq guide to honosca in Australia. Retrieved October, 2005, from /www. amhocn.org/home/faqsS. Burns, A., Beevor, A., Lelliott, P., Wing, J., Blakey, A., Orrell, M., Mulinga, J., Hadden, S., 1999. Health of the nation outcome scales for elderly people (honos 65þ ). British Journal of Psychiatry 174, 424–427. Callaly, T., von Treuer, K., van Hamond, T., Windle, K., 2011. Forming and sustaining partnerships to provide integrated services for young people: An overview based on the headspace Geelong experience. Early Intervention in Psychiatry 5 (Suppl. 1), 28–33. Cobb-Clark, D.A., 2008. Leaving home: What economics has to say about the living arrangements of young Australians. Australian Economic Review 41, 160. Commonwealth of Australia., 2009. Fourth national mental health plan—an agenda for collaborative government action in mental health 2009–2014. Commonwealth of Australia, Canberra. Department of Health and Ageing. (2003). Mental health national outcomes and casemix collection: Overview of clinician-rated and consumer self-report measures. Version 1.50. Department of Health and Ageing, Canberra. Department of Health and Ageing. (2009). Mental health national outcomes and casemix collection: Technical specification of state and territory reporting requirements. Version 1.60. Commonwealth of Australia, Canberra. Department of Health and Ageing. (2011). National health reform: Progress and delivery. Commonwealth of Australia, Canberra. Garralda, E., Yates, P., 2000. Honosca: Uses and limitations. Child Psychology and Psychiatry Review 5, 131–132. Gowers, S.G., Harrington, R.C., Whitton, A., Beevor, A., Lelliott, P., Jezzard, R., Wing, J., 1999a. Health of the nation outcome scales for children and adolescents (honosca). Glossary for honosca score sheet. British Journal of Psychiatry 174, 428–431. Gowers, S.G., Harrington, R.C., Whitton, A., Lelliott, P., Beevor, A., Wing, J., Jezzard, R., 1999b. Brief scale for measuring the outcomes of emotional and behavioural disorders in children. Health of the nation outcome scales for children and adolescents (honosca). British Journal of Psychiatry 174, 413–416. Hanssen-Bauer, K., Aalen, O.O., Ruud, T., Heyerdahl, S., 2007a. Inter-rater reliability of clinician-rated outcome measures in child and adolescent mental health services. Administration & Policy in Mental Health 34, 504–512. Hanssen-Bauer, K., Gowers, S., Aalen, O.O., Bilenberg, N., Brann, P., Garralda, E., Merry, S., Heyerdahl, S., 2007b. Cross-national reliability of clinician-rated
889
outcome measures in child and adolescent mental health services. Administration & Policy in Mental Health 34, 513–518. Jivanjee, P., Kruzich, J., Gordon, L., 2008. Community integration of transition-age individuals: Views of young with mental health disorders. Journal of Behavioral Health Services and Research 35, 402–418. Jivanjee, P., Kruzich, J., Gordon, L., 2009. The age of uncertainty: Parent perspectives on the transitions of young people with mental health difficulties to adulthood. Journal of Child and Family Studies 18, 435–446. Kisely, S., Campbell, L.A., Crossman, D., Gleich, S., Campbell, J., 2007. Are the health of the nation outcome scales a valid and practical instrument to measure outcomes in North America? A three-site evaluation across Nova Scotia. Community Mental Health Journal 43, 91–107. McGorry, P., 1998. Beyond adolescent psychiatry: The logic of a youth mental health model. The Australian and New Zealand Journal of Psychiatry 32, 138–140. McGorry, P., Purcell, R., Hickie, I.B., Jorm, A.F., 2007. Investing in youth mental health is a best buy. Medical Journal of Australia 187 (Suppl. 7), S5–7. Patel, V., Flisher, A.J., Hetrick, S., McGorry, P., 2007. Mental health of young people: A global public-health challenge. Lancet 369, 1302–1313. Pirkis, J., Burgess, P., Kirk, P., Dodson, S., Coombs, T., 2005a. Review of standardised measures used in the national outcomes and casemix collection (NOCC). Version 1.2. Australian Mental Health Outcomes and Classification Network, Sydney. Pirkis, J., Burgess, P.M., Kirk, P.K., Dodson, S., Coombs, T.J., Williamson, M.K., 2005b. A review of the psychometric properties of the health of the nation outcome scales (honos) family of measures. Health & Quality of Life Outcomes 3, 76. Scott, E., Naismith, S., Whitwell, B., Hamilton, B., Chudleigh, C., Hickie, I., 2009. Delivering youth-specific mental health services: The advantages of a collaborative, multi-disciplinary system. Australasian Psychiatry 17, 189–194. Slade, M., Thornicroft, G., Glover, G., 1999. The feasibility of routine outcome measures in mental health. Social Psychiatry and Psychiatric Epidemiology 34, 243–249. Tabachnick, B.G., Fidell, L.S., 1996. Using Multivariate Statistics, 3rd ed. Harper Collins College Publishers, New York. Trauer, T. (Ed.), 2010. Outcome Measurement in Mental Health: Theory and Practice. Cambridge University Press, Cambridge. Turner, S., 2004. Are the health of the nation outcome scales (honos) useful for measuring outcomes in older people’s mental health services? Aging & Mental Health 8, 387–396. Wing, J.K., Beevor, A.S., Curtis, R.H., Park, S.B.G., Hadden, S., Burns, A., 1998. Health of the nation outcome scales (honos). British Journal of Psychiatry 172, 11–18. Wing, J.K., Curtis, R.H., Beevor, A.S., 1999. Health of the nation outcome scales (honos). Glossary for honos score sheet. British Journal of Psychiatry 174, 432–434. Yates, P., Kramer, T., Garralda, M.E., 2006. Use of a routine mental health measure in an adolescent secure unit. British Journal of Psychiatry 188, 583–584.