- Email: [email protected]
Impact of client feedback on clinical outcomes for young people using public mental health services: A pilot study
Psychiatry Research 229 (2015) 617–619
Contents lists available at ScienceDirect
Psychiatry Research journal homepage: www.elsevier.com/locate/psychres
Brief report
Impact of client feedback on clinical outcomes for young people using public mental health services: A pilot study Benjamin Hansen a,n, Annette Howe a, Paul Sutton a, Kevin Ronan b a b
Child & Youth Mental Health, Division of Mental Health, Alcohol and Other Drugs, Mackay Hospital and Health Service, Queensland, Australia CQ University, Rockhampton, Queensland, Australia
art ic l e i nf o
a b s t r a c t
Article history: Received 30 November 2014 Received in revised form 21 January 2015 Accepted 4 May 2015 Available online 12 May 2015
This paper reports on research conducted in public health settings with young people who have serious mental health issues. An easy to use feedback system for clinicians providing psychotherapy was assessed against treatment as usual. Data were collected on four widely used outcome measures. There has been little previous research in this area. We found evidence for treatment effects. There was some evidence supportive of the feedback system but further study is needed. Crown Copyright & 2015 Published by Elsevier Ireland Ltd. All rights reserved.
Keywords: Psychotherapy Feedback Youth
1. Introduction This paper reports on an experiment conducted in a Child & Youth Mental Health Service (CYMHS) in Queensland, Australia. The purpose was to determine whether the use of a simple consumer driven feedback system as an adjunct to routine clinical care led to better outcomes as assessed by common measures for a real world clinical population. To date there has been little research published on this topic. CYMHS is part of the Queensland public health system and is funded to meet the needs of young people with moderate to severe mental health issues. Clients who meet service criteria are allocated to a case manager who provides psychotherapy to clients amongst a range of other tasks. In Australia the Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA) and the Children's Global Assessment Scale (CGAS) are to be completed by clinicians in public settings and the Strengths and Difficulties Questionnaire (SDQ) is offered to gain client and parent perspectives on outcome (Mental Health National Outcomes and Casemix Collection, 2013). The feedback system employed the Outcome Rating Scale (ORS) and the Session Rating Scale (SRS). The ORS (Miller et al., 2003) is a brief measure of client functioning that is sensitive to the effects of psychotherapy. The SRS (Duncan et al., 2003) is a brief measure of the strength of the therapeutic alliance that a therapist has with his or her client. The ORS is administered at the start of a
n
Corresponding author. Tel.: þ 61 7 4968 3893; fax: þ61 7 4968 3894. E-mail address: [email protected] (B. Hansen).
psychotherapy session and the SRS at the end. As part of a feedback-informed service, scores and implications including possible changes to the therapeutic approach are discussed with the client. Sundet (2009) conducted a qualitative study with the ORS and SRS in a Norwegian family therapy unit. Therapists and clients who participated in the study agreed that the measures facilitated collaborative practice. Timimi et al. (2013) reported on the use of the ORS and SRS in a Child & Adolescent Mental Health Service in the UK. The utility of the system was assessed via efficiency data – throughput, numbers of referrals going on to more specialised service and so forth. Results were favourable but did not include data on outcomes.
2. Methods Data for this project were collected with ethics approval between 2009 and 2012. Permission to use the ORS and SRS for research was granted in 2006. Data were initially collected from a treatment as usual group (n¼ 35) and later from an experimental, feedback-enhanced group (n¼ 38) with whom the ORS and SRS were used. The age range for the TAU group was nine to 17. The age range for the ORS group was 10–17. The TAU group was 40% female and 60% male. The ORS group was 50% female and 50% male. The most common diagnoses in both groups were mood and anxiety disorders. CYMHS case managers were instructed in the use of the ORS and SRS as per Duncan et al. (2004, pp. 97–103). Essentially the measures and the rationale for feedback are explained to the client from first contact and are subsequently used to guide sessions. Case manager/therapist and client participation was voluntary. The TAU and ORS group outcomes were collected at the start of treatment and then again at either the first three monthly multidisciplinary case review or at discharge; whichever came first. Subjects in both groups received an average of
http://dx.doi.org/10.1016/j.psychres.2015.05.007 0165-1781/Crown Copyright & 2015 Published by Elsevier Ireland Ltd. All rights reserved.
618
B. Hansen et al. / Psychiatry Research 229 (2015) 617–619
Table 1 Dependent variables. Dependent variable
Pre-treatment mean and S.D.
Post-treatment mean and S.D.
HoNOSCA TAU ORS CGAS TAU ORS SDQP TAU ORS SDQY TAU ORS
23.36/10.05 26.56/8.26 44.55/3.90 44.33/6.84 19.45/10.56 21.56/5.9 19.18/7.19 21.56/5.90
12.82/7.43 11.67/7.09 60.45/12.86 62.11/15.24 20.18/9.1 20/4.56 15.55/6.38 18.33/6.73
a
F value and probability for between Effect size for between subjects subjectsa 4.106(1, 14)/0.017
0.595
Yes
3.169(1, 14)/0.04
0.531
No
16.168(1, 14)/0.001
0.852
No
3.060(1,14)/0.045
0.522
No
Partial eta squared, small effect ¼ 0.01, medium ¼ 0.06, large¼ 0.14.
eight therapy sessions. The ORS and SRS were used on average four times with subjects in the ORS group.
3. Results We assessed that parametric statistics provided the best avenue of detecting relevant effects. Scores on the dependent measures were acceptably close to normally distributed (Shapiro Wilk test, p¼0.05) but pre-treatment score differences necessitated a combination of tests. To assess for change without respect to condition, mixed factorial ANOVA with the HoNOSCA, CGAS and Youth SDQ summary scores revealed significant within subjects effects – HoNOSCA (F¼26.864[1,18], po0.001, partial eta squared¼ 0.599), CGAS (F¼37.442[1,18], po0.001, partial eta squared¼ 0.675) and Youth SDQ (F¼5.869[1,18], p¼0.026, partial eta squared¼0.246). There were no significant effects found on the Parent SDQ (F¼0.213 [1,18], p¼0.650, partial eta squared¼0.012). Then, to assess change as a function of condition, we conducted a multivariate analysis using pre-treatment scores as covariates. The results are detailed in Table 1. The SRS scores were not normally distributed so we used the Wilcoxon signed ranks test to analyse the ORS and SRS data. We found significant differences between first and final summary scores – ORS (Z¼ 4.192, p o0.001, r ¼0.721), SRS (Z¼ 4.285, p o0.001). An analysis of clinically significant change in ORS results (Miller, 2012a) showed this to be true in 16 cases, whilst two cases showed clinically significant deterioration and 16 cases did not meet clinically significant criteria.
4. Discussion Weisz and Gray (2008) assessed that one percent of the studies that comprise the current evidence base for psychotherapeutic treatments for young people involve real world data. The current research therefore has strong external validity. There were however a number of methodological problems. We decided not to use random allocation to one of two simultaneous groups because the ORS and SRS are supposed to be used as part of a “culture of feedback” (Miller, 2012b, p. 29). We thought that it might be difficult for staff to move between conditions in these circumstances. We felt it was important to include SDQ data in our analysis but our rate of return for SDQs at the time of the study was 30%. This is close to the national average (Mental Health National Outcomes and Casemix Collection, 2013) and accurately reflects real world conditions but it lowers the power of our study. 1
Clinically significant and reliable change
Pearson's r, small effect ¼0.1, medium ¼0.3, large¼ 0.5.
The only outcome measure that showed statistically and clinically significant change, the HoNOSCA, is clinician rated. This is again reflective of real world practice. Our clinicians have been assessed as competent raters and the HoNOSCA data they produce closely matches those of peers across the state and the nation. The correlations between HoNOSCA scores, CGAS and SDQs are generally modest. Hanssen-Bauer et al. (2010) report correlations of between 0.35 and 0.64 for the HoNOSCA and the CGAS. They report that the correlations between the HoNOSCA and Parent and Youth SDQs are 0.38 and 0.36 respectively. Overall outcomes in this study were modest but our end measures were collected from clients who were often still in treatment and not ready for discharge. There were three clinicians who were enthusiastic about the study and who contributed most of the clients to both conditions (the proportions were very similar). Staff who participated in a limited way reported that they were busy with core duties and had little time to devote to a research project. This impacted on the number of clients recruited. A related issue is that one half of possible collection occasions with the ORS and SRS were missed. We believe that this was due to clinicians being time poor. Clinicians have a large number of tasks to complete and we have not seen a situation where a case manager was up to date with every task. There were no instances reported of clients refusing the ORS and SRS. The data produced by this study have led to a greater interest in using these measures amongst the larger work group. This study suggests that short-term psychological interventions in public mental health services can assist young people with serious mental health issues. We found an effect for feedback on a clinician rated measure but more research is needed. The most obvious way to do this with methodological rigour would be to have a university working closely with mental health services and supervising protocol adherence and rating outcomes.
Contributions Ben Hansen – project design, ethics approvals, participant therapist, statistics, text preparation. Annette Howe – participant therapist, data coding, text preparation. Paul Sutton – participant therapist, data coding, text preparation. Dr Kevin Ronan – text preparation, statistics.
Acknowledgements CYMHS clients and staff who participated in the study, John Dillon – North Queensland Clinical Improvement Coordinator
B. Hansen et al. / Psychiatry Research 229 (2015) 617–619
(Queensland Health) for data coding assistance, Dr Alison Pighills – Health Practitioner Research Capacity Development Program (Queensland Health/James Cook University) for advice and support, Dr Alistair Campbell for advice on design, Dr Scott Miller for his support and comments. References Duncan, B.L., Miller, S.D., Sparks, J.A., Claud, D.A., Reynolds, L.R., Brown, J., 2003. The Session Rating Scale: preliminary psychometric properties of a working alliance measure. Journal of Brief Therapy 3, 3–12. Duncan, B.L., Miller, S.D., Sparks, J.A., 2004. The Heroic Client: A Revolutionary Way to Improve Effectiveness Through Client-Directed, Outcome-Informed Therapy. Jossey-Bass, CA. Hanssen-Bauer, K., Langsrud, Ø., Kvernmo, S., Heyerdahl, S., 2010. Clinician-rated mental health in outpatient child and adolescent mental health services: association with parent, teacher and adolescent ratings Retrieved from. Child & Adolescent Psychiatry & Mental Health 4, 29 〈http://www.capmh.com/ content/4/1/29〉. Mental Health National Outcomes and Casemix Collection, 2013. NOCC Strategic Directions 2014–2024. National Mental Health Information Development
619
Expert Advisory Panel. Retrieved from 〈http://amhocn.Org/static/files/assets/ 7dc3e26b/NOCC_Strategic_Directions_2014-2024.pdf〉. Miller, S.D., 2012a. ORS Reliable Change Chart. Retrieved from 〈http://www.Slide share.net/scottdmiller/ors-reliable-change-chartn〉. Miller, S.D., 2012b. The Outcome Rating Scale (ORS) and the Session Rating Scale (SRS). Integrating Science and Practice 2 (2), 28–31, Retrieved from〈https:// www.ordrepsy.qc.ca/pdf/2012_11_01_Integrating_SandP_Dossier_06_Miller_ Bargmann_En.pdf〉. Miller, S.D., Duncan, B.L., Brown, J., Sparks, J.A., Claud, D.A., 2003. The Outcome Rating Scale: a preliminary study of the reliability, validity and feasibility of a brief visual analog measure. Journal of Brief Therapy 2, 91–100. Sundet, R., 2009. Client Directed, Outcome Informed Therapy in An Intensive Family Therapy Unit – A Study if The Use of Research Generated Knowledge in Clinical Practice. University of Oslo/Buskerud University College, Retrieved from https:// www.duo.uio.no/bitstream/handle/10852/18579/3/Sundet-publ.pdf. Timimi, S., Tetley, D., Burgoine, W., Walker, G., 2013. Outcome Oriented Child and Adolescent Mental Health Services (OO-CAMHS): A whole service model. Clinical Child Psychology and Psychiatry 18 (2), 169–184. Weisz, J.R., Gray, J.S., 2008. Evidence-based psychotherapies for children and adolescents: data from the present and a model for the future. Child & Adolescent Mental Health 13 (2), 54–65.
Contents lists available at ScienceDirect
Psychiatry Research journal homepage: www.elsevier.com/locate/psychres
Brief report
Impact of client feedback on clinical outcomes for young people using public mental health services: A pilot study Benjamin Hansen a,n, Annette Howe a, Paul Sutton a, Kevin Ronan b a b
Child & Youth Mental Health, Division of Mental Health, Alcohol and Other Drugs, Mackay Hospital and Health Service, Queensland, Australia CQ University, Rockhampton, Queensland, Australia
art ic l e i nf o
a b s t r a c t
Article history: Received 30 November 2014 Received in revised form 21 January 2015 Accepted 4 May 2015 Available online 12 May 2015
This paper reports on research conducted in public health settings with young people who have serious mental health issues. An easy to use feedback system for clinicians providing psychotherapy was assessed against treatment as usual. Data were collected on four widely used outcome measures. There has been little previous research in this area. We found evidence for treatment effects. There was some evidence supportive of the feedback system but further study is needed. Crown Copyright & 2015 Published by Elsevier Ireland Ltd. All rights reserved.
Keywords: Psychotherapy Feedback Youth
1. Introduction This paper reports on an experiment conducted in a Child & Youth Mental Health Service (CYMHS) in Queensland, Australia. The purpose was to determine whether the use of a simple consumer driven feedback system as an adjunct to routine clinical care led to better outcomes as assessed by common measures for a real world clinical population. To date there has been little research published on this topic. CYMHS is part of the Queensland public health system and is funded to meet the needs of young people with moderate to severe mental health issues. Clients who meet service criteria are allocated to a case manager who provides psychotherapy to clients amongst a range of other tasks. In Australia the Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA) and the Children's Global Assessment Scale (CGAS) are to be completed by clinicians in public settings and the Strengths and Difficulties Questionnaire (SDQ) is offered to gain client and parent perspectives on outcome (Mental Health National Outcomes and Casemix Collection, 2013). The feedback system employed the Outcome Rating Scale (ORS) and the Session Rating Scale (SRS). The ORS (Miller et al., 2003) is a brief measure of client functioning that is sensitive to the effects of psychotherapy. The SRS (Duncan et al., 2003) is a brief measure of the strength of the therapeutic alliance that a therapist has with his or her client. The ORS is administered at the start of a
n
Corresponding author. Tel.: þ 61 7 4968 3893; fax: þ61 7 4968 3894. E-mail address: [email protected] (B. Hansen).
psychotherapy session and the SRS at the end. As part of a feedback-informed service, scores and implications including possible changes to the therapeutic approach are discussed with the client. Sundet (2009) conducted a qualitative study with the ORS and SRS in a Norwegian family therapy unit. Therapists and clients who participated in the study agreed that the measures facilitated collaborative practice. Timimi et al. (2013) reported on the use of the ORS and SRS in a Child & Adolescent Mental Health Service in the UK. The utility of the system was assessed via efficiency data – throughput, numbers of referrals going on to more specialised service and so forth. Results were favourable but did not include data on outcomes.
2. Methods Data for this project were collected with ethics approval between 2009 and 2012. Permission to use the ORS and SRS for research was granted in 2006. Data were initially collected from a treatment as usual group (n¼ 35) and later from an experimental, feedback-enhanced group (n¼ 38) with whom the ORS and SRS were used. The age range for the TAU group was nine to 17. The age range for the ORS group was 10–17. The TAU group was 40% female and 60% male. The ORS group was 50% female and 50% male. The most common diagnoses in both groups were mood and anxiety disorders. CYMHS case managers were instructed in the use of the ORS and SRS as per Duncan et al. (2004, pp. 97–103). Essentially the measures and the rationale for feedback are explained to the client from first contact and are subsequently used to guide sessions. Case manager/therapist and client participation was voluntary. The TAU and ORS group outcomes were collected at the start of treatment and then again at either the first three monthly multidisciplinary case review or at discharge; whichever came first. Subjects in both groups received an average of
http://dx.doi.org/10.1016/j.psychres.2015.05.007 0165-1781/Crown Copyright & 2015 Published by Elsevier Ireland Ltd. All rights reserved.
618
B. Hansen et al. / Psychiatry Research 229 (2015) 617–619
Table 1 Dependent variables. Dependent variable
Pre-treatment mean and S.D.
Post-treatment mean and S.D.
HoNOSCA TAU ORS CGAS TAU ORS SDQP TAU ORS SDQY TAU ORS
23.36/10.05 26.56/8.26 44.55/3.90 44.33/6.84 19.45/10.56 21.56/5.9 19.18/7.19 21.56/5.90
12.82/7.43 11.67/7.09 60.45/12.86 62.11/15.24 20.18/9.1 20/4.56 15.55/6.38 18.33/6.73
a
F value and probability for between Effect size for between subjects subjectsa 4.106(1, 14)/0.017
0.595
Yes
3.169(1, 14)/0.04
0.531
No
16.168(1, 14)/0.001
0.852
No
3.060(1,14)/0.045
0.522
No
Partial eta squared, small effect ¼ 0.01, medium ¼ 0.06, large¼ 0.14.
eight therapy sessions. The ORS and SRS were used on average four times with subjects in the ORS group.
3. Results We assessed that parametric statistics provided the best avenue of detecting relevant effects. Scores on the dependent measures were acceptably close to normally distributed (Shapiro Wilk test, p¼0.05) but pre-treatment score differences necessitated a combination of tests. To assess for change without respect to condition, mixed factorial ANOVA with the HoNOSCA, CGAS and Youth SDQ summary scores revealed significant within subjects effects – HoNOSCA (F¼26.864[1,18], po0.001, partial eta squared¼ 0.599), CGAS (F¼37.442[1,18], po0.001, partial eta squared¼ 0.675) and Youth SDQ (F¼5.869[1,18], p¼0.026, partial eta squared¼0.246). There were no significant effects found on the Parent SDQ (F¼0.213 [1,18], p¼0.650, partial eta squared¼0.012). Then, to assess change as a function of condition, we conducted a multivariate analysis using pre-treatment scores as covariates. The results are detailed in Table 1. The SRS scores were not normally distributed so we used the Wilcoxon signed ranks test to analyse the ORS and SRS data. We found significant differences between first and final summary scores – ORS (Z¼ 4.192, p o0.001, r ¼0.721), SRS (Z¼ 4.285, p o0.001). An analysis of clinically significant change in ORS results (Miller, 2012a) showed this to be true in 16 cases, whilst two cases showed clinically significant deterioration and 16 cases did not meet clinically significant criteria.
4. Discussion Weisz and Gray (2008) assessed that one percent of the studies that comprise the current evidence base for psychotherapeutic treatments for young people involve real world data. The current research therefore has strong external validity. There were however a number of methodological problems. We decided not to use random allocation to one of two simultaneous groups because the ORS and SRS are supposed to be used as part of a “culture of feedback” (Miller, 2012b, p. 29). We thought that it might be difficult for staff to move between conditions in these circumstances. We felt it was important to include SDQ data in our analysis but our rate of return for SDQs at the time of the study was 30%. This is close to the national average (Mental Health National Outcomes and Casemix Collection, 2013) and accurately reflects real world conditions but it lowers the power of our study. 1
Clinically significant and reliable change
Pearson's r, small effect ¼0.1, medium ¼0.3, large¼ 0.5.
The only outcome measure that showed statistically and clinically significant change, the HoNOSCA, is clinician rated. This is again reflective of real world practice. Our clinicians have been assessed as competent raters and the HoNOSCA data they produce closely matches those of peers across the state and the nation. The correlations between HoNOSCA scores, CGAS and SDQs are generally modest. Hanssen-Bauer et al. (2010) report correlations of between 0.35 and 0.64 for the HoNOSCA and the CGAS. They report that the correlations between the HoNOSCA and Parent and Youth SDQs are 0.38 and 0.36 respectively. Overall outcomes in this study were modest but our end measures were collected from clients who were often still in treatment and not ready for discharge. There were three clinicians who were enthusiastic about the study and who contributed most of the clients to both conditions (the proportions were very similar). Staff who participated in a limited way reported that they were busy with core duties and had little time to devote to a research project. This impacted on the number of clients recruited. A related issue is that one half of possible collection occasions with the ORS and SRS were missed. We believe that this was due to clinicians being time poor. Clinicians have a large number of tasks to complete and we have not seen a situation where a case manager was up to date with every task. There were no instances reported of clients refusing the ORS and SRS. The data produced by this study have led to a greater interest in using these measures amongst the larger work group. This study suggests that short-term psychological interventions in public mental health services can assist young people with serious mental health issues. We found an effect for feedback on a clinician rated measure but more research is needed. The most obvious way to do this with methodological rigour would be to have a university working closely with mental health services and supervising protocol adherence and rating outcomes.
Contributions Ben Hansen – project design, ethics approvals, participant therapist, statistics, text preparation. Annette Howe – participant therapist, data coding, text preparation. Paul Sutton – participant therapist, data coding, text preparation. Dr Kevin Ronan – text preparation, statistics.
Acknowledgements CYMHS clients and staff who participated in the study, John Dillon – North Queensland Clinical Improvement Coordinator
B. Hansen et al. / Psychiatry Research 229 (2015) 617–619
(Queensland Health) for data coding assistance, Dr Alison Pighills – Health Practitioner Research Capacity Development Program (Queensland Health/James Cook University) for advice and support, Dr Alistair Campbell for advice on design, Dr Scott Miller for his support and comments. References Duncan, B.L., Miller, S.D., Sparks, J.A., Claud, D.A., Reynolds, L.R., Brown, J., 2003. The Session Rating Scale: preliminary psychometric properties of a working alliance measure. Journal of Brief Therapy 3, 3–12. Duncan, B.L., Miller, S.D., Sparks, J.A., 2004. The Heroic Client: A Revolutionary Way to Improve Effectiveness Through Client-Directed, Outcome-Informed Therapy. Jossey-Bass, CA. Hanssen-Bauer, K., Langsrud, Ø., Kvernmo, S., Heyerdahl, S., 2010. Clinician-rated mental health in outpatient child and adolescent mental health services: association with parent, teacher and adolescent ratings Retrieved from. Child & Adolescent Psychiatry & Mental Health 4, 29 〈http://www.capmh.com/ content/4/1/29〉. Mental Health National Outcomes and Casemix Collection, 2013. NOCC Strategic Directions 2014–2024. National Mental Health Information Development
619
Expert Advisory Panel. Retrieved from 〈http://amhocn.Org/static/files/assets/ 7dc3e26b/NOCC_Strategic_Directions_2014-2024.pdf〉. Miller, S.D., 2012a. ORS Reliable Change Chart. Retrieved from 〈http://www.Slide share.net/scottdmiller/ors-reliable-change-chartn〉. Miller, S.D., 2012b. The Outcome Rating Scale (ORS) and the Session Rating Scale (SRS). Integrating Science and Practice 2 (2), 28–31, Retrieved from〈https:// www.ordrepsy.qc.ca/pdf/2012_11_01_Integrating_SandP_Dossier_06_Miller_ Bargmann_En.pdf〉. Miller, S.D., Duncan, B.L., Brown, J., Sparks, J.A., Claud, D.A., 2003. The Outcome Rating Scale: a preliminary study of the reliability, validity and feasibility of a brief visual analog measure. Journal of Brief Therapy 2, 91–100. Sundet, R., 2009. Client Directed, Outcome Informed Therapy in An Intensive Family Therapy Unit – A Study if The Use of Research Generated Knowledge in Clinical Practice. University of Oslo/Buskerud University College, Retrieved from https:// www.duo.uio.no/bitstream/handle/10852/18579/3/Sundet-publ.pdf. Timimi, S., Tetley, D., Burgoine, W., Walker, G., 2013. Outcome Oriented Child and Adolescent Mental Health Services (OO-CAMHS): A whole service model. Clinical Child Psychology and Psychiatry 18 (2), 169–184. Weisz, J.R., Gray, J.S., 2008. Evidence-based psychotherapies for children and adolescents: data from the present and a model for the future. Child & Adolescent Mental Health 13 (2), 54–65.