- Email: [email protected]
Does the Veterans Affairs Depression Performance Measure Predict Quality Care?
The Joint Commission Journal on Quality and Patient Safety Performance Measures
Does the Veterans Affairs Depression Performance Measure Predict Quality Care? Brian Shiner, M.D., M.P.H.; Bradley V. Watts, M.D., M.P.H.; Marcy K. Traum, M.D.; Samuel J. Huber, M.D., M.P.H.; Yinong Young-Xu, Sc.D.
D
epression is a common illness among veterans treated in the Department of Veterans Affairs (VA) medical centers (VAMCs). Examination of electronic medical records (EMRs) at a VA primary care clinic revealed that 5%–7% of patients had a diagnosis of depression,1 although the prevalence of depression among VA outpatients was found to be as high as 31% in a comprehensive screening program.2 Depression and related psychiatric disability continue to affect many soldiers returning from current conflicts in Iraq and Afghanistan.3,4 In addition to complicating the treatment of other neuropsychiatric difficulties common among veterans, such as posttraumatic stress disorder and persistent postconcussive syndrome,5,6 depression can complicate the care of chronic medical conditions, with worse outcomes and increased costs.7 Depression is associated with low adherence to cardiac medications in patients with coronary artery disease,8 and higher depression severity is associated with poor diet and low adherence to oral hypoglycemic medications in patients with diabetes.9 Fortunately, depression treatment can ameliorate these problems. Chronic disease management programs that identify and treat depression in a structured, proactive manner have been found to be efficacious10 and cost-effective.11 In addition to promoting the effective treatment of depression, these programs have reduced mortality in depressed patients with comorbid diabetes or cancer.12, 13 Although the mechanism by which effective depression treatment improves medical mortality is unknown, it is clear that patients with depression who adhere to their treatment regimen for depression are more likely to adhere to their treatment regimen for medical illnesses such as coronary artery disease, dislipidemia, and diabetes.14 Given the high prevalence and serious consequences of depression among veterans treated in the VA, the identification and evidencebased treatment of depression have become a high priority, with significant investments made in the adaptation and spread of evidence-based models of care for VA treatment settings.15 170
April 2011
Article-at-a-Glance Background: In 2000, the Department of Veterans
Affairs (VA) instituted a performance measure to improve the quality of depression care. The measure evaluated adequacy of follow-up for depressed patients but was removed from clinic directors’ performance plans in fiscal year (FY) 2009 because it had not been empirically validated. The VA depression performance measure was compared with an empirically validated model for assessing adherence to important depression treatment processes. Methods: VA medical centers (VAMCs) whose performance on the VA depression measure was in the top or bottom quartile nationally for all four quarters in FY2008 were selected for inclusion. A blinded interviewer attempted to contact clinical directors of both primary care and mental health at each VAMC and conducted telephone interviews using a protocol designed to employ the 3-Component Model (3CM) fidelity measure, which assesses domains of evidence-based depression care. Results: There were 9 sites in the “high-performing” group and 10 sites in the “low-performing” group. At least one interview was completed at 8 of the 9 sites in the highperforming group and 9 of the 10 sites in the low-performing group. There was a significant difference in the percentage of patients meeting the VA depression performance measure between the high- and low-performing groups (47.5% versus 14.7%; 2 = 837.5, p < .001). The adapted version of the 3CM fidelity scale detected a significant difference in process of depression care between the high- and low-performing sites (82.3 versus 71.4; z =2.4, p = .018). Conclusions: The highest-performing sites on the VA depression performance measure adhered to important care processes more often than did the lowest-performing sites.
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Accordingly, implementing a useful measure of depression care quality is important to systemwide improvement. To track the quality of depression care at VAMCs, a depression performance measure was introduced in 2000.16 This measure was designed to assess the adequacy of treatment for patients newly diagnosed with depression and treated with antidepressant medications. It was assessed using automated collection from the EMR for the 12 weeks following a new diagnosis.17 Adequacy was defined in terms of two separate items, each of which employed the same denominator: (1) the percentage of patients who received at least three follow-up encounters and (2) the percentage of patients who received a 90-day medication supply. For the follow-up portion, three encounters had to be for mental health purposes; up to two could be over the telephone, and one had to be with a practitioner who could write prescriptions. Patients could be diagnosed or treated by mental health, primary care, or both services. The VA depression performance measure was patterned after a Healthcare Effectiveness Data and Information Set (HEDIS) measure implemented in the private sector at the same time.18 The HEDIS measure also measured the adequacy of medication prescription and follow-up in the 84 days following a new diagnosis of depression (acute phase) but had an additional component measuring whether patients continued on their medications for 180 days (continuation phase). The VA depression performance measure was removed from primary care and mental health directors’ performance plans in FY2009 because of concerns that it had not been empirically validated, as well as ongoing concerns that the measure was not appropriate for chronically ill patients in the VA population.19 However, there may have been organizational benefit. In a retrospective analysis examining the relationship between adequate follow-up and mortality, Cully et al. showed that the measure was associated with a gradual improvement in the adequacy of follow-up but with no change in adequacy of medication supply from FY2000 to FY2005.20 Using national VA administrative data sets, they considered adequate follow-up to be three visits in the 84 days subsequent to a new diagnosis of depression where there was either (1) a current procedural technology code indicative of a mental health visit or (2) a diagnosis of depression associated with a non–mental health visit. Although they determined the adequacy of follow-up differently than this study, they propose that the improvements in adequacy of follow-up were related to the adoption of the VA depression performance measure. The VA depression performance measure was assessed by the
April 2011
VA Office of Quality and Performance and reported quarterly at the VAMC level. Leaders at individual VAMCs could, in turn, drill down to the list of individual patients who had been included in the measure to analyze their performance. The measure included the percentage of patients who met the adequacy of the follow-up standard and the percentage of patients who met the adequacy of the medication supply standard rather than the percentage of patients who met both standards simultaneously. Therefore, because we used the central performance measure reporting system to obtain our data, we were unable to meaningfully combine the two portions of the measure, as Culley et al. did, using raw administrative data.20 We chose to focus on the follow-up portion of the measure for the following three reasons: 1. There was a greater range of performance, permitting greater discrimination between high and low performers. 2. VAMCs did more poorly on this part, so we thought that lessons on how to do better would be more important. 3. We were interested in active clinical processes that followed the prescription of antidepressants rather than the prescriptions themselves. Although adequacy of follow-up is an important component of optimal care for depression, additional clinical processes have been shown to improve depression outcomes. Oxman and colleagues developed a 10-item fidelity measure to assess the performance of primary care depression programs in delivering effective care.21 Practices scoring highly on this scale—the 3Component Model (3CM) fidelity scale—obtained consistently better outcomes for depressed patients (higher response and remission rates). Whereas Oxman and colleagues’ model has three components, the fidelity scale has 10 items, which represent 10 important processes associated with clinical improvement in the related randomized trial of the 3CM model.22 We sought to compare performance on the follow-up portion of the VA measure with performance on the 3CM fidelity measure to gain an understanding of whether assessing adequacy of follow-up alone is a useful proxy for other effective depression management practices. Our goal was to establish a link between the score on the VA depression performance measure and improved outcomes (Figure 1, page 172). We hypothesized that VAMCs that most frequently met the three-encounter follow-up measure would also follow other procedures outlined by the 3CM fidelity scale for effective care of depression. We further hypothesized that the opposite would be true at VAMCs that least frequently met the three-encounter follow-up measure.
Volume 37 Number 4
Copyright 2011 © The Joint Commission
171
The Joint Commission Journal on Quality and Patient Safety Model for Evaluation
Figure 1. The goal of the study was to establish a link between the score on the Department of Veterans Affairs (VA) depression performance measure and improved outcomes; Dietrich 2004 represents reference 22; Oxman 2006, reference 21. 3CM, 3-component model.
Methods ETHICAL ISSUES This project was reviewed and approved by the Dartmouth Medical School Committee for the Protection of Human Subjects (CPHS No. 21499) and by the White River Junction VAMC Research and Development Committee.
CLASSIFICATION OF SITES Performance measure data were obtained from the Veterans Health Administration (VHA) Support Service Center Web site.17 We attempted to identify 10 consistently highperforming facilities and 10 consistently low-performing facilities. We selected VAMCs that were in either the top or bottom quartile nationally for all four quarters on the VA depression performance measure in FY 2008. This yielded 9 “high-performing” sites and 10 “low-performing” sites. We considered using the yearly mean to identify sites, but our method of using quartiles resulted in the identification of more stable groups of high and low performers. For example, more inclusive criteria such as selecting the top 10% of sites for the high-performing group would have meant that some “top sites” would have met the performance measure less than 40% of the time in one or more quarters. To find individuals who would best understand the overall process of depression care at their respective sites, we used a yearly directory23 to identify the clinical service line chiefs for both mental health and primary care at each of the relevant VAMCs. Service line chiefs are senior clinicians who are 172
April 2011
responsible for clinical care delivered by their disciplinary colleagues. Primary care and mental health service line chiefs shared responsibility for meeting the depression performance measure at the medical center level. We obtained verbal consent to perform audiotaped telephone interviews. An interviewer who was blinded to whether sites were in the high- or lowperforming group [M.K.T.] conducted semistructured interviews separately with both the primary care service chief and the mental health service chief at each hospital. Interviews were conducted in March and April 2009; FY2008 data were not fully reported until January 2009.
DEVELOPMENT OF
THE
ASSESSMENT PROTOCOL
The 3CM fidelity measure is scored from 0 to 100, with different weights for each item. The measure was adapted for use in our interview in the following ways: 1. Item 10, which related to follow-up, was removed to avoid overlap with follow-up as measured by the VA depression performance measure. 2. To adapt this patient-level tool to suit an institutionwide process assessment approach, we removed item 8, which required the review of individual patient logs. 3. We split item six, which related to standardized reassessment and subsequent treatment changes into two separate items to make the process of care more explicit. 4. We reordered the remaining items into interview domains that could be addressed in a logical order using case scenarios (see page 173).
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Table 1. Semistructured Interview Domains Interview Domain
Weight
3CM Fidelity Item
Using the Patient Health Questionnaire-9 (PHQ-9) or similar tool for diagnosis and baseline severity
18.33
1
D2
Using the PHQ-9 or similar tool to monitor treatment response
8.97
6
D3
Determining the adequacy of initial treatment response at four weeks and making alterations to treatment if necessary
8.97
6
D4
Continuing to make treatment recommendations to the point of remission
13.09
7
D5
Asking focused questions about suicide risk
8.91
2
D6
Providing educational materials
7.29
3
D7
Identifying self-management goals
12.92
4
D8
Making follow-up calls to assess treatment barriers
14.84
5
D9
Communication between primary care and mental health providers
6.67
9
D1
Descriptor
To account for items that were removed, we scaled up the domains proportionally to keep the total possible score at 100. The resulting nine domains are presented in Table 1 (above). The 3CM fidelity measure was designed for assessment of case management logs in a randomized trial of depression management in primary care.22 To adapt this scale for our study, we used case scenarios modeled after the Joint Commission “tracer” methodology to frame our semistructured interview.24 We used an iterative process to develop our final scenarios; the interviewer practiced early versions with colleagues at VAMCs not included in the study and we incorporated their feedback. The first case scenario, which assessed facilities’ systems of depression care in the primary care setting, was reviewed with both the primary care service chief and the mental health service chief to generate data to examine our primary hypothesis. Because patients cared for in the mental health service were also included in the VA follow-up measure, we developed a second case scenario that was reviewed only with the mental health service chief. The same domains were assessed, but because some 3CM domains were not highly relevant to mental health care, this part of the interview was used simply to gain a better understanding of individual sites in relevant domains and for hypothesis generation. After the interviews, the blinded interviewer reviewed the tapes and scored the individual domains using predetermined decision rules. Questionable items were resolved with group coding of the interview tapes [M.K.T., B.S., B.V.W.]. When we spoke with both the primary care service line chief and the mental health service line chief about the process of depression care in primary care and they disagreed on individual domains, we resolved differences by reviewing tran-
April 2011
scripts and tapes and taking the answer that was more fully explained. The case scenarios with domain scoring are presented in Appendix 1 (available in online article).
STATISTICAL ANALYSIS High- and low-performing groups on the VA three-visit depression follow-up measure were compared using a 2 test. For the primary care case scenario, comparison between the high- and low-performing groups on the adapted 3CM fidelity scale was made using a Wilcoxon Signed-Rank Test for matched, nonparametric data (means of individual item scores were matched). Comparisons between high- and low-performing groups in individual domains were made using Wilcoxon Rank-Sum Tests for unmatched, nonparametric data (individual scores in the top group were compared to individual scores in the bottom group for each item). All statistical analyses were performed with STATA version 10.1 (Stata Corp, L.P.; College Station, Texas). For the mental health case scenario, we simply looked at adherence to specific domains that were problematic in the primary care case scenario to determine if deficits were consistent across treatment settings.
Results INTERVIEW PARTICIPATION We interviewed at least one clinical service line chief about systems of depression care in the primary care setting at 8 out of the 9 sites in the high-performing group and at 9 out of the 10 sites in the low-performing group. The site in the highperforming group that declined to consent to interview
Volume 37 Number 4
Copyright 2011 © The Joint Commission
173
The Joint Commission Journal on Quality and Patient Safety expressed concern that the data we collected might attract undue scrutiny. The site in the low-performing group that declined to consent to interview expressed concerns about participating in research; there were no affiliated research groups or Institutional Review Board facilities at their site with whom they could discuss the merits and ethics of participating in research. High-Performing Group. In the high-performing group, we obtained complete information from both the primary care service line chief and the mental health service line chief at five out of the remaining eight sites. Among these sites, data were incomplete for the following reasons: 1. The primary care service line chief indicated that he did not know about depression care beyond the screening program because all depression cases were referred directly to mental health (one site). 2. The mental health service line chief did not know about depression care in primary care other than the fact that there was a screening program (one site). 3. The mental health service line chief could not address two domains: alterations in treatment due to nonresponse and making recommendations to the point of remission (one site). Low-Performing Group. We obtained complete information from both the primary care service line chief and the mental health service line chief in only one of the nine sites in the lowperforming group. Among these sites, data were incomplete for the following reasons: 1. The primary care service line chief was interviewed, but the mental health service line chief did not respond to multiple requests for an interview (one site). 2. The mental health service line chief was interviewed, but the primary care service line chief did not respond to multiple requests for interviews (one site). 3. The primary care service line chief indicated that he did not know about depression care beyond the screening program because all depression cases were referred directly to mental health (one site). 4. The mental health service line chief did not know about depression care in primary care other than the fact that there was a screening program (one site). 5. The mental health service line chief did not know about depression care in primary care other than the fact that there was a screening program and that there was not a standardized method to monitor patient progress (one site). 6. The mental health service line chief could not address one domain—alterations in treatment due to nonresponse (one site) and providing educational materials (one site). 174
April 2011
Primary Study Results
* p < .001 in 2 test comparing top and bottom groups †
p < .05 in Wilcoxon Signed Rank Test comparing top and bottom groups
Figure 2. There was a significant difference in the percentage of patients meeting the adequacy of follow-up on the Department of Veterans Affairs depression performance measure between the high- and low-performing groups, and the adapted version of the 3-Component Model (3CM) fidelity scale detected a significant difference in the process of depression treatment between the highand low-performing groups.
Agreement Between Primary Care and Mental Health Clinical Directors. At the 6 sites where complete information was obtained from both interviewees, there was high degree of agreement on the reports between the primary care and mental health clinical directors at each site (Kappa, 0.74). Primary care and mental health service line chiefs’ responses were combined into a complete data set for the 17 sites where we completed at least one interview. As a result, all domains contain some information on the basis of the report of a single interviewee.
DISCRIMINATION BETWEEN GROUPS Figure 2 (above) displays the primary study results. Not surprisingly, there was a significant difference in the percentage of patients meeting the adequacy of follow-up on the VA depression performance measure between the high- and lowperforming groups, given that this was the basis of their grouping (47.5% versus 14.7%; 2 = 837.5, p < .001). Our adapted version of the 3CM fidelity scale detected a significant difference in process of depression treatment between the high- and low-performing groups (82.3 versus 71.4; z =2.4, p = .018).
PROCESS OF CARE: PRIMARY CARE When individual domains were examined (Figure 3, page 175) clinics in the high-performing group consistently adhered
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Positive Responses: Depression Treatment in Primary Care for Nine Dimensions of Evidence-Based Depression Care
Figure 3. Clinics in the high-performing group consistently adhered better than clinics in the low-performing group, but the differences were not statistically significant. D, dimension; MH, mental health; PC, primary care.
better than clinics in the low-performing group, but the differences were not statistically significant, partially because of small sample size. Overall, the largest differences between high- and low-performing groups were in monitoring progress with a standardized depression measure and in altering treatment due to nonresponse. Half or fewer sites in both groups indicated that recommendations were made to the point of remission. The sites consistently performed well in other domains, such as depression screening, suicide assessment, identifying self-management goals, and providing educational materials.
PROCESS OF CARE: MENTAL HEALTH Eight mental health service chiefs in the high-performing group and eight mental health service chiefs in the lowperforming group were interviewed using the second tracer, which assessed depression care provided in mental health clinics. There were no consistent performance patterns between high- and low-performing groups. Interestingly, the greatest problems in primary care clinics were also the consistent deficits in mental health clinics; at the 16 sites, only 3 (19%) indicated that they were monitoring progress with a standardized depression measure, only 5 (31%) indicated that they altered treatment due to nonresponse, and only 8 (50%) indicated that they made recommendations to the point of remission.
April 2011
Discussion The highest-performing sites on the VA depression performance measure adhered to important care processes more often than did the lowest-performing sites. Therefore, this measure may be a useful proxy for other important aspects of depression treatment. Sites in the high-performing group consistently performed better than sites in the low-performing group, especially in their use of a standardized depression measure to monitor progress and in making alterations to treatment on the basis of treatment response. Another important finding was that in both groups, remission was commonly not identified as a goal of depression treatment. This indicates that leaders frequently did not have a conceptual model of what recovery from depression looks like and how to measure it. However, even sites in the low-performing group performed well on many domains; this work identifies the few domains that are areas of consistent deficit that may require more focus. In previous work using the 3CM scale, differences of 2 points resulted in clinically significant improvements in outcomes, and differences ranging from 71 to 82 would suggest clinically robust differences in depression outcomes.21 Although depression outcomes are not collected in a uniform fashion across VAMCs, this indicates that sites in the high-performing group may have better outcomes than sites in the lowperforming group. However, previous work using the 3CM scale evaluated individual patients’ care and related it to out-
Volume 37 Number 4
Copyright 2011 © The Joint Commission
175
The Joint Commission Journal on Quality and Patient Safety comes, whereas we applied the scale to a service as a whole. This interpretation must be made with caution because this is a new application of the 3CM scale. The greatest limitation to this study was the small sample size. This was related to the overall poor performance on the follow-up portion of the VA depression performance measure and paucity of consistently high performers. Expanding the high-performing group was not possible; doing so would have added sites without excellence on this measure to the group. A second major limitation is that we only spoke with two practitioners, at most, from each site. Those practitioners shared responsibility for the quality of depression care at their sites, and there was a high degree of agreement when they were able to answer all the questions about treatment in primary care clinics. However, it is possible that interviewing individual providers or patients may have yielded different results. At some sites, especially in the low-performing group, leaders had less shared knowledge about the process of depression care in primary care. It is especially interesting to note that we were able to obtain complete information with a high level of agreement from both the primary care service line chief and the mental health service line chief at five of the eight sites that we interviewed in the high-performing group but at only one of the nine sites that we interviewed in the low-performing group. It appears that service chiefs at high-performing VAMCs are more likely to have a shared conceptual model of the process of depression care in primary care. Concordance between primary care and mental health service line chiefs itself could affect quality of depression care in the primary care setting, creating systematic bias in our study—the assessments would be most accurate at the best VAMCs and least accurate at the worst VAMCs. The third major limitation was that clinical processes were evaluated with phone interviews rather than with site visits. Although site visits may have yielded richer observations and enabled us to adhere strictly to the tracer methodology, phone interviews did permit us to follow a standardized protocol. Ideally, a method to objectively measure the other components of successful care of depression using queries of the EMR could be developed, thereby avoiding the need for either interviews or site visits. The findings of this study should be seen as a preliminary attempt to understand what the VA depression performance measure tells us about depression care at individual VAMCs. Other studies have compared patient-level processes and outcomes (such as the VA depression performance measure, which is assessed on each patient with a new diagnosis of depression) 176
April 2011
with facility-level quality data (such our design of using service line chiefs as representative agents instead of individual provider and patient interviews or chart reviews). These studies have yielded mixed results in using this method to evaluate other measures of quality, as was seen in three recent studies correlating hospital-level quality of care with mortality.25–27 Although our interviews revealed a high level of system awareness and systems-based practice in primary care clinics, the mental health clinics were notably less systematized and processes were more difficult to describe; frequently, mental health service line chiefs noted that depression care in mental health was “provider-dependent.” This was even true at sites doing well on the performance measure. Although this is an interesting finding that deserves more attention, one must remember that comparison of mental health and primary care settings were not the primary purpose or design of this study. Therefore, further confirmation is indicated, and generalization beyond the 16 mental health clinics that were assessed in this study is not appropriate. This is an area for future exploration, either in the development of new, more systematic mental health models or adaptation of systematic depression care management programs developed in primary care to mental health settings.
Conclusions The highest-performing sites on the follow-up portion of the VA depression performance measure adhere to important care processes more often than the lowest-performing sites. Therefore, this measure may be a useful proxy for other important aspects of depression treatment. Although this measure is no longer used to assess clinic directors’ performance, the data are still collected systematically and are available to individual VAMCs for use in local improvement activities. Our work suggests that primary care clinics wishing to improve their depression treatment performance should consider using the data, which continue to be available as a “supporting measure” as part of a balanced scorecard to assess their gains.* J The authors thank Pamela W. Lee, Ph.D., for her review of an early version of this manuscript, and Thomas E. Oxman, M.D., and Joseph Francis, M.D., M.P.H., for their review of later versions. The views expressed in this article do not necessarily represent the views of the Department of Veterans Affairs (VA) or the United States government. This project was reviewed and approved by Dartmouth Medical School’s Committee for the Protection of Human Subjects, Hanover, NH, USA (CPHS No. 21499), and by the White River Junction VAMC Research and Development Committee.
* Veterans Health Administration Office of Quality and Performance: Measure Master. http://vaww.pdw.med.va.gov/MeasureMaster/MMIndex.asp (last accessed Mar. 2, 2011; available only to members).
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Online-Only Content
8
See the online version of this article for Appendix 1. Interview Protocols
Brian Shiner, M.D., M.P.H., is a Psychiatrist, Department of Veterans Affairs (VA) Medical Center, White River Junction, Vermont, and Assistant Professor, Dartmouth Medical School, Hanover, New Hampshire, and The Dartmouth Institute for Health Policy and Clinical Practice, Lebanon, New Hampshire. Bradley V. Watts, M.D., M.P.H., is Fellowship Director, VA–National Center for Patient Safety Field Office, White River Junction, and Assistant Professor of Psychiatry, Dartmouth Medical School. Marcy K. Traum, M.D., is a Resident in Psychiatry, Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire. Samuel J. Huber, M.D., M.P.H., formerly Fellow, VA National Quality Scholars Program, White River Junction, is Associate Medical Director for Inpatient Psychiatry and Senior Instructor of Psychiatry, University of Rochester School of Medicine and Dentistry, Rochester, New York. Yinong Young-Xu, Sc.D., is a Statistician, VA National Center for Patient Safety Field Office, White River Junction, and Assistant Professor of Psychiatry, Dartmouth Medical School. Please address request for reprints to Brian Shiner, [email protected].
References 1. Kirchner J.E., Curran G.M., Aikens J.: Datapoints: Detecting depression in VA primary care clinics. Psychiatr Serv 55:350, Apr. 2004. 2. Hankin C.S., et al.: Mental disorders and mental health treatment among U.S. Department of Veterans Affairs outpatients: The Veterans Health Study. Am J Psychiatry 156:1924–1930, Dec. 1999. 3. Fontana A., Rosenheck R.: Treatment-seeking veterans of Iraq and Afghanistan: Comparison with veterans of previous wars. J Nerv Ment Dis 196:513–521, Jul. 2008. 4. Milliken C.S., Auchterlonie J.L., Hoge C.W.: Longitudinal assessment of mental health problems among active and reserve component soldiers returning from the Iraq war. JAMA 298:2141–2148, Nov. 14, 2007. 5. Hoge C.W., Goldberg H.M., Castro C.A.: Care of war veterans with mild traumatic brain injury—Flawed perspectives. N Engl J Med 360:1588–1591, Apr. 16, 2009. 6. Stein M.B., McAllister T.W.: Exploring the convergence of posttraumatic stress disorder and mild traumatic brain injury. Am J Psychiatry 166:768–776, Jul. 2009. 7. Donohue J.M., Pincus H.A.: Reducing the societal burden of depression: A review of economic costs, quality of care and effects of treatment. Pharmacoeconomics 25:7–24, Jan. 2010. 8. Gehi A., et al.: Depression and medication adherence in outpatients with coronary heart disease: Findings from the Heart and Soul Study. Arch Intern Med 165:2508–2513, Nov. 28, 2005. 9. Ciechanowski P.S., Katon W.J., Russo J.E.: Depression and diabetes: Impact of depressive symptoms on adherence, function, and costs. Arch Intern Med 160:3278–3285, Nov. 27, 2000. 10. Oxman T.E., Dietrich A.J., Schulberg H.C.: Evidence-based models of integrated management of depression in primary care. Psychiatr Clin North Am 28:1061–1077, Dec. 2005.
April 2011
11. Wang P.S., Simon G.E., Kessler R.C.: Making the business case for enhanced depression care: The National Institute of Mental Health–Harvard Work Outcomes Research and Cost-effectiveness Study. J Occup Environ Med 50:468–475, Apr. 2008. 12. Bogner H.R., et al.: Diabetes, depression, and death: A randomized controlled trial of a depression treatment program for older adults based in primary care (PROSPECT). Diabetes Care 30:3005–3010, Dec. 2007. 13. Gallo J.J., et al.: The effect of a primary care practice-based depression intervention on mortality in older adults: A randomized trial. Ann Intern Med 146:689–698, May 15, 2007. 14. Katon W., et al.: Impact of antidepressant drug adherence on comorbid medication use and resource utilization. Arch Intern Med 165:2497–2503, Nov. 28, 2005. 15. Smith J.L., et al.: Developing a national dissemination plan for collaborative care for depression: QUERI Series. Implement Sci 3, Dec. 21, 2008. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631596/?tool=pubmed (last accessed Feb. 17, 2011). 16. Department of Veterans Affairs, Management of MDD [Major Depressive Disorder] Working Group: Clinical Practice Guideline: Management of Major Depressive Disorder. Pub. No. 10Q-CPG/MDD-00. Washington, DC: May 2009. http://www.healthquality.va.gov/mdd/mdd_full09_c.pdf (last accessed Feb. 17, 2011). 17. The Veterans Health Administration (VHA) Service Support Center: Performance Measures and Monitors. http://vssc/products.asp?PgmArea=9 (last accessed Jan. 15, 2008; available only to members). 18. National Committee for Quality Assurance: HEDIS 2000: Technical Specifications, vol 2. Washington, DC: National Committee for Quality Assurance, 2000. 19. Liu C.F., Chaney E.: Using HEDIS to measure quality of care in VA primary care: The case of antidepressant medication management. Abstr Acad Health Serv Res Health Policy Meet 18, Jan. 2001. http://gateway.nlm.nih.gov/MeetingAbstracts/ma?f=102273097.html (last accessed Feb. 17, 2011). 20. Cully J.A., et al.: Quality of depression care and its impact on health service use and mortality among veterans. Psychiatr Serv 59:1399–1405, Dec. 2008. 21. Oxman T.E., et al.: A fidelity measure for integrated management of depression in primary care. Med Care 44:1030–1037, Nov. 2006. 22. Dietrich A.J., et al.: Re-engineering systems for the treatment of depression in primary care: Cluster randomised controlled trial. BMJ 329(7466):602–607, Sep. 11, 2004. Epub Sep. 2, 2004. 23. 2008 Directory of VA and DoD Health Care Facilities. Federal Practitioner, Dec. 2007. 24. Joint Commission International: Applied Tracer Methodology: Tips and Strategies for Continuous Quality Improvement. Oak Brook, IL: Joint Commission Resources, 2007. 25. Bradley E.H., et al.: Hospital quality for acute myocardial infarction: Correlation among process measures and relationship with short-term mortality. JAMA 296:72–78, Jul. 5, 2008. 26. Peterson E.D., et al.: Association between hospital process performance and outcomes among patients with acute coronary syndromes. JAMA 295:1912–1920, Apr. 26, 2006. 27. Werner R.M., Bradlow E.T.: Relationship between Medicare’s hospital compare performance measures and mortality rates. JAMA 296:2694–2702, Dec. 13, 2006.
Volume 37 Number 4
Copyright 2011 © The Joint Commission
177
The Joint Commission Journal on Quality and Patient Safety Online-Only Content
8 Appendix 1. Interview Protocols*
Tracer 1: As an example, let’s talk about the process of care for a 26-year-old Iraq Veteran who presents for an intake in the primary care clinic and screens positive for depression. (PC and MH chiefs) Target Information D1 Using the Patient Health Questionnaire-9 (PHQ-9) or similar tool for diagnosis and baseline severity. Q:
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
What do you tell the patients about depression; do you write anything down for them?
Target Information D7 Identifying self-management goals. Q:
No
What else do you assess when someone has depression?
Target Information D6 Providing educational materials. Q:
Yes
When is treatment finished? How do we know when patients relapse?
Target Information D5 Asking focused questions about suicide risk. Q:
No
What do you do if patients are not getting better? How long do you wait before reevaluating the treatment plan?
Target Information D4 Continuing to make treatment recommendations to the point of remission. Q:
Yes
What are the typical treatment options in primary care? How is progress monitored? Who monitors progress?
Target Information D3 Determining the adequacy of initial treatment response at four weeks and making alterations to treatment if necessary. Q:
No
Who performs the screen? Who interprets the screen? What is the next step in assessment?
Target Information D2 Using the PHQ-9 or similar tool to monitor treatment response. Q:
Yes
How do you assess whether care is consistent with patients’ goals; how are they engaged in treatment planning?
Target Information D8 Making follow-up calls to assess treatment barriers. Q: Does anyone call patients outside of scheduled appointments? What do they assess? Target Information D9 Communication between primary care and mental health providers. PC: When do you get mental health involved, and how do you work together? MH: What is your working relationship with primary care?
(continued on page AP2)
AP1
April 2011
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Online-Only Content
8 Appendix 1. Interview Protocols* (continued)
Tracer 2: As an example, let’s talk about the process of care for a 56-year-old Vietnam Veteran referred to mental health for treatment of depression after unsuccessful treatment trials of Prozac (fluoxetine) and Wellbutrin (bupropion) in the primary care clinic. (MH chief only) Target Information D1 Using the Patient Health Questionnaire-9 (PHQ-9) or similar tool for diagnosis and baseline severity. Q:
Yes
No
Yes
No
Who performs the intake? Is there a standard assessment routine? Which types of providers might be involved?
Target Information D2 Using the PHQ-9 or similar tool to monitor treatment response.
Q: What are the typical treatment options for depression in mental health? How is progress monitored? Who monitors progress? Do you offer inpatient or outpatient ECT? Target Information D3 Determining the adequacy of initial treatment response at four weeks and making alterations to treatment if necessary. Q:
Yes
No
Yes
No
Yes
No
Yes
No
Does anyone call patients outside of scheduled appointments? What do they assess?
Target Information D9 Communication between primary care and mental health providers. Q:
No
How do you assess whether care is consistent with patients’ goals; how are they engaged in treatment planning?
Target Information D8 Making follow-up calls to assess treatment barriers. Q:
Yes
What do you tell the patients about depression; do you write anything down for them?
Target Information D7 Identifying self-management goals. Q:
No
What else do you assess when someone has depression?
Target Information D6 Providing educational materials. Q:
Yes
When is treatment finished? How do you know when patients relapse?
Target Information D5 Asking focused questions about suicide risk. Q:
No
What do you do if patients are not getting better? How long do you wait before reevaluating the treatment plan?
Target Information D4 Continuing to make treatment recommendations to the point of remission. Q:
Yes
What is your working relationship with primary care? Do you refer patients back to primary care when they are better?
*PC, primary care; MH, mental health; ECT, electroconvulsive therapy; D, domain.
April 2011
Volume 37 Number 4
Copyright 2011 © The Joint Commission
AP2
Does the Veterans Affairs Depression Performance Measure Predict Quality Care? Brian Shiner, M.D., M.P.H.; Bradley V. Watts, M.D., M.P.H.; Marcy K. Traum, M.D.; Samuel J. Huber, M.D., M.P.H.; Yinong Young-Xu, Sc.D.
D
epression is a common illness among veterans treated in the Department of Veterans Affairs (VA) medical centers (VAMCs). Examination of electronic medical records (EMRs) at a VA primary care clinic revealed that 5%–7% of patients had a diagnosis of depression,1 although the prevalence of depression among VA outpatients was found to be as high as 31% in a comprehensive screening program.2 Depression and related psychiatric disability continue to affect many soldiers returning from current conflicts in Iraq and Afghanistan.3,4 In addition to complicating the treatment of other neuropsychiatric difficulties common among veterans, such as posttraumatic stress disorder and persistent postconcussive syndrome,5,6 depression can complicate the care of chronic medical conditions, with worse outcomes and increased costs.7 Depression is associated with low adherence to cardiac medications in patients with coronary artery disease,8 and higher depression severity is associated with poor diet and low adherence to oral hypoglycemic medications in patients with diabetes.9 Fortunately, depression treatment can ameliorate these problems. Chronic disease management programs that identify and treat depression in a structured, proactive manner have been found to be efficacious10 and cost-effective.11 In addition to promoting the effective treatment of depression, these programs have reduced mortality in depressed patients with comorbid diabetes or cancer.12, 13 Although the mechanism by which effective depression treatment improves medical mortality is unknown, it is clear that patients with depression who adhere to their treatment regimen for depression are more likely to adhere to their treatment regimen for medical illnesses such as coronary artery disease, dislipidemia, and diabetes.14 Given the high prevalence and serious consequences of depression among veterans treated in the VA, the identification and evidencebased treatment of depression have become a high priority, with significant investments made in the adaptation and spread of evidence-based models of care for VA treatment settings.15 170
April 2011
Article-at-a-Glance Background: In 2000, the Department of Veterans
Affairs (VA) instituted a performance measure to improve the quality of depression care. The measure evaluated adequacy of follow-up for depressed patients but was removed from clinic directors’ performance plans in fiscal year (FY) 2009 because it had not been empirically validated. The VA depression performance measure was compared with an empirically validated model for assessing adherence to important depression treatment processes. Methods: VA medical centers (VAMCs) whose performance on the VA depression measure was in the top or bottom quartile nationally for all four quarters in FY2008 were selected for inclusion. A blinded interviewer attempted to contact clinical directors of both primary care and mental health at each VAMC and conducted telephone interviews using a protocol designed to employ the 3-Component Model (3CM) fidelity measure, which assesses domains of evidence-based depression care. Results: There were 9 sites in the “high-performing” group and 10 sites in the “low-performing” group. At least one interview was completed at 8 of the 9 sites in the highperforming group and 9 of the 10 sites in the low-performing group. There was a significant difference in the percentage of patients meeting the VA depression performance measure between the high- and low-performing groups (47.5% versus 14.7%; 2 = 837.5, p < .001). The adapted version of the 3CM fidelity scale detected a significant difference in process of depression care between the high- and low-performing sites (82.3 versus 71.4; z =2.4, p = .018). Conclusions: The highest-performing sites on the VA depression performance measure adhered to important care processes more often than did the lowest-performing sites.
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Accordingly, implementing a useful measure of depression care quality is important to systemwide improvement. To track the quality of depression care at VAMCs, a depression performance measure was introduced in 2000.16 This measure was designed to assess the adequacy of treatment for patients newly diagnosed with depression and treated with antidepressant medications. It was assessed using automated collection from the EMR for the 12 weeks following a new diagnosis.17 Adequacy was defined in terms of two separate items, each of which employed the same denominator: (1) the percentage of patients who received at least three follow-up encounters and (2) the percentage of patients who received a 90-day medication supply. For the follow-up portion, three encounters had to be for mental health purposes; up to two could be over the telephone, and one had to be with a practitioner who could write prescriptions. Patients could be diagnosed or treated by mental health, primary care, or both services. The VA depression performance measure was patterned after a Healthcare Effectiveness Data and Information Set (HEDIS) measure implemented in the private sector at the same time.18 The HEDIS measure also measured the adequacy of medication prescription and follow-up in the 84 days following a new diagnosis of depression (acute phase) but had an additional component measuring whether patients continued on their medications for 180 days (continuation phase). The VA depression performance measure was removed from primary care and mental health directors’ performance plans in FY2009 because of concerns that it had not been empirically validated, as well as ongoing concerns that the measure was not appropriate for chronically ill patients in the VA population.19 However, there may have been organizational benefit. In a retrospective analysis examining the relationship between adequate follow-up and mortality, Cully et al. showed that the measure was associated with a gradual improvement in the adequacy of follow-up but with no change in adequacy of medication supply from FY2000 to FY2005.20 Using national VA administrative data sets, they considered adequate follow-up to be three visits in the 84 days subsequent to a new diagnosis of depression where there was either (1) a current procedural technology code indicative of a mental health visit or (2) a diagnosis of depression associated with a non–mental health visit. Although they determined the adequacy of follow-up differently than this study, they propose that the improvements in adequacy of follow-up were related to the adoption of the VA depression performance measure. The VA depression performance measure was assessed by the
April 2011
VA Office of Quality and Performance and reported quarterly at the VAMC level. Leaders at individual VAMCs could, in turn, drill down to the list of individual patients who had been included in the measure to analyze their performance. The measure included the percentage of patients who met the adequacy of the follow-up standard and the percentage of patients who met the adequacy of the medication supply standard rather than the percentage of patients who met both standards simultaneously. Therefore, because we used the central performance measure reporting system to obtain our data, we were unable to meaningfully combine the two portions of the measure, as Culley et al. did, using raw administrative data.20 We chose to focus on the follow-up portion of the measure for the following three reasons: 1. There was a greater range of performance, permitting greater discrimination between high and low performers. 2. VAMCs did more poorly on this part, so we thought that lessons on how to do better would be more important. 3. We were interested in active clinical processes that followed the prescription of antidepressants rather than the prescriptions themselves. Although adequacy of follow-up is an important component of optimal care for depression, additional clinical processes have been shown to improve depression outcomes. Oxman and colleagues developed a 10-item fidelity measure to assess the performance of primary care depression programs in delivering effective care.21 Practices scoring highly on this scale—the 3Component Model (3CM) fidelity scale—obtained consistently better outcomes for depressed patients (higher response and remission rates). Whereas Oxman and colleagues’ model has three components, the fidelity scale has 10 items, which represent 10 important processes associated with clinical improvement in the related randomized trial of the 3CM model.22 We sought to compare performance on the follow-up portion of the VA measure with performance on the 3CM fidelity measure to gain an understanding of whether assessing adequacy of follow-up alone is a useful proxy for other effective depression management practices. Our goal was to establish a link between the score on the VA depression performance measure and improved outcomes (Figure 1, page 172). We hypothesized that VAMCs that most frequently met the three-encounter follow-up measure would also follow other procedures outlined by the 3CM fidelity scale for effective care of depression. We further hypothesized that the opposite would be true at VAMCs that least frequently met the three-encounter follow-up measure.
Volume 37 Number 4
Copyright 2011 © The Joint Commission
171
The Joint Commission Journal on Quality and Patient Safety Model for Evaluation
Figure 1. The goal of the study was to establish a link between the score on the Department of Veterans Affairs (VA) depression performance measure and improved outcomes; Dietrich 2004 represents reference 22; Oxman 2006, reference 21. 3CM, 3-component model.
Methods ETHICAL ISSUES This project was reviewed and approved by the Dartmouth Medical School Committee for the Protection of Human Subjects (CPHS No. 21499) and by the White River Junction VAMC Research and Development Committee.
CLASSIFICATION OF SITES Performance measure data were obtained from the Veterans Health Administration (VHA) Support Service Center Web site.17 We attempted to identify 10 consistently highperforming facilities and 10 consistently low-performing facilities. We selected VAMCs that were in either the top or bottom quartile nationally for all four quarters on the VA depression performance measure in FY 2008. This yielded 9 “high-performing” sites and 10 “low-performing” sites. We considered using the yearly mean to identify sites, but our method of using quartiles resulted in the identification of more stable groups of high and low performers. For example, more inclusive criteria such as selecting the top 10% of sites for the high-performing group would have meant that some “top sites” would have met the performance measure less than 40% of the time in one or more quarters. To find individuals who would best understand the overall process of depression care at their respective sites, we used a yearly directory23 to identify the clinical service line chiefs for both mental health and primary care at each of the relevant VAMCs. Service line chiefs are senior clinicians who are 172
April 2011
responsible for clinical care delivered by their disciplinary colleagues. Primary care and mental health service line chiefs shared responsibility for meeting the depression performance measure at the medical center level. We obtained verbal consent to perform audiotaped telephone interviews. An interviewer who was blinded to whether sites were in the high- or lowperforming group [M.K.T.] conducted semistructured interviews separately with both the primary care service chief and the mental health service chief at each hospital. Interviews were conducted in March and April 2009; FY2008 data were not fully reported until January 2009.
DEVELOPMENT OF
THE
ASSESSMENT PROTOCOL
The 3CM fidelity measure is scored from 0 to 100, with different weights for each item. The measure was adapted for use in our interview in the following ways: 1. Item 10, which related to follow-up, was removed to avoid overlap with follow-up as measured by the VA depression performance measure. 2. To adapt this patient-level tool to suit an institutionwide process assessment approach, we removed item 8, which required the review of individual patient logs. 3. We split item six, which related to standardized reassessment and subsequent treatment changes into two separate items to make the process of care more explicit. 4. We reordered the remaining items into interview domains that could be addressed in a logical order using case scenarios (see page 173).
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Table 1. Semistructured Interview Domains Interview Domain
Weight
3CM Fidelity Item
Using the Patient Health Questionnaire-9 (PHQ-9) or similar tool for diagnosis and baseline severity
18.33
1
D2
Using the PHQ-9 or similar tool to monitor treatment response
8.97
6
D3
Determining the adequacy of initial treatment response at four weeks and making alterations to treatment if necessary
8.97
6
D4
Continuing to make treatment recommendations to the point of remission
13.09
7
D5
Asking focused questions about suicide risk
8.91
2
D6
Providing educational materials
7.29
3
D7
Identifying self-management goals
12.92
4
D8
Making follow-up calls to assess treatment barriers
14.84
5
D9
Communication between primary care and mental health providers
6.67
9
D1
Descriptor
To account for items that were removed, we scaled up the domains proportionally to keep the total possible score at 100. The resulting nine domains are presented in Table 1 (above). The 3CM fidelity measure was designed for assessment of case management logs in a randomized trial of depression management in primary care.22 To adapt this scale for our study, we used case scenarios modeled after the Joint Commission “tracer” methodology to frame our semistructured interview.24 We used an iterative process to develop our final scenarios; the interviewer practiced early versions with colleagues at VAMCs not included in the study and we incorporated their feedback. The first case scenario, which assessed facilities’ systems of depression care in the primary care setting, was reviewed with both the primary care service chief and the mental health service chief to generate data to examine our primary hypothesis. Because patients cared for in the mental health service were also included in the VA follow-up measure, we developed a second case scenario that was reviewed only with the mental health service chief. The same domains were assessed, but because some 3CM domains were not highly relevant to mental health care, this part of the interview was used simply to gain a better understanding of individual sites in relevant domains and for hypothesis generation. After the interviews, the blinded interviewer reviewed the tapes and scored the individual domains using predetermined decision rules. Questionable items were resolved with group coding of the interview tapes [M.K.T., B.S., B.V.W.]. When we spoke with both the primary care service line chief and the mental health service line chief about the process of depression care in primary care and they disagreed on individual domains, we resolved differences by reviewing tran-
April 2011
scripts and tapes and taking the answer that was more fully explained. The case scenarios with domain scoring are presented in Appendix 1 (available in online article).
STATISTICAL ANALYSIS High- and low-performing groups on the VA three-visit depression follow-up measure were compared using a 2 test. For the primary care case scenario, comparison between the high- and low-performing groups on the adapted 3CM fidelity scale was made using a Wilcoxon Signed-Rank Test for matched, nonparametric data (means of individual item scores were matched). Comparisons between high- and low-performing groups in individual domains were made using Wilcoxon Rank-Sum Tests for unmatched, nonparametric data (individual scores in the top group were compared to individual scores in the bottom group for each item). All statistical analyses were performed with STATA version 10.1 (Stata Corp, L.P.; College Station, Texas). For the mental health case scenario, we simply looked at adherence to specific domains that were problematic in the primary care case scenario to determine if deficits were consistent across treatment settings.
Results INTERVIEW PARTICIPATION We interviewed at least one clinical service line chief about systems of depression care in the primary care setting at 8 out of the 9 sites in the high-performing group and at 9 out of the 10 sites in the low-performing group. The site in the highperforming group that declined to consent to interview
Volume 37 Number 4
Copyright 2011 © The Joint Commission
173
The Joint Commission Journal on Quality and Patient Safety expressed concern that the data we collected might attract undue scrutiny. The site in the low-performing group that declined to consent to interview expressed concerns about participating in research; there were no affiliated research groups or Institutional Review Board facilities at their site with whom they could discuss the merits and ethics of participating in research. High-Performing Group. In the high-performing group, we obtained complete information from both the primary care service line chief and the mental health service line chief at five out of the remaining eight sites. Among these sites, data were incomplete for the following reasons: 1. The primary care service line chief indicated that he did not know about depression care beyond the screening program because all depression cases were referred directly to mental health (one site). 2. The mental health service line chief did not know about depression care in primary care other than the fact that there was a screening program (one site). 3. The mental health service line chief could not address two domains: alterations in treatment due to nonresponse and making recommendations to the point of remission (one site). Low-Performing Group. We obtained complete information from both the primary care service line chief and the mental health service line chief in only one of the nine sites in the lowperforming group. Among these sites, data were incomplete for the following reasons: 1. The primary care service line chief was interviewed, but the mental health service line chief did not respond to multiple requests for an interview (one site). 2. The mental health service line chief was interviewed, but the primary care service line chief did not respond to multiple requests for interviews (one site). 3. The primary care service line chief indicated that he did not know about depression care beyond the screening program because all depression cases were referred directly to mental health (one site). 4. The mental health service line chief did not know about depression care in primary care other than the fact that there was a screening program (one site). 5. The mental health service line chief did not know about depression care in primary care other than the fact that there was a screening program and that there was not a standardized method to monitor patient progress (one site). 6. The mental health service line chief could not address one domain—alterations in treatment due to nonresponse (one site) and providing educational materials (one site). 174
April 2011
Primary Study Results
* p < .001 in 2 test comparing top and bottom groups †
p < .05 in Wilcoxon Signed Rank Test comparing top and bottom groups
Figure 2. There was a significant difference in the percentage of patients meeting the adequacy of follow-up on the Department of Veterans Affairs depression performance measure between the high- and low-performing groups, and the adapted version of the 3-Component Model (3CM) fidelity scale detected a significant difference in the process of depression treatment between the highand low-performing groups.
Agreement Between Primary Care and Mental Health Clinical Directors. At the 6 sites where complete information was obtained from both interviewees, there was high degree of agreement on the reports between the primary care and mental health clinical directors at each site (Kappa, 0.74). Primary care and mental health service line chiefs’ responses were combined into a complete data set for the 17 sites where we completed at least one interview. As a result, all domains contain some information on the basis of the report of a single interviewee.
DISCRIMINATION BETWEEN GROUPS Figure 2 (above) displays the primary study results. Not surprisingly, there was a significant difference in the percentage of patients meeting the adequacy of follow-up on the VA depression performance measure between the high- and lowperforming groups, given that this was the basis of their grouping (47.5% versus 14.7%; 2 = 837.5, p < .001). Our adapted version of the 3CM fidelity scale detected a significant difference in process of depression treatment between the high- and low-performing groups (82.3 versus 71.4; z =2.4, p = .018).
PROCESS OF CARE: PRIMARY CARE When individual domains were examined (Figure 3, page 175) clinics in the high-performing group consistently adhered
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Positive Responses: Depression Treatment in Primary Care for Nine Dimensions of Evidence-Based Depression Care
Figure 3. Clinics in the high-performing group consistently adhered better than clinics in the low-performing group, but the differences were not statistically significant. D, dimension; MH, mental health; PC, primary care.
better than clinics in the low-performing group, but the differences were not statistically significant, partially because of small sample size. Overall, the largest differences between high- and low-performing groups were in monitoring progress with a standardized depression measure and in altering treatment due to nonresponse. Half or fewer sites in both groups indicated that recommendations were made to the point of remission. The sites consistently performed well in other domains, such as depression screening, suicide assessment, identifying self-management goals, and providing educational materials.
PROCESS OF CARE: MENTAL HEALTH Eight mental health service chiefs in the high-performing group and eight mental health service chiefs in the lowperforming group were interviewed using the second tracer, which assessed depression care provided in mental health clinics. There were no consistent performance patterns between high- and low-performing groups. Interestingly, the greatest problems in primary care clinics were also the consistent deficits in mental health clinics; at the 16 sites, only 3 (19%) indicated that they were monitoring progress with a standardized depression measure, only 5 (31%) indicated that they altered treatment due to nonresponse, and only 8 (50%) indicated that they made recommendations to the point of remission.
April 2011
Discussion The highest-performing sites on the VA depression performance measure adhered to important care processes more often than did the lowest-performing sites. Therefore, this measure may be a useful proxy for other important aspects of depression treatment. Sites in the high-performing group consistently performed better than sites in the low-performing group, especially in their use of a standardized depression measure to monitor progress and in making alterations to treatment on the basis of treatment response. Another important finding was that in both groups, remission was commonly not identified as a goal of depression treatment. This indicates that leaders frequently did not have a conceptual model of what recovery from depression looks like and how to measure it. However, even sites in the low-performing group performed well on many domains; this work identifies the few domains that are areas of consistent deficit that may require more focus. In previous work using the 3CM scale, differences of 2 points resulted in clinically significant improvements in outcomes, and differences ranging from 71 to 82 would suggest clinically robust differences in depression outcomes.21 Although depression outcomes are not collected in a uniform fashion across VAMCs, this indicates that sites in the high-performing group may have better outcomes than sites in the lowperforming group. However, previous work using the 3CM scale evaluated individual patients’ care and related it to out-
Volume 37 Number 4
Copyright 2011 © The Joint Commission
175
The Joint Commission Journal on Quality and Patient Safety comes, whereas we applied the scale to a service as a whole. This interpretation must be made with caution because this is a new application of the 3CM scale. The greatest limitation to this study was the small sample size. This was related to the overall poor performance on the follow-up portion of the VA depression performance measure and paucity of consistently high performers. Expanding the high-performing group was not possible; doing so would have added sites without excellence on this measure to the group. A second major limitation is that we only spoke with two practitioners, at most, from each site. Those practitioners shared responsibility for the quality of depression care at their sites, and there was a high degree of agreement when they were able to answer all the questions about treatment in primary care clinics. However, it is possible that interviewing individual providers or patients may have yielded different results. At some sites, especially in the low-performing group, leaders had less shared knowledge about the process of depression care in primary care. It is especially interesting to note that we were able to obtain complete information with a high level of agreement from both the primary care service line chief and the mental health service line chief at five of the eight sites that we interviewed in the high-performing group but at only one of the nine sites that we interviewed in the low-performing group. It appears that service chiefs at high-performing VAMCs are more likely to have a shared conceptual model of the process of depression care in primary care. Concordance between primary care and mental health service line chiefs itself could affect quality of depression care in the primary care setting, creating systematic bias in our study—the assessments would be most accurate at the best VAMCs and least accurate at the worst VAMCs. The third major limitation was that clinical processes were evaluated with phone interviews rather than with site visits. Although site visits may have yielded richer observations and enabled us to adhere strictly to the tracer methodology, phone interviews did permit us to follow a standardized protocol. Ideally, a method to objectively measure the other components of successful care of depression using queries of the EMR could be developed, thereby avoiding the need for either interviews or site visits. The findings of this study should be seen as a preliminary attempt to understand what the VA depression performance measure tells us about depression care at individual VAMCs. Other studies have compared patient-level processes and outcomes (such as the VA depression performance measure, which is assessed on each patient with a new diagnosis of depression) 176
April 2011
with facility-level quality data (such our design of using service line chiefs as representative agents instead of individual provider and patient interviews or chart reviews). These studies have yielded mixed results in using this method to evaluate other measures of quality, as was seen in three recent studies correlating hospital-level quality of care with mortality.25–27 Although our interviews revealed a high level of system awareness and systems-based practice in primary care clinics, the mental health clinics were notably less systematized and processes were more difficult to describe; frequently, mental health service line chiefs noted that depression care in mental health was “provider-dependent.” This was even true at sites doing well on the performance measure. Although this is an interesting finding that deserves more attention, one must remember that comparison of mental health and primary care settings were not the primary purpose or design of this study. Therefore, further confirmation is indicated, and generalization beyond the 16 mental health clinics that were assessed in this study is not appropriate. This is an area for future exploration, either in the development of new, more systematic mental health models or adaptation of systematic depression care management programs developed in primary care to mental health settings.
Conclusions The highest-performing sites on the follow-up portion of the VA depression performance measure adhere to important care processes more often than the lowest-performing sites. Therefore, this measure may be a useful proxy for other important aspects of depression treatment. Although this measure is no longer used to assess clinic directors’ performance, the data are still collected systematically and are available to individual VAMCs for use in local improvement activities. Our work suggests that primary care clinics wishing to improve their depression treatment performance should consider using the data, which continue to be available as a “supporting measure” as part of a balanced scorecard to assess their gains.* J The authors thank Pamela W. Lee, Ph.D., for her review of an early version of this manuscript, and Thomas E. Oxman, M.D., and Joseph Francis, M.D., M.P.H., for their review of later versions. The views expressed in this article do not necessarily represent the views of the Department of Veterans Affairs (VA) or the United States government. This project was reviewed and approved by Dartmouth Medical School’s Committee for the Protection of Human Subjects, Hanover, NH, USA (CPHS No. 21499), and by the White River Junction VAMC Research and Development Committee.
* Veterans Health Administration Office of Quality and Performance: Measure Master. http://vaww.pdw.med.va.gov/MeasureMaster/MMIndex.asp (last accessed Mar. 2, 2011; available only to members).
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Online-Only Content
8
See the online version of this article for Appendix 1. Interview Protocols
Brian Shiner, M.D., M.P.H., is a Psychiatrist, Department of Veterans Affairs (VA) Medical Center, White River Junction, Vermont, and Assistant Professor, Dartmouth Medical School, Hanover, New Hampshire, and The Dartmouth Institute for Health Policy and Clinical Practice, Lebanon, New Hampshire. Bradley V. Watts, M.D., M.P.H., is Fellowship Director, VA–National Center for Patient Safety Field Office, White River Junction, and Assistant Professor of Psychiatry, Dartmouth Medical School. Marcy K. Traum, M.D., is a Resident in Psychiatry, Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire. Samuel J. Huber, M.D., M.P.H., formerly Fellow, VA National Quality Scholars Program, White River Junction, is Associate Medical Director for Inpatient Psychiatry and Senior Instructor of Psychiatry, University of Rochester School of Medicine and Dentistry, Rochester, New York. Yinong Young-Xu, Sc.D., is a Statistician, VA National Center for Patient Safety Field Office, White River Junction, and Assistant Professor of Psychiatry, Dartmouth Medical School. Please address request for reprints to Brian Shiner, [email protected].
References 1. Kirchner J.E., Curran G.M., Aikens J.: Datapoints: Detecting depression in VA primary care clinics. Psychiatr Serv 55:350, Apr. 2004. 2. Hankin C.S., et al.: Mental disorders and mental health treatment among U.S. Department of Veterans Affairs outpatients: The Veterans Health Study. Am J Psychiatry 156:1924–1930, Dec. 1999. 3. Fontana A., Rosenheck R.: Treatment-seeking veterans of Iraq and Afghanistan: Comparison with veterans of previous wars. J Nerv Ment Dis 196:513–521, Jul. 2008. 4. Milliken C.S., Auchterlonie J.L., Hoge C.W.: Longitudinal assessment of mental health problems among active and reserve component soldiers returning from the Iraq war. JAMA 298:2141–2148, Nov. 14, 2007. 5. Hoge C.W., Goldberg H.M., Castro C.A.: Care of war veterans with mild traumatic brain injury—Flawed perspectives. N Engl J Med 360:1588–1591, Apr. 16, 2009. 6. Stein M.B., McAllister T.W.: Exploring the convergence of posttraumatic stress disorder and mild traumatic brain injury. Am J Psychiatry 166:768–776, Jul. 2009. 7. Donohue J.M., Pincus H.A.: Reducing the societal burden of depression: A review of economic costs, quality of care and effects of treatment. Pharmacoeconomics 25:7–24, Jan. 2010. 8. Gehi A., et al.: Depression and medication adherence in outpatients with coronary heart disease: Findings from the Heart and Soul Study. Arch Intern Med 165:2508–2513, Nov. 28, 2005. 9. Ciechanowski P.S., Katon W.J., Russo J.E.: Depression and diabetes: Impact of depressive symptoms on adherence, function, and costs. Arch Intern Med 160:3278–3285, Nov. 27, 2000. 10. Oxman T.E., Dietrich A.J., Schulberg H.C.: Evidence-based models of integrated management of depression in primary care. Psychiatr Clin North Am 28:1061–1077, Dec. 2005.
April 2011
11. Wang P.S., Simon G.E., Kessler R.C.: Making the business case for enhanced depression care: The National Institute of Mental Health–Harvard Work Outcomes Research and Cost-effectiveness Study. J Occup Environ Med 50:468–475, Apr. 2008. 12. Bogner H.R., et al.: Diabetes, depression, and death: A randomized controlled trial of a depression treatment program for older adults based in primary care (PROSPECT). Diabetes Care 30:3005–3010, Dec. 2007. 13. Gallo J.J., et al.: The effect of a primary care practice-based depression intervention on mortality in older adults: A randomized trial. Ann Intern Med 146:689–698, May 15, 2007. 14. Katon W., et al.: Impact of antidepressant drug adherence on comorbid medication use and resource utilization. Arch Intern Med 165:2497–2503, Nov. 28, 2005. 15. Smith J.L., et al.: Developing a national dissemination plan for collaborative care for depression: QUERI Series. Implement Sci 3, Dec. 21, 2008. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631596/?tool=pubmed (last accessed Feb. 17, 2011). 16. Department of Veterans Affairs, Management of MDD [Major Depressive Disorder] Working Group: Clinical Practice Guideline: Management of Major Depressive Disorder. Pub. No. 10Q-CPG/MDD-00. Washington, DC: May 2009. http://www.healthquality.va.gov/mdd/mdd_full09_c.pdf (last accessed Feb. 17, 2011). 17. The Veterans Health Administration (VHA) Service Support Center: Performance Measures and Monitors. http://vssc/products.asp?PgmArea=9 (last accessed Jan. 15, 2008; available only to members). 18. National Committee for Quality Assurance: HEDIS 2000: Technical Specifications, vol 2. Washington, DC: National Committee for Quality Assurance, 2000. 19. Liu C.F., Chaney E.: Using HEDIS to measure quality of care in VA primary care: The case of antidepressant medication management. Abstr Acad Health Serv Res Health Policy Meet 18, Jan. 2001. http://gateway.nlm.nih.gov/MeetingAbstracts/ma?f=102273097.html (last accessed Feb. 17, 2011). 20. Cully J.A., et al.: Quality of depression care and its impact on health service use and mortality among veterans. Psychiatr Serv 59:1399–1405, Dec. 2008. 21. Oxman T.E., et al.: A fidelity measure for integrated management of depression in primary care. Med Care 44:1030–1037, Nov. 2006. 22. Dietrich A.J., et al.: Re-engineering systems for the treatment of depression in primary care: Cluster randomised controlled trial. BMJ 329(7466):602–607, Sep. 11, 2004. Epub Sep. 2, 2004. 23. 2008 Directory of VA and DoD Health Care Facilities. Federal Practitioner, Dec. 2007. 24. Joint Commission International: Applied Tracer Methodology: Tips and Strategies for Continuous Quality Improvement. Oak Brook, IL: Joint Commission Resources, 2007. 25. Bradley E.H., et al.: Hospital quality for acute myocardial infarction: Correlation among process measures and relationship with short-term mortality. JAMA 296:72–78, Jul. 5, 2008. 26. Peterson E.D., et al.: Association between hospital process performance and outcomes among patients with acute coronary syndromes. JAMA 295:1912–1920, Apr. 26, 2006. 27. Werner R.M., Bradlow E.T.: Relationship between Medicare’s hospital compare performance measures and mortality rates. JAMA 296:2694–2702, Dec. 13, 2006.
Volume 37 Number 4
Copyright 2011 © The Joint Commission
177
The Joint Commission Journal on Quality and Patient Safety Online-Only Content
8 Appendix 1. Interview Protocols*
Tracer 1: As an example, let’s talk about the process of care for a 26-year-old Iraq Veteran who presents for an intake in the primary care clinic and screens positive for depression. (PC and MH chiefs) Target Information D1 Using the Patient Health Questionnaire-9 (PHQ-9) or similar tool for diagnosis and baseline severity. Q:
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
What do you tell the patients about depression; do you write anything down for them?
Target Information D7 Identifying self-management goals. Q:
No
What else do you assess when someone has depression?
Target Information D6 Providing educational materials. Q:
Yes
When is treatment finished? How do we know when patients relapse?
Target Information D5 Asking focused questions about suicide risk. Q:
No
What do you do if patients are not getting better? How long do you wait before reevaluating the treatment plan?
Target Information D4 Continuing to make treatment recommendations to the point of remission. Q:
Yes
What are the typical treatment options in primary care? How is progress monitored? Who monitors progress?
Target Information D3 Determining the adequacy of initial treatment response at four weeks and making alterations to treatment if necessary. Q:
No
Who performs the screen? Who interprets the screen? What is the next step in assessment?
Target Information D2 Using the PHQ-9 or similar tool to monitor treatment response. Q:
Yes
How do you assess whether care is consistent with patients’ goals; how are they engaged in treatment planning?
Target Information D8 Making follow-up calls to assess treatment barriers. Q: Does anyone call patients outside of scheduled appointments? What do they assess? Target Information D9 Communication between primary care and mental health providers. PC: When do you get mental health involved, and how do you work together? MH: What is your working relationship with primary care?
(continued on page AP2)
AP1
April 2011
Volume 37 Number 4
Copyright 2011 © The Joint Commission
The Joint Commission Journal on Quality and Patient Safety Online-Only Content
8 Appendix 1. Interview Protocols* (continued)
Tracer 2: As an example, let’s talk about the process of care for a 56-year-old Vietnam Veteran referred to mental health for treatment of depression after unsuccessful treatment trials of Prozac (fluoxetine) and Wellbutrin (bupropion) in the primary care clinic. (MH chief only) Target Information D1 Using the Patient Health Questionnaire-9 (PHQ-9) or similar tool for diagnosis and baseline severity. Q:
Yes
No
Yes
No
Who performs the intake? Is there a standard assessment routine? Which types of providers might be involved?
Target Information D2 Using the PHQ-9 or similar tool to monitor treatment response.
Q: What are the typical treatment options for depression in mental health? How is progress monitored? Who monitors progress? Do you offer inpatient or outpatient ECT? Target Information D3 Determining the adequacy of initial treatment response at four weeks and making alterations to treatment if necessary. Q:
Yes
No
Yes
No
Yes
No
Yes
No
Does anyone call patients outside of scheduled appointments? What do they assess?
Target Information D9 Communication between primary care and mental health providers. Q:
No
How do you assess whether care is consistent with patients’ goals; how are they engaged in treatment planning?
Target Information D8 Making follow-up calls to assess treatment barriers. Q:
Yes
What do you tell the patients about depression; do you write anything down for them?
Target Information D7 Identifying self-management goals. Q:
No
What else do you assess when someone has depression?
Target Information D6 Providing educational materials. Q:
Yes
When is treatment finished? How do you know when patients relapse?
Target Information D5 Asking focused questions about suicide risk. Q:
No
What do you do if patients are not getting better? How long do you wait before reevaluating the treatment plan?
Target Information D4 Continuing to make treatment recommendations to the point of remission. Q:
Yes
What is your working relationship with primary care? Do you refer patients back to primary care when they are better?
*PC, primary care; MH, mental health; ECT, electroconvulsive therapy; D, domain.
April 2011
Volume 37 Number 4
Copyright 2011 © The Joint Commission
AP2