- Email: [email protected]
Outcomes Research in Hand Surgery: Where Have We Been and Where Should We Go?
Outcomes Research in Hand Surgery: Where Have We Been and Where Should We Go? Kevin C. Chung, MD, MS, Patricia B. Burns, MPH, Erika Davis Sears, MD From the Section of Plastic Surgery, Department of Surgery, The University of Michigan Health System, Ann Arbor, MI.
Purpose: The outcomes movement began in response to a national emphasis to control cost and to limit geographic variation in medical services. The impact of the outcomes movement on hand surgery research is unknown. We conducted a systematic review of hand surgery studies to assess the past and current states of outcomes research in this specialty. Methods: A systematic review of hand surgery outcomes studies was conducted in The Journal of Hand Surgery (American and British volumes) from 1988 to 2004. A Medline search using the 11 Agency for Healthcare Research and Quality categories of outcomes studies was performed. Studies were rated for study design, data endpoints, and level of impact using the Agency for Healthcare Research and Quality– designated impact levels. Results: A total of 2,236 studies were identified, and 1,188 were included in the analysis. Most studies were rated as level 1 impact (N ⫽ 1090, 92%), and a small number of studies were level 4 impact (N ⫽ 98, 8%). Thirty-nine percent were prospective and 45% (N ⫽ 529) were retrospective descriptive studies. Data gathered in studies included objective clinical measures (N ⫽ 672, 57%), quality of life (N ⫽ 374, 31%), morbidity (N ⫽ 401, 34%), subjective clinical measures (N ⫽ 27, 16%), cost (N ⫽ 185, 2%), and mortality (N ⫽ 11, 1%). Conclusions: Our systematic review of hand surgery outcomes studies found that much of the research is confined to testing new or existing surgical techniques (level 1 impact). We found a small number of studies that did show improvement in patient outcomes as a result of an intervention (level 4 impact). The future direction of outcomes studies should consider the impact of the research goals to change patient or physician behavior to enhance health parameters. More research on economic analysis needs to be conducted in hand surgery to meet the national goal of containing cost and improving the quality of health care. (J Hand Surg 2006;31A:1373–1379. Copyright © 2006 by the American Society for Surgery of the Hand.) Key words: Outcomes, hand surgery, clinical studies, audit, impact.
he outcomes movement began in response to the expansion of medical services in the 1950s and 1960s, which resulted in a demand by third-party payors for cost containment.1 Concerns were raised about the unknown end results or outcomes of these medical services and whether the push to reduce cost would decrease the quality of health care. Fueling these concerns were the reported geographic differences in the use of medical services. Wennberg et al2 compared hospital utilization rates in 2 demographically similar areas. They found considerable differences in the use of certain medical
T
procedures, but they detected no difference in the mortality rates for these 2 areas.3 This study showed that the increased use of medical services did not translate into an improvement in outcomes. The movement toward a system to assess and account for medical outcomes was termed the “third revolution of medical care” by Arnold Relman,1 former editor for The New England Journal of Medicine. In response to these concerns, the US Congress approved the organization of the Agency for Health Care Policy and Research (AHCPR), now known as the Agency for Healthcare Research and Quality The Journal of Hand Surgery
1373
1374
The Journal of Hand Surgery / Vol. 31A No. 8 October 2006
(AHRQ). The AHCRR’s mission was to identify research that would directly improve patient outcomes, result in changes in clinical practice, and lead to the development of guidelines for the appropriate use of medical services.4,5 When questioned about the influence of its research on patient outcomes, the AHCPR developed a model to show impact. Impact levels range from 1 to 4, with level 1 studies confirming the value of existing treatment but having little impact on patient outcomes, and level 4 having the greatest impact.5 In 1999, a decade after the movement began, the AHCPR reported on the outcome of outcomes research.6,7 Physicians funded by the AHCPR were surveyed about the results of their outcomes research. Most studies were retrospective studies of administrative records; few studies had results that would directly change patient outcomes, clinical practice, or public policy. To assess the past performance of outcomes research in hand surgery, we undertook a systematic review of outcomes publications in The Journal of Hand Surgery, American volume and The Journal of Hand Surgery, British/European volume. Although the outcomes movement has produced a surge in hand surgery outcomes research, no overall assessment of this effort has been performed. The status of past and present hand surgery outcomes research directions is unknown. Has hand surgery research improved patient outcomes and influenced physician decision making? Have guidelines for specific hand surgery conditions been developed? What is the impact of these studies in enhancing the quality of health care for this specialty? This article addresses these issues by reviewing outcomes articles in hand surgery to examine the types of research being conducted, the types of data gathered, and the level of impact of the studies. Equally important, if we understand where we have been, we can propose future directions for hand surgery outcomes research.
Materials and Methods We conducted a systematic review of outcomes publications in hand surgery from 1988 to 2004. The review was restricted to The Journal of Hand Surgery (American and British volumes). We performed Medline key word searches for the following 11 categories of outcomes studies as outlined by the AHRQ: (1) comparative effectiveness (key word: treatment outcome); (2) descriptive epidemiology (key words: epidemiology, incidence, prevalence); (3) economic assessment (key words: cost, cost analysis); (4) legal, legislative, or regulatory (key words:
legislation, formal social control); (5) methodologic development (key words: risk adjustment, severity of illness index, methodologic development, questionnaires); (6) modeling (key words: decision support techniques, outcome prediction, theoretical models); (7) patient-reported outcomes (key words: outcome assessment (healthcare), quality of life, patient satisfaction, health status); (8) practice variation (key word: physician’s practice patterns); (9) quality of health care (key word: quality of healthcare); (10) sociology of health care (key words: medical sociology, attitude of health personnel); and (11) systematic review or meta-analysis (key word: systematic review, meta-analysis). Articles were excluded if they were editorials, comments, letters, historical articles, case reports with fewer than 20 subjects, tutorials, review articles without primary data, journal articles without primary data, and nonhuman studies. The articles were reviewed by research personnel who had been trained in clinical epidemiology. Each article was then assessed for study design, type of data gathered, and level of impact. The levelof-impact categories are defined by the AHRQ and range from 1 to 4. We found these impact categories difficult to use, especially when distinguishing between level 1 and level 4 impacts. Modifications were made to the impact levels and are described as follows. Level 1 Level 1 describes a research study that presents new clinical information on an existing treatment or a new technique or a new instrument that has the potential to change current clinical decision making. This new information will likely suggest the need for further research to determine the utility or importance of the information presented. Level 1 studies do not directly change clinical practice or policy. An example of a level 1 study is the prospective multicenter study of a new plate for internal fixation of complex fractures of the distal radius.8 The study found the new plate to be safe and effective. A level 1 study does not have a control group to compare a new treatment with an existing treatment. Level 2 Level 2 studies describe the use of evidence-based data by a governing body (eg, law-makers, professional organizations) to justify the creation of a new policy or program. For example, AHRQ-funded research led to the development of a screening measure for Chlamydia species5 and was included in the 2000
Chung, Burns, and Davis Sears / Outcomes Research in Hand Surgery
edition of the Health Plan Employer Data and Information Set (HEDIS). Level 3 Level 3 describes a research study (3a) or historical trend (3b) that shows a change in physician and/or patient decision-making as a direct result of the dissemination of specific research data, clinical guidelines, or established policy. For example, the VF-14 measure, an instrument used to quantify visual impairment, has replaced visual acuity (a measure of clarity of vision, e.g., 20/20 vision) for outcome assessment after cataract surgery and is now used routinely in clinical practice.9 Level 4 Level 4 describes a research study (4a) or historical trend (4b) that shows an improvement in health outcomes (clinical, economic, quality of life, satisfaction) as a direct result of a specific intervention. The intervention must be compared with a control group and show an overall improvement in health outcomes and benefit to society. An example is a clinical trial comparing the open with the percutaneous technique for treating trigger fingers.10 Although both techniques are effective in treating this condition, the percutaneous technique resulted in less pain and a shorter recovery time. We created the following decision-making algorithm to use when it was difficult to differentiate between level 1 and level 4 impact: does the study show a clear advantage of the intervention presented over a control intervention? If the study shows an improvement of a treatment over an existing treatment, the study is level 4. If it does not, the study is level 1. The studies were assessed by 2 reviewers. To determine the reliability of impact-level rating categories, a random sample of 32 studies were selected
Table 1. Study Design for Hand Surgery Outcomes Studies, 1988 to 2004 Study Design Retrospective cohort/case-control Prospective cohort Cross-sectional Randomized clinical trial Meta-analysis/systematic review/ guidelines Cost analysis Total
Number of Studies
%
529 463 95 91
45 39 8 8
5 5 1,188
⬍1 ⬍1 100
1375
Table 2. Outcome Categories in Audit of Hand Surgery Studies From 1988 to 2004 Outcome Category
%
Quality of health care/comparative effectiveness Descriptive epidemiology Economic analysis Total
81 17 2 100
and independently coded by the 2 reviewers. The sample was based on our initial reading of the first 500 articles. After excluding studies that did not fit our inclusion criteria, 320 articles remained. A 10% sample of these 320 papers was used for the random sample. The level of agreement between the reviewers was 91%, showing good agreement in this rating scale.
Results A total of 2,233 articles were identified, and 1,188 fit the inclusion criteria. Most studies in our review were retrospective or case control studies (N ⫽ 529, 45%), followed by prospective studies (N ⫽ 463, 39%). Additional study categories included randomized clinical trials (N ⫽ 91, 8%) and cross-sectional studies (N ⫽ 95, 8%). Five articles were meta-analysis/systematic reviews, and 5 were cost studies (Table 1). Using the outcome categories outlined by the AHRQ, we found that most studies assessed quality of health care and comparative effectiveness (N ⫽ 965, 81%). Seventeen percent (N ⫽ 194) of the studies had descriptive epidemiology outcomes, and 2% (N ⫽ 27) were economic analyses. None of the studies discussed practice variation, quality of health care, sociology of health care, or any of the other categories (Table 2). Clinical measures such as grip strength, range of motion, functional status, and diagnostic accuracy were the most common data gathered in the articles (N ⫽ 672, 57%). Other data endpoint categories included quality of life (N ⫽ 374, 31%), morbidity (N ⫽ 401, 34%), subjective clinical measures (N ⫽ 185, 16%), and mortality (N ⫽ 11, 1%). Some studies had more than 1 endpoint; therefore, the percentage presented does not add up to 100%. Only 2% (N ⫽ 27) of the studies gathered data on cost (Table 3). Ninety-two percent (N ⫽ 1090) of the studies were determined to have a level 1 impact, and 8% (N ⫽ 98) were determined to have level 4 impacts. No studies were found to have level 2 or level 3 impacts (Table 4).
1376
The Journal of Hand Surgery / Vol. 31A No. 8 October 2006
Table 3. Result Endpoint for Hand Surgery Outcomes Studies, 1988 to 2004* Endpoint
n
%
Objective clinical measures Morbidity Quality of life Subjective clinical measures Mortality Cost
672 401 374 185 11 27
57 34 32 16 1 2
*Percentages total more than 100% because of studies with multiple endpoints.
Discussion The outcomes movement began with high hopes of reducing costs by eliminating unnecessary procedures, enhancing the dissemination of information to patients and physicians, improving patient health and functioning, and developing guidelines for clinical practice.1,4,11,12 A review of AHRQ-funded research did not find any reduction in costs or improvement in quality of patient outcomes.6,7 Although a great deal of knowledge had been generated, there had been few changes in policy and clinical practice; many of the goals of the movement had not been accomplished. Using a similar audit strategy as that proposed by the AHRQ, we performed an assessment of outcomes research in hand surgery. We reviewed hand surgery research from the beginning of the outcomes movement, 1988, through 2004 —a course of 17 years. Our systematic review found that outcomes research in hand surgery consisted of articles that generate hypotheses or test new or existing procedures (92% level 1 impact). Although a level 1 impact contributes to the existing knowledge base, it does not directly change patient outcomes and decision making in clinical practice or public policy. An example of a level 1 impact is a retrospective study of 2 groups who had either endoscopic or open decompression of the carpal tunnel.13 Questionnaires sent to patients found no differences between the 2 groups for symptom relief, time to return to work, or complications. This study was a level 1 study because it was not able to show an improvement in patient outcomes of one procedure over the other, nor did it have the potential to change clinical decision making. A few studies (9%) did show an improvement in patient outcomes as a result of a specific intervention (level 4 impact). An example of level 4 impact is a retrospective study testing the diagnostic accuracy of
ultrasound for solid and cystic lesions of the hand and wrist.14 Ultrasound reports were compared with the initial clinical impression recorded on patients’ medical records. Ultrasound was very accurate (83%) in diagnosing lesions of the hand and wrist compared with initial clinical impression (54%). Ultrasound had a clear advantage in diagnosing these types of lesions, and therefore the study has a level 4 impact. We found it difficult to distinguish between level 1 and level 4 impacts using the categories for impact levels developed by the AHRQ. We made modifications by expanding the definitions for each level with particular emphasis on levels 1 and 4. Even with our modifications, we believed there was still room for error in the decision process for level 1 versus level 4. This led to the development of a decisionmaking algorithm to assist in assigning level 1 or level 4. An intervention group had to be compared with a control group and show a clear advantage over the control group to have a level 4 impact. The percentage of type 4 impact studies was low. Even with the potential imprecision of the AHRQ definition, type 4 impact studies are still a small percentage of the overall outcomes studies in hand surgery. The absence of level 2 (changing policy) or level 3 (changing patient or physician decision-making) impact studies indicates that hand surgery outcomes research has not progressed to the point of changing physician or patient decision-making or creating guidelines that have policy implications. Level 2 or 3 studies are particularly difficult to conduct, because these studies challenge existing practice. Examples of these studies as highlighted by the AHRQ include the development of guidelines by the American Academy of Pediatrics for the use of less expensive antibiotics in treating otitis media (level 2).15 Another example is how the use of an interactive video program resulted in lower rates of surgery for benign prostatic hyperplasia (level 3).16 The AHRQ reported on the difficulties in changing physician decisionmaking and clinical practice. Many physicians lack
Table 4. Level of Impact of Hand Surgery Outcome Studies, 1988 to 2004 Level of Impact
n
%
1 2 3 4 Total
1,090 0 0 98 1,188
92 0 0 8 100
Chung, Burns, and Davis Sears / Outcomes Research in Hand Surgery
the time and skills to synthesize research results and use those results to make critical decisions.11 Physicians have been shown to be reluctant to make changes in clinical practice even when they agree with proposed guidelines.4 The development of guidelines is an expensive and time-consuming process.4,17 Therefore, guidelines should be restricted to conditions that affect large numbers of people, have high costs, and show large variations in health outcomes.17 The pay-for-performance mandate by the Center for Medicare and Medicaid Services18 will impose some form of practice guidelines that may or may not change practice. Ideally, health policy makers hope that the institution of carefully constructed, specialty-specific practice guidelines will reduce practice variation, decrease cost, or ultimately, enhance the quality of health care. The challenge for hand surgery is to conduct these types of studies that have impact on policy and practice. Similar to the AHRQ impact levels are the levelof-evidence categories used in The Journal of Bone and Joint Surgery and The Journal of Hand Surgery. The level-of-evidence categories are based on the strength of evidence provided by different study designs. The 4 types of impact studies as proposed by the AHRQ aim to evaluate how these clinical studies improve the quality of health care. Overall, these 2 types of clinical study categorization are different. The clinical trial has often been considered the gold standard in study designs. One of the difficulties with this study design is that the results of an intervention are from a controlled environment and may not reflect outcomes in everyday clinical practice.4,12 In addition, there is not enough time or money to perform a clinical trial for every medical procedure.19 Clinical trials are mostly efficacy studies that evaluate practices in ideal conditions, typically from tertiary referral centers. Many argue that clinical trials do not reflect real-life experiences and may not be applicable to everyday practice. The attraction of outcomes studies is that they are effectiveness studies that are conducted by observing overall practice patterns and outcomes in the health care system. Outcomes studies can be more relevant to the actual practice. Descriptive studies such as retrospective and prospective cohort are commonly used in outcomes research, but their potential to determine the effectiveness of procedures is limited.6 Because descriptive studies often lack a control group, it is difficult to determine if a procedure is effective. We found that
1377
most studies being conducted in the hand surgery outcomes field are descriptive, with retrospective (45%) and prospective cohort studies (39%) predominating. Randomized clinical trials (8%) and crosssectional studies (8%) made up the rest of the studies. Clinical trials were more likely to be assigned a level 4 impact. The small number of economic analysis studies shows there has been little emphasis on examining cost in hand surgery research. There are virtually no articles on variation in hand surgery care. Considering that the outcomes movement is spurred by practice variations and a need to contain cost, future outcomes studies in hand surgery must focus on these 2 areas of study that are central to the outcomes initiative. Hand surgeons have traditionally relied on objective clinical measures such as range of motion, grip strength, and key pinch as health outcomes.20 –22 Our review of hand surgery studies found that 57% of the studies involved collected objective clinical measures. Some of the objective measures were direct measurements of grip strength and range of motion, but this group also included studies that gathered data on functional status through questionnaires. Morbidity data, specifically complications and persistence of conditions, was another frequent data endpoint (34%). The shift toward patient-reported outcomes in hand surgery is indicated by the number of studies reporting subjective outcomes. Quality-of-life outcomes were reported in 31% of the studies. Data included symptoms, patient satisfaction, and time to return to work and data from quality-of-life–related questionnaires. Subjective clinical measures (ie, outcomes rated as good, fair, poor) were reported in 16% of the studies. Cost data made up a small proportion (2%) of the data gathered. To reach the goal of using outcomes data for cost containment, more cost data need to be gathered. Our review, based on almost 2 decades of outcomes research, had similar findings to the AHRQ report. Most hand surgery outcomes studies do not indicate a direct improvement in health outcomes as a result of an intervention or changes in clinical practice or public policy. Viewed over time, however, the number of level 4 impact studies increased. For example, in 1988 none of the studies had level 4 impact, but in 2004, 11% of the studies had level 4 impact. According to the AHRQ model for impacts, the level 1 impacts will be used as the foundation for level 4 impacts.6 Based on our systematic review, it appears that hand surgery is headed in the right direction. The lack of studies showing a change in physician (level 2) or patient– decision-making (level 3) or the
1378
The Journal of Hand Surgery / Vol. 31A No. 8 October 2006
development of guidelines shows there is still more work to be done. For example, a level 2 impact study, which may have the potential to change physician behavior, could relate to the development of a practice guideline to treat chronic wrist pain. This practice guideline can be constructed using evidencebased data and cost consideration of various tests to derive a most optimal diagnostic and treatment algorithm for patients presenting with chronic wrist pain. The universal adoption of this wrist practice guideline may decrease practice variations among primary care physicians and hand surgeons in managing this difficult condition. The report that this practice guideline has changed physician behavior in decreasing variability in managing chronic wrist pain will have a level 2 impact. An example of a level 3 impact study, which changes patient decision making, may involve an education program for patients with rheumatoid arthritis regarding hand reconstruction. This education program may encourage patients to seek hand surgery evaluations for their hand deformities. The impact of this education program can be shown by increased patient interest or an increased national trend toward patients with rheumatoid arthritis being evaluated by hand surgeons. Of course, these impacttype studies may be difficult to plan and to perform, but their ability to change physician or patient behavior toward improving the quality of health care is particularly important to health policy makers. Many medical conditions are competing for national attention and funding. Although other conditions may have higher profiles because of their impact on the life of patients, the field of hand surgery has an important role in enhancing quality of life. Only through high-impact studies can hand surgery raise the awareness of the American public toward diseases and conditions afflicting the hand and upper limb. Only through concerted efforts by hand surgeons in producing these impact studies can hand surgery receive equal national support commensurate with its contribution to the well-being of the public. To answer our question regarding the status of hand surgery outcomes research, it appears that a great deal of research has produced new clinical information and surgical techniques. Hand surgeons, however, need to move research to the next level of impact. More studies are needed that can show improvements in patient outcomes as a result of surgical techniques. In addition, researchers must expand the scope of outcomes research by incorporating all of the 11 outcomes study categories suggested by the AHRQ. Important areas of studies should include economic analyses, practice vari-
ations, sociology of health care, and quality of health care. Hand surgery has a rich and productive interest in outcomes research. The future of this specialty’s research direction relies on developing imaginative, highimpact studies to change and enhance the quality of hand surgery care. Received for publication June 1, 2006; accepted in revised form June 16, 2006. No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article. Corresponding author: Kevin C. Chung, MD, MS, Section of Plastic Surgery, University of Michigan Health System, 2130 Taubman Center, 1500 E Medical Center Dr, Ann Arbor, MI 48109-0340; e-mail: [email protected]. Copyright © 2006 by the American Society for Surgery of the Hand 0363-5023/06/31A08-0019$32.00/0 doi:10.1016/j.jhsa.2006.06.012
References 1. Relman AS. Assessment and accountability: the third revolution in medical care. N Engl J Med 1988;319:1220 –1222. 2. Wennberg JE, Freeman JL, Culp WJ. Are hospital services rationed in New Haven or over-utilized in Boston? Lancet 1987;1:1185–1189. 3. Wennberg JE, Freeman JL, Shelton RM, Bubolz TA. Hospital use and mortality among Medicare beneficiaries in Boston and New Haven. N Engl J Med 1989;321:1168 –1173. 4. Epstein AM. The outcomes movement—will it get us where we want to go? N Engl J Med 1990;323:266 –270. 5. Eisenberg JM. Putting research to work: reporting and enhancing the impact of health services research. Health Serv Res 2001;36:x–xvii. 6. The outcome of outcomes research at AHCPR: final report. Agency for Health Care Policy and Research (No. 99-R044). Rockville, MD, 1999. Available at: http://www.ahrq.gov/ clinic/out2res. Accessed June 13, 2006. 7. Stryer D, Tunis S, Hubbard H, Clancy C. The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade. Health Serv Res 2000;35:977–993. 8. Ring D, Jupiter JB, Brennwald J, Buchler U, Hastings H II. Prospective multicenter trial of a plate for dorsal fixation of distal radius fractures. J Hand Surg 1997;22A:777–784. 9. Steinberg EP, Tielsch JM, Schein OD, Javitt JC, Sharkey P, Cassard SD, et al. The VF-14. An index of functional impairment in patients with cataract. Arch Ophthalmol 1994; 112:630 – 638. 10. Gilberts EC, Beekman WH, Stevens HJ, Wereldsman JC. Prospective randomized trial of open versus percutaneous surgery for trigger digits. J Hand Surg 2001;26A:497–500. 11. Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care. An initiative to evaluate and improve medical practice. N Engl J Med 1988;319:1197– 1202. 12. Ellwood PM. Shattuck lecture— outcomes management. A technology of patient experience. N Engl J Med 1988;318: 1549 –1556. 13. Bande S, DeSmet L, Fabry G. The results of carpal tunnel release: open versus endoscopic technique. J Hand Surg 1994;19B:14 –17.
Chung, Burns, and Davis Sears / Outcomes Research in Hand Surgery 14. Teefey SA, Middleton WD, Patel V, Hildebolt CF, Boyer MI. The accuracy of high-resolution ultrasound for evaluating focal lesions of the hand and wrist. J Hand Surg 2004; 29A:393–399. 15. Berman S, Byrns PJ, Bondy J, Smith PJ, Lezotte D. Otitis media-related antibiotic prescribing patterns, outcomes, and expenditures in a pediatric Medicaid population. Pediatrics 1997;100:585–592. 16. Barry MJ, Cherkin DC, Chang Y, Fowler FJ, Skates S. A randomized trial of a multimedia shared decision making program for men facing a treatment decision for benign prostatic hypertrophy. Dis Management Clin Outcomes 1997; 1:5–14. 17. Perfetto EM, Morris LS. Agency for Health Care Policy and
18. 19. 20.
21.
22.
1379
Research clinical practice guidelines. Ann Pharmacother 1996;30:1117–1121. Peterson ED. Should we link payment to quality? Am Heart J 2004;148:S56 –S58. Greenfield S. The state of outcome research: are we on target? N Engl J Med 1989;320:1142–1143. Bindra RR, Dias JJ, Heras-Palau C, Amadio PC, Chung KC, Burke FD. Assessing outcome after hand surgery: the current state. J Hand Surg 2003;28B:289 –294. Macey AC, Burke FD, Abbot K, Barton NJ, Bradbury E, Bradley A, et al. Outcomes of hand surgery. British Society for Surgery of the Hand. J Hand Surg 1995;20B:841– 855. Szabo RM. Outcomes assessments in hand surgery: when are they meaningful? J Hand Surg 2001;26A:993–1002.
Purpose: The outcomes movement began in response to a national emphasis to control cost and to limit geographic variation in medical services. The impact of the outcomes movement on hand surgery research is unknown. We conducted a systematic review of hand surgery studies to assess the past and current states of outcomes research in this specialty. Methods: A systematic review of hand surgery outcomes studies was conducted in The Journal of Hand Surgery (American and British volumes) from 1988 to 2004. A Medline search using the 11 Agency for Healthcare Research and Quality categories of outcomes studies was performed. Studies were rated for study design, data endpoints, and level of impact using the Agency for Healthcare Research and Quality– designated impact levels. Results: A total of 2,236 studies were identified, and 1,188 were included in the analysis. Most studies were rated as level 1 impact (N ⫽ 1090, 92%), and a small number of studies were level 4 impact (N ⫽ 98, 8%). Thirty-nine percent were prospective and 45% (N ⫽ 529) were retrospective descriptive studies. Data gathered in studies included objective clinical measures (N ⫽ 672, 57%), quality of life (N ⫽ 374, 31%), morbidity (N ⫽ 401, 34%), subjective clinical measures (N ⫽ 27, 16%), cost (N ⫽ 185, 2%), and mortality (N ⫽ 11, 1%). Conclusions: Our systematic review of hand surgery outcomes studies found that much of the research is confined to testing new or existing surgical techniques (level 1 impact). We found a small number of studies that did show improvement in patient outcomes as a result of an intervention (level 4 impact). The future direction of outcomes studies should consider the impact of the research goals to change patient or physician behavior to enhance health parameters. More research on economic analysis needs to be conducted in hand surgery to meet the national goal of containing cost and improving the quality of health care. (J Hand Surg 2006;31A:1373–1379. Copyright © 2006 by the American Society for Surgery of the Hand.) Key words: Outcomes, hand surgery, clinical studies, audit, impact.
he outcomes movement began in response to the expansion of medical services in the 1950s and 1960s, which resulted in a demand by third-party payors for cost containment.1 Concerns were raised about the unknown end results or outcomes of these medical services and whether the push to reduce cost would decrease the quality of health care. Fueling these concerns were the reported geographic differences in the use of medical services. Wennberg et al2 compared hospital utilization rates in 2 demographically similar areas. They found considerable differences in the use of certain medical
T
procedures, but they detected no difference in the mortality rates for these 2 areas.3 This study showed that the increased use of medical services did not translate into an improvement in outcomes. The movement toward a system to assess and account for medical outcomes was termed the “third revolution of medical care” by Arnold Relman,1 former editor for The New England Journal of Medicine. In response to these concerns, the US Congress approved the organization of the Agency for Health Care Policy and Research (AHCPR), now known as the Agency for Healthcare Research and Quality The Journal of Hand Surgery
1373
1374
The Journal of Hand Surgery / Vol. 31A No. 8 October 2006
(AHRQ). The AHCRR’s mission was to identify research that would directly improve patient outcomes, result in changes in clinical practice, and lead to the development of guidelines for the appropriate use of medical services.4,5 When questioned about the influence of its research on patient outcomes, the AHCPR developed a model to show impact. Impact levels range from 1 to 4, with level 1 studies confirming the value of existing treatment but having little impact on patient outcomes, and level 4 having the greatest impact.5 In 1999, a decade after the movement began, the AHCPR reported on the outcome of outcomes research.6,7 Physicians funded by the AHCPR were surveyed about the results of their outcomes research. Most studies were retrospective studies of administrative records; few studies had results that would directly change patient outcomes, clinical practice, or public policy. To assess the past performance of outcomes research in hand surgery, we undertook a systematic review of outcomes publications in The Journal of Hand Surgery, American volume and The Journal of Hand Surgery, British/European volume. Although the outcomes movement has produced a surge in hand surgery outcomes research, no overall assessment of this effort has been performed. The status of past and present hand surgery outcomes research directions is unknown. Has hand surgery research improved patient outcomes and influenced physician decision making? Have guidelines for specific hand surgery conditions been developed? What is the impact of these studies in enhancing the quality of health care for this specialty? This article addresses these issues by reviewing outcomes articles in hand surgery to examine the types of research being conducted, the types of data gathered, and the level of impact of the studies. Equally important, if we understand where we have been, we can propose future directions for hand surgery outcomes research.
Materials and Methods We conducted a systematic review of outcomes publications in hand surgery from 1988 to 2004. The review was restricted to The Journal of Hand Surgery (American and British volumes). We performed Medline key word searches for the following 11 categories of outcomes studies as outlined by the AHRQ: (1) comparative effectiveness (key word: treatment outcome); (2) descriptive epidemiology (key words: epidemiology, incidence, prevalence); (3) economic assessment (key words: cost, cost analysis); (4) legal, legislative, or regulatory (key words:
legislation, formal social control); (5) methodologic development (key words: risk adjustment, severity of illness index, methodologic development, questionnaires); (6) modeling (key words: decision support techniques, outcome prediction, theoretical models); (7) patient-reported outcomes (key words: outcome assessment (healthcare), quality of life, patient satisfaction, health status); (8) practice variation (key word: physician’s practice patterns); (9) quality of health care (key word: quality of healthcare); (10) sociology of health care (key words: medical sociology, attitude of health personnel); and (11) systematic review or meta-analysis (key word: systematic review, meta-analysis). Articles were excluded if they were editorials, comments, letters, historical articles, case reports with fewer than 20 subjects, tutorials, review articles without primary data, journal articles without primary data, and nonhuman studies. The articles were reviewed by research personnel who had been trained in clinical epidemiology. Each article was then assessed for study design, type of data gathered, and level of impact. The levelof-impact categories are defined by the AHRQ and range from 1 to 4. We found these impact categories difficult to use, especially when distinguishing between level 1 and level 4 impacts. Modifications were made to the impact levels and are described as follows. Level 1 Level 1 describes a research study that presents new clinical information on an existing treatment or a new technique or a new instrument that has the potential to change current clinical decision making. This new information will likely suggest the need for further research to determine the utility or importance of the information presented. Level 1 studies do not directly change clinical practice or policy. An example of a level 1 study is the prospective multicenter study of a new plate for internal fixation of complex fractures of the distal radius.8 The study found the new plate to be safe and effective. A level 1 study does not have a control group to compare a new treatment with an existing treatment. Level 2 Level 2 studies describe the use of evidence-based data by a governing body (eg, law-makers, professional organizations) to justify the creation of a new policy or program. For example, AHRQ-funded research led to the development of a screening measure for Chlamydia species5 and was included in the 2000
Chung, Burns, and Davis Sears / Outcomes Research in Hand Surgery
edition of the Health Plan Employer Data and Information Set (HEDIS). Level 3 Level 3 describes a research study (3a) or historical trend (3b) that shows a change in physician and/or patient decision-making as a direct result of the dissemination of specific research data, clinical guidelines, or established policy. For example, the VF-14 measure, an instrument used to quantify visual impairment, has replaced visual acuity (a measure of clarity of vision, e.g., 20/20 vision) for outcome assessment after cataract surgery and is now used routinely in clinical practice.9 Level 4 Level 4 describes a research study (4a) or historical trend (4b) that shows an improvement in health outcomes (clinical, economic, quality of life, satisfaction) as a direct result of a specific intervention. The intervention must be compared with a control group and show an overall improvement in health outcomes and benefit to society. An example is a clinical trial comparing the open with the percutaneous technique for treating trigger fingers.10 Although both techniques are effective in treating this condition, the percutaneous technique resulted in less pain and a shorter recovery time. We created the following decision-making algorithm to use when it was difficult to differentiate between level 1 and level 4 impact: does the study show a clear advantage of the intervention presented over a control intervention? If the study shows an improvement of a treatment over an existing treatment, the study is level 4. If it does not, the study is level 1. The studies were assessed by 2 reviewers. To determine the reliability of impact-level rating categories, a random sample of 32 studies were selected
Table 1. Study Design for Hand Surgery Outcomes Studies, 1988 to 2004 Study Design Retrospective cohort/case-control Prospective cohort Cross-sectional Randomized clinical trial Meta-analysis/systematic review/ guidelines Cost analysis Total
Number of Studies
%
529 463 95 91
45 39 8 8
5 5 1,188
⬍1 ⬍1 100
1375
Table 2. Outcome Categories in Audit of Hand Surgery Studies From 1988 to 2004 Outcome Category
%
Quality of health care/comparative effectiveness Descriptive epidemiology Economic analysis Total
81 17 2 100
and independently coded by the 2 reviewers. The sample was based on our initial reading of the first 500 articles. After excluding studies that did not fit our inclusion criteria, 320 articles remained. A 10% sample of these 320 papers was used for the random sample. The level of agreement between the reviewers was 91%, showing good agreement in this rating scale.
Results A total of 2,233 articles were identified, and 1,188 fit the inclusion criteria. Most studies in our review were retrospective or case control studies (N ⫽ 529, 45%), followed by prospective studies (N ⫽ 463, 39%). Additional study categories included randomized clinical trials (N ⫽ 91, 8%) and cross-sectional studies (N ⫽ 95, 8%). Five articles were meta-analysis/systematic reviews, and 5 were cost studies (Table 1). Using the outcome categories outlined by the AHRQ, we found that most studies assessed quality of health care and comparative effectiveness (N ⫽ 965, 81%). Seventeen percent (N ⫽ 194) of the studies had descriptive epidemiology outcomes, and 2% (N ⫽ 27) were economic analyses. None of the studies discussed practice variation, quality of health care, sociology of health care, or any of the other categories (Table 2). Clinical measures such as grip strength, range of motion, functional status, and diagnostic accuracy were the most common data gathered in the articles (N ⫽ 672, 57%). Other data endpoint categories included quality of life (N ⫽ 374, 31%), morbidity (N ⫽ 401, 34%), subjective clinical measures (N ⫽ 185, 16%), and mortality (N ⫽ 11, 1%). Some studies had more than 1 endpoint; therefore, the percentage presented does not add up to 100%. Only 2% (N ⫽ 27) of the studies gathered data on cost (Table 3). Ninety-two percent (N ⫽ 1090) of the studies were determined to have a level 1 impact, and 8% (N ⫽ 98) were determined to have level 4 impacts. No studies were found to have level 2 or level 3 impacts (Table 4).
1376
The Journal of Hand Surgery / Vol. 31A No. 8 October 2006
Table 3. Result Endpoint for Hand Surgery Outcomes Studies, 1988 to 2004* Endpoint
n
%
Objective clinical measures Morbidity Quality of life Subjective clinical measures Mortality Cost
672 401 374 185 11 27
57 34 32 16 1 2
*Percentages total more than 100% because of studies with multiple endpoints.
Discussion The outcomes movement began with high hopes of reducing costs by eliminating unnecessary procedures, enhancing the dissemination of information to patients and physicians, improving patient health and functioning, and developing guidelines for clinical practice.1,4,11,12 A review of AHRQ-funded research did not find any reduction in costs or improvement in quality of patient outcomes.6,7 Although a great deal of knowledge had been generated, there had been few changes in policy and clinical practice; many of the goals of the movement had not been accomplished. Using a similar audit strategy as that proposed by the AHRQ, we performed an assessment of outcomes research in hand surgery. We reviewed hand surgery research from the beginning of the outcomes movement, 1988, through 2004 —a course of 17 years. Our systematic review found that outcomes research in hand surgery consisted of articles that generate hypotheses or test new or existing procedures (92% level 1 impact). Although a level 1 impact contributes to the existing knowledge base, it does not directly change patient outcomes and decision making in clinical practice or public policy. An example of a level 1 impact is a retrospective study of 2 groups who had either endoscopic or open decompression of the carpal tunnel.13 Questionnaires sent to patients found no differences between the 2 groups for symptom relief, time to return to work, or complications. This study was a level 1 study because it was not able to show an improvement in patient outcomes of one procedure over the other, nor did it have the potential to change clinical decision making. A few studies (9%) did show an improvement in patient outcomes as a result of a specific intervention (level 4 impact). An example of level 4 impact is a retrospective study testing the diagnostic accuracy of
ultrasound for solid and cystic lesions of the hand and wrist.14 Ultrasound reports were compared with the initial clinical impression recorded on patients’ medical records. Ultrasound was very accurate (83%) in diagnosing lesions of the hand and wrist compared with initial clinical impression (54%). Ultrasound had a clear advantage in diagnosing these types of lesions, and therefore the study has a level 4 impact. We found it difficult to distinguish between level 1 and level 4 impacts using the categories for impact levels developed by the AHRQ. We made modifications by expanding the definitions for each level with particular emphasis on levels 1 and 4. Even with our modifications, we believed there was still room for error in the decision process for level 1 versus level 4. This led to the development of a decisionmaking algorithm to assist in assigning level 1 or level 4. An intervention group had to be compared with a control group and show a clear advantage over the control group to have a level 4 impact. The percentage of type 4 impact studies was low. Even with the potential imprecision of the AHRQ definition, type 4 impact studies are still a small percentage of the overall outcomes studies in hand surgery. The absence of level 2 (changing policy) or level 3 (changing patient or physician decision-making) impact studies indicates that hand surgery outcomes research has not progressed to the point of changing physician or patient decision-making or creating guidelines that have policy implications. Level 2 or 3 studies are particularly difficult to conduct, because these studies challenge existing practice. Examples of these studies as highlighted by the AHRQ include the development of guidelines by the American Academy of Pediatrics for the use of less expensive antibiotics in treating otitis media (level 2).15 Another example is how the use of an interactive video program resulted in lower rates of surgery for benign prostatic hyperplasia (level 3).16 The AHRQ reported on the difficulties in changing physician decisionmaking and clinical practice. Many physicians lack
Table 4. Level of Impact of Hand Surgery Outcome Studies, 1988 to 2004 Level of Impact
n
%
1 2 3 4 Total
1,090 0 0 98 1,188
92 0 0 8 100
Chung, Burns, and Davis Sears / Outcomes Research in Hand Surgery
the time and skills to synthesize research results and use those results to make critical decisions.11 Physicians have been shown to be reluctant to make changes in clinical practice even when they agree with proposed guidelines.4 The development of guidelines is an expensive and time-consuming process.4,17 Therefore, guidelines should be restricted to conditions that affect large numbers of people, have high costs, and show large variations in health outcomes.17 The pay-for-performance mandate by the Center for Medicare and Medicaid Services18 will impose some form of practice guidelines that may or may not change practice. Ideally, health policy makers hope that the institution of carefully constructed, specialty-specific practice guidelines will reduce practice variation, decrease cost, or ultimately, enhance the quality of health care. The challenge for hand surgery is to conduct these types of studies that have impact on policy and practice. Similar to the AHRQ impact levels are the levelof-evidence categories used in The Journal of Bone and Joint Surgery and The Journal of Hand Surgery. The level-of-evidence categories are based on the strength of evidence provided by different study designs. The 4 types of impact studies as proposed by the AHRQ aim to evaluate how these clinical studies improve the quality of health care. Overall, these 2 types of clinical study categorization are different. The clinical trial has often been considered the gold standard in study designs. One of the difficulties with this study design is that the results of an intervention are from a controlled environment and may not reflect outcomes in everyday clinical practice.4,12 In addition, there is not enough time or money to perform a clinical trial for every medical procedure.19 Clinical trials are mostly efficacy studies that evaluate practices in ideal conditions, typically from tertiary referral centers. Many argue that clinical trials do not reflect real-life experiences and may not be applicable to everyday practice. The attraction of outcomes studies is that they are effectiveness studies that are conducted by observing overall practice patterns and outcomes in the health care system. Outcomes studies can be more relevant to the actual practice. Descriptive studies such as retrospective and prospective cohort are commonly used in outcomes research, but their potential to determine the effectiveness of procedures is limited.6 Because descriptive studies often lack a control group, it is difficult to determine if a procedure is effective. We found that
1377
most studies being conducted in the hand surgery outcomes field are descriptive, with retrospective (45%) and prospective cohort studies (39%) predominating. Randomized clinical trials (8%) and crosssectional studies (8%) made up the rest of the studies. Clinical trials were more likely to be assigned a level 4 impact. The small number of economic analysis studies shows there has been little emphasis on examining cost in hand surgery research. There are virtually no articles on variation in hand surgery care. Considering that the outcomes movement is spurred by practice variations and a need to contain cost, future outcomes studies in hand surgery must focus on these 2 areas of study that are central to the outcomes initiative. Hand surgeons have traditionally relied on objective clinical measures such as range of motion, grip strength, and key pinch as health outcomes.20 –22 Our review of hand surgery studies found that 57% of the studies involved collected objective clinical measures. Some of the objective measures were direct measurements of grip strength and range of motion, but this group also included studies that gathered data on functional status through questionnaires. Morbidity data, specifically complications and persistence of conditions, was another frequent data endpoint (34%). The shift toward patient-reported outcomes in hand surgery is indicated by the number of studies reporting subjective outcomes. Quality-of-life outcomes were reported in 31% of the studies. Data included symptoms, patient satisfaction, and time to return to work and data from quality-of-life–related questionnaires. Subjective clinical measures (ie, outcomes rated as good, fair, poor) were reported in 16% of the studies. Cost data made up a small proportion (2%) of the data gathered. To reach the goal of using outcomes data for cost containment, more cost data need to be gathered. Our review, based on almost 2 decades of outcomes research, had similar findings to the AHRQ report. Most hand surgery outcomes studies do not indicate a direct improvement in health outcomes as a result of an intervention or changes in clinical practice or public policy. Viewed over time, however, the number of level 4 impact studies increased. For example, in 1988 none of the studies had level 4 impact, but in 2004, 11% of the studies had level 4 impact. According to the AHRQ model for impacts, the level 1 impacts will be used as the foundation for level 4 impacts.6 Based on our systematic review, it appears that hand surgery is headed in the right direction. The lack of studies showing a change in physician (level 2) or patient– decision-making (level 3) or the
1378
The Journal of Hand Surgery / Vol. 31A No. 8 October 2006
development of guidelines shows there is still more work to be done. For example, a level 2 impact study, which may have the potential to change physician behavior, could relate to the development of a practice guideline to treat chronic wrist pain. This practice guideline can be constructed using evidencebased data and cost consideration of various tests to derive a most optimal diagnostic and treatment algorithm for patients presenting with chronic wrist pain. The universal adoption of this wrist practice guideline may decrease practice variations among primary care physicians and hand surgeons in managing this difficult condition. The report that this practice guideline has changed physician behavior in decreasing variability in managing chronic wrist pain will have a level 2 impact. An example of a level 3 impact study, which changes patient decision making, may involve an education program for patients with rheumatoid arthritis regarding hand reconstruction. This education program may encourage patients to seek hand surgery evaluations for their hand deformities. The impact of this education program can be shown by increased patient interest or an increased national trend toward patients with rheumatoid arthritis being evaluated by hand surgeons. Of course, these impacttype studies may be difficult to plan and to perform, but their ability to change physician or patient behavior toward improving the quality of health care is particularly important to health policy makers. Many medical conditions are competing for national attention and funding. Although other conditions may have higher profiles because of their impact on the life of patients, the field of hand surgery has an important role in enhancing quality of life. Only through high-impact studies can hand surgery raise the awareness of the American public toward diseases and conditions afflicting the hand and upper limb. Only through concerted efforts by hand surgeons in producing these impact studies can hand surgery receive equal national support commensurate with its contribution to the well-being of the public. To answer our question regarding the status of hand surgery outcomes research, it appears that a great deal of research has produced new clinical information and surgical techniques. Hand surgeons, however, need to move research to the next level of impact. More studies are needed that can show improvements in patient outcomes as a result of surgical techniques. In addition, researchers must expand the scope of outcomes research by incorporating all of the 11 outcomes study categories suggested by the AHRQ. Important areas of studies should include economic analyses, practice vari-
ations, sociology of health care, and quality of health care. Hand surgery has a rich and productive interest in outcomes research. The future of this specialty’s research direction relies on developing imaginative, highimpact studies to change and enhance the quality of hand surgery care. Received for publication June 1, 2006; accepted in revised form June 16, 2006. No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article. Corresponding author: Kevin C. Chung, MD, MS, Section of Plastic Surgery, University of Michigan Health System, 2130 Taubman Center, 1500 E Medical Center Dr, Ann Arbor, MI 48109-0340; e-mail: [email protected]. Copyright © 2006 by the American Society for Surgery of the Hand 0363-5023/06/31A08-0019$32.00/0 doi:10.1016/j.jhsa.2006.06.012
References 1. Relman AS. Assessment and accountability: the third revolution in medical care. N Engl J Med 1988;319:1220 –1222. 2. Wennberg JE, Freeman JL, Culp WJ. Are hospital services rationed in New Haven or over-utilized in Boston? Lancet 1987;1:1185–1189. 3. Wennberg JE, Freeman JL, Shelton RM, Bubolz TA. Hospital use and mortality among Medicare beneficiaries in Boston and New Haven. N Engl J Med 1989;321:1168 –1173. 4. Epstein AM. The outcomes movement—will it get us where we want to go? N Engl J Med 1990;323:266 –270. 5. Eisenberg JM. Putting research to work: reporting and enhancing the impact of health services research. Health Serv Res 2001;36:x–xvii. 6. The outcome of outcomes research at AHCPR: final report. Agency for Health Care Policy and Research (No. 99-R044). Rockville, MD, 1999. Available at: http://www.ahrq.gov/ clinic/out2res. Accessed June 13, 2006. 7. Stryer D, Tunis S, Hubbard H, Clancy C. The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade. Health Serv Res 2000;35:977–993. 8. Ring D, Jupiter JB, Brennwald J, Buchler U, Hastings H II. Prospective multicenter trial of a plate for dorsal fixation of distal radius fractures. J Hand Surg 1997;22A:777–784. 9. Steinberg EP, Tielsch JM, Schein OD, Javitt JC, Sharkey P, Cassard SD, et al. The VF-14. An index of functional impairment in patients with cataract. Arch Ophthalmol 1994; 112:630 – 638. 10. Gilberts EC, Beekman WH, Stevens HJ, Wereldsman JC. Prospective randomized trial of open versus percutaneous surgery for trigger digits. J Hand Surg 2001;26A:497–500. 11. Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care. An initiative to evaluate and improve medical practice. N Engl J Med 1988;319:1197– 1202. 12. Ellwood PM. Shattuck lecture— outcomes management. A technology of patient experience. N Engl J Med 1988;318: 1549 –1556. 13. Bande S, DeSmet L, Fabry G. The results of carpal tunnel release: open versus endoscopic technique. J Hand Surg 1994;19B:14 –17.
Chung, Burns, and Davis Sears / Outcomes Research in Hand Surgery 14. Teefey SA, Middleton WD, Patel V, Hildebolt CF, Boyer MI. The accuracy of high-resolution ultrasound for evaluating focal lesions of the hand and wrist. J Hand Surg 2004; 29A:393–399. 15. Berman S, Byrns PJ, Bondy J, Smith PJ, Lezotte D. Otitis media-related antibiotic prescribing patterns, outcomes, and expenditures in a pediatric Medicaid population. Pediatrics 1997;100:585–592. 16. Barry MJ, Cherkin DC, Chang Y, Fowler FJ, Skates S. A randomized trial of a multimedia shared decision making program for men facing a treatment decision for benign prostatic hypertrophy. Dis Management Clin Outcomes 1997; 1:5–14. 17. Perfetto EM, Morris LS. Agency for Health Care Policy and
18. 19. 20.
21.
22.
1379
Research clinical practice guidelines. Ann Pharmacother 1996;30:1117–1121. Peterson ED. Should we link payment to quality? Am Heart J 2004;148:S56 –S58. Greenfield S. The state of outcome research: are we on target? N Engl J Med 1989;320:1142–1143. Bindra RR, Dias JJ, Heras-Palau C, Amadio PC, Chung KC, Burke FD. Assessing outcome after hand surgery: the current state. J Hand Surg 2003;28B:289 –294. Macey AC, Burke FD, Abbot K, Barton NJ, Bradbury E, Bradley A, et al. Outcomes of hand surgery. British Society for Surgery of the Hand. J Hand Surg 1995;20B:841– 855. Szabo RM. Outcomes assessments in hand surgery: when are they meaningful? J Hand Surg 2001;26A:993–1002.