- Email: [email protected]
Using Information From Databases to Improve Clinical Practice: Lessons Learned Under Fire
Using Information From Databases to Improve Clinical Practice: Lessons Learned Under Fire Stanley W. Dziuban, Jr, MD Department of Thoracic Surgery, St Peter’s Hospital, Albany, New York
Background. Information derived from a clinical database can be used to produce reports with valid, applicable data, to assemble outcomes data for purposes of marketing or practice survival, and to improve and optimize patient care. Methods. In response to lay-press figures suggesting a high risk-adjusted mortality at St. Peter’s Hospital (Albany, NY), a multidisciplinary group at the hospital undertook a collaborative review of information from our clinical database to identify any patterns that could explain this disturbing summary statistic. Results. This review showed that for the vast majority (>95%) of cases mortality was on a par with or lower than the statewide average. The elevated mortality was confined to a small and specific subset of high-risk patients.
Once identified, attention was focused on this group of patients, and both cardiologists and surgeons discussed practice changes based on information from the clinical database. Since program changes were instituted in 1993, mortality has decreased in both high-risk and all other patients. Overall mortality for coronary artery bypass graft patients has decreased from 4.5% to about 1.5%. Conclusions. Use of a database to extract a single number reflecting patient-care performance, eg, shortterm mortality, can conceal more information than it conveys. Only in-depth analysis of the information in the database can identify areas for improvement in clinical practice. (Ann Thorac Surg 1997;64:S64 –7) © 1997 by The Society of Thoracic Surgeons
T
A clinical database is really just a model of this complex reality, the result of input from all phases of the process and all staff components. Of course, no model is perfect. In fact, it has been said that all models are wrong but some are useful. The important measure of a model is how useful it is, not how complex or sophisticated it is.
here is power in data. Currently our data are being used by federal and state governments, business coalitions, managed-care organizations, payers, and consumer groups, with ourselves often the passive victims and targets of this use. This presentation addresses the issue of how we can best use our own data. It makes sense that we should be the experts in using these data and should assert leadership in showing others how to use them well. There are three ways in which a database might be used: (1) for purposes of internal reporting, (2) for external purposes such as marketing and contracting, unavoidable necessities in this era of competitive market forces, and (3) to improve practice through feedback. This last use is in the best tradition of our profession: using our resources for the improvement of patient care. The reality of patient care is very complex, even in a relatively homogenous field like coronary artery bypass grafting (CABG). The overall process involves the patient with some severity of disease, multiple interventions including catheterization, anesthesia, operation, and postoperative care, all contributing to patient outcome. This complexity is compounded by the number of patients passing through the process. Further, many hospital staff components are involved in the process, including administration, cardiology, anesthesiology, and nursing, as well as surgery.
Presented at Risk Management in CABG: Analysis of Critical Issues, San Diego, CA, Feb 1, 1997. Address reprint requests to Dr Dziuban, Thoracic Surgery, St Peter’s Hospital, 319 S Manning Blvd, Suite 301, Albany, NY 12208.
© 1997 by The Society of Thoracic Surgeons Published by Elsevier Science Inc
Bottom-Line Numbers Are Not Enough A database will provide some bottom-line results, eg, mortality after a cardiac operation. These bottom-line numbers are very popular with everyone, including the government, the lay press, and consumer groups. They are easy to understand, and they give the appearance of being able to summarize or grade a program. But one bottom-line number cannot possibly accurately reflect the information in a database, much less the very complex care process that stands behind it. It does not tell much about the patient-care process, and it does not reveal how patient care might be improved. It is like the bottom-line number on a financial report: It shows that a corporation is either in the black or in the red, but it does not tell why. I like to say that bottom-line numbers raise more questions than they answer. And beyond that, they also can be downright misleading. Bottom-line numbers can be used as a prod to go back to the database to search for meaningful information. Think of this as an exploratory operation. One does not always know what is going to be found, or even know what one is looking for. But, like an exploratory operation, a database search is not likely to be totally random. Familiarity with the clinical situation at hand may pro0003-4975/97/$17.00 PII S0003-4975(97)01159-4
Ann Thorac Surg 1997;64:S64 –7
vide some expectations of what might be found. Nevertheless, a thorough search is needed to rule out anything unexpected. It is essential to compare one’s own database with some external database that can serve as a norm or standard. Note that not just bottom lines are compared, but also the details in the databases are. Of course, this works best if one is part of a larger database and is working with the same definitions. Open access to the details of the larger database is also needed. This kind of comparison is so important that I believe it makes a relatively simple database that has a good comparable external database more useful than a relatively sophisticated database that is either closed or proprietary. The information gained from a database search can serve as useful feedback to all staff components. But a greater purpose is served if all the staff come together as an integrated team to respond to this information. If each specialty works separately to optimize only its own part of the care process, the overall process remains suboptimized. Today’s specialties tend to be highly evolved. It is more likely that improvements in the care process can be found in the transitions and boundaries between specialties than within each one. These interfaces can only be addressed by an integrated teamwork approach.
Practical Experience: St. Peter’s Hospital I am a member of a large surgical group that developed cardiac surgery programs at three hospitals in Albany, New York. As far as we could tell, St. Peter’s Hospital was just like the others, with the possible exception that it had a particularly large and aggressive cardiology group. Then one morning, we awoke to a headline in a local paper announcing that St. Peter’s had the highest death rate for heart bypass in the state. The staff reacted with intense shock and disbelief. After all, this was where we worked diligently, doing what we thought was good cardiac surgery. First reactions were that the data could not be right, that the risk adjustment could not be fair. Frustration levels were high because nothing about our rating gave any clue about the nature of the problem. Predictably, we initially responded with the approach in which we had been trained, and instituted an intense mortality and morbidity review. But repeated reviews by many groups all arrived at the same conclusion: Virtually all the deaths had occurred in high-risk patients, and there were no specific errors to be found in their care. Then the first positive development occurred. Everyone realized that this was not just the surgeons’ problem, that all hospital staff were in this together and had to figure out where the problem was together. The decision was therefore made to undertake a collaborative review of our database [1, 2]. The bottom line for CABG mortality in our 1992 database showed that actual mortality was 4.6%. Predicted mortality was 2.1%. Risk-adjusted mortality, which was the bottom-line result on which our headline-making state ranking was based, was 6.6%; this compared with the New York State average of 3.1%.
RISK MANAGEMENT DZIUBAN DATABASES IN CLINICAL PRACTICE
S65
It was generally assumed by the public, and even in health-care circles, that these figures meant that we were achieving poor results even in low-risk patients. But when we explored our database and compared our CABG data with statewide figures, we found that our mortality was largely concentrated in our emergency cases, which represented about 10% of our volume. Mortality in our 1992 emergency cases was 26%; this compared with a 1991 statewide average of 7.7%. Elective cases represented more than a third of our 1992 volume; our mortality rate of 1.2% in these cases was somewhat below the statewide average. Urgent cases represented more than half of our 1992 volume; our mortality rate in these cases of 3.2% also was somewhat below the statewide average. This comparison certainly shed new light on our situation. First, it showed that 90% of our CABG operations were being done with a mortality that was lower than average. Second, it identified a problem in our emergency surgery. Our prior impression, which was that our high-risk patients had a higher mortality that was proportional to their risk, was modified by the database information. When we looked further into our database, we found that deaths in emergency patients were heavily concentrated in those cases classified as high-acuity; these are patients who had a myocardial infarction within 6 hours before the operation, or were in shock or hemodynamically unstable at the time they went into the operating room. High-acuity patients represent 5% of our total CABG volume. In a 2-year pool of deaths in all emergency patients, 13 of 16 deaths had occurred in highacuity patients. Now we had arrived at a focus that was useful to address. We took this information back to the staff and focused attention on our high-acuity subset of patients. In accord with the Hawthorn effect, which states that as soon as you start looking at something it begins to change, this focused attention led to many changes in our practices. The cardiologists changed their thinking. They previously had been moving these unstable patients into operation quickly, considering it important to have them revascularized as quickly as possible. Now they began to stabilize them for a short time before having them undergo operation. Our database had shown that only 24% of the high-acuity patients had been given a preoperative intraaortic balloon pump; this was due both to the desire to rush them into operation and to the desire to avoid complications of leg edema. With new awareness of the risk level of these patients, this approach was reconsidered and the following year 89% of the comparable patients had a preoperative intraaortic balloon pump. The surgeons discussed their cardioplegia practices and realized they were all using different methods, with not all using retrograde cardioplegia. Practices were standardized in a move toward greater use of retrograde cardioplegia to provide myocardial protection. We became aware that the use of intravenous nitroglycerin while patients were in critical care posed a problem when
S66
RISK MANAGEMENT DZIUBAN DATABASES IN CLINICAL PRACTICE
Ann Thorac Surg 1997;64:S64 –7
Fig 3. Impact of program changes introduced beginning in 1993 on mortality in all coronary artery bypass graft patients. Fig 1. Impact of program changes introduced beginning in 1993 on mortality in high-acuity coronary artery bypass graft patients.
they were transferred to the floor, with patients sometimes destabilizing when they were switched to a nitropaste patch, so this policy was changed. This improvement process eventually extended throughout our entire program. For example, concern about balloon complications prompted our critical-care nurses to begin tracking all balloon patients and identifying the types of complications that occurred, leading to changes in our balloon practices that reduced the frequency of complications. We established joint conferences with all relevant staff groups to provide integrated direction to our program. We also became much more open about sharing data and found that this defused much suspicion and distrust. Holding data close to the chest makes others assume there is something in the data to hide.
Impact of Program Changes Since the introduction of program changes beginning in 1993, mortality has decreased in both high-acuity pa-
Fig 2. Impact of program changes introduced beginning in 1993 on mortality in coronary artery bypass graft patients other than highacuity patients.
tients and all other CABG patients (Figs 1, 2) [1, 2]. Annual program volume has increased. Overall CABG mortality has decreased from 4.5% to about 1.5%. Overall mortality has been consistently under expected mortality, and expected mortality has remained fairly constant, indicating that we have not been avoiding operating on high-risk patients (Fig 3). Initiatives that we undertook directed at length of stays were effective in reducing both preoperative and postoperative stays (Fig 4).
Summary: Lessons Learned Under Fire Experience at St. Peter’s Hospital teaches a number of important lessons: (1) We need to become the experts in using our own data, thus avoiding becoming the victim of others who will use the data less knowledgeably and less appropriately. We need to take the initiative, to acknowledge our data honestly, and to use these data effectively and responsibly. (2) Everyone in a program both influences the program and shares in its fate. (3) Bottom-line results produce more questions than answers; they represent only the beginning of the search for information. (4) Explore your database to find areas for improvement. Even preformed impressions of where problems or opportunities exist in your program should be confirmed by
Fig 4. Impact of program changes introduced beginning in 1993 on length of stay for all coronary artery bypass graft patients.
Ann Thorac Surg 1997;64:S64 –7
information in your database. Objective data always are more effective than personal opinion in convincing others. (5) Use an external database to gain perspective on your own data. The most comparable data come from others using the same database information, and hence the same definitions. (6) Seek ways to optimize care rather than seeking just to avoid errors in care. Errors in care probably occur in a small minority of patients. The payoff is greater in affecting care for the majority of your patient population. (7) Integrate everyone as a team to optimize the overall care process. If each specialty optimizes only its own role, the overall process remains suboptimized. In today’s world of advanced specialties, opportunities for the improvement of care are more likely to be found in the boundaries between specialties.
RISK MANAGEMENT DZIUBAN DATABASES IN CLINICAL PRACTICE
S67
We thought our program at St. Peter’s was a good one. It took information from our database to tell us that although the program was not bad, it could be improved. And it took information from the database to help us determine the areas of needed improvement. Are there any such areas in your program? Develop a database now, start using it, and you will know.
References 1. Chassin MR, Hannan EL, DeBuono BA. Benefits and hazards of reporting medical outcomes publicly. N Engl J Med 1996; 334:394– 8. 2. Dziuban SW Jr, McIlduff JB, Miller SJ, Dal Col RH. How a New York cardiac surgery program uses outcomes data. Ann Thorac Surg 1994;58:1871– 6.
Background. Information derived from a clinical database can be used to produce reports with valid, applicable data, to assemble outcomes data for purposes of marketing or practice survival, and to improve and optimize patient care. Methods. In response to lay-press figures suggesting a high risk-adjusted mortality at St. Peter’s Hospital (Albany, NY), a multidisciplinary group at the hospital undertook a collaborative review of information from our clinical database to identify any patterns that could explain this disturbing summary statistic. Results. This review showed that for the vast majority (>95%) of cases mortality was on a par with or lower than the statewide average. The elevated mortality was confined to a small and specific subset of high-risk patients.
Once identified, attention was focused on this group of patients, and both cardiologists and surgeons discussed practice changes based on information from the clinical database. Since program changes were instituted in 1993, mortality has decreased in both high-risk and all other patients. Overall mortality for coronary artery bypass graft patients has decreased from 4.5% to about 1.5%. Conclusions. Use of a database to extract a single number reflecting patient-care performance, eg, shortterm mortality, can conceal more information than it conveys. Only in-depth analysis of the information in the database can identify areas for improvement in clinical practice. (Ann Thorac Surg 1997;64:S64 –7) © 1997 by The Society of Thoracic Surgeons
T
A clinical database is really just a model of this complex reality, the result of input from all phases of the process and all staff components. Of course, no model is perfect. In fact, it has been said that all models are wrong but some are useful. The important measure of a model is how useful it is, not how complex or sophisticated it is.
here is power in data. Currently our data are being used by federal and state governments, business coalitions, managed-care organizations, payers, and consumer groups, with ourselves often the passive victims and targets of this use. This presentation addresses the issue of how we can best use our own data. It makes sense that we should be the experts in using these data and should assert leadership in showing others how to use them well. There are three ways in which a database might be used: (1) for purposes of internal reporting, (2) for external purposes such as marketing and contracting, unavoidable necessities in this era of competitive market forces, and (3) to improve practice through feedback. This last use is in the best tradition of our profession: using our resources for the improvement of patient care. The reality of patient care is very complex, even in a relatively homogenous field like coronary artery bypass grafting (CABG). The overall process involves the patient with some severity of disease, multiple interventions including catheterization, anesthesia, operation, and postoperative care, all contributing to patient outcome. This complexity is compounded by the number of patients passing through the process. Further, many hospital staff components are involved in the process, including administration, cardiology, anesthesiology, and nursing, as well as surgery.
Presented at Risk Management in CABG: Analysis of Critical Issues, San Diego, CA, Feb 1, 1997. Address reprint requests to Dr Dziuban, Thoracic Surgery, St Peter’s Hospital, 319 S Manning Blvd, Suite 301, Albany, NY 12208.
© 1997 by The Society of Thoracic Surgeons Published by Elsevier Science Inc
Bottom-Line Numbers Are Not Enough A database will provide some bottom-line results, eg, mortality after a cardiac operation. These bottom-line numbers are very popular with everyone, including the government, the lay press, and consumer groups. They are easy to understand, and they give the appearance of being able to summarize or grade a program. But one bottom-line number cannot possibly accurately reflect the information in a database, much less the very complex care process that stands behind it. It does not tell much about the patient-care process, and it does not reveal how patient care might be improved. It is like the bottom-line number on a financial report: It shows that a corporation is either in the black or in the red, but it does not tell why. I like to say that bottom-line numbers raise more questions than they answer. And beyond that, they also can be downright misleading. Bottom-line numbers can be used as a prod to go back to the database to search for meaningful information. Think of this as an exploratory operation. One does not always know what is going to be found, or even know what one is looking for. But, like an exploratory operation, a database search is not likely to be totally random. Familiarity with the clinical situation at hand may pro0003-4975/97/$17.00 PII S0003-4975(97)01159-4
Ann Thorac Surg 1997;64:S64 –7
vide some expectations of what might be found. Nevertheless, a thorough search is needed to rule out anything unexpected. It is essential to compare one’s own database with some external database that can serve as a norm or standard. Note that not just bottom lines are compared, but also the details in the databases are. Of course, this works best if one is part of a larger database and is working with the same definitions. Open access to the details of the larger database is also needed. This kind of comparison is so important that I believe it makes a relatively simple database that has a good comparable external database more useful than a relatively sophisticated database that is either closed or proprietary. The information gained from a database search can serve as useful feedback to all staff components. But a greater purpose is served if all the staff come together as an integrated team to respond to this information. If each specialty works separately to optimize only its own part of the care process, the overall process remains suboptimized. Today’s specialties tend to be highly evolved. It is more likely that improvements in the care process can be found in the transitions and boundaries between specialties than within each one. These interfaces can only be addressed by an integrated teamwork approach.
Practical Experience: St. Peter’s Hospital I am a member of a large surgical group that developed cardiac surgery programs at three hospitals in Albany, New York. As far as we could tell, St. Peter’s Hospital was just like the others, with the possible exception that it had a particularly large and aggressive cardiology group. Then one morning, we awoke to a headline in a local paper announcing that St. Peter’s had the highest death rate for heart bypass in the state. The staff reacted with intense shock and disbelief. After all, this was where we worked diligently, doing what we thought was good cardiac surgery. First reactions were that the data could not be right, that the risk adjustment could not be fair. Frustration levels were high because nothing about our rating gave any clue about the nature of the problem. Predictably, we initially responded with the approach in which we had been trained, and instituted an intense mortality and morbidity review. But repeated reviews by many groups all arrived at the same conclusion: Virtually all the deaths had occurred in high-risk patients, and there were no specific errors to be found in their care. Then the first positive development occurred. Everyone realized that this was not just the surgeons’ problem, that all hospital staff were in this together and had to figure out where the problem was together. The decision was therefore made to undertake a collaborative review of our database [1, 2]. The bottom line for CABG mortality in our 1992 database showed that actual mortality was 4.6%. Predicted mortality was 2.1%. Risk-adjusted mortality, which was the bottom-line result on which our headline-making state ranking was based, was 6.6%; this compared with the New York State average of 3.1%.
RISK MANAGEMENT DZIUBAN DATABASES IN CLINICAL PRACTICE
S65
It was generally assumed by the public, and even in health-care circles, that these figures meant that we were achieving poor results even in low-risk patients. But when we explored our database and compared our CABG data with statewide figures, we found that our mortality was largely concentrated in our emergency cases, which represented about 10% of our volume. Mortality in our 1992 emergency cases was 26%; this compared with a 1991 statewide average of 7.7%. Elective cases represented more than a third of our 1992 volume; our mortality rate of 1.2% in these cases was somewhat below the statewide average. Urgent cases represented more than half of our 1992 volume; our mortality rate in these cases of 3.2% also was somewhat below the statewide average. This comparison certainly shed new light on our situation. First, it showed that 90% of our CABG operations were being done with a mortality that was lower than average. Second, it identified a problem in our emergency surgery. Our prior impression, which was that our high-risk patients had a higher mortality that was proportional to their risk, was modified by the database information. When we looked further into our database, we found that deaths in emergency patients were heavily concentrated in those cases classified as high-acuity; these are patients who had a myocardial infarction within 6 hours before the operation, or were in shock or hemodynamically unstable at the time they went into the operating room. High-acuity patients represent 5% of our total CABG volume. In a 2-year pool of deaths in all emergency patients, 13 of 16 deaths had occurred in highacuity patients. Now we had arrived at a focus that was useful to address. We took this information back to the staff and focused attention on our high-acuity subset of patients. In accord with the Hawthorn effect, which states that as soon as you start looking at something it begins to change, this focused attention led to many changes in our practices. The cardiologists changed their thinking. They previously had been moving these unstable patients into operation quickly, considering it important to have them revascularized as quickly as possible. Now they began to stabilize them for a short time before having them undergo operation. Our database had shown that only 24% of the high-acuity patients had been given a preoperative intraaortic balloon pump; this was due both to the desire to rush them into operation and to the desire to avoid complications of leg edema. With new awareness of the risk level of these patients, this approach was reconsidered and the following year 89% of the comparable patients had a preoperative intraaortic balloon pump. The surgeons discussed their cardioplegia practices and realized they were all using different methods, with not all using retrograde cardioplegia. Practices were standardized in a move toward greater use of retrograde cardioplegia to provide myocardial protection. We became aware that the use of intravenous nitroglycerin while patients were in critical care posed a problem when
S66
RISK MANAGEMENT DZIUBAN DATABASES IN CLINICAL PRACTICE
Ann Thorac Surg 1997;64:S64 –7
Fig 3. Impact of program changes introduced beginning in 1993 on mortality in all coronary artery bypass graft patients. Fig 1. Impact of program changes introduced beginning in 1993 on mortality in high-acuity coronary artery bypass graft patients.
they were transferred to the floor, with patients sometimes destabilizing when they were switched to a nitropaste patch, so this policy was changed. This improvement process eventually extended throughout our entire program. For example, concern about balloon complications prompted our critical-care nurses to begin tracking all balloon patients and identifying the types of complications that occurred, leading to changes in our balloon practices that reduced the frequency of complications. We established joint conferences with all relevant staff groups to provide integrated direction to our program. We also became much more open about sharing data and found that this defused much suspicion and distrust. Holding data close to the chest makes others assume there is something in the data to hide.
Impact of Program Changes Since the introduction of program changes beginning in 1993, mortality has decreased in both high-acuity pa-
Fig 2. Impact of program changes introduced beginning in 1993 on mortality in coronary artery bypass graft patients other than highacuity patients.
tients and all other CABG patients (Figs 1, 2) [1, 2]. Annual program volume has increased. Overall CABG mortality has decreased from 4.5% to about 1.5%. Overall mortality has been consistently under expected mortality, and expected mortality has remained fairly constant, indicating that we have not been avoiding operating on high-risk patients (Fig 3). Initiatives that we undertook directed at length of stays were effective in reducing both preoperative and postoperative stays (Fig 4).
Summary: Lessons Learned Under Fire Experience at St. Peter’s Hospital teaches a number of important lessons: (1) We need to become the experts in using our own data, thus avoiding becoming the victim of others who will use the data less knowledgeably and less appropriately. We need to take the initiative, to acknowledge our data honestly, and to use these data effectively and responsibly. (2) Everyone in a program both influences the program and shares in its fate. (3) Bottom-line results produce more questions than answers; they represent only the beginning of the search for information. (4) Explore your database to find areas for improvement. Even preformed impressions of where problems or opportunities exist in your program should be confirmed by
Fig 4. Impact of program changes introduced beginning in 1993 on length of stay for all coronary artery bypass graft patients.
Ann Thorac Surg 1997;64:S64 –7
information in your database. Objective data always are more effective than personal opinion in convincing others. (5) Use an external database to gain perspective on your own data. The most comparable data come from others using the same database information, and hence the same definitions. (6) Seek ways to optimize care rather than seeking just to avoid errors in care. Errors in care probably occur in a small minority of patients. The payoff is greater in affecting care for the majority of your patient population. (7) Integrate everyone as a team to optimize the overall care process. If each specialty optimizes only its own role, the overall process remains suboptimized. In today’s world of advanced specialties, opportunities for the improvement of care are more likely to be found in the boundaries between specialties.
RISK MANAGEMENT DZIUBAN DATABASES IN CLINICAL PRACTICE
S67
We thought our program at St. Peter’s was a good one. It took information from our database to tell us that although the program was not bad, it could be improved. And it took information from the database to help us determine the areas of needed improvement. Are there any such areas in your program? Develop a database now, start using it, and you will know.
References 1. Chassin MR, Hannan EL, DeBuono BA. Benefits and hazards of reporting medical outcomes publicly. N Engl J Med 1996; 334:394– 8. 2. Dziuban SW Jr, McIlduff JB, Miller SJ, Dal Col RH. How a New York cardiac surgery program uses outcomes data. Ann Thorac Surg 1994;58:1871– 6.