- Email: [email protected]
Measurement, Certification, and Quality
Measurement, Certification, and Quality Meeting Enduring Challenges with Modern Tools Robert M. Wachter, MD The author, a former chair of the ABIM, describes the challenges that the board certification enterprise is experiencing as medicine shifts from being a paper-based to a digital industry. While there are clearly threats to board certification, he argues that boards can remain highly relevant if they focus on areas in which they can make unique contributions, such as the measurement of cognitive skills, diagnostic accuracy, “keeping up,” and procedural skills. Ophthalmology 2016;123:S46-S49 ª 2016 by the American Academy of Ophthalmology.
With its founding in 1916, the American Board of Ophthalmology (ABO) launched the entire enterprise of physician quality assessment, an enterprise that, I believe, has saved many thousands of lives. Despite its many contributions to patient care and education, today we find the entire board enterprise under assault from a variety of forces that are unsympathetic to the premise that physician quality and safety need to be measured, or that the boards are the best organizations to carry out such measurement. I was privileged to spend a decade on the board of directors of the American Board of Internal Medicine (ABIM), culminating with my term as chair from 2012 through 2013. During my period of service with the board, we certainly had a sense of the growing disquiet among our diplomates, many increasingly unhappy with a variety of regulatory requirements that some believed were unproductive and a distraction from their core work. However, I did not fully understand the depth of the concerns until we, in collaboration with many of the other boards under the umbrella of the American Board of Medical Specialties, launched our new process of continuous maintenance of certification in 2013. The firestorm that resulted threatened the very existence of the ABIM, leading to a famous February 2015 letter by our chief executive officer, Dr. Richard Baron: “Dear Internal Medicine Community,” it began. “ABIM clearly got it wrong. We launched programs that weren’t ready and we didn’t deliver a [maintenance of certification] program that physicians found meaningful. We want to change that.”1 Despite this apology, the ABIM continues to be under assault by physicians who question the legitimacy of the board certification process and who believe that the quality of their practice is either self-evident or should be judged by their participation in continuing medical education activities. At this writing, it is not clear how this battle will end. Of course, the measurement of physician quality has always been contentious. The founding of the ABO in 1916 came soon after the seminal work of Ernest Codman, the Massachusetts General Hospital surgeon who aspired to create a so-called end results hospital, in which the outcomes of patients would be
S46
2016 by the American Academy of Ophthalmology Published by Elsevier Inc.
measured and reported publicly. In 1914, 2 years before the ABO’s founding, Codman received a letter from a colleague that began, “The very enemies who lurk in second story windows with muffled rifles are waiting your passing.”2 One wonders what would have happened to Codman had social media been around a century ago! Although he was spared a Twitter onslaught, his experience illustrates that the work of measuring the quality of physicians has always stirred up significant passion within our profession. In this article, I describe today’s context for the boards’ work as the ABO enters its second century. I highlight 2 major trends: the growing pressure for value in American health care and the information technology revolution. After laying out these trends, I add some thoughts about where the boards fit into this increasingly crowded, fast-paced, and fractious landscape.
The Case for Change Why are we being pressured to change? Well, that one is easy. There is growing evidence that American health care does not produce a high-value (that is, quality divided by cost) product. This impression has been bolstered by a substantial body of evidence and several influential reports. For example, the Institute of Medicine’s 2000 report on patient safety, To Err Is Human, presented data showing that 44 000 to 98 000 Americans die as a result of medical mistakes each year, the equivalent of a large jumbo jet crashing every day.3 In 2003, McGlynn et al4 published a study in the New England Journal of Medicine that found that American medicine adheres to evidence-based practice 54% of the time. There are equally compelling data showing that access and patient satisfaction are problematic and highlighting enormous variations in care and significant health care disparities. On top of that, of course, is the staggering cost of American health care, which recently topped $3 trillion, Statement of Potential Conflict of Interest and Funding/Support: See page S49. http://dx.doi.org/10.1016/j.ophtha.2016.06.015 ISSN 0161-6420/16
Wachter
Measurement, Certification, and Quality
representing a far greater fraction of our gross domestic product (17%) than that of any other country. The pressure for value is only one of the sea changes in the American medical landscape. In addition, in the last few years we have gone from a primarily analog, paper-based industry to one that is fundamentally digital. This movement was prompted by $30 billion of federal incentive payments dispersed between 2010 and 2015, incentives that succeeded in increasing the penetration of electronic health records from approximately 10% to nearly 80% in both doctor’s offices and hospitals.5 My own belief is that these 2 transformational trendsdthe pressure to deliver high-value care and the digitization of health caredare rapidly shifting the entire medical landscape. Today, the demand for value is the greater of the 2 forces: physicians and health care delivery organizations are feeling increasing pressure from a variety of fronts to deliver better, safer, more satisfying care at a lower cost. But I predict that, 10 years from now, we will look back and say that the change from analog to digital was the larger catalyst for transformation. Why? Because this is the history of every other industry that has been touched by digitization. Just consider the plight of taxi drivers, hotel operators, or camera manufacturers in the era of Uber, Airbnb, and digital photography. In medicine, we are seeing the earliest stages of our digital transformation, and it’s not surprising that we’ve not yet witnessed the full impact of digital disruption. I believe we will, within a few years. Why have we been somewhat sheltered by the disruptive forces of digitization? I believe it is because our digitization has been limited, focused mostly on transformation within our professional sphere. But that is changing rapidly. The $30 billion of federal incentive payments led to 2 major upticks in computerization, one predicted and intended, the other less predicted but perhaps ultimately more important. The predicted one was convincing doctors and hospitals to buy electronic health records. That is what the money was designed to support, and it did, with a marked increase in the penetration of electronic health records built by familiar companies such as Epic, Cerner, and Allscripts. But the federal incentives did another thing, one that ultimately may be more important: awakening Silicon Valley to the health care market, which, after all, represents 17% of the gross domestic product and, up to that point, was the last large swath of the economy to remain stubbornly analog. As soon as hospitals and doctor’s offices went digital, the funders and developers of Silicon Valley saw their chance to enter the health care world in a decisive way. You see that in the form of Fitbits, Apple Watches, and a multitude of other apps, wearables, and sensors. Even with this, the impact of these consumer-facing information technologies is relatively limited, because they’re currently all siloed. There is no ubiquitous flow of data through the entire health care system, and it is often such liquid flow of data that catalyzes the massive disruptions we’ve seen in other fields.
Interoperability The concept of interoperability is familiar to us: it is why you can have an AT&T phone and effortlessly call a
Verizon phone, or you can put your Bank of America card into a Citibank machine and withdraw $100. Health care today is like the Transcontinental Railroad, with one set of tracks being the enterprise electronic health records, and the other (coming from the other coast) being the consumerfacing apps, sensors, and wearables. Today, those tracks mostly don’t connect. But within 5 years, I believe they will, as regulatory and business pressures will lead someone to lay the metaphorical golden spike. When that happens, disruptive innovation is likely to be unleashed throughout the system. One part of the health care system likely to be disrupted will be the work of those, like the boards, who are in the business of measuring and influencing the quality of care. In my book The Digital Doctor, I explored our early awkward stage of health care digitization. Based on nearly 100 interviews, I described the lack of user-centered design; the impact on physicians, who find themselves serving as very expensive, very unhappy data entry clerks; and the capacity for information technology to facilitate new kinds of medical mistakes. The latter category included a staggering error at my own hospital: we gave a 16-year-old teenager a 39-fold overdose of a common antibiotic, despite a state-of-the-art electronic health record.6 What was going on? I came to believe that Harvard political scientist Ronald Heifetz had it right when he described 2 kinds of changes in organizations: technical change and adaptive change. Technical change is straightforward: simply follow a set of directions and you get it right. Adaptive changes are, in Heifetz’s words, “Problems that require people themselves to change. In adaptive problems, the people are the problem and the people are the solution. And leadership is about mobilizing and engaging the people with the problem rather than trying to anesthetize them so you can go off and solve it on your own.”7 In health information technology, we treated the entry of technology into our extraordinarily complex world as technical change, whereas it is truly the mother of all adaptive changes. If we are to get it right, we need to understand that you can’t simply put in a computer into a complex health care ecosystem and have it work right. The computers change everything about the work, and therefore we have to re-envision the roles of the people and the workflow to take full advantage of our new digital tools.
Digital Medicine Meets Board Certification All of this will have an important impact on the area of board certification. As other stakeholders, including delivery systems, payers, and journalists, become interested in measuring whether a doctor is any good, they are likely to succeed in certain areas. The nearly complete penetration of electronic health records and the availability of big data analytics will allow others to measure certain dimensions of physician quality, such as patient experience, adherence to evidence, appropriateness, and maybe even teamwork. We are left to struggle with a question: what are the unique competencies that boards have in measuring the quality of physician care? I come up with the following:
S47
Ophthalmology Volume 123, Number 9, Supplement, September 2016 core knowledge, whether physicians are “keeping up,” diagnostic reasoning, and procedural skills. The boards also may have a role as a trusted agent for analyzing and reporting measures created by other parties. The issue of big data is interesting. Think about how Amazon and Netflix tell you what books or movies “customers like you” also liked. It’s not a large leap from that kind of analysis to an electronic health record analyzing a patient’s data and saying: “patients like this turned out to have lupus.” Or, “patients like this did better on one antibiotic or another.” Or, “patients like this did better when they went to a different hospital or saw a different physician.” This use of big data methods to analyze electronic health records may well be a game changer when it comes to measuring the quality of physicians. Moreover, there is always a tension between secrecy and transparency. Very few professions are unambiguously enthusiastic about full transparency, believing that customers (in our case, patients) will not interpret the data correctly, or that the data will give mistaken impressions of quality, perhaps because of the inability to adjust for the baseline characteristics that may influence outcomes. In this tension between secrecy and transparency, historically the professionals have had the edge because they control their own records. However, in this particular tug of war, digitization changes the balance considerably. Think about it: patients have had the right to access their medical records since the mid 1990s (it’s part of the Health Insurance Portability and Accountability Act). Notwithstanding this right, in a paper world, a patient who went to a hospital and asked to see a copy of her record usually would have been told that it would take 1 month to photocopy and would cost $1 per page. Today, all that separates a patient from his or her medical record is a password, and more than 5 million patients in the United States now have complete access to their medical records electronically, not only laboratory and radiologic results, but full physician notes.8 As these data become available, new organizations are entering the space of physician measurement. Last year, the investigative journalism organization ProPublica published its Surgeon’s Scorecard, a system based on publicly available electronic data.9 We will see more and more of these kinds of efforts, and with it the boards’ monopolies on measurement and reporting of physician quality will erode. Unwanted transparency is only one of the things that unsettles today’s physicians. Many physicians report a massive measurement burden, because electronic health records are enablers for quality measurers, accreditors, and boards to require physicians to document “just one more thing,” without recognizing that the ultimate impact of this thinking is several dozen additional quality measures or billing requirements. In a recent op-ed in the New York Times, I wrote of the plight of physicians and K-12 teachers, both suffering under the weight of this new effort to measure the quality of their work via student test scores and quality measures. I fretted that we are “hitting the target but missing the point.”10 In both medicine and teaching, we are witnessing skyrocketing rates of professional burnout. Among physicians, burnout rates now top 50%, and some fields have rates in the 70% to 80% range.11 Ophthalmology is somewhat better,
S48
with burnout rates of approximately 40%. But even it has seen a marked increase in burnout rates over the past 4 years.
What Can Boards Do? In this data-rich environment, what are the unique roles of boards? I’d focus particularly on diagnostic accuracy and procedural skills. It is difficult for me to envision electronic health record-derived data being able to demonstrate clearly whether a physician is keeping up, or is an excellent diagnostician, or is a competent proceduralist. In the past few years the issue of diagnostic reasoning has received more attention, in part because it was largely ignored in the early days of the patient safety movement. This began at the start of the movement. In the seminal Institute of Medicine report To Err Is Human, the term medication errors is used 70 times, whereas the term diagnostic errors comes up twice, although most studies show that diagnostic errors are the most common type of medical mistake.3 Many of today’s quality measures ask questions like, “For the patient with heart failure, did you give the right treatment?” The problem with this measurement schema is that a physician can look like a superb practitioner by giving patients with pneumonia the correct antibiotics or doing the correct test for patients with heart failure, even if every diagnosis is wrong.12 It is wonderful that the issue is attracting far more attention, as evidenced by a 2015 National Academy of Medicine report, but we continue to lag behind in our ability to measure such errors.13 As long as this is the case, the boards retain a unique role in reassuring the public that their physicians are keeping up and have the analytical and cognitive skills to be effective diagnosticians. Another unique role for boards is in the assessment of procedural skills. An important 2013 study by Birkmeyer et al14 involved review of videotapes of bariatric surgeons by peers. The peers assessed the technical skills of the bariatric surgeons based on the video reviews, rating them in areas such as fluidity and exposure. The surgeons varied quite considerably in their perceived technical skills as judged by their peers. Strikingly, when subsequently looking at the outcomes of the surgeons’ patients, the technical skill ratings (drawn from the video peer reviews) were correlated highly with quality and safety outcomes, including mortality rates, infection rates, and return to operating room rates. In fact, the technical scores were far more predictive of outcomes than were variables such as years in practice or the presence of fellowship training. Clearly, we need to do a better job assessing these technical skills, and if anyone is going to figure out how to do this, it will have to be the boards. Yet, at least so far, this has been an area that largely has been neglected.
Conclusions The bottom line is that we find ourselves in a new world, with the combination of intense pressure to improve health
Wachter
Measurement, Certification, and Quality
care value and widespread digitization meaning that others are invading what traditionally has been the boards’ space. In some cases, these others have significant advantages, including investment capital and perhaps even the lack of a legacy, which may encourage them to adapt new approaches or to be somewhat unencumbered by political realities. However, they lack the board’s status of being “of the profession, but for the public.” In my own view, just the fact that we come out of the profession and enjoy a century of history will not be enough to win the hearts, minds, and market in the world of quality assessment. It is vitally important that we understand these competitive threats and see the pressures to improve value and our new digital tools as opportunities for us to conduct our work in innovative new ways. We need to take full advantage of what we alone can do while partnering with other organizations, perhaps even ceding parts of our work to others able to do some of what we do as well as or better than we can. It will be by building on our rich traditions while embracing the new tools that a digital health care world has to offer that we can create the foundation for another 100 years of important work.
4. 5.
6. 7.
8. 9. 10.
11.
References 1. American Board of Internal Medicine. ABIM announces changes to MOC program. Available at: http://www.abim. org/news/abim-announces-immediate-changes-to-moc-program. aspx. Accessed February 3, 2015. 2. Donabedian A. The end results of health care: Ernest Codman’s contribution to quality assessment and beyond. Milbank Q 1989;67:245. 3. Institute of Medicine (US) Committee on Quality of Health Care in America; Kohn LT, Corrigan JM, Donaldson MS, eds.
12. 13. 14.
To Err Is Human: Building a Safer Healthcare System. Washington, DC: National Academies Press; 2000. McGlynn EG, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348:2635–45. Charles D, Gabriel M, Searcy M. Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008e2014. ONC Data Brief. Federal Report; April 23, 2015. Wachter RM. The Digital Doctor: Hope, Hype, and Harm at the Dawn of Medicine’s Computer Age. New York: McGrawHill; 2015. Vedantam S. Lessons in leadership: it’s not about you (it’s about them). Morning Edition. National Public Radio. Available at: http://www.npr.org/2013/11/11/230841224/lessons-in-leadershipits-not-about-you-its-about-them. Accessed: November 11, 2013. Delbanco T, Walker J, Bell SK, et al. Inviting patients to read their doctors’ notes: a quasi-experimental study and a look ahead. Ann Intern Med 2012;157:461–70. ProPublica. Surgeon scorecard. Available at: https://projects. propublica.org/surgeons/. Accessed June 30, 2016. Wachter RM. How measurement fails doctors and teachers. New York Times. Availabe at: http://www.nytimes.com/2016/ 01/17/opinion/sunday/how-measurement-fails-doctors-andteachers.html; Accessed June 30, 2016. Shanafelt T, Hasan O, Dyrbye LN, et al. Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014. Mayo Clin Proc 2015;90:1600–13. Wachter RM. Why diagnostic errors don’t get any respectdand what can be done about them. Health Aff (Millwood) 2010;29:1605–10. National Academies of Sciences, Engineering, and Medicine. Improving Diagnosis in Health Care. Washington, DC: The National Academies Press; 2015. Birkmeyer JD, Finks JF, O’Reilly A, et al. Michigan Bariatric Surgery Collaborative. Surgical skill and complication rates after bariatric surgery. N Engl J Med 2013;369:1434–42.
Footnotes and Financial Disclosures Originally received: April 18, 2016. Final revision: June 3, 2016. Accepted: June 3, 2016.
Analysis and interpretation: Wachter Data collection: Wachter Manuscript no. 2016-795.
Department of Medicine, University of California, San Francisco, San Francisco, California. Presented at: American Board of Ophthalmology 100th Anniversary Symposium, March 2016, San Francisco, California. Financial Disclosure(s): The author(s) have no proprietary or commercial interest in any materials discussed in this article. Author Contributions: Conception and design: Wachter
Obtained funding: none Overall responsibility: Wachter Abbreviations and Acronyms: ABIM ¼ American Board of Internal Medicine; ABO ¼ American Board of Ophthalmology. Correspondence: Robert M. Wachter, MD, Department of Medicine, University of California, San Francisco, 505 Parnassus Avenue, Room M994, San Francisco, CA 94143-0120. E-mail: [email protected].
S49
With its founding in 1916, the American Board of Ophthalmology (ABO) launched the entire enterprise of physician quality assessment, an enterprise that, I believe, has saved many thousands of lives. Despite its many contributions to patient care and education, today we find the entire board enterprise under assault from a variety of forces that are unsympathetic to the premise that physician quality and safety need to be measured, or that the boards are the best organizations to carry out such measurement. I was privileged to spend a decade on the board of directors of the American Board of Internal Medicine (ABIM), culminating with my term as chair from 2012 through 2013. During my period of service with the board, we certainly had a sense of the growing disquiet among our diplomates, many increasingly unhappy with a variety of regulatory requirements that some believed were unproductive and a distraction from their core work. However, I did not fully understand the depth of the concerns until we, in collaboration with many of the other boards under the umbrella of the American Board of Medical Specialties, launched our new process of continuous maintenance of certification in 2013. The firestorm that resulted threatened the very existence of the ABIM, leading to a famous February 2015 letter by our chief executive officer, Dr. Richard Baron: “Dear Internal Medicine Community,” it began. “ABIM clearly got it wrong. We launched programs that weren’t ready and we didn’t deliver a [maintenance of certification] program that physicians found meaningful. We want to change that.”1 Despite this apology, the ABIM continues to be under assault by physicians who question the legitimacy of the board certification process and who believe that the quality of their practice is either self-evident or should be judged by their participation in continuing medical education activities. At this writing, it is not clear how this battle will end. Of course, the measurement of physician quality has always been contentious. The founding of the ABO in 1916 came soon after the seminal work of Ernest Codman, the Massachusetts General Hospital surgeon who aspired to create a so-called end results hospital, in which the outcomes of patients would be
S46
2016 by the American Academy of Ophthalmology Published by Elsevier Inc.
measured and reported publicly. In 1914, 2 years before the ABO’s founding, Codman received a letter from a colleague that began, “The very enemies who lurk in second story windows with muffled rifles are waiting your passing.”2 One wonders what would have happened to Codman had social media been around a century ago! Although he was spared a Twitter onslaught, his experience illustrates that the work of measuring the quality of physicians has always stirred up significant passion within our profession. In this article, I describe today’s context for the boards’ work as the ABO enters its second century. I highlight 2 major trends: the growing pressure for value in American health care and the information technology revolution. After laying out these trends, I add some thoughts about where the boards fit into this increasingly crowded, fast-paced, and fractious landscape.
The Case for Change Why are we being pressured to change? Well, that one is easy. There is growing evidence that American health care does not produce a high-value (that is, quality divided by cost) product. This impression has been bolstered by a substantial body of evidence and several influential reports. For example, the Institute of Medicine’s 2000 report on patient safety, To Err Is Human, presented data showing that 44 000 to 98 000 Americans die as a result of medical mistakes each year, the equivalent of a large jumbo jet crashing every day.3 In 2003, McGlynn et al4 published a study in the New England Journal of Medicine that found that American medicine adheres to evidence-based practice 54% of the time. There are equally compelling data showing that access and patient satisfaction are problematic and highlighting enormous variations in care and significant health care disparities. On top of that, of course, is the staggering cost of American health care, which recently topped $3 trillion, Statement of Potential Conflict of Interest and Funding/Support: See page S49. http://dx.doi.org/10.1016/j.ophtha.2016.06.015 ISSN 0161-6420/16
Wachter
Measurement, Certification, and Quality
representing a far greater fraction of our gross domestic product (17%) than that of any other country. The pressure for value is only one of the sea changes in the American medical landscape. In addition, in the last few years we have gone from a primarily analog, paper-based industry to one that is fundamentally digital. This movement was prompted by $30 billion of federal incentive payments dispersed between 2010 and 2015, incentives that succeeded in increasing the penetration of electronic health records from approximately 10% to nearly 80% in both doctor’s offices and hospitals.5 My own belief is that these 2 transformational trendsdthe pressure to deliver high-value care and the digitization of health caredare rapidly shifting the entire medical landscape. Today, the demand for value is the greater of the 2 forces: physicians and health care delivery organizations are feeling increasing pressure from a variety of fronts to deliver better, safer, more satisfying care at a lower cost. But I predict that, 10 years from now, we will look back and say that the change from analog to digital was the larger catalyst for transformation. Why? Because this is the history of every other industry that has been touched by digitization. Just consider the plight of taxi drivers, hotel operators, or camera manufacturers in the era of Uber, Airbnb, and digital photography. In medicine, we are seeing the earliest stages of our digital transformation, and it’s not surprising that we’ve not yet witnessed the full impact of digital disruption. I believe we will, within a few years. Why have we been somewhat sheltered by the disruptive forces of digitization? I believe it is because our digitization has been limited, focused mostly on transformation within our professional sphere. But that is changing rapidly. The $30 billion of federal incentive payments led to 2 major upticks in computerization, one predicted and intended, the other less predicted but perhaps ultimately more important. The predicted one was convincing doctors and hospitals to buy electronic health records. That is what the money was designed to support, and it did, with a marked increase in the penetration of electronic health records built by familiar companies such as Epic, Cerner, and Allscripts. But the federal incentives did another thing, one that ultimately may be more important: awakening Silicon Valley to the health care market, which, after all, represents 17% of the gross domestic product and, up to that point, was the last large swath of the economy to remain stubbornly analog. As soon as hospitals and doctor’s offices went digital, the funders and developers of Silicon Valley saw their chance to enter the health care world in a decisive way. You see that in the form of Fitbits, Apple Watches, and a multitude of other apps, wearables, and sensors. Even with this, the impact of these consumer-facing information technologies is relatively limited, because they’re currently all siloed. There is no ubiquitous flow of data through the entire health care system, and it is often such liquid flow of data that catalyzes the massive disruptions we’ve seen in other fields.
Interoperability The concept of interoperability is familiar to us: it is why you can have an AT&T phone and effortlessly call a
Verizon phone, or you can put your Bank of America card into a Citibank machine and withdraw $100. Health care today is like the Transcontinental Railroad, with one set of tracks being the enterprise electronic health records, and the other (coming from the other coast) being the consumerfacing apps, sensors, and wearables. Today, those tracks mostly don’t connect. But within 5 years, I believe they will, as regulatory and business pressures will lead someone to lay the metaphorical golden spike. When that happens, disruptive innovation is likely to be unleashed throughout the system. One part of the health care system likely to be disrupted will be the work of those, like the boards, who are in the business of measuring and influencing the quality of care. In my book The Digital Doctor, I explored our early awkward stage of health care digitization. Based on nearly 100 interviews, I described the lack of user-centered design; the impact on physicians, who find themselves serving as very expensive, very unhappy data entry clerks; and the capacity for information technology to facilitate new kinds of medical mistakes. The latter category included a staggering error at my own hospital: we gave a 16-year-old teenager a 39-fold overdose of a common antibiotic, despite a state-of-the-art electronic health record.6 What was going on? I came to believe that Harvard political scientist Ronald Heifetz had it right when he described 2 kinds of changes in organizations: technical change and adaptive change. Technical change is straightforward: simply follow a set of directions and you get it right. Adaptive changes are, in Heifetz’s words, “Problems that require people themselves to change. In adaptive problems, the people are the problem and the people are the solution. And leadership is about mobilizing and engaging the people with the problem rather than trying to anesthetize them so you can go off and solve it on your own.”7 In health information technology, we treated the entry of technology into our extraordinarily complex world as technical change, whereas it is truly the mother of all adaptive changes. If we are to get it right, we need to understand that you can’t simply put in a computer into a complex health care ecosystem and have it work right. The computers change everything about the work, and therefore we have to re-envision the roles of the people and the workflow to take full advantage of our new digital tools.
Digital Medicine Meets Board Certification All of this will have an important impact on the area of board certification. As other stakeholders, including delivery systems, payers, and journalists, become interested in measuring whether a doctor is any good, they are likely to succeed in certain areas. The nearly complete penetration of electronic health records and the availability of big data analytics will allow others to measure certain dimensions of physician quality, such as patient experience, adherence to evidence, appropriateness, and maybe even teamwork. We are left to struggle with a question: what are the unique competencies that boards have in measuring the quality of physician care? I come up with the following:
S47
Ophthalmology Volume 123, Number 9, Supplement, September 2016 core knowledge, whether physicians are “keeping up,” diagnostic reasoning, and procedural skills. The boards also may have a role as a trusted agent for analyzing and reporting measures created by other parties. The issue of big data is interesting. Think about how Amazon and Netflix tell you what books or movies “customers like you” also liked. It’s not a large leap from that kind of analysis to an electronic health record analyzing a patient’s data and saying: “patients like this turned out to have lupus.” Or, “patients like this did better on one antibiotic or another.” Or, “patients like this did better when they went to a different hospital or saw a different physician.” This use of big data methods to analyze electronic health records may well be a game changer when it comes to measuring the quality of physicians. Moreover, there is always a tension between secrecy and transparency. Very few professions are unambiguously enthusiastic about full transparency, believing that customers (in our case, patients) will not interpret the data correctly, or that the data will give mistaken impressions of quality, perhaps because of the inability to adjust for the baseline characteristics that may influence outcomes. In this tension between secrecy and transparency, historically the professionals have had the edge because they control their own records. However, in this particular tug of war, digitization changes the balance considerably. Think about it: patients have had the right to access their medical records since the mid 1990s (it’s part of the Health Insurance Portability and Accountability Act). Notwithstanding this right, in a paper world, a patient who went to a hospital and asked to see a copy of her record usually would have been told that it would take 1 month to photocopy and would cost $1 per page. Today, all that separates a patient from his or her medical record is a password, and more than 5 million patients in the United States now have complete access to their medical records electronically, not only laboratory and radiologic results, but full physician notes.8 As these data become available, new organizations are entering the space of physician measurement. Last year, the investigative journalism organization ProPublica published its Surgeon’s Scorecard, a system based on publicly available electronic data.9 We will see more and more of these kinds of efforts, and with it the boards’ monopolies on measurement and reporting of physician quality will erode. Unwanted transparency is only one of the things that unsettles today’s physicians. Many physicians report a massive measurement burden, because electronic health records are enablers for quality measurers, accreditors, and boards to require physicians to document “just one more thing,” without recognizing that the ultimate impact of this thinking is several dozen additional quality measures or billing requirements. In a recent op-ed in the New York Times, I wrote of the plight of physicians and K-12 teachers, both suffering under the weight of this new effort to measure the quality of their work via student test scores and quality measures. I fretted that we are “hitting the target but missing the point.”10 In both medicine and teaching, we are witnessing skyrocketing rates of professional burnout. Among physicians, burnout rates now top 50%, and some fields have rates in the 70% to 80% range.11 Ophthalmology is somewhat better,
S48
with burnout rates of approximately 40%. But even it has seen a marked increase in burnout rates over the past 4 years.
What Can Boards Do? In this data-rich environment, what are the unique roles of boards? I’d focus particularly on diagnostic accuracy and procedural skills. It is difficult for me to envision electronic health record-derived data being able to demonstrate clearly whether a physician is keeping up, or is an excellent diagnostician, or is a competent proceduralist. In the past few years the issue of diagnostic reasoning has received more attention, in part because it was largely ignored in the early days of the patient safety movement. This began at the start of the movement. In the seminal Institute of Medicine report To Err Is Human, the term medication errors is used 70 times, whereas the term diagnostic errors comes up twice, although most studies show that diagnostic errors are the most common type of medical mistake.3 Many of today’s quality measures ask questions like, “For the patient with heart failure, did you give the right treatment?” The problem with this measurement schema is that a physician can look like a superb practitioner by giving patients with pneumonia the correct antibiotics or doing the correct test for patients with heart failure, even if every diagnosis is wrong.12 It is wonderful that the issue is attracting far more attention, as evidenced by a 2015 National Academy of Medicine report, but we continue to lag behind in our ability to measure such errors.13 As long as this is the case, the boards retain a unique role in reassuring the public that their physicians are keeping up and have the analytical and cognitive skills to be effective diagnosticians. Another unique role for boards is in the assessment of procedural skills. An important 2013 study by Birkmeyer et al14 involved review of videotapes of bariatric surgeons by peers. The peers assessed the technical skills of the bariatric surgeons based on the video reviews, rating them in areas such as fluidity and exposure. The surgeons varied quite considerably in their perceived technical skills as judged by their peers. Strikingly, when subsequently looking at the outcomes of the surgeons’ patients, the technical skill ratings (drawn from the video peer reviews) were correlated highly with quality and safety outcomes, including mortality rates, infection rates, and return to operating room rates. In fact, the technical scores were far more predictive of outcomes than were variables such as years in practice or the presence of fellowship training. Clearly, we need to do a better job assessing these technical skills, and if anyone is going to figure out how to do this, it will have to be the boards. Yet, at least so far, this has been an area that largely has been neglected.
Conclusions The bottom line is that we find ourselves in a new world, with the combination of intense pressure to improve health
Wachter
Measurement, Certification, and Quality
care value and widespread digitization meaning that others are invading what traditionally has been the boards’ space. In some cases, these others have significant advantages, including investment capital and perhaps even the lack of a legacy, which may encourage them to adapt new approaches or to be somewhat unencumbered by political realities. However, they lack the board’s status of being “of the profession, but for the public.” In my own view, just the fact that we come out of the profession and enjoy a century of history will not be enough to win the hearts, minds, and market in the world of quality assessment. It is vitally important that we understand these competitive threats and see the pressures to improve value and our new digital tools as opportunities for us to conduct our work in innovative new ways. We need to take full advantage of what we alone can do while partnering with other organizations, perhaps even ceding parts of our work to others able to do some of what we do as well as or better than we can. It will be by building on our rich traditions while embracing the new tools that a digital health care world has to offer that we can create the foundation for another 100 years of important work.
4. 5.
6. 7.
8. 9. 10.
11.
References 1. American Board of Internal Medicine. ABIM announces changes to MOC program. Available at: http://www.abim. org/news/abim-announces-immediate-changes-to-moc-program. aspx. Accessed February 3, 2015. 2. Donabedian A. The end results of health care: Ernest Codman’s contribution to quality assessment and beyond. Milbank Q 1989;67:245. 3. Institute of Medicine (US) Committee on Quality of Health Care in America; Kohn LT, Corrigan JM, Donaldson MS, eds.
12. 13. 14.
To Err Is Human: Building a Safer Healthcare System. Washington, DC: National Academies Press; 2000. McGlynn EG, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med 2003;348:2635–45. Charles D, Gabriel M, Searcy M. Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008e2014. ONC Data Brief. Federal Report; April 23, 2015. Wachter RM. The Digital Doctor: Hope, Hype, and Harm at the Dawn of Medicine’s Computer Age. New York: McGrawHill; 2015. Vedantam S. Lessons in leadership: it’s not about you (it’s about them). Morning Edition. National Public Radio. Available at: http://www.npr.org/2013/11/11/230841224/lessons-in-leadershipits-not-about-you-its-about-them. Accessed: November 11, 2013. Delbanco T, Walker J, Bell SK, et al. Inviting patients to read their doctors’ notes: a quasi-experimental study and a look ahead. Ann Intern Med 2012;157:461–70. ProPublica. Surgeon scorecard. Available at: https://projects. propublica.org/surgeons/. Accessed June 30, 2016. Wachter RM. How measurement fails doctors and teachers. New York Times. Availabe at: http://www.nytimes.com/2016/ 01/17/opinion/sunday/how-measurement-fails-doctors-andteachers.html; Accessed June 30, 2016. Shanafelt T, Hasan O, Dyrbye LN, et al. Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014. Mayo Clin Proc 2015;90:1600–13. Wachter RM. Why diagnostic errors don’t get any respectdand what can be done about them. Health Aff (Millwood) 2010;29:1605–10. National Academies of Sciences, Engineering, and Medicine. Improving Diagnosis in Health Care. Washington, DC: The National Academies Press; 2015. Birkmeyer JD, Finks JF, O’Reilly A, et al. Michigan Bariatric Surgery Collaborative. Surgical skill and complication rates after bariatric surgery. N Engl J Med 2013;369:1434–42.
Footnotes and Financial Disclosures Originally received: April 18, 2016. Final revision: June 3, 2016. Accepted: June 3, 2016.
Analysis and interpretation: Wachter Data collection: Wachter Manuscript no. 2016-795.
Department of Medicine, University of California, San Francisco, San Francisco, California. Presented at: American Board of Ophthalmology 100th Anniversary Symposium, March 2016, San Francisco, California. Financial Disclosure(s): The author(s) have no proprietary or commercial interest in any materials discussed in this article. Author Contributions: Conception and design: Wachter
Obtained funding: none Overall responsibility: Wachter Abbreviations and Acronyms: ABIM ¼ American Board of Internal Medicine; ABO ¼ American Board of Ophthalmology. Correspondence: Robert M. Wachter, MD, Department of Medicine, University of California, San Francisco, 505 Parnassus Avenue, Room M994, San Francisco, CA 94143-0120. E-mail: [email protected].
S49