- Email: [email protected]
Diagnostic Errors in Congenital Echocardiography: Importance of Study Conditions
Diagnostic Errors in Congenital Echocardiography: Importance of Study Conditions Oscar J. Benavidez, MD, MPP, Kimberlee Gauvreau, ScD, and Tal Geva, MD, Boston, Massachusetts
Background: Diagnostic errors are unwanted clinical events that place patients at risk for injury. The authors have previously reported that a majority of congenital echocardiography errors have clinical impacts and, on the basis of a small cohort, identified factors associated with diagnostic error. The objectives of this study were (1) to evaluate patient risk factors for diagnostic errors in a large contemporary cohort and (2) to identify risk factors for situation-related diagnostic errors. Methods: Diagnostic errors were identified at a large academic pediatric cardiac center from 2004 to 2011. Clinical and situational variables were collected from diagnostic error cases and controls. Results: Among the 254 diagnostic error cases, 66% affected clinical management or patients experienced adverse events; 77% of errors were preventable or possibly preventable. Coronary arteries, pulmonary veins, and the aortic arch were most commonly involved with diagnostic errors. Multivariate analysis identified the following patient-related risk factors: rare or very rare diagnoses (adjusted odds ratio [AOR], 6.3; P < .001), high anatomic complexity (AOR, 3.4; P < .001), and weight < 5 kg (AOR, 2.7; P < .001). Risk factors related to the setting of the echocardiographic study included evening or night (7 PM to 6:59 AM) study interpretation (AOR, 2.6; P = .005) and weekend studies (Friday through Sunday) (AOR, 1.6; P = .04). The model area under the receiver operating characteristic curve was 0.833. Conclusions: In addition to patient risk factors, the setting of an echocardiographic study and interpretation contribute to risk for a diagnostic error. Studies interpreted overnight or performed during a weekend should be considered for a quality improvement activity to reduce diagnostic errors or their impact. (J Am Soc Echocardiogr 2014;-:---.) Keywords: Congenital echocardiography, Diagnostic error, Quality, Outcomes
Diagnostic errors are unwanted clinical events that place patients at risk for injury and are a leading cause of malpractice claims.1,2 Echocardiography is the first line of diagnostic investigation among patients suspected of having congenital heart disease,3,4 and diagnostic errors may place patients at risk for suboptimal outcomes.5,6 We have previously reported that a majority of congenital echocardiographic diagnostic errors have clinical impacts.5 That analysis, however, was based on a relatively small cohort. Using a large contemporary cohort, the objectives of the present study were (1) to determine patient-related risk factors for diagnostic errors and (2) to identify independent risk factors related to the situation in which echocardiographic studies are performed and interpreted. From the Department of Cardiology, Boston Children’s Hospital, Harvard Medical School, Boston, Massachusetts (O.J.B.); Division of Pediatric/Congenital Cardiology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts (O.J.B., K.G., T.G.); and Department of Pediatrics, Harvard Medical School, Boston, Massachusetts (O.J.B., K.G., T.G.). Dr Benavidez is supported a grant from the Robert Wood Johnson Foundation’s Harold Amos Medical Faculty Program. Reprint requests: Oscar J. Benavidez, MD, MPP, Massachusetts General Hospital, Division of Pediatric/Congenital Cardiology, 175 Cambridge Street, Boston, MA 02114 (E-mail: [email protected]). 0894-7317/$36.00 Copyright 2014 by the American Society of Echocardiography. http://dx.doi.org/10.1016/j.echo.2014.03.001
Situational risk factors may be responsive to quality improvement strategies to reduce diagnostic errors or to mitigate their impact.
METHODS The Scientific Review Committee of the Department of Cardiology and the Institutional Review Board at Boston Children’s Hospital approved this study. The authors had full access to and take full responsibility for the integrity of the data. All authors have read and agree to the report as written. Data Source The Echocardiography Laboratory at Boston Children’s Hospital performs approximately 22,000 echocardiographic studies annually. These studies are performed in locations such as echocardiography suites, satellite cardiology clinics, operating rooms, intensive care units, recovery rooms, catheterization laboratories, cardiology wards, emergency departments, and general pediatric wards. Trained pediatric and congenital sonographers, cardiology fellows, and echocardiography staff cardiologists perform the examinations. The cardiac sonographers are provided with a requisition to perform a study, and they review the requisition and study questions before performing the study. Before study completion, the sonographer will discuss the pertinent findings with the noninvasive cardiologist. 1
2 Benavidez et al
The echocardiography staff cardiologists interpret all studies AOR = Adjusted odds ratio and issue reports, which are stored in the hospital’s electronic CI = Confidence interval medical records. The final diagnosis reported by the staff cardiologist after the interpretation of the echocardiogram was the outcome of interest in this study; preliminary results reported by cardiac sonographers or trainees were not examined. Postnatal pediatric and congenital echocardiograms were the subject of this study. Abbreviations
Diagnostic Error Case Ascertainment Between December 2004 and March 2011, as part of a continuous quality improvement initiative in the Noninvasive Cardiology Division, data related to diagnostic error cases were collected prospectively through voluntary reporting and active quality assurance mechanisms. We have previously reported these methods.5 A diagnostic error was defined as a diagnosis that was unintentionally delayed, wrong, or missed as judged from eventual appreciation of the existing data or more definitive information.5 The sources of diagnostic error case discovery included information obtained from other tests (e.g., cardiac catheterization, magnetic resonance imaging), operative observations, subsequent echocardiographic examinations, and autopsy. Data Collected The following patient demographic and case data were collected: age, referral diagnosis, weight, race, prematurity, anatomy involved in diagnostic error, and study indication or question. Other data obtained included study location, use of sedation or anesthesia, comments on image quality, day of week, and time of day the study was performed and interpreted. Data related to the number of interim studies or procedures performed before the diagnostic error discovery was also noted, including the method of discovery. Diagnostic frequency, defined as the frequency a diagnosis is encountered in the echocardiography laboratory, was categorized as follows: (1) frequent (diagnosis is observed more than once per week; e.g., patent ductus arteriosus), (2) intermediate frequency (diagnosis is observed more than once a month but less than once weekly; e.g., coarctation of the aorta), (3) rare (diagnosis is observed more than once per year but less than once monthly; e.g., inferiortype sinus venosus defect), and (4) very rare (diagnosis is observed less than once yearly; e.g., aortic–left ventricular tunnel). Anatomic complexity data were divided into 3 categories: (1) low (no significant heart disease or a single, simple structural cardiovascular anomaly; e.g., atrial septal defect or single membranous ventricular septal defect), (2) moderate (abnormalities involving more than one cardiovascular structure or diagnoses with moderately common characteristics; e.g., complete common atrioventricular canal, hypoplastic left heart syndrome, transposition of the great arteries), and (3) high (uncommon variants of moderately complex anatomic diagnoses or rare, complex anomalies; e.g., dextrocardia, superior-inferior ventricles with crisscross atrioventricular relations, hypoplastic right ventricle, and straddling mitral valve).7 Diagnostic Error Case Review Process Staff echocardiographers reviewed diagnostic error cases and other diagnostic data in addition to the images of the studies in question, as previously described5 (Figure 1). Briefly, diagnostic error cases
Journal of the American Society of Echocardiography - 2014
were reviewed by staff pediatric echocardiographers, including review of medical records and other diagnostic images in addition to the images of the studies in question. Interviews with sonographers, trainees, cardiologists, and other involved health care providers were conducted to understand the conditions and the context in which the examinations were performed and interpreted, to determine the clinical impact, and to identify primary causes of the diagnostic errors. The relevant clinical and image data were presented at a monthly noninvasive morbidity and mortality conference. This process involves a systematic review of the echocardiographic process to identify contributors to the diagnostic error in question. Case discussion included categorization of diagnostic error type, severity, preventability, and contributors or root causes. Participants in the conference include attending physicians, trainees, and cardiac sonographers from the noninvasive division, as well representatives from other divisions within the Department of Cardiology and other disciplines (e.g., cardiac anesthesia and cardiovascular surgery). A consensus based on a review of the case and the ensuing discussion was used to finalize categorization of the diagnostic error type, severity, preventability, and root cause.
Definitions and Classification of Diagnostic Errors A diagnostic error was defined as a diagnosis that was unintentionally delayed, wrong, or missed, as judged from eventual appreciation of the existing data or of more definitive information.8 Diagnostic Error Categorization. False-negative: An error that omits a finding or states that a finding is normal (or absent) when an abnormality is present, or the reader failed to include a significant diagnostic possibility (e.g., patent ductus arteriosus is ruled out or omitted when it is evident on the study images). False-positive: An error that reports an abnormality but no abnormality is present, or the reader overemphasized the significance of a finding (e.g., atrial septal defect is diagnosed when the atrial septum is intact). Discrepant diagnosis: The actual diagnosis is different from the one made (e.g., diagnosis of double-inlet left ventricle is made when the actual diagnosis is tricuspid atresia). Severity Categorization. Diagnostic errors were categorized by severity into the following categories on the basis of clinical impact: minor, moderate, severe, and catastrophic (Table 1). Minor: A diagnostic error or discrepancy that does not change patient management or affect clinical course, with little or no potential for adverse event (e.g., missed left superior vena cava to an intact coronary sinus in a patient with an otherwise structurally normal heart). Moderate: A diagnostic error or discrepancy with an impact on management, whereby the patient may be placed at risk and/or experience a transient adverse event (e.g., missed primum atrial septal defect in an infant). Major: Diagnostic error discrepancy with impact on management that results in an adverse event, including the performance of unnecessary invasive procedure, or a long-lasting or permanent adverse event (e.g., false-positive diagnosis of an atrial septal defect leading to an unnecessary surgery to close a defect that was not present). Catastrophic: A diagnostic error or discrepancy that contributed to patient death (e.g., missed anomalous coronary artery from
Benavidez et al 3
Journal of the American Society of Echocardiography Volume - Number -
Figure 1 Diagnostic error review process. The echocardiographic process is reviewed in detail from study requisition through study reporting. Each step of this process is systematically reviewed for contribution to diagnostic error. Example considerations based on common causes of diagnostic error contributors are listed below each step.
Table 1 Categorization of diagnostic error severity Category
Description
Minor
A diagnostic error or discrepancy that does not change patient management or affect clinic course, with little or no potential for adverse event (e.g., missed left superior vena cava to an intact coronary sinus in a patient with an otherwise structurally normal heart).
Moderate
A diagnostic error or discrepancy with an impact on management, whereby the patient may be placed at risk and/or experience a transient adverse event (e.g., missed primum atrial septal defect in an infant). Diagnostic error discrepancy with impact on management that results in an adverse event, including the performance of unnecessary invasive procedure, or a long-lasting or permanent adverse event (e.g., false-positive diagnosis of an atrial septal defect leading to an unnecessary surgery to close a defect that was not present).
Major
Catastrophic
A diagnostic error or discrepancy that contributed to patient death (e.g., missed anomalous coronary artery from pulmonary artery that resulted in intraoperative myocardial ischemia leading to patient death).
pulmonary artery that resulted in intraoperative myocardial ischemia leading to patient death).
conditions (e.g., an echocardiogram clearly demonstrates a patent ductus arteriosus, but the study is interpreted as showing none).
Categorization of Preventability. Diagnostic errors were categorized by preventability as not preventable, possibly preventable, or preventable (Table 2).
Diagnostic Error Cause Categorization. The underlying causes or contributors to diagnostic errors belonged to one of the following categories: administrative error, procedural error, communication error, cognitive error, technical factor, and patient-related or diseaserelated factors5 (Table 3).
Not preventable: A diagnostic error is not preventable if the images, imaging modality, or imaging conditions (e.g., attempted imaging during active cardiopulmonary resuscitation) do not permit diagnosis (e.g., failure to image a ligamentum arteriosum contributing to a vascular ring). Possibly preventable: A diagnostic error is possibly preventable if accurate diagnosis may be expected given the imaging modality and/or imaging conditions but may have required a different technique(s) or condition(s) (e.g., aortic arch anomaly that was missed because the aortic arch branching was incompletely examined). Preventable: A diagnosis is preventable if accurate diagnosis is expected given the available images, imaging modality, and/or imaging
Administrative or data entry errors: Errors that typically precede the actual examination (e.g., entering an examination report under an incorrect patient name). Procedural or conditional factors: These are factors that relate to the performance of the study or the conditions under which the examination is performed (e.g., failure to adequately interrogate the entire ventricular septum by color Doppler in a cooperative patient, resulting in a missed apical ventricular septal defect). Communication or information errors: Errors in information transfer to those performing and interpreting the examination or from
4 Benavidez et al
Journal of the American Society of Echocardiography - 2014
Table 2 Categorization of diagnostic error preventability Category
Description
Not preventable
A diagnostic error is not preventable if the images, imaging modality, or imaging conditions (e.g., attempted imaging during active cardiopulmonary resuscitation) do not permit diagnosis (e.g., failure to image a ligamentum arteriosum contributing to a vascular ring).
Possibly preventable
A diagnostic error is possibly preventable if accurate diagnosis may be expected given the imaging modality and/or imaging conditions but may have required a different technique(s) or condition(s) (e.g., aortic arch anomaly that was missed because the aortic arch branching was incompletely examined). A diagnosis is preventable if accurate diagnosis is expected or readily apparent given the study images, imaging modality, and/or imaging conditions (e.g., an echocardiogram clearly demonstrates a patent ductus arteriosus, but the study is interpreted as showing none).
Preventable
Table 3 Contributors to diagnostic error Administrative or data entry errors Incorrect name assigned to imaging data Scheduling error Incorrect data entry Procedural or conditional factors Failure to confirm patient identity or diagnosis Incomplete examination of anatomy or physiology Poor imaging environment Failure to improve imaging conditions when possible Communication or information errors Lacking or misleading patient history No access to prior studies Failure to report critical findings in a timely fashion to referring physician Incorrect requisition (unintended or no clinical question asked) Cognitive errors Insufficient knowledge base Insufficient technical skills Faulty data synthesis Lack of consideration a patient’s situation/condition that is relevant to diagnosis Misidentification/interpretation of a finding on echocardiography Premature closure of case Distraction by other diagnoses, findings, or focused question Underappreciation/consideration of a finding Overappreciation of a finding Confirmation bias Incorrect or improper calculation Technical factors Artifact Modality limitation Poor acoustic windows Equipment malfunction Patient-related or disease-related factors Rare or complex anatomy Misleading anatomy or physiology
those interpreting the study to the referring clinician. This includes providing inaccurate or misleading clinical history or incorrect study questions. For example, a treating cardiologist was concerned about an intracardiac thrombus in a patient after Fontan palliation, but this information was not communicated to the
echocardiography laboratory. Instead, the echocardiography laboratory was incorrectly asked to rule out pericardial effusion. As a result, a focused echocardiogram was performed to rule out a pericardial effusion, missing an intracardiac thrombus. In this case, failure to communicate the intentioned study indication (concern for intracardiac clot) by providing an inaccurate study question (rule out pericardial effusion) contributed to the diagnostic error. Cognitive errors: These are errors that occur during the analysis of the imaging and clinical data. For example, an echocardiographic study is requested for estimation of the pressure gradient across a right ventricle–to–pulmonary artery conduit in a patient who had an endovascular stent placed in the conduit. The conduit gradient is reported correctly, but the readily apparent free-floating fractured stent in the right ventricle is not noted. Technical factors: These are factors involving equipment malfunction or inherent limitations of echocardiography. Examples include an imaging artifact creating the appearance of a vascular structure that is then interpreted as an anomaly. Postoperative changes or obesity are examples of technical factors that can result in ‘‘poor acoustic windows,’’ which produce degraded, or nondiagnostic image quality. Patient-related or disease-related factors: Factors related to the patient’s underlying diagnosis or physiology. For example, a tortuous left atrial appendage is mistaken for an aneurysm of the left coronary artery in a patient with Kawasaki disease.
Statistical Analysis The goal of the analysis was to identify patient-related and situationrelated risk factors associated with diagnostic errors. Control cases were randomly selected (with a ratio of approximately 2.5 controls per diagnostic error case) by a statistical program (Stata/SE version 9.2 for Windows; StataCorp LP, College Station, TX) from all echocardiographic studies performed during the same calendar week the diagnostic error occurred. The imaging data, reports, and medical records of control cases were examined for the presence of unrecognized diagnostic errors and were excluded from the control group if an error was identified. The findings from the categorization of diagnostic error type, preventability, severity, anatomic factors, and potential contributors were tabulated and reported as percentages of all cases of diagnostic error. Univariate and multivariate analyses compared diagnostic error cases with controls. Demographic and situational variables were compared for diagnostic error cases versus controls using Fisher’s exact test. Variables with P values < .20 in univariate analysis were considered for inclusion in a multivariate logistic regression model. A P value of
Benavidez et al 5
Journal of the American Society of Echocardiography Volume - Number -
Table 4 Factors contributing to diagnostic errors (n = 254)
Figure 2 (A) Diagnostic error severity: only 33% of the diagnostic errors (n = 85) were of minor severity and had no impact on patient management. (B) Diagnostic error preventability: only 22% (n = 57) of the errors were not preventable. .05 was required for inclusion of a variable in the final model. Adjusted odds ratios (AORs) for having a diagnostic error with 95% confidence intervals (CIs) were calculated for variables retained in the final model.
Contributor
Cases
%
Cognitive errors Misidentification/interpretation of finding Underinterpretation of a finding Distraction by other diagnoses Overinterpretation of a finding Incorrect or improper calculation Technical factors Poor acoustic windows Modality limitation Imaging artifact Patient-related or disease-related factors Misleading anatomy or physiology Rare or complex anatomy or physiology Procedural or conditional factors Incomplete examination Procedure situation or practice Poor imaging conditions Communication or information errors Lacking or misleading patient history Incorrect requisition Administrative/other errors Incorrect height and weight data entry Incorrect patient imaged/reported
94 30 53 6 4 1 71 37 26 8 30 15 15 53 35 5 13 3 2 1 3 2 1
37 12 21 2 2 <1 28 15 10 3 10 6 6 21 14 2 5 1 <1 <1 1 <1 <1
(n = 112 [45%]) (Figure 2B). Of note, the majority of preventable errors (73%) were of moderate severity or greater.
Contributors to Diagnostic Errors Table 4 lists the factors contributing to diagnostic errors. The 3 most common factors contributing to diagnostic errors were cognitive (38%), technical (28%), and procedure related (21%).
RESULTS Between December 2004 and March 2011, 254 postnatal echocardiography diagnostic error cases were identified. During that period, approximately 147,000 postnatal echocardiograms were obtained and interpreted in our laboratory, yielding an overall error rate of 0.17%. Diagnostic Error Categorization Of the 254 diagnostic error cases identified, 70% were falsenegatives, 15% were false-positives, and 15% discrepant diagnoses. Diagnostic Error Severity Only 33% of the diagnostic errors (n = 85) were of minor severity and had no impact on patient management; 67% affected patient management, with 59% (n = 151) of moderate severity, 6% (n = 16) of major severity, and 1% (n = 2) catastrophic. Both catastrophic cases involved coronary arteries (Figure 2A). Diagnostic Error Preventability Only 22% of the diagnostic errors (n = 57) were not preventable, while 78% were preventable (n = 85 [33%]) or possibly preventable
Anatomy Involved with Diagnostic Errors Common anatomic segments involved with diagnostic errors were coronary arteries (n = 52 [20%]), pulmonary veins (n = 31 [12%]), and the aortic arch (n = 26 [10%]).
Diagnostic Error Discovery The correct diagnoses were discovered by echocardiography in 28% of cases (either from new studies or reexamination of the initial studies), intraoperative inspection in 27%, cardiac catheterization in 18%, cardiac magnetic resonance imaging in 17%, transesophageal echocardiography in 7%, and by another modality in 4% of cases. Of the 254 cases, 38% had interim imaging studies before identification of the error: 17% had one interim study, 11% had two or three interim studies, and 10% had more than three studies. Only 3% of the cases had interim cardiac magnetic resonance imaging before diagnostic error discovery. Regarding the number of invasive procedures before discovery of diagnostic error, 9% had at least one interim cardiac catheterization, and 8% had an interim cardiac surgery. Stated differently, these invasive procedures were performed without discovery of the diagnostic errors.
6 Benavidez et al
Journal of the American Society of Echocardiography - 2014
Table 5 Univariate analysis comparison of characteristics for patients with diagnostic errors versus controls with no diagnostic errors Variable
Weight (kg) <5 5–19 $20 Premature Yes No Study location CICU NICU Echocardiography lab Satellite/outpatient Other Weekend* Yes No Echocardiography performance time 7 AM to 12:59 PM 1 PM to 6:59 PM 7 PM to 6:59 PM Echocardiogram read time 7 AM to 12:59 PM 1 PM to 6:59 PM 7 PM to 6:59 AM Not reported Prevalence Frequent Intermediate Rare Very rare Complexity Low Medium High Limited study† Yes No
Total (n = 798)
Cases (n = 254)
Controls (n = 544)
200 (25%) 232 (29%) 359 (45%)
103 (41%) 89 (35%) 62 (24%)
97 (18%) 143 (26%) 297 (55%)
47 (6%) 751 (94%)
18 (7%) 236 (93%)
29 (5%) 515 (95%)
100 (13%) 40 (5%) 364 (46%) 72 (9%) 222 (28%)
59 (23%) 22 (9%) 101 (40%) 13 (5%) 59 (23%)
41 (8%) 18 (3%) 263 (48%) 59 (11%) 163 (30%)
140 (18%) 658 (82%)
65 (26%) 189 (74%)
75 (14%) 469 (86%)
436 (55%) 324 (41%) 38 (5%)
127 (50%) 109 (43%) 18 (7%)
309 (57%) 215 (39%) 20 (4%)
262 (33%) 474 (59%) 61 (8%) 1 (<1%)
70 (28%) 145 (57%) 39 (15%) 0 (0%)
192 (35%) 329 (60%) 22 (4%) 1 (<1%)
410 (51%) 177 (22%) 169 (21%) 25 (3%)
55 (22%) 68 (27%) 108 (43%) 23 (9%)
355 (65%) 109 (20%) 61 (11%) 2 (<1%)
387 (50%) 280 (36%) 113 (14%)
50 (20%) 131 (51%) 73 (29%)
337 (62%) 149 (28%) 40 (7%)
115 (14%) 683 (86%)
26 (10%) 228 (90%)
89 (16%) 455 (84%)
P
<.001
.34
<.001
<.001
.05
<.001
<.001
<.001
.02
CICU, Cardiac intensive care unit; NICU, neonatal intensive care unit. *Friday night after 6 PM to Monday, 6:59 AM. † An echocardiographic study that yields few images or on which the image quality is such that the study questions cannot be answered; the echocardiographic report is written as a summary sentence or brief paragraph with few details.
Comparison of Diagnostic Error Cases with Controls Results of the univariate analysis are summarized in Table 5. Compared with controls, diagnostic error cases were more likely to involve a lower body weight, to be performed in a critical care unit, to be performed and interpreted at night or on a weekend, to involve uncommon diagnoses, and to involve greater anatomic complexity. Patient-Related Risk Factors for Diagnostic Errors Multivariate analysis identified the following patient-related variables as risk factors diagnostic errors (Table 6): rare or very rare diagnoses (AOR, 6.2; 95% CI, 3.8–10.1, P < .001), moderate level of anatomic complexity (AOR, 3.3; 95% CI, 2.1–5.1; P < .001), high level of
anatomic complexity (AOR, 3.0; 95% CI, 1.7–5.4, P < .001), patient weight < 5 kg (AOR, 3.5; 95% CI, 1.1–10.6; P = .031), and patient weight 5 to 19 kg (AOR, 2.2; 95% CI, 1.4–3.4, P < .001).
Situation-Related Risk Factors for Diagnostic Errors While controlling for patient-related risk factors for diagnostic error, multivariate analysis identified the following situation-specific variables as risk factors diagnostic errors (Table 6): evening echocardiographic interpretation (between 7 PM and 6:59 AM) (AOR, 2.6; 95% CI, 1.3–5.1; P < .001) and weekend studies (Friday through Sunday) (AOR, 1.6; 95% CI, 1.0–2.6; P = .04). The model area under
Benavidez et al 7
Journal of the American Society of Echocardiography Volume - Number -
Table 6 Multivariable logistic regression analysis of risk factors for patient-related and situation-related diagnostic errors Variable
Patient factors Prevalence Frequent/unknown Intermediate Rare/very rare Complexity Low Medium High Weight (kg) <5 5–19 $20 Situational factors Echocardiogram read time 7 PM to 6:59 AM (evening) Other Weekend echocardiography* Yes No
Odds Ratio (95% CI)
P
1.0 1.9 (1.2–3.1) 6.2 (3.8–10)
.008 <.001
1.0 3.3 (2.1–5.1) 3.0 (1.7–5.4)
<.001 <.001
2.7 (1.7–4.3) 2.2 (1.4–3.4) 1.0
<.001 <.001
2.6 (1.3–4.8) 1.0
.006
1.6 (1.0–2.6) 1.0
.04
Area under receiver operating characteristic curve = 0.833. *Friday night after 6 PM to Monday, 6:59 AM.
the receiver operating characteristic curve was 0.833, indicating a good explanatory model. We performed additional analyses to examine potential contributors to diagnostic errors in studies performed at night versus daytime. We found a trend suggesting that night echocardiographic errors were more likely to be preventable than studies performed during the day (44% vs 32%, P = .09). Consistent with this finding, we also found a trend suggesting that cognitive errors may be more prevalent during nighttime studies than daytime studies (51% vs 34%, P = .12). Another noteworthy finding was that poor imaging conditions were more frequent with weekend studies than weekday studies, although that difference did not reach statistical significance (11% vs 3%, P = .32).
DISCUSSION This study builds on our prior research, in which we developed and applied a diagnostic error taxonomy.5 This study differs in the larger number of diagnostic error cases included and our examination of situational variables as potential targets for quality improvement activities. With a substantially larger sample size, we were able, for the first time in patients with pediatric or congenial heart disease, to identify patientrelated and situation-related factors associated with diagnostic errors. Patient-Level Risk Factors Patient-related risk factors are important in predicting the risk for diagnostic error. These risk factors include higher anatomic complexity, rarity of diagnosis, and smaller patient size. These factors have been previously identified as important in predicting diagnostic error. Smaller patient size was previously reported and was suspected to
be attributable to changing physiology in the very young and to patient movement, which may complicate the ability to perform a complete echocardiographic assessment. Additional observations suggest that when clinical circumstances interfere with access to the patient or limit the time available to perform a study, an incomplete study is more likely. Further study may be indicated to elucidate this phenomenon. Given the persistence of these findings, however, congenital cardiac centers may choose to target quality improvement activities among those patients who have one or more risk factors to reduce the diagnostic error rate. For example, a center may implement a policy that echocardiograms obtained in patients weighting < 5 kg with moderate or higher complexity congenital heart disease will undergo second reads. The risk factors for diagnostic error identified in this study may also assist noninvasive laboratories in accounting for study complexity when assessing the performance of individual echocardiographers or the entire laboratory. In other words, such data can be used to develop a method for risk adjustment that can be applied to individual readers and/or to the entire laboratory. Specifically, those cases with multiple risk factors would be weighed as more diagnostically challenging compared with those without any risk factors. Situation-Related Risk Factors We achieved the second aim of our study and identified two independent risk factors for diagnostic error related to the setting in which a study was performed and interpreted. We found that evening interpretation of a congenital echocardiogram and weekend echocardiography were independent risk factors for diagnostic error, while controlling for patient-level risk factors. Further subanalysis suggested that these errors might be more likely to be caused by cognitive errors and to be preventable. Nights and weekends potentially represent a high-risk period for patients. Although data regarding night and weekend clinical activities and their impact on outcomes are limited, some studies have demonstrated a weekend effect. A study by Buckley and Bulger9 of medical error reports during general medical admissions found greater than twofold odds of medical errors occurring on weekends compared with weekdays among hospitalized patients. Additionally, a study of medication errors in a pediatric emergency department reported that the majority of medication dosage errors occurred during the evening and night shifts.10 Our findings are consistent with these reports and now expand these findings to pediatric and congenital echocardiography for the first time. The implication of these findings is that they lend themselves to quality improvement actions. One could envision a quality improvement action that involves second reviews of weekend and evening echocardiograms to mitigate diagnostic errors or their impact. Diagnostic Errors and Targeted Quality Improvement Actions In our previous study, echocardiographic studies performed in the recovery suite (postanesthesia care unit setting) were associated with diagnostic errors.5 This finding was not significant in this study. On the basis of our previous findings that echocardiographic studies performed in the recovery area had higher rates of diagnostic errors, we implemented a strategy to reduce these errors, called attending (cardiologist) supervised echocardiograms. The following procedure was implemented: echocardiographic studies performed in the recovery area were identified as ‘‘high-risk’’ studies and were assigned to a supervising cardiologist. Before each such study, the supervising cardiologist discussed the imaging plan with the sonographer performing the
8 Benavidez et al
study. During and after the study and before the patient left the imaging area, the supervising cardiologist reviewed the images with the sonographer and, at his or her discretion, acquired additional images. The supervising cardiologist then interpreted the study and completed the report. The goals of this quality improvement strategy were to ensure continuity of care by the supervising attending cardiologist and to minimize the likelihood of missing vital diagnostic information.11 It is possible that this targeted data-driven quality improvement action resulted in the elimination of the recovery area as a risk factor for diagnostic error in this cohort. Quality improvement actions directed at evening and weekend echocardiography may deliver similar results. Limitations Selection bias and incomplete case ascertainment are notable limitations of this study. The majority of cases were identified by second imaging studies or procedures. Patients who did not undergo these imaging studies or procedures may have had unidentified diagnostic errors. Therefore, the total number of diagnostic errors might be underestimated, although it is less likely that major errors affecting clinical outcomes were not detected.
CONCLUSIONS The identification of patient-related risk factors and the presence of situational risk factors that are universal, such as weekend and evening echocardiography, may assist programs in reducing diagnostic errors. It may also provide a foundation to develop a risk adjustment model for diagnostic errors. Although diagnostic accuracy may represent the pinnacle of quality of diagnostic imaging, it is important to recognize other situational and process components that may influence diagnostic accuracy. A multifaceted approach acting on these risk factors and contributors to diagnostic error would likely yield the best result
Journal of the American Society of Echocardiography - 2014
in improving imaging outcomes. Echocardiography is presently an extremely useful, safe, and readily available modality to diagnose congenital heart disease, and our present study represents an attempt to improve the efficacy of this important imaging modality.
REFERENCES 1. CRICO/RMF. Malpractice high risk areas. Available at: http://www.rmf. harvard.edu/clinician-resoures/article/2011/high-risk-areas. Accessed October 16, 2012. 2. Berner ES, Miller RA, Graber ML. Missed and delayed diagnoses in the ambulatory setting. Ann Intern Med 2007;146:470. 3. Dorfman AL, Levine JC, Colan SD, Geva T. Accuracy of echocardiography in low birth weight infants with congenital heart disease. Pediatrics 2005;115:102-7. 4. Gutgesell HP, Huhta JC, Latson LA, Huffines D, McNamara DG. Accuracy of two-dimensional echocardiography in the diagnosis of congenital heart disease. Am J Cardiol 1985;55:514-8. 5. Benavidez OJ, Gauvreau K, Jenkins KJ, Geva T. Diagnostic errors in pediatric echocardiography: development of taxonomy and identification of risk factors. Circulation 2008;117:2995-3001. 6. Sholler G. Echocardiography in congenital heart disease: diagnosis, misdiagnosis, and ownership. J Paediatr Child Health 2001;37:321-2. 7. Geva T, Van Praagh S, Sanders SP, Mayer JE Jr., Van Praagh R. Straddling mitral valve with hypoplastic right ventricle, crisscross atrioventricular relations, double outlet right ventricle and dextrocardia: morphologic, diagnostic and surgical considerations. J Am Coll Cardiol 1991;17:1603-12. 8. Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med 2005;165:1493-9. 9. Buckley D, Bulger D. Trends and weekly and seasonal cycles in the rate of errors in the clinical management of hospitalized patients. Chronobiol Int 2012;29:947-54. 10. Selbst SM, Fein JA, Osterhoudt K, Ho W. Medication errors in a pediatric emergency department. Pediatr Emerg Care 1999;15:1-4. 11. Lai WW. Pediatric and congenital heart disease council communication improving the chances of being right. J Am Soc Echocardiogr 2009;22:A39.
Background: Diagnostic errors are unwanted clinical events that place patients at risk for injury. The authors have previously reported that a majority of congenital echocardiography errors have clinical impacts and, on the basis of a small cohort, identified factors associated with diagnostic error. The objectives of this study were (1) to evaluate patient risk factors for diagnostic errors in a large contemporary cohort and (2) to identify risk factors for situation-related diagnostic errors. Methods: Diagnostic errors were identified at a large academic pediatric cardiac center from 2004 to 2011. Clinical and situational variables were collected from diagnostic error cases and controls. Results: Among the 254 diagnostic error cases, 66% affected clinical management or patients experienced adverse events; 77% of errors were preventable or possibly preventable. Coronary arteries, pulmonary veins, and the aortic arch were most commonly involved with diagnostic errors. Multivariate analysis identified the following patient-related risk factors: rare or very rare diagnoses (adjusted odds ratio [AOR], 6.3; P < .001), high anatomic complexity (AOR, 3.4; P < .001), and weight < 5 kg (AOR, 2.7; P < .001). Risk factors related to the setting of the echocardiographic study included evening or night (7 PM to 6:59 AM) study interpretation (AOR, 2.6; P = .005) and weekend studies (Friday through Sunday) (AOR, 1.6; P = .04). The model area under the receiver operating characteristic curve was 0.833. Conclusions: In addition to patient risk factors, the setting of an echocardiographic study and interpretation contribute to risk for a diagnostic error. Studies interpreted overnight or performed during a weekend should be considered for a quality improvement activity to reduce diagnostic errors or their impact. (J Am Soc Echocardiogr 2014;-:---.) Keywords: Congenital echocardiography, Diagnostic error, Quality, Outcomes
Diagnostic errors are unwanted clinical events that place patients at risk for injury and are a leading cause of malpractice claims.1,2 Echocardiography is the first line of diagnostic investigation among patients suspected of having congenital heart disease,3,4 and diagnostic errors may place patients at risk for suboptimal outcomes.5,6 We have previously reported that a majority of congenital echocardiographic diagnostic errors have clinical impacts.5 That analysis, however, was based on a relatively small cohort. Using a large contemporary cohort, the objectives of the present study were (1) to determine patient-related risk factors for diagnostic errors and (2) to identify independent risk factors related to the situation in which echocardiographic studies are performed and interpreted. From the Department of Cardiology, Boston Children’s Hospital, Harvard Medical School, Boston, Massachusetts (O.J.B.); Division of Pediatric/Congenital Cardiology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts (O.J.B., K.G., T.G.); and Department of Pediatrics, Harvard Medical School, Boston, Massachusetts (O.J.B., K.G., T.G.). Dr Benavidez is supported a grant from the Robert Wood Johnson Foundation’s Harold Amos Medical Faculty Program. Reprint requests: Oscar J. Benavidez, MD, MPP, Massachusetts General Hospital, Division of Pediatric/Congenital Cardiology, 175 Cambridge Street, Boston, MA 02114 (E-mail: [email protected]). 0894-7317/$36.00 Copyright 2014 by the American Society of Echocardiography. http://dx.doi.org/10.1016/j.echo.2014.03.001
Situational risk factors may be responsive to quality improvement strategies to reduce diagnostic errors or to mitigate their impact.
METHODS The Scientific Review Committee of the Department of Cardiology and the Institutional Review Board at Boston Children’s Hospital approved this study. The authors had full access to and take full responsibility for the integrity of the data. All authors have read and agree to the report as written. Data Source The Echocardiography Laboratory at Boston Children’s Hospital performs approximately 22,000 echocardiographic studies annually. These studies are performed in locations such as echocardiography suites, satellite cardiology clinics, operating rooms, intensive care units, recovery rooms, catheterization laboratories, cardiology wards, emergency departments, and general pediatric wards. Trained pediatric and congenital sonographers, cardiology fellows, and echocardiography staff cardiologists perform the examinations. The cardiac sonographers are provided with a requisition to perform a study, and they review the requisition and study questions before performing the study. Before study completion, the sonographer will discuss the pertinent findings with the noninvasive cardiologist. 1
2 Benavidez et al
The echocardiography staff cardiologists interpret all studies AOR = Adjusted odds ratio and issue reports, which are stored in the hospital’s electronic CI = Confidence interval medical records. The final diagnosis reported by the staff cardiologist after the interpretation of the echocardiogram was the outcome of interest in this study; preliminary results reported by cardiac sonographers or trainees were not examined. Postnatal pediatric and congenital echocardiograms were the subject of this study. Abbreviations
Diagnostic Error Case Ascertainment Between December 2004 and March 2011, as part of a continuous quality improvement initiative in the Noninvasive Cardiology Division, data related to diagnostic error cases were collected prospectively through voluntary reporting and active quality assurance mechanisms. We have previously reported these methods.5 A diagnostic error was defined as a diagnosis that was unintentionally delayed, wrong, or missed as judged from eventual appreciation of the existing data or more definitive information.5 The sources of diagnostic error case discovery included information obtained from other tests (e.g., cardiac catheterization, magnetic resonance imaging), operative observations, subsequent echocardiographic examinations, and autopsy. Data Collected The following patient demographic and case data were collected: age, referral diagnosis, weight, race, prematurity, anatomy involved in diagnostic error, and study indication or question. Other data obtained included study location, use of sedation or anesthesia, comments on image quality, day of week, and time of day the study was performed and interpreted. Data related to the number of interim studies or procedures performed before the diagnostic error discovery was also noted, including the method of discovery. Diagnostic frequency, defined as the frequency a diagnosis is encountered in the echocardiography laboratory, was categorized as follows: (1) frequent (diagnosis is observed more than once per week; e.g., patent ductus arteriosus), (2) intermediate frequency (diagnosis is observed more than once a month but less than once weekly; e.g., coarctation of the aorta), (3) rare (diagnosis is observed more than once per year but less than once monthly; e.g., inferiortype sinus venosus defect), and (4) very rare (diagnosis is observed less than once yearly; e.g., aortic–left ventricular tunnel). Anatomic complexity data were divided into 3 categories: (1) low (no significant heart disease or a single, simple structural cardiovascular anomaly; e.g., atrial septal defect or single membranous ventricular septal defect), (2) moderate (abnormalities involving more than one cardiovascular structure or diagnoses with moderately common characteristics; e.g., complete common atrioventricular canal, hypoplastic left heart syndrome, transposition of the great arteries), and (3) high (uncommon variants of moderately complex anatomic diagnoses or rare, complex anomalies; e.g., dextrocardia, superior-inferior ventricles with crisscross atrioventricular relations, hypoplastic right ventricle, and straddling mitral valve).7 Diagnostic Error Case Review Process Staff echocardiographers reviewed diagnostic error cases and other diagnostic data in addition to the images of the studies in question, as previously described5 (Figure 1). Briefly, diagnostic error cases
Journal of the American Society of Echocardiography - 2014
were reviewed by staff pediatric echocardiographers, including review of medical records and other diagnostic images in addition to the images of the studies in question. Interviews with sonographers, trainees, cardiologists, and other involved health care providers were conducted to understand the conditions and the context in which the examinations were performed and interpreted, to determine the clinical impact, and to identify primary causes of the diagnostic errors. The relevant clinical and image data were presented at a monthly noninvasive morbidity and mortality conference. This process involves a systematic review of the echocardiographic process to identify contributors to the diagnostic error in question. Case discussion included categorization of diagnostic error type, severity, preventability, and contributors or root causes. Participants in the conference include attending physicians, trainees, and cardiac sonographers from the noninvasive division, as well representatives from other divisions within the Department of Cardiology and other disciplines (e.g., cardiac anesthesia and cardiovascular surgery). A consensus based on a review of the case and the ensuing discussion was used to finalize categorization of the diagnostic error type, severity, preventability, and root cause.
Definitions and Classification of Diagnostic Errors A diagnostic error was defined as a diagnosis that was unintentionally delayed, wrong, or missed, as judged from eventual appreciation of the existing data or of more definitive information.8 Diagnostic Error Categorization. False-negative: An error that omits a finding or states that a finding is normal (or absent) when an abnormality is present, or the reader failed to include a significant diagnostic possibility (e.g., patent ductus arteriosus is ruled out or omitted when it is evident on the study images). False-positive: An error that reports an abnormality but no abnormality is present, or the reader overemphasized the significance of a finding (e.g., atrial septal defect is diagnosed when the atrial septum is intact). Discrepant diagnosis: The actual diagnosis is different from the one made (e.g., diagnosis of double-inlet left ventricle is made when the actual diagnosis is tricuspid atresia). Severity Categorization. Diagnostic errors were categorized by severity into the following categories on the basis of clinical impact: minor, moderate, severe, and catastrophic (Table 1). Minor: A diagnostic error or discrepancy that does not change patient management or affect clinical course, with little or no potential for adverse event (e.g., missed left superior vena cava to an intact coronary sinus in a patient with an otherwise structurally normal heart). Moderate: A diagnostic error or discrepancy with an impact on management, whereby the patient may be placed at risk and/or experience a transient adverse event (e.g., missed primum atrial septal defect in an infant). Major: Diagnostic error discrepancy with impact on management that results in an adverse event, including the performance of unnecessary invasive procedure, or a long-lasting or permanent adverse event (e.g., false-positive diagnosis of an atrial septal defect leading to an unnecessary surgery to close a defect that was not present). Catastrophic: A diagnostic error or discrepancy that contributed to patient death (e.g., missed anomalous coronary artery from
Benavidez et al 3
Journal of the American Society of Echocardiography Volume - Number -
Figure 1 Diagnostic error review process. The echocardiographic process is reviewed in detail from study requisition through study reporting. Each step of this process is systematically reviewed for contribution to diagnostic error. Example considerations based on common causes of diagnostic error contributors are listed below each step.
Table 1 Categorization of diagnostic error severity Category
Description
Minor
A diagnostic error or discrepancy that does not change patient management or affect clinic course, with little or no potential for adverse event (e.g., missed left superior vena cava to an intact coronary sinus in a patient with an otherwise structurally normal heart).
Moderate
A diagnostic error or discrepancy with an impact on management, whereby the patient may be placed at risk and/or experience a transient adverse event (e.g., missed primum atrial septal defect in an infant). Diagnostic error discrepancy with impact on management that results in an adverse event, including the performance of unnecessary invasive procedure, or a long-lasting or permanent adverse event (e.g., false-positive diagnosis of an atrial septal defect leading to an unnecessary surgery to close a defect that was not present).
Major
Catastrophic
A diagnostic error or discrepancy that contributed to patient death (e.g., missed anomalous coronary artery from pulmonary artery that resulted in intraoperative myocardial ischemia leading to patient death).
pulmonary artery that resulted in intraoperative myocardial ischemia leading to patient death).
conditions (e.g., an echocardiogram clearly demonstrates a patent ductus arteriosus, but the study is interpreted as showing none).
Categorization of Preventability. Diagnostic errors were categorized by preventability as not preventable, possibly preventable, or preventable (Table 2).
Diagnostic Error Cause Categorization. The underlying causes or contributors to diagnostic errors belonged to one of the following categories: administrative error, procedural error, communication error, cognitive error, technical factor, and patient-related or diseaserelated factors5 (Table 3).
Not preventable: A diagnostic error is not preventable if the images, imaging modality, or imaging conditions (e.g., attempted imaging during active cardiopulmonary resuscitation) do not permit diagnosis (e.g., failure to image a ligamentum arteriosum contributing to a vascular ring). Possibly preventable: A diagnostic error is possibly preventable if accurate diagnosis may be expected given the imaging modality and/or imaging conditions but may have required a different technique(s) or condition(s) (e.g., aortic arch anomaly that was missed because the aortic arch branching was incompletely examined). Preventable: A diagnosis is preventable if accurate diagnosis is expected given the available images, imaging modality, and/or imaging
Administrative or data entry errors: Errors that typically precede the actual examination (e.g., entering an examination report under an incorrect patient name). Procedural or conditional factors: These are factors that relate to the performance of the study or the conditions under which the examination is performed (e.g., failure to adequately interrogate the entire ventricular septum by color Doppler in a cooperative patient, resulting in a missed apical ventricular septal defect). Communication or information errors: Errors in information transfer to those performing and interpreting the examination or from
4 Benavidez et al
Journal of the American Society of Echocardiography - 2014
Table 2 Categorization of diagnostic error preventability Category
Description
Not preventable
A diagnostic error is not preventable if the images, imaging modality, or imaging conditions (e.g., attempted imaging during active cardiopulmonary resuscitation) do not permit diagnosis (e.g., failure to image a ligamentum arteriosum contributing to a vascular ring).
Possibly preventable
A diagnostic error is possibly preventable if accurate diagnosis may be expected given the imaging modality and/or imaging conditions but may have required a different technique(s) or condition(s) (e.g., aortic arch anomaly that was missed because the aortic arch branching was incompletely examined). A diagnosis is preventable if accurate diagnosis is expected or readily apparent given the study images, imaging modality, and/or imaging conditions (e.g., an echocardiogram clearly demonstrates a patent ductus arteriosus, but the study is interpreted as showing none).
Preventable
Table 3 Contributors to diagnostic error Administrative or data entry errors Incorrect name assigned to imaging data Scheduling error Incorrect data entry Procedural or conditional factors Failure to confirm patient identity or diagnosis Incomplete examination of anatomy or physiology Poor imaging environment Failure to improve imaging conditions when possible Communication or information errors Lacking or misleading patient history No access to prior studies Failure to report critical findings in a timely fashion to referring physician Incorrect requisition (unintended or no clinical question asked) Cognitive errors Insufficient knowledge base Insufficient technical skills Faulty data synthesis Lack of consideration a patient’s situation/condition that is relevant to diagnosis Misidentification/interpretation of a finding on echocardiography Premature closure of case Distraction by other diagnoses, findings, or focused question Underappreciation/consideration of a finding Overappreciation of a finding Confirmation bias Incorrect or improper calculation Technical factors Artifact Modality limitation Poor acoustic windows Equipment malfunction Patient-related or disease-related factors Rare or complex anatomy Misleading anatomy or physiology
those interpreting the study to the referring clinician. This includes providing inaccurate or misleading clinical history or incorrect study questions. For example, a treating cardiologist was concerned about an intracardiac thrombus in a patient after Fontan palliation, but this information was not communicated to the
echocardiography laboratory. Instead, the echocardiography laboratory was incorrectly asked to rule out pericardial effusion. As a result, a focused echocardiogram was performed to rule out a pericardial effusion, missing an intracardiac thrombus. In this case, failure to communicate the intentioned study indication (concern for intracardiac clot) by providing an inaccurate study question (rule out pericardial effusion) contributed to the diagnostic error. Cognitive errors: These are errors that occur during the analysis of the imaging and clinical data. For example, an echocardiographic study is requested for estimation of the pressure gradient across a right ventricle–to–pulmonary artery conduit in a patient who had an endovascular stent placed in the conduit. The conduit gradient is reported correctly, but the readily apparent free-floating fractured stent in the right ventricle is not noted. Technical factors: These are factors involving equipment malfunction or inherent limitations of echocardiography. Examples include an imaging artifact creating the appearance of a vascular structure that is then interpreted as an anomaly. Postoperative changes or obesity are examples of technical factors that can result in ‘‘poor acoustic windows,’’ which produce degraded, or nondiagnostic image quality. Patient-related or disease-related factors: Factors related to the patient’s underlying diagnosis or physiology. For example, a tortuous left atrial appendage is mistaken for an aneurysm of the left coronary artery in a patient with Kawasaki disease.
Statistical Analysis The goal of the analysis was to identify patient-related and situationrelated risk factors associated with diagnostic errors. Control cases were randomly selected (with a ratio of approximately 2.5 controls per diagnostic error case) by a statistical program (Stata/SE version 9.2 for Windows; StataCorp LP, College Station, TX) from all echocardiographic studies performed during the same calendar week the diagnostic error occurred. The imaging data, reports, and medical records of control cases were examined for the presence of unrecognized diagnostic errors and were excluded from the control group if an error was identified. The findings from the categorization of diagnostic error type, preventability, severity, anatomic factors, and potential contributors were tabulated and reported as percentages of all cases of diagnostic error. Univariate and multivariate analyses compared diagnostic error cases with controls. Demographic and situational variables were compared for diagnostic error cases versus controls using Fisher’s exact test. Variables with P values < .20 in univariate analysis were considered for inclusion in a multivariate logistic regression model. A P value of
Benavidez et al 5
Journal of the American Society of Echocardiography Volume - Number -
Table 4 Factors contributing to diagnostic errors (n = 254)
Figure 2 (A) Diagnostic error severity: only 33% of the diagnostic errors (n = 85) were of minor severity and had no impact on patient management. (B) Diagnostic error preventability: only 22% (n = 57) of the errors were not preventable. .05 was required for inclusion of a variable in the final model. Adjusted odds ratios (AORs) for having a diagnostic error with 95% confidence intervals (CIs) were calculated for variables retained in the final model.
Contributor
Cases
%
Cognitive errors Misidentification/interpretation of finding Underinterpretation of a finding Distraction by other diagnoses Overinterpretation of a finding Incorrect or improper calculation Technical factors Poor acoustic windows Modality limitation Imaging artifact Patient-related or disease-related factors Misleading anatomy or physiology Rare or complex anatomy or physiology Procedural or conditional factors Incomplete examination Procedure situation or practice Poor imaging conditions Communication or information errors Lacking or misleading patient history Incorrect requisition Administrative/other errors Incorrect height and weight data entry Incorrect patient imaged/reported
94 30 53 6 4 1 71 37 26 8 30 15 15 53 35 5 13 3 2 1 3 2 1
37 12 21 2 2 <1 28 15 10 3 10 6 6 21 14 2 5 1 <1 <1 1 <1 <1
(n = 112 [45%]) (Figure 2B). Of note, the majority of preventable errors (73%) were of moderate severity or greater.
Contributors to Diagnostic Errors Table 4 lists the factors contributing to diagnostic errors. The 3 most common factors contributing to diagnostic errors were cognitive (38%), technical (28%), and procedure related (21%).
RESULTS Between December 2004 and March 2011, 254 postnatal echocardiography diagnostic error cases were identified. During that period, approximately 147,000 postnatal echocardiograms were obtained and interpreted in our laboratory, yielding an overall error rate of 0.17%. Diagnostic Error Categorization Of the 254 diagnostic error cases identified, 70% were falsenegatives, 15% were false-positives, and 15% discrepant diagnoses. Diagnostic Error Severity Only 33% of the diagnostic errors (n = 85) were of minor severity and had no impact on patient management; 67% affected patient management, with 59% (n = 151) of moderate severity, 6% (n = 16) of major severity, and 1% (n = 2) catastrophic. Both catastrophic cases involved coronary arteries (Figure 2A). Diagnostic Error Preventability Only 22% of the diagnostic errors (n = 57) were not preventable, while 78% were preventable (n = 85 [33%]) or possibly preventable
Anatomy Involved with Diagnostic Errors Common anatomic segments involved with diagnostic errors were coronary arteries (n = 52 [20%]), pulmonary veins (n = 31 [12%]), and the aortic arch (n = 26 [10%]).
Diagnostic Error Discovery The correct diagnoses were discovered by echocardiography in 28% of cases (either from new studies or reexamination of the initial studies), intraoperative inspection in 27%, cardiac catheterization in 18%, cardiac magnetic resonance imaging in 17%, transesophageal echocardiography in 7%, and by another modality in 4% of cases. Of the 254 cases, 38% had interim imaging studies before identification of the error: 17% had one interim study, 11% had two or three interim studies, and 10% had more than three studies. Only 3% of the cases had interim cardiac magnetic resonance imaging before diagnostic error discovery. Regarding the number of invasive procedures before discovery of diagnostic error, 9% had at least one interim cardiac catheterization, and 8% had an interim cardiac surgery. Stated differently, these invasive procedures were performed without discovery of the diagnostic errors.
6 Benavidez et al
Journal of the American Society of Echocardiography - 2014
Table 5 Univariate analysis comparison of characteristics for patients with diagnostic errors versus controls with no diagnostic errors Variable
Weight (kg) <5 5–19 $20 Premature Yes No Study location CICU NICU Echocardiography lab Satellite/outpatient Other Weekend* Yes No Echocardiography performance time 7 AM to 12:59 PM 1 PM to 6:59 PM 7 PM to 6:59 PM Echocardiogram read time 7 AM to 12:59 PM 1 PM to 6:59 PM 7 PM to 6:59 AM Not reported Prevalence Frequent Intermediate Rare Very rare Complexity Low Medium High Limited study† Yes No
Total (n = 798)
Cases (n = 254)
Controls (n = 544)
200 (25%) 232 (29%) 359 (45%)
103 (41%) 89 (35%) 62 (24%)
97 (18%) 143 (26%) 297 (55%)
47 (6%) 751 (94%)
18 (7%) 236 (93%)
29 (5%) 515 (95%)
100 (13%) 40 (5%) 364 (46%) 72 (9%) 222 (28%)
59 (23%) 22 (9%) 101 (40%) 13 (5%) 59 (23%)
41 (8%) 18 (3%) 263 (48%) 59 (11%) 163 (30%)
140 (18%) 658 (82%)
65 (26%) 189 (74%)
75 (14%) 469 (86%)
436 (55%) 324 (41%) 38 (5%)
127 (50%) 109 (43%) 18 (7%)
309 (57%) 215 (39%) 20 (4%)
262 (33%) 474 (59%) 61 (8%) 1 (<1%)
70 (28%) 145 (57%) 39 (15%) 0 (0%)
192 (35%) 329 (60%) 22 (4%) 1 (<1%)
410 (51%) 177 (22%) 169 (21%) 25 (3%)
55 (22%) 68 (27%) 108 (43%) 23 (9%)
355 (65%) 109 (20%) 61 (11%) 2 (<1%)
387 (50%) 280 (36%) 113 (14%)
50 (20%) 131 (51%) 73 (29%)
337 (62%) 149 (28%) 40 (7%)
115 (14%) 683 (86%)
26 (10%) 228 (90%)
89 (16%) 455 (84%)
P
<.001
.34
<.001
<.001
.05
<.001
<.001
<.001
.02
CICU, Cardiac intensive care unit; NICU, neonatal intensive care unit. *Friday night after 6 PM to Monday, 6:59 AM. † An echocardiographic study that yields few images or on which the image quality is such that the study questions cannot be answered; the echocardiographic report is written as a summary sentence or brief paragraph with few details.
Comparison of Diagnostic Error Cases with Controls Results of the univariate analysis are summarized in Table 5. Compared with controls, diagnostic error cases were more likely to involve a lower body weight, to be performed in a critical care unit, to be performed and interpreted at night or on a weekend, to involve uncommon diagnoses, and to involve greater anatomic complexity. Patient-Related Risk Factors for Diagnostic Errors Multivariate analysis identified the following patient-related variables as risk factors diagnostic errors (Table 6): rare or very rare diagnoses (AOR, 6.2; 95% CI, 3.8–10.1, P < .001), moderate level of anatomic complexity (AOR, 3.3; 95% CI, 2.1–5.1; P < .001), high level of
anatomic complexity (AOR, 3.0; 95% CI, 1.7–5.4, P < .001), patient weight < 5 kg (AOR, 3.5; 95% CI, 1.1–10.6; P = .031), and patient weight 5 to 19 kg (AOR, 2.2; 95% CI, 1.4–3.4, P < .001).
Situation-Related Risk Factors for Diagnostic Errors While controlling for patient-related risk factors for diagnostic error, multivariate analysis identified the following situation-specific variables as risk factors diagnostic errors (Table 6): evening echocardiographic interpretation (between 7 PM and 6:59 AM) (AOR, 2.6; 95% CI, 1.3–5.1; P < .001) and weekend studies (Friday through Sunday) (AOR, 1.6; 95% CI, 1.0–2.6; P = .04). The model area under
Benavidez et al 7
Journal of the American Society of Echocardiography Volume - Number -
Table 6 Multivariable logistic regression analysis of risk factors for patient-related and situation-related diagnostic errors Variable
Patient factors Prevalence Frequent/unknown Intermediate Rare/very rare Complexity Low Medium High Weight (kg) <5 5–19 $20 Situational factors Echocardiogram read time 7 PM to 6:59 AM (evening) Other Weekend echocardiography* Yes No
Odds Ratio (95% CI)
P
1.0 1.9 (1.2–3.1) 6.2 (3.8–10)
.008 <.001
1.0 3.3 (2.1–5.1) 3.0 (1.7–5.4)
<.001 <.001
2.7 (1.7–4.3) 2.2 (1.4–3.4) 1.0
<.001 <.001
2.6 (1.3–4.8) 1.0
.006
1.6 (1.0–2.6) 1.0
.04
Area under receiver operating characteristic curve = 0.833. *Friday night after 6 PM to Monday, 6:59 AM.
the receiver operating characteristic curve was 0.833, indicating a good explanatory model. We performed additional analyses to examine potential contributors to diagnostic errors in studies performed at night versus daytime. We found a trend suggesting that night echocardiographic errors were more likely to be preventable than studies performed during the day (44% vs 32%, P = .09). Consistent with this finding, we also found a trend suggesting that cognitive errors may be more prevalent during nighttime studies than daytime studies (51% vs 34%, P = .12). Another noteworthy finding was that poor imaging conditions were more frequent with weekend studies than weekday studies, although that difference did not reach statistical significance (11% vs 3%, P = .32).
DISCUSSION This study builds on our prior research, in which we developed and applied a diagnostic error taxonomy.5 This study differs in the larger number of diagnostic error cases included and our examination of situational variables as potential targets for quality improvement activities. With a substantially larger sample size, we were able, for the first time in patients with pediatric or congenial heart disease, to identify patientrelated and situation-related factors associated with diagnostic errors. Patient-Level Risk Factors Patient-related risk factors are important in predicting the risk for diagnostic error. These risk factors include higher anatomic complexity, rarity of diagnosis, and smaller patient size. These factors have been previously identified as important in predicting diagnostic error. Smaller patient size was previously reported and was suspected to
be attributable to changing physiology in the very young and to patient movement, which may complicate the ability to perform a complete echocardiographic assessment. Additional observations suggest that when clinical circumstances interfere with access to the patient or limit the time available to perform a study, an incomplete study is more likely. Further study may be indicated to elucidate this phenomenon. Given the persistence of these findings, however, congenital cardiac centers may choose to target quality improvement activities among those patients who have one or more risk factors to reduce the diagnostic error rate. For example, a center may implement a policy that echocardiograms obtained in patients weighting < 5 kg with moderate or higher complexity congenital heart disease will undergo second reads. The risk factors for diagnostic error identified in this study may also assist noninvasive laboratories in accounting for study complexity when assessing the performance of individual echocardiographers or the entire laboratory. In other words, such data can be used to develop a method for risk adjustment that can be applied to individual readers and/or to the entire laboratory. Specifically, those cases with multiple risk factors would be weighed as more diagnostically challenging compared with those without any risk factors. Situation-Related Risk Factors We achieved the second aim of our study and identified two independent risk factors for diagnostic error related to the setting in which a study was performed and interpreted. We found that evening interpretation of a congenital echocardiogram and weekend echocardiography were independent risk factors for diagnostic error, while controlling for patient-level risk factors. Further subanalysis suggested that these errors might be more likely to be caused by cognitive errors and to be preventable. Nights and weekends potentially represent a high-risk period for patients. Although data regarding night and weekend clinical activities and their impact on outcomes are limited, some studies have demonstrated a weekend effect. A study by Buckley and Bulger9 of medical error reports during general medical admissions found greater than twofold odds of medical errors occurring on weekends compared with weekdays among hospitalized patients. Additionally, a study of medication errors in a pediatric emergency department reported that the majority of medication dosage errors occurred during the evening and night shifts.10 Our findings are consistent with these reports and now expand these findings to pediatric and congenital echocardiography for the first time. The implication of these findings is that they lend themselves to quality improvement actions. One could envision a quality improvement action that involves second reviews of weekend and evening echocardiograms to mitigate diagnostic errors or their impact. Diagnostic Errors and Targeted Quality Improvement Actions In our previous study, echocardiographic studies performed in the recovery suite (postanesthesia care unit setting) were associated with diagnostic errors.5 This finding was not significant in this study. On the basis of our previous findings that echocardiographic studies performed in the recovery area had higher rates of diagnostic errors, we implemented a strategy to reduce these errors, called attending (cardiologist) supervised echocardiograms. The following procedure was implemented: echocardiographic studies performed in the recovery area were identified as ‘‘high-risk’’ studies and were assigned to a supervising cardiologist. Before each such study, the supervising cardiologist discussed the imaging plan with the sonographer performing the
8 Benavidez et al
study. During and after the study and before the patient left the imaging area, the supervising cardiologist reviewed the images with the sonographer and, at his or her discretion, acquired additional images. The supervising cardiologist then interpreted the study and completed the report. The goals of this quality improvement strategy were to ensure continuity of care by the supervising attending cardiologist and to minimize the likelihood of missing vital diagnostic information.11 It is possible that this targeted data-driven quality improvement action resulted in the elimination of the recovery area as a risk factor for diagnostic error in this cohort. Quality improvement actions directed at evening and weekend echocardiography may deliver similar results. Limitations Selection bias and incomplete case ascertainment are notable limitations of this study. The majority of cases were identified by second imaging studies or procedures. Patients who did not undergo these imaging studies or procedures may have had unidentified diagnostic errors. Therefore, the total number of diagnostic errors might be underestimated, although it is less likely that major errors affecting clinical outcomes were not detected.
CONCLUSIONS The identification of patient-related risk factors and the presence of situational risk factors that are universal, such as weekend and evening echocardiography, may assist programs in reducing diagnostic errors. It may also provide a foundation to develop a risk adjustment model for diagnostic errors. Although diagnostic accuracy may represent the pinnacle of quality of diagnostic imaging, it is important to recognize other situational and process components that may influence diagnostic accuracy. A multifaceted approach acting on these risk factors and contributors to diagnostic error would likely yield the best result
Journal of the American Society of Echocardiography - 2014
in improving imaging outcomes. Echocardiography is presently an extremely useful, safe, and readily available modality to diagnose congenital heart disease, and our present study represents an attempt to improve the efficacy of this important imaging modality.
REFERENCES 1. CRICO/RMF. Malpractice high risk areas. Available at: http://www.rmf. harvard.edu/clinician-resoures/article/2011/high-risk-areas. Accessed October 16, 2012. 2. Berner ES, Miller RA, Graber ML. Missed and delayed diagnoses in the ambulatory setting. Ann Intern Med 2007;146:470. 3. Dorfman AL, Levine JC, Colan SD, Geva T. Accuracy of echocardiography in low birth weight infants with congenital heart disease. Pediatrics 2005;115:102-7. 4. Gutgesell HP, Huhta JC, Latson LA, Huffines D, McNamara DG. Accuracy of two-dimensional echocardiography in the diagnosis of congenital heart disease. Am J Cardiol 1985;55:514-8. 5. Benavidez OJ, Gauvreau K, Jenkins KJ, Geva T. Diagnostic errors in pediatric echocardiography: development of taxonomy and identification of risk factors. Circulation 2008;117:2995-3001. 6. Sholler G. Echocardiography in congenital heart disease: diagnosis, misdiagnosis, and ownership. J Paediatr Child Health 2001;37:321-2. 7. Geva T, Van Praagh S, Sanders SP, Mayer JE Jr., Van Praagh R. Straddling mitral valve with hypoplastic right ventricle, crisscross atrioventricular relations, double outlet right ventricle and dextrocardia: morphologic, diagnostic and surgical considerations. J Am Coll Cardiol 1991;17:1603-12. 8. Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med 2005;165:1493-9. 9. Buckley D, Bulger D. Trends and weekly and seasonal cycles in the rate of errors in the clinical management of hospitalized patients. Chronobiol Int 2012;29:947-54. 10. Selbst SM, Fein JA, Osterhoudt K, Ho W. Medication errors in a pediatric emergency department. Pediatr Emerg Care 1999;15:1-4. 11. Lai WW. Pediatric and congenital heart disease council communication improving the chances of being right. J Am Soc Echocardiogr 2009;22:A39.