Common errors in computer electrocardiogram interpretation

International Journal of Cardiology 106 (2006) 232 – 237 www.elsevier.com/locate/ijcard

Common errors in computer electrocardiogram interpretation Maya E. GuglinT, Deepak Thatai Wayne State University, John D. Dingell VA Medical Center, 4646 John R. Street, Detroit, MI 48034, United States Received 13 September 2004; received in revised form 5 February 2005; accepted 6 February 2005 Available online 9 April 2005

Abstract Objective: The aim of the study was to determine the frequency and nature of errors in computer electrocardiogram (ECG) reading. Methods: The ECGs were collected in the tertiary care VA Hospital from both inpatients and outpatients. They were read by the electrocardiograph built-in computer software, and then reread by two cardiologists. Statistical analysis was performed using sensitivity, specificity, positive and negative predicted value to analyze the data. An error index was formulated as indicator of diagnostic accuracy. Results: Out of 2072 ECGs, 776 (37.5%) were normal, and 1296 (62.5%) were abnormal. In 9.9% of all ECGs and in 15.9% of abnormal ECGs there were significant disagreements between the computer and cardiologists. The errors in diagnosis of arrhythmia, conduction disorders and electronic pacemakers accounted for 178 cases, or 86.4% of all errors. The rest was represented by misdetection of chamber enlargement (7 cases, 3.4%), misdiagnosis of ischemia and acute myocardial infarction (16 cases, 7.8%), and lead misplacement (5 cases, 2.4%). Conclusions: The most frequent errors in computer ECG interpretation are related to arrhythmias, conduction disorders, and electronic pacemakers. Computer ECG diagnosis of life threatening conditions e.g. acute myocardial infarction or high degree AV blocks are frequently not accurate (40.7% and 75.0% errors, respectively). Improvement in the diagnostic algorithms should focus on these areas. Error index is a convenient and informative tool for evaluation of diagnostic accuracy. D 2005 Elsevier Ireland Ltd. All rights reserved. Keywords: ECG; Diagnostic errors; Computer

1. Introduction Part of the regular work of a cardiologist is reading of electrocardiograms (ECGs). Commonly, the reader has a computer generated diagnostic interpretation, which can be accepted or rejected, partially or in full. This analysis by the computer is extremely helpful. It can considerably speed up the process of physician ECG interpretation and help prevent errors [3]. However, frequently inaccuracies are encountered which need manual correction. Certain types of errors are encountered more frequently. The aim

T Corresponding author. Tel.: +1 313 576 3221; fax: +1 313 576 1121. E-mail addresses: [email protected], [email protected] (M.E. Guglin). 0167-5273/$ - see front matter D 2005 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijcard.2005.02.007

of this study was to determine the nature and frequency of errors by computer analysis of ECG. The reliability and accuracy of computer programs has been previously analyzed. In the setting of an emergency room, the computer made 4% of major errors (2 out of 50) [1]. On a large sample size greater than 5000 ECGs, Thompson et al. [2] concluded that overall sensitivity of computer interpretation was 90.1%, specificity 89.6%, positive predictive accuracy 87.1%, and negative predicted accuracy 92.2%. The readings of ST–T wave changes demonstrated the lowest sensitivity and specificity. A significant observation was that ECGs read as normal by computer did not require further checking. The nature and frequency of typical errors has not been in the focus of previous studies. This study focuses on the types of errors most commonly encountered. We sought to

M.E. Guglin, D. Thatai / International Journal of Cardiology 106 (2006) 232–237

identify areas of weakness which can help refine the ECG reading algorithms.

233

plus false negatives. Thus the EI was calculated as ratio of (FP + FN) / (2TP + FP + FN). True negatives are not included; therefore, the index does not depend on the prevalence of the disease.

2. Methods The ECGs were collected in the tertiary care VA Hospital from both inpatients (36.4%), outpatients (47.6%) and in the emergency room (16.0%). There were 2194 consecutive ECGs recorded on 1856 patients. Patients age ranged from 33 to 96 years, mean 73.5. Nearly all of them (98.3%) were male. The decision to record an ECG was made by the treating health care provider. No ECGs were taken for the purpose of this study or added to the sample to make it more representative. All ECGs were initially analyzed by the electrocardiograph built-in computer software (12SL). Each tracing was reread sequentially by two independent cardiologists. The first reader was one of the staff cardiologists who did not know that it was part of a study. The ECGs were read for clinical purposes only. The second reader independently analyzed each ECG and was blinded to the final interpretation of the first cardiologist. Computer ECG analysis error was identified if both cardiologist interpretations were in agreement. The final readings were changed accordingly. For the purpose of this study discordance between the two physician readers was excluded as a computer error. The 12 SL is the only diagnostic algorithm used at our institution. No other algorithms were analyzed. ECGs read as bborderlineQ, i.e. bnormal except borderline left ventricular hypertrophyQ (LVH) or bnormal except first degree atrioventricular (AV) blockQ, were considered abnormal. All cardiac rhythms except normal sinus rhythm, sinus tachycardia, sinus bradycardia, and sinus arrhythmia were also considered abnormal. Sensitivity, specificity, positive and negative predicted values were determined. A new concept for diagnostic accuracy, error index (EI) was introduced. To calculate the EI, each final diagnosis was counted, made by either the computer or cardiologists. As a result, each ECG had two diagnoses, one made by the computer, and the other by the cardiologists. The number of diagnoses of any given condition was equal to the sum of computer and cardiologists diagnoses. The percent of erroneous diagnoses (false positive plus false negative) was calculated as a percent of this number. In accepted terms, error index calculates the ratio of all diagnostic errors (false positive and false negative), which is the numerator of the formula, to all diagnoses for a given condition made by either computer or cardiologists. The denominator includes all diagnoses made by computer, which are true positives and false positives, and all diagnoses made by cardiologists, which include true positives and false negatives. Hence the denominator of the equation becomes 2 true positives plus false positives

3. Results A total of 2194 ECGs were included for analysis in the study. One hundred twenty two ECGs with a disagreement between the two cardiologists were excluded from analysis. Out of 2072 remaining cases, 776 (37.5%) were read by the computer as normal, and 1296 were abnormal. In 206 cases, there were discordances between the computer and cardiologists readings (9.9%), which constituted the computer error group. There were no discordances in the ECGs read as normal. Therefore, discordances occurred in 15.9 % of all ECGs read as abnormal. Out of 206 ECGs which computer read incorrectly, the errors in diagnosis of arrhythmia, conduction disorders and electronic pacemakers accounted for 178 cases, or 86.4%. The rest was represented by misdetection of chamber enlargement (7 cases, 3.4%), misdiagnosis of ischemia and acute myocardial infarction (MI) (16 cases, 7.8%), and erroneous interpretation of lead misplacement (5 cases, 2.4%). Primary errors lead to secondary errors, for example, unrecognized electronic pacemaker led to erroneous reading of paced beats as myocardial infarction, bundle branch block or hypertrophy. Therefore, total number of errors exceeded 206. On 52 ECGs there was an artifact which could potentially cause misinterpretation, however disagreements between computer and cardiologists occurred in only six of these ECGs representing less than 0.5% of errors. Diagnostic errors made by computer for each condition are summarized in Table 1. This data identifies the most frequent errors which mainly included arrhythmias, conduction disorders, and electronic pacemakers. Within the barrhythmiaQ group, the most common misreading was bundetermined rhythmQ (23 cases). These were finally interpreted by the cardiologists as atrial fibrillation (AF) in 5 cases, AF with premature ventricular contractions (PVCs) [4], normal sinus rhythm [4], atrial flutter [2], normal sinus rhythm with complete AV block and ventricular escape [2], and also AF with electronic ventricular pacemaker and PVCs, atrial flutter with PVCs, junctional rhythm with PVCs, normal sinus rhythm with first degree AV block, normal sinus rhythm with electronic ventricular pacemaker tracking P waves, and sequential AV pacing, one case of each. The next most common misinterpretation was bsinus rhythm with sinus arrhythmiaQ, which was, in 22 cases, thought to be a normal sinus rhythm with premature atrial contractions (PACs) by both cardiologists. In 5 cases, atrial flutter was read as normal sinus rhythm. Accelerated idioventricular rhythm (on the tracing with predominantly sinus rhythm) went completely unnoticed by the computer

234

M.E. Guglin, D. Thatai / International Journal of Cardiology 106 (2006) 232–237

Table 1 Diagnostic errors, made by computer in different conditions

Ischemia Left ventricular hypertrophy Old myocardial infarction Left atrial abnormality 1M atrio-ventricular block Right atrial abnormality Right bundle branch block Premature ventricular contraction Left bundle branch block Permanent pacemaker Atrial fibrillation Right ventricular hypertrophy Atrial flutter Premature atrial contractions Acute myocardial infarction 2M Atrio-ventricular block

Computer reading

Cardiologist reading

Overall

FP

FN

Errors FP + FN

Error Index %

Sensitivity %

Specificity %

PPV %

NPV %

203 402 349 93 141 31 114 150 33 44 60 9 29 72 19 7

199 399 344 97 138 29 118 150 33 56 67 6 41 94 8 1

402 801 693 190 279 60 232 300 66 100 127 15 70 166 27 8

4 8 9 0 6 2 4 9 3 3 9 3 2 10 11 6

0 5 4 4 3 0 8 9 3 15 16 0 14 32 0 0

4 13 13 4 9 2 12 18 6 18 25 3 16 42 11 6

1 1.6 1.9 2.1 3.2 3.3 5.2 6 9.1 18 19.7 20 22.9 25.3 40.7 75

100 98.7 98.8 95.9 97.8 100 93.2 94 90.9 73.2 76.1 100 65.9 66 100 100

99.8 99.5 99.5 100 99.7 99.9 99.8 99.5 99.9 99.9 99.6 99.9 99.9 99.5 99.5 99.7

98 98 97.4 100 95.7 93.5 96.5 94 90.9 93.2 85 66.7 93.1 86.1 42.1 14.3

100 99.7 99.8 99.8 99.8 100 99.6 99.5 99.9 99.3 99.2 100 99.3 98.4 100 100

FP-false positive (read by computer but not confirmed by cardiologists), FN-false negative (missed by computer but read by cardiologists), PPV-positive predicted value (TP / (TP + FP), NPV-negative predicted value (TN / (TN + FN), Error Index–Error Index (FP + FN) / (2TP + FP + FN).

twice. In six ECGs fusion beats were read by the computer when they were not present. In three cases the computer read bblocked PACsQ when they were not blocked. Other commonly misinterpreted conditions are presented in the Table 2. The other cases of diagnostic disagreement in arrhythmia and conduction occurred only once for each arrhythmia and were not included into the table. For instance, a wide complex tachycardia was present on two ECGs. In one case, read as bwide QRS rhythmQ, it was sinus tachycardia with bundle branch block. In the second case, ventricular tachycardia in a patient with cardiomyopathy, just resuscitated from cardiac arrest, was read as an atrial flutter with bundle branch block. One case of a first degree AV block we included as an error, because the PR interval was misread by computer as 554 ms, but when measured by cardiologists it was 300 ms. Table 2 Common rhythm misinterpretations by computer Computer Cardiologists Computer Cardiologists

Computer Cardiologists

Junctional rhythm Normal sinus rhythm Wandering atrial pacemaker Unusual P axis, possible ectopic atrial rhythm Normal sinus rhythm Sinus rhythm with premature atrial contractions Atrial flutter Atrial fibrillation Atrial fibrillation Normal sinus rhythm Sinus rhythm with premature atrial contractions Sinus rhythm with premature ventricular contractions Atrial flutter Multifocal atrial tachycardia

8 6 2 19 11 2 3 3 9 3 3 1 1 1

Second largest area of computer errors involves ECGs with electronic pacemaker. The most common error was the failure of the computer to recognize the presence of the pacemaker. In 14 cases, the computer did not recognize electronic pacemaker. It read the ECGs in a variety of ways, including banterior MI, intraventricular conduction delayQ [3], binferior MI, interventricular conduction delayQ [1], bWPW type BQ [1], bright bundle branch block (RBBB), lead reversalQ [1], bLVH, anteroseptal MIQ [1]Q, bintraventricular conduction blockQ [3], bLVHQ [1], bpulmonary disease pattern, ST elevation, consider anterior injury or ACUTE MIQ [1], banterior injury or acute MIQ [1], bpulmonary disease pattern, ST elevation, consider anterior injury or ACUTE MI, consider inferior injury or acute MIQ [1]. The rhythm was read as sinus in 11 cases and undetermined in 3. In reality, it was sinus with tracking (DDD) in 5, AF with ventricular pacing in 1, sequential AV pacing in 6, and ventricular pacing in one case. In 13 cases the computer ignored the intrinsic P waves and read belectronic ventricular pacemakerQ, which was, in reality, a normal sinus rhythm with electronic ventricular pacemaker tracking P waves. In 5 more cases, the computer did recognize electronic pacemaker, but made the interpretation based on bintrinsic beatsQ. Since there were no intrinsic beats on any of these ECGs, the same errors were repeated. In 2 more cases, dual chamber pacemaker was mistaken for atrial pacemaker, and QRS was interpreted as if it was conducted, with diagnoses of right ventricular hypertrophy, lateral MI, anterior MI etc, when in reality it was paced. Twice the computer brecognizedQ electronic pacemaker when there was none. The presence of background artifact—bnoiseQ—was the likely cause of the error, although it was obvious for the trained human eye that there was no electronic pacemaker. A small proportion in the list of errors consists of disagreement in atrial enlargement or ventricular hyper-

M.E. Guglin, D. Thatai / International Journal of Cardiology 106 (2006) 232–237

trophy. The common finding in computer-based conclusion is an binfarct, age undeterminedQ. On most of the occasions, the cardiologists chose not to argue with it and left the corresponding line in the conclusion intact. In such cases, we did not consider the diagnosis binfarctQ an error, although in many cases criteria for the diagnosis were soft. The diagnosis of bacute MIQ or binjuryQ of all localizations was analyzed. It was read by the computer in 19 cases: in 8 cases the reading was correct, and in 11 cases it was false positive (early repolarization in 3 cases, atrial flutter in 2, LVH in 4, and electronic ventricular pacemaker in 2). On one ECG, the computer read both acute anterior and acute inferior myocardial infarction, while there were paced beats. The conclusion bST elevation, consider early repolarization, acute pericarditis, or injuryQ was not considered an error, even if cardiologists were sure which of those condition took place. In two cases, the computer did not recognize old infarct, while both readers agreed that it was present. There were 5 cases of limb leads misplacement, unrecognized by the computer and read as bright axis deviationQ [2], bpulmonary disease pattern, biventricular hypertrophyQ [2], and one case of a precordial lead misplacement, read as anterior MI.

4. Discussion Most of the initial data on computerized ECG reading were published in the 1970s–1980s, when it was a new modality, and there was a need for the evidence that this method was acceptable. It is currently used ubiquitously, saving innumerable hours for cardiologists. It was shown that the computer-assisted ECG reading cuts the physicianTs time by 28% and improves the quality of the diagnosis [3]. In cases when the computer was correct, the availability of computer interpretation improved the accuracy of the diagnosis. It was also noted to be extremely beneficial in the VA computer program series [4]. In the series of Endou et al. [5], when the share of abnormal ECGs was 45.8%, computer and physician agreed in 93.8% cases. The IBM Bonner-2 (V2 MO) program provided sensitivity 88% and specificity 90% for myocardial infarctions and 93.4% and 98.7%, respectively for conduction abnormalities. Sensitivity for left ventricular hypertrophy was 90%, for AF 87%, to electronic pacemakers 65% [6]. In Japan, the computer gave 10.5% false-negative reports and 16.5% of false-positive reports.[7] The rate of detection of left ventricular hypertrophy was more accurate by physician’s interpretation than by the computer, when compared with the calculated left ventricular mass. The computer provided 45% sensitivity, 83% specificity; cardiologists provided 56% and 92%, respectively [8]. The superiority of a real expert over computer was brilliantly demonstrated by J.W. Hurst et al. [9].

235

However, incorrect reading by the computer deteriorates performance of inexperienced readers. Medical residents in general read ECGs better when they see a correct computer drawn analysis, but erroneous conclusions dropped their accuracy from 56.7% to 48.3% [10]. Ironically, a computer makes most of the mistakes in the most serious clinical situations. This was demonstrated elegantly while comparing ECG-diagnostic skills of computer versus an emergency room physician [11]. All ECGs were divided into four categories according to their clinical significance. The computer interpreted the ECGs of moderate significance better than doctors, but in the group of critical significance physicians did better (28% of correct diagnoses) than computer (14%), although overall performance was poor in both. Serial comparison of the ECGs represents additional source of errors. Insertion of any corrections into a computer database before the next comparison substantially reduces the number of errors [12]. It was also described that the computer system bsignificantly disagreed with itself in 36 of 92 pairs of unselected electrocardiograms which had not changed when recorded 1 min apartQ [13]. Willems et al. [14] concluded that the best computer programs approach best cardiologists, but overall computer performance was significantly lower. Whereas in some previously published studies, cardiologists were blinded to the computer conclusions, in others they were not. The current study focused on real life situation: ECGs are analyzed by the computer first followed by reading by two cardiologists with full access to the computer interpretation. Moreover, emphasis was placed on the most significant errors, which cardiologists corrected for real clinical purposes. bIf real-world test performance is to be evaluated, test reviewers should be given access to the information they would normally receive in the clinical settingQ [15]. In our experience, the sensitivity of computer reading for different conditions ranges between 65.9% and 100%, while the specificity for all diagnoses is above 90%. The overall error rate is 15.9% for all abnormal ECGs. Analyzing the accuracy of computer reading, one can see that the sensitivity of the diagnosis of acute MI is 100% (no false negative diagnoses); the specificity is 99.5%, negative predictive value 100%. Only the dismal low positive predictive value of 42.1% reveals a significant inaccuracy in diagnosis. Electronic pacemakers interpretation yielded a high specificity of 99.9%, positive predictive value of 93.2%, and negative predictive value of 99.3%. The only area of deficiency was in relatively lower sensitivity value of 73.2%. To compare the accuracy of the diagnosis for various different conditions, all four statistical indices (sensitivity, specificity, positive and negative predictive values) were utilized. However these different statistical modalities, as shown above, are unable to confirm the ability of the computer reading to identify one condition better than the

236

M.E. Guglin, D. Thatai / International Journal of Cardiology 106 (2006) 232–237

Table 3 Calculation of the error index for atrial fibrillation Subsets of diagnoses

Number of ECGs

%

Overall (computer and physicians reading) Correct (physicians reading) False positive (computer reading not confirmed by physicians) False negative (diagnosis made by physicians but missed by computer) Total errors / Error index (FP + FN) / (2TP + FP + FN)

127

100.0

102 9

80.3 7.1

16

12.6

25 (9 + 16)

19.7 (7.1 + 12.6)

other. To overcome this, the concept of EI [16] was utilized. An example of calculating the EI is described. Computer read the rhythm as atrial fibrillation (AF) in 60 cases. Cardiologists made the same diagnosis in 67 cases. Overall the diagnosis of AF was made in 127 cases, which represents 100% for the diagnosis of AF. In 16 cases the computer did not recognize AF, so there were 16 false negative results, or 12.6% of 127. In 9 cases, computer read bAFQ although it was not present, so there were 9 false positive diagnoses, consisting 7.1% of 127. Therefore, the overall number of errors made by computer in the diagnosis of AF was 19.7%. In the remaining 51 cases, the rhythm was interpreted as AF both by computer and cardiologist, giving overall 102 diagnoses, or 80.3% of 127. So, there were 80.3% of correct diagnoses of AF and 19.7% of erroneous diagnoses, total 100% (Table 3). To our knowledge, no other method gives such an opportunity to directly sum the diagnostic errors (falsepositive and false negative diagnoses), and to add on correct and incorrect diagnoses to get overall 100%. For second degree AV blocks (Table 1), sensitivity is 100% (there were no false negative results) and specificity is 99.7%. Meanwhile, only one reading of this condition out of seven, made by computer, was correct. By our method, the EI was 75%, which, from our standpoint, gives much better estimate of diagnostic accuracy. Based on this method, it is easy to see, from the Table 1, that next highest number of errors (40.7%) is in the diagnosis of the most clinically important area of acute MI (sensitivity 100%, specificity 99.5%). In our subset, we had only hyperdiagnostic bias, with 11 out of 19 computer readings being false positive. It is surely better than a hypodiagnostic bias, but the accuracy can barely be classified as acceptable. Trying to estimate the accuracy of computer reading of acute myocardial infarction, Porela et al. [17] matched the computer reading with elevation of cardiospecific enzymes. They found that correlation was very poor. Other areas of typical mistakes include premature atrial contractions (25.3% of errors), atrial flutter (22.9%), right ventricular hypertrophy (20%), AF (19.7%) and electronic pacemakers (18.1%). The frequency of the rest of the errors was within single digits. Insufficient quality of computer reading on ECGs with an electronic pacemaker was analyzed before, and new

algorithms were suggested [18]. In the dataset of Endou et al. [5] electronic pacemakers were the weakest area of interpretation, with the sensitivity of only 3% (computer did not recognize 32 electronic pacemakers out of 33). In our data, in 44 out of 56 cases, electronic pacemaker was recognized by the computer reflecting a significant improvement in the diagnostic accuracy. Within this subset, multiple mistakes were made: mode was diagnosed inappropriately or paced beats were considered intrinsic. With increasing number of patients with implanted pacemakers, there is a need to design better algorithms. The main limitation of this study is the use of only one of several commercially available algorithms for ECG interpretation. Having the access to the only algorithm available at our institution we cannot comment on performance of other diagnostic software. There are very few studies in the literature comparing different ECG reading algorithms, and the one we used was not included in the analysis [19]. Comparison of performance of different algorithms could be useful in the future.

5. Conclusions The computer analysis of ECGs is an extremely helpful tool and has gained widespread acceptance. It is used frequently in order to save valuable physician time without compromising on the quality of final interpretation. However there seem to be obvious deficiencies still in the computer analysis. These errors are more common in abnormal ECGs and especially unacceptable in high risk life threatening conditions. Improvement in the diagnostic algorithms should be focused on these areas. The new index for diagnostic accuracy, an error index, suggested for calculation of the percent of erroneous diagnoses gives more information then traditional application of sensitivity and specificity.

References [1] Goodacre S, Webster A, Morris F. Do computer generated ECG reports improve interpretation by accident and emergency senior house officers? Postgrad Med J 2001;77(909):455 – 7. [2] Thomson A, Mitchell S, Harris PJ. Computerized electrocardiographic interpretation: an analysis of clinical utility in 5110 electrocardiograms. Med J Aust 1989;151(8):428 – 30. [3] Brailer DJ, Kroch E, Pauly MV. The impact of computer-assisted test interpretation on physician decision making: the case of electrocardiograms. Med Decis Mak 1997;17(1):80 – 6. [4] Milliken JA, Pipberger H, Pipberger HV, et al. The impact of an ECG computer analysis program on the cardiologist’s interpretation. A cooperative study. J Electrocardiol 1983;16(2):141 – 9. [5] Endou K, Miyahara H, Sato T. Clinical usefulness of computer diagnosis in automated electrocardiography. Cardiology 1980;66(3): 174 – 89. [6] Garcia R, Breneman GM, Goldstein S. Electrocardiogram computer analysis. Practical value of the IBM Bonner-2 (V2 MO) program. J Electrocardiol 1981;14(3):283 – 8.

M.E. Guglin, D. Thatai / International Journal of Cardiology 106 (2006) 232–237 [7] Sekiguchi K, Kanda T, Osada M, Tsunoda Y, Kodajima N, Fukumura Y, et al. Comparative accuracy of automated computer analysis versus physicians in training in the interpretation of electrocardiograms. J Med 1999;30(1-2):75 – 81. [8] Devereux RB, Casale PN, Eisenberg RR, Miller DH, Kligfield P. Electrocardiographic detection of left ventricular hypertrophy using echocardiographic determination of left ventricular mass as the reference standard. Comparison of standard criteria, computer diagnosis and physician interpretation. J Am Coll Cardiol 1984; 3(1):82 – 7. [9] Hurst JW, Treasure CB, Sathavorn CS. Computer errors in electrocardiography. Clin Cardiol 1996;19(7):580 – 6. [10] Tsai TL, Fridsma DB, Gatti G. Computer decision support as a source of interpretation error: the case of electrocardiograms. J Am Med Inform Assoc 2003;10(5):478 – 83. [11] Snyder CS, Fenrich AL, Friedman RA, Macias C, O’Reilly K, Kertesz NJ. The emergency department versus the computer: which is the better electrocardiographer? Pediatr Cardiol 2003;24(4):364 – 8. [12] Bonner RE, Crevasse L, Ferrer MI, Greenfield Jr JC. The influence of editing on the performance of a computer program for serial comparison of electrocardiograms. J Electrocardiol 1983;16(2):181 – 9.

237

[13] Spodick DH, Bishop RL. Computer treason: intraobserver variability of an electrocardiographic computer system. Am J Cardiol 1997; 80(1):102 – 3. [14] Willems JL, Abreu-Lima C, Arnaud P, et al. The diagnostic performance of computer programs for the interpretation of electrocardiograms. N Engl J Med 1991;325(25):1767 – 73. [15] Mower W. Evaluating bias and variability in diagnostic test reports. Ann Emerg Med 1999;33(1):85 – 91. [16] Guglin ER. On a method of comparing diagnoses. Arkh Patol 1967; 29(2):79 – 81. [17] Porela P, Hanninen KP, Vuorenmaa T, et al. Computer-assisted electrocardiography in structured diagnosis of acute myocardial infarction. Scand Cardiovasc J 1999;33(2):89 – 96. [18] Greenhut SE, Jenkins JM, DiCarlo LA. Computerized interpretation of the paced ECG. J Electrocardiol 1992;24(Suppl):146 – 52. [19] Andresen A, Dobkin J, Maynard C, Myers R, Wagner GS, Warner RA, et al. Validation of advanced ECG diagnostic software for the detection of prior myocardial infarction by using nuclear cardiac imaging. J Electrocardiol 2001;34(Suppl):243 – 8.