- Email: [email protected]
The influence of bias on the subjective interpretation of cardiac angiograms
January,
de Caprio
et al.
American
Measurements in exercise electrocardiography. The Ernest Simonson Conference. Snrinafield, Ill, 1969. Charles C Thomas, Publisher, p 363.* 32. Rerych SK, Scholz PM, Newman GE, Sabiston DC, Jones RH: Cardiac function at rest and during exercise in normals and in patients with coronary heart disease: Evaluation by radionucleide angiocardiography. Ann Surg 197:443, 1978. 33. Stein RA, Michelli D, Fox L, Krasnow N: Continuous ventricular dimensions in man during supine exercise and recovery. An echocardiographic study. Am J Cardiol 41:655, 1978. 34. Poliner LR, Dehmer GJ, Lewis SE, Parkey RW, Blomquist CG, Willerson JT: Left ventricular performance in normal subjects: A comparison of the responses in the upright and supine position. Circulation 62:528, 1980.
The influence interpretation
Heart
1994 Journal
David D, Naito M, Chen C, Morganroth J, Schaffenburg M: R-wave amplitude variations during acute myocardial ischemia: An inadequate index for changes in intracardiac volume. Circulation 63:1364, 1981. 36. David D, Naito M, Michelson E, Watanabe Y, Chen CC, Morganroth J, Schaffenburg M, Blenko T: Intracardiac conduction: A major determinant of R-wave amplitude during acute myocardial ischemia. Circulation 65:161, 1981. 37. Gerson ML, Morris SM, McHenry PL: Relation on exerciseinduced physiologic ST segment depression to R wave amplitude in normal subjects. Am J Cardiol 46:778, 1980. 35.
of bias on the subjective of cardiac angiograms
Subjective interpretation of angiographic left ventricular regional wall motion is routinely performed with knowledge of the location and extent of coronary artery stenosis. We studied 100 patients with coronary artery disease in order to determine the accuracy of such wall motion assessment relative to a more objective standard based upon computer-assisted left ventricular (LV) ejection fraction and end-systolic fractional shortening referenced to the end-diastolic area centroid. Only 379 of 700 (54%) region-by-region comparisons of wall motion were in precise agreement. Computer-assisted wall motion analysis correlated significantly better with ejection fraction than did subjective analysis (r = 0.82 vs r = 0.61, p < 0.002). In 56 patients, in whom major discordance was noted, subjective assessment of wall motion correlated significantly better with the presence of coronary artery stenosis (p < 0.05), but objective assessment correlated significantly better with ejection fraction in these same patients (p < 0.02). These data suggest that the accuracy of subjective assessment of regional wall motion, relative to global ejection fraction, can be adversely biased by knowledge of the patient’s coronary anatomy. Because of the inherently reproducible nature of the algorithmic process, and in light of the better correlation with global function, computer-assisted analysis of regional wall motion might be preferable to conventional subjective assessment. (AM HEART J 107:68, 1984.)
George A. Diamond, M.D., Ran Vas, Ph.D., James S. Forrester, M.D., Hu Zhen Xiang, M.D., James Whiting, Ph.D., Martin Pfaff, MS., and H. J. C. Swan, M.D., Ph.D. Los Angeles and Culver City, Calif.
Medical diagnostic test results that cannot be readily quantified are generally assessed subjectively, often by a physician who possessesprior knowledge
From the Division of Cardiology, Department of Medicine, Cedars-Sinai Medical Center, UCLA School of Medicine, and the Cardiac Catheterization Laboratory, Brotman Medical Center. Received, for publication June 24, 1982; revision received Sept. 2, 1982; accepted Oct. 1, 1982. Reprint requests: George A. Diamond, M.D., Division of Cardiology, Cedars-Sinai Medical Center, 8700 Beverly Bvd., Schuman 6, Los Angeles, CA 90048. 68
of other clinical findings. Although this custom is usually considered “good medical practice,” it carries the potential for abuse, whereby unrecognizably biased interpretations improperly influence subsequent clinical decisions. For example, subjective qualitative interpretation of left ventricular wall motion and subjective quantitative assessment of the magnitude of coronary artery stenosis are fundamental to decisions concerning coronary bypass surgery. Interobserver variability for each of these radiologic interpretations, however, is in the range of 40 5%.*,2
Volume Number
107 1
Analysis
of
subjective
ON
interpretation
of
FRACTION
=
64%
FRACTION
=
34;:
LV angiograms
69
IB
e
2
.4
F F: A 11 f I D td ci L
.6
.8
1
S H 13 R T E t.4 I N G
1. Beta frequency’ distribution of normal left ventricular end-systolic fractional shortening. The shaded areas of the curve are the 5% tails of the distribution which define a 90 % confidenceinterval ranging from 19% to 61% along the x axis. Fig.
I
EJECTION
AL
A / + AA
16
Since angiographic left ventricular (LV) function is customarily assessed not in isolation but with knowledge of coronary artery anatomy, this study
was designed to determine the influence of such knowledge on the accuracy of subjective wall motion analysis and to compare subjective assessment with a more objective computer-assisted format. METHODS Quantification of subjective interpretations. The cardiac catheterization reports for 100patients with a primary diagnosisof coronary artery diseasewere selected in alphabetically consecutive order from the cardiac catheterization laboratory files of Brotman Medical Center. The only selection criterion was the presenceof at least one abnormally contracting segmentin the description of a technically adequate left ventriculogram. Each report had been dictated by the cardiologist who had performed the angiographic study. All reports were reviewed by a single cardiologist who graded left ventricular wall motion, in each of seven regions, on a semiquantitative scalebasedon the performing physician’s dictated interpretations. The seven left ventricular regionswere: anterolateral (AL), midanterior (A), anteroapical (AA), apical (X), inferoapical (IA), midinferior (I), and inferobasal (IB). Each region was graded by a five-point scoring system3where: 0 = dyskinesis,1 = akinesis,2 = hypokinesis, 3 = normal, and 4 = hyperkinesis. The potential sum of the scoresfor the seven regions,therefore, ranged from 0 to 28. The “anterior” wall motion scorewas defined as the sum of the individual scores for the anterolateral, midanterior, and anteroapical regions.A similar “inferior” wall motion scorewasderived as the sum of the scoresfor the inferobasal, midinferior, and inferoapical regions. The main left, anterior descending, circumflex, and
:
Q
I
\\ IA
X
2. Computer-assisted “objective” analysis of LV function for a representative patient with normal ejection fraction and regional systolic shortening (top) and a patient with reduced ejection fraction and anteroapical akinesis (bottom). Seetext for further discussion. Fig.
right coronary arteries were also graded on a five-point system based upon the reported estimate of percent diameter narrowing of the most stenotic segmentof that vessel:0 = normal, 1 = <40%, 2 = 40% to 59%, 3 = 60% to 90%) 4 = >90%. The total potential coronary score, therefore (anterior + inferior), ranged from 0 to 8. The coronary scorefor the “anterior circulation” wasrecorded as the larger of the scoresfor the left main and anterior descendingvessels.The scorefor the “inferior circulation” was recorded as the larger of the scoresfor the left main and circumflex if the right coronary artery was nondominant or as the right coronary score if the right coronary artery was dominant. Quantification of ejection fraction and regional wall motion. The hypothesis underlying our study required a
reanalysis of the ventriculographic wall motion assessment without concomitant knowledge of coronary anatomy. The most obvious meansof obtaining suchdata would appear to require a repeat-but now blinded-subjective assessment.The magnitude of variability for subjective assessmentof regional wall motion and coronary artery stenosisis so large,Is2however, that it was not considered advisable to reinterpret the data subjectively. Thus, we choseinstead to develop a method of reanalysiswhich was
Diamond
70
January,
et al.
American
SUBJECTIVE
WALL
MOTION
28
SCORE
21
r
. *a**
t
cso
14
**,/a* l
OBJECTIVE
WALL
MOTION
SCORE
iij
7
8
9
10
11
12
13
14
I6
16
17
18
16
PO
21
22
23
24
Fig. 3. Frequency distribution of the sum of the left ventricular wall motion scores in 100 patients. A wall motion score of 21 represents normal function of all seven ventricular regions (3 x 7). Both distributions are shifted toward the abnormal side of this value.
inherently independent of knowledge of coronary anatomy and would be subject to less variability than that of dynamic visual assessment. Computer-assisted analysis provided the basis for this method. The end-diastolic and end-systolic outline of each left ventricular angiogram was traced from the rear-projected right anterior oblique images, and left ventricular ejection fraction was calculated by area planimetry with the use of a Versawriter digitizer interfaced to an Apple II microcomputer supported by software employing analytic algorithms for determination of both left ventricular ejection fraction and regional wall motion. Extrasystolic and postextrasystolic beats were excluded from analysis. A single operator selected the end-diastolic and endsystolic frame to be digitized by visual inspection. The computer software defined the ventricular perimeter by a series of n adjacent points in the first quandrant of a rectangular X-Y plane (280 X 160 pixels). For each adjacent pair of points, i and i + 1, a triangle was defined by i, i + 1, and the origin. The area of this triangle (in square pixels) was calculated analytically, and the sum of the areas of the n triangles then represented the area enclosed by the n points. The volume represented by this area was determined by assuming the left ventricle to be an ellipsoid of revolution4.5 about a long axis, L, defined as the pixel distance from the midpoint of the last adjacent (i, i + 1) pair to the operator-selected most distant point on the digitized perimeter: V = 8A2/37rL. Ejection fraction
l
7
l
.
,* /*
I
0
6
l
.
1984 Journal
l
ebb
0,‘. ,**:
l
m
*,’
l ****‘* . l ****‘* me***’ .. 0 . **a’*** .
I
l
Heart
l .
I
14 css
Fig. 4. Correlation between the sum wall motion scores (CS) and the sum of motion scores (CO) in 100 patients. The illustrated as a reference. Overlapping illustration of all 100 points.
I
21
I
28
of the subjective the objective wall line of identity is data prevent the
was calculated from end-diastolic volume (V,) and endsystolic volume (V,): EF = (V, - V,)/V,. The magnitude of fractional systolic shortening was quantified in relation to the end-diastolic area centroid, defined as the weighted average of the n X, Y coordinates.3,6 The mean value (-t 1 SD) of end-systolic fractional shortening relative to the end-diastolic centroid was estimated from previous reports of 92 normal patients in the medical literature7-I0 to be 39.6 * 13.5%. A confidence interval for this “normal range” was determined from these data by assuming that fractional shortening, like ejection fraction, is distributed as a beta variable.” The beta function describes the distribution of a continuous probabilistic variable, such as fractional shortening, over a range of values from 0 to 1. The parameters of the distribution were determined from the mean and standard deviation of the literature-based fractional shortening, and the lower (a) and upper (b) bounds of a 90% confidence interval for fractional shortening were determined analytically from the distribution function.6 The 90% confidence interval for normal fractional shortening so determined ranged from a = 19% to b = 61% (Fig. 1). The following definition+‘* and associated scores were then established. (0) Dyskinesis: regional end-systolic displacement exceeded the end-diastolic boundary. (1) Akinesis: regional end-systolic displacement was identical (2 1 pixel) to the end-diastolic boundary. (2) Hypokinesis: regional end-systolic displacement ranged between the 90 % lower bound for normal and the end-diastolic boundary. (3) Normal: regional end-systolic displacement lay within the 90% confidence interval for normal. (4) Hyperkinesis: regional end-systolic displacement exceeded the
Volume Number
107 1
Analysis
r
28
I 0
of subjective 28
I
I
20 40 EJECTION
I
I
I
60 80 FRACTION
100
interpretation
of LV angiograms
71
r
I 0
I
I
20 40 EJECTION
I
I
60 80 FRACTION
00
I
5. Correlation between left ventricular ejection fraction and wall motion scores. The linear regressionfor objective analysis (right panel) is significantly better than for subjective analysis (left panel). The largey intercept in both graphsis expected, sincea totally akinetic ventricle with an ejection fraction of 0 would be expected to have a wall motion scoreof 7 (1 X 7). Fig.
upper 90% bound for normal. Each of the sevenventricular regionsassessed by subjective analysiswasalsoassessed by this “objective” computer algorithm. Each of the 700 ventricular regions was thus characterized by both a subjective (S) score(obtained with knowledgeof coronary anatomy) and an objective (0) score(independent of such knowledge). Statistics. Statistical comparisons employed linear regressionand Student’s t test. Comparisonsof the linear correlation coefficient were performed by meansof the z transformation. A central goal of this study was to assess the comparative accuracy of subjective and objective interpretation. Since there is no accepted“gold standard” for wall motion interpretation, we choseejection fraction as the operative reference by which “accuracy” is defined. This choice is based upon the assumption that ejection fraction represents the three-dimensional integration of all two-dimensional regional fractional shortening events.
the selection criteria employed (at least one abnormal segment by subjective analysis). There was no
RESULTS
tion implies that the poor correlation of objective and subjective interpretation did not relate to undefined implicit differences between ventricular
Objective
vs subjective
wall
motion
interpretation.
Fig. 2 illustrates the end-diastolic and end-systolic silhouettes from a typical normal and abnormal
ventriculogram. The small cross at the center of each figure represents the centroid of the end-diastolic perimeter to which regional wall motion was referenced. The line segments which radiate from the centroid toward the ventricular perimeter are the 90 % confidence intervals of end-systolic fractional shortening for each of the seven regions analyzed. Fig. 3 illustrates the frequency distribution for the sum of the seven “objective” wall motion scores (CO), based upon fractional shortening derived from these ventricular silhouettes, and for the sum of the seven “subjective” scores (CS) obtained from the physician’s interpretive report. Both distributions are skewed toward the abnormal range, reflecting
significant
difference in the distribution
of wall
motion scores analyzed subjectively and objectively (CS = 16.5 IL 3.2 [I SD] vs CO = 16.7 _+ 2.9). Although there was broad overall correlation in subjective and objective wall motion assessment in the 100 patients (Fig. 4), the correlation between individual scores, while significant 0, < O.OOOOOl), was poor (r =0.66). Of the 700 region-by-region correlations, only 379 (54%) were in precise agreement, and in 50 cases (7%) the disagreement was major (two or more grades on the five-point scale). The majority (63%) of the 321 discordant interpretations involved the distinction between grade 2 (hypokinesis) and grade 3 (normal), and the rate of discordance from region-to-region increased linearly as a function of the number of patients exhibiting these two grades (r = 0.88, p < 0.005). This observa-
regions. To further analyze the possible cause of the discordance between subjective interpretation and objective interpretation of regional wall motion, the wall motion scores were compared to coronary anatomy and to ejection fraction. Correlation motion. Left
between
ejection
fraction
and
wall
ventricular ejection fraction ranged widely in the 100 patients, from 13% to 78% (mean 53.4 +- 14.9%). The correlation between subjective assessment of wall motion and left ventricular ejection fraction was modest (r = 0.61), while the parallel correlation between objective assessment of wall motion and ejection fraction was substantially better (r = 0.82), the difference between the two corre-
72
Diamond et al.
American
.
21 :. css
January, 1984 Heart Journal
14
:
. .A,. =. =, T’--.‘?.
‘: .
.
. .
l
.
.
‘.a
. .
. 7
.
I /
.
.
u-i-i 0
I 20 40 EJECTION
60 80 FRACTION
100
0
I I 20 40 EJECTION
I I 60 00 FRACTION
I 100
Fig. 6. Correlation between left ventricular ejection fraction and wall motion scoresin the 56 patients in whom the “circulation-specific” wall motion scores(anterior and/or inferior) were discordant by more than one grade. The linear regression for objective analysis remained significantly better than for subjective analysis.See text for further discussion.
lations being highly significant (p < 0.002) (Fig. 5). These data indicate that the subjective interpretation of wall motion is less accurate and less precise
than objective interpretation, when judged against a reference that represents the functional sum of the individual regional movements. Correlation wall motion.
between
coronary
artery
anatomy
and
There was a slight negative correlation between the 200 “circulation-specific” coronary wall motion scores for the 100 anterior and 100 inferior regions (r = - 0.32, p < 0.002 for the objective scores and r = - 0.33, p < 0.001 for the subjective scores). The major source of discrepancy lay in the recording of a normal wall motion score with an abnormal coronary score, representing the presence of normal
resting regional motion in the face of significant
coronary artery stenosis. In 70 of the 200 regions (56 patients), however, there was a difference of two or more grades between the subjective and objective wall motion scores. In these cases (37 anterior, 33 inferior), the correlation of subjective wall motion with coronary anatomy was similar to the overall group (r = -0.39, p < 0.002), but the objective wall motion score correlation was no longer significant (r = - 0.19, p > 0.1). In 43 of the 70 instances (61%), the subjective wall motion score was in closer agreement to the magnitude of coronary artery stenosis than was the objective score (p < 0.05). These data suggested the possibility that the angiographers might have systematically modified their subjective interpretations of wall motion in response to their knowledge of coronary anatomy. If this did occur, one would expect the correlation between subjective wall motion score and ejection fraction in this subgroup of the patients to be significantly lower than in the remaining 44 patients. This was the case: The correlation coefficients were significantly different
(r = 0.55 for the 56 discordant
patients vs r = 0.75 for the 44 concordant patients; p < 0.05). A parallel analysis of the objective interpretations demonstrated no significant differences (r = 0.78 for the discordant group vs r = 0.88 for the concordant group). Thus, in this discordant population, objective assessment of wall motion correlated more closely with its global functional analogue-ejection fraction-than did subjective assessment (r = 0.78 vs r = 0.55, p < 0.02). These data imply that the better correlation for subjective assessment of wall
motion with coronary anatomy represents a dependent interpretive bias and that objective interpretation of wall motion is significantly more accurate than is subjective interpretation relative to global ejection fraction (Fig. 6). DISCUSSION
Contrast ventriculography is the conventional standard to which a number of newer noninvasive methods have been referenced.13-I5 Unfortunately, assessment of wall motion is fundamentally subjective and is associated with a large interpretive variability, even in the hands of highly experienced observers. Zir, et al.’ for instance, reported a 42% variability when using a scoring system similar to that employed in this study. One might think, then, that wall motion assessment would be improved by a more “informed” interpretation: that is, by knowledge of other related clinical factors such as the patient’s chest pain history, resting ECG, stress test results, and coronary anatomy. With this information, the interpreter might better “detect” a borderline region of hypokinesis that would otherwise go unnoticed. Interpreter
knowledge
of coronary
anatomy.
Our
results suggest that interpreters with prior knowledge of at least one of these clinical factors, coronary
Volume
107
Number
1
anatomy, do indeed incorporate such information into their judgements, but not necessarily in a manner which leads to more suitable assessment. Thus, knowledge of the coronary anatomy appeared to introduce a significant degree of interpretive bias into the subjective assessment of left ventricular wall motion. When subjective wall motion analysis was compared to a more objective method which was based upon an explicit, graphic confidence interval for normal motion, interpreters were observed to have overestimated the degree of regional dysfunction when the coronary artery supplying that left ventricular region was judged to be more than moderately narrowed (34 of 56 instances), and they underestimated the degree of dysfunction when the vessels were judged to be less than moderately narrowed (9 of 14 instances), each by a ratio of 2:l. Limitations. The objective method employed in this study did not consider the frame-by-frame temporal pattern of wall motion. Such phasic information might contribute importantly to the accuracy of wall motion assessment and would not necessarily be reflected in our correlation with global ejection fraction. Thus, subjective visual assessment might be more sensitive than objective analysis but, of necessity, would be less specific. An optimal balance between sensitivity and specificity might be achieved by defining the range of fractional shortening associated with various degrees of subjective regional dysfunction. Preliminary data from our laboratory indicate that the distribution of fractional shortening in subjectively normal regions overlaps substantially with that for subjectively abnormal regions. Previous studies from our laboratory3 employed a regional wall motion algorithm which computed a centroid for each frame of the cardiac cycle. Distance from this centroid to a selected point on the cardiac perimeter was determined by triangulation, and wall motion was expressed as the difference in this distance on successive frames. This algorithm, therefore, was based upon a moving internal reference, which would not be expected to correspond to subjective assessment of wall motion. For example, if a single region of the left ventricle is rendered akinetic, the centroid vector shifts toward this region during systole, decreasing the distance between the frame-specific centroid and the akinetic perimeter, resulting in fictitious “shortening,” relative to visual interpretation. This abberation is largely offset by referencing wall motion to a fixed centroid (defined in this case from the end-diastolic frame). It should be noted that the heart does not
Analysis
of
subjective interpretation
of
LV angiograms
73
actually contract in relation to any definable point or set of points. In this context, all reference systems are arbitrary, and the selection of any one is a matter of convenience. Our use of a fixed reference is supported, however, not only by its theoretic practicality, but also by studies using external reference systems which have demonstrated that translational and rotational motions of the heart are small in magnitude relative to actual segment shortening.“j Clinical implications. These data have major clinical implications. Physicians are taught to employ diagnostic tests to guide or confirm their clinical judgement, not to dictate it. The rule has merit. For example, we are thereby readily able to identify a “positive” electrocardiographic stress test in an asymptomatic young woman as a probable false positive. Viewed in this way, such clinical bias is a positive influence. But, there are limits to its practicality. Our prior information is often not easily quantified or appropriately weighted. When we allow our jugement to be unduly influenced by prior information, we step across the invisible line separating sagacity and self-deception. For example, might not objectively normal-but subjectively borderline-inferior wall motion be overinterpreted as “abnormal” to justify one’s belief that a questionable right coronary stenosis is “functionally significant?” An equally insidious problem can arise in the conduct of a clinical trial, where soft, subjective data, such as the estimated magnitude of coronary artery stenosis, are employed as primary defining characteristics of the study population. Minor differences in this subjective judgement could thereby majorly influence the ultimate conclusions of the trial. This potentially critical limitation often goes unstated when the results are published and has been termed by Meier,17 “the dirty little secret.” We conclude that subjective assessments of left ventricular regional wall motion should be supplemented by more objective and, therefore, inherently more reproducible methods. In this regard, the method of wall motion analysis reported herein appears suitably practical for routine clinical application, requiring less than 3 minutes per study. REFERENCES
1. Zir LM, Miller SW, Dinsmore RE, Gilbert JP, Harthorne JW: Interobserver variability in coronary angiography. Circulation 33:627, 1976. 2. Chaitman BR, DeMots H, Bristow JD, Rosch J, Rahimtoola SH: Objective and subjective analysis of left ventricular angiograms. Circulation 52:420, 1975. 3. Tzivoni D, Diamond GA, Pichler M, Stankus K, Vas R, Forrester JS: Analysis of regional ischemic left ventricular dysfunction by quantitative cineangiography. Circulation 60:1278, 1979.
74
Diamond
et al.
4. Kennedy
5.
6.
7.
8.
9.
10.
11.
JW, Trenholme SE, Kasser IS: Left ventricular volume and mass from single plane cineangiograms: A comparison of anteroposterior and right anterior oblique methods. AM HEART J 80:343, 1979. Sandler H, Dodge HT: The use of single plane angiograms for the calculation of left ventricular volume in man. AM HEART J 75:325, 1968. Vas R, Diamond GA, Forrester JS, Whiting JS, Pfaff MJ, Levisman JA, Nakano FS, Swan HJC: Computer-enhanced digital angiography: Correlation of clinical assessment of left ventricular ejection fraction and regional wall motion. AM HEART J 104:732, 1982. Bove AA, Kreulen TH, Span JF: Computer analysis of left ventricular dynamic geometery in man. Am J Cardiol 41:1239, 1978. Dove JT, Shah PM, Schreiner BF: Effects of nitroglycerin on left ventricular wall motion in coronary artery disease. Circulation 49:682, 1974. Leighton RF, Wilt SM, Lewis RP: Detection of hypokinesis by quantitative analysis of left ventricular cineangiograms. Circulation 50:121, 1974. Gelberg HJ, Brundage GH, Glantz S, Parmley WW: Quantitative left ventricular wall motion analysis: A comparison of area chord and radial methods. Circulation 59:991, 1979. Diamond GA, Forrester JS: Improved interpretation of a
American
12.
13.
14.
15.
16.
17.
January, 1994 Heart Journal
continuous variable in diagnostic testing: Probabilistic analysis of scintigraphic rest and exercise left ventricular ejection frac ion for coronary disease detection. AM HEART J 102:189, 1981. Forrester JS, Wyatt HL, da Luz PL, Tyberg JV, Diamond GA, Swan HJC: Functional significance of regional ischemic contraction abnormalities. Circulation 54:64, 1976. Johnstone DE. Sands MJ. Bereer HF. Reduto LA. Lachman AS, Wackers FJ, Cohen LS, Goitschalk A, Zaret BL: Comparison of exercise radionuclide angiography and thallium 201 myocardial perfusion imaging in coronary artery disease. Am J Cardiol 45:1113, 1980. Folland E, Parisi AF, Moynihan PF, Jones RD, Feldman CL, Tow DE: Assessment of left ventricular ejection fraction and volumes by real time two-dimensional echocardiography. Circulation 60:760, 1979. Silverberg RA, Diamond GA, Vas R, Tzivoni D, Swan HJC, Forrester JS: Noninvasive diagnosis of coronary artery disease: The cardiokymographic stress test. Circulation 61:579, 1980. Chaitman BR, Bristpw JD, Rahimtoola SH: Left ventricular wall motion assessed by using fixed external reference system. Circulation 48:1043, 1973. Meier P: Stratification in the design of a clinical trial. Controlled Clin Trials 1:355, 1981.
de Caprio
et al.
American
Measurements in exercise electrocardiography. The Ernest Simonson Conference. Snrinafield, Ill, 1969. Charles C Thomas, Publisher, p 363.* 32. Rerych SK, Scholz PM, Newman GE, Sabiston DC, Jones RH: Cardiac function at rest and during exercise in normals and in patients with coronary heart disease: Evaluation by radionucleide angiocardiography. Ann Surg 197:443, 1978. 33. Stein RA, Michelli D, Fox L, Krasnow N: Continuous ventricular dimensions in man during supine exercise and recovery. An echocardiographic study. Am J Cardiol 41:655, 1978. 34. Poliner LR, Dehmer GJ, Lewis SE, Parkey RW, Blomquist CG, Willerson JT: Left ventricular performance in normal subjects: A comparison of the responses in the upright and supine position. Circulation 62:528, 1980.
The influence interpretation
Heart
1994 Journal
David D, Naito M, Chen C, Morganroth J, Schaffenburg M: R-wave amplitude variations during acute myocardial ischemia: An inadequate index for changes in intracardiac volume. Circulation 63:1364, 1981. 36. David D, Naito M, Michelson E, Watanabe Y, Chen CC, Morganroth J, Schaffenburg M, Blenko T: Intracardiac conduction: A major determinant of R-wave amplitude during acute myocardial ischemia. Circulation 65:161, 1981. 37. Gerson ML, Morris SM, McHenry PL: Relation on exerciseinduced physiologic ST segment depression to R wave amplitude in normal subjects. Am J Cardiol 46:778, 1980. 35.
of bias on the subjective of cardiac angiograms
Subjective interpretation of angiographic left ventricular regional wall motion is routinely performed with knowledge of the location and extent of coronary artery stenosis. We studied 100 patients with coronary artery disease in order to determine the accuracy of such wall motion assessment relative to a more objective standard based upon computer-assisted left ventricular (LV) ejection fraction and end-systolic fractional shortening referenced to the end-diastolic area centroid. Only 379 of 700 (54%) region-by-region comparisons of wall motion were in precise agreement. Computer-assisted wall motion analysis correlated significantly better with ejection fraction than did subjective analysis (r = 0.82 vs r = 0.61, p < 0.002). In 56 patients, in whom major discordance was noted, subjective assessment of wall motion correlated significantly better with the presence of coronary artery stenosis (p < 0.05), but objective assessment correlated significantly better with ejection fraction in these same patients (p < 0.02). These data suggest that the accuracy of subjective assessment of regional wall motion, relative to global ejection fraction, can be adversely biased by knowledge of the patient’s coronary anatomy. Because of the inherently reproducible nature of the algorithmic process, and in light of the better correlation with global function, computer-assisted analysis of regional wall motion might be preferable to conventional subjective assessment. (AM HEART J 107:68, 1984.)
George A. Diamond, M.D., Ran Vas, Ph.D., James S. Forrester, M.D., Hu Zhen Xiang, M.D., James Whiting, Ph.D., Martin Pfaff, MS., and H. J. C. Swan, M.D., Ph.D. Los Angeles and Culver City, Calif.
Medical diagnostic test results that cannot be readily quantified are generally assessed subjectively, often by a physician who possessesprior knowledge
From the Division of Cardiology, Department of Medicine, Cedars-Sinai Medical Center, UCLA School of Medicine, and the Cardiac Catheterization Laboratory, Brotman Medical Center. Received, for publication June 24, 1982; revision received Sept. 2, 1982; accepted Oct. 1, 1982. Reprint requests: George A. Diamond, M.D., Division of Cardiology, Cedars-Sinai Medical Center, 8700 Beverly Bvd., Schuman 6, Los Angeles, CA 90048. 68
of other clinical findings. Although this custom is usually considered “good medical practice,” it carries the potential for abuse, whereby unrecognizably biased interpretations improperly influence subsequent clinical decisions. For example, subjective qualitative interpretation of left ventricular wall motion and subjective quantitative assessment of the magnitude of coronary artery stenosis are fundamental to decisions concerning coronary bypass surgery. Interobserver variability for each of these radiologic interpretations, however, is in the range of 40 5%.*,2
Volume Number
107 1
Analysis
of
subjective
ON
interpretation
of
FRACTION
=
64%
FRACTION
=
34;:
LV angiograms
69
IB
e
2
.4
F F: A 11 f I D td ci L
.6
.8
1
S H 13 R T E t.4 I N G
1. Beta frequency’ distribution of normal left ventricular end-systolic fractional shortening. The shaded areas of the curve are the 5% tails of the distribution which define a 90 % confidenceinterval ranging from 19% to 61% along the x axis. Fig.
I
EJECTION
AL
A / + AA
16
Since angiographic left ventricular (LV) function is customarily assessed not in isolation but with knowledge of coronary artery anatomy, this study
was designed to determine the influence of such knowledge on the accuracy of subjective wall motion analysis and to compare subjective assessment with a more objective computer-assisted format. METHODS Quantification of subjective interpretations. The cardiac catheterization reports for 100patients with a primary diagnosisof coronary artery diseasewere selected in alphabetically consecutive order from the cardiac catheterization laboratory files of Brotman Medical Center. The only selection criterion was the presenceof at least one abnormally contracting segmentin the description of a technically adequate left ventriculogram. Each report had been dictated by the cardiologist who had performed the angiographic study. All reports were reviewed by a single cardiologist who graded left ventricular wall motion, in each of seven regions, on a semiquantitative scalebasedon the performing physician’s dictated interpretations. The seven left ventricular regionswere: anterolateral (AL), midanterior (A), anteroapical (AA), apical (X), inferoapical (IA), midinferior (I), and inferobasal (IB). Each region was graded by a five-point scoring system3where: 0 = dyskinesis,1 = akinesis,2 = hypokinesis, 3 = normal, and 4 = hyperkinesis. The potential sum of the scoresfor the seven regions,therefore, ranged from 0 to 28. The “anterior” wall motion scorewas defined as the sum of the individual scores for the anterolateral, midanterior, and anteroapical regions.A similar “inferior” wall motion scorewasderived as the sum of the scoresfor the inferobasal, midinferior, and inferoapical regions. The main left, anterior descending, circumflex, and
:
Q
I
\\ IA
X
2. Computer-assisted “objective” analysis of LV function for a representative patient with normal ejection fraction and regional systolic shortening (top) and a patient with reduced ejection fraction and anteroapical akinesis (bottom). Seetext for further discussion. Fig.
right coronary arteries were also graded on a five-point system based upon the reported estimate of percent diameter narrowing of the most stenotic segmentof that vessel:0 = normal, 1 = <40%, 2 = 40% to 59%, 3 = 60% to 90%) 4 = >90%. The total potential coronary score, therefore (anterior + inferior), ranged from 0 to 8. The coronary scorefor the “anterior circulation” wasrecorded as the larger of the scoresfor the left main and anterior descendingvessels.The scorefor the “inferior circulation” was recorded as the larger of the scoresfor the left main and circumflex if the right coronary artery was nondominant or as the right coronary score if the right coronary artery was dominant. Quantification of ejection fraction and regional wall motion. The hypothesis underlying our study required a
reanalysis of the ventriculographic wall motion assessment without concomitant knowledge of coronary anatomy. The most obvious meansof obtaining suchdata would appear to require a repeat-but now blinded-subjective assessment.The magnitude of variability for subjective assessmentof regional wall motion and coronary artery stenosisis so large,Is2however, that it was not considered advisable to reinterpret the data subjectively. Thus, we choseinstead to develop a method of reanalysiswhich was
Diamond
70
January,
et al.
American
SUBJECTIVE
WALL
MOTION
28
SCORE
21
r
. *a**
t
cso
14
**,/a* l
OBJECTIVE
WALL
MOTION
SCORE
iij
7
8
9
10
11
12
13
14
I6
16
17
18
16
PO
21
22
23
24
Fig. 3. Frequency distribution of the sum of the left ventricular wall motion scores in 100 patients. A wall motion score of 21 represents normal function of all seven ventricular regions (3 x 7). Both distributions are shifted toward the abnormal side of this value.
inherently independent of knowledge of coronary anatomy and would be subject to less variability than that of dynamic visual assessment. Computer-assisted analysis provided the basis for this method. The end-diastolic and end-systolic outline of each left ventricular angiogram was traced from the rear-projected right anterior oblique images, and left ventricular ejection fraction was calculated by area planimetry with the use of a Versawriter digitizer interfaced to an Apple II microcomputer supported by software employing analytic algorithms for determination of both left ventricular ejection fraction and regional wall motion. Extrasystolic and postextrasystolic beats were excluded from analysis. A single operator selected the end-diastolic and endsystolic frame to be digitized by visual inspection. The computer software defined the ventricular perimeter by a series of n adjacent points in the first quandrant of a rectangular X-Y plane (280 X 160 pixels). For each adjacent pair of points, i and i + 1, a triangle was defined by i, i + 1, and the origin. The area of this triangle (in square pixels) was calculated analytically, and the sum of the areas of the n triangles then represented the area enclosed by the n points. The volume represented by this area was determined by assuming the left ventricle to be an ellipsoid of revolution4.5 about a long axis, L, defined as the pixel distance from the midpoint of the last adjacent (i, i + 1) pair to the operator-selected most distant point on the digitized perimeter: V = 8A2/37rL. Ejection fraction
l
7
l
.
,* /*
I
0
6
l
.
1984 Journal
l
ebb
0,‘. ,**:
l
m
*,’
l ****‘* . l ****‘* me***’ .. 0 . **a’*** .
I
l
Heart
l .
I
14 css
Fig. 4. Correlation between the sum wall motion scores (CS) and the sum of motion scores (CO) in 100 patients. The illustrated as a reference. Overlapping illustration of all 100 points.
I
21
I
28
of the subjective the objective wall line of identity is data prevent the
was calculated from end-diastolic volume (V,) and endsystolic volume (V,): EF = (V, - V,)/V,. The magnitude of fractional systolic shortening was quantified in relation to the end-diastolic area centroid, defined as the weighted average of the n X, Y coordinates.3,6 The mean value (-t 1 SD) of end-systolic fractional shortening relative to the end-diastolic centroid was estimated from previous reports of 92 normal patients in the medical literature7-I0 to be 39.6 * 13.5%. A confidence interval for this “normal range” was determined from these data by assuming that fractional shortening, like ejection fraction, is distributed as a beta variable.” The beta function describes the distribution of a continuous probabilistic variable, such as fractional shortening, over a range of values from 0 to 1. The parameters of the distribution were determined from the mean and standard deviation of the literature-based fractional shortening, and the lower (a) and upper (b) bounds of a 90% confidence interval for fractional shortening were determined analytically from the distribution function.6 The 90% confidence interval for normal fractional shortening so determined ranged from a = 19% to b = 61% (Fig. 1). The following definition+‘* and associated scores were then established. (0) Dyskinesis: regional end-systolic displacement exceeded the end-diastolic boundary. (1) Akinesis: regional end-systolic displacement was identical (2 1 pixel) to the end-diastolic boundary. (2) Hypokinesis: regional end-systolic displacement ranged between the 90 % lower bound for normal and the end-diastolic boundary. (3) Normal: regional end-systolic displacement lay within the 90% confidence interval for normal. (4) Hyperkinesis: regional end-systolic displacement exceeded the
Volume Number
107 1
Analysis
r
28
I 0
of subjective 28
I
I
20 40 EJECTION
I
I
I
60 80 FRACTION
100
interpretation
of LV angiograms
71
r
I 0
I
I
20 40 EJECTION
I
I
60 80 FRACTION
00
I
5. Correlation between left ventricular ejection fraction and wall motion scores. The linear regressionfor objective analysis (right panel) is significantly better than for subjective analysis (left panel). The largey intercept in both graphsis expected, sincea totally akinetic ventricle with an ejection fraction of 0 would be expected to have a wall motion scoreof 7 (1 X 7). Fig.
upper 90% bound for normal. Each of the sevenventricular regionsassessed by subjective analysiswasalsoassessed by this “objective” computer algorithm. Each of the 700 ventricular regions was thus characterized by both a subjective (S) score(obtained with knowledgeof coronary anatomy) and an objective (0) score(independent of such knowledge). Statistics. Statistical comparisons employed linear regressionand Student’s t test. Comparisonsof the linear correlation coefficient were performed by meansof the z transformation. A central goal of this study was to assess the comparative accuracy of subjective and objective interpretation. Since there is no accepted“gold standard” for wall motion interpretation, we choseejection fraction as the operative reference by which “accuracy” is defined. This choice is based upon the assumption that ejection fraction represents the three-dimensional integration of all two-dimensional regional fractional shortening events.
the selection criteria employed (at least one abnormal segment by subjective analysis). There was no
RESULTS
tion implies that the poor correlation of objective and subjective interpretation did not relate to undefined implicit differences between ventricular
Objective
vs subjective
wall
motion
interpretation.
Fig. 2 illustrates the end-diastolic and end-systolic silhouettes from a typical normal and abnormal
ventriculogram. The small cross at the center of each figure represents the centroid of the end-diastolic perimeter to which regional wall motion was referenced. The line segments which radiate from the centroid toward the ventricular perimeter are the 90 % confidence intervals of end-systolic fractional shortening for each of the seven regions analyzed. Fig. 3 illustrates the frequency distribution for the sum of the seven “objective” wall motion scores (CO), based upon fractional shortening derived from these ventricular silhouettes, and for the sum of the seven “subjective” scores (CS) obtained from the physician’s interpretive report. Both distributions are skewed toward the abnormal range, reflecting
significant
difference in the distribution
of wall
motion scores analyzed subjectively and objectively (CS = 16.5 IL 3.2 [I SD] vs CO = 16.7 _+ 2.9). Although there was broad overall correlation in subjective and objective wall motion assessment in the 100 patients (Fig. 4), the correlation between individual scores, while significant 0, < O.OOOOOl), was poor (r =0.66). Of the 700 region-by-region correlations, only 379 (54%) were in precise agreement, and in 50 cases (7%) the disagreement was major (two or more grades on the five-point scale). The majority (63%) of the 321 discordant interpretations involved the distinction between grade 2 (hypokinesis) and grade 3 (normal), and the rate of discordance from region-to-region increased linearly as a function of the number of patients exhibiting these two grades (r = 0.88, p < 0.005). This observa-
regions. To further analyze the possible cause of the discordance between subjective interpretation and objective interpretation of regional wall motion, the wall motion scores were compared to coronary anatomy and to ejection fraction. Correlation motion. Left
between
ejection
fraction
and
wall
ventricular ejection fraction ranged widely in the 100 patients, from 13% to 78% (mean 53.4 +- 14.9%). The correlation between subjective assessment of wall motion and left ventricular ejection fraction was modest (r = 0.61), while the parallel correlation between objective assessment of wall motion and ejection fraction was substantially better (r = 0.82), the difference between the two corre-
72
Diamond et al.
American
.
21 :. css
January, 1984 Heart Journal
14
:
. .A,. =. =, T’--.‘?.
‘: .
.
. .
l
.
.
‘.a
. .
. 7
.
I /
.
.
u-i-i 0
I 20 40 EJECTION
60 80 FRACTION
100
0
I I 20 40 EJECTION
I I 60 00 FRACTION
I 100
Fig. 6. Correlation between left ventricular ejection fraction and wall motion scoresin the 56 patients in whom the “circulation-specific” wall motion scores(anterior and/or inferior) were discordant by more than one grade. The linear regression for objective analysis remained significantly better than for subjective analysis.See text for further discussion.
lations being highly significant (p < 0.002) (Fig. 5). These data indicate that the subjective interpretation of wall motion is less accurate and less precise
than objective interpretation, when judged against a reference that represents the functional sum of the individual regional movements. Correlation wall motion.
between
coronary
artery
anatomy
and
There was a slight negative correlation between the 200 “circulation-specific” coronary wall motion scores for the 100 anterior and 100 inferior regions (r = - 0.32, p < 0.002 for the objective scores and r = - 0.33, p < 0.001 for the subjective scores). The major source of discrepancy lay in the recording of a normal wall motion score with an abnormal coronary score, representing the presence of normal
resting regional motion in the face of significant
coronary artery stenosis. In 70 of the 200 regions (56 patients), however, there was a difference of two or more grades between the subjective and objective wall motion scores. In these cases (37 anterior, 33 inferior), the correlation of subjective wall motion with coronary anatomy was similar to the overall group (r = -0.39, p < 0.002), but the objective wall motion score correlation was no longer significant (r = - 0.19, p > 0.1). In 43 of the 70 instances (61%), the subjective wall motion score was in closer agreement to the magnitude of coronary artery stenosis than was the objective score (p < 0.05). These data suggested the possibility that the angiographers might have systematically modified their subjective interpretations of wall motion in response to their knowledge of coronary anatomy. If this did occur, one would expect the correlation between subjective wall motion score and ejection fraction in this subgroup of the patients to be significantly lower than in the remaining 44 patients. This was the case: The correlation coefficients were significantly different
(r = 0.55 for the 56 discordant
patients vs r = 0.75 for the 44 concordant patients; p < 0.05). A parallel analysis of the objective interpretations demonstrated no significant differences (r = 0.78 for the discordant group vs r = 0.88 for the concordant group). Thus, in this discordant population, objective assessment of wall motion correlated more closely with its global functional analogue-ejection fraction-than did subjective assessment (r = 0.78 vs r = 0.55, p < 0.02). These data imply that the better correlation for subjective assessment of wall
motion with coronary anatomy represents a dependent interpretive bias and that objective interpretation of wall motion is significantly more accurate than is subjective interpretation relative to global ejection fraction (Fig. 6). DISCUSSION
Contrast ventriculography is the conventional standard to which a number of newer noninvasive methods have been referenced.13-I5 Unfortunately, assessment of wall motion is fundamentally subjective and is associated with a large interpretive variability, even in the hands of highly experienced observers. Zir, et al.’ for instance, reported a 42% variability when using a scoring system similar to that employed in this study. One might think, then, that wall motion assessment would be improved by a more “informed” interpretation: that is, by knowledge of other related clinical factors such as the patient’s chest pain history, resting ECG, stress test results, and coronary anatomy. With this information, the interpreter might better “detect” a borderline region of hypokinesis that would otherwise go unnoticed. Interpreter
knowledge
of coronary
anatomy.
Our
results suggest that interpreters with prior knowledge of at least one of these clinical factors, coronary
Volume
107
Number
1
anatomy, do indeed incorporate such information into their judgements, but not necessarily in a manner which leads to more suitable assessment. Thus, knowledge of the coronary anatomy appeared to introduce a significant degree of interpretive bias into the subjective assessment of left ventricular wall motion. When subjective wall motion analysis was compared to a more objective method which was based upon an explicit, graphic confidence interval for normal motion, interpreters were observed to have overestimated the degree of regional dysfunction when the coronary artery supplying that left ventricular region was judged to be more than moderately narrowed (34 of 56 instances), and they underestimated the degree of dysfunction when the vessels were judged to be less than moderately narrowed (9 of 14 instances), each by a ratio of 2:l. Limitations. The objective method employed in this study did not consider the frame-by-frame temporal pattern of wall motion. Such phasic information might contribute importantly to the accuracy of wall motion assessment and would not necessarily be reflected in our correlation with global ejection fraction. Thus, subjective visual assessment might be more sensitive than objective analysis but, of necessity, would be less specific. An optimal balance between sensitivity and specificity might be achieved by defining the range of fractional shortening associated with various degrees of subjective regional dysfunction. Preliminary data from our laboratory indicate that the distribution of fractional shortening in subjectively normal regions overlaps substantially with that for subjectively abnormal regions. Previous studies from our laboratory3 employed a regional wall motion algorithm which computed a centroid for each frame of the cardiac cycle. Distance from this centroid to a selected point on the cardiac perimeter was determined by triangulation, and wall motion was expressed as the difference in this distance on successive frames. This algorithm, therefore, was based upon a moving internal reference, which would not be expected to correspond to subjective assessment of wall motion. For example, if a single region of the left ventricle is rendered akinetic, the centroid vector shifts toward this region during systole, decreasing the distance between the frame-specific centroid and the akinetic perimeter, resulting in fictitious “shortening,” relative to visual interpretation. This abberation is largely offset by referencing wall motion to a fixed centroid (defined in this case from the end-diastolic frame). It should be noted that the heart does not
Analysis
of
subjective interpretation
of
LV angiograms
73
actually contract in relation to any definable point or set of points. In this context, all reference systems are arbitrary, and the selection of any one is a matter of convenience. Our use of a fixed reference is supported, however, not only by its theoretic practicality, but also by studies using external reference systems which have demonstrated that translational and rotational motions of the heart are small in magnitude relative to actual segment shortening.“j Clinical implications. These data have major clinical implications. Physicians are taught to employ diagnostic tests to guide or confirm their clinical judgement, not to dictate it. The rule has merit. For example, we are thereby readily able to identify a “positive” electrocardiographic stress test in an asymptomatic young woman as a probable false positive. Viewed in this way, such clinical bias is a positive influence. But, there are limits to its practicality. Our prior information is often not easily quantified or appropriately weighted. When we allow our jugement to be unduly influenced by prior information, we step across the invisible line separating sagacity and self-deception. For example, might not objectively normal-but subjectively borderline-inferior wall motion be overinterpreted as “abnormal” to justify one’s belief that a questionable right coronary stenosis is “functionally significant?” An equally insidious problem can arise in the conduct of a clinical trial, where soft, subjective data, such as the estimated magnitude of coronary artery stenosis, are employed as primary defining characteristics of the study population. Minor differences in this subjective judgement could thereby majorly influence the ultimate conclusions of the trial. This potentially critical limitation often goes unstated when the results are published and has been termed by Meier,17 “the dirty little secret.” We conclude that subjective assessments of left ventricular regional wall motion should be supplemented by more objective and, therefore, inherently more reproducible methods. In this regard, the method of wall motion analysis reported herein appears suitably practical for routine clinical application, requiring less than 3 minutes per study. REFERENCES
1. Zir LM, Miller SW, Dinsmore RE, Gilbert JP, Harthorne JW: Interobserver variability in coronary angiography. Circulation 33:627, 1976. 2. Chaitman BR, DeMots H, Bristow JD, Rosch J, Rahimtoola SH: Objective and subjective analysis of left ventricular angiograms. Circulation 52:420, 1975. 3. Tzivoni D, Diamond GA, Pichler M, Stankus K, Vas R, Forrester JS: Analysis of regional ischemic left ventricular dysfunction by quantitative cineangiography. Circulation 60:1278, 1979.
74
Diamond
et al.
4. Kennedy
5.
6.
7.
8.
9.
10.
11.
JW, Trenholme SE, Kasser IS: Left ventricular volume and mass from single plane cineangiograms: A comparison of anteroposterior and right anterior oblique methods. AM HEART J 80:343, 1979. Sandler H, Dodge HT: The use of single plane angiograms for the calculation of left ventricular volume in man. AM HEART J 75:325, 1968. Vas R, Diamond GA, Forrester JS, Whiting JS, Pfaff MJ, Levisman JA, Nakano FS, Swan HJC: Computer-enhanced digital angiography: Correlation of clinical assessment of left ventricular ejection fraction and regional wall motion. AM HEART J 104:732, 1982. Bove AA, Kreulen TH, Span JF: Computer analysis of left ventricular dynamic geometery in man. Am J Cardiol 41:1239, 1978. Dove JT, Shah PM, Schreiner BF: Effects of nitroglycerin on left ventricular wall motion in coronary artery disease. Circulation 49:682, 1974. Leighton RF, Wilt SM, Lewis RP: Detection of hypokinesis by quantitative analysis of left ventricular cineangiograms. Circulation 50:121, 1974. Gelberg HJ, Brundage GH, Glantz S, Parmley WW: Quantitative left ventricular wall motion analysis: A comparison of area chord and radial methods. Circulation 59:991, 1979. Diamond GA, Forrester JS: Improved interpretation of a
American
12.
13.
14.
15.
16.
17.
January, 1994 Heart Journal
continuous variable in diagnostic testing: Probabilistic analysis of scintigraphic rest and exercise left ventricular ejection frac ion for coronary disease detection. AM HEART J 102:189, 1981. Forrester JS, Wyatt HL, da Luz PL, Tyberg JV, Diamond GA, Swan HJC: Functional significance of regional ischemic contraction abnormalities. Circulation 54:64, 1976. Johnstone DE. Sands MJ. Bereer HF. Reduto LA. Lachman AS, Wackers FJ, Cohen LS, Goitschalk A, Zaret BL: Comparison of exercise radionuclide angiography and thallium 201 myocardial perfusion imaging in coronary artery disease. Am J Cardiol 45:1113, 1980. Folland E, Parisi AF, Moynihan PF, Jones RD, Feldman CL, Tow DE: Assessment of left ventricular ejection fraction and volumes by real time two-dimensional echocardiography. Circulation 60:760, 1979. Silverberg RA, Diamond GA, Vas R, Tzivoni D, Swan HJC, Forrester JS: Noninvasive diagnosis of coronary artery disease: The cardiokymographic stress test. Circulation 61:579, 1980. Chaitman BR, Bristpw JD, Rahimtoola SH: Left ventricular wall motion assessed by using fixed external reference system. Circulation 48:1043, 1973. Meier P: Stratification in the design of a clinical trial. Controlled Clin Trials 1:355, 1981.