- Email: [email protected]
Diffusion-weighted Imaging in Ischemic Stroke
Diffusion-weighted Imaging in Ischemic Stroke: Effect of Display Method on Observers’ Diagnostic Performance1 Toshinori Hirai, MD, Makoto Sasaki, MD, Masayuki Maeda, MD, Masahiro Ida, MD, Shigehiko Katsuragawa, PhD, Masaharu Sakoh, MD, Koichi Takano, MD, Shoichi Arai, MD, Teruyuki Hirano, MD, Yutaka Kai, MD, Shingo Kakeda, MD, Ryuji Murakami, MD, Ryuji Ikeda, Hirofumi Fukuoka, MD, Akira Sasao, MD, Yasuyuki Yamashita, MD, for the Acute Stroke Imaging Standardization Group–Japan (ASIST-Japan) Investigators
Rationale and Objectives. When evaluating ischemic stroke on diffusion-weighted magnetic resonance imaging (DWI), the display method has not been investigated. The purpose of this study was to determine whether standardization of the display method for DWI affects observers’ diagnostic performance in detecting ischemic stroke on DWI. Materials and Methods. Twenty-six observers evaluated 40 DWI studies in 20 patients with acute (<6 hours) middle cerebral arterial strokes and 20 controls for the presence of hyperintense lesions in 10 areas using the Alberta Stroke Programme Early CT Score (ASPECTS) system and one area in the corona radiata using a modified version of the ASPECTS system (ASPECTS-DWI). The images were reviewed using a standardized display method (SDM) and a conventional display method (CDM). The reading time was recorded for each session. The observers’ performance was evaluated with receiver-operating characteristic analysis. Results. In all observers with ASPECTS-DWI scores of #8 points, the value of the mean average area under the receiveroperating characteristic curve was slightly higher for the SDM than the CDM, but the difference was not statistically significant. In the insular ribbon, diagnostic accuracy was significantly higher with the SDM than the CDM (P = .036). In the other locations, there were no significant differences. With the SDM, the mean reading time was reduced by 7.5 seconds (P = .024). Conclusion. The SDM improved diagnostic accuracy for the insular ribbon and shortened the reading time, although it did not improve observers’ performance with the ASPECTS-DWI system. Key Words. Diffusion-weighted imaging; ischemic stroke; display method. ª AUR, 2009
Acad Radiol 2009; 16:305–312 1
From the Departments of Diagnostic Radiology (T. Hirai, H.F., A.S., Y.Y.), Neurology (T. Hirano), Neurosurgery (Y.K.), and Radiation Oncology (R.M.), Graduate School of Medical Sciences, and Kumamoto University Hospital (R.I.), Kumamoto University, 1-1-1 Honjo, Kumamoto 860-8556, Japan; the Advanced Medical Research Center, Iwate Medical University, Uchimaru, Morioka, Japan (M. Sasaki); the Department of Radiology, Mie University School of Medicine, Tsu, Mie, Japan (M.M.); the Department of Radiology, Ebara Hospital, Tokyo, Japan (M.I.); the Department of Radiological Technology, Kumamoto University School of Health Sciences, Kumamoto, Japan (S. Katsuragawa); the Department of Rehabilitation, Hatsudai Rehabilitation Hospital, Tokyo, Japan (M. Sakoh); the Department of Radiology, Fukuoka University, School of Medicine, Fukuoka, Japan (K.T., S.A.); and the Department of Radiology, University of Occupational and Environmental Health, School of Medicine, Kitakyushu, Japan (S. Kakeda). This work was partly supported by a Grant for Standardization Projects from the Japanese Society for Magnetic Resonance in Medicine and by a Research Grant for Cardiovascular Diseases (17C-3) from the Ministry of Heath, Labour, and Welfare of Japan. Received July 11, 2008; accepted September 14, 2008. Address correspondence to: T. Hirai e-mail: [email protected]
ª AUR, 2009 doi:10.1016/j.acra.2008.09.012
With respect to the detection and localization of ischemic stroke, diffusion-weighted magnetic resonance imaging (DWI) yields good interrater homogeneity and better sensitivity and accuracy than computed tomography, even among raters with limited experience (1–3). In patients with ischemic strokes, tissue plasminogen activator improves outcomes and yields benefits, as long as administration is begun within 9 hours of the insult (4–6). Perfusion-DWI mismatch and clinical-DWI mismatch have been used as diagnostic criteria for thrombolytic therapy (5–9). In the evaluation of the extent of ischemic areas, the Alberta Stroke Programme Early CT Score (ASPECTS) system has been applied to DWI (7–9). The interpretation of the extent of acute ischemic lesions on DWI is crucial for estimating tissues at risk and for determining treatment. Because DWI display conditions such as the window width and level vary among subjects, operators, and magnetic
305
HIRAI ET AL
resonance scanners (10), there is a risk for missing acute ischemic lesions and misinterpreting normal brain regions as infarcted. To our knowledge, the possible effect of the display method on the diagnostic performance of raters evaluating DWI in patients with ischemic strokes has not been studied in detail. We developed a simple standardized method to set DWI display conditions to evaluate acute ischemic lesions (10). The hypothesis in our study was that the standardized display method (SDM) would affect diagnostic performance in detecting acute cerebral infarction. The purpose of our study was to verify this hypothesis using observers’ performance evaluation with receiver-operating characteristic (ROC) analysis.
MATERIALS AND METHODS Selection of Patients and Controls A total of 30 magnetic resonance imaging (MRI) studies on 30 patients who underwent MRI between January 2001 and July 2003 were collected from seven Japanese centers for this retrospective study of the effect of the DWI display method on the diagnosis of acute cerebral infarction. The inclusion criteria were acquisition of MRI scans within 6 hours of ischemic stroke onset; an age range of 40 to 85 years; the use of a 1.5-T MRI scanner featuring a head coil, echoplanar capability, and identical sequence parameters; a diagnosis of ischemic stroke based on focal neurologic deficits measurable on the National Institutes of Health Stroke Scale (11); high–signal intensity (SI) areas on DWI in only one middle cerebral artery (MCA) territory of the cerebral hemisphere; and the availability of high-quality images for interpretation and of follow-up computed tomographic or MRI scans. The institutional review boards of the participating centers approved this retrospective study; patient informed consent was waived. From the 30 collected studies, two neuroradiologists (T. Hirai, M. Sasaki) selected 20 DWI studies performed in 10 men and 10 women (age range, 54–82 years; mean age, 70.7) with acute infarctions in the MCA territory. Ten studies were excluded for insufficient image quality from motion artifacts, the presence of previous subacute to chronic infarctions, or the presence of nonspecific white matter lesions exhibiting T2 shine-through on DWI, and these studies were not included in the subsequent observer performance study. Of the 20 selected patients, 12 had undergone MRI within 3 hours of stroke onset. Using databases maintained at the seven centers, the two neuroradiologists also selected 20 DWI studies in 20 patients (9 men, 11 women; age range, 57–82 years; mean age, 70.2) without brain infarctions. These served as controls in the observer performance study.
306
Academic Radiology, Vol 16, No 3, March 2009
MRI All MRI studies were performed on 1.5-T MRI scanners (Magnetom Vision or Symphony, Siemens Medical Systems, Erlangen, Germany; Signa MR/i, GE Healthcare, Milwaukee, WI; Gyroscan Intera, Philips Medical Systems, Best, The Netherlands); quadrature detection head coils or multichannel head coils were used. DWI was performed using the single-shot spin-echo echo-planar imaging technique, with the following parameters: repetition time, 4000 to 8000 ms; echo time, 70 to 100 ms; b value, 1000 s/mm2; slice thickness, 5 to 7 mm; matrix size, 128 80 to 128; and field of view, 220 to 230 mm; 16 to 20 contiguous axial sections parallel to a line through the anterior and posterior commissure were obtained. The acquired data were collected in the Digital Imaging and Communications in Medicine format. Scoring System of DWI Abnormalities for the Observer Performance Study For the evaluation of early ischemic change on DWI, identified by hyperintense signals, the existence of lesions in the MCA territory was assessed in 11 locations: 10 locations according to the ASPECTS system (12), as well as the corona radiata (Fig 1). We used this modified version of the ASPECTS system (ASPECTS-DWI) because the corona radiata is often affected by ischemic stroke and because lesions in this area can be identified easily on DWI. Although the ASPECTS-DWI regions were idealized from two standardized axial sections, in the performance study, each reader reviewed the entire sequence of slices to determine the score. The MCA territory was allotted 11 points; DWI scans without the involvement of this area received ASPECTS-DWI scores of 11 points. A score of zero indicated ischemic involvement throughout the MCA territory. Using the ASPECTS-DWI system, focal ischemic areas in one MCA territory were selected by the two neuroradiologists who chose the DWI studies for inclusion in the observer performance study. Follow-up computed tomographic or MRI studies were used as the gold standard to determine final infarct extent. The number of lesions by location on the basis of the ASPECTS-DWI system in 20 patients with ischemic strokes is shown in Figure 2. A total of 75 focal hyperintense areas were identified on DWI. Because all the cases we collected did not have internal capsule infarctions, lesions in this location were not included in this study. The number of hyperintense lesions per patient ranged from 1 to 6 (mean, 3.75), resulting in ASPECTS-DWI scores ranging from 5 to 10 points (mean, 7.25 points). Display Methods Two display methods were used. In the conventional display method (CDM), the window and level settings were operator dependent (ie, each observer was able to change this
Academic Radiology, Vol 16, No 3, March 2009
EFFECT OF DISPLAY METHOD ON DWI
Figure 1. Modified Alberta Stroke Programme Early CT Score (ASPECTS) system applied to diffusion-weighted imaging (DWI). A hyperintense lesion (asterisk) is seen in the lentiform (L) and corona radiata (CR). Because the middle cerebral artery (MCA) territory was allotted 11 points on the basis of the ASPECTS-DWI system, the ASPECTSDWI score was 9 points in this case. C, caudate; I, insular ribbon; IC, internal capsule; M1, anterior portion of the MCA cortex; M2, MCA cortex lateral to the insular ribbon; M3, posterior MCA cortex; M4, M5, and M6, anterior, lateral, and posterior MCA territories immediately superior to M1, M2, and M3, rostral to the basal ganglia.
setting on the monitor according to preference). In the SDM, the window width and level settings were constant in all evaluations and could not be changed. Details of the SDM have been reported elsewhere (10). For the standardized display of DWI, the window width and level were as follows: and
window width ¼ SIb0 ; window level ¼ ðSIb0 Þ=2;
where SIb0 is the SI in the normal-appearing thalamus on b0 images (10). For the SDM, one neuroradiologist (T. Hirai) manually measured the SI of the normal-appearing thalamus on b0 images with a circular region of interest (ROI); on each scan, one 60-mm2 ROI was placed within the thalamus (Fig 3). The measured window width and level were applied to all DWI studies for the SDM.
Observer Performance Study A total of 26 board-certified observers comprising eight neuroradiologists (12–20 years of experience; mean, 15.5), eight general radiologists (9–30 years of experience; mean, 17.4), five neurologists (7–20 years of experience; mean, 12.6), and five neurosurgeons (18–26 years of experience; mean, 22.2) participated in the observer performance study. All were blinded to the consensual diagnosis of the two experienced neuroradiologists who selected the DWI studies.
Figure 2. The number of ischemic lesions by location on the basis of the Alberta Stroke Programme Early CT Score diffusionweighted imaging (DWI) system in 20 patients with ischemic strokes. A total of 75 focal hyperintense areas on DWI were included. Lesions in the internal capsule were not included in this study. M1, anterior portion of the MCA cortex; M2, MCA cortex lateral to the insular ribbon; M3, posterior MCA cortex; M4, M5, and M6, anterior, lateral, and posterior MCA territories immediately superior to M1, M2, and M3, rostral to the basal ganglia.
Before the test, the observers were informed that the purpose of the study was to evaluate their performance in detecting ischemic brain lesions on DWI displayed by two different methods, that the study included 20 patients with and 20 without acute cerebral infarctions in the MCA territory in one cerebral hemisphere, and that the evaluation included 22 areas (two pairs of 11 areas in one cerebral hemisphere) per case according to the ASPECTS-DWI system. Two sessions were conducted, one using the CDM and the other using the SDM. To assess the results of the
307
HIRAI ET AL
Academic Radiology, Vol 16, No 3, March 2009
Figure 3. Diffusion-weighted imaging display methods. (a) Diffusion-weighted image with a narrow window width shows hyperintensity in the cingulated gyrus and insula (arrowheads) as well as an infracted area (arrow). (b) In the standardized display method, the window width and level settings were determined by manually measuring the signal intensity of a circular region of interest within the normal-appearing thalamus (circle) on the b0 image. (c) Diffusion-weighted image with window width and level determined by the standardized display method shows hyperintensity in an infarcted area (arrow).
observer performance study, we used an independent test (13,14). To reduce the effects of learning, the interval between reading sessions was $2 months. During the first session, half of the observers (four neuroradiologists, four general radiologists, two neurologists, and three neurosurgeons) interpreted DWI studies using the CDM, and the other half interpreted studies using the SDM. Using computer randomization, images of 20 patients with and without acute infarctions were intermixed. All 40 cases were presented in the same randomized order to the observers. During the second session, the observers interpreted the images under conditions that were different from their first sessions. The 26 observers viewed the DWI studies on a monitor (RadiForce, R22-S; EIZO, Ishikawa, Japan) with a spatial resolution of 1600 1200. Before the observer performance test, each reader underwent a training session with two training cases not included in the subsequent observer performance test to become familiar with the monitor and the test procedure. The observers were allowed to adjust the image window level and width only in CDM sessions. In the SDM sessions, they viewed each image at preset window level and width. Observers were blinded to clinical information, and they recorded the presence or absence of ischemic lesions in 11 separate areas of each cerebral hemisphere (Fig 1). They then rated the confidence levels of their determinations with respect to three or more areas (an ASPECTSDWI score of #8 points). An ASPECTS of 7 points is the cutoff value to predict hemorrhagic complications (12). Because a total allotted area of the ASPECTS-DWI system was 1 point higher than that of ASPECTS system, we used a score of 8 points as the cutoff value for the ASPECTS-DWI system in our observer performance study. Each observer’s confidence level was recorded on a continuous rating scale using a line-marking method (15). In each series, the recorded
308
points were converted into a confidence level ranging from zero to 100. Each observer also recorded the reading time from first seeing the image to the marking of the confidence level. Statistical Analyses Observer performance was evaluated using ROC analysis (LABROC; C. Metz, University of Chicago, Chicago, IL) (16). The ROC curves for each observer displayed the truepositive fraction against the false-positive fraction at each confidence level. The area under the ROC curve (Az) was used to compare the observers’ performance in detecting an ASPECTS-DWI score of #8 points. To investigate the effect of the image display method on all observers and each specialty group, the significance of the difference between the Az values obtained with the CDM and the SDM was evaluated with Wilcoxon’s matched-pairs signed-ranks test. To assess the difference of diagnostic performance between specialty groups, the significance of the difference in the Az values was evaluated using the MannWhitney test. To evaluate the effects of the two image display methods at each brain location, we calculated and compared diagnostic accuracy at each location using the paired Student’s t test. The diagnostic accuracy at each brain location was defined as the number of true-positive and true-negative recordings per total number of tests. The number of falsepositive and false-negative recordings per session was also calculated. To evaluate the effects of the display methods on reading time, the reading times recorded for the two image display methods were compared; the statistical significance of differences was determined using the paired Student’s t test. In all analyses, P values < .05 were considered to indicate significant differences.
EFFECT OF DISPLAY METHOD ON DWI
Academic Radiology, Vol 16, No 3, March 2009
Table 1 Comparison of Mean Az Values Obtained with the CDM and the SDM Method
Neuroradiologists
General Radiologists
Neurologists
Neurosurgeons
All Observers
CDM SDM P value
0.996 0.005* 0.990 0.012y .226
0.961 0.047 0.979 0.036 .196
0.975 0.020* 0.966 0.027y .560
0.947 0.045 0.974 0.018 .385
0.971 0.037 0.979 0.026 .332
Az, area under the receiver-operating characteristic curve; CDM, conventional display method; SDM, standardized display method. Data are expressed as mean standard deviation. * P = .016 y P = .048.
RESULTS Diagnostic Performance With respect to the detection of a modified ASPECTS score of #8 points, we compared the mean Az values obtained with the CDM and the SDM (Table 1). In all observers, the mean Az value was slightly higher with the SDM than the CDM (0.979 0.026 vs 0.971 0.037); the difference was not statistically significant (P = .33) (Fig 4). There was no significant difference in the mean Az values obtained with the CDM and the SDM by readers grouped according to their specialties. On the other hand, neuroradiologists had significantly higher mean Az values than neurologists, irrespective of whether the images were displayed by the CDM or the SDM (P = .048 vs P = .016). In the other specialty groups, there was no significant difference in the mean Az values. Diagnostic accuracy by brain location for all observers is shown in Table 2. In the right insular ribbon areas, accuracy was significantly higher with the SDM than the CDM (P = .036). In the other locations, there was no significant difference. The number of false-positive assessments per session was highest for the right insular ribbon, followed by the left corona radiata. The number of false-negative assessments per session was highest for the right corona radiata, followed by the right MCA cortex lateral to the insular ribbon. Reading Times The mean reading times per assessment with the CDM and the SDM are shown in Table 3. For all observers, the mean reading time per case was reduced by 7.5 seconds when the SDM was used (P = .024). Among specialty groups, there was no significant difference in the reading time required with the two display methods. DISCUSSION The diagnostic accuracy of the insular ribbon was significantly increased when the images were displayed with the SDM. The highest number of false-positive assessments per
Figure 4. Comparison of mean areas under the receiver-operating characteristic curves (Az) between the standardized and conventional display methods. Although the mean Az value was slightly higher for the standardized method, there was no significant difference between them (P = .33).
case per session was recorded for the insular ribbon. On DWI, the brain cortices manifest physiologic regional signal variation; in healthy subjects, the insular and the cingulate cortices show higher SI than the other cerebral cortices (17). This phenomenon is probably due to T2 shine-through (18). In addition, DWI display conditions such as the window width and level vary among subjects, operators, and MRI scanners (10). Thus, physiologic DWI hyperintensity and DWI display conditions may have affected the result of the evaluation for the insular ribbon. Among the brain locations evaluated in our study, diagnostic accuracy was lowest for the corona radiata. Slight hyperintense areas are often seen in the corona radiata without lesions on DWI (Fig 1). The area in the corona radiata we evaluated in this study was located in the white matter lateral to the body of the lateral ventricle, which includes the
309
HIRAI ET AL
310 Table 2 Diagnostic Accuracy According to the Location of the Brain for All Observers Caudate Right Variable
Lentiform Left
Right
Internal Capsule Left
Right
Insular Ribbon
Left
Right
M1
Left
Right
M2 Left
Right
Left
CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM
No. of FP cases 0.62 per session No. of FN cases 0.31 per session Accuracy (%) 97.7
0.65
0.65
0.23
0.31
0.58
0.23
0.31
1.38
1.22
0.92
0.85
1.58
0.92
1.04
0.73
1.08
0.96
1.00
0.92
1.12
1.04
0.85
0.73
0.46
0.12
0
1.65
1.38
1.38
1.19
NA
NA
NA
NA
0.77
0.42
1.35
1.27
0.19
0.12
0.27
0.19
2.35
1.81
1.00
0.96
97.2
98.1
99.4
95.1
95.1
96.0
96.3
NA
NA
NA
NA
94.1* 96.6* 94.0
95.0
96.8
97.3
96.8
97.2
91.3
92.9
95.4
95.8
M3 Right
M4 Left
M5
Right
Left
Right
M6 Left
Right
Corona Radiata Left
Right
Left
CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM 0.92
0.50
0.65
0.77
0.42
0.46
0.49
0.69
0.65
0.46
0.15
1.19
1.35
0.92
0.50
1.42
1.08
1.57
1.35
1.50
0.04
0.08
1.38
1.38
0
0
2.23
1.85
0.69
0.50
0.58
0.35
0.62
0.73
2.46
2.27
1.69
1.65
93.9
98.7
98.2
94.6
95.5
98.8
98.3
92.7
93.8
97.1
98.4
95.6
95.8
96.2
96.9
90.3
91.6
91.3
92.5
CDM, conventional display method; FN, false negative; FP, false positive; M1, anterior portion of the middle cerebral artery (MCA) cortex; M2, MCA cortex lateral to insular ribbon; M3, posterior MCA cortex; M4, M5, and M6, anterior, lateral, and posterior MCA territories immediately superior to M1, M2, and M3, rostral to the basal ganglia; NA, not available; SDM, standardized display method. * P = .036.
Academic Radiology, Vol 16, No 3, March 2009
No. of FP cases 0.42 per session No. of FN cases 1.77 per session Accuracy (%) 94.5
EFFECT OF DISPLAY METHOD ON DWI
Academic Radiology, Vol 16, No 3, March 2009
Table 3 Comparison of Mean Reading Times per Case with the CDM and the SDM Method
Neuroradiologists
General Radiologists
Neurologists
Neurosurgeons
All Observers
CDM (seconds) SDM (seconds)
82.1 12 71.8 8
60.8 14 54.5 9
66.6 11 60.3 7
68.7 12 54.6 7
68.5 13* 61.0 10*
CDM, conventional display method; SDM, standardized display method. Data are expressed as mean standard deviation. * P = .024.
corticospinal tracts that are visualized as slight hyperintensity on T2-weighted images in healthy subjects (18). Because T2 affects the SI on DWI (19), the T2 shine-through effect may have resulted in the high number of false-positive assessments. For an ASPECTS-DWI score of #8 points, we analyzed the Az values separately in four specialty groups to examine whether the effect of the display method on observer performance depended on clinical experience. Our results indicated that the use of the SDM did not significantly affect observer performance in the four groups. Although we included patients with very early infarctions (12 of 20 had undergone MRI within 3 hours of stroke onset), the conspicuity of the lesions might have been high. In addition, in our series of 20 patients with early infarctions and 20 patients without infarctions, the number of hyperintense areas on DWI ranged from one to six; the mean number per case was 3.75. Because this number was slightly higher than the cutoff value, this difference may have affected our observer performance analysis. Neuroradiologists, who had highest mean Az value, did show higher performance with the CDM than with the SDM. This was probably because they had more experience with DWI and selected more appropriate window and level settings. The assessment of ischemic stroke on DWI requires knowledge regarding the anatomy of the brain and the etiology and classification of ischemic stroke, the diagnostic accuracy of DWI, and the presence of physiologic hyperintense areas and artifacts in the images. With both the CDM and the SDM, diagnostic performance was significantly higher for neuroradiologists than neurologists. The mean age of the neurologist group was lower than that of the other groups. Their experience may have affected the results. In addition, our neurologist observers may not have been familiar with methods that use computers in diagnosis. For all observers, the mean reading time was significantly shorter with the SDM than the CDM. Because the SDM does not require adjustments to the image window level and width on the monitor, the observers could concentrate on image evaluation. The presentation of all DWI studies at the same window level and width settings may have shortened the reading time. We also found that neuroradiologists took longer to read than other groups. Neuroradiologists may have
had more caution because of more experience reviewing DWI studies. Our study had some limitations. First, a bias in observer selection cannot be ruled out. However, because all 26 of our observers were board certified, irrespective of their specialties, and all exhibited high diagnostic performance, we think that the selection of observers was appropriate for our observer performance study. Second, apparent diffusion coefficient (ADC) maps were not used in this observer performance study. We performed only a semiquantitative evaluation of the diffusion-weighted images, for the following reasons: (1) diffusion-weighted images are widely used for evaluating ischemic brain lesions, (2) quantitative ROI measurement on ADC maps may be observer dependent, (3) ADC maps have relatively poorer tissue contrast than diffusion-weighted images, and (4) early brain infarctions of <6 hours’ onset may not always exhibit significant ADC changes (20,21). However, ADC maps may have been helpful in differentiating between early brain infarctions and T2 shine-through areas in our study. Third, a cutoff of 8 points was used with the ASPECTSDWI system in this observer performance test, although the cutoff value to predict hemorrhagic complications is 7 points in the ASPECTS system using computed tomography (12). Because a total allotted area of the ASPECTS-DWI system was 1 point higher than that of ASPECTS system, we used a score of 8 points as the cutoff value for the ASPECTS-DWI system. To date, the assessment system of DWI abnormalities for early ischemic stroke is not established. Because we often encounter DWI abnormalities in the corona radiata in patients with early ischemic lesions, we added this area to the 10 areas of the ASPECT system in our observer performance tests. Further investigations are needed to clarify the usefulness of this system. Fourth, there is a difference in the environment of an observer performance and the clinical setting. Our observers were informed that only one cerebral hemisphere was affected in the patients with strokes; this may have influenced their performance, because upon finding a lesion in one cerebral hemisphere, they would not proceed to inspect the other hemisphere. Therefore, potential false-positive assessments may have been excluded.
311
HIRAI ET AL
The SDM improved diagnostic accuracy for the insular ribbon and shortened the reading time, although it did not improve observer performance in evaluating DWI abnormalities using the ASPECTS-DWI system. Standardization of the DWI display method may be helpful in multicenter clinical trials of thrombolytic therapy. ACKNOWLEDGMENTS
We are grateful to Shoji Morishita, Yoshiko Hayashida, Ichiro Ikushima, Yoshinori Shigematsu, Mika Kitajima, Tomoko Okuda, Masayuki Yamura, Koichi Kawanaka, Yoshiharu Nakayama, Masanori Imuta, Takanori Negishi, Shinichiro Kusunoki, Shinya Shiraishi, Masaki Watanabe, Tomoo Hirahara, Sadahisa Okamoto, Akihiko Ueda, Motohiro Morioka, Keishi Makino, Hideo Takeshima, and Shigetoshi Yano for participating in our observer performance study. We gratefully thank the investigators and collaborators of Standardization Project #2 of the Japanese Society for Magnetic Resonance in Medicine and of the Acute Stroke Imaging Standardization Group Japan for their generous helps and suggestions for this study. REFERENCES 1. Lansberg MG, Albers GW, Beaulieu C, Marks MP. Comparison of diffusion-weighted MRI and CT in acute stroke. Neurology 2000; 54: 1557–1561. 2. Fiebach JB, Schellinger PD, Jansen O, et al. CT and diffusion-weighted MR imaging in randomized order: diffusion-weighted imaging results in higher accuracy and lower interrater variability in the diagnosis of hyperacute ischemic stroke. Stroke 2002; 33:2206–2210. 3. Saur D, Kucinski T, Grzyska U, et al. Sensitivity and interrater agreement of CT and diffusion-weighted MR imaging in hyperacute stroke. AJNR Am J Neuroradiol 2003; 24:878–885. 4. Hacke W, Donnan G, Fieschi C, et al. Association of outcome with early stroke treatment: pooled analysis of ATLANTIS, ECASS, and NINDS rt-PA stroke trials. Lancet 2004; 363:768–774. 5. Hacke W, Albers G, Al-Rawi Y, et al. The Desmoteplase in Acute Ischemic Stroke Trial (DIAS): a phase II MRI-based 9-hour window acute stroke thrombolysis trial with intravenous desmoteplase. Stroke 2005; 36: 66–73.
312
Academic Radiology, Vol 16, No 3, March 2009
6. Furlan AJ, Eyding D, Albers GW, et al. Dose Escalation of Desmoteplase for Acute Ischemic Stroke (DEDAS): evidence of safety and efficacy 3 to 9 hours after stroke onset. Stroke 2006; 37:1227–1231. 7. Da´valos A, Blanco M, Pedraza S, et al. The clinical-DWI mismatch. A new diagnostic approach to the brain tissue at risk of infarction. Neurology 2004; 62:2187–2192. 8. Prosser J, Butcher K, Allport L, et al. Clinical-diffusion mismatch predicts the putative penumbra with high specificity. Stroke 2005; 36:1700–1704. 9. Tei H, Uchiyama S, Usai T. Clinical-diffusion mismatch defined by NIHSS and ASPECTS in non-lacunar anterior circulation infarction. J Neurol 2007; 254:340–346. 10. Sasaki M, Ida M, Yamada K, Watanabe Y, Matsui M. Standardizing display conditions of diffusion-weighted images using concurrent b0 images: a multi-vendor multi-institutional study. Magn Reson Med Sci 2007; 6:133–137. 11. Brott T, Adams HP, Olinger CP, et al. Measurements of acute cerebral infarction: a clinical examination scale. Stroke 1989; 20:864–870. 12. Barber PA, Demchuk AM, Zhang J, et al; ASPECTS Study Group. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy. Lancet 2000; 355:1670–1674. 13. Kobayashi T, Xu XW, MacMahon H, Metz CE, Doi K. Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs. Radiology 1996; 199:843–848. 14. Kakeda S, Kamada K, Hatakeyama Y, et al. Effect of temporal subtraction technique on interpretation time and diagnostic accuracy of chest radiography. AJR Am J Roentgenol 2006; 187:1253–1259. 15. Hirai T, Korogi Y, Arimura H, et al. Intracranial aneurysms at MR angiography: effect of computer-aided diagnosis on radiologists’ detection performance. Radiology 2005; 237:605–610. 16. Metz CE, Herman BA, Shen JH. Maximum-likelihood estimation of receiver operating (ROC) characteristic curves from continuously-distributed data. Stat Med 1998; 17:1033–1053. 17. Asao C, Hirai T, Yoshimatsu S, et al. Human cerebral cortices: signal variation on diffusion-weighted MR imaging. Neuroradiology 2008; 50: 205–211. 18. Hofmann E, Ochs G, Pelzl A, Warmuth-Metz M. The corticospinal tract in amyotrophic lateral sclerosis: an MRI study. Neuroradiology 1998; 40: 71–75. 19. Schaefer PW, Grant PE, Gonzalez RG. Diffusion-weighted MR imaging of the brain. Radiology 2000; 217:331–345. 20. Desmond PM, Lovell AC, Rawlinson AA, et al. The value of apparent diffusion coefficient maps in early cerebral ischemia. AJNR Am J Neuroradiol 2001; 22:1260–1267. 21. Loh PS, Butcher KS, Parsons MW, et al. Apparent diffusion coefficient thresholds do not predict the response to acute stroke thrombolysis. Stroke 2005; 36:2626–2631.
Rationale and Objectives. When evaluating ischemic stroke on diffusion-weighted magnetic resonance imaging (DWI), the display method has not been investigated. The purpose of this study was to determine whether standardization of the display method for DWI affects observers’ diagnostic performance in detecting ischemic stroke on DWI. Materials and Methods. Twenty-six observers evaluated 40 DWI studies in 20 patients with acute (<6 hours) middle cerebral arterial strokes and 20 controls for the presence of hyperintense lesions in 10 areas using the Alberta Stroke Programme Early CT Score (ASPECTS) system and one area in the corona radiata using a modified version of the ASPECTS system (ASPECTS-DWI). The images were reviewed using a standardized display method (SDM) and a conventional display method (CDM). The reading time was recorded for each session. The observers’ performance was evaluated with receiver-operating characteristic analysis. Results. In all observers with ASPECTS-DWI scores of #8 points, the value of the mean average area under the receiveroperating characteristic curve was slightly higher for the SDM than the CDM, but the difference was not statistically significant. In the insular ribbon, diagnostic accuracy was significantly higher with the SDM than the CDM (P = .036). In the other locations, there were no significant differences. With the SDM, the mean reading time was reduced by 7.5 seconds (P = .024). Conclusion. The SDM improved diagnostic accuracy for the insular ribbon and shortened the reading time, although it did not improve observers’ performance with the ASPECTS-DWI system. Key Words. Diffusion-weighted imaging; ischemic stroke; display method. ª AUR, 2009
Acad Radiol 2009; 16:305–312 1
From the Departments of Diagnostic Radiology (T. Hirai, H.F., A.S., Y.Y.), Neurology (T. Hirano), Neurosurgery (Y.K.), and Radiation Oncology (R.M.), Graduate School of Medical Sciences, and Kumamoto University Hospital (R.I.), Kumamoto University, 1-1-1 Honjo, Kumamoto 860-8556, Japan; the Advanced Medical Research Center, Iwate Medical University, Uchimaru, Morioka, Japan (M. Sasaki); the Department of Radiology, Mie University School of Medicine, Tsu, Mie, Japan (M.M.); the Department of Radiology, Ebara Hospital, Tokyo, Japan (M.I.); the Department of Radiological Technology, Kumamoto University School of Health Sciences, Kumamoto, Japan (S. Katsuragawa); the Department of Rehabilitation, Hatsudai Rehabilitation Hospital, Tokyo, Japan (M. Sakoh); the Department of Radiology, Fukuoka University, School of Medicine, Fukuoka, Japan (K.T., S.A.); and the Department of Radiology, University of Occupational and Environmental Health, School of Medicine, Kitakyushu, Japan (S. Kakeda). This work was partly supported by a Grant for Standardization Projects from the Japanese Society for Magnetic Resonance in Medicine and by a Research Grant for Cardiovascular Diseases (17C-3) from the Ministry of Heath, Labour, and Welfare of Japan. Received July 11, 2008; accepted September 14, 2008. Address correspondence to: T. Hirai e-mail: [email protected]
ª AUR, 2009 doi:10.1016/j.acra.2008.09.012
With respect to the detection and localization of ischemic stroke, diffusion-weighted magnetic resonance imaging (DWI) yields good interrater homogeneity and better sensitivity and accuracy than computed tomography, even among raters with limited experience (1–3). In patients with ischemic strokes, tissue plasminogen activator improves outcomes and yields benefits, as long as administration is begun within 9 hours of the insult (4–6). Perfusion-DWI mismatch and clinical-DWI mismatch have been used as diagnostic criteria for thrombolytic therapy (5–9). In the evaluation of the extent of ischemic areas, the Alberta Stroke Programme Early CT Score (ASPECTS) system has been applied to DWI (7–9). The interpretation of the extent of acute ischemic lesions on DWI is crucial for estimating tissues at risk and for determining treatment. Because DWI display conditions such as the window width and level vary among subjects, operators, and magnetic
305
HIRAI ET AL
resonance scanners (10), there is a risk for missing acute ischemic lesions and misinterpreting normal brain regions as infarcted. To our knowledge, the possible effect of the display method on the diagnostic performance of raters evaluating DWI in patients with ischemic strokes has not been studied in detail. We developed a simple standardized method to set DWI display conditions to evaluate acute ischemic lesions (10). The hypothesis in our study was that the standardized display method (SDM) would affect diagnostic performance in detecting acute cerebral infarction. The purpose of our study was to verify this hypothesis using observers’ performance evaluation with receiver-operating characteristic (ROC) analysis.
MATERIALS AND METHODS Selection of Patients and Controls A total of 30 magnetic resonance imaging (MRI) studies on 30 patients who underwent MRI between January 2001 and July 2003 were collected from seven Japanese centers for this retrospective study of the effect of the DWI display method on the diagnosis of acute cerebral infarction. The inclusion criteria were acquisition of MRI scans within 6 hours of ischemic stroke onset; an age range of 40 to 85 years; the use of a 1.5-T MRI scanner featuring a head coil, echoplanar capability, and identical sequence parameters; a diagnosis of ischemic stroke based on focal neurologic deficits measurable on the National Institutes of Health Stroke Scale (11); high–signal intensity (SI) areas on DWI in only one middle cerebral artery (MCA) territory of the cerebral hemisphere; and the availability of high-quality images for interpretation and of follow-up computed tomographic or MRI scans. The institutional review boards of the participating centers approved this retrospective study; patient informed consent was waived. From the 30 collected studies, two neuroradiologists (T. Hirai, M. Sasaki) selected 20 DWI studies performed in 10 men and 10 women (age range, 54–82 years; mean age, 70.7) with acute infarctions in the MCA territory. Ten studies were excluded for insufficient image quality from motion artifacts, the presence of previous subacute to chronic infarctions, or the presence of nonspecific white matter lesions exhibiting T2 shine-through on DWI, and these studies were not included in the subsequent observer performance study. Of the 20 selected patients, 12 had undergone MRI within 3 hours of stroke onset. Using databases maintained at the seven centers, the two neuroradiologists also selected 20 DWI studies in 20 patients (9 men, 11 women; age range, 57–82 years; mean age, 70.2) without brain infarctions. These served as controls in the observer performance study.
306
Academic Radiology, Vol 16, No 3, March 2009
MRI All MRI studies were performed on 1.5-T MRI scanners (Magnetom Vision or Symphony, Siemens Medical Systems, Erlangen, Germany; Signa MR/i, GE Healthcare, Milwaukee, WI; Gyroscan Intera, Philips Medical Systems, Best, The Netherlands); quadrature detection head coils or multichannel head coils were used. DWI was performed using the single-shot spin-echo echo-planar imaging technique, with the following parameters: repetition time, 4000 to 8000 ms; echo time, 70 to 100 ms; b value, 1000 s/mm2; slice thickness, 5 to 7 mm; matrix size, 128 80 to 128; and field of view, 220 to 230 mm; 16 to 20 contiguous axial sections parallel to a line through the anterior and posterior commissure were obtained. The acquired data were collected in the Digital Imaging and Communications in Medicine format. Scoring System of DWI Abnormalities for the Observer Performance Study For the evaluation of early ischemic change on DWI, identified by hyperintense signals, the existence of lesions in the MCA territory was assessed in 11 locations: 10 locations according to the ASPECTS system (12), as well as the corona radiata (Fig 1). We used this modified version of the ASPECTS system (ASPECTS-DWI) because the corona radiata is often affected by ischemic stroke and because lesions in this area can be identified easily on DWI. Although the ASPECTS-DWI regions were idealized from two standardized axial sections, in the performance study, each reader reviewed the entire sequence of slices to determine the score. The MCA territory was allotted 11 points; DWI scans without the involvement of this area received ASPECTS-DWI scores of 11 points. A score of zero indicated ischemic involvement throughout the MCA territory. Using the ASPECTS-DWI system, focal ischemic areas in one MCA territory were selected by the two neuroradiologists who chose the DWI studies for inclusion in the observer performance study. Follow-up computed tomographic or MRI studies were used as the gold standard to determine final infarct extent. The number of lesions by location on the basis of the ASPECTS-DWI system in 20 patients with ischemic strokes is shown in Figure 2. A total of 75 focal hyperintense areas were identified on DWI. Because all the cases we collected did not have internal capsule infarctions, lesions in this location were not included in this study. The number of hyperintense lesions per patient ranged from 1 to 6 (mean, 3.75), resulting in ASPECTS-DWI scores ranging from 5 to 10 points (mean, 7.25 points). Display Methods Two display methods were used. In the conventional display method (CDM), the window and level settings were operator dependent (ie, each observer was able to change this
Academic Radiology, Vol 16, No 3, March 2009
EFFECT OF DISPLAY METHOD ON DWI
Figure 1. Modified Alberta Stroke Programme Early CT Score (ASPECTS) system applied to diffusion-weighted imaging (DWI). A hyperintense lesion (asterisk) is seen in the lentiform (L) and corona radiata (CR). Because the middle cerebral artery (MCA) territory was allotted 11 points on the basis of the ASPECTS-DWI system, the ASPECTSDWI score was 9 points in this case. C, caudate; I, insular ribbon; IC, internal capsule; M1, anterior portion of the MCA cortex; M2, MCA cortex lateral to the insular ribbon; M3, posterior MCA cortex; M4, M5, and M6, anterior, lateral, and posterior MCA territories immediately superior to M1, M2, and M3, rostral to the basal ganglia.
setting on the monitor according to preference). In the SDM, the window width and level settings were constant in all evaluations and could not be changed. Details of the SDM have been reported elsewhere (10). For the standardized display of DWI, the window width and level were as follows: and
window width ¼ SIb0 ; window level ¼ ðSIb0 Þ=2;
where SIb0 is the SI in the normal-appearing thalamus on b0 images (10). For the SDM, one neuroradiologist (T. Hirai) manually measured the SI of the normal-appearing thalamus on b0 images with a circular region of interest (ROI); on each scan, one 60-mm2 ROI was placed within the thalamus (Fig 3). The measured window width and level were applied to all DWI studies for the SDM.
Observer Performance Study A total of 26 board-certified observers comprising eight neuroradiologists (12–20 years of experience; mean, 15.5), eight general radiologists (9–30 years of experience; mean, 17.4), five neurologists (7–20 years of experience; mean, 12.6), and five neurosurgeons (18–26 years of experience; mean, 22.2) participated in the observer performance study. All were blinded to the consensual diagnosis of the two experienced neuroradiologists who selected the DWI studies.
Figure 2. The number of ischemic lesions by location on the basis of the Alberta Stroke Programme Early CT Score diffusionweighted imaging (DWI) system in 20 patients with ischemic strokes. A total of 75 focal hyperintense areas on DWI were included. Lesions in the internal capsule were not included in this study. M1, anterior portion of the MCA cortex; M2, MCA cortex lateral to the insular ribbon; M3, posterior MCA cortex; M4, M5, and M6, anterior, lateral, and posterior MCA territories immediately superior to M1, M2, and M3, rostral to the basal ganglia.
Before the test, the observers were informed that the purpose of the study was to evaluate their performance in detecting ischemic brain lesions on DWI displayed by two different methods, that the study included 20 patients with and 20 without acute cerebral infarctions in the MCA territory in one cerebral hemisphere, and that the evaluation included 22 areas (two pairs of 11 areas in one cerebral hemisphere) per case according to the ASPECTS-DWI system. Two sessions were conducted, one using the CDM and the other using the SDM. To assess the results of the
307
HIRAI ET AL
Academic Radiology, Vol 16, No 3, March 2009
Figure 3. Diffusion-weighted imaging display methods. (a) Diffusion-weighted image with a narrow window width shows hyperintensity in the cingulated gyrus and insula (arrowheads) as well as an infracted area (arrow). (b) In the standardized display method, the window width and level settings were determined by manually measuring the signal intensity of a circular region of interest within the normal-appearing thalamus (circle) on the b0 image. (c) Diffusion-weighted image with window width and level determined by the standardized display method shows hyperintensity in an infarcted area (arrow).
observer performance study, we used an independent test (13,14). To reduce the effects of learning, the interval between reading sessions was $2 months. During the first session, half of the observers (four neuroradiologists, four general radiologists, two neurologists, and three neurosurgeons) interpreted DWI studies using the CDM, and the other half interpreted studies using the SDM. Using computer randomization, images of 20 patients with and without acute infarctions were intermixed. All 40 cases were presented in the same randomized order to the observers. During the second session, the observers interpreted the images under conditions that were different from their first sessions. The 26 observers viewed the DWI studies on a monitor (RadiForce, R22-S; EIZO, Ishikawa, Japan) with a spatial resolution of 1600 1200. Before the observer performance test, each reader underwent a training session with two training cases not included in the subsequent observer performance test to become familiar with the monitor and the test procedure. The observers were allowed to adjust the image window level and width only in CDM sessions. In the SDM sessions, they viewed each image at preset window level and width. Observers were blinded to clinical information, and they recorded the presence or absence of ischemic lesions in 11 separate areas of each cerebral hemisphere (Fig 1). They then rated the confidence levels of their determinations with respect to three or more areas (an ASPECTSDWI score of #8 points). An ASPECTS of 7 points is the cutoff value to predict hemorrhagic complications (12). Because a total allotted area of the ASPECTS-DWI system was 1 point higher than that of ASPECTS system, we used a score of 8 points as the cutoff value for the ASPECTS-DWI system in our observer performance study. Each observer’s confidence level was recorded on a continuous rating scale using a line-marking method (15). In each series, the recorded
308
points were converted into a confidence level ranging from zero to 100. Each observer also recorded the reading time from first seeing the image to the marking of the confidence level. Statistical Analyses Observer performance was evaluated using ROC analysis (LABROC; C. Metz, University of Chicago, Chicago, IL) (16). The ROC curves for each observer displayed the truepositive fraction against the false-positive fraction at each confidence level. The area under the ROC curve (Az) was used to compare the observers’ performance in detecting an ASPECTS-DWI score of #8 points. To investigate the effect of the image display method on all observers and each specialty group, the significance of the difference between the Az values obtained with the CDM and the SDM was evaluated with Wilcoxon’s matched-pairs signed-ranks test. To assess the difference of diagnostic performance between specialty groups, the significance of the difference in the Az values was evaluated using the MannWhitney test. To evaluate the effects of the two image display methods at each brain location, we calculated and compared diagnostic accuracy at each location using the paired Student’s t test. The diagnostic accuracy at each brain location was defined as the number of true-positive and true-negative recordings per total number of tests. The number of falsepositive and false-negative recordings per session was also calculated. To evaluate the effects of the display methods on reading time, the reading times recorded for the two image display methods were compared; the statistical significance of differences was determined using the paired Student’s t test. In all analyses, P values < .05 were considered to indicate significant differences.
EFFECT OF DISPLAY METHOD ON DWI
Academic Radiology, Vol 16, No 3, March 2009
Table 1 Comparison of Mean Az Values Obtained with the CDM and the SDM Method
Neuroradiologists
General Radiologists
Neurologists
Neurosurgeons
All Observers
CDM SDM P value
0.996 0.005* 0.990 0.012y .226
0.961 0.047 0.979 0.036 .196
0.975 0.020* 0.966 0.027y .560
0.947 0.045 0.974 0.018 .385
0.971 0.037 0.979 0.026 .332
Az, area under the receiver-operating characteristic curve; CDM, conventional display method; SDM, standardized display method. Data are expressed as mean standard deviation. * P = .016 y P = .048.
RESULTS Diagnostic Performance With respect to the detection of a modified ASPECTS score of #8 points, we compared the mean Az values obtained with the CDM and the SDM (Table 1). In all observers, the mean Az value was slightly higher with the SDM than the CDM (0.979 0.026 vs 0.971 0.037); the difference was not statistically significant (P = .33) (Fig 4). There was no significant difference in the mean Az values obtained with the CDM and the SDM by readers grouped according to their specialties. On the other hand, neuroradiologists had significantly higher mean Az values than neurologists, irrespective of whether the images were displayed by the CDM or the SDM (P = .048 vs P = .016). In the other specialty groups, there was no significant difference in the mean Az values. Diagnostic accuracy by brain location for all observers is shown in Table 2. In the right insular ribbon areas, accuracy was significantly higher with the SDM than the CDM (P = .036). In the other locations, there was no significant difference. The number of false-positive assessments per session was highest for the right insular ribbon, followed by the left corona radiata. The number of false-negative assessments per session was highest for the right corona radiata, followed by the right MCA cortex lateral to the insular ribbon. Reading Times The mean reading times per assessment with the CDM and the SDM are shown in Table 3. For all observers, the mean reading time per case was reduced by 7.5 seconds when the SDM was used (P = .024). Among specialty groups, there was no significant difference in the reading time required with the two display methods. DISCUSSION The diagnostic accuracy of the insular ribbon was significantly increased when the images were displayed with the SDM. The highest number of false-positive assessments per
Figure 4. Comparison of mean areas under the receiver-operating characteristic curves (Az) between the standardized and conventional display methods. Although the mean Az value was slightly higher for the standardized method, there was no significant difference between them (P = .33).
case per session was recorded for the insular ribbon. On DWI, the brain cortices manifest physiologic regional signal variation; in healthy subjects, the insular and the cingulate cortices show higher SI than the other cerebral cortices (17). This phenomenon is probably due to T2 shine-through (18). In addition, DWI display conditions such as the window width and level vary among subjects, operators, and MRI scanners (10). Thus, physiologic DWI hyperintensity and DWI display conditions may have affected the result of the evaluation for the insular ribbon. Among the brain locations evaluated in our study, diagnostic accuracy was lowest for the corona radiata. Slight hyperintense areas are often seen in the corona radiata without lesions on DWI (Fig 1). The area in the corona radiata we evaluated in this study was located in the white matter lateral to the body of the lateral ventricle, which includes the
309
HIRAI ET AL
310 Table 2 Diagnostic Accuracy According to the Location of the Brain for All Observers Caudate Right Variable
Lentiform Left
Right
Internal Capsule Left
Right
Insular Ribbon
Left
Right
M1
Left
Right
M2 Left
Right
Left
CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM
No. of FP cases 0.62 per session No. of FN cases 0.31 per session Accuracy (%) 97.7
0.65
0.65
0.23
0.31
0.58
0.23
0.31
1.38
1.22
0.92
0.85
1.58
0.92
1.04
0.73
1.08
0.96
1.00
0.92
1.12
1.04
0.85
0.73
0.46
0.12
0
1.65
1.38
1.38
1.19
NA
NA
NA
NA
0.77
0.42
1.35
1.27
0.19
0.12
0.27
0.19
2.35
1.81
1.00
0.96
97.2
98.1
99.4
95.1
95.1
96.0
96.3
NA
NA
NA
NA
94.1* 96.6* 94.0
95.0
96.8
97.3
96.8
97.2
91.3
92.9
95.4
95.8
M3 Right
M4 Left
M5
Right
Left
Right
M6 Left
Right
Corona Radiata Left
Right
Left
CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM CDM SDM 0.92
0.50
0.65
0.77
0.42
0.46
0.49
0.69
0.65
0.46
0.15
1.19
1.35
0.92
0.50
1.42
1.08
1.57
1.35
1.50
0.04
0.08
1.38
1.38
0
0
2.23
1.85
0.69
0.50
0.58
0.35
0.62
0.73
2.46
2.27
1.69
1.65
93.9
98.7
98.2
94.6
95.5
98.8
98.3
92.7
93.8
97.1
98.4
95.6
95.8
96.2
96.9
90.3
91.6
91.3
92.5
CDM, conventional display method; FN, false negative; FP, false positive; M1, anterior portion of the middle cerebral artery (MCA) cortex; M2, MCA cortex lateral to insular ribbon; M3, posterior MCA cortex; M4, M5, and M6, anterior, lateral, and posterior MCA territories immediately superior to M1, M2, and M3, rostral to the basal ganglia; NA, not available; SDM, standardized display method. * P = .036.
Academic Radiology, Vol 16, No 3, March 2009
No. of FP cases 0.42 per session No. of FN cases 1.77 per session Accuracy (%) 94.5
EFFECT OF DISPLAY METHOD ON DWI
Academic Radiology, Vol 16, No 3, March 2009
Table 3 Comparison of Mean Reading Times per Case with the CDM and the SDM Method
Neuroradiologists
General Radiologists
Neurologists
Neurosurgeons
All Observers
CDM (seconds) SDM (seconds)
82.1 12 71.8 8
60.8 14 54.5 9
66.6 11 60.3 7
68.7 12 54.6 7
68.5 13* 61.0 10*
CDM, conventional display method; SDM, standardized display method. Data are expressed as mean standard deviation. * P = .024.
corticospinal tracts that are visualized as slight hyperintensity on T2-weighted images in healthy subjects (18). Because T2 affects the SI on DWI (19), the T2 shine-through effect may have resulted in the high number of false-positive assessments. For an ASPECTS-DWI score of #8 points, we analyzed the Az values separately in four specialty groups to examine whether the effect of the display method on observer performance depended on clinical experience. Our results indicated that the use of the SDM did not significantly affect observer performance in the four groups. Although we included patients with very early infarctions (12 of 20 had undergone MRI within 3 hours of stroke onset), the conspicuity of the lesions might have been high. In addition, in our series of 20 patients with early infarctions and 20 patients without infarctions, the number of hyperintense areas on DWI ranged from one to six; the mean number per case was 3.75. Because this number was slightly higher than the cutoff value, this difference may have affected our observer performance analysis. Neuroradiologists, who had highest mean Az value, did show higher performance with the CDM than with the SDM. This was probably because they had more experience with DWI and selected more appropriate window and level settings. The assessment of ischemic stroke on DWI requires knowledge regarding the anatomy of the brain and the etiology and classification of ischemic stroke, the diagnostic accuracy of DWI, and the presence of physiologic hyperintense areas and artifacts in the images. With both the CDM and the SDM, diagnostic performance was significantly higher for neuroradiologists than neurologists. The mean age of the neurologist group was lower than that of the other groups. Their experience may have affected the results. In addition, our neurologist observers may not have been familiar with methods that use computers in diagnosis. For all observers, the mean reading time was significantly shorter with the SDM than the CDM. Because the SDM does not require adjustments to the image window level and width on the monitor, the observers could concentrate on image evaluation. The presentation of all DWI studies at the same window level and width settings may have shortened the reading time. We also found that neuroradiologists took longer to read than other groups. Neuroradiologists may have
had more caution because of more experience reviewing DWI studies. Our study had some limitations. First, a bias in observer selection cannot be ruled out. However, because all 26 of our observers were board certified, irrespective of their specialties, and all exhibited high diagnostic performance, we think that the selection of observers was appropriate for our observer performance study. Second, apparent diffusion coefficient (ADC) maps were not used in this observer performance study. We performed only a semiquantitative evaluation of the diffusion-weighted images, for the following reasons: (1) diffusion-weighted images are widely used for evaluating ischemic brain lesions, (2) quantitative ROI measurement on ADC maps may be observer dependent, (3) ADC maps have relatively poorer tissue contrast than diffusion-weighted images, and (4) early brain infarctions of <6 hours’ onset may not always exhibit significant ADC changes (20,21). However, ADC maps may have been helpful in differentiating between early brain infarctions and T2 shine-through areas in our study. Third, a cutoff of 8 points was used with the ASPECTSDWI system in this observer performance test, although the cutoff value to predict hemorrhagic complications is 7 points in the ASPECTS system using computed tomography (12). Because a total allotted area of the ASPECTS-DWI system was 1 point higher than that of ASPECTS system, we used a score of 8 points as the cutoff value for the ASPECTS-DWI system. To date, the assessment system of DWI abnormalities for early ischemic stroke is not established. Because we often encounter DWI abnormalities in the corona radiata in patients with early ischemic lesions, we added this area to the 10 areas of the ASPECT system in our observer performance tests. Further investigations are needed to clarify the usefulness of this system. Fourth, there is a difference in the environment of an observer performance and the clinical setting. Our observers were informed that only one cerebral hemisphere was affected in the patients with strokes; this may have influenced their performance, because upon finding a lesion in one cerebral hemisphere, they would not proceed to inspect the other hemisphere. Therefore, potential false-positive assessments may have been excluded.
311
HIRAI ET AL
The SDM improved diagnostic accuracy for the insular ribbon and shortened the reading time, although it did not improve observer performance in evaluating DWI abnormalities using the ASPECTS-DWI system. Standardization of the DWI display method may be helpful in multicenter clinical trials of thrombolytic therapy. ACKNOWLEDGMENTS
We are grateful to Shoji Morishita, Yoshiko Hayashida, Ichiro Ikushima, Yoshinori Shigematsu, Mika Kitajima, Tomoko Okuda, Masayuki Yamura, Koichi Kawanaka, Yoshiharu Nakayama, Masanori Imuta, Takanori Negishi, Shinichiro Kusunoki, Shinya Shiraishi, Masaki Watanabe, Tomoo Hirahara, Sadahisa Okamoto, Akihiko Ueda, Motohiro Morioka, Keishi Makino, Hideo Takeshima, and Shigetoshi Yano for participating in our observer performance study. We gratefully thank the investigators and collaborators of Standardization Project #2 of the Japanese Society for Magnetic Resonance in Medicine and of the Acute Stroke Imaging Standardization Group Japan for their generous helps and suggestions for this study. REFERENCES 1. Lansberg MG, Albers GW, Beaulieu C, Marks MP. Comparison of diffusion-weighted MRI and CT in acute stroke. Neurology 2000; 54: 1557–1561. 2. Fiebach JB, Schellinger PD, Jansen O, et al. CT and diffusion-weighted MR imaging in randomized order: diffusion-weighted imaging results in higher accuracy and lower interrater variability in the diagnosis of hyperacute ischemic stroke. Stroke 2002; 33:2206–2210. 3. Saur D, Kucinski T, Grzyska U, et al. Sensitivity and interrater agreement of CT and diffusion-weighted MR imaging in hyperacute stroke. AJNR Am J Neuroradiol 2003; 24:878–885. 4. Hacke W, Donnan G, Fieschi C, et al. Association of outcome with early stroke treatment: pooled analysis of ATLANTIS, ECASS, and NINDS rt-PA stroke trials. Lancet 2004; 363:768–774. 5. Hacke W, Albers G, Al-Rawi Y, et al. The Desmoteplase in Acute Ischemic Stroke Trial (DIAS): a phase II MRI-based 9-hour window acute stroke thrombolysis trial with intravenous desmoteplase. Stroke 2005; 36: 66–73.
312
Academic Radiology, Vol 16, No 3, March 2009
6. Furlan AJ, Eyding D, Albers GW, et al. Dose Escalation of Desmoteplase for Acute Ischemic Stroke (DEDAS): evidence of safety and efficacy 3 to 9 hours after stroke onset. Stroke 2006; 37:1227–1231. 7. Da´valos A, Blanco M, Pedraza S, et al. The clinical-DWI mismatch. A new diagnostic approach to the brain tissue at risk of infarction. Neurology 2004; 62:2187–2192. 8. Prosser J, Butcher K, Allport L, et al. Clinical-diffusion mismatch predicts the putative penumbra with high specificity. Stroke 2005; 36:1700–1704. 9. Tei H, Uchiyama S, Usai T. Clinical-diffusion mismatch defined by NIHSS and ASPECTS in non-lacunar anterior circulation infarction. J Neurol 2007; 254:340–346. 10. Sasaki M, Ida M, Yamada K, Watanabe Y, Matsui M. Standardizing display conditions of diffusion-weighted images using concurrent b0 images: a multi-vendor multi-institutional study. Magn Reson Med Sci 2007; 6:133–137. 11. Brott T, Adams HP, Olinger CP, et al. Measurements of acute cerebral infarction: a clinical examination scale. Stroke 1989; 20:864–870. 12. Barber PA, Demchuk AM, Zhang J, et al; ASPECTS Study Group. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy. Lancet 2000; 355:1670–1674. 13. Kobayashi T, Xu XW, MacMahon H, Metz CE, Doi K. Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs. Radiology 1996; 199:843–848. 14. Kakeda S, Kamada K, Hatakeyama Y, et al. Effect of temporal subtraction technique on interpretation time and diagnostic accuracy of chest radiography. AJR Am J Roentgenol 2006; 187:1253–1259. 15. Hirai T, Korogi Y, Arimura H, et al. Intracranial aneurysms at MR angiography: effect of computer-aided diagnosis on radiologists’ detection performance. Radiology 2005; 237:605–610. 16. Metz CE, Herman BA, Shen JH. Maximum-likelihood estimation of receiver operating (ROC) characteristic curves from continuously-distributed data. Stat Med 1998; 17:1033–1053. 17. Asao C, Hirai T, Yoshimatsu S, et al. Human cerebral cortices: signal variation on diffusion-weighted MR imaging. Neuroradiology 2008; 50: 205–211. 18. Hofmann E, Ochs G, Pelzl A, Warmuth-Metz M. The corticospinal tract in amyotrophic lateral sclerosis: an MRI study. Neuroradiology 1998; 40: 71–75. 19. Schaefer PW, Grant PE, Gonzalez RG. Diffusion-weighted MR imaging of the brain. Radiology 2000; 217:331–345. 20. Desmond PM, Lovell AC, Rawlinson AA, et al. The value of apparent diffusion coefficient maps in early cerebral ischemia. AJNR Am J Neuroradiol 2001; 22:1260–1267. 21. Loh PS, Butcher KS, Parsons MW, et al. Apparent diffusion coefficient thresholds do not predict the response to acute stroke thrombolysis. Stroke 2005; 36:2626–2631.