- Email: [email protected]
Evaluation of Computer-aided Diagnosis (CAD) Software for the Detection of Lung Nodules on Multidetector Row Computed Tomography (MDCT)
Evaluation of Computer-aided Diagnosis (CAD) Software for the Detection of Lung Nodules on Multidetector Row Computed Tomography (MDCT): JAFROC Study for the Improvement in Radiologists’ Diagnostic Accuracy1 Tomohiro Hirose, MD, Norihisa Nitta, MD, PhD, Junji Shiraishi, PhD, Yukihiro Nagatani, MD Masashi Takahashi, MD, Kiyoshi Murata, MD, PhD
Rationale and Objectives. The aim of this study was to evaluate the usefulness of computer-aided diagnosis (CAD) software for the detection of lung nodules on multidetector-row computed tomography (MDCT) in terms of improvement in radiologists’ diagnostic accuracy in detecting lung nodules, using jackknife free-response receiver-operating characteristic (JAFROC) analysis. Materials and Methods. Twenty-one patients (6 without and 15 with lung nodules) were selected randomly from 120 consecutive thoracic computed tomographic examinations. The gold standard for the presence or absence of nodules in the observer study was determined by consensus of two radiologists. Six expert radiologists participated in a free-response receiver operating characteristic study for the detection of lung nodules on MDCT, in which cases were interpreted first without and then with the output of CAD software. Radiologists were asked to indicate the locations of lung nodule candidates on the monitor with their confidence ratings for the presence of lung nodules. Results. The performance of the CAD software indicated that the sensitivity in detecting lung nodules was 71.4%, with 0.95 false-positive results per case. When radiologists used the CAD software, the average sensitivity improved from 39.5% to 81.0%, with an increase in the average number of false-positive results from 0.14 to 0.89 per case. The average figure-of-merit values for the six radiologists were 0.390 without and 0.845 with the output of the CAD software, and there was a statistically significant difference (P ⬍ .0001) using the JAFROC analysis. Conclusion. The CAD software for the detection of lung nodules on MDCT has the potential to assist radiologists by increasing their accuracy. Key Words. Computer-aided diagnosis; CAD; lung nodule; jackknife free-response receiver operating characteristic analysis; JAFROC; multidetector-row computed tomography; MDCT. ©
2008 AUR. Published by Elsevier Inc. All rights reserved.
Acad Radiol 2008; 15:1505–1512 1
From the Department of Radiology, Shiga University of Medical Science, Seta-Tsukinowa-cho, Otsu-shi, Shiga 520-2192, Japan (T.H., N.N., Y.N., M.T., K.M.); and the Department of Radiology, University of Chicago, Chicago, IL (J.S.). Received May 16, 2008; accepted June 12, 2008. Address correspondence to: T.H. e-mail: [email protected]
©
2008 AUR. Published by Elsevier Inc. All rights reserved. doi:10.1016/j.acra.2008.06.009
Because some evidence suggests that the early detection of lung cancer may allow timely therapeutic intervention and thus favorable prognoses for patients (1–5), lung cancer screening using low-dose computed tomography has been proposed (6 – 8). For example, in a 2006 report by the International Early Lung Cancer Action Program (9), computed tomographic (CT) lung cancer screening resulted in diagnoses of lung cancer in 484 of 31,567 participants. Of these 484 participants, 412 (85%) had clini-
1505
HIROSE ET AL
cal stage I lung cancer, and the estimated 10-year survival rate was 88% in this subgroup. Among the 484 participants, 302 had clinical stage I cancer and underwent surgical resection within 1 month after diagnosis. Their survival rate was 92%. The development of multidetector-row computed tomography (MDCT) has made it easier to cover the whole lung with thin-section images. In addition, it has become possible to obtain isotropic voxel data, which can be useful in the observation of lesions and/or vessels from multiple directions. As a result, more than 300 images with thin-section thickness in millimeters per thoracic CT examination are reconstructed for radiologists’ routine work. Although computed tomography is very sensitive in detecting small, noncalcified nodules (8), the interpretation of a large number of CT images is time-consuming for radiologists. Therefore, a number of investigators have been developing computer-aided diagnosis (CAD) methods for the detection of lung nodules on CT images (10 – 16). In addition, a number of observer performance studies have demonstrated that CAD is expected to make a significant contribution to the improvement of radiologists’ diagnostic accuracy (17,18). In these respects, CAD could be considered markedly useful in assisting radiologists in the detection of small lung nodules. In clinical practice, when a massive number of images are interpreted, the benefit of CAD could be better than expected. Moreover, CAD is expected to reduce the variability in detecting lesions due to differences in the experience of radiologists in image interpretation, to maintain a consistently high level of diagnostic accuracy, and to shorten image interpretation time. In this study, we evaluated the usefulness of CAD software in terms of improvement in radiologists’ diagnostic accuracy in detecting lung nodules on MDCT, using jackknife free-response receiver operating characteristic (JAFROC) analysis (19,20). MATERIALS AND METHODS Our institutional review board approved the use of this database and the participation of radiologists in this observer performance study. Image Database A total of 120 consecutive thoracic CT studies were examined at our hospital in January 2006. The true presence of nodules was diagnosed in 80 of these cases. The remaining 40 were diagnosed as nodule-absent cases.
1506
Academic Radiology, Vol 15, No 12, December 2008
To determine the gold standard for the existence of lung nodules, two expert radiologists (T.H. and N.N., with 7 and 18 years of experience, respectively) individually diagnosed all cases and recorded nodule locations initially and then reached consensus or agreement by taking into account all image information, such as coronal and sagittal views and multiplanar reconstruction (MPR) images in an axial view provided by the CAD software. Inconsistencies and the presence or absence of nodules were resolved by discussion. Note that the radiologists used the output of the CAD software only retrospectively for confirmation of the gold standard results. The clinical significance and the question of whether nodules were benign or malignant were not examined in this study. Therefore, in the retrospective selection of the study patients, we did not use any specific diagnosis or pathologic condition of pulmonary nodules. The nodules could be associated with metastasis, primary lung cancer, or granuloma. Both calcified and noncalcified nodules were included. No cavitary, ground-glass, or subsolid nodules were present in our patient population. Patients in whom CT studies showed substantial parenchymal or pleural diseases, such as consolidation, fibrosis, or pleural effusion, were excluded from our study population. From the 80 patients with lung nodules, 15 and three patients with lung nodules were selected randomly as test and training cases in the observer study, respectively. From the 40 nodule-absent patients, six were selected randomly as test cases. No nodule-absent patient was selected as a training case. The 21 test cases with and without nodules used in this study included eight male and 13 female patients (age range, 11 to 77 years; average, 53.7). A total of 49 gold standard lung nodules were included in 15 test nodule-present cases. There were one to eight nodules found in these cases, ranging in diameter from 1.5 to 15 mm, with an average diameter of 4.5 mm. Twenty-five of the 49 nodules (51.0%) were 4 mm or larger in diameter. Figure 1 shows the distribution of the 49 nodules according to the range of their diameters. There were 10 to 12 nodules each in the three cases used in the training session. In terms of the anatomic locations of the 49 lung nodules, 17 were located in the upper lobe, 24 in the middle lobe, and eight in the lower lobe. We used two multidetector-row CT systems with 16 multidetectors (Sensation Cardiac 16, Siemens Medical Systems, Erlangen, Germany; and Aquilion, Toshiba Medical, Nasu, Japan) to obtain images of the whole lung. Technical parameters for the Siemens system were as follows: rotation time, 0.37 seconds; collimation, 0.75
Academic Radiology, Vol 15, No 12, December 2008
CAD SOFTWARE FOR LUNG NODULE DETECTION
Figure 1. The distribution of the 49 nodules according to the nodule diameter ranged from 1.5 to 15.0 mm, with an average diameter of 4.5 mm. Twenty-five of 49 nodules were 4 mm or larger in diameter.
mm; tube voltage, 120 kV; tube current, 100 effective mA (223 mA); and feed rotation, 10 mm (beam pitch, 0.833). Technical parameters for the Toshiba system were as follows: rotation time, 0.5 seconds; collimation, 1.0 mm; tube voltage, 120 kV; tube current, 280 mA; and helical pitch, 10.0 (beam pitch, 1.28). A reconstruction formula targeted for lungs was used. Reconstruction kernels used in this study were B70f for the Siemens system and FC52 for the Toshiba system. All images were reconstructed with a section thickness of 1 mm and a slice interval of 1 mm. The image matrix size was 512 ⫻ 512 pixels. CAD Software ZIOCAD LE version 1.15 (Ziosoft, Inc., Tokyo, Japan) was used. Both solid and ground-glass opacity nodules can be detected by this software. The characteristics of this CAD system are as follows: (1) technology for calculating lesions in a 3-dimensional manner in 3-dimensional space; (2) high-speed technology focusing on calculations in situ; (3) technology for automatically detecting lungs, the thoracic wall, and the heart; and (4) ground-glass opacity segmentation technology focusing on the CT value distribution. The CAD method for nodule detection consisted of four steps: (1) filtration for low-frequency bands to decrease variations in the calculation results using a standard reconstruction kernel and a high-frequency intensified reconstruction kernel; (2) lung segmentation by limit-
ing the calculation area to calculate more quickly, to decrease the number of false-positive results, and to segment nodules attached to the thoracic wall; (3) the identification of initial candidates using a local maximum CT value and the identification of the relative distance between initial candidates; and (4) calculation of the point concentration for each initial candidate using a proprietary evaluation formula with a combination of geometric and statistical methods to calculate nodularity, which is used to determine the final candidates. Observer Study Six expert radiologists (Ryutaro Takazakura, MD, Shinichi Ohta, MD, Akinaga Sonoda, MD, Ayumi Seko, MD, Y.N., and M.T.) with 8 to 23 years of experience participated in an observer study for diagnosing lung nodules on MDCT without and with the output of the CAD software. Before the observer study, the radiologists were given the following instructions: (1) your performance in detecting lung nodules, first without and then with CAD output, is to be evaluated; (2) the role of the CAD output is that of a “second opinion”; (3) 21 CT studies (with 1-mm-thick sections) that did or did not show lung nodules are included in this observer study; (4) the observer in this study will be blinded to the number of patients with lung nodules and the performance level of the CAD software; and (5) click on the screen using the mouse to locate the most likely position and indicate on a bar your
1507
HIROSE ET AL
Academic Radiology, Vol 15, No 12, December 2008
Figure 2. Display of images without display of computer-aided diagnosis results. An axial image is shown at the center of the monitor, and slices can be moved in any direction, from head to toe, by rotation of the mouse wheel. Coronal (upper left), sagittal (middle left), and multiplanar reconstruction (lower left) images are displayed by moving the cross line on the axial image at the center of the monitor and left-clicking the mouse. The location corresponding to the respective site on the monitor image is displayed by left-clicking of the mouse or moving the cross line. Furthermore, candidate nodules detected by a radiologist can be displayed in the volume-rendering (upper right) and maximum-intensity projection (middle left) images and saved to the list on the right side of the monitor (lower right).
confidence level regarding the presence (or absence) of a lung nodule. For a training session before the test, three training cases, each of which had 10 to 12 nodules, were provided to the radiologists for learning how to operate the cine mode interface and how to take into account computer data for making their decisions. The radiologists began by diagnosing cases without the CAD software (Fig 2) and then rediagnosed them using the results provided by the CAD software (Fig 3). They carried out their interpretations repeatedly by following routine diagnostic procedures. For each nodule, the location was stored in the software, and the radiologist’s confidence for the pres-
1508
ence of lung nodules was recorded using a rating bar scale (7 cm in width), on which the right end corresponded to “absolute presence of a lung nodule” and the left end corresponded to “absolute absence of a lung nodule.” As shown in Figures 2 and 3, the software used in this study allowed the arbitrary selection of section thickness. In addition, it was possible to observe coronal and sagittal views and arbitrary slice sections as well as axial views in MPR images. During the interpretation of CT images with CAD software, the radiologists were allowed to make full use of functions, such as the 3-dimensional adjacent rendering mode and MPR for diagnosis. The win-
Academic Radiology, Vol 15, No 12, December 2008
CAD SOFTWARE FOR LUNG NODULE DETECTION
Figure 3. Display of images with computer-aided diagnosis (CAD) results. In the list at the bottom of the monitor screen, candidate nodules identified by CAD and nodules detected by radiologists are displayed. The former and latter nodules are indicated as red and green circles, respectively, on the axial view (upper left) by selection of items in the list. The same location can be displayed on the axial (upper center), maximum-intensity projection (upper right), coronal (middle left), and sagittal (lower center) images without displaying CAD results. Arbitrary locations in corresponding slices can be displayed by double-clicking or moving the cross line.
dow level was set at ⫺650 hu, and the window width was set at 1500 HU. JAFROC analysis (19) was used for the evaluation of the radiologists’ performance in the detection of lung nodules without and with the output of the CAD software. JAFROC analysis has been proposed for estimating statistically significant differences between modalities when location issues are relevant. JAFROC analysis is based on a free-response receiver operating characteristic (FROC) paradigm and accounts for reader variation (19,21). Conventional ROC analysis is of limited value for this kind of application because only one signal can be used per case, and the location of the signal cannot be taken into account in the evaluation. In contrast, FROC
analysis allows one to evaluate the performance of radiologists in diagnosing medical images using multiple responses, each with information on the confidence level and location (22). For statistical testing of differences between radiologists’ performance without and with the output of the CAD software, we applied JAFROC analysis with method 2 of Chakraborty and Berbaum (19) to estimate figure-of-merit (FOM) values (the analogue of the area under the ROC curve) for each reader. Chakraborty and Berbaum (19) proposed two implementations (methods 1 and 2) of JAFROC analysis in their study; we used method 2, which showed better properties in that study. In this statistical procedure, we calculated a sequence of FOM values for the JAFROC analysis
1509
HIROSE ET AL
and then built a pseudovalue matrix with these FOM values for multireaders and multicases according to the equations shown in the original investigation by Chakraborty and Berbaum (19). Finally, the FOM pseudovalue matrix was analyzed using analysis of variance techniques for estimating statistically significant differences between the two data sets obtained without and with the output of the CAD software. We also obtained sensitivities and the number of falsepositive results for all of the radiologists to compare their performance in the detection of lung nodules without and with the output of the CAD software.
RESULTS The CAD software correctly identified 35 of the 49 nodules of the gold standard, with 20 false-positive results for 21 cases. Therefore, the sensitivity of the CAD software for the cases used in this study was 71.4%, with 0.95 false-positive results per case. Among 14 lung nodules (mean diameter, 4.5 mm; range, 2.3–12.0) overlooked by the CAD software, nine were small nodules attached to the chest wall, and five were in contact with the trachea and/or blood vessels. Four nodules in one case, two nodules in three cases, and one nodule in four cases were overlooked. Among 25 nodules 4 mm or larger in diameter, the CAD software correctly identified 20 (80.0%). Thus, the CAD software identified 15 of the 24 (62.5%) nodules smaller than 4 mm in diameter. Of 20 no-nodule findings incorrectly identified by the CAD software, 16 were bronchial bifurcations or bifurcations of blood vessels, two were inflammatory scars in the lung apex, and one was interlobar pleural thickening. The remaining false-positive finding was located outside the lung. Using the CAD software, all radiologists remarkably improved their sensitivity for the detection of lung nodules from an average sensitivity of 39.5% to 81.0%, as shown in Table 1. However, the average number of falsepositive results increased from 0.14 to 0.89 per case. Eighteen of the 20 no-nodule findings (90.0%) were incorrectly identified by the CAD software and were also identified incorrectly as lung nodules by more than four of the six readers using the CAD software. Although the average rating score (0.405) for these 18 false-positive findings was relatively low compared to that for the actual lung nodules (0.863), the radiologists were likely to adopt the output of the CAD software.
1510
Academic Radiology, Vol 15, No 12, December 2008
Table 1 FOM Values Obtained from JAFROC Analysis (Method 2) without and with Output of the CAD Software for the Six Radiologists Without CAD FP Results/ Reader Sensitivity Case 1 2 3 4 5 6 Average
24.5% 30.6% 59.2% 40.8% 38.8% 42.9% 39.5%
0.10 0.14 0.24 0.19 0.05 0.14 0.14
With CAD
FOM 0.236 0.312 0.563 0.433 0.364 0.434 0.390*
FP Results/ Sensitivity Case 75.5% 75.5% 85.7% 79.6% 85.7% 83.7% 81.0%
0.90 1.00 1.10 0.52 0.95 0.86 0.89
FOM 0.808 0.792 0.870 0.819 0.881 0.901 0.845*
CAD, computer-aided diagnosis; FOM, figure-of-merit; FP, false-positive; JAFROC, jackknife free-response receiver operating characteristic. *Statistically significant difference (P ⬍ .0001) between average FOM values.
JAFROC analysis revealed that the CAD software improved the FOM values for all readers. There was a statistically significant difference (P ⬍ .0001) when the average FOM of the six radiologists with and without the CAD software was compared by taking into account reader and case variations (Table 1).
DISCUSSION It is widely known that computed tomography is superior to chest x-rays for the detection of small, noncalcified nodules (8). However, for radiologists to detect minute lung nodules on a number of CT slice images requires time and manpower. CAD software, as the “second radiologist,” has been developed for enhancing the capability of radiologists to detect lung nodules. A variety of CAD systems for the detection of lung nodules have been developed. Single-detector-row CT images were used for diagnosis in previously developed CAD systems, but recently MDCT has been used increasingly for evaluation. The CAD software used in this study was designed for multidetector-row CT images. Previously, it was common to calculate the characteristic of each nodule on the basis of 2-dimensional CT slice images because of the limitations of the CT equipment. However, with technological advances in computed tomography, it has become possible to obtain 3-dimensional volume data with a markedly smaller slice pitch. The algorithm of the CAD software
Academic Radiology, Vol 15, No 12, December 2008
CAD SOFTWARE FOR LUNG NODULE DETECTION
Figure 4. Two examples of lung nodules (arrows), which were correctly identified by 5 of the 6 radiologists as lung nodules but were not identified by the computed-aided diagnosis software.
used in this study used 3-dimensional volume data for calculating the characteristics of lung nodule candidates. Thus, more accurate recognition of nodule shapes could be explored by the calculation of CT volume data in a 3-dimensional manner. It has been reported that some nodule detection CAD software for multidetector-row CT images is now commercially available. Yuan et al (23) reported a sensitivity of 72.6% for Image Checker CT (R2 Technologies, Sunnyvale, CA). Das et al (24) also used Image Checker CT and reported a sensitivity of 73%. They also used Nodule Enhanced Viewing (Siemens Medical Systems) and achieved a sensitivity of 75% (24). Although the database used in our study was different and included a limited number of cases, there were differences in the sensitivity between the CAD software used in our study (71.4%) and their CAD software. In addition, it should be noted that the number of false-positive results per case (0.95) in the CAD software used in our study was very small compared with that in Yuan et al’s (23) study (3.19) and Das et al’s (24) study (six for Image Checker CT and eight for Nodule Enhanced Viewing). Bae et al (25) reported that the overall sensitivity for nodule detection with a CAD program was 95.1% (156 of 164 nodules). The sensitivity according to the nodule diameter was 91.2% (52 of 57 nodules) for nodules 3 to 5 mm and 97.2% (104 of 107 nodules) for nodules 5 mm and larger. The number of false-positive detections per patient was 6.9 for structures 3 mm and larger and 4.0 for structures 5 mm and larger. The CAD software used by Bae et al (25) had high sensitivity; however, their number of false-
positive results would not be recommended for routine clinical use. Our observer study revealed that radiologists could improve their sensitivity for the detection of lung nodules using the CAD software from 39.5% to 81.0% on average. It should be noted that the average sensitivity for the six radiologists with CAD software exceeded that of CAD software only (71.4%). In addition, although the average number of false-positive results for the six radiologists (0.89) was increased with the CAD software, this number was smaller than that for the CAD software only (0.95). Therefore, radiologists could use the CAD output very effectively by selecting valuable information provided by CAD. Figure 4 shows two examples of lung nodules that were identified correctly by five of the six radiologists but were not identified by the CAD software. One limitation of this observer study was that only 21 cases were used for the evaluation. To obtain reliable evidence that CAD software can be used routinely for improving radiologists’ diagnostic accuracy for the detection of lung nodules on MDCT, more cases provided consecutively in practical situations would be required in future studies. In the present study, 24 nodules smaller than 4 mm in diameter were included as the gold standard by the consensus of two radiologists (T.H. and N.N.), and 15 of the 24 small nodules (62.5%) were identified correctly by the CAD software. Because it was difficult for radiologists to detect small nodules, a number of small nodules were likely to be overlooked by the radiologists. This would be one of the reasons why their sensitivities were relatively
1511
HIROSE ET AL
low without the use of CAD software. Actually, the average sensitivity of the six radiologists for these 24 small nodules was 31.3% without and 72.2% with the CAD software. However, according to the guidelines for the management of small pulmonary nodules detected on CT scans proposed by the Fleischner Society in 2005 (26), patients with nodules no larger than 5 mm in diameter on baseline screening CT scans should be referred for repeat annual screening in 12 months, with no interval scans. In addition, in the protocol of the ongoing National Lung Screening Trial, a diameter of 4 mm has been used as a criterion for avoiding short-term interval scans of less than 12 months. This means that the detection of lung nodules larger than 4 (or 5) mm in diameter would be more important in routine work for detecting lung cancers at an early stage. In terms of the improvement of radiologists’ sensitivity for the 25 lung nodules 4 mm or larger in diameter using the CAD software, the average sensitivity of the six radiologists was also improved, from 47.3% to 89.3%. In conclusion, the CAD software used in this study significantly improved radiologists’ diagnostic accuracy compared with that in conventional image diagnosis. ACKNOWLEDGMENT
We are grateful to Ryutaro Takazakura, MD, Shinichi Ohta, MD, Akinaga Sonoda, MD, and Ayumi Seko, MD, for participating as observers; to Kazuhiko Matsumoto (Ziosoft, Inc.) for technical assistance with the CAD software; and to Elisabeth Lanzl for improving the manuscript. REFERENCES 1. Heelan RT, Flehinger BJ, Melamed MR, et al. Non-small-cell lung cancer: results of the New York screening program. Radiology 1984; 151:289 –293. 2. Sone S, Takashima S, Li F, et al. Mass screening for lung cancer with mobile spiral computed tomography scanner. Lancet 1998; 351:1242– 1245. 3. Flehinger BJ, Kimmel M, Melamed MR. The effect of surgical treatment on survival from early lung cancer. Implications for screening. Chest 1992; 101:1013–1018. 4. Sobue T, Suzuki T, Matsuda M, Kuroishi T, Ikeda S, Naruke T. Survival for clinical stage I lung cancer not surgically treated. Comparison between screen-detected and symptom-detected cases. The Japanese Lung Cancer Screening Research Group. Cancer 1992; 69:685– 692. 5. Miettinen OS. Screening for lung cancer. Radiol Clin North Am 2000; 38:479 – 486.
1512
Academic Radiology, Vol 15, No 12, December 2008
6. Muhm JR, Miller WE, Fontana RS, Sanderson DR, Uhlenhopp MA. Lung cancer detected during a screening program using four-month chest radiographs. Radiology 1983; 148:609 – 615. 7. Henschke CI, McCauley DI, Yankelevitz DF, et al. Early Lung Cancer Action Project: overall design and findings from baseline screening. Lancet 1999; 354:99 –105. 8. Austin JH, Romney BM, Goldsmith LS. Missed bronchogenic carcinoma: radiographic findings in 27 patients with a potentially resectable lesion evident in retrospect. Radiology 1992; 182:115–122. 9. Henschke CI, Yankelevitz DF, Libby DM, et al. Survival of patients with stage I lung cancer detected on CT screening. N Engl J Med 2006; 355:1763–1771. 10. Giger ML, Bae KT, MacMahon H. Computerized detection of pulmonary nodules in computed tomography images. Invest Radiol 1994; 29: 459 – 465. 11. Lee Y, Hara T, Fujita H, Itoh S, Ishigaki T. Automated detection of pulmonary nodules in helical CT images based on an improved templatematching technique. IEEE Trans Med Imaging 2001; 20:595– 604. 12. Armato SG III, Giger ML, Moran CJ, Blackburn JT, Doi K, MacMahon H. Computerized detection of pulmonary nodules on CT scans. RadioGraphics 1999; 19:1303–1311. 13. Armato SG III, Giger ML, MacMahon H. Automated detection of lung nodules in CT scans: preliminary results. Med Phys 2001; 28:1552– 1561. 14. Gurcan MN, Sahiner B, Petrick N, et al. Lung nodule detection on thoracic computed tomography images: preliminary evaluation of a computer-aided diagnosis system. Med Phys 2002; 29:2552–2558. 15. Wormanns D, Fiebich M, Saidi M, Diederich S, Heindel W. Automatic detection of pulmonary nodules at spiral CT: clinical application of a computer-aided diagnosis system. Eur Radiol 2002; 12:1052–1057. 16. Arimura H, Katsuragawa S, Suzuki K, et al. Computerized scheme for automated detection of lung nodules in low-dose computed tomography images for lung cancer screening. Acad Radiol 2004; 11:617– 629. 17. Awai K, Murao K, Ozawa A, et al. Pulmonary nodules at chest CT: effect of computer-aided diagnosis on radiologists’ detection performance. Radiology 2004; 230:347–352. 18. Wormanns D, Beyer F, Diederich S, Ludwig K, Heindel W. Diagnostic performance of a commercially available computer-aided diagnosis system for automatic detection of pulmonary nodules: comparison with single and double reading. Rofo 2004; 176:953–958. 19. Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys 2004; 31:2313–2330. 20. Chakraborty DP. Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol 2006; 13:1187–1193. 21. Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys 1989; 16:561–568. 22. Wagner RF, Metz CE, Campbell G. Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 2007; 14: 723–748. 23. Yuan R, Vos PM, Cooperberg PL. Computer-aided detection in screening CT for pulmonary nodules. AJR Am J Roentgenol 2006; 186:1280 – 1287. 24. Das M, Muhlenbruch G, Mahnken AH, et al. Small pulmonary nodules: effect of two computer-aided detection systems on radiologist performance. Radiology 2006; 241:564 –571. 25. Bae KT, Kim JS, Na YH, Kim KG, Kim JH. Pulmonary nodules: automated detection on CT images with morphologic matching algorithm— preliminary results. Radiology 2005; 236:286 –293. 26. MacMahon H, Austin JH, Gamsu G, et al. Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology 2005; 237:395– 400.
Rationale and Objectives. The aim of this study was to evaluate the usefulness of computer-aided diagnosis (CAD) software for the detection of lung nodules on multidetector-row computed tomography (MDCT) in terms of improvement in radiologists’ diagnostic accuracy in detecting lung nodules, using jackknife free-response receiver-operating characteristic (JAFROC) analysis. Materials and Methods. Twenty-one patients (6 without and 15 with lung nodules) were selected randomly from 120 consecutive thoracic computed tomographic examinations. The gold standard for the presence or absence of nodules in the observer study was determined by consensus of two radiologists. Six expert radiologists participated in a free-response receiver operating characteristic study for the detection of lung nodules on MDCT, in which cases were interpreted first without and then with the output of CAD software. Radiologists were asked to indicate the locations of lung nodule candidates on the monitor with their confidence ratings for the presence of lung nodules. Results. The performance of the CAD software indicated that the sensitivity in detecting lung nodules was 71.4%, with 0.95 false-positive results per case. When radiologists used the CAD software, the average sensitivity improved from 39.5% to 81.0%, with an increase in the average number of false-positive results from 0.14 to 0.89 per case. The average figure-of-merit values for the six radiologists were 0.390 without and 0.845 with the output of the CAD software, and there was a statistically significant difference (P ⬍ .0001) using the JAFROC analysis. Conclusion. The CAD software for the detection of lung nodules on MDCT has the potential to assist radiologists by increasing their accuracy. Key Words. Computer-aided diagnosis; CAD; lung nodule; jackknife free-response receiver operating characteristic analysis; JAFROC; multidetector-row computed tomography; MDCT. ©
2008 AUR. Published by Elsevier Inc. All rights reserved.
Acad Radiol 2008; 15:1505–1512 1
From the Department of Radiology, Shiga University of Medical Science, Seta-Tsukinowa-cho, Otsu-shi, Shiga 520-2192, Japan (T.H., N.N., Y.N., M.T., K.M.); and the Department of Radiology, University of Chicago, Chicago, IL (J.S.). Received May 16, 2008; accepted June 12, 2008. Address correspondence to: T.H. e-mail: [email protected]
©
2008 AUR. Published by Elsevier Inc. All rights reserved. doi:10.1016/j.acra.2008.06.009
Because some evidence suggests that the early detection of lung cancer may allow timely therapeutic intervention and thus favorable prognoses for patients (1–5), lung cancer screening using low-dose computed tomography has been proposed (6 – 8). For example, in a 2006 report by the International Early Lung Cancer Action Program (9), computed tomographic (CT) lung cancer screening resulted in diagnoses of lung cancer in 484 of 31,567 participants. Of these 484 participants, 412 (85%) had clini-
1505
HIROSE ET AL
cal stage I lung cancer, and the estimated 10-year survival rate was 88% in this subgroup. Among the 484 participants, 302 had clinical stage I cancer and underwent surgical resection within 1 month after diagnosis. Their survival rate was 92%. The development of multidetector-row computed tomography (MDCT) has made it easier to cover the whole lung with thin-section images. In addition, it has become possible to obtain isotropic voxel data, which can be useful in the observation of lesions and/or vessels from multiple directions. As a result, more than 300 images with thin-section thickness in millimeters per thoracic CT examination are reconstructed for radiologists’ routine work. Although computed tomography is very sensitive in detecting small, noncalcified nodules (8), the interpretation of a large number of CT images is time-consuming for radiologists. Therefore, a number of investigators have been developing computer-aided diagnosis (CAD) methods for the detection of lung nodules on CT images (10 – 16). In addition, a number of observer performance studies have demonstrated that CAD is expected to make a significant contribution to the improvement of radiologists’ diagnostic accuracy (17,18). In these respects, CAD could be considered markedly useful in assisting radiologists in the detection of small lung nodules. In clinical practice, when a massive number of images are interpreted, the benefit of CAD could be better than expected. Moreover, CAD is expected to reduce the variability in detecting lesions due to differences in the experience of radiologists in image interpretation, to maintain a consistently high level of diagnostic accuracy, and to shorten image interpretation time. In this study, we evaluated the usefulness of CAD software in terms of improvement in radiologists’ diagnostic accuracy in detecting lung nodules on MDCT, using jackknife free-response receiver operating characteristic (JAFROC) analysis (19,20). MATERIALS AND METHODS Our institutional review board approved the use of this database and the participation of radiologists in this observer performance study. Image Database A total of 120 consecutive thoracic CT studies were examined at our hospital in January 2006. The true presence of nodules was diagnosed in 80 of these cases. The remaining 40 were diagnosed as nodule-absent cases.
1506
Academic Radiology, Vol 15, No 12, December 2008
To determine the gold standard for the existence of lung nodules, two expert radiologists (T.H. and N.N., with 7 and 18 years of experience, respectively) individually diagnosed all cases and recorded nodule locations initially and then reached consensus or agreement by taking into account all image information, such as coronal and sagittal views and multiplanar reconstruction (MPR) images in an axial view provided by the CAD software. Inconsistencies and the presence or absence of nodules were resolved by discussion. Note that the radiologists used the output of the CAD software only retrospectively for confirmation of the gold standard results. The clinical significance and the question of whether nodules were benign or malignant were not examined in this study. Therefore, in the retrospective selection of the study patients, we did not use any specific diagnosis or pathologic condition of pulmonary nodules. The nodules could be associated with metastasis, primary lung cancer, or granuloma. Both calcified and noncalcified nodules were included. No cavitary, ground-glass, or subsolid nodules were present in our patient population. Patients in whom CT studies showed substantial parenchymal or pleural diseases, such as consolidation, fibrosis, or pleural effusion, were excluded from our study population. From the 80 patients with lung nodules, 15 and three patients with lung nodules were selected randomly as test and training cases in the observer study, respectively. From the 40 nodule-absent patients, six were selected randomly as test cases. No nodule-absent patient was selected as a training case. The 21 test cases with and without nodules used in this study included eight male and 13 female patients (age range, 11 to 77 years; average, 53.7). A total of 49 gold standard lung nodules were included in 15 test nodule-present cases. There were one to eight nodules found in these cases, ranging in diameter from 1.5 to 15 mm, with an average diameter of 4.5 mm. Twenty-five of the 49 nodules (51.0%) were 4 mm or larger in diameter. Figure 1 shows the distribution of the 49 nodules according to the range of their diameters. There were 10 to 12 nodules each in the three cases used in the training session. In terms of the anatomic locations of the 49 lung nodules, 17 were located in the upper lobe, 24 in the middle lobe, and eight in the lower lobe. We used two multidetector-row CT systems with 16 multidetectors (Sensation Cardiac 16, Siemens Medical Systems, Erlangen, Germany; and Aquilion, Toshiba Medical, Nasu, Japan) to obtain images of the whole lung. Technical parameters for the Siemens system were as follows: rotation time, 0.37 seconds; collimation, 0.75
Academic Radiology, Vol 15, No 12, December 2008
CAD SOFTWARE FOR LUNG NODULE DETECTION
Figure 1. The distribution of the 49 nodules according to the nodule diameter ranged from 1.5 to 15.0 mm, with an average diameter of 4.5 mm. Twenty-five of 49 nodules were 4 mm or larger in diameter.
mm; tube voltage, 120 kV; tube current, 100 effective mA (223 mA); and feed rotation, 10 mm (beam pitch, 0.833). Technical parameters for the Toshiba system were as follows: rotation time, 0.5 seconds; collimation, 1.0 mm; tube voltage, 120 kV; tube current, 280 mA; and helical pitch, 10.0 (beam pitch, 1.28). A reconstruction formula targeted for lungs was used. Reconstruction kernels used in this study were B70f for the Siemens system and FC52 for the Toshiba system. All images were reconstructed with a section thickness of 1 mm and a slice interval of 1 mm. The image matrix size was 512 ⫻ 512 pixels. CAD Software ZIOCAD LE version 1.15 (Ziosoft, Inc., Tokyo, Japan) was used. Both solid and ground-glass opacity nodules can be detected by this software. The characteristics of this CAD system are as follows: (1) technology for calculating lesions in a 3-dimensional manner in 3-dimensional space; (2) high-speed technology focusing on calculations in situ; (3) technology for automatically detecting lungs, the thoracic wall, and the heart; and (4) ground-glass opacity segmentation technology focusing on the CT value distribution. The CAD method for nodule detection consisted of four steps: (1) filtration for low-frequency bands to decrease variations in the calculation results using a standard reconstruction kernel and a high-frequency intensified reconstruction kernel; (2) lung segmentation by limit-
ing the calculation area to calculate more quickly, to decrease the number of false-positive results, and to segment nodules attached to the thoracic wall; (3) the identification of initial candidates using a local maximum CT value and the identification of the relative distance between initial candidates; and (4) calculation of the point concentration for each initial candidate using a proprietary evaluation formula with a combination of geometric and statistical methods to calculate nodularity, which is used to determine the final candidates. Observer Study Six expert radiologists (Ryutaro Takazakura, MD, Shinichi Ohta, MD, Akinaga Sonoda, MD, Ayumi Seko, MD, Y.N., and M.T.) with 8 to 23 years of experience participated in an observer study for diagnosing lung nodules on MDCT without and with the output of the CAD software. Before the observer study, the radiologists were given the following instructions: (1) your performance in detecting lung nodules, first without and then with CAD output, is to be evaluated; (2) the role of the CAD output is that of a “second opinion”; (3) 21 CT studies (with 1-mm-thick sections) that did or did not show lung nodules are included in this observer study; (4) the observer in this study will be blinded to the number of patients with lung nodules and the performance level of the CAD software; and (5) click on the screen using the mouse to locate the most likely position and indicate on a bar your
1507
HIROSE ET AL
Academic Radiology, Vol 15, No 12, December 2008
Figure 2. Display of images without display of computer-aided diagnosis results. An axial image is shown at the center of the monitor, and slices can be moved in any direction, from head to toe, by rotation of the mouse wheel. Coronal (upper left), sagittal (middle left), and multiplanar reconstruction (lower left) images are displayed by moving the cross line on the axial image at the center of the monitor and left-clicking the mouse. The location corresponding to the respective site on the monitor image is displayed by left-clicking of the mouse or moving the cross line. Furthermore, candidate nodules detected by a radiologist can be displayed in the volume-rendering (upper right) and maximum-intensity projection (middle left) images and saved to the list on the right side of the monitor (lower right).
confidence level regarding the presence (or absence) of a lung nodule. For a training session before the test, three training cases, each of which had 10 to 12 nodules, were provided to the radiologists for learning how to operate the cine mode interface and how to take into account computer data for making their decisions. The radiologists began by diagnosing cases without the CAD software (Fig 2) and then rediagnosed them using the results provided by the CAD software (Fig 3). They carried out their interpretations repeatedly by following routine diagnostic procedures. For each nodule, the location was stored in the software, and the radiologist’s confidence for the pres-
1508
ence of lung nodules was recorded using a rating bar scale (7 cm in width), on which the right end corresponded to “absolute presence of a lung nodule” and the left end corresponded to “absolute absence of a lung nodule.” As shown in Figures 2 and 3, the software used in this study allowed the arbitrary selection of section thickness. In addition, it was possible to observe coronal and sagittal views and arbitrary slice sections as well as axial views in MPR images. During the interpretation of CT images with CAD software, the radiologists were allowed to make full use of functions, such as the 3-dimensional adjacent rendering mode and MPR for diagnosis. The win-
Academic Radiology, Vol 15, No 12, December 2008
CAD SOFTWARE FOR LUNG NODULE DETECTION
Figure 3. Display of images with computer-aided diagnosis (CAD) results. In the list at the bottom of the monitor screen, candidate nodules identified by CAD and nodules detected by radiologists are displayed. The former and latter nodules are indicated as red and green circles, respectively, on the axial view (upper left) by selection of items in the list. The same location can be displayed on the axial (upper center), maximum-intensity projection (upper right), coronal (middle left), and sagittal (lower center) images without displaying CAD results. Arbitrary locations in corresponding slices can be displayed by double-clicking or moving the cross line.
dow level was set at ⫺650 hu, and the window width was set at 1500 HU. JAFROC analysis (19) was used for the evaluation of the radiologists’ performance in the detection of lung nodules without and with the output of the CAD software. JAFROC analysis has been proposed for estimating statistically significant differences between modalities when location issues are relevant. JAFROC analysis is based on a free-response receiver operating characteristic (FROC) paradigm and accounts for reader variation (19,21). Conventional ROC analysis is of limited value for this kind of application because only one signal can be used per case, and the location of the signal cannot be taken into account in the evaluation. In contrast, FROC
analysis allows one to evaluate the performance of radiologists in diagnosing medical images using multiple responses, each with information on the confidence level and location (22). For statistical testing of differences between radiologists’ performance without and with the output of the CAD software, we applied JAFROC analysis with method 2 of Chakraborty and Berbaum (19) to estimate figure-of-merit (FOM) values (the analogue of the area under the ROC curve) for each reader. Chakraborty and Berbaum (19) proposed two implementations (methods 1 and 2) of JAFROC analysis in their study; we used method 2, which showed better properties in that study. In this statistical procedure, we calculated a sequence of FOM values for the JAFROC analysis
1509
HIROSE ET AL
and then built a pseudovalue matrix with these FOM values for multireaders and multicases according to the equations shown in the original investigation by Chakraborty and Berbaum (19). Finally, the FOM pseudovalue matrix was analyzed using analysis of variance techniques for estimating statistically significant differences between the two data sets obtained without and with the output of the CAD software. We also obtained sensitivities and the number of falsepositive results for all of the radiologists to compare their performance in the detection of lung nodules without and with the output of the CAD software.
RESULTS The CAD software correctly identified 35 of the 49 nodules of the gold standard, with 20 false-positive results for 21 cases. Therefore, the sensitivity of the CAD software for the cases used in this study was 71.4%, with 0.95 false-positive results per case. Among 14 lung nodules (mean diameter, 4.5 mm; range, 2.3–12.0) overlooked by the CAD software, nine were small nodules attached to the chest wall, and five were in contact with the trachea and/or blood vessels. Four nodules in one case, two nodules in three cases, and one nodule in four cases were overlooked. Among 25 nodules 4 mm or larger in diameter, the CAD software correctly identified 20 (80.0%). Thus, the CAD software identified 15 of the 24 (62.5%) nodules smaller than 4 mm in diameter. Of 20 no-nodule findings incorrectly identified by the CAD software, 16 were bronchial bifurcations or bifurcations of blood vessels, two were inflammatory scars in the lung apex, and one was interlobar pleural thickening. The remaining false-positive finding was located outside the lung. Using the CAD software, all radiologists remarkably improved their sensitivity for the detection of lung nodules from an average sensitivity of 39.5% to 81.0%, as shown in Table 1. However, the average number of falsepositive results increased from 0.14 to 0.89 per case. Eighteen of the 20 no-nodule findings (90.0%) were incorrectly identified by the CAD software and were also identified incorrectly as lung nodules by more than four of the six readers using the CAD software. Although the average rating score (0.405) for these 18 false-positive findings was relatively low compared to that for the actual lung nodules (0.863), the radiologists were likely to adopt the output of the CAD software.
1510
Academic Radiology, Vol 15, No 12, December 2008
Table 1 FOM Values Obtained from JAFROC Analysis (Method 2) without and with Output of the CAD Software for the Six Radiologists Without CAD FP Results/ Reader Sensitivity Case 1 2 3 4 5 6 Average
24.5% 30.6% 59.2% 40.8% 38.8% 42.9% 39.5%
0.10 0.14 0.24 0.19 0.05 0.14 0.14
With CAD
FOM 0.236 0.312 0.563 0.433 0.364 0.434 0.390*
FP Results/ Sensitivity Case 75.5% 75.5% 85.7% 79.6% 85.7% 83.7% 81.0%
0.90 1.00 1.10 0.52 0.95 0.86 0.89
FOM 0.808 0.792 0.870 0.819 0.881 0.901 0.845*
CAD, computer-aided diagnosis; FOM, figure-of-merit; FP, false-positive; JAFROC, jackknife free-response receiver operating characteristic. *Statistically significant difference (P ⬍ .0001) between average FOM values.
JAFROC analysis revealed that the CAD software improved the FOM values for all readers. There was a statistically significant difference (P ⬍ .0001) when the average FOM of the six radiologists with and without the CAD software was compared by taking into account reader and case variations (Table 1).
DISCUSSION It is widely known that computed tomography is superior to chest x-rays for the detection of small, noncalcified nodules (8). However, for radiologists to detect minute lung nodules on a number of CT slice images requires time and manpower. CAD software, as the “second radiologist,” has been developed for enhancing the capability of radiologists to detect lung nodules. A variety of CAD systems for the detection of lung nodules have been developed. Single-detector-row CT images were used for diagnosis in previously developed CAD systems, but recently MDCT has been used increasingly for evaluation. The CAD software used in this study was designed for multidetector-row CT images. Previously, it was common to calculate the characteristic of each nodule on the basis of 2-dimensional CT slice images because of the limitations of the CT equipment. However, with technological advances in computed tomography, it has become possible to obtain 3-dimensional volume data with a markedly smaller slice pitch. The algorithm of the CAD software
Academic Radiology, Vol 15, No 12, December 2008
CAD SOFTWARE FOR LUNG NODULE DETECTION
Figure 4. Two examples of lung nodules (arrows), which were correctly identified by 5 of the 6 radiologists as lung nodules but were not identified by the computed-aided diagnosis software.
used in this study used 3-dimensional volume data for calculating the characteristics of lung nodule candidates. Thus, more accurate recognition of nodule shapes could be explored by the calculation of CT volume data in a 3-dimensional manner. It has been reported that some nodule detection CAD software for multidetector-row CT images is now commercially available. Yuan et al (23) reported a sensitivity of 72.6% for Image Checker CT (R2 Technologies, Sunnyvale, CA). Das et al (24) also used Image Checker CT and reported a sensitivity of 73%. They also used Nodule Enhanced Viewing (Siemens Medical Systems) and achieved a sensitivity of 75% (24). Although the database used in our study was different and included a limited number of cases, there were differences in the sensitivity between the CAD software used in our study (71.4%) and their CAD software. In addition, it should be noted that the number of false-positive results per case (0.95) in the CAD software used in our study was very small compared with that in Yuan et al’s (23) study (3.19) and Das et al’s (24) study (six for Image Checker CT and eight for Nodule Enhanced Viewing). Bae et al (25) reported that the overall sensitivity for nodule detection with a CAD program was 95.1% (156 of 164 nodules). The sensitivity according to the nodule diameter was 91.2% (52 of 57 nodules) for nodules 3 to 5 mm and 97.2% (104 of 107 nodules) for nodules 5 mm and larger. The number of false-positive detections per patient was 6.9 for structures 3 mm and larger and 4.0 for structures 5 mm and larger. The CAD software used by Bae et al (25) had high sensitivity; however, their number of false-
positive results would not be recommended for routine clinical use. Our observer study revealed that radiologists could improve their sensitivity for the detection of lung nodules using the CAD software from 39.5% to 81.0% on average. It should be noted that the average sensitivity for the six radiologists with CAD software exceeded that of CAD software only (71.4%). In addition, although the average number of false-positive results for the six radiologists (0.89) was increased with the CAD software, this number was smaller than that for the CAD software only (0.95). Therefore, radiologists could use the CAD output very effectively by selecting valuable information provided by CAD. Figure 4 shows two examples of lung nodules that were identified correctly by five of the six radiologists but were not identified by the CAD software. One limitation of this observer study was that only 21 cases were used for the evaluation. To obtain reliable evidence that CAD software can be used routinely for improving radiologists’ diagnostic accuracy for the detection of lung nodules on MDCT, more cases provided consecutively in practical situations would be required in future studies. In the present study, 24 nodules smaller than 4 mm in diameter were included as the gold standard by the consensus of two radiologists (T.H. and N.N.), and 15 of the 24 small nodules (62.5%) were identified correctly by the CAD software. Because it was difficult for radiologists to detect small nodules, a number of small nodules were likely to be overlooked by the radiologists. This would be one of the reasons why their sensitivities were relatively
1511
HIROSE ET AL
low without the use of CAD software. Actually, the average sensitivity of the six radiologists for these 24 small nodules was 31.3% without and 72.2% with the CAD software. However, according to the guidelines for the management of small pulmonary nodules detected on CT scans proposed by the Fleischner Society in 2005 (26), patients with nodules no larger than 5 mm in diameter on baseline screening CT scans should be referred for repeat annual screening in 12 months, with no interval scans. In addition, in the protocol of the ongoing National Lung Screening Trial, a diameter of 4 mm has been used as a criterion for avoiding short-term interval scans of less than 12 months. This means that the detection of lung nodules larger than 4 (or 5) mm in diameter would be more important in routine work for detecting lung cancers at an early stage. In terms of the improvement of radiologists’ sensitivity for the 25 lung nodules 4 mm or larger in diameter using the CAD software, the average sensitivity of the six radiologists was also improved, from 47.3% to 89.3%. In conclusion, the CAD software used in this study significantly improved radiologists’ diagnostic accuracy compared with that in conventional image diagnosis. ACKNOWLEDGMENT
We are grateful to Ryutaro Takazakura, MD, Shinichi Ohta, MD, Akinaga Sonoda, MD, and Ayumi Seko, MD, for participating as observers; to Kazuhiko Matsumoto (Ziosoft, Inc.) for technical assistance with the CAD software; and to Elisabeth Lanzl for improving the manuscript. REFERENCES 1. Heelan RT, Flehinger BJ, Melamed MR, et al. Non-small-cell lung cancer: results of the New York screening program. Radiology 1984; 151:289 –293. 2. Sone S, Takashima S, Li F, et al. Mass screening for lung cancer with mobile spiral computed tomography scanner. Lancet 1998; 351:1242– 1245. 3. Flehinger BJ, Kimmel M, Melamed MR. The effect of surgical treatment on survival from early lung cancer. Implications for screening. Chest 1992; 101:1013–1018. 4. Sobue T, Suzuki T, Matsuda M, Kuroishi T, Ikeda S, Naruke T. Survival for clinical stage I lung cancer not surgically treated. Comparison between screen-detected and symptom-detected cases. The Japanese Lung Cancer Screening Research Group. Cancer 1992; 69:685– 692. 5. Miettinen OS. Screening for lung cancer. Radiol Clin North Am 2000; 38:479 – 486.
1512
Academic Radiology, Vol 15, No 12, December 2008
6. Muhm JR, Miller WE, Fontana RS, Sanderson DR, Uhlenhopp MA. Lung cancer detected during a screening program using four-month chest radiographs. Radiology 1983; 148:609 – 615. 7. Henschke CI, McCauley DI, Yankelevitz DF, et al. Early Lung Cancer Action Project: overall design and findings from baseline screening. Lancet 1999; 354:99 –105. 8. Austin JH, Romney BM, Goldsmith LS. Missed bronchogenic carcinoma: radiographic findings in 27 patients with a potentially resectable lesion evident in retrospect. Radiology 1992; 182:115–122. 9. Henschke CI, Yankelevitz DF, Libby DM, et al. Survival of patients with stage I lung cancer detected on CT screening. N Engl J Med 2006; 355:1763–1771. 10. Giger ML, Bae KT, MacMahon H. Computerized detection of pulmonary nodules in computed tomography images. Invest Radiol 1994; 29: 459 – 465. 11. Lee Y, Hara T, Fujita H, Itoh S, Ishigaki T. Automated detection of pulmonary nodules in helical CT images based on an improved templatematching technique. IEEE Trans Med Imaging 2001; 20:595– 604. 12. Armato SG III, Giger ML, Moran CJ, Blackburn JT, Doi K, MacMahon H. Computerized detection of pulmonary nodules on CT scans. RadioGraphics 1999; 19:1303–1311. 13. Armato SG III, Giger ML, MacMahon H. Automated detection of lung nodules in CT scans: preliminary results. Med Phys 2001; 28:1552– 1561. 14. Gurcan MN, Sahiner B, Petrick N, et al. Lung nodule detection on thoracic computed tomography images: preliminary evaluation of a computer-aided diagnosis system. Med Phys 2002; 29:2552–2558. 15. Wormanns D, Fiebich M, Saidi M, Diederich S, Heindel W. Automatic detection of pulmonary nodules at spiral CT: clinical application of a computer-aided diagnosis system. Eur Radiol 2002; 12:1052–1057. 16. Arimura H, Katsuragawa S, Suzuki K, et al. Computerized scheme for automated detection of lung nodules in low-dose computed tomography images for lung cancer screening. Acad Radiol 2004; 11:617– 629. 17. Awai K, Murao K, Ozawa A, et al. Pulmonary nodules at chest CT: effect of computer-aided diagnosis on radiologists’ detection performance. Radiology 2004; 230:347–352. 18. Wormanns D, Beyer F, Diederich S, Ludwig K, Heindel W. Diagnostic performance of a commercially available computer-aided diagnosis system for automatic detection of pulmonary nodules: comparison with single and double reading. Rofo 2004; 176:953–958. 19. Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys 2004; 31:2313–2330. 20. Chakraborty DP. Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol 2006; 13:1187–1193. 21. Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys 1989; 16:561–568. 22. Wagner RF, Metz CE, Campbell G. Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol 2007; 14: 723–748. 23. Yuan R, Vos PM, Cooperberg PL. Computer-aided detection in screening CT for pulmonary nodules. AJR Am J Roentgenol 2006; 186:1280 – 1287. 24. Das M, Muhlenbruch G, Mahnken AH, et al. Small pulmonary nodules: effect of two computer-aided detection systems on radiologist performance. Radiology 2006; 241:564 –571. 25. Bae KT, Kim JS, Na YH, Kim KG, Kim JH. Pulmonary nodules: automated detection on CT images with morphologic matching algorithm— preliminary results. Radiology 2005; 236:286 –293. 26. MacMahon H, Austin JH, Gamsu G, et al. Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology 2005; 237:395– 400.