Computer-aided detection of pulmonary nodules: influence of nodule characteristics on detection performance

Clinical Radiology (2005) 60, 196–206

Computer-aided detection of pulmonary nodules: influence of nodule characteristics on detection performance ¨sla, S. Obenauerc, K. Martena,*, C. Engelkea, T. Seyfarthb, A. Grillho E.J. Rummenya a

Department of Radiology, Technical University Munich, Munich, bSiemens Medical Solutions, Forchheim, ¨ttingen, Germany and cDepartment of Radiology, Georg August University, Go

Received 3 March 2004; received in revised form 4 May 2004; accepted 25 May 2004

KEYWORDS Computed tomography (CT); Thin-section; Computers; Diagnostic aid; Computed tomography (CT); Image processing; Lung; Nodule; Diagnostic radiology; Observer performance

AIM: To evaluate prospectively the influence of pulmonary nodule characteristics on detection performances of a computer-aided diagnosis (CAD) tool and experienced chest radiologists using multislice CT (MSCT). MATERIALS AND METHODS: MSCT scans of 20 consecutive patients were evaluated by a CAD system and two independent chest radiologists for presence of pulmonary nodules. Nodule size, position, margin, matrix characteristics, vascular and pleural attachments and reader confidence were recorded and data compared with an independent standard of reference. Statistical analysis for predictors influencing nodule detection or reader performance included chi-squared, retrograde stepwise conditional logistic regression with odds ratios and nodule detection proportion estimates (DPE), and ROC analysis. RESULTS: For 135 nodules, detection rates for CAD and readers were 76.3, 52.6 and 52.6%, respectively; false-positive rates were 0.55, 0.25 and 0.15 per examination, respectively. In consensus with CAD the reader detection rate increased to 93.3%, and the false-positive rate dropped to 0.1/scan. DPEs for nodules # 5 mm were significantly higher for ICAD than for the readers ðp , 0:05Þ: Absence of vascular attachment was the only significant predictor of nodule detection by CAD ðp ¼ 0:0006 – 0:008Þ: There were no predictors of nodule detection for reader consensus with CAD. In contrast, vascular attachment predicted nodule detection by the readers ðp ¼ 0:0001 – 0:003Þ: Reader sensitivity was higher for nodules with vascular attachment than for unattached nodules (sensitivities 0.768 and 0.369; 95% confidence intervals ¼ 0.651 – 0.861 and 0.253 –0.498, respectively). CONCLUSION: CAD increases nodule detection rates, decreases false-positive rates and compensates for deficient reader performance in detection of smallest lesions and of nodules without vascular attachment. q 2005 The Royal College of Radiologists. Published by Elsevier Ltd. All rights reserved.

Introduction Spiral computed tomography is the most sensitive imaging technique for depiction of the smallest *Guarantor and correspondent: K. Marten, Department of Radiology, Institut fu ¨ntgendiagnosik, Klinikum rechts der ¨r Ro Isar, Technical University, Ismaningerstr. 22, 81675 Mu ¨nchen, Germany. Tel.: þ49-89-4140-2621; fax: þ 49-89-4140-4834. E-mail address: [email protected]

pulmonary nodules.1 – 8 In particular, the introduction of multislice CT scanners (MSCTs) offering simultaneous acquisition of up to 32.0 £ 0.75 mm sections allows for sub-millimetre isotropic imaging of the entire chest within a single breathhold, thereby yielding significantly higher detection rates of small pulmonary nodules than thick collimation CT. Additionally MSCT has a higher accuracy in the distinction of nodule growth suggestive of malignancy.9 However, interpretation of thin-section

0009-9260/$ - see front matter q 2005 The Royal College of Radiologists. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.crad.2004.05.014

Computer-aided detection of pulmonary nodules

MSCT examinations of the chest comprising up to some 750 axial images is tedious and may result in decreasing confidence levels even with experienced chest radiologists, causing higher false-negative rates. Accordingly, in a retrospective study of first annual lung cancer CT screening examinations, nodules were missed in 26% of cases.7 In more than half of these cases the retrospectively identified nodules were smaller than 4 mm (62%) and over one third (37%) measured 4 –7 mm. Conversely, radiologists’ detection performance of pulmonary nodules may be enhanced by computer-assisted diagnosis (CAD) tools that automatically and effectively search radiological data. Preliminary CAD improvements have been suggested by several clinical investigators using comparative studies concepts.10 – 15 Ideally, a CAD system would yield a general increase in detection rates regardless of nodule morphology, size or location, and would therefore outweigh reader performance inconsistencies in depiction of inconspicuous lesions. However, the problem of possible influence of nodule location, morphology or size on this performance gain remains essentially unaddressed. Therefore, the purpose of this study was to investigate the influence of nodule location, morphology and size on CAD and reader performance, and to evaluate whether an increased diagnostic value of CAD can be attributed to comparatively greater morphological and size robustness in depiction of pulmonary nodules.

Materials and methods Study design, patients and scan techniques This was a prospective blinded observational study comparing the efficiency of a prototype interactive CAD system, an automatic tool for multislice CT-based detection of pulmonary nodules, with the performance of two experienced chest radiologists, in order to evaluate the influence of nodule size, location and morphology on reader and ICAD performance. Clinical routine multislice CT (MSCT) scans of 20 individuals (mean age 62.4 years, range 29 – 84 years) having investigations for pulmonary metastatic disease were prospectively reconstructed and evaluated separately from the clinical routine by ICAD and by two experienced chest radiologists for solid pulmonary nodules. The results were compared against an independent gold standard, which was determined through a consensus reading by two additional experienced chest radiologists

197

who had knowledge of the results of the two study readers and the results of ICAD. The resulting study data were not used for clinical decision making and at our centre no institutional review board approval is required for this type of study. All scans were performed using a 16-row MSCT unit (Sensation 16, Siemens Medical Solutions, Erlangen, Germany). The scan parameters used were tube voltage 120 kV, tube current 80 mAs at 0.5 s gantry revolution, 0.75 mm slice collimation and table feed of 19.2 mm/rot. As part of the routine staging protocol, single-phase peripheral intravenous power injection was performed using 80 ml nonionic contrast material with 300 mg/ml iodine concentration (Ultravist 300, Schering AG, Berlin, Germany), and subsequently 30 ml of normal saline solution at a flow of 2 – 3 ml/s in all cases. Image data were reconstructed using a lung filter kernel (B60f) at a slice thickness setting of 0.75 mm at 0.6 mm reconstruction increment. Image data were stored and analyzed on a dedicated workstation not accessible for routine clinical work.

CT evaluation by radiologists All CT image data were evaluated by the two blinded radiologists (A.G. and S.O.) in random order. The readers had at least 5 years’ experience in clinical chest CT interpretation. CT datasets were interactively assessed using axial cine mode without the help of additional 3D tools. Reader confidence in the diagnosis of each pulmonary nodule was assessed and documented on a threepoint scale (0 ¼ negative, 1 ¼ uncertain, 2 ¼ positive) independently by each reader. Underlying pulmonary pathologies potentially interfering with CAD detection were documented and included atelectasis, consolidation and diffusely decreased lung density. The presence of considerable breathing or pulsation-related movement artefacts was recorded. The readers documented nodule position, diameter, solidity (solid, part solid and “ground glass”), inherent nodule characteristics (matrix) such as calcification and cavitation, margin characteristics (well- or ill-defined), and vascular and pleural attachment.

CAD tool and CT evaluation by ICAD ICAD (Siemens Medical Solutions, Erlangen, Germany; Siemens Corporate Research, Princeton, New Jersey, USA) is a knowledge-based automatic lung nodule detection prototype system comprising multiple “expert processing modules” to detect solitary nodules or nodules with pleural or vascular attachment. The ICAD segmentation algorithm has

198 been described previously.16 ICAD offers a percentual confidence level for the diagnosis of each lung nodule, which we took for subsequent statistical analysis. Additionally, each nodule diameter is displayed on the output screen (Fig. 1). We performed the ICAD image data evaluation for all image datasets after initial evaluation of the two study readers. Nodule positions, ICAD confidence levels and diameters were recorded.

Gold standard The gold standard was established by consensus of two independent experienced chest radiologists (K.M. and C.E.) who first evaluated all datasets separately and subsequently included the results of ICAD and the two study radiologists by consensus. On clinical grounds—all patients had a histologically confirmed extrapulmonary primary malignancy—no surgical or histological correlation of the nodule diagnosis was performed and therefore such data

K. Marten et al.

were not included in our reference standard. The nodule diameters were adapted in consensus from ICAD measurement data or, in cases of failed ICAD detection, revaluated manually or adapted from the study reader data. The criterion for the diagnosis of a pulmonary nodule was the presence of a well-demarcated, solid or part solid or groundglass, spherical or more irregular ellipsoid or complex structure. This, if solid, should display a density in the range of soft tissue or above, or well above the density of the surrounding lung parenchyma. The nodule could be attached to any neighbouring structure. Longitudinal or linear densities without nodular or mass-like aspect, and small subpleural densities that were attributable to pleural adhesions without nodular aspect, were excluded from the diagnosis and the gold standard. Bronchial wall thickening, thickening of interlobular septa, and linear or reticular interstitial changes were excluded, as were areas of air space consolidation and regions displaying significant

Figure 1 Output screen of ICAD showing detected pulmonary nodule on main display (right,) and central evaluation unit with confidence level bar (left middle). Additional 3D and rotating multiplanar displays of nodule for better spatial appreciation (left top and bottom).

Computer-aided detection of pulmonary nodules

movement artefacts that would not allow for safe differentiation between pulmonary nodules and bronchial or vascular structures.

Statistical methods Our institutional statistician was consulted to ensure the use of the appropriate statistical tests. All calculations were performed using a spreadsheetbased statistical software package (StatsDirecte release 2.3.7, CamCode, Herts, UK). Abnormality of nodule size distribution was tested using the Shapiro-Wilk method. Univariate analysis of predictors for nodule detection was performed using Chi-square with two-tailed Fisher’s exact probabilities. Because univariate and multivariate tests for binary response data (chi square and standard logistic regression) assume the independence of each observation, which possibly deteriorated in our dataset comprising 135 nodules from 20 patients, chi-square testing was coupled with a univariate version of a conditional logistic regression allowing for each factor tested by the chi square to estimate an additional patient effect. Conditional logistic regression fits and analyses conditional logistic models where the observations are not independent but are matched in some way and, therefore, includes the introduction of a stratum (patients). StatsDirect fits the regression by maximization of the natural logarithm of the conditional likelihood function using the Newton-Raphson iteration as described by Krailo et al., Smith et al. and Howard.17 – 19 Because dichotomous outcome data are required for chi square and logistic regression analyses, uncertain reader results (rated 1) were grouped together with the negative group (rated 0). Conditional logistic regression was calculated using odds ratio (OR) estimates with 95% confidence intervals. Factors tested included morphological features such as nodule margin or matrix characteristics (Table 1). Factors present in less than 6 nodules among the whole collective were excluded from statistical work-up. Multivariate analysis of combined factors predicting nodule detection, including nodule morphology, size and location, was performed using similar conditional logistic regression with subsequent retrograde stepwise conditional logistic regression which selects the best predictors until all remaining variables of the tested model are significant. In order to estimate the influence of nodule size on detection rates of ICAD and readers, nodule detection proportion estimates were calculated using the fitted logistic model coefficients with a specified 1 mm stepwise increment of values within the nodule diameter range (1 – 30 mm) as

199

well as the regression mean. Finally, the penetrance of morphological predictor effects on overall reader and ICAD performances was stratified by ROC analysis obtaining values of the area under the curve, sensitivities and specificities for each nodule subgroup of present versus absent predictive factors. StatsDirect performs the calculation of the area under the ROC curve directly by an extended trapezoidal rule20 and by a non-parametric method analogous to the Wilcoxon/Mann Whitney test;21 two-tailed confidence intervals are calculated using DeLong’s variance estimate.22 All tests were carried out in two-tailed fashion with p values less than 0.05 or non-overlapping 95% confidence intervals indicating statistical significance.

Results Nodule characteristics and overall nodule detection rates of ICAD and readers A total of 135 nodules were included by the independent review panel in the gold standard. Minor breathing or other pulmonary movement artefacts were recorded in 4 of 20 patients, small atelectasis in 3 patients and consolidation in 3 patients. Emphysema or other pathological diffuse low pulmonary attenuation was not recorded. In no case was the scan quality substantially reduced to allow for assessment and comparison of small nodular densities (0.7 – 4.0 mm) with RST image data. Of 133 solid nodules, 6 had a totally calcified matrix and 2 were partially solid. The median nodule size was 4.4 mm (range: 1.0 – 29.6 mm), with an upper quartile of 6.6 mm and a lower quartile of 3.0 mm. Nodule sizes were not distributed normally (p , 0:0001; Fig. 2). ICAD detected 76.3% of nodules, with 0.55 false-positive findings per examination. The best results were achieved by consensus reading of reader 1 with ICAD, which increased the detection rate to 93.3% and reduced the false-positive findings to 0.1 per examination. Readers 1 and 2 detected 52.6% of nodules, with 0.25 and 0.15 false-positive findings per examination, respectively.

Influence of nodule size on nodule detection rates and predictors of nodule detection Nodule size significantly affected nodule detection by the readers (p ¼ 0:0002 – 0:002; OR ¼ 1.50 – 1.60) but it did not influence detection by ICAD with or without consensus of reader 1 (Table 1 and Fig. 3).

200

K. Marten et al.

Table 1 Multivariate analysis I of factors predicting nodule detection Model

Comparison

p

OR

95% CI

r2

CLR Nodule diameter Calcification Margin definition Pleural attachment Vascular attachment

ICAD Increasing mm Yes/no Sharp/ill Yes/no Yes/no

0.560 0.123 0.474 0.981 0.001

1.046 0.089 1.709 0.986 0.099

0.900–1.215 0.004–1.924 0.394–7.412 0.314–3.098 0.025–0.396

0.211

BCLR Vascular attachment

ICAD Yes/no

0.001

0.145

0.048–0.435

0.174

CLR Nodule diameter Calcification Margin definition Pleural attachment Vascular attachment

ICAD þ reader 1 Increasing mm Yes/no Sharp/ill Yes/no Yes/no

0.109 0.978 0.567 0.183 0.111

0.891 1.21 £ 106 0.581 3.387 4.670

0.774–1.026 n.e. 0.091–3.731 0.562–20.428 0.703–31.043

0.174

BCLR

ICAD þ reader 1 No significant predictors

–

CLR Nodule diameter Calcification Margin definition Pleural attachment Vascular attachment

Reader 1 Increasing mm Yes/no Sharp/ill Yes/no Yes/no

0.001 0.149 0.296 0.342 0.011

1.545 6.975 0.482 1.734 4.437

1.183–2.017 0.499–97.467 0.123–1.894 0.557–5.398 1.411–13.947

BCLR Nodule diameter Vascular attachment

Reader 1 Increasing mm Yes/no

0.0002 0.027

1.602 3.076

1.253–2.046 1.135–8.336

CLR Nodule diameter Calcification Margin definition Pleural attachment Vascular attachment

Reader 2 Increasing mm Yes/no Sharp/ill Yes/no Yes/no

0.002 0.223 0.678 0.191 0.0006

1.493 4.351 0.756 2.100 8.356

1.158–1.925 0.409–46.310 0.203–2.823 0.691–6.387 2.504–27.879

BCLR Nodule diameter Vascular attachment

Reader 2 Increasing mm Yes/no

0.0003 0.001

1.564 5.498

1.228–1.990 1.985–15.227

0.309

0.271

0.342

0.313

p, conditional logistic regression probability; OR, odds ratio; CI, confidence interval of OR; r 2, pseudo (McFadden) R-square for conditional logistic regression; CLR, conditional logistic regression; BCLR, endpoint of backward stepwise CLR; n.e., not estimated.

The proportion of nodules detected by readers 1 and 2 dropped significantly below the ICAD values for nodules smaller than 6 mm diameter (for ICAD and readers 1 and 2, respectively: p , 0:05; detection proportion 0.75, 0.53 and 0.54; 95% CI ¼ 0.67 –0.82, 0.44 – 0.63 and 0.44 – 0.63) (Fig. 3). The regression mean for readers 1 and 2 and ICAD was 7.19 mm diameter, indicating identical response proportions for all observers for nodules of this diameter. Above the regression mean the two readers performed better than ICAD (for ICAD and readers 1 and 2, respectively: p , 0:05; for nodules . 12 mm, detection proportion 0.68, 0.96 and 0.97; 95% CI ¼ 0.53 –0.81, 0.82 –0.99 and 0.86 –0.99) (Fig. 3). Nodule location had no influence on detection by ICAD or any of the readers (Table 2). Among the morphological factors tested for influence on

nodule detection, only vessel attachment significantly predicted detection by ICAD (p ¼ 0:0006 – 0:008; OR ¼ 0.10 –0.15, Tables 1 and 3, and Fig. 4). Hence, ICAD recognized more of the nodules without vascular attachment. However, the overall performance of ICAD with or without consensus of reader 1 was not influenced by any morphological factor tested (Table 4). For both readers vessel attachment predicted nodule detection (p ¼ 0:0001 – 0:003; OR ¼ 3.07 – 8.36, Tables 1 and 3). Therefore, readers recognized more of the nodules with vascular attachment. This affected the overall sensitivities of readers 1 and 2, which on ROC analysis were significantly higher with nodular vessel attachment (for vessel attachments and no attachments, respectively, reader 1: sensitivity 0.768; 95%

Computer-aided detection of pulmonary nodules

201

performance on ROC analysis (Tables 1 and 4). For consensus of reader 1 with ICAD results none of the factors tested influenced nodule detection or the overall performance on ROC (Tables 1, 3 and 4).

Discussion

Figure 2 Size distribution of nodules. Note range (horizontal line), upper and lower quartiles (rectangle), and median (vertical line) showing abnormal size distribution.

CI ¼ 0.651 – 0.861, and sensitivity 0.369; 95% CI ¼ 0.253 – 0.498; reader 2: sensitivity 0.725; 95% CI ¼ 0.604 – 0.825, and sensitivity 0.354; 95% CI ¼ 0.239 – 0.482) (Table 4). Pleural attachment had a predictive trend towards significance using univariate methods (p ¼ 0:084 – 0:138; OR ¼ 1.78 – 2.05) (Table 3), which was not demonstrated on multivariate testing and did not influence overall

Computer-aided detection of pulmonary nodules has been evaluated in a limited number of studies.12 – 15,23 – 26 In the studies of Lee et al., Awai et al. and Giger et al. the authors referenced true-positive rates of 94, 80 and 72%, respectively, and false-positive rates of 0.08, 0.87 and 1.10 nodules per image section, respectively.10,25,26 Other investigators showed wider ranges of sensitivities of 38 –84% with false-positive rates of 1.0 – 5.8 per examination.27 – 30 However, it is difficult to compare the performance of different published CAD systems from literature data, owing to the diversity of detection algorithms and the lack of a standardized nodule database. In keeping with these results, our study demonstrated a comparable detection rate of 76.3%, with a comparatively lower false-positive rate of 0.55 per examination. As expected, CAD outperformed the readers who showed a nodule detection rate of only 52.6% and false-positive rates of 0.25 and 0.15 per examination, respectively. Consensus reading of one reader with the CAD tool increased the detection

Figure 3 Influence of nodule size on reader and ICAD detection rates using fitted logistic model estimation from logistic regression statistics (error bars depict 95% confidence intervals; nodule size intervals 1 mm). Note significant detection rate decline of readers for nodules smaller than 7.19 mm (logistic regression mean). For ICAD there is no significant detection rate change for any nodule sizes.

202

K. Marten et al.

Table 2 Multivariate analysis II of factors predicting nodule detection Model

Comparison

p

OR

95% CI

r2

CLR Side S1-10 S5-7 S5-7, 10 S1, 5-7, 10

ICAD L vs. R All segments Yes/no Yes/no Yes/no

0.663 0.929 0.792 0.996 0.880

1.276 0.991 1.439 1.010 1.204

0.427 –3.810 0.805 –1.219 0.097 –21.406 0.026 –40.063 0.109 –13.353

0.139

CLR Side S1-10 S5-7 S5-7, 10 S1, 5-7, 10

ICAD þ reader 1 L vs. R All segments Yes/no Yes/no Yes/no

0.168 0.819 0.979 1.0 0.986

0.211 0.965 0.001 0.733 5.21 £ 105

0.023 –1.926 0.713 –1.307 n.e. n.e. n.e.

0.154

CLR Side S1-10 S5-7 S5-7, 10 S1, 5-7, 10

Reader 1 L vs. R All segments Yes/no Yes/no Yes/no

0.902 0.408 0.511 0.679 0.850

0.940 1.076 0.530 0.556 1.219

0.351 –2.516 0.904 –1.281 0.080 –3.524 0.035 –8.923 0.157 –9.462

0.351

CLR Side S1-10 S5-7 S5-7, 10 S1, 5-7, 10

Reader 2 L vs. R All segments Yes/no Yes/no Yes/no

0.903 0.377 0.615 0.561 0.365

0.945 1.080 0.619 2.265 0.390

0.376 –2.373 0.911 –1.280 0.095 –4.019 0.144 –35.591 0.051 –2.994

0.254

p, conditional logistic regression probability; OR, odds ratio; L, left; R, right; CLR, conditional logistic regression; CI, confidence interval of OR; n.e., not estimated; S, lung segment; r 2, pseudo (McFadden) R-square for conditional logistic regression.

rate to 93.3%, with a reduction of false-positives to only 0.1 per examination, demonstrating a clear potential benefit for a clinical reader using computer-aided diagnosis in consensus. To the best of our knowledge, the influence of nodule characteristics such as size, morphology and localization on this performance advantage of CAD has largely remained unaddressed. The influence of nodule size is particularly interesting in the light of the low detection rates of human readers investigating smallest nodules, which is well recognized in previous studies.7,31 Evaluating nodules of 1 – 7 mm in size, Naidich et al. demonstrated an overall detection rate of only 63% by readers and, particularly in diameter ranges below 3 and 1.5 mm, detection rates fell to 48 and 1%, respectively.7,31 Similar observations including a comparison with low-dose CT were repeated by the same group using simulated nodules on chest CT scans.31,32 Therefore, with use of a CAD system clinical users should gain robust detection rates over the whole nodule size range, particularly of smallest lesions, and over the entire spectrum of morphological nodule characteristics, overcoming this limitation of human reader performance. Accordingly, the results of our study revealed a constantly good performance of the ICAD tool over the full range of nodule sizes tested,

whereas the nodule detection rates of readers were significantly dependent on the nodule diameter, being worse than those of the automated detection tool for nodule diameters less than 6 mm, but significantly superior to the CAD system for nodules larger than 12 mm. It remains to be demonstrated whether the presence of such small nodules will have an impact on clinical decision-making for persons with pulmonary metastasis. The performance of human readers can be influenced by morphological nodule characteristics, as well as size, and their relationship to surrounding anatomical structures.31,32 We therefore designed this study to evaluate a potential benefit of the CAD tool by demonstration of performance robustness within the morphological nodule spectrum. Our study demonstrated that, although nodules without vascular attachment were more likely to be detected by the CAD tool, this effect was not sufficiently marked to influence overall CAD performance. However, this specific property of the investigated CAD system should undergo further improvement in the future. No other morphological factors such as nodule relationship to the pleura, matrix calcification or margin characteristics could predict nodule detection, and there was no influence by these

Localization/morphology

n/[%]

Nodules detected

p1

p2

OR

95% CI

Nodules detected

p1

p2

OR

95% CI

Reader 1 þ ICAD

ICAD Part solid or GGO Ill defined margin Pleural attachment Vessel attachment Bronchus attachment Calcification Cavity

2 [1.5] 18 [13.3] 56 [41.5] 69 [51.1] 4 [2.9] 6 [4.4] 0

2 13 42 43 2 3 0 Reader 1

f.e. 1 0.703 0.008 f.e. 0.363 f.e.

f.e. 0.975 0.324 0.0006 f.e. 0.279 f.e.

f.e. 0.980 1.619 0.145 f.e. 0.249 f.e.

f.e. 0.283 –3.397 0.622 –4.217 0.048 –0.435 f.e. 0.020 –3.084 f.e.

2 16 53 65 2 6 0 Reader 2

f.e. 1 0.741 0.198 f.e. 1 f.e.

f.e. 0.678 0.438 0.635 f.e. 0.977 f.e.

f.e. 0.684 1,780 1,410 f.e. 7.79E þ 6 f.e.

f.e. 0.114–4.09 0.414–7.660 0.341–5.833 f.e. n.e. f.e.

Part solid or GGO Ill defined margin Pleural attachment Vessel attachment Bronchus attachment Calcification Cavity

2 [1.5] 18 [13.3] 56 [41.5] 69 [51.1] 4 [2.9] 6 [4.4] 0

2 6 37 47 2 5 0

f.e. 0.082 0.014 ,0.001 f.e. 0.213 f.e.

f.e. 0.498 0.084 0.003 f.e. 0.111 f.e.

f.e. 0.663 2.054 3.665 f.e. 6.296 f.e.

f.e. 0.202 –2.178 0.909 –4.644 1.539 –8.727 f.e. 0.654 –60.619 f.e.

1 9 36 50 2 4 0

f.e. 0.805 0.035 ,0.001 f.e. 0.684 f.e.

f.e. 0.703 0.138 ,0.0001 f.e. 0.224 f.e.

f.e. 1,232 1,784 6,060 f.e. 3,036 f.e.

f.e. 0.420–3.612 0.831–3.831 2.495–14.715 f.e. 0.506–18.201 f.e.

Computer-aided detection of pulmonary nodules

Table 3 Univariate analysis of factors predicting nodule detection

p1, Fisher’s exact test probability; p2, conditional logistic regression probability; OR, odds ratio of p2; CI, confidence interval of OR; n, nodule number; [%], percent of 135 detectable nodules; f.e., factor excluded from analysis; n.e., not estimated; GGO, ground glass opacity

203

204

K. Marten et al.

Figure 4 CTs of pulmonary nodules detected or missed by CAD and the readers. (a) 2 mm pulmonary nodule (arrow) in the apical segment of the upper lobe without vascular attachment. This nodule was missed by both readers, but was detected by CAD. (b) 2 mm pulmonary nodule (arrow) in the right lateral basal lower lobe segment. CAD missed this lesion, whereas both readers correctly identified it. (c) 2 mm nodule in the laterobasal segment of the right lower lobe which was detected by CAD but missed by the readers. (d) 8 mm nodule in the lateral segment of the middle lobe. This nodule was depicted by the readers but missed by CAD.

features on CAD performance, indicating excellent morphological robustness of ICAD. In contrast, the sensitivity for nodule detection by both readers was significantly increased by the presence of vascular nodule attachment. Interestingly, this influence of vessel attachment on reader sensitivity was offset by an opposing effect of the CAD tool during consensus reading, where no significant difference in sensitivity or detection performance of nodules with or without vascular attachment could be demonstrated.

The following limitations of our study need to be addressed. First, the potential influence of nodule localization on detection performance was evaluated focusing on anatomical segments of the lungs instead of differentiating between central and peripheral non-segmental lung zones; but our results do not indicate any influence of nodule localization on the CAD performance supporting the robustness of findings from other tests. Secondly, owing to small numbers, which were not regarded as sufficient for statistical analysis, certain

Computer-aided detection of pulmonary nodules

205

Table 4 ROC analysis of morphological factors predicting nodule detection Nodule group

ICAD

95% CI

Smooth margin

0.805 0.612 1 0.767 0.667 1 0.781 0.750 0.786 0.807 0.680 1 0.710 0.536 1 0.835 0.708 0.889

0.738 –0.873 0.517 –0.701 0.846 –1 0.582 –0.952 0.410 –0.867 0.478 –1 0.678 –0.885 0.616 –0.856 0.492 –0.953 0.725 –0.889 0.564 –0.781 0.753 –1 0.612 –0.808 0.412 –0.657 0.815 –1 0.719 –0.951 0.582 –0.814 0.518 –0.997

auc sens spec Ill defined margin auc sens spec Pleural attachment auc sens spec No pleural attachment auc sens spec Vessel attachment auc sens spec No vessel attachment auc sens spec

ICAD þ reader 1 95% CI 0.922 0.931 0.909 0.944 0.889 1 0.937 0.946 0.929 0.920 0.923 0.923 0.940 0.928 0.944 0.906 0.923 0.889

0.857 –0.988 0.869 –0.970 0.708 –0.989 0.870 –1 0.653 –0.986 0.478 –1 0.860 –1 0.851 –0.989 0.661 –0.998 0.839 –1 0.840 –0.971 0.640 –0.998 0.878 –1 0.839 –0.976 0.727 –0.999 0.792 –1 0.830 –0.975 0.518 –0.997

Reader 1 95% CI

reader 2 95% CI

0.673 0.578 0.773 0.778 0.556 1 0.782 0.714 0.857 0.617 0.474 0.769 0.794 0.768 0.833 0.574 0.369 0.778

0.718 0.578 0.864 0.678 0.500 1 0.798 0.643 0.929 0.643 0.449 0.846 0.817 0.725 0.889 0.620 0.354 0.889

0.572 – 0.775 0.482 – 0.669 0.546 – 0.922 0.660 – 0.896 0.308 – 0.785 0.478 – 1 0.666 – 0.898 0.578 – 0.827 0.572 – 0.982 0.482 – 0.753 0.360 – 0.591 0.462 – 0.950 0.685 – 0.902 0.651 – 0.861 0.586 – 0.964 0.418 – 0.729 0.253 – 0.498 0.400 – 0.972

0.629 – 0.806 0.482 – 0.669 0.651 – 0.970 0.476 – 0.879 0.260 – 0.740 0.478 – 1 0.698 – 0.898 0.504 – 0.766 0.661 – 0.998 0.518 – 0.767 0.336 – 0.566 0.546 – 0.981 0.721 – 0.913 0.604 – 0.825 0.652 – 0.986 0.493 – 0.746 0.239 – 0.482 0.518 – 0.997

CI, confidence interval; auc, estimated area under ROC-curve; sens, sensitivity; spec, specificity; p , 0:05 for comparison between two nodule groups, i.e. feature present vs. feature absent.

morphological nodule characteristics such as part solid matrix, cavitations and bronchus attachment had to be excluded from the study. The rarity of these nodule characteristics reflects our patient collective which consisted of subjects with pulmonary metastatic disease. However, the basic ICAD algorithm was developed for detection of solid lesions and, in keeping with our study data, we do not propose to expand its specification for sub-solid matrix characteristics. In conclusion, our study results add weight to the hypothesis that the evaluated CAD system may be used to replace the second clinical radiologist for evaluation of pulmonary nodules on MSCT, as it increases nodule detection rates, decreases the rates of false-positive findings, and outperforms readers in the detection of smallest lesions or lesions without vascular attachment.

References 1. Kaneko M, Eguchi K, Ohmatsu H, et al. Peripheral lung cancer: screening and detection with low-dose spiral CT versus radiography. Radiology 1996;210:798—802. 2. Sone S, Takashima S, Li F, et al. Mass screening for lung cancer with mobile spiral computed tomography scanner. Lancet 1998;351:1242—5. 3. Henschke CI, McCauley DI, Yankelevitz DF, et al. Early lung cancer action project: overall design and findings from baseline screening. Lancet 1999;354:99—104. 4. Sone S, Li F, Yang ZG, et al. Characteristics of small lung cancers invisible on conventional chest radiography and detected by population based screening using spiral CT. Br J Radiol 2000;73:137—45.

5. Nawa T, Nakagawa T, Kusano S, Kawasaki Y, Sugawara Y, Nakata H. Lung cancer screening using low-dose spiral CT: results of baseline and 1-year follow-up studies. Chest 2002; 122:15—20. 6. Diederich S, Wormanns D, Semik M, et al. Screening for early lung cancer with low-dose spiral CT: prevalence in 817 asymptomatic smokers. Radiology 2002;222:773—81. 7. Swensen SJ, Jett JR, Sloan JA, et al. Screeening for lung cancer with low-dose spiral computed tomography. Am J Respir Crit Care Med 2002;165:508—13. 8. Sobue T, Moriyama N, Kaneko M, et al. Screening for lung cancer with low-dose helical computed tomography: antilung cancer association project. J Clin Oncol 2002;20: 911—20. 9. Yankelevitz DF, Gupta R, Zhao B, Henschke CI. Small pulmonary nodules: evaluation with repeat CT—preliminary experience. Radiology 1999;212:561—6. 10. Giger ML, Bae KT, MacMahon H. Computerized detection of pulmonary nodules in computed tomography images. Invest Radiol 1994;29:459—65. 11. Armato III SG, Giger ML, MacMahon H. Automated detection of lung nodules in CT scans: preliminary results. Med Phys 2001;28:1552—61. 12. Fiebich M, Wietholt C, Renger BC, et al. Automatic detection of pulmonary nodules in low-dose screening thoracic CT examinations. Proc SPIE 1999;3661:1434—9. 13. Fan L, Novak CL, Qian J, Kohl G, Naidich DP. Automatic detection of lung nodules from multi-slice low-dose CT images. Proc SPIE 2001;4322:1828—35. 14. Satoh H, Ukai Y, Niki N, et al. Computer aided diagnosis system for lung cancer based on retrospective helical CT images. Proc SPIE 1999;3661:1324—35. 15. Okumura T, Miwa T, Kako J, Yamamoto S, et al. Image processing for computer-aided diagnosis of lung cancer screening system by CT (LSCT). Proc SPIE 1998;3338: 1314—22. 16. Fan L, Novak CL, Naidich DP, Wei G, Quian J, Ko JP. Improving optimal radiologic interpretation of low-dose multi-slice lung CT studies using ICAD. Radiology 2002;225: 475.

206

17. Krailo MD, Pike MC. Algorithm AS 196. Conditional multivariate logistic analysis of stratified case-control studies. Appl Stat 1984;33:95—103. 18. Smith PG, Pike MC, Hill P, Breslow NE, Day NE. Algorithm AS 162. Multivariate conditional logistic analysis of stratummatched case-control studies. Appl Stat 1981;30:190—7. 19. Howard S. Remark on the paper by Cox D.R.: Regression methods and life tables. J R Stat Soc, Ser B 1972;34:187—220. 20. Press WH, Flamery BP, Teukolsky SA, Vetterling WT. Numerical recipes, the art of scientific computing. Cambridge, UK: Cambridge University Press; 1992. 21. Hanley JA, McNeil BJ. The meaning and use of area under a Receiver Operating Characteristic (ROC) curve. Radiology 1982;143:29—36. 22. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating curves: a nonparametric approach. Biometrics 1988;44: 837—45. 23. Ko JP, Betke M. Chest CT automated nodule detection and assessment of change over time—preliminary experience. Radiology 2001;218:267—73. 24. Croisille P, Souto M, Cova M, et al. Pulmonary nodules: improved detection with vascular segmentation and extraction with spiral CT. Radiology 1995;197:397—401. 25. Awai K, Murao K, Ozawa A, et al. Pulmonary nodules at chest CT: effect of computer-aided diagnosis on radiologists’ detection performance. Radiology 2004;230:347—52.

K. Marten et al.

26. Lee Y, Hara T, Fujita H, Itoh S, Ishigaki T. Automated detection of pulmonary nodules in helical CT images based on an improved templated matching technique. IEEE Trans Med Imaging 2001;20:595—604. 27. Zhao B, Gamsu G, Ginsber MS, Jiang L, Schwartz LH. Automatic detection of small lung nodules on CT utilizing a local density maximum algorithm. J Appl Clin Med Phys 2003;4:248—60. 28. Wormanns D, Fiebich M, Saidi M, Diederich S, Heindel W. Automatic detection of pulmonary nodules at spiral CT: clinical application of a computer-aided diagnosis system. Eur Radiol 2002;12:1052—7. 29. Armato III SG, Giger ML, Moran CJ, Blackburn JT, Doi K, MacMahon H. Computerized detection of pulmonary nodules on CT scans. RadioGraphics 1999;19:1303—11. 30. Armato III SG, Li F, Giger ML, MacMahon H, Sone S, Doi K. Lung cancer: performance of automated lung nodule detection applied to cancers missed in a CT screening program. Radiology 2002;225:685—92. 31. Naidich DP, Rusinek H, McGuinness G, Leitman B, McCauley DI, Henschke CI. Variables affecting pulmonary nodule detection with computed tomography: evaluation with three-dimensional computer simulation. J Thorac Imaging 1993;8:291—9. 32. Rusinek H, Naidich DP, McGuinness G, et al. Pulmonary nodule detection: low-dose versus conventional CT. Radiology 1998;209:243—9.