- Email: [email protected]
Diagnostic accuracy of Fischer Senoscan Digital Mammography versus screen-film mammography in a diagnostic mammography population1
Diagnostic Accuracy of Fischer SenoScan Digital Mammography Versus Screen-Film Mammography in a Diagnostic Mammography Population1 Elodia Cole, MS, Etta D. Pisano, MD, Mary Brown, BS, Cherie Kuzmiak, DO, M. Patricia Braeuning, MD, Hak Hee Kim, MD, Roberta Jong, MD, FRCPC, Ruth Walsh, MD
Rationale and Objectives. To compare the diagnostic accuracy of the Fischer Senoscan Digital Mammography System with that of standard screen–film mammography in a population of women presenting for screening or diagnostic mammography. Materials and Methods. Enrollment of patients took place at six different breast-imaging centers between 1997 and 1999. A total of 247 cases were selected for inclusion in the final reader study. All known cancer cases were included (111) from all six participating sites representing 45% of the total cases. The remaining 136 cases (55%) were randomly selected from all available benign or negative cases from three of the six sites. A complete case consisted of both a (unilateral or bilateral) digital and screen–film mammogram of the same patient. Eight radiologists interpreted the cases in laser-printed digital and screen–film hardcopy formats. The study was designed to detect differences of 0.05 in the ROC area under the curve (AUC) between digital and screen–film radiologist interpretation performance. Results. The average AUC for the Senoscan digital was 0.715 for the 8 readers. The average AUC for screen–film was 0.765. The difference AUC of ⫺0.05 falls within the 95% confidence interval (⫺0.101, 0.002). The average sensitivity was 66% and specificity 67% for SenoScan full-field digital mammography. The average screen–film mammography sensitivity and specificity were 74% and 60%, respectively. Conclusion. No statistically significant difference in diagnostic accuracy between the Fischer Senoscan and screen–film mammography was detected in this study. Key Words. Digital mammography; FDA clinical trial; ROC analysis; observer performance; diagnostic accuracy. ©
AUR, 2004
Acad Radiol 2004; 11:879 – 886 1 From the Department of Radiology, 106 Mason Farm Road, CB# 7515, Chapel Hill, NC 27599-7510 (E.C., E.D.P., M.B., C.K.) and Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill (E.C., E.D.P., M.B.), Department of Radiology, Christ Hospital, Cincinnati, OH, (M.P.B.) Department of Radiology, Kangnam St. Mary’s Hospital, The Catholic University of Korea, Seoul, South Korea (H.H.K.), Department of Medical Imaging, Sunnybrook & Women’s College Health Sciences Centre, The University of Toronto, Toronto, Canada (R.J.), and Department of Radiology, Duke University Medical Center, Durham, NC (R.W.) Received November 18, 2003; revision requested and received February 23, 2004; accepted April 13. Address correspondence to E.C. e-mail: [email protected]
©
AUR, 2004 doi:10.1016/j.acra.2004.04.003
The widespread use of mammography as a tool for early detection of breast cancer has resulted in a decrease in mortality due to breast cancer (1). The current gold standard, screen–film mammography, has undergone many technical improvements in its over–30-year history leading to better image quality, acquisition techniques (positioning and compression), and interpretation protocols (1, 2). While screen–film mammography was being improved, new research efforts emerged focusing on image quality improvement after acquisition, such as image processing algorithms and computer-aided diagnosis and detection systems (CAD) (3,4,5), which would hopefully result in improved visibility of lesions. Many of these
879
COLE ET AL
research efforts depended on having images in a digital format, which meant digital conversion of screen–film mammograms. Limitations of screen–film mammography, as a result of the acquisition medium and display medium being the same (film), led to the development of direct digital acquisition systems for mammography, which provided decoupling of acquisition from display, allowing for ease of application of image processing or CAD and independence from the method of display, either on high-resolution monitors (softcopy) or printed to film (hardcopy). As with any new medical technology, the clinical utility of digital mammography must be established through clinical trials before Food and Drug Administration (FDA) approval can be obtained for widespread clinical implementation. This report documents an FDA clinical trial used to assess the accuracy of the Fischer SenoScan Digital Mammography System. This study was performed for the sole purpose of obtaining Food and Drug Administration (FDA) approval for the Fischer SenoScan. As such, the analysis presented in this paper was designed to determine the diagnostic accuracy of the Fischer SenoScan Digital Mammography System compared with standard screen–film mammography in the population of women scheduled to undergo biopsy presenting for screening or problem-solving mammography, using Receiver Operating Characteristic Curve (ROC) methodology. The hypothesis for this research was that the difference in the area under the ROC curve (AUC) will be no greater than 0.05 when SenoScan digital mammography is compared with screen–film mammography.
METHODS AND MATERIALS Image Acquisition System The digital mammography system used in this study is the Fischer SenoScan Full-field Digital Mammography System (Fischer Imaging Corporation, Denver, CO). This device is a scanning slot system. The term “scanning slot” describes the geometry and the motion of the detector. The prototype system used in this study had a slot area of 1 cm ⫻ 22 cm acquired in less than 6 seconds (6). The movable detector was scanned over a 30-cm area resulting in a 22 cm ⫻ 30 cm image (or 3072 ⫻ 4800 pixels). One of the benefits of the scanning slot architecture is that the X-ray beam is confined to a 1-cm width, resulting in images with very little scatter and no dose penalty associated with absorption of primary radiation by a grid (6). The spatial resolution of the SenoScan was 9
880
Academic Radiology, Vol 11, No 8, August 2004
line pairs/mm in the standard view mode used in this study and a contrast resolution of 4096 discernable grey values. The slot area, scanning area, and image size were the same for all sites. Exposure parameters were set using Fischer SenoScan technique charts based on breast thickness and parenchymal density. Enrollment Eligibility and Exclusion Criteria There were three enrollment phases to this study, each with its own eligibility criteria. The first phase of case acquisition involved the enrollment of women who had been recommended for breast biopsy, who had had abnormal screening screen–film mammograms, or who had symptoms that led to their referral for diagnostic mammography at Brooke Army Hospital, the University of North Carolina at Chapel Hill (UNC), Sally Jobe Clinic, and Thomas Jefferson University. Five hundred and sixty women were enrolled during phase 1. Phase 1 enrollment was terminated due to changes in the Food and Drug Administration (FDA) approval guidelines for Digital Mammography systems (7), which changed from a requirement of agreement between readers without reference to the true breast cancer status of patients to a standard which required the manufacturers of digital mammography systems to prove equivalence to screen–film mammography based on reader performance with reference to breast cancer status. Standard clinical practice was used in determination of breast cancer status. After diagnostic imaging, for some patients, short-term followup may have been requested by the interpreting radiologist as opposed to biopsy. If after one year of followup the mammograms of these patients were stable, they were classified as negative for the purposes of this study. With only 25 cancers in the 560 patients enrolled in phase 1, the criteria for eligibility were changed to increase the number of biopsy-proven lesions, thereby increasing the number of cancers. Women were eligible for phase 2 enrollment if they were scheduled to undergo breast biopsy, either percutaneous or open surgical. An additional 101 cancers were added to the case set under phase 2. A third and final phase was initiated to try to obtain more cancers, this time drawing from pre-existing SenoScan digital mammography case sets obtained from the University of California at San Francisco and the University of Toronto. These mammograms were taken of women who were not recruited to this study but were recruited to other clinical trials that had the same eligibility criteria as this study. All of the women whose mam-
Academic Radiology, Vol 11, No 8, August 2004
mograms were included in Phase 3 of case acquisition signed consent forms that permitted the use of their images in additional research as needed. A total of 15 cancer cases were added to the case totals under Phase 3. IRB approval was obtained for all three phases of patient recruitment at all institutions contributing mammograms to this study. A woman who otherwise met the eligibility criteria was excluded from the trial if she was under the age of 21 years, if she was pregnant or thought she might be pregnant, or if she was unable to give informed consent for any reason (e.g., psychiatric or neurologic disability or language barriers). At each of the participating institutions, the recruiting radiologist or his or her designee determined the eligibility of the women presenting for problem-solving mammography. Attempts were made to recruit all consecutive eligible women at the participating institutions. Research assistants at each institution approached eligible women regarding participation in the clinical trial when they presented for problem-solving mammography. Patients gave informed consent to participate at the time of recruitment to the study. Each patient prior to imaging with the digital mammography system signed an IRB-approved consent form. Imaging Sequence At the time of the diagnostic visit, the patient underwent screen–film mammography as clinically indicated. At the same diagnostic visit, the patient then underwent digital mammography using the Fischer SenoScan fullfield digital mammography system. All women underwent two-view mammograms (both cranio-caudal and mediolateral oblique views) of one or both breasts using the digital system. Standard clinical protocols were used to determine whether each woman required unilateral or bilateral mammograms. For large-breasted women, as many cranio-caudal and medio-lateral oblique views were performed as were determined necessary by the imaging technologist to include each breast in its entirety. This is in accordance with standard clinical practice for the performance of screen–film mammography. All medio-lateral oblique and cranio-caudal digital mammograms obtained on the enrolled patients were considered the digital mammogram for the experimental reading study. The original eligibility screen–film mammograms were copied at the clinical sites and the originals were transmitted to the University of North Carolina for use in the reader study.
DIAGNOSTIC ACCURACY OF FISCHER SENOSCAN
All digital mammograms were archived from the SenoScan systems at each of the participating sites to optical disk and sent to the Fischer Imaging Corporation where a mammography technologist with 5 years of digital mammography experience manually adjusted the contrast and brightness of (intensity-windowed) all digital images. At the time of this study, there was no automatic image-processing algorithm for the Fischer SenoScan in preparation for film printing. The digital images were then printed on Kodak Dryview DVM film with a Kodak Dryview laser imager 43 microns pixel size, 4096 ⫻ 5120 pixels (Kodak Corporation, Rochester, NY). Of the 676 total cases available for the reader study, only 247 cases were included in the reader study because of time constraints for windowing and printing cases. The smaller number of cases also minimized the time it would take each individual reader to interpret the cases in a controlled environment and therefore the entire length of the reader study portion of the project. One hundred twenty cases originated from a screening visit and one hundred twenty-six cases originated from a diagnostic visit. For one case, the visit type was not recorded. Of the 247 reader study cases, there were 136 benign or otherwise negative cases and 111 malignant cases. For each of the 111 malignant cases, there was only one malignant lesion identified per case. There could have been none, one, or more than one incidental benign lesions for the 247 cases. There were 51 unilateral mammograms and 196 bilateral mammograms in the case set. Non-cancer cases were selected by taking a stratified random sample from the remaining cases. The stratification was by institution so that cases would be included in proportion to the number of cases recruited to the protocol at each institution. The truth about breast cancer status for each patient whose mammograms were included in the reader study was determined by either biopsy or one-year clinical follow-up using screen–film mammography. A single experienced radiologist with 19 years of mammography experience (EDP) at the University of North Carolina (UNC) reviewed the pathology reports on all patients and coded the available histopathological diagnosis. The radiologists at the enrolling sites, who interpreted the patient’s screen–film studies, coded lesion locations. In the event of missing lesion location information, the same radiologist at UNC (EDP) coded lesion location using needle localization and imaging-guided core biopsy clinical reports and patient records. All patients classified as negative or benign for this study underwent follow-up screen– film mammography one year after they received their dig-
881
COLE ET AL
ital mammogram. None of these patients showed evidence of malignancy by screen–film mammography or clinically for a minimum of one year after their Fischer SenoScan digital mammogram. Research assistants at each of the participating sites coded the results of the follow-up screen–film mammogram. For the 111 malignant cases, the distribution of lesion sizes and tumor stage using TNM criteria were collected for 72 lesions that underwent core biopsy. The remaining 39 lesions were found cancerous based on fine needle aspiration (FNA). Of the malignancies, 42% were less than 1 cm in size (Table 1) and 75% were stage 0 or 1 (Table 2). Reader Study Eight radiologists participated in a reader study, which was conducted at the University of North Carolina. Six readers were ABR-certified, one Canadian-certified, and one Korean-certified with years of mammography experience ranging from 2-25 years (mean 8.5 yrs). Of those, six had experience in interpreting digital mammograms through direct clinical practice or through participation in other digital mammography studies at UNC. The remaining two radiologists, who had no experience reading digital mammograms in our research lab, were trained in interpreting SenoScan digital mammograms by reading ten printed digital mammograms that were not included in this reader study but that were prepared and printed by the same imaging technologist who prepared and printed the actual study cases. The two readers were provided with immediate feedback regarding the pathologically proven lesions present in these ten training mammograms. All readers were trained in the use of the forms used in the study just before interpreting the study examinations. There were 247 cases, each consisting of a screen–film mammogram and a digital mammogram. The images were divided into two groups, A and B, each consisting of 247 cases. The screen–film or digital mammogram for each case set was randomly assigned to one of the two groups using standard statistical randomization procedures. All readers read group A cases first during their individual reading sessions, followed by a minimum 4-week washout period to minimize memory effects followed by the reading of group B cases. A research assistant (EBC) loaded 50 to 100 cases at a time onto a multiviewer utilizing appropriate masking for extraneous light. Readers were required to take 5-minute breaks every 50 minutes, or as necessary.
882
Academic Radiology, Vol 11, No 8, August 2004
Table 1 Size Distribution of Core-Biopsied Cancerous Lesions Size mm
Frequency count
Percent
0 1 2 3 4 5 7 8 9 10 11 12 13 14 15 16 17 18 20 21 23 25 26 30 35 37 42 44 45 50 52 56 60 71 109
4 2 2 2 3 2 2 1 5 7 1 3 2 1 4 1 2 4 4 1 1 2 1 4 1 1 1 1 1 1 1 1 1 1 1
5.56 2.78 2.78 2.78 4.17 2.78 2.78 1.39 6.94 9.72 1.39 4.17 2.78 1.39 5.56 1.39 2.78 5.56 5.56 1.39 1.39 2.78 1.39 5.56 1.39 1.39 1.39 1.39 1.39 1.39 1.39 1.39 1.39 1.39 1.39
***The research assistant also recorded data onto paper forms as readers interpreted the examinations. The readers were required to provide the information in a standardized format following a specifically designed, structured report. If clinically relevant abnormalities were found, the research assistant recorded this information on lesion-specific forms. The lesion location and probability for malignancy (based on a 5-point scale, where the ratings of 1— definitely not malignant, 2—probably not malignant, 3—possibly malignant, 4 —probably malignant, and 5— definitely malignant) were used to classify the likelihood of cancer in each case. The BI-RADS standard scale for likelihood of cancer classification was not used
DIAGNOSTIC ACCURACY OF FISCHER SENOSCAN
Academic Radiology, Vol 11, No 8, August 2004
Table 2 T-stage of core biopsied cancerous lesions Size
Frequency count
Percent
Tis T1 T1a T1b T1c T2 T3 T4 TX
15 11 4 11 13 11 4 2 1
20.83 15.28 5.56 15.28 18.06 15.28 5.56 2.78 1.39
because it does not readily lend itself to ROC analysis since it is not a continuous scale. While all of this information was collected during the course of the study, only the lesion location and probability for malignancy ratings were used for the statistical analysis. As the study progressed, a subset of screen–film mammograms was removed either for patient care purposes or because the originating site requested their return, so some readers did not read all screen–film mammograms that were initially selected for inclusion in the study. No digital cases were removed from the study once it began. Table 3 shows the case distributions of paired digital and screen–film cases interpreted by each reader. Standard statistical methods were used to address missing data. The readers’ interpretations were compared with the lesion locations identified in the clinical reports at each of the enrolling sites. Diagnosis was obtained from the pathology reports if the lesions were biopsied, or from one year of follow-up for those lesions that did not go to biopsy. Both lesion location and diagnosis were the only relevant pieces of information collected during the course of the reader study that were required for conducting the AUC analysis planned. First the lesion location provided by the readers was checked with the known true location for each case. Second the probability of malignancy, where a rating of 1 or 2 was called benign and a rating of 3, 4, or 5 was called malignant, was checked against the true diagnosis for each case. Statistical Analysis The 95% confidence interval for the difference of the mean AUCs (digital–film) was determined by applying the approach described by Obuchowski (8). Only the methods for determining the confidence interval were used from the Obuchowski paper even though there are
additional tests for equivalence that are mentioned. The confidence interval for the difference in mean AUC was the only information required by the FDA for proving equivalence. The hypothesis that the difference in AUC between SenoScan digital and screen–film mammography is no larger than 0.05 was selected in accordance with FDA guidelines for pre-market approval (7), where the low limit of ⫺0.05 (digital AUC⫺screen–film AUC) (representative of screen–film providing better performance than digital) would still be considered acceptable for FDA approval of a digital mammography system. Both the screen–film interpretation forms as well as the digital interpretation forms were required for each case. If the reader did not read a case in the screen–film modality due to it being removed from the study, it was excluded from the analysis for that particular reader’s case set. The determination of sensitivity and specificity was based on the probability of malignancy ratings described above. The cut point used for calculating sensitivity and specificity was 1–2 for benign and 3–5 for malignant. ROCKit statistical software (C. Metz, University of Chicago, Chicago IL) was used for the primary analysis of estimating the area under the ROC curves. SAS (SAS Institute, Cary, NC) was used for generating the summary statistics.
RESULTS The average AUC for SenoScan digital mammography was 0.715. The average AUC for screen–film mammography was 0.765. The 95% confidence interval for the average difference of the mean AUCs [(SenoScan digital)⫺(screen film)] was (⫺0.101 to 0.002). Table 4 shows the individual AUCs and standard error for Sen-
Table 3 Total number of cases read per reader Reader ID
Negative cases
Positive cases
Total cases
2 3 4 5 6 7 8 9
112 124 110 112 112 111 112 111
93 106 90 90 89 91 91 90
205 230 200 200 201 202 203 201
883
COLE ET AL
Academic Radiology, Vol 11, No 8, August 2004
Table 4 Areas Under the ROC Curve for Each Reader for SenoScan Digital and Screen-Film
Reader ID
Senoscan digital AUC(SE)
Screen film AUC(SE)
95% CI for difference in mean AUCs Digital–Screen Film
2 3 4 5 6 7 8 9
.678 (.038) .758 (.033) .694 (.036) .777 (.031) .730 (.038) .696 (.042) .653 (.051) .736 (.039)
.737 (.037) .807 (.029) .748 (.036) .769 (.034) .810 (.031) .800 (.033) .681 (.045) .766 (.035)
(⫺.1411, .0244) (⫺.1134, .0161) (⫺.1387, .0304) (⫺.0634, .0780) (⫺.1492, ⫺.0096)* (⫺.1836, ⫺.0246)* (⫺.1280, .0725) (⫺.1081, .0480)
SE ⫽ Standard Error *Readers for which the AUC for digital is significantly less than the AUC for screen film
oScan digital and screen–film along with individual 95% confidence intervals for difference between SenoScan digital and screen–film for each of the 8 readers. The average sensitivity for SenoScan digital mammography was 0.66. The average sensitivity for screen–film mammography was 0.74. A comparison of each of the eight readers’ sensitivities for digital and screen–film mammography is shown in Figure 1. The average specificity for SenoScan digital mammography was 0.67. The average specificity for screen–film mammography was 0.60. A comparison of each of the eight readers’ specificities for digital and screen–film mammography is shown in Figure 2.
DISCUSSION All of the cases included in this study had some type of abnormality that led to their inclusion in this study, making the population under study a diagnostic one with some cases entering the study without symptoms at the time of an abnormal screening mammogram. The results described here are consistent with similar studies comparing digital to screen–film mammography. Another study conducted on Fischer SenoScan digital mammography in a population of patients with dense breasts resulted in AUC, sensitivity, and specificity scores of 0.663, 54.1%, and 64.6%, respectively, for masses; and AUC, sensitivity, and specificity results of 0.633, 59.9%, and 76.9%, respectively, for calcifications (9). Preliminary results
884
from a digital mammography versus screen–film mammography study conducted by Lewin et al (10) with the General Electric digital mammography system in a screening population showed sensitivity results of 60% for digital mammography versus 63% for screen–film mammography. One of the limitations of this study is that enrollment was based on findings that were seen on the eligibility screen–film mammograms. This leads to a sensitivity bias toward screen–film mammography and a specificity bias toward digital. This sensitivity bias toward screen–film mammography and specificity bias toward digital was also noted by Lewin et al (10) in their work. There is a large digital versus screen–film study currently underway that has corrected for this bias by enrolling women who are not known to have abnormal screen–film mammograms and random assignment of the initial mammogram to digital or screen–film at enrollment. This study is also limited by the presentation display of the images in printed format instead of softcopy using high-resolution monitors. The softcopy display benefits of image contrast adjustment, panning and zooming to full resolution, and image processing were not available for our readers. This, of course, may have affected reader interpretation performance. Another study comparing softcopy display to hardcopy display of Fischer SenoScan digital mammograms showed no significant difference in performance between the two modalities (11). Larger case sets with more readers would be ideal for a follow-up study for the diagnostic population, and one is underway for various digital mammography systems including the SenoScan using softcopy display. Another limitation of this study is in the possible bias introduced as a result of different numbers of cases used to determine the individual reader AUCs due to the loss of screen–film mammograms intermittently throughout the course of the study. As original screen–film mammograms were used for this study, they were always available if needed for clinical purposes, or in the case of one site, for an overlapping reader study. This does point out one of the benefits of digital mammography over screen– film mammography: there is no loss in image quality with copy films. The statistical analysis presented here includes the entire analysis presented to the FDA for an equivalence study to screen–film mammography of the Fischer SenoScan digital mammography system in hardcopy presentation. As such, this presents a limitation of the study in that more rigorous analyses, such as lesion-specific per-
Academic Radiology, Vol 11, No 8, August 2004
DIAGNOSTIC ACCURACY OF FISCHER SENOSCAN
Figure 1. The individual reader mean sensitivity with SenoScan Digital in a diagnostic population trend toward lower sensitivity for SenoScan Digital.
formance, breast density–specific performance, and the reporting of P values were not conducted. Again it must be pointed out that the analysis presented here represents the data analysis requirements for FDA approval; only the
statistical analysis required was reported here. As our results are specific to the Fischer SenoScan digital mammography system, it would be inappropriate to extrapolate these results to the performance of other digital mammog-
Figure 2. The individual reader mean specificity with SenoScan Digital in a diagnostic population trend toward higher specificity than screen–film.
885
COLE ET AL
Academic Radiology, Vol 11, No 8, August 2004
raphy systems. Since the time of this trial, there have been hardware and software modifications that could possibly lead to different outcomes then those obtained here. Based on the AUC of this study, given the size of the 95% confidence interval and its inclusion of zero, the Fischer SenoScan Digital Mammography System is minimally equivalent to screen–film mammography for a diagnostic or problem-solving mammography population. As most of the confidence interval is negative, it is possible that a study with more power would show the Fischer SenoScan as inferior to screen–film mammography. REFERENCES 1. Sickles EA. Breast imaging: from 1965 to the present. Radiology 2000; 215:1–16. 2. Hendee WR. History and status of x-ray mammography. Health Phys 1995; 69:636 – 648. 3. Chan HP, Vyborny CJ, MacMahon H, et al. digital mammography. ROC studies of the effects of pixel size and unsharp-mask filtering on the detection of subtle microcalcifications. Invest Radiol 1987; 22:581– 589. 4. Nawano S, Murakami K, Moriyama N, et al. Computer-aided diagnosis in full digital mammography. Invest Radiol 1999; 34:310 –316.
5. Betal D, Roberts N, Whitehouse GH. Segmentation and numerical analysis of microcalcifications on mammograms using mathematical morphology. Br J Radiol 1997; 70:903–917. 6. Tesic MM, Piccaro MF, Munier B. Full field digital mammography scanner. Eur J Radiol 1999; 31:2–17. 7. Premarket Applications for Digital Mammography Systems: Final Guidance for Industry and FDA. Rockville, MD: Center for Devices and Radiological Health, U.S. Dept of Health and Human Services Food and Drug Administration; 2001. CDRH publication 983. 8. Obuchowski NA. Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol1995;2 Suppl 1:S22-9. 9. Cole EB, Pisano ED, Kistner EO, et al. Diagnostic accuracy of digital mammography in patients with dense breasts presenting for problemsolving mammography-image processing and lesion type effects. Radiology 2003; 226:153–160. 10. Lewin JM, Hendrick RE, D’Orsi CJ, et al. Comparison of full-field digital mammography with screen–film mammography for cancer detection: results of 4,945 paired examinations. Radiology 2001; 218: 873– 880. 11. Pisano ED, Cole EB, Kistner EO, et al. Digital mammography interpretation: Comparison of the speed and accuracy of softcopy versus printed film display. Radiology 2002; 223:483– 488.
The Association of University Radiologists wishes to thank the following companies for their generous support of the AUR Junior Membership Program:
886
4th Year Radiology Residents
3rd Year Radiology Residents
Siemens Medical Systems, Inc.
Berlex Laboratories
Rationale and Objectives. To compare the diagnostic accuracy of the Fischer Senoscan Digital Mammography System with that of standard screen–film mammography in a population of women presenting for screening or diagnostic mammography. Materials and Methods. Enrollment of patients took place at six different breast-imaging centers between 1997 and 1999. A total of 247 cases were selected for inclusion in the final reader study. All known cancer cases were included (111) from all six participating sites representing 45% of the total cases. The remaining 136 cases (55%) were randomly selected from all available benign or negative cases from three of the six sites. A complete case consisted of both a (unilateral or bilateral) digital and screen–film mammogram of the same patient. Eight radiologists interpreted the cases in laser-printed digital and screen–film hardcopy formats. The study was designed to detect differences of 0.05 in the ROC area under the curve (AUC) between digital and screen–film radiologist interpretation performance. Results. The average AUC for the Senoscan digital was 0.715 for the 8 readers. The average AUC for screen–film was 0.765. The difference AUC of ⫺0.05 falls within the 95% confidence interval (⫺0.101, 0.002). The average sensitivity was 66% and specificity 67% for SenoScan full-field digital mammography. The average screen–film mammography sensitivity and specificity were 74% and 60%, respectively. Conclusion. No statistically significant difference in diagnostic accuracy between the Fischer Senoscan and screen–film mammography was detected in this study. Key Words. Digital mammography; FDA clinical trial; ROC analysis; observer performance; diagnostic accuracy. ©
AUR, 2004
Acad Radiol 2004; 11:879 – 886 1 From the Department of Radiology, 106 Mason Farm Road, CB# 7515, Chapel Hill, NC 27599-7510 (E.C., E.D.P., M.B., C.K.) and Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill (E.C., E.D.P., M.B.), Department of Radiology, Christ Hospital, Cincinnati, OH, (M.P.B.) Department of Radiology, Kangnam St. Mary’s Hospital, The Catholic University of Korea, Seoul, South Korea (H.H.K.), Department of Medical Imaging, Sunnybrook & Women’s College Health Sciences Centre, The University of Toronto, Toronto, Canada (R.J.), and Department of Radiology, Duke University Medical Center, Durham, NC (R.W.) Received November 18, 2003; revision requested and received February 23, 2004; accepted April 13. Address correspondence to E.C. e-mail: [email protected]
©
AUR, 2004 doi:10.1016/j.acra.2004.04.003
The widespread use of mammography as a tool for early detection of breast cancer has resulted in a decrease in mortality due to breast cancer (1). The current gold standard, screen–film mammography, has undergone many technical improvements in its over–30-year history leading to better image quality, acquisition techniques (positioning and compression), and interpretation protocols (1, 2). While screen–film mammography was being improved, new research efforts emerged focusing on image quality improvement after acquisition, such as image processing algorithms and computer-aided diagnosis and detection systems (CAD) (3,4,5), which would hopefully result in improved visibility of lesions. Many of these
879
COLE ET AL
research efforts depended on having images in a digital format, which meant digital conversion of screen–film mammograms. Limitations of screen–film mammography, as a result of the acquisition medium and display medium being the same (film), led to the development of direct digital acquisition systems for mammography, which provided decoupling of acquisition from display, allowing for ease of application of image processing or CAD and independence from the method of display, either on high-resolution monitors (softcopy) or printed to film (hardcopy). As with any new medical technology, the clinical utility of digital mammography must be established through clinical trials before Food and Drug Administration (FDA) approval can be obtained for widespread clinical implementation. This report documents an FDA clinical trial used to assess the accuracy of the Fischer SenoScan Digital Mammography System. This study was performed for the sole purpose of obtaining Food and Drug Administration (FDA) approval for the Fischer SenoScan. As such, the analysis presented in this paper was designed to determine the diagnostic accuracy of the Fischer SenoScan Digital Mammography System compared with standard screen–film mammography in the population of women scheduled to undergo biopsy presenting for screening or problem-solving mammography, using Receiver Operating Characteristic Curve (ROC) methodology. The hypothesis for this research was that the difference in the area under the ROC curve (AUC) will be no greater than 0.05 when SenoScan digital mammography is compared with screen–film mammography.
METHODS AND MATERIALS Image Acquisition System The digital mammography system used in this study is the Fischer SenoScan Full-field Digital Mammography System (Fischer Imaging Corporation, Denver, CO). This device is a scanning slot system. The term “scanning slot” describes the geometry and the motion of the detector. The prototype system used in this study had a slot area of 1 cm ⫻ 22 cm acquired in less than 6 seconds (6). The movable detector was scanned over a 30-cm area resulting in a 22 cm ⫻ 30 cm image (or 3072 ⫻ 4800 pixels). One of the benefits of the scanning slot architecture is that the X-ray beam is confined to a 1-cm width, resulting in images with very little scatter and no dose penalty associated with absorption of primary radiation by a grid (6). The spatial resolution of the SenoScan was 9
880
Academic Radiology, Vol 11, No 8, August 2004
line pairs/mm in the standard view mode used in this study and a contrast resolution of 4096 discernable grey values. The slot area, scanning area, and image size were the same for all sites. Exposure parameters were set using Fischer SenoScan technique charts based on breast thickness and parenchymal density. Enrollment Eligibility and Exclusion Criteria There were three enrollment phases to this study, each with its own eligibility criteria. The first phase of case acquisition involved the enrollment of women who had been recommended for breast biopsy, who had had abnormal screening screen–film mammograms, or who had symptoms that led to their referral for diagnostic mammography at Brooke Army Hospital, the University of North Carolina at Chapel Hill (UNC), Sally Jobe Clinic, and Thomas Jefferson University. Five hundred and sixty women were enrolled during phase 1. Phase 1 enrollment was terminated due to changes in the Food and Drug Administration (FDA) approval guidelines for Digital Mammography systems (7), which changed from a requirement of agreement between readers without reference to the true breast cancer status of patients to a standard which required the manufacturers of digital mammography systems to prove equivalence to screen–film mammography based on reader performance with reference to breast cancer status. Standard clinical practice was used in determination of breast cancer status. After diagnostic imaging, for some patients, short-term followup may have been requested by the interpreting radiologist as opposed to biopsy. If after one year of followup the mammograms of these patients were stable, they were classified as negative for the purposes of this study. With only 25 cancers in the 560 patients enrolled in phase 1, the criteria for eligibility were changed to increase the number of biopsy-proven lesions, thereby increasing the number of cancers. Women were eligible for phase 2 enrollment if they were scheduled to undergo breast biopsy, either percutaneous or open surgical. An additional 101 cancers were added to the case set under phase 2. A third and final phase was initiated to try to obtain more cancers, this time drawing from pre-existing SenoScan digital mammography case sets obtained from the University of California at San Francisco and the University of Toronto. These mammograms were taken of women who were not recruited to this study but were recruited to other clinical trials that had the same eligibility criteria as this study. All of the women whose mam-
Academic Radiology, Vol 11, No 8, August 2004
mograms were included in Phase 3 of case acquisition signed consent forms that permitted the use of their images in additional research as needed. A total of 15 cancer cases were added to the case totals under Phase 3. IRB approval was obtained for all three phases of patient recruitment at all institutions contributing mammograms to this study. A woman who otherwise met the eligibility criteria was excluded from the trial if she was under the age of 21 years, if she was pregnant or thought she might be pregnant, or if she was unable to give informed consent for any reason (e.g., psychiatric or neurologic disability or language barriers). At each of the participating institutions, the recruiting radiologist or his or her designee determined the eligibility of the women presenting for problem-solving mammography. Attempts were made to recruit all consecutive eligible women at the participating institutions. Research assistants at each institution approached eligible women regarding participation in the clinical trial when they presented for problem-solving mammography. Patients gave informed consent to participate at the time of recruitment to the study. Each patient prior to imaging with the digital mammography system signed an IRB-approved consent form. Imaging Sequence At the time of the diagnostic visit, the patient underwent screen–film mammography as clinically indicated. At the same diagnostic visit, the patient then underwent digital mammography using the Fischer SenoScan fullfield digital mammography system. All women underwent two-view mammograms (both cranio-caudal and mediolateral oblique views) of one or both breasts using the digital system. Standard clinical protocols were used to determine whether each woman required unilateral or bilateral mammograms. For large-breasted women, as many cranio-caudal and medio-lateral oblique views were performed as were determined necessary by the imaging technologist to include each breast in its entirety. This is in accordance with standard clinical practice for the performance of screen–film mammography. All medio-lateral oblique and cranio-caudal digital mammograms obtained on the enrolled patients were considered the digital mammogram for the experimental reading study. The original eligibility screen–film mammograms were copied at the clinical sites and the originals were transmitted to the University of North Carolina for use in the reader study.
DIAGNOSTIC ACCURACY OF FISCHER SENOSCAN
All digital mammograms were archived from the SenoScan systems at each of the participating sites to optical disk and sent to the Fischer Imaging Corporation where a mammography technologist with 5 years of digital mammography experience manually adjusted the contrast and brightness of (intensity-windowed) all digital images. At the time of this study, there was no automatic image-processing algorithm for the Fischer SenoScan in preparation for film printing. The digital images were then printed on Kodak Dryview DVM film with a Kodak Dryview laser imager 43 microns pixel size, 4096 ⫻ 5120 pixels (Kodak Corporation, Rochester, NY). Of the 676 total cases available for the reader study, only 247 cases were included in the reader study because of time constraints for windowing and printing cases. The smaller number of cases also minimized the time it would take each individual reader to interpret the cases in a controlled environment and therefore the entire length of the reader study portion of the project. One hundred twenty cases originated from a screening visit and one hundred twenty-six cases originated from a diagnostic visit. For one case, the visit type was not recorded. Of the 247 reader study cases, there were 136 benign or otherwise negative cases and 111 malignant cases. For each of the 111 malignant cases, there was only one malignant lesion identified per case. There could have been none, one, or more than one incidental benign lesions for the 247 cases. There were 51 unilateral mammograms and 196 bilateral mammograms in the case set. Non-cancer cases were selected by taking a stratified random sample from the remaining cases. The stratification was by institution so that cases would be included in proportion to the number of cases recruited to the protocol at each institution. The truth about breast cancer status for each patient whose mammograms were included in the reader study was determined by either biopsy or one-year clinical follow-up using screen–film mammography. A single experienced radiologist with 19 years of mammography experience (EDP) at the University of North Carolina (UNC) reviewed the pathology reports on all patients and coded the available histopathological diagnosis. The radiologists at the enrolling sites, who interpreted the patient’s screen–film studies, coded lesion locations. In the event of missing lesion location information, the same radiologist at UNC (EDP) coded lesion location using needle localization and imaging-guided core biopsy clinical reports and patient records. All patients classified as negative or benign for this study underwent follow-up screen– film mammography one year after they received their dig-
881
COLE ET AL
ital mammogram. None of these patients showed evidence of malignancy by screen–film mammography or clinically for a minimum of one year after their Fischer SenoScan digital mammogram. Research assistants at each of the participating sites coded the results of the follow-up screen–film mammogram. For the 111 malignant cases, the distribution of lesion sizes and tumor stage using TNM criteria were collected for 72 lesions that underwent core biopsy. The remaining 39 lesions were found cancerous based on fine needle aspiration (FNA). Of the malignancies, 42% were less than 1 cm in size (Table 1) and 75% were stage 0 or 1 (Table 2). Reader Study Eight radiologists participated in a reader study, which was conducted at the University of North Carolina. Six readers were ABR-certified, one Canadian-certified, and one Korean-certified with years of mammography experience ranging from 2-25 years (mean 8.5 yrs). Of those, six had experience in interpreting digital mammograms through direct clinical practice or through participation in other digital mammography studies at UNC. The remaining two radiologists, who had no experience reading digital mammograms in our research lab, were trained in interpreting SenoScan digital mammograms by reading ten printed digital mammograms that were not included in this reader study but that were prepared and printed by the same imaging technologist who prepared and printed the actual study cases. The two readers were provided with immediate feedback regarding the pathologically proven lesions present in these ten training mammograms. All readers were trained in the use of the forms used in the study just before interpreting the study examinations. There were 247 cases, each consisting of a screen–film mammogram and a digital mammogram. The images were divided into two groups, A and B, each consisting of 247 cases. The screen–film or digital mammogram for each case set was randomly assigned to one of the two groups using standard statistical randomization procedures. All readers read group A cases first during their individual reading sessions, followed by a minimum 4-week washout period to minimize memory effects followed by the reading of group B cases. A research assistant (EBC) loaded 50 to 100 cases at a time onto a multiviewer utilizing appropriate masking for extraneous light. Readers were required to take 5-minute breaks every 50 minutes, or as necessary.
882
Academic Radiology, Vol 11, No 8, August 2004
Table 1 Size Distribution of Core-Biopsied Cancerous Lesions Size mm
Frequency count
Percent
0 1 2 3 4 5 7 8 9 10 11 12 13 14 15 16 17 18 20 21 23 25 26 30 35 37 42 44 45 50 52 56 60 71 109
4 2 2 2 3 2 2 1 5 7 1 3 2 1 4 1 2 4 4 1 1 2 1 4 1 1 1 1 1 1 1 1 1 1 1
5.56 2.78 2.78 2.78 4.17 2.78 2.78 1.39 6.94 9.72 1.39 4.17 2.78 1.39 5.56 1.39 2.78 5.56 5.56 1.39 1.39 2.78 1.39 5.56 1.39 1.39 1.39 1.39 1.39 1.39 1.39 1.39 1.39 1.39 1.39
***The research assistant also recorded data onto paper forms as readers interpreted the examinations. The readers were required to provide the information in a standardized format following a specifically designed, structured report. If clinically relevant abnormalities were found, the research assistant recorded this information on lesion-specific forms. The lesion location and probability for malignancy (based on a 5-point scale, where the ratings of 1— definitely not malignant, 2—probably not malignant, 3—possibly malignant, 4 —probably malignant, and 5— definitely malignant) were used to classify the likelihood of cancer in each case. The BI-RADS standard scale for likelihood of cancer classification was not used
DIAGNOSTIC ACCURACY OF FISCHER SENOSCAN
Academic Radiology, Vol 11, No 8, August 2004
Table 2 T-stage of core biopsied cancerous lesions Size
Frequency count
Percent
Tis T1 T1a T1b T1c T2 T3 T4 TX
15 11 4 11 13 11 4 2 1
20.83 15.28 5.56 15.28 18.06 15.28 5.56 2.78 1.39
because it does not readily lend itself to ROC analysis since it is not a continuous scale. While all of this information was collected during the course of the study, only the lesion location and probability for malignancy ratings were used for the statistical analysis. As the study progressed, a subset of screen–film mammograms was removed either for patient care purposes or because the originating site requested their return, so some readers did not read all screen–film mammograms that were initially selected for inclusion in the study. No digital cases were removed from the study once it began. Table 3 shows the case distributions of paired digital and screen–film cases interpreted by each reader. Standard statistical methods were used to address missing data. The readers’ interpretations were compared with the lesion locations identified in the clinical reports at each of the enrolling sites. Diagnosis was obtained from the pathology reports if the lesions were biopsied, or from one year of follow-up for those lesions that did not go to biopsy. Both lesion location and diagnosis were the only relevant pieces of information collected during the course of the reader study that were required for conducting the AUC analysis planned. First the lesion location provided by the readers was checked with the known true location for each case. Second the probability of malignancy, where a rating of 1 or 2 was called benign and a rating of 3, 4, or 5 was called malignant, was checked against the true diagnosis for each case. Statistical Analysis The 95% confidence interval for the difference of the mean AUCs (digital–film) was determined by applying the approach described by Obuchowski (8). Only the methods for determining the confidence interval were used from the Obuchowski paper even though there are
additional tests for equivalence that are mentioned. The confidence interval for the difference in mean AUC was the only information required by the FDA for proving equivalence. The hypothesis that the difference in AUC between SenoScan digital and screen–film mammography is no larger than 0.05 was selected in accordance with FDA guidelines for pre-market approval (7), where the low limit of ⫺0.05 (digital AUC⫺screen–film AUC) (representative of screen–film providing better performance than digital) would still be considered acceptable for FDA approval of a digital mammography system. Both the screen–film interpretation forms as well as the digital interpretation forms were required for each case. If the reader did not read a case in the screen–film modality due to it being removed from the study, it was excluded from the analysis for that particular reader’s case set. The determination of sensitivity and specificity was based on the probability of malignancy ratings described above. The cut point used for calculating sensitivity and specificity was 1–2 for benign and 3–5 for malignant. ROCKit statistical software (C. Metz, University of Chicago, Chicago IL) was used for the primary analysis of estimating the area under the ROC curves. SAS (SAS Institute, Cary, NC) was used for generating the summary statistics.
RESULTS The average AUC for SenoScan digital mammography was 0.715. The average AUC for screen–film mammography was 0.765. The 95% confidence interval for the average difference of the mean AUCs [(SenoScan digital)⫺(screen film)] was (⫺0.101 to 0.002). Table 4 shows the individual AUCs and standard error for Sen-
Table 3 Total number of cases read per reader Reader ID
Negative cases
Positive cases
Total cases
2 3 4 5 6 7 8 9
112 124 110 112 112 111 112 111
93 106 90 90 89 91 91 90
205 230 200 200 201 202 203 201
883
COLE ET AL
Academic Radiology, Vol 11, No 8, August 2004
Table 4 Areas Under the ROC Curve for Each Reader for SenoScan Digital and Screen-Film
Reader ID
Senoscan digital AUC(SE)
Screen film AUC(SE)
95% CI for difference in mean AUCs Digital–Screen Film
2 3 4 5 6 7 8 9
.678 (.038) .758 (.033) .694 (.036) .777 (.031) .730 (.038) .696 (.042) .653 (.051) .736 (.039)
.737 (.037) .807 (.029) .748 (.036) .769 (.034) .810 (.031) .800 (.033) .681 (.045) .766 (.035)
(⫺.1411, .0244) (⫺.1134, .0161) (⫺.1387, .0304) (⫺.0634, .0780) (⫺.1492, ⫺.0096)* (⫺.1836, ⫺.0246)* (⫺.1280, .0725) (⫺.1081, .0480)
SE ⫽ Standard Error *Readers for which the AUC for digital is significantly less than the AUC for screen film
oScan digital and screen–film along with individual 95% confidence intervals for difference between SenoScan digital and screen–film for each of the 8 readers. The average sensitivity for SenoScan digital mammography was 0.66. The average sensitivity for screen–film mammography was 0.74. A comparison of each of the eight readers’ sensitivities for digital and screen–film mammography is shown in Figure 1. The average specificity for SenoScan digital mammography was 0.67. The average specificity for screen–film mammography was 0.60. A comparison of each of the eight readers’ specificities for digital and screen–film mammography is shown in Figure 2.
DISCUSSION All of the cases included in this study had some type of abnormality that led to their inclusion in this study, making the population under study a diagnostic one with some cases entering the study without symptoms at the time of an abnormal screening mammogram. The results described here are consistent with similar studies comparing digital to screen–film mammography. Another study conducted on Fischer SenoScan digital mammography in a population of patients with dense breasts resulted in AUC, sensitivity, and specificity scores of 0.663, 54.1%, and 64.6%, respectively, for masses; and AUC, sensitivity, and specificity results of 0.633, 59.9%, and 76.9%, respectively, for calcifications (9). Preliminary results
884
from a digital mammography versus screen–film mammography study conducted by Lewin et al (10) with the General Electric digital mammography system in a screening population showed sensitivity results of 60% for digital mammography versus 63% for screen–film mammography. One of the limitations of this study is that enrollment was based on findings that were seen on the eligibility screen–film mammograms. This leads to a sensitivity bias toward screen–film mammography and a specificity bias toward digital. This sensitivity bias toward screen–film mammography and specificity bias toward digital was also noted by Lewin et al (10) in their work. There is a large digital versus screen–film study currently underway that has corrected for this bias by enrolling women who are not known to have abnormal screen–film mammograms and random assignment of the initial mammogram to digital or screen–film at enrollment. This study is also limited by the presentation display of the images in printed format instead of softcopy using high-resolution monitors. The softcopy display benefits of image contrast adjustment, panning and zooming to full resolution, and image processing were not available for our readers. This, of course, may have affected reader interpretation performance. Another study comparing softcopy display to hardcopy display of Fischer SenoScan digital mammograms showed no significant difference in performance between the two modalities (11). Larger case sets with more readers would be ideal for a follow-up study for the diagnostic population, and one is underway for various digital mammography systems including the SenoScan using softcopy display. Another limitation of this study is in the possible bias introduced as a result of different numbers of cases used to determine the individual reader AUCs due to the loss of screen–film mammograms intermittently throughout the course of the study. As original screen–film mammograms were used for this study, they were always available if needed for clinical purposes, or in the case of one site, for an overlapping reader study. This does point out one of the benefits of digital mammography over screen– film mammography: there is no loss in image quality with copy films. The statistical analysis presented here includes the entire analysis presented to the FDA for an equivalence study to screen–film mammography of the Fischer SenoScan digital mammography system in hardcopy presentation. As such, this presents a limitation of the study in that more rigorous analyses, such as lesion-specific per-
Academic Radiology, Vol 11, No 8, August 2004
DIAGNOSTIC ACCURACY OF FISCHER SENOSCAN
Figure 1. The individual reader mean sensitivity with SenoScan Digital in a diagnostic population trend toward lower sensitivity for SenoScan Digital.
formance, breast density–specific performance, and the reporting of P values were not conducted. Again it must be pointed out that the analysis presented here represents the data analysis requirements for FDA approval; only the
statistical analysis required was reported here. As our results are specific to the Fischer SenoScan digital mammography system, it would be inappropriate to extrapolate these results to the performance of other digital mammog-
Figure 2. The individual reader mean specificity with SenoScan Digital in a diagnostic population trend toward higher specificity than screen–film.
885
COLE ET AL
Academic Radiology, Vol 11, No 8, August 2004
raphy systems. Since the time of this trial, there have been hardware and software modifications that could possibly lead to different outcomes then those obtained here. Based on the AUC of this study, given the size of the 95% confidence interval and its inclusion of zero, the Fischer SenoScan Digital Mammography System is minimally equivalent to screen–film mammography for a diagnostic or problem-solving mammography population. As most of the confidence interval is negative, it is possible that a study with more power would show the Fischer SenoScan as inferior to screen–film mammography. REFERENCES 1. Sickles EA. Breast imaging: from 1965 to the present. Radiology 2000; 215:1–16. 2. Hendee WR. History and status of x-ray mammography. Health Phys 1995; 69:636 – 648. 3. Chan HP, Vyborny CJ, MacMahon H, et al. digital mammography. ROC studies of the effects of pixel size and unsharp-mask filtering on the detection of subtle microcalcifications. Invest Radiol 1987; 22:581– 589. 4. Nawano S, Murakami K, Moriyama N, et al. Computer-aided diagnosis in full digital mammography. Invest Radiol 1999; 34:310 –316.
5. Betal D, Roberts N, Whitehouse GH. Segmentation and numerical analysis of microcalcifications on mammograms using mathematical morphology. Br J Radiol 1997; 70:903–917. 6. Tesic MM, Piccaro MF, Munier B. Full field digital mammography scanner. Eur J Radiol 1999; 31:2–17. 7. Premarket Applications for Digital Mammography Systems: Final Guidance for Industry and FDA. Rockville, MD: Center for Devices and Radiological Health, U.S. Dept of Health and Human Services Food and Drug Administration; 2001. CDRH publication 983. 8. Obuchowski NA. Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol1995;2 Suppl 1:S22-9. 9. Cole EB, Pisano ED, Kistner EO, et al. Diagnostic accuracy of digital mammography in patients with dense breasts presenting for problemsolving mammography-image processing and lesion type effects. Radiology 2003; 226:153–160. 10. Lewin JM, Hendrick RE, D’Orsi CJ, et al. Comparison of full-field digital mammography with screen–film mammography for cancer detection: results of 4,945 paired examinations. Radiology 2001; 218: 873– 880. 11. Pisano ED, Cole EB, Kistner EO, et al. Digital mammography interpretation: Comparison of the speed and accuracy of softcopy versus printed film display. Radiology 2002; 223:483– 488.
The Association of University Radiologists wishes to thank the following companies for their generous support of the AUR Junior Membership Program:
886
4th Year Radiology Residents
3rd Year Radiology Residents
Siemens Medical Systems, Inc.
Berlex Laboratories