- Email: [email protected]
Artificial intelligence and deep learning – Radiology's next frontier?
Accepted Manuscript Artificial intelligence and deep learning – Radiology's next frontier?
R.C. Mayo, J.W.T. Leung PII: DOI: Reference:
S0899-7071(17)30227-9 doi:10.1016/j.clinimag.2017.11.007 JCT 8346
To appear in: Received date: Revised date: Accepted date:
28 July 2017 2 November 2017 9 November 2017
Please cite this article as: R.C. Mayo, J.W.T. Leung , Artificial intelligence and deep learning – Radiology's next frontier?. The address for the corresponding author was captured as affiliation for all authors. Please check if appropriate. Jct(2017), doi:10.1016/ j.clinimag.2017.11.007
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT Title Page
Artificial Intelligence and Deep Learning – Radiology’s Next Frontier? RC Mayo, MD1, JWT Leung, MD1 1
AC
CE
PT
ED
M
AN
US
Please address correspondence to: Ray Cody Mayo III, MD Assistant Professor, Department of Diagnostic Radiology TELEPHONE: 713-745-4555 FAX: 713-563-9779 EMAIL: [email protected]
CR
IP
T
The University of Texas MD Anderson Cancer Center 1515 Holcombe Blvd., Unit 1350 Houston, TX 77030
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN
US
CR
IP
T
Artificial Intelligence and Deep Learning – Radiology’s Next Frontier? Introduction Computers have revolutionized the field of diagnostic imaging and are indispensable in our current working environment [1]. A brief review of the history of computers in the radiology department is a useful framework to analyze their future role in imaging. First introduced into the radiology department in the 1960’s, computers’ role drastically expanded in the 1970’s with Radiology Information Systems (RIS). These systems focused primarily on administrative tasks such as registration and billing [2]. The earliest uses of computer technology related to imaging acquisition were nuclear medicine, digital subtraction angiography (DSA) within interventional radiology, computerized tomography (CT) in the 1970’s, and magnetic resonance imaging (MRI) in the 1980’s. The next phase of computer driven improvement was the application of picture archiving and communication systems (PACS). While often started as home-grown, one-of-akind systems, commercial versions were widely in use by the end of the 1990’s and have become more sophisticated as image volumes have grown [3]. Currently some functions of PACS may be performed in the cloud [4]. For example, images may be stored and transmitted by offsite computer networks rather than by onsite servers. This allows increased image access from remote locations while reducing the impact of local IT infrastructure failure, reliance on compact discs, and siloed data. By the mid 2000’s, many practices had replaced human transcriptionist with voice recognition software. Also in the 2000’s congress included reimbursement of screening mammography computer aided detection (CAD) in Medicare coverage which precipitated its widespread adoption [5-7]. An area of current intense focus is the intersection of artificial intelligence (AI) and image recognition. What does the future of computers in radiology department look like? Body The most robust outcomes data come of computers assisting with interpretation is from screening mammograms [8]. Numerous studies in the early and mid-2000’s showed variable modest ability of CAD to improve radiologist performance [5-14]. However several more recent studies have shown that CAD may not improve the diagnostic ability of mammography in any performance metric including sensitivity, specificity, positive predictive value, recall rate or benign biopsy rate [15-16]. False positives flagged by the CAD on the mammograms may even distract the interpreting radiologist with too much “noise” and could lead to unnecessary workups and biopsies [17]. Based on this mixed data, it is reasonable to summarize the current influence of CAD on mammographic interpretation performance as controversial with room for improvement. One promising new technology with the potential to propel the next era of progress regarding medical image interpretation is artificial intelligence, which is the science of engineering intelligent machines and computer programs. Under the umbrella of AI, a process called machine learning allows a program to learn and improve from experience without being explicitly programmed. This contrasts with traditional computer programming which requires written code that details every step the program will take. Deep learning refers to a powerful subset of machine learning techniques which uses algorithms inspired by the structure and function of the human brain that are called artificial neural networks (ANNs). These ANNs are composed of multiple layers with each layer receiving the output of the prior layer, performing a discrete task, then passing its output to the next layer. This is the structure of a computer
ACCEPTED MANUSCRIPT
AC
Applications
CE
PT
ED
M
AN
US
CR
IP
T
program that teaches itself how to learn from reviewing a large amount of supplied data. Computer aided detection of medical imaging in the future will not involve teams of developers creating rules to help the computer find edges or pixels as was performed in the past. Rather, a large amount of image data will be provided to a computer system which will then learn to detect patterns from that data in a training session. A final important requirement during the training phase is that the computer must be told what it is looking – meaning there must be a descriptive label for each image. Radiology exams with structured report will allow much more efficient training than non-structured reports. Although this may sound difficult to implement, the technology has been functional for some time. Key developments promoting recent advancements in AI include huge digital data sets that now exist (big data), Moore’s law (computational power doubles every two years), and increasing ranks of computer scientist. One of the first examples of this concept is entertaining and will be briefly summarized: In 2012 the Google Artificial Brain project connected a network of 16,000 computers. The Google Artificial Brain successfully trained itself to recognize cats based on entry of 10 million YouTube videos. As more and more images of cats were reviewed, the software became better and better at recognizing the imaging features of cats on its own. After three days of learning, the program could predict with 75% accuracy if a picture contained a cat [18-19]. More recently in 2016, this concept was emulated by the “The Digital Mammography DREAM Challenge.” This event was sponsored by several private technology and healthcare entities and the image data was supplied by the Breast Cancer Surveillance Consortium and the Icahn School of Medicine at Mount Sinai [20]. It was a competition to develop a useful AI detection algorithm from 640,000 digital mammograms. The winner of the challenge demonstrated a specificity of 0.81, sensitivity of 0.80. A receiver operating characteristic (ROC) curve plots the true positive against the false positive at different thresholds and gives area under the curve (AUC). The AUC is an objective measure of performance with a score of 1.0 represents a perfect test. The winning algorithm had an AUC of 0.87 [20]. For context, this performance is roughly equivalent to the bottom 10% of radiologists [21]. We can conclude that the technology may have potential, but there is still much to be desired. One way of improving performance of AI is to increase the amount of training data.
It has been proposed that CAD attempt the following four goals: improve radiologist performance, save time, be seamlessly integrated into workflow, and have negligible incremental costs [23]. Marked improvement in each of these four areas will be required for a clinically viable product. Screening mammograms represents a fertile area for further development for several reasons. First the exam is standardized, always containing CC and MLO views. Second, associated structured reports which use standardized lexicon each have a final assigned BI-RADS code. These two factors allow computers to learn quickly and efficiently. In contrast, chest X-ray images are standardized but the interpretations are not. Without a discrete label, the learning process is obscured. Learning occurs more efficiently with a binary assignment of cancer/no cancer than a subjective label in a free text report (infiltrate, opacity, or density). Hospital systems and radiology groups possess the largest repositories of normal and abnormal medical images. When coupled with structured reporting, this represents a tremendous untapped source of labeled data
ACCEPTED MANUSCRIPT
CR
IP
T
AI could also be used to “re-read” prior exams archived in PACS. Working in the background, it could retrospectively identify potential critical findings. A worklist with only the significant images that were flagged could be quickly reviewed by a radiologist to determine if the whole study needed a closer look. Even if only a small percentage of cases had critical findings identified, this could be a worthwhile practice. In addition to finding abnormal cases, a potential prospective use would be AI algorithms that identify “quick negative” exams. This would be useful in high volume screening settings. If even 10% of the normal chest x-rays, mammograms, lung CT screens, and head CTs could be immediately identified as normal, a substantial amount of radiologist time and effort could be reallocated. A standardized report would be automatically generated for review and sign off. This concept is only viable if AI’s performance could result in a sensitivity (and negative predictive value) approaching 100%. The concept of the “quick negative” would also be useful in underserved countries without easy access to local medical expertise.
ED
M
AN
US
Conclusion As this technology becomes increasingly discussed in our literature and national meetings, a familiarity with the terms and concepts of AI will facilitate improved engagement and result in novel projects. Artificial intelligence will find its way into medical imaging. Given radiology’s rich history of leveraging contemporary computing abilities, AI represents radiology’s opportunity to unlock previously unrealized value. Guided by the ACR’s Imaging 3.0 initiative, radiologists should be the ones expanding the frontier of AI in medical imaging by positioning ourselves to become image consultants with expertise in image interpretation and artificial intelligence [22].
PT
References
Mixdorf MA, Goldsworthy RE. A history of computers and computerized imaging. Radiol Technol. 1996 MarApr;67(4):291-6
2.
Arenson RL, Andriole KP, Avrin DE, Gould RG. Computers in imaging and health care: now and in the future. J Digit Imaging. 2000 Nov; 13(4):145-56
3.
Arenson RL, Avrin DE, Wong A, et al. Second Generation Folder Manager for PACS. Proceedings of SCAR 1994. Winston-Salem, NC, 1994 Jun12-15:601-05
4.
Mayo RC, Pearson KL, Avrin DE, Leung JWT. The Economic and Social Value of an Image Exchange Network: A Case for the Cloud. JACR. 2017 Jan; 14(1):130-4.
5.
Cupples TE, Cunningham JE, Reynolds JC. Impact of computer-aided detection in a regional screening mammography program. AJR Am J Roentgenol. 2005 Oct;185(4):944-50
6.
Freer TW, Ulissey MJ. Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology. 2001 Sep;220(3):781-6
7.
Morton MJ, Whaley DH, Brandt KR, Amrami KK. Screening mammograms: interpretation with computeraided detection--prospective evaluation. Radiology. 2006 May;239(2):375-83. Epub 2006 Mar 28
AC
CE
1.
ACCEPTED MANUSCRIPT 8.
Castellino RA.Computer aided detection (CAD): an overview. Cancer Imaging. 2005; 5(1): 17–19.
9.
Bandodkar P, Birdwell R, Ikeda D. Computer aided detection (CAD) with screening mammography in an academic institution: preliminary findings. Radiology. 2002;225(P):458
10. Nicholas MJ, Slanetz PJ, Mendel JB. Prospective assessment of computer-aided detection in interpretation of screening mammography: work in progress. AJR. 2004;182(P):32–3
T
11. Cupples TE. Impact of computer-aided detection (CAD) in a regional screening mammography program. Radiology. 2001;221(P):520
IP
12. Gur D, Sumkin JH, Rockette HE, et al. Changes in breast cancer detection and mammography recall rates after introduction of a computer-aided detection system. J Natl Cancer Inst. 2004;96:185–90
CR
13. Feig SA, Sickles EA, Evans WP, Linver MN. Letter to the Editor. Re: Changes in breast cancer detection and mammography recall rates after introduction of a computer-aided detection system. J Natl Cancer Inst. 2004;96:1260–1
US
14. Burhenne LJW, et al. Potential Contribution of Computer-aided Detection to the Sensitivity of Screening Mammography. Radiology. 2000 May; 215(2)
AN
15. Lehman CD, et al. Diagnostic Accuracy of Digital Screening Mammography With and Without ComputerAided Detection. JAMA Intern Med. 2015 Nov;175(11)
M
16. Fenton JJ, Taplin N. Influence of computer-aided detection on performance of screening mammography. Engl J Med 2007; 356(14):1399-409
ED
17. Alcusky M, Philpotts L, Bonafede M, et al. The Patient Burden of Screening Mammography Recall. J Wom Health. 2014;23(S1)
PT
18. Google’s Artificial Brain Learns to Find Cat Videos. Clark L. Wired UK Science {Internet}. 2012 Jun 26. Available from http://www.wired.com/2012/06/google-x-neural-network Accessed 9/13/17
CE
19. How Many Computers to Identify a Cat? 16,000. Markoff J. New York Times http://www.nytimes.com/2012/06/26/technology/in-a-big-network-of-computers-evidence-of-machinelearning.html Accessed 4/27/17
AC
20. The Digital Mammography DREAM challenge. https://www.synapse.org/#!Synapse:syn4224222/wiki/401744. Accessed 11/2/17. 21. Sprague BL, et al. National Performance Benchmarks for Modern Diagnostic Digital Mammography: Update from the Breast Cancer Surveillance Consortium. Radiology. 2017 Apr;283(1):59-69. 22. Van Ginneken, et al. Computer-aided Diagnosis: How to Move from the Laboratory to the Clinic. Radiology. 2011 Dec; 261(3) 23. Mayo RC, Parikh J. Breast Imaging: The Face of Imaging 3.0. J Am Coll Radiol. 2016 Aug;13(8):1003-7
R.C. Mayo, J.W.T. Leung PII: DOI: Reference:
S0899-7071(17)30227-9 doi:10.1016/j.clinimag.2017.11.007 JCT 8346
To appear in: Received date: Revised date: Accepted date:
28 July 2017 2 November 2017 9 November 2017
Please cite this article as: R.C. Mayo, J.W.T. Leung , Artificial intelligence and deep learning – Radiology's next frontier?. The address for the corresponding author was captured as affiliation for all authors. Please check if appropriate. Jct(2017), doi:10.1016/ j.clinimag.2017.11.007
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT Title Page
Artificial Intelligence and Deep Learning – Radiology’s Next Frontier? RC Mayo, MD1, JWT Leung, MD1 1
AC
CE
PT
ED
M
AN
US
Please address correspondence to: Ray Cody Mayo III, MD Assistant Professor, Department of Diagnostic Radiology TELEPHONE: 713-745-4555 FAX: 713-563-9779 EMAIL: [email protected]
CR
IP
T
The University of Texas MD Anderson Cancer Center 1515 Holcombe Blvd., Unit 1350 Houston, TX 77030
ACCEPTED MANUSCRIPT
AC
CE
PT
ED
M
AN
US
CR
IP
T
Artificial Intelligence and Deep Learning – Radiology’s Next Frontier? Introduction Computers have revolutionized the field of diagnostic imaging and are indispensable in our current working environment [1]. A brief review of the history of computers in the radiology department is a useful framework to analyze their future role in imaging. First introduced into the radiology department in the 1960’s, computers’ role drastically expanded in the 1970’s with Radiology Information Systems (RIS). These systems focused primarily on administrative tasks such as registration and billing [2]. The earliest uses of computer technology related to imaging acquisition were nuclear medicine, digital subtraction angiography (DSA) within interventional radiology, computerized tomography (CT) in the 1970’s, and magnetic resonance imaging (MRI) in the 1980’s. The next phase of computer driven improvement was the application of picture archiving and communication systems (PACS). While often started as home-grown, one-of-akind systems, commercial versions were widely in use by the end of the 1990’s and have become more sophisticated as image volumes have grown [3]. Currently some functions of PACS may be performed in the cloud [4]. For example, images may be stored and transmitted by offsite computer networks rather than by onsite servers. This allows increased image access from remote locations while reducing the impact of local IT infrastructure failure, reliance on compact discs, and siloed data. By the mid 2000’s, many practices had replaced human transcriptionist with voice recognition software. Also in the 2000’s congress included reimbursement of screening mammography computer aided detection (CAD) in Medicare coverage which precipitated its widespread adoption [5-7]. An area of current intense focus is the intersection of artificial intelligence (AI) and image recognition. What does the future of computers in radiology department look like? Body The most robust outcomes data come of computers assisting with interpretation is from screening mammograms [8]. Numerous studies in the early and mid-2000’s showed variable modest ability of CAD to improve radiologist performance [5-14]. However several more recent studies have shown that CAD may not improve the diagnostic ability of mammography in any performance metric including sensitivity, specificity, positive predictive value, recall rate or benign biopsy rate [15-16]. False positives flagged by the CAD on the mammograms may even distract the interpreting radiologist with too much “noise” and could lead to unnecessary workups and biopsies [17]. Based on this mixed data, it is reasonable to summarize the current influence of CAD on mammographic interpretation performance as controversial with room for improvement. One promising new technology with the potential to propel the next era of progress regarding medical image interpretation is artificial intelligence, which is the science of engineering intelligent machines and computer programs. Under the umbrella of AI, a process called machine learning allows a program to learn and improve from experience without being explicitly programmed. This contrasts with traditional computer programming which requires written code that details every step the program will take. Deep learning refers to a powerful subset of machine learning techniques which uses algorithms inspired by the structure and function of the human brain that are called artificial neural networks (ANNs). These ANNs are composed of multiple layers with each layer receiving the output of the prior layer, performing a discrete task, then passing its output to the next layer. This is the structure of a computer
ACCEPTED MANUSCRIPT
AC
Applications
CE
PT
ED
M
AN
US
CR
IP
T
program that teaches itself how to learn from reviewing a large amount of supplied data. Computer aided detection of medical imaging in the future will not involve teams of developers creating rules to help the computer find edges or pixels as was performed in the past. Rather, a large amount of image data will be provided to a computer system which will then learn to detect patterns from that data in a training session. A final important requirement during the training phase is that the computer must be told what it is looking – meaning there must be a descriptive label for each image. Radiology exams with structured report will allow much more efficient training than non-structured reports. Although this may sound difficult to implement, the technology has been functional for some time. Key developments promoting recent advancements in AI include huge digital data sets that now exist (big data), Moore’s law (computational power doubles every two years), and increasing ranks of computer scientist. One of the first examples of this concept is entertaining and will be briefly summarized: In 2012 the Google Artificial Brain project connected a network of 16,000 computers. The Google Artificial Brain successfully trained itself to recognize cats based on entry of 10 million YouTube videos. As more and more images of cats were reviewed, the software became better and better at recognizing the imaging features of cats on its own. After three days of learning, the program could predict with 75% accuracy if a picture contained a cat [18-19]. More recently in 2016, this concept was emulated by the “The Digital Mammography DREAM Challenge.” This event was sponsored by several private technology and healthcare entities and the image data was supplied by the Breast Cancer Surveillance Consortium and the Icahn School of Medicine at Mount Sinai [20]. It was a competition to develop a useful AI detection algorithm from 640,000 digital mammograms. The winner of the challenge demonstrated a specificity of 0.81, sensitivity of 0.80. A receiver operating characteristic (ROC) curve plots the true positive against the false positive at different thresholds and gives area under the curve (AUC). The AUC is an objective measure of performance with a score of 1.0 represents a perfect test. The winning algorithm had an AUC of 0.87 [20]. For context, this performance is roughly equivalent to the bottom 10% of radiologists [21]. We can conclude that the technology may have potential, but there is still much to be desired. One way of improving performance of AI is to increase the amount of training data.
It has been proposed that CAD attempt the following four goals: improve radiologist performance, save time, be seamlessly integrated into workflow, and have negligible incremental costs [23]. Marked improvement in each of these four areas will be required for a clinically viable product. Screening mammograms represents a fertile area for further development for several reasons. First the exam is standardized, always containing CC and MLO views. Second, associated structured reports which use standardized lexicon each have a final assigned BI-RADS code. These two factors allow computers to learn quickly and efficiently. In contrast, chest X-ray images are standardized but the interpretations are not. Without a discrete label, the learning process is obscured. Learning occurs more efficiently with a binary assignment of cancer/no cancer than a subjective label in a free text report (infiltrate, opacity, or density). Hospital systems and radiology groups possess the largest repositories of normal and abnormal medical images. When coupled with structured reporting, this represents a tremendous untapped source of labeled data
ACCEPTED MANUSCRIPT
CR
IP
T
AI could also be used to “re-read” prior exams archived in PACS. Working in the background, it could retrospectively identify potential critical findings. A worklist with only the significant images that were flagged could be quickly reviewed by a radiologist to determine if the whole study needed a closer look. Even if only a small percentage of cases had critical findings identified, this could be a worthwhile practice. In addition to finding abnormal cases, a potential prospective use would be AI algorithms that identify “quick negative” exams. This would be useful in high volume screening settings. If even 10% of the normal chest x-rays, mammograms, lung CT screens, and head CTs could be immediately identified as normal, a substantial amount of radiologist time and effort could be reallocated. A standardized report would be automatically generated for review and sign off. This concept is only viable if AI’s performance could result in a sensitivity (and negative predictive value) approaching 100%. The concept of the “quick negative” would also be useful in underserved countries without easy access to local medical expertise.
ED
M
AN
US
Conclusion As this technology becomes increasingly discussed in our literature and national meetings, a familiarity with the terms and concepts of AI will facilitate improved engagement and result in novel projects. Artificial intelligence will find its way into medical imaging. Given radiology’s rich history of leveraging contemporary computing abilities, AI represents radiology’s opportunity to unlock previously unrealized value. Guided by the ACR’s Imaging 3.0 initiative, radiologists should be the ones expanding the frontier of AI in medical imaging by positioning ourselves to become image consultants with expertise in image interpretation and artificial intelligence [22].
PT
References
Mixdorf MA, Goldsworthy RE. A history of computers and computerized imaging. Radiol Technol. 1996 MarApr;67(4):291-6
2.
Arenson RL, Andriole KP, Avrin DE, Gould RG. Computers in imaging and health care: now and in the future. J Digit Imaging. 2000 Nov; 13(4):145-56
3.
Arenson RL, Avrin DE, Wong A, et al. Second Generation Folder Manager for PACS. Proceedings of SCAR 1994. Winston-Salem, NC, 1994 Jun12-15:601-05
4.
Mayo RC, Pearson KL, Avrin DE, Leung JWT. The Economic and Social Value of an Image Exchange Network: A Case for the Cloud. JACR. 2017 Jan; 14(1):130-4.
5.
Cupples TE, Cunningham JE, Reynolds JC. Impact of computer-aided detection in a regional screening mammography program. AJR Am J Roentgenol. 2005 Oct;185(4):944-50
6.
Freer TW, Ulissey MJ. Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology. 2001 Sep;220(3):781-6
7.
Morton MJ, Whaley DH, Brandt KR, Amrami KK. Screening mammograms: interpretation with computeraided detection--prospective evaluation. Radiology. 2006 May;239(2):375-83. Epub 2006 Mar 28
AC
CE
1.
ACCEPTED MANUSCRIPT 8.
Castellino RA.Computer aided detection (CAD): an overview. Cancer Imaging. 2005; 5(1): 17–19.
9.
Bandodkar P, Birdwell R, Ikeda D. Computer aided detection (CAD) with screening mammography in an academic institution: preliminary findings. Radiology. 2002;225(P):458
10. Nicholas MJ, Slanetz PJ, Mendel JB. Prospective assessment of computer-aided detection in interpretation of screening mammography: work in progress. AJR. 2004;182(P):32–3
T
11. Cupples TE. Impact of computer-aided detection (CAD) in a regional screening mammography program. Radiology. 2001;221(P):520
IP
12. Gur D, Sumkin JH, Rockette HE, et al. Changes in breast cancer detection and mammography recall rates after introduction of a computer-aided detection system. J Natl Cancer Inst. 2004;96:185–90
CR
13. Feig SA, Sickles EA, Evans WP, Linver MN. Letter to the Editor. Re: Changes in breast cancer detection and mammography recall rates after introduction of a computer-aided detection system. J Natl Cancer Inst. 2004;96:1260–1
US
14. Burhenne LJW, et al. Potential Contribution of Computer-aided Detection to the Sensitivity of Screening Mammography. Radiology. 2000 May; 215(2)
AN
15. Lehman CD, et al. Diagnostic Accuracy of Digital Screening Mammography With and Without ComputerAided Detection. JAMA Intern Med. 2015 Nov;175(11)
M
16. Fenton JJ, Taplin N. Influence of computer-aided detection on performance of screening mammography. Engl J Med 2007; 356(14):1399-409
ED
17. Alcusky M, Philpotts L, Bonafede M, et al. The Patient Burden of Screening Mammography Recall. J Wom Health. 2014;23(S1)
PT
18. Google’s Artificial Brain Learns to Find Cat Videos. Clark L. Wired UK Science {Internet}. 2012 Jun 26. Available from http://www.wired.com/2012/06/google-x-neural-network Accessed 9/13/17
CE
19. How Many Computers to Identify a Cat? 16,000. Markoff J. New York Times http://www.nytimes.com/2012/06/26/technology/in-a-big-network-of-computers-evidence-of-machinelearning.html Accessed 4/27/17
AC
20. The Digital Mammography DREAM challenge. https://www.synapse.org/#!Synapse:syn4224222/wiki/401744. Accessed 11/2/17. 21. Sprague BL, et al. National Performance Benchmarks for Modern Diagnostic Digital Mammography: Update from the Breast Cancer Surveillance Consortium. Radiology. 2017 Apr;283(1):59-69. 22. Van Ginneken, et al. Computer-aided Diagnosis: How to Move from the Laboratory to the Clinic. Radiology. 2011 Dec; 261(3) 23. Mayo RC, Parikh J. Breast Imaging: The Face of Imaging 3.0. J Am Coll Radiol. 2016 Aug;13(8):1003-7