Artificial intelligence in medical imaging

Journal Pre-proof Artificial intelligence in medical imaging

John C. Gore PII:

S0730-725X(19)30755-6

DOI:

https://doi.org/10.1016/j.mri.2019.12.006

Reference:

MRI 9356

To appear in:

Magnetic Resonance Imaging

Received date:

9 December 2019

Accepted date:

9 December 2019

Please cite this article as: J.C. Gore, Artificial intelligence in medical imaging, Magnetic Resonance Imaging(2018), https://doi.org/10.1016/j.mri.2019.12.006

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

© 2018 Published by Elsevier.

Journal Pre-proof

Artificial Intelligence in Medical Imaging John C. Gore

-p

ro

of

Vanderbilt University Institute of Imaging Science

lP

re

Corresponding address:

Vanderbilt University Institute of Imaging Science

na

1161 21st Ave S

Jo

TN, 37232

ur

Nashville,

USA

Tel: 615 322 8359 [email protected]

Journal Pre-proof Abstract The medical specialty radiology has experienced a number of extremely important and influential technical developments in the past that have affected how medical imaging is deployed. Artificial Intelligence (AI) is potentially another such development that will

of

introduce fundamental changes into the practice of radiology. In this commentary the

ro

historical evolution of some major changes in radiology are traced as background to how AI

-p

may also be embraced into practice. Potential new capabilities provided by AI offer exciting

Jo

ur

na

lP

re

prospects for more efficient and effective use of medical images.

Journal Pre-proof

Our December 2019 issue, Volume 64 edited by Dr. Bennett Landman of Vanderbilt

of

University, was dedicated to recent developments in the use of artificial intelligence (AI) in

ro

MRI. This timely collection of papers reflects the current state of research and applications

-p

of AI in our field, and highlights the rapid advances being made to improve radiological

lP

re

diagnoses and analyses of MR images used advanced computer algorithms. To highlight the relevance of these exemplars, Figure 1 shows the growth of publications indexed in PubMed

na

when searching on the terms “radiology” with “artificial intelligence”, “machine learning”,

ur

“deep learning” or “convolutional neural networks”. The curves for the past 5 years illustrate

Jo

how these concepts are strongly coupled and how rapid recent growth has been. This commentary in intended to place the advent of AI within radiology in a historical context, and to emphasize to users of MRI the likely future impact of AI beyond specific technical advances being currently implemented. Recall that radiology originated with the remarkable discovery of X-rays. Within a few weeks of their detection by Roentgen, X-rays were being applied for clinical purposes, and

Journal Pre-proof X-ray technology rapidly disseminated throughout medicine. Thus, radiology as a profession began as an early adopter of a breakthrough technology. Since then there has been a stream of technological innovations that have been embraced by radiologists in the form of new or more advanced imaging modalities - nuclear PET and SPECT, MRI, digital

of

radiography, CT, ultrasound and optical imaging - that have evolved largely from new

ro

discoveries in physics and engineering or other technical developments. In parallel there has

-p

been a massive growth in the use and diversity of imaging in clinical practice. The global

Given the widespread use and significant costs of radiological

lP

about 5% annually [1].

re

medical equipment market today is roughly US$35 billion a year and growing at a rate of

na

imaging, there are important questions being raised regarding how Radiology will evolve in

ur

the future, especially in the era of AI.

Jo

The term Artificial Intelligence (AI) was coined by John McCarthy of Dartmouth College in 1956 [2], and today serves as a general description of the capabilities and operations of machines (computers) that mimic or emulate human intelligence. Without entering into a semantic discourse on the precise meanings of machine intelligence and machine learning (topics that are regularly debated by the cognoscenti of AI) it is intuitive to interpret AI as embracing all those developments in computing (hardware and software) that harness

Journal Pre-proof information to address problems and provide solutions or new knowledge based on algorithms or designs that themselves are derived, often only loosely, from the manner in which humans think and behave. AI has made dramatic advances in recent years in many spheres, and clearly medicine is one domain in which AI is currently making major inroads.

of

Thomas Kuhn, the Harvard-based philosopher of science, in his classic book “The

ro

Structure of Scientific Revolutions” [3] coined the phrase “paradigm shift” to describe how

-p

major changes (revolutions) arise in science. He conceived that “normal” science progresses

re

steadily, within an accepted set of rules and assumptions (the paradigm), but may

lP

fundamentally change direction when some radical new idea or technology arises that

na

disrupts the prevailing status. The term “disruptive technologies” was coined by Joseph

ur

Bower and Clayton Christensen in an 1995 article published in the Harvard Business Review

Jo

[4]. Examples of disruptive technologies include smart phones, digital cameras and (many years ago) the motor car. AI has the potential to cause a paradigm shift in medical imaging. Radiology itself has been subject to a small number of extremely disruptive paradigm shifts since its inception. The original technologies of X ray tubes, film-screen cassettes, film developers and viewing boxes served as the basis for most radiology up until the 1970s. It was the invention of CT scanning by Hounsfield and colleagues at EMI Ltd. in the UK that

Journal Pre-proof forever changed the face and practice of radiology [5]. CT was inherently digital - it had to wait for the development of adequate computers and digital technology to be feasible and it produced images in cross-sections, a view that ultrasound also provided but (then) with much less impact. CT changed Radiology in the most fundamental manner, it ushered

of

in the current environment in which all modalities are inherently digital, usually providing

ro

cross-sectional and three dimensional data, and it enabled other fundamental changes in

-p

the way radiologists performed their work. It was no longer necessary for physicians to be at

re

or even close to the site at which images were acquired. Viewing boxes were replaced by

lP

work stations, film developers by storage discs, and for the first time numbers (Hounsfield

na

Units, originally EMI Units) had specific meaning within clinical imaging. With improvements

ur

in hardware the concept of an imaging network evolved, allowing central storage and

Jo

distribution of images via PACS systems (Picture Archive and Communication Systems), remote reading and reporting, eventually teleradiology. This was truly a paradigm shift induced by a disruptive technology. A second such shift occurred in the 1980s with the advent of MRI. MRI introduced the concept of an imaging protocol in which a set of specified exams would be programmed and performed to fill the time available. Imaging was no longer limited by radiation

Journal Pre-proof exposures or safety considerations so cost and time were the only major constraints on image acquisitions. Moreover, MRI produced a rich and still-increasing array of types of information - images in which the contrast could be modified by relaxation times, water content, or diffusion rates, or used to measure flow, perfusion, neural activity, iron

of

deposition, fat content and so on - so specific new skills and experience were needed to

ro

design imaging protocols for maximal efficacy. A dizzying array of methodological choices -

-p

pulse sequences - replaced the simpler choices of kV and mAs and accelerated the

re

introduction and applications of imaging biomarkers that pushed radiology beyond

lP

diagnosis into a greater engagement in patient management.

na

Since the advent of MRI, there has been steady progress in all imaging modalities.

ur

Ultrasound imaging has made remarkable strides in resolution and artifact reduction,

Jo

exemplified by the high resolution 4D images of fetuses in vivo that are treasured by parents and obstetricians alike. PET imaging has moved into the mainstream of cancer care and made molecular imaging a reality. The speed and resolution of CT scanning have improved by orders of magnitude with a concomitant reduction of radiation dose. MRI technological innovations have included higher field magnets, better gradients and parallel RF coil arrays that have increased the SNR of all MR methods. However, while there are

Journal Pre-proof continuing technological improvements in all types of imaging, it is also clear that we are reaching the limits of what can be done within the constraints of safety, cost, the laws of physics and known technology. For example, there are fundamental limits on how low radiation doses for CT can go once all photons are counted and their energies and

of

directions analyzed simply because information theory dictates a minimum number of

ro

detection events must occur to be able to reconstruct a useful image. Similarly, the ultimate

-p

sensitivity of ultrasound is limited by the receiver electronic noise and the shapes and

re

fidelity of the sound beams used, which again are limited ultimately by laws of physics and

lP

constrained practically by safety considerations that limit sound intensities used in imaging.

na

MRI continues to improve but there is a fundamental limit on the strength of the signals

ur

available and practical limits on magnet strengths and RF technology, so while clever

Jo

innovations in image acquisition, correction and reconstruction will continue to have impact, there are no obvious ways in which the quality of data routinely available to radiologists will increase hugely. Indeed, even if major gains in signal to noise are realized, there are also practical questions as to how much we should push the spatial resolution of clinical images because the time required to read very high resolution images (presumably using digital magnification in order to make use of the information) will increase, and the interpretation

Journal Pre-proof of structures at very fine scale may not increase diagnostic reliability or even be relevant to most pathologies. It seems highly plausible that, unlike the recent past in which technical advances have generally been used to improve image quality (contrast, signal to noise and resolution), in the future such advances will mainly be used to increase speed, reduce cost

of

or improve safety.

ro

If the question is asked “what is the next or coming paradigm shift?” then clearly the

-p

most obvious response is Artificial Intelligence. AI has multiple potential applications in

re

medicine and medical uses will attract attention and resources, but there are strong

lP

arguments to support the contention that radiology should be in the vanguard of clinical

na

applications of AI. Radiology generates billions of images every year that are acquired,

ur

evaluated, stored and transmitted from location to location by computer networks. All

Jo

radiology today is inherently digital. Computers permeate every aspect of imaging, and as discussed earlier, radiology has a tradition of being early adopters of technologies. These factors alone may be enough to catalyze the development of AI within imaging but AI also offers clear enhancements to current capabilities and provides solutions to important current problems such that the adoption of AI based techniques becomes compelling. For example, there is a constant push-pull interaction between two attributes of radiological practice,

Journal Pre-proof namely, efficiency and efficacy. Efficiency connotes the proper management of resources and the manner in which workflows are implemented to ensure maximal utilization and minimal waste, and ultimately affects the financial health of an enterprise. In an era of increasing numbers of images per patient and decreasing reimbursements, the workload of

of

each radiologist inevitably increases. For example, recall that in the 1960s a chest exam

ro

might involve viewing only two radiographs. With modern CT lung screening, that may

-p

easily become 50 - 100 images, and even more with MRI exams with multiple pulse

re

sequences. This increase in workload is especially visible outside the US and Europe - for

lP

example, in China there is only one qualified radiologist for every 40,000 people [6], each

na

MR scanner may examine up to 8 patients per hour, and so the workload easily becomes

ur

unmanageable. This in turn affects the other factor, the efficacy, which connotes the

Jo

accuracy and reliability of providing correct interpretations and clinical decisions, which in current models of practice are negatively impacted by increased efficiencies. As workloads increase, medical errors also increase. If AI is able to alleviate this classic conflict then it will be welcomed by physicians, hospital administrators, insurers and patients alike. The potential of AI increases if we consider the current limitations on imaging in practice. Complex systems such as MRI scanners are operated by human technologists who

Journal Pre-proof may be stressed, busy, inadequately trained or simply have limited expertise to always perform an optimal procedure. Images are interpreted by radiologists and even the most expert and diligent make errors or may be biased by experience or fatigue. Radiologists on average have only a few seconds to examine each image in a series before focusing on

of

those that really matter for making decisions or diagnoses. There is very little practical use

ro

of quantitative information - deriving and using numerical data is often time-consuming and

-p

physicians instead rely on subjective impressions that are prone to error. There are also

re

several technical factors that affect radiological decisions in practice such as the lack of

lP

standardization of equipment or protocols, limitations on the accuracy of co-registering

na

studies performed at different times, image artifacts or the influence of major variations in

ur

normal anatomy that may appear to be pathology. These limit the reliability of current

Jo

imaging but in addition there are missed opportunities for doing better. For example, while experienced practitioners may accrue a wealth of expertise from a lifetime of interpreting images, the actual number of cases on which they may base a decision is, in terms of big data, trivial - perhaps hundreds of cases, whereas a computer may be trained on thousands or even millions of datasets and have perfect recall. Similarly, even expert human observers are relatively poor at integrating multi-modal, complex data from different domains, such as

Journal Pre-proof visual images, genotypical data, numerical information and so forth. Computers, on the other hand, treat all such data, once appropriately coded, as essentially equivalent, and are agnostic to the format or nature of the underlying content. Unlike for humans, extracting quantitative metrics or assigning numerical values such as probabilities are not time-

of

intensive for machines but are inherently embedded in computer handling of information.

ro

From this it can be predicted that algorithms and digital hardware should be able to

-p

overcome some current limitations and create new capabilities.

re

The precise roles of AI are still evolving but it is likely that they will not be limited to

lP

single aspects of radiological practice. The traditional imaging chain of events starts with the

na

selection of one or more modality. Patients are imaged using equipment that require some

ur

initial calibration and positioning, and in the case of complex procedures such as MRI, there

Jo

needs to be an informed decision made about the precise scanning protocol. The data that are acquired must be reconstructed or reformatted into images that in turn may be processed, corrected and then presented for interpretation. Finally images are read and decisions are made regarding patient management. Practically every aspect of this chain lends itself to the engagement of intelligent algorithms to assist or replace human interventions. For example, we already are on the verge of accepting self-driving cars

Journal Pre-proof equipped with dozens of sensors that can replace human drivers. Intelligent “self-driving” MRI scanners are similarly possible, with machine controlled intelligent choices of protocols and calibrations, the selection of optimal imaging techniques for targeted imaging, with intelligent adaptation to individual variations in anatomy and patient, and avoiding

of

technologist errors. Image acquisitions themselves can be made faster, with more reliable

ro

reconstructions, fewer artifacts and reduced noise because machine learning obviates the

-p

need of some data acquisitions and machines can be taught to recognize essential

re

differences between noise, artifact and anatomy. The result will be higher throughputs and

lP

improved image qualities.

na

AI will also contribute after images have been acquired. AI will permit the automated

ur

integration of images with electronic medical records and other data in such as way that

Jo

decision support will no longer be limited to referencing similar images but rather identical phenotypes selected from a large array of options. AI will be useful if it reduces the amount of time spent looking at innocuous details and highlights suspicious abnormalities of greater interest. Machines can be trained to be much less distracted by normal variants and image artifacts, and will be able to access vast amounts of data from other exams quickly and

Journal Pre-proof effortlessly. These abilities cannot be replicated by armies of human physicians and thus represent a disruptive change in practice. Moore’s law, the result of analysis by Gordon Moore in the 1960s [7], predicted the number of transistors in a dense integrated circuit would double about every two years, and

of

this has proven the case for the past few decades, so that the speed and power of

ro

computers have increased over a million-fold since the advent of CT, MRI and digital

-p

radiology. However, although we have emphasized how radiology has become inherently

re

computer-based, we actually have tapped little of the extraordinary increase in computing

lP

power that has occurred since those early developments. It is true that processes such as

na

reconstruction and visualization have increased in complexity and speed but otherwise little

ur

of the potential power of modern computing has had any impact. However, it is clear how

Jo

this is changing with the advent of AI and powerful new concepts such as deep learning and convolutional neural networks. Current uses of computing are complementary to human activities and include (i) image acquisition and reconstructions; (2) image processing and enhancement; (3) image analysis with some quantification (often under human supervision); (4) image display and visualization; (5) image storage and transmission; and (6) some integration of imaging with medical records. Contrast these uses with the disruptive

Journal Pre-proof enhancements, some of which were illustrated in our recent Special Issue, that may include (1) the intelligent design and management of imaging protocols by scanners able to identify and calibrate for individual patients; (2) reduced image acquisition and improved reconstructions based on machine learning, including synthesizing images that were not

of

specifically acquired [8]; (3) automated corrections in imperfect images along with image

ro

enhancements and noise reduction [9] [10], [11]; (4) automated identification and detection

-p

of features within images, quantified objectively and without specific hypotheses, that can

re

feed into decision support algorithms [12]; (5) data-driven interpretation and automated

lP

diagnoses [13]; and (6) pattern recognition across different multi-parametric data sets that

na

bring together imaging with other types of information. These capabilities are increasingly

ur

made possible by the continuing evolution of digital hardware and software. They address

Jo

the conventional push-pull conflict because they promise to increase both efficiency and efficacy, making better use of resources and decreasing medical errors. To reach the full potential of AI there are still several substantial challenges to overcome. Integration of imaging data with other records and for automated analyses will require improvements in natural language processing and structured reporting. As emphasized by Schilling and Landman [14] there is an urgent need for greater transparency as to how some

Journal Pre-proof algorithms used in the diagnostic decision chain make inferences, an essential requirement for broad acceptance and user confidence. Algorithms must be evaluated on realistic data, imperfect datasets that are not carefully selected but which replicate practical settings. Their performance must be robust for different sites, vendors, and populations. In order to

of

achieve these goals, AI may require a much enhanced digital infrastructure within radiology

ro

departments, and new image reconstruction methods may challenge the use of traditional

-p

metrics of image quality and require new theoretical approaches to evaluate information

re

content. These challenges are being met as the appropriate roles of AI are being recognized

lP

and accepted, but whether AI remains a complementary addition to human abilities or a

na

disruptive influence on how images are made and interpreted is a debate that has not yet

ur

been resolved. However, the forces of innovation, economics and technology seem sure to

Jo

produce a fundamental shift in the relative roles of radiologist and machine. It is plausible that every image will one day be pre-screened and analyzed by a computer prior to human interaction. The extent to which this obviates the need for specific professional skills remains to be established.

Journal Pre-proof Figure Caption Figure 1. The growth of AI, deep learning and convolutional neural networks as reflected in

Jo

ur

na

lP

re

-p

ro

of

the number of papers published each year in radiology in PubMed.

Journal Pre-proof References [1]

https://www.fortunebusinessinsights.com/industry-reports/medical-imaging-equipment-

market-100382. Medical Imaging Equipment Market: Global Market Analysis, Insights and Forecasts, 2018-2025, Fortune Business Insights, 2019

of

[2] McCorduck, P. Machines Who Think, 2nd ed., Natick, MA: A. K. Peters; 2004

ro

[3] Kuhn, TS. The Structure of Scientific Revolutions, University of Chicago Press; 1962

-p

[4] Bower, JL and Christensen, CM. Disruptive technologies: catching the wave, Harvard

lP

re

Business Review 1995; 73:43-53.

[5] Hounsfield, GN. Computerised transverse axial scanning (tomography) Part 1: Description

education

ur

https://www.diagnosticimaging.com/articles/radiologys-future-rests-training-and-

Jo

[6]

na

of system. British Journal of Radiology 1973; 46:1016-1022

[7] Moore, GE. Cramming more components onto integrated circuits. Electronics 1965; 38(8):114-117 [8] Ryu K, Shin NY, Kim DH and Nam Y. Synthesizing T1 weighted MPRAGE image from multi echo GRE images via deep neural network, Magn Reson Imaging 2019; 64:13-20

Journal Pre-proof [9] Schilling KG, Blaber J, Huo Y, Newton AT, Hansen C, Nath V, Shafer AT, williams O, Resnick SM, Rogers BP, Anderson AW and Landman BA. Synthesized b0 diffusion distortion correction (Synb0-DisCo), Magn Reson Imaging 2019; 64:62-70 [10] Zhao C, Shaoa M, Carassa A, Li H, Dewey BE, Ellingsen LM, Woo J, Guttman MA, Blitz

of

AM, Stone M, Calabresi PA, Halperin H and Prince JL. Applications of a deep learning method

ro

for anti-aliasing and super-resolution in MRI, Magn Reson Imaging 2019; 64:132-141

-p

[11] You X, Cao N, Lu H, Mao M, and Wang W. Denoising of MR images with Rician noise

lP

re

using a wider neural network and noise range division, Magn Reson Imaging 2019; 64:154159

na

[12] Liu CF, Paddy S, Ramachandran S, Wang VX, Efimov A, Bernal A, Shi L, Vaillant M,

ur

Ratnanather T, Faria AV, Caffo B, Albert M and Miller MI. Using deep Siamese neural

Jo

networks for detection of brain asymmetries associated with Alzheimer's Disease and Mild Cognitive Impairment, Magn Reson Imaging 2019; 64:190-199 [13] Davatzikos C, Sotiras A, Fan Y, Habes M, Erus G, Rathore S, Bakas S, Chitalia R, Gastounioti A and Kontos D. Precision diagnostics based on machine learning-derived imaging signatures, Magn Reson Imaging 2019; 64:49-61

Journal Pre-proof [14] Schilling KG and Landman BA. AI in MRI: A case for grassroots deep learning, Magn

Jo

ur

na

lP

re

-p

ro

of

Reson Imaging 2019; 64:1-3

Figure 1