- Email: [email protected]
The Application of Machine Learning to Quality Improvement Through the Lens of the Radiology Value Network
ORIGINAL ARTICLE
The Application of Machine Learning to Quality Improvement Through the Lens of the Radiology Value Network Valeria Makeeva, MD a , Judy Gichoya, MD a, C. Matthew Hawkins, MD a, Alexander J. Towbin, MD b, Marta Heilbrun, MD a, Adam Prater, MD a Abstract Recent advances in machine learning and artificial intelligence offer promising applications to radiology quality improvement initiatives as they relate to the radiology value network. Coordination within the interlocking web of systems, events, and stakeholders in the radiology value network may be mitigated though standardization, automation, and a focus on workflow efficiency. In this article the authors present applications of these various strategies via use cases for quality improvement projects at different points in the radiology value network. In addition, the authors discuss opportunities for machine-learning applications in data aggregation as opposed to traditional applications in data extraction. Key Words: Machine learning, artificial intelligence, radiology quality improvement, radiology value network, data aggregation J Am Coll Radiol 2019;16:1254-1258. Copyright 2019 American College of Radiology
INTRODUCTION The term value chain was originally used in business and is defined as “any business operation that exists to provide one or more products (including services) that are of value to others,” including “stages” that represent the cluster of business actions that transform inputs into products and “interrelationships” that represent interdependencies between stages [1]. In radiology, the definition encompasses a
Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, Georgia. b Department of Radiology and Medical Imaging, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio. Corresponding author and reprints: Valeria Makeeva, MD, Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, GA 30322-1007; e-mail: [email protected]. Dr Hawkins is the associate editor for practice management at JACR, a member of the ACR Board of Chancellors, a member of the RADPAD Board of Directors, an alternate CPT adviser for the Society of Interventional Radiology, and sole proprietor of Hawkins Healthcare Consulting. Dr Towbin has received grants from Guerbet, Siemens, and the Cystic Fibrosis Foundation; has received personal fees from and is an advisory board member for IBM Watson Health; is an advisory board member for KLAS; has received personal fees from Applied Radiology; and has received author royalties from Elsevier. Dr Heilbrun is a member of the RSNA Radiology Informatics Committee and the Quality Improvement Committee and of the ACR Data Science Institute Panel for Non-Interpretive Skills. All other authors state that they have no conflict of interest related to the material discussed in this article.
communication related to imaging interpretation and is “value added step-by-step when information is acquired, interpreted, and communicated back to the referring clinician” for the purpose of guiding clinical care and predicting patient outcomes [2]. It is traditionally thought of as a linear process and defined by a series of sequential events progressing through positing a clinical question, order placement, patient scheduling, patient check-in, patient identification, image acquisition, image interpretation, report upload into the electronic medical record, care decision, billing, and patient experience [2,3]. Because the value chain intersects with multiple stakeholders, the process is a complex, interrelated web of many-to-many communication. Consequently, it has been proposed that the term chain be replaced by the more accurate network [2]. Stakeholders include radiologists, technologists, ordering providers, patients, schedulers, and billing support staff members [2]. This is further complicated when an array of information systems is introduced to the series of sequential events and multiple stakeholders. The radiology information system, PACS, speech recognition system, electronic health record, and health information system must all seamlessly work together for the value network to ª 2019 American College of Radiology
1254
1546-1440/19/$36.00
n
https://doi.org/10.1016/j.jacr.2019.05.039
function. Although we imagine that data flow through each system in a seamless and continuous stream, the separate operating systems introduce further complexity. Recent advances in machine learning (ML) and artificial intelligence (AI) offer promising applications to radiology quality improvement (QI) initiatives as they relate to the radiology value network. The purpose of this article is to present applications of various strategies via use cases for QI projects at different points in the radiology value network, including ordering, scheduling, and improving the patient experience.
NAVIGATING THE SYSTEM SUCCESSFULLY Because of the interlocking web of systems, events, and stakeholders in the radiology value network, coordination can be hampered by workflow inefficiencies and communication breakdowns [4-6]. The inherent complexity of multiple systems and multiple stakeholders may be mitigated though standardization, automation, and a focus on workflow efficiency [3]. Labor-intensive data organization tasks can be reduced when there is standardization for both systems and users. Automation of routine information minimizes opportunity for error from repetitive tasks and frees users to use human strengths in critical thinking. Efficiency reduces complexity and adds value to the system as a whole. Although mitigation strategies seem straightforward, their implementation can be challenging. For instance, standardization can be plagued by lack of agreement on the operational definition of a term. The lack of standard definitions occurs frequently in QI projects and departmental operations. Consider the example of patient wait time. Although the concept of a patient waiting is simple, standard measurement can be difficult. Start and stop times may be collected by different processes, some of which are electronic and some of which are manually recorded. Even if the selection of time stamps is standardized, it can still be difficult to operationalize the definition. For example, if patient wait time is defined as the time between the patient check-in and beginexamination steps, there needs to be standardization regarding the time at which each step is triggered. Without standard work, there is considerable variability in when the begin-examination step is initiated; some technologists and practices might begin an examination before going to the waiting room to collect a patient, others once the patient is in the examination room, and some once the imaging is complete and documentation steps are initiated in the electronic health record. The measurement associated with the begin-examination step Journal of the American College of Radiology Makeeva et al n Application of Machine Learning to QI
variability among different locations and different personnel may be such that meaningful comparisons cannot be made. Thus, although data can be easily collected, managers must work with their employees to ensure consistent operations. Once a term such as patient wait time has been defined, a QI team can work to improve wait times. At this point, the problem of data complexity occurs. Many different factors can affect patient wait time in radiology. Some potential factors affecting wait time include the technologists working that day, the radiologists working that day, local weather, local traffic, and the other patients who have arrived for imaging on a particular day. This type of data is stored in multiple systems, some internal to the organization and some external to the organization. Collecting these data and organizing them in a uniform way is labor and resource intensive. However, once the data are stored in a consistent manner, ML strategies can be used to predict patient wait times, allowing the QI team to model and implement improvements.
ML IN QI: “DATA EXTRACTION” VERSUS “DATA AGGREGATION” QI involves methods such as Lean, which aims to improve process performance through waste elimination, and Six Sigma, which improves process performance on the basis of customer-specific focus at different points of the value network. There has been increasing interest in using ML, an interdisciplinary method of data science in which computer algorithms can learn complex relationships from empirical data and make accurate decisions, in QI [7]. Although much of the ML literature in radiology to date focuses on imaging interpretation (tasks also known as “data extraction”), little attention has been given to “data aggregation.” Data extraction is the process of pulling preidentified fields from a database or, in diagnostic imaging, pulling structured information from an unstructured source, as in feature extraction. One commonly used example is using natural language processing to data mine nonstandardized clinical and imaging reports [8,9]. Another example is the ability to classify images. Significant attention has been devoted to developing data extraction algorithms in medical imaging interpretation with great success, focusing on narrow and specific tasks such as automated bone age assessment, knee cartilage segmentation, and tuberculosis classification [10-12]. The ImageNet Large Scale Visual Recognition Competition Challenge is a data set containing more 1255
than 15 million high-resolution images categorized into 22,000 categories. Its goal is to “evaluate algorithms for object detection and image classification at large scale” [13,14]. Artificial neural networks, processing units, and artificial neurons connected in a network and trained with a back-propagation algorithm were used to increase top-five accuracy in image classification from 75% in 2011 to 97% in 2016 [13,15,16]. QI is an area that could also benefit from ML methods, specifically by aiding in “data aggregation.” Data aggregation is the collection and organization of data sets from multiple sources. A well-recognized strength of ML is the ability to analyze more data sets and then take it a step further than pure data aggregation to extract pattern information from this morass of data [7]. ML principles have been proposed to capture the complexity of multiple-view biologic data, such as clinical and genomic data [17].
USE CASES: ML IN QI Although it is important to focus improvement efforts along all portions of the value network, deconstructing the value network into component parts to think of potential ML use cases is necessary. Throughout the remainder of this article we provide examples of potential ML use cases during the order placement, patient scheduling, and patient experience steps of the value network. Order Placement Clinical decision support (CDS) systems support health care providers in making decisions regarding both diagnosis and treatment. CDS systems are structured according to two broad principles: rule based and data driven. Rule-based systems, also known as “if-then” systems, operate by finding a relevant rule and then producing a recommendation. The weakness of such systems is that they become difficult to operate when the number of rules is large or when separate rules contradict each other. Data-driven systems operate in large data sets through data mining, and their ability to learn has been successfully applied in health care [18,19]. A proof-of-concept study in chronic disease showed that an AI-based CDS tool may attain improved patient outcomes at a reduced cost [20]. Using a Markov decision process, a method for performing probabilistic inferences over time, and dynamic decision networks to learn from clinical data and determine optimal sequential treatment decisions in treating chronic disorders, the authors 1256
showed a cost per unit outcome change of $189 for AI methods compared with $497 for treatment-as-usual methods [20]. In the future, AI algorithms may review problem lists, progress notes, and laboratory values to diagnose previously unsuspected conditions. Another application of AI-based CDS may be automated screening for renal insufficiency when placing orders for contrast-enhanced studies such that patients with poor renal function may be redirected to alternative imaging tests. A third application may be machine screening for duplicate examination orders within a predetermined time frame to reduce redundant imaging.
Patient Scheduling Patient scheduling and missed appointments are a challenge to efficiency. The “no-show” problem is complex and involves factors such as time of day, day of the week, month of the year, patient socioeconomic demographics, weather, and traffic patterns. ML has been used to predict missed outpatient clinic appointments in a tertiary care center setting in Japan using analysis from approximately 16,000 clinic appointments [21]. The authors demonstrated that among a variety of factors influencing the likelihood of missed appointments, day of the week was most strongly associated with missed appointment prediction [21]. Similar work predicting no-shows at a US tertiary care center analyzed approximately 90,000 clinic appointments using a ML algorithm and showed that race and socioeconomic status were independent predictors of missed appointments [22]. Work is currently under way to develop individualized patient-targeted solutions to reduce the chance of missing care, such as texting reminders sent to patients at high risk for missing their appointments [23]. Patient Experience Radiology reports are routinely shared with patients through online portals. Although reports written for provider-to-provider communication are not traditionally viewed as patient education material, their accessibility represents an opportunity to enhance patient education, patient satisfaction, and patient-centered care overall. Breast imaging has set a precedent because the Mammography Quality Standards Act requires mammography facilities to send written summaries of their radiology reports in lay terms [24]. However, radiology reports outside mammography are written at greater than a 12th grade reading level, with only 4% Journal of the American College of Radiology Volume 16 n Number 9PB n September 2019
of reports readable at the 8th grade level, the level of the average US adult [25,26]. This is significantly higher than the Centers for Disease Control and Prevention recommendation that patients are more likely to understand health care materials if written below the 8th grade level, which is the level at which the Centers for Disease Control and Prevention endorse patient education materials [27]. Patient satisfaction and customer retention have been linked with the ability to view health care information online [28]. From the patient satisfaction perspective, understanding the details of radiology reports may enhance patient satisfaction metrics further. Early efforts to provide this resource, such as websites like RadiologyInfo.org, are written at a 14th grade reading level [29]. Subsequent work has provided patient materials at a reading level closer to the US average. A system that provides definitions at or below the 10th grade level along with illustrations within MRI knee reports in an outpatient academic medical setting showed improved patient understanding of their diagnoses [30,31]. Within pathology, “patient-friendly” supplemental material has been distributed along with official pathology reports of breast atypia to assist with patient understanding and ease anxiety [32]. A potential future application of an ML algorithm in the patient education and satisfaction space would incorporate natural language processing tools to translate the radiology report to a format easily understood by patients.
CONCLUSIONS ML may be optimally applied to improve standardization, automation, and efficiency in the radiology imaging value network. We highlight several potential use cases to describe the feasibility and potential effectiveness of this approach. As ML becomes more prevalent, the prospect for more widespread adoption of these techniques has potential to improve the patient experience and enhance patient care. TAKE-HOME POINTS -
-
Recent advances in ML and AI offer promising applications to radiology QI initiatives as they relate to the radiology value network. The inherent complexity of multiple systems and multiple stakeholders may be mitigated though standardization, automation, and a focus on workflow efficiency.
Journal of the American College of Radiology Makeeva et al n Application of Machine Learning to QI
-
-
-
QI is an area that could benefit from ML methods, specifically by aiding in “data aggregation.” Deconstructing the value network into component parts to think of potential ML use cases is one mechanism for optimal application of ML toward improving standardization, automation, and efficiency. ML use cases as applied to the order placement, patient scheduling, and patient experience steps of the value network have potential to improve the patient experience and enhance patient care.
REFERENCES 1. Porter M. Competitive advantage: creating and sustaining superior performance. New York: Free Press; 1985. 2. Larson DB, Froehle CM, Johnson ND, Towbin AJ. Communication in diagnostic radiology: meeting the challenges of complexity. AJR Am J Roentgenol 2014;203:957-64. 3. Towbin AJ, Perry LA, Larson DB. Improving efficiency in the radiology department. Pediatr Radiol 2017;47:783-92. 4. Fitzgerald R. Error in radiology. Clin Radiol 2001;56:938-46. 5. Gandhi TK. Fumbled handoffs: one dropped ball after another. Ann Intern Med 2005;142:352-8. 6. Whang JS, Baker SR, Patel R, Luk L, Castro A III. The causes of medical malpractice suits against radiologists in the United States. Radiology 2013;266:548-54. 7. Wang S, Summers RM. Machine learning and radiology. Med Image Anal 2012;16:933-51. 8. Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform 2009;42:760-72. 9. Reiner B. Uncovering and improving upon the inherent deficiencies of radiology reporting through data mining. J Digit Imaging 2010;23: 109-18. 10. Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 2017;284:574-82. 11. Spampinato C, Palazzo S, Giordano D, Aldinucci M, Leonardi R. Deep learning for automated skeletal bone age assessment in X-ray images. Med Image Anal 2017;36:41-51. 12. Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M. Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Med Image Comput Comput Assist Interv 2013;16:246-53. 13. Krizhevsky A, SI, Hinton GE. ImageNet classification with deep convolutional neural networks. Adv Neural Inform Process Syst 2012: 1097-105. 14. Large Scale Visual Recognition Challenge (ILSVRC). Available at: http://www.image-net.org/challenges/LSVRC/. Accessed January 28, 2019. 15. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. Inception-v4, InceptionResNet and the impact of residual connections on learning. arXiv Available at: https://arxiv.org/abs/1602.07261. Accessed June 6, 2019. 16. Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G. Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology 2019;291:272. 17. Li Y, Wu FX, Ngom A. A review on machine learning principles for multi-view biological data integration. Brief Bioinform 2018;19: 325-40. 18. Bal M, Amasyali MF, Sever H, Kose G, Demirhan A. Performance evaluation of the machine learning algorithms used in inference
1257
19.
20.
21.
22.
23.
24. 25.
mechanism of a medical decision support system. Sci World J 2014;2014:137896. Yan H, JY, Zheng J, Peng C, Li Q. A multilayer perceptron-based medical decision support system for heart disease diagnosis. Expert Syst Appl 2006;30:272-81. Bennett CC, Hauser K. Artificial intelligence framework for simulating clinical decision-making: a Markov decision process approach. Artif Intel Med 2013;57:9-19. Kurasawa H, Hayashi K, Fujino A, et al. Machine-learning-based prediction of a missed scheduled clinical appointment by patients with diabetes. J Diabetes Sci Tech 2016;10:730-6. Glover MT, Daye D, Khalilzadeh O, et al. Socioeconomic and demographic predictors of missed opportunities to provide advanced imaging services. J Am Coll Radiol 2017;14:1403-11. Harvey HB, Liu C, Ai J, et al. Predicting no-shows in radiology using regression modeling of data available in the electronic medical record. J Am Coll Radiol 2017;14:1303-9. US Food and Drug Administration. Mammography Quality Standards Reauthorization Act of 1998. Yi PH, Golden SK, Harringa JB, Kliewer MA. Readability of lumbar spine MRI reports: will patients understand? AJR Am J Roentgenol 2019;212:602-6.
1258
26. Martin-Carreras T, Cook TS, Kahn CE Jr. Readability of radiology reports: implications for patient-centered care. Clin Imaging 2018;54: 116-20. 27. Centers for Disease Control and Prevention. Scientific and Technical Information Simply Put. Available at: http://www.cdc.gov/ healthliteracy/pdf/Simply_Put.pdf. Accessed January 23, 2019. 28. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015;17:e44. 29. Hansberry DR, John A, John E, Agarwal N, Gonzales SF, Baker SR. A critical review of the readability of online patient education resources from RadiologyInfo.org. AJR Am J Roentgenol 2014;202: 566-75. 30. Cook TS, Oh SC, Kahn CE Jr. Patients’ use and evaluation of an online system to annotate radiology reports with lay language definitions. Acad Radiol 2017;24:1169-74. 31. Oh SC, Cook TS, Kahn CE Jr. PORTER: a Prototype System for Patient-Oriented Radiology Reporting. J Digit Imaging 2016;29: 450-4. 32. Rooney SP, Hoffman S, Perrin JC, Milliron KJ, Nees AV, Jorns JM. Patient-friendly pathology reports for patients with breast atypias. Breast J 2018;24:855-7.
Journal of the American College of Radiology Volume 16 n Number 9PB n September 2019
The Application of Machine Learning to Quality Improvement Through the Lens of the Radiology Value Network Valeria Makeeva, MD a , Judy Gichoya, MD a, C. Matthew Hawkins, MD a, Alexander J. Towbin, MD b, Marta Heilbrun, MD a, Adam Prater, MD a Abstract Recent advances in machine learning and artificial intelligence offer promising applications to radiology quality improvement initiatives as they relate to the radiology value network. Coordination within the interlocking web of systems, events, and stakeholders in the radiology value network may be mitigated though standardization, automation, and a focus on workflow efficiency. In this article the authors present applications of these various strategies via use cases for quality improvement projects at different points in the radiology value network. In addition, the authors discuss opportunities for machine-learning applications in data aggregation as opposed to traditional applications in data extraction. Key Words: Machine learning, artificial intelligence, radiology quality improvement, radiology value network, data aggregation J Am Coll Radiol 2019;16:1254-1258. Copyright 2019 American College of Radiology
INTRODUCTION The term value chain was originally used in business and is defined as “any business operation that exists to provide one or more products (including services) that are of value to others,” including “stages” that represent the cluster of business actions that transform inputs into products and “interrelationships” that represent interdependencies between stages [1]. In radiology, the definition encompasses a
Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, Georgia. b Department of Radiology and Medical Imaging, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio. Corresponding author and reprints: Valeria Makeeva, MD, Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, GA 30322-1007; e-mail: [email protected]. Dr Hawkins is the associate editor for practice management at JACR, a member of the ACR Board of Chancellors, a member of the RADPAD Board of Directors, an alternate CPT adviser for the Society of Interventional Radiology, and sole proprietor of Hawkins Healthcare Consulting. Dr Towbin has received grants from Guerbet, Siemens, and the Cystic Fibrosis Foundation; has received personal fees from and is an advisory board member for IBM Watson Health; is an advisory board member for KLAS; has received personal fees from Applied Radiology; and has received author royalties from Elsevier. Dr Heilbrun is a member of the RSNA Radiology Informatics Committee and the Quality Improvement Committee and of the ACR Data Science Institute Panel for Non-Interpretive Skills. All other authors state that they have no conflict of interest related to the material discussed in this article.
communication related to imaging interpretation and is “value added step-by-step when information is acquired, interpreted, and communicated back to the referring clinician” for the purpose of guiding clinical care and predicting patient outcomes [2]. It is traditionally thought of as a linear process and defined by a series of sequential events progressing through positing a clinical question, order placement, patient scheduling, patient check-in, patient identification, image acquisition, image interpretation, report upload into the electronic medical record, care decision, billing, and patient experience [2,3]. Because the value chain intersects with multiple stakeholders, the process is a complex, interrelated web of many-to-many communication. Consequently, it has been proposed that the term chain be replaced by the more accurate network [2]. Stakeholders include radiologists, technologists, ordering providers, patients, schedulers, and billing support staff members [2]. This is further complicated when an array of information systems is introduced to the series of sequential events and multiple stakeholders. The radiology information system, PACS, speech recognition system, electronic health record, and health information system must all seamlessly work together for the value network to ª 2019 American College of Radiology
1254
1546-1440/19/$36.00
n
https://doi.org/10.1016/j.jacr.2019.05.039
function. Although we imagine that data flow through each system in a seamless and continuous stream, the separate operating systems introduce further complexity. Recent advances in machine learning (ML) and artificial intelligence (AI) offer promising applications to radiology quality improvement (QI) initiatives as they relate to the radiology value network. The purpose of this article is to present applications of various strategies via use cases for QI projects at different points in the radiology value network, including ordering, scheduling, and improving the patient experience.
NAVIGATING THE SYSTEM SUCCESSFULLY Because of the interlocking web of systems, events, and stakeholders in the radiology value network, coordination can be hampered by workflow inefficiencies and communication breakdowns [4-6]. The inherent complexity of multiple systems and multiple stakeholders may be mitigated though standardization, automation, and a focus on workflow efficiency [3]. Labor-intensive data organization tasks can be reduced when there is standardization for both systems and users. Automation of routine information minimizes opportunity for error from repetitive tasks and frees users to use human strengths in critical thinking. Efficiency reduces complexity and adds value to the system as a whole. Although mitigation strategies seem straightforward, their implementation can be challenging. For instance, standardization can be plagued by lack of agreement on the operational definition of a term. The lack of standard definitions occurs frequently in QI projects and departmental operations. Consider the example of patient wait time. Although the concept of a patient waiting is simple, standard measurement can be difficult. Start and stop times may be collected by different processes, some of which are electronic and some of which are manually recorded. Even if the selection of time stamps is standardized, it can still be difficult to operationalize the definition. For example, if patient wait time is defined as the time between the patient check-in and beginexamination steps, there needs to be standardization regarding the time at which each step is triggered. Without standard work, there is considerable variability in when the begin-examination step is initiated; some technologists and practices might begin an examination before going to the waiting room to collect a patient, others once the patient is in the examination room, and some once the imaging is complete and documentation steps are initiated in the electronic health record. The measurement associated with the begin-examination step Journal of the American College of Radiology Makeeva et al n Application of Machine Learning to QI
variability among different locations and different personnel may be such that meaningful comparisons cannot be made. Thus, although data can be easily collected, managers must work with their employees to ensure consistent operations. Once a term such as patient wait time has been defined, a QI team can work to improve wait times. At this point, the problem of data complexity occurs. Many different factors can affect patient wait time in radiology. Some potential factors affecting wait time include the technologists working that day, the radiologists working that day, local weather, local traffic, and the other patients who have arrived for imaging on a particular day. This type of data is stored in multiple systems, some internal to the organization and some external to the organization. Collecting these data and organizing them in a uniform way is labor and resource intensive. However, once the data are stored in a consistent manner, ML strategies can be used to predict patient wait times, allowing the QI team to model and implement improvements.
ML IN QI: “DATA EXTRACTION” VERSUS “DATA AGGREGATION” QI involves methods such as Lean, which aims to improve process performance through waste elimination, and Six Sigma, which improves process performance on the basis of customer-specific focus at different points of the value network. There has been increasing interest in using ML, an interdisciplinary method of data science in which computer algorithms can learn complex relationships from empirical data and make accurate decisions, in QI [7]. Although much of the ML literature in radiology to date focuses on imaging interpretation (tasks also known as “data extraction”), little attention has been given to “data aggregation.” Data extraction is the process of pulling preidentified fields from a database or, in diagnostic imaging, pulling structured information from an unstructured source, as in feature extraction. One commonly used example is using natural language processing to data mine nonstandardized clinical and imaging reports [8,9]. Another example is the ability to classify images. Significant attention has been devoted to developing data extraction algorithms in medical imaging interpretation with great success, focusing on narrow and specific tasks such as automated bone age assessment, knee cartilage segmentation, and tuberculosis classification [10-12]. The ImageNet Large Scale Visual Recognition Competition Challenge is a data set containing more 1255
than 15 million high-resolution images categorized into 22,000 categories. Its goal is to “evaluate algorithms for object detection and image classification at large scale” [13,14]. Artificial neural networks, processing units, and artificial neurons connected in a network and trained with a back-propagation algorithm were used to increase top-five accuracy in image classification from 75% in 2011 to 97% in 2016 [13,15,16]. QI is an area that could also benefit from ML methods, specifically by aiding in “data aggregation.” Data aggregation is the collection and organization of data sets from multiple sources. A well-recognized strength of ML is the ability to analyze more data sets and then take it a step further than pure data aggregation to extract pattern information from this morass of data [7]. ML principles have been proposed to capture the complexity of multiple-view biologic data, such as clinical and genomic data [17].
USE CASES: ML IN QI Although it is important to focus improvement efforts along all portions of the value network, deconstructing the value network into component parts to think of potential ML use cases is necessary. Throughout the remainder of this article we provide examples of potential ML use cases during the order placement, patient scheduling, and patient experience steps of the value network. Order Placement Clinical decision support (CDS) systems support health care providers in making decisions regarding both diagnosis and treatment. CDS systems are structured according to two broad principles: rule based and data driven. Rule-based systems, also known as “if-then” systems, operate by finding a relevant rule and then producing a recommendation. The weakness of such systems is that they become difficult to operate when the number of rules is large or when separate rules contradict each other. Data-driven systems operate in large data sets through data mining, and their ability to learn has been successfully applied in health care [18,19]. A proof-of-concept study in chronic disease showed that an AI-based CDS tool may attain improved patient outcomes at a reduced cost [20]. Using a Markov decision process, a method for performing probabilistic inferences over time, and dynamic decision networks to learn from clinical data and determine optimal sequential treatment decisions in treating chronic disorders, the authors 1256
showed a cost per unit outcome change of $189 for AI methods compared with $497 for treatment-as-usual methods [20]. In the future, AI algorithms may review problem lists, progress notes, and laboratory values to diagnose previously unsuspected conditions. Another application of AI-based CDS may be automated screening for renal insufficiency when placing orders for contrast-enhanced studies such that patients with poor renal function may be redirected to alternative imaging tests. A third application may be machine screening for duplicate examination orders within a predetermined time frame to reduce redundant imaging.
Patient Scheduling Patient scheduling and missed appointments are a challenge to efficiency. The “no-show” problem is complex and involves factors such as time of day, day of the week, month of the year, patient socioeconomic demographics, weather, and traffic patterns. ML has been used to predict missed outpatient clinic appointments in a tertiary care center setting in Japan using analysis from approximately 16,000 clinic appointments [21]. The authors demonstrated that among a variety of factors influencing the likelihood of missed appointments, day of the week was most strongly associated with missed appointment prediction [21]. Similar work predicting no-shows at a US tertiary care center analyzed approximately 90,000 clinic appointments using a ML algorithm and showed that race and socioeconomic status were independent predictors of missed appointments [22]. Work is currently under way to develop individualized patient-targeted solutions to reduce the chance of missing care, such as texting reminders sent to patients at high risk for missing their appointments [23]. Patient Experience Radiology reports are routinely shared with patients through online portals. Although reports written for provider-to-provider communication are not traditionally viewed as patient education material, their accessibility represents an opportunity to enhance patient education, patient satisfaction, and patient-centered care overall. Breast imaging has set a precedent because the Mammography Quality Standards Act requires mammography facilities to send written summaries of their radiology reports in lay terms [24]. However, radiology reports outside mammography are written at greater than a 12th grade reading level, with only 4% Journal of the American College of Radiology Volume 16 n Number 9PB n September 2019
of reports readable at the 8th grade level, the level of the average US adult [25,26]. This is significantly higher than the Centers for Disease Control and Prevention recommendation that patients are more likely to understand health care materials if written below the 8th grade level, which is the level at which the Centers for Disease Control and Prevention endorse patient education materials [27]. Patient satisfaction and customer retention have been linked with the ability to view health care information online [28]. From the patient satisfaction perspective, understanding the details of radiology reports may enhance patient satisfaction metrics further. Early efforts to provide this resource, such as websites like RadiologyInfo.org, are written at a 14th grade reading level [29]. Subsequent work has provided patient materials at a reading level closer to the US average. A system that provides definitions at or below the 10th grade level along with illustrations within MRI knee reports in an outpatient academic medical setting showed improved patient understanding of their diagnoses [30,31]. Within pathology, “patient-friendly” supplemental material has been distributed along with official pathology reports of breast atypia to assist with patient understanding and ease anxiety [32]. A potential future application of an ML algorithm in the patient education and satisfaction space would incorporate natural language processing tools to translate the radiology report to a format easily understood by patients.
CONCLUSIONS ML may be optimally applied to improve standardization, automation, and efficiency in the radiology imaging value network. We highlight several potential use cases to describe the feasibility and potential effectiveness of this approach. As ML becomes more prevalent, the prospect for more widespread adoption of these techniques has potential to improve the patient experience and enhance patient care. TAKE-HOME POINTS -
-
Recent advances in ML and AI offer promising applications to radiology QI initiatives as they relate to the radiology value network. The inherent complexity of multiple systems and multiple stakeholders may be mitigated though standardization, automation, and a focus on workflow efficiency.
Journal of the American College of Radiology Makeeva et al n Application of Machine Learning to QI
-
-
-
QI is an area that could benefit from ML methods, specifically by aiding in “data aggregation.” Deconstructing the value network into component parts to think of potential ML use cases is one mechanism for optimal application of ML toward improving standardization, automation, and efficiency. ML use cases as applied to the order placement, patient scheduling, and patient experience steps of the value network have potential to improve the patient experience and enhance patient care.
REFERENCES 1. Porter M. Competitive advantage: creating and sustaining superior performance. New York: Free Press; 1985. 2. Larson DB, Froehle CM, Johnson ND, Towbin AJ. Communication in diagnostic radiology: meeting the challenges of complexity. AJR Am J Roentgenol 2014;203:957-64. 3. Towbin AJ, Perry LA, Larson DB. Improving efficiency in the radiology department. Pediatr Radiol 2017;47:783-92. 4. Fitzgerald R. Error in radiology. Clin Radiol 2001;56:938-46. 5. Gandhi TK. Fumbled handoffs: one dropped ball after another. Ann Intern Med 2005;142:352-8. 6. Whang JS, Baker SR, Patel R, Luk L, Castro A III. The causes of medical malpractice suits against radiologists in the United States. Radiology 2013;266:548-54. 7. Wang S, Summers RM. Machine learning and radiology. Med Image Anal 2012;16:933-51. 8. Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform 2009;42:760-72. 9. Reiner B. Uncovering and improving upon the inherent deficiencies of radiology reporting through data mining. J Digit Imaging 2010;23: 109-18. 10. Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 2017;284:574-82. 11. Spampinato C, Palazzo S, Giordano D, Aldinucci M, Leonardi R. Deep learning for automated skeletal bone age assessment in X-ray images. Med Image Anal 2017;36:41-51. 12. Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M. Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Med Image Comput Comput Assist Interv 2013;16:246-53. 13. Krizhevsky A, SI, Hinton GE. ImageNet classification with deep convolutional neural networks. Adv Neural Inform Process Syst 2012: 1097-105. 14. Large Scale Visual Recognition Challenge (ILSVRC). Available at: http://www.image-net.org/challenges/LSVRC/. Accessed January 28, 2019. 15. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA. Inception-v4, InceptionResNet and the impact of residual connections on learning. arXiv Available at: https://arxiv.org/abs/1602.07261. Accessed June 6, 2019. 16. Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G. Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology 2019;291:272. 17. Li Y, Wu FX, Ngom A. A review on machine learning principles for multi-view biological data integration. Brief Bioinform 2018;19: 325-40. 18. Bal M, Amasyali MF, Sever H, Kose G, Demirhan A. Performance evaluation of the machine learning algorithms used in inference
1257
19.
20.
21.
22.
23.
24. 25.
mechanism of a medical decision support system. Sci World J 2014;2014:137896. Yan H, JY, Zheng J, Peng C, Li Q. A multilayer perceptron-based medical decision support system for heart disease diagnosis. Expert Syst Appl 2006;30:272-81. Bennett CC, Hauser K. Artificial intelligence framework for simulating clinical decision-making: a Markov decision process approach. Artif Intel Med 2013;57:9-19. Kurasawa H, Hayashi K, Fujino A, et al. Machine-learning-based prediction of a missed scheduled clinical appointment by patients with diabetes. J Diabetes Sci Tech 2016;10:730-6. Glover MT, Daye D, Khalilzadeh O, et al. Socioeconomic and demographic predictors of missed opportunities to provide advanced imaging services. J Am Coll Radiol 2017;14:1403-11. Harvey HB, Liu C, Ai J, et al. Predicting no-shows in radiology using regression modeling of data available in the electronic medical record. J Am Coll Radiol 2017;14:1303-9. US Food and Drug Administration. Mammography Quality Standards Reauthorization Act of 1998. Yi PH, Golden SK, Harringa JB, Kliewer MA. Readability of lumbar spine MRI reports: will patients understand? AJR Am J Roentgenol 2019;212:602-6.
1258
26. Martin-Carreras T, Cook TS, Kahn CE Jr. Readability of radiology reports: implications for patient-centered care. Clin Imaging 2018;54: 116-20. 27. Centers for Disease Control and Prevention. Scientific and Technical Information Simply Put. Available at: http://www.cdc.gov/ healthliteracy/pdf/Simply_Put.pdf. Accessed January 23, 2019. 28. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015;17:e44. 29. Hansberry DR, John A, John E, Agarwal N, Gonzales SF, Baker SR. A critical review of the readability of online patient education resources from RadiologyInfo.org. AJR Am J Roentgenol 2014;202: 566-75. 30. Cook TS, Oh SC, Kahn CE Jr. Patients’ use and evaluation of an online system to annotate radiology reports with lay language definitions. Acad Radiol 2017;24:1169-74. 31. Oh SC, Cook TS, Kahn CE Jr. PORTER: a Prototype System for Patient-Oriented Radiology Reporting. J Digit Imaging 2016;29: 450-4. 32. Rooney SP, Hoffman S, Perrin JC, Milliron KJ, Nees AV, Jorns JM. Patient-friendly pathology reports for patients with breast atypias. Breast J 2018;24:855-7.
Journal of the American College of Radiology Volume 16 n Number 9PB n September 2019