- Email: [email protected]
Just-in-Time Radiologist Decision Support: The Importance of PACS-Integrated Workflow
Just-in-Time Radiologist Decision Support: The Importance of PACS-Integrated Workflow Matthew B. Morgan, MD, MSa,e, Barton F. Branstetter IV, MDb,c,d, Chuck Clark, BSe, Jonathan House, BSe, Dustin Baker, BSe, and H. Ric Harnsberger, MDa,e
Purpose: Decision support systems for radiologists can provide information during image interpretation that may efficiently improve diagnostic accuracy and increase radiologists’ confidence. However, most decision support systems require radiologists to exit PACS, which may deter busy radiologists from pursuing decision support. The purpose of this study was to determine whether radiologists would use a PACS-integrated decision support tool more frequently than an equivalent nonintegrated system. Methods: Forty-eight radiology residents were randomly assigned to one of two groups: the control group was provided access to a radiology clinical decision support tool via Web access, which required the resident to launch a Web browser from a desktop icon and then log in to the decision support application. The experimental group was provided access to the same tool but was allowed to launch from a PACS-integrated portal with automated login and authentication. Halfway through the 10-month study period, the groups were switched. The main outcome measure was the average number of decision support sessions initiated each month over the study period. Results: The experimental (integrated) group had higher use than the control (nonintegrated) group by a factor of 3.0 (P ⬍ .05). When integrated access was removed from the experimental group, their use fell by 52%. When integrated access was granted to the control group, their use rose by only 20%. Conclusion: Integration with PACS improves radiologists’ use of clinical decision support tools. Integrated access is critical at the time of initial deployment, or acceptance of the decision support tools may be undermined. Key Words: Decision support, informatics, online tools, just-in-time learning J Am Coll Radiol 2011;8:497-500. Copyright © 2011 American College of Radiology
INTRODUCTION
The digital transformation of radiology has revolutionized the methods that we use for information access and information exchange. In the recent past, hard copy film, paper requisitions, and faxes were the usual media for information exchange. Reference materials included a
Department of Radiology, University of Utah School of Medicine, Salt Lake City, Utah. b Department of Radiology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania. c
Department of Otolaryngology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania. d Department of Biomedical Informatics, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania. e Amirsys, Inc, Salt Lake City, Utah. Corresponding author and reprints: Matthew B. Morgan, MD, MS, University of Utah School of Medicine, Department of Radiology, 50 N Medical Drive, Salt Lake City, UT 84132; e-mail: [email protected]. Disclosure: C.C, J.H and D.B are employees of Amirsys, Inc., and H.R.H. is CEO of Amirsys, Inc., in Salt Lake City. © 2011 American College of Radiology 0091-2182/11/$36.00 ● DOI 10.1016/j.jacr.2011.01.003
textbooks and hard copy journal articles. Because of the substantial workflow disruption and inefficiency inherent in physically moving to a new medium to access information, there was a necessary separation of work time and learning time. When confronted with an unfamiliar imaging appearance in their clinical work, radiologists were usually forced to rely on memorized facts and personal experience, or perhaps consult with more senior colleagues. With the introduction of PACS and the World Wide Web, radiologists now sit at powerful computer workstations with live Internet connections and access to seemingly limitless amounts of online information. The separation between work time and learning time has nearly disappeared as the physical barriers to information access have been markedly reduced. Short bursts of learning can now be injected into radiologists’ normal workflow without substantial disruption. This “just-in-time” or “pointof-care” learning may in fact be more potent than traditional textbooks because of the added clinical context. 497
498 Journal of the American College of Radiology/ Vol. 8 No. 7 July 2011
Moreover, because this on-demand learning occurs at the point of care, it also has the potential to directly improve quality of patient care. Furthermore, radiologists and other physicians have shown a strong interest in this method of learning [1-4]. Online decision support systems can provide just-intime supplemental information such as a differential diagnosis, comparison images, access to recent literature, or a review of pertinent anatomy. The use of a decision support tool may improve diagnostic accuracy, increase radiologists’ confidence, and improve the quality of interpretations, especially for pathology that is infrequently encountered or is outside radiologists’ area of expertise. However, most clinical decision support tools are standalone systems that require the radiologist exit the PACS environment and launch an additional system, which can disrupt the clinical workflow. This seemingly small workflow hurdle may be enough to deter busy radiologists from pursuing decision support. The purpose of this study was to determine whether a PACS-integrated decision support tool would be more frequently used by radiologists than an equivalent nonintegrated system. Two secondary hypotheses were also tested: (1) Does use decrease when PACS-integrated access is withdrawn? (2) Does use improve when PACSintegrated access is granted to users who were already using a nonintegrated tool?
number of learning sessions for each group within a calendar month. The control and experimental groups were compared over the first 5 months of the study using an analysis of covariance intended to control for the expected increase in use over time (outcome variable: number of sessions; main independent variable: control or experimental group; covariate: month of study). P values ⬍ .05 were considered significant. After the fifth month of the study, the access methods for the 2 groups were crossed over and followed for another 5 months. Since this crossover was not amenable to a repeated analysis of covariance because of the substantial carryover effects, 2 additional hypotheses were tested: (1) Does use decrease when PACS-integrated access is withdrawn? (2) Does use improve when PACS-integrated access is granted to users who were already using a nonintegrated tool? To determine whether use had changed after the change in PACS-integrated access, a 95% confidence interval for the average monthly use after crossover was compared with the peak use at the end of the initial 5-month study period for each group. All results were calculated using SPSS version 16.0 (SPSS, Inc, Chicago, Illinois). Our institutional review board approved this study. RESULTS
METHODS
The study was conducted at a large, university-based radiology department using all 50 radiology residents as subjects. One resident on medical leave and one coauthor of the study (M.B.M.) were excluded, for a total of 48 residents. The residents were randomly assigned to either the control or the experimental group, such that each study group had the same number of residents from each postgraduate year. The control group was provided access to an online radiology decision support tool (STATdx; Amirsys, Inc, Salt Lake City, Utah) via Web access, which required residents to minimize the PACS interface, launch a browser from a desktop icon, and then log in to the decision support Web page on the 3-monitor workstations used at our institution. The experimental group was provided access to the same tool but was given a PACS-integrated portal that launched the application directly from within the PACS environment with automated login and authentication. The experimental group was also allowed to use the more cumbersome login, which might be preferable when the PACS is not active or the resident is not on the hospital campus. A learning session for an individual was defined as a successful login, via either interface, that occurred ⱖ10 minutes after the previous login for that individual (logins within 10 minutes of each other were considered to be a single session). Monthly use was defined as the total
The numbers of sessions recorded in each of the 10 months of the study for each group are listed in Table 1. During the first 5-month study period, use of the decision support tool increased over time in both groups (Figure 1). Use of the integrated decision support showed a jump by an order of magnitude starting after the second month. In the fifth month, use reached 604 sessions in the experimental group, compared with 229 sessions in the control group. The experimental group averaged 279 sessions/month, compared with 92 sessions/month for the control group (P ⫽ .02). In the second 5-month study period, when PACSintegrated access was switched between the groups, average use dropped from 604 to 289 sessions/month (⫺52%; 95% confidence interval, ⫺50% to ⫺55%) in the group that lost PACS-integrated access and rose from 229 to 274 sessions/month (⫹20%; 95% confidence interval, ⫹14% to ⫹25%) in the group that gained integrated access (Figure 2) The change was statistically significant for both groups. DISCUSSION
Our results indicate that PACS-integrated decision support software significantly increased use compared with nonintegrated software. Although both types of decision support showed steady increases in use over the first 5 months of the study, the group with integrated access logged 3 times the number of sessions as the noninte-
Morgan et al/Radiologist Decision Support 499
Table 1. Number of logins per month to a Web-based clinical decision support tool, for PACS-integrated and nonintegrated users PACS Month Integrated Nonintegrated 1 2 3 4 5 6 7 8 9 10
50 164 169 410 604 229 279 315 303 321
0 97 55 79 229 308 272 309 248 231
Note: PACS ⫽ picture archiving and communication system. In the second half of the study (months 6-10), access was withdrawn from the PACS-integrated group and granted to the nonintegrated group to determine the effect of postdeployment PACS integration. There were 24 radiologists in each group.
grated group, confirming a strong initial association between integration and use. However, it is not known if this difference would have persisted, partially resolved, or equalized over time. The order of magnitude jump in use starting after month 2 likely follows the “technology adoption s curve” seen in diffusion of innovations theory [5]. Users tend to adopt technologies in a typical pattern moving from initial contact and awareness to trial use and adoption. In the second study period, the group that lost PACSintegrated access fell to about half of its prior use, whereas the group that gained integrated access increased its use by about 35% initially but then fell back to its preintegration baseline. Reasons for this behavior are not completely clear but may have to do with the difficulty of overcoming formed behaviors. Once introduced to one method of accomplishing a task, individuals tend to repeat that method because it is familiar or habitual. In fact, physicians may be
Fig 1. Results of the first 5-month period. There was a significant increase in the use of the decision support tool by the group with PACS-integrated access over the group without integrated access.
Fig 2. Analysis of percentage change in the use of decision support when the groups were switched. The users who lost PACS-integrated access dropped their use by 52%. The users who gained PACS-integrated access increased their use by an average of 19%, but after 5 months, use had returned to the previous baseline.
more prone to this tendency given the importance of methodic analysis in their daily work. There are many potential barriers to accessing decision support. Small technical factors can be significant deterrents to the use of decision support. Each step required to access a given support tool is a potential point of failure. For example, in a nonintegrated system, getting to the tool can be a problem if the link to launch the tool is lost amid numerous icons on a cluttered PC desktop, which is not uncommon on the shared workstations at our institution. If the link cannot be found, or if the link is inadvertently deleted from the desktop, one would have to remember the address of the system and manually type it in to the browser’s address bar. Once the tool is launched, the authentication process is another possible point of failure. With all the systems radiologists use, it can be difficult to recall all the unique usernames and passwords. If after several attempts the login to the decision support tool is unsuccessful, the radiologist may abandon the system altogether rather than take the time to track down the correct login information. Finally, even the number of clicks to get to the decision support tool can also be a significant factor. If access to the system requires exiting the PACS environment to launch another application that requires a distinct login, the user may decide that the extra hassle is not worth the benefit. Any risk for wasted time without successful payoff is a deterrent in a busy practice environment. As with all computerized systems, the designers of PACS and decision support systems should “make it easy to do it right” [6]. Embedding the decision support tool within the PACS is a convenient and effective way to inject learning into the workflow, and the closer the information access is to the workflow, the better chance it has to improve outcomes and affect clinical care. Future
500 Journal of the American College of Radiology/ Vol. 8 No. 7 July 2011
innovations in decision support will move beyond integration and explore ways for decision support to become context and patient specific. There were several limitations of this study. At the termination of the first phase, the numbers for both groups were trending up. Therefore, it is not known if either group would start trending down, continue to rise, or eventually plateau over time. The study population consisted of radiology residents, not practicing radiologists. Patterns of use by residents who are in training may vary from those of established practicing radiologists, but we believe that the beneficial effects of integrated workflow are generally applicable. In addition, other institutions and practices may have different technology environments that could limit the broad applicability of the study’s conclusions. For example, some practices may not have Internet access through PACS workstations. Finally, we recognize that use of a tool is only a surrogate indicator. Ultimately, the indicator of success is not just the use of modern technology but its impact on patient care. We did not measure outcomes on the basis of actual improvement of patient care or resident education. CONCLUSIONS
With the transition to digital workflows, radiologists are in a perfect position to take advantage of unprecedented
access to online information. Integration with PACS can increase the use of clinical decision support tools by embedding the tools in the clinical workflow. However, adding integration after users have learned an alternate means of access provides less benefit. Decision support tools that are embedded into the clinical workflow have the best chance of improving quality of care and, in the current climate of radiology outsourcing, may also help preserve radiologists’ added value.
REFERENCES 1. Kahn CE Jr, Ehlers KC, Wood BP. Radiologists’ preferences for justin-time learning. J Digit Imaging 2006;19:202-6. 2. Kahn CE Jr. Improving outcomes in radiology: bringing computer-based decision support and education to the point of care. Acad Radiol 2005;12: 409-14. 3. Galvin JR, D’Alessandro MP, Erkonen WE, Smith WL, el-Khoury GY, Weinstein JN. The virtual hospital. Providing multimedia decision support tools via the Internet. Spine 1995;20:1735-8. 4. D’Alessandro MP, Galvin JR, Santer DM, Erkonen WE. Hand-held digital books in radiology: convenient access to information. AJR Am J Roentgenol 1995;164:485-8. 5. Rogers EM. Diffusion of innovations. New York: Free Press; 1983. 6. James BC. Making it easy to do it right. N Engl J Med 2001;345:991-3.
Purpose: Decision support systems for radiologists can provide information during image interpretation that may efficiently improve diagnostic accuracy and increase radiologists’ confidence. However, most decision support systems require radiologists to exit PACS, which may deter busy radiologists from pursuing decision support. The purpose of this study was to determine whether radiologists would use a PACS-integrated decision support tool more frequently than an equivalent nonintegrated system. Methods: Forty-eight radiology residents were randomly assigned to one of two groups: the control group was provided access to a radiology clinical decision support tool via Web access, which required the resident to launch a Web browser from a desktop icon and then log in to the decision support application. The experimental group was provided access to the same tool but was allowed to launch from a PACS-integrated portal with automated login and authentication. Halfway through the 10-month study period, the groups were switched. The main outcome measure was the average number of decision support sessions initiated each month over the study period. Results: The experimental (integrated) group had higher use than the control (nonintegrated) group by a factor of 3.0 (P ⬍ .05). When integrated access was removed from the experimental group, their use fell by 52%. When integrated access was granted to the control group, their use rose by only 20%. Conclusion: Integration with PACS improves radiologists’ use of clinical decision support tools. Integrated access is critical at the time of initial deployment, or acceptance of the decision support tools may be undermined. Key Words: Decision support, informatics, online tools, just-in-time learning J Am Coll Radiol 2011;8:497-500. Copyright © 2011 American College of Radiology
INTRODUCTION
The digital transformation of radiology has revolutionized the methods that we use for information access and information exchange. In the recent past, hard copy film, paper requisitions, and faxes were the usual media for information exchange. Reference materials included a
Department of Radiology, University of Utah School of Medicine, Salt Lake City, Utah. b Department of Radiology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania. c
Department of Otolaryngology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania. d Department of Biomedical Informatics, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania. e Amirsys, Inc, Salt Lake City, Utah. Corresponding author and reprints: Matthew B. Morgan, MD, MS, University of Utah School of Medicine, Department of Radiology, 50 N Medical Drive, Salt Lake City, UT 84132; e-mail: [email protected]. Disclosure: C.C, J.H and D.B are employees of Amirsys, Inc., and H.R.H. is CEO of Amirsys, Inc., in Salt Lake City. © 2011 American College of Radiology 0091-2182/11/$36.00 ● DOI 10.1016/j.jacr.2011.01.003
textbooks and hard copy journal articles. Because of the substantial workflow disruption and inefficiency inherent in physically moving to a new medium to access information, there was a necessary separation of work time and learning time. When confronted with an unfamiliar imaging appearance in their clinical work, radiologists were usually forced to rely on memorized facts and personal experience, or perhaps consult with more senior colleagues. With the introduction of PACS and the World Wide Web, radiologists now sit at powerful computer workstations with live Internet connections and access to seemingly limitless amounts of online information. The separation between work time and learning time has nearly disappeared as the physical barriers to information access have been markedly reduced. Short bursts of learning can now be injected into radiologists’ normal workflow without substantial disruption. This “just-in-time” or “pointof-care” learning may in fact be more potent than traditional textbooks because of the added clinical context. 497
498 Journal of the American College of Radiology/ Vol. 8 No. 7 July 2011
Moreover, because this on-demand learning occurs at the point of care, it also has the potential to directly improve quality of patient care. Furthermore, radiologists and other physicians have shown a strong interest in this method of learning [1-4]. Online decision support systems can provide just-intime supplemental information such as a differential diagnosis, comparison images, access to recent literature, or a review of pertinent anatomy. The use of a decision support tool may improve diagnostic accuracy, increase radiologists’ confidence, and improve the quality of interpretations, especially for pathology that is infrequently encountered or is outside radiologists’ area of expertise. However, most clinical decision support tools are standalone systems that require the radiologist exit the PACS environment and launch an additional system, which can disrupt the clinical workflow. This seemingly small workflow hurdle may be enough to deter busy radiologists from pursuing decision support. The purpose of this study was to determine whether a PACS-integrated decision support tool would be more frequently used by radiologists than an equivalent nonintegrated system. Two secondary hypotheses were also tested: (1) Does use decrease when PACS-integrated access is withdrawn? (2) Does use improve when PACSintegrated access is granted to users who were already using a nonintegrated tool?
number of learning sessions for each group within a calendar month. The control and experimental groups were compared over the first 5 months of the study using an analysis of covariance intended to control for the expected increase in use over time (outcome variable: number of sessions; main independent variable: control or experimental group; covariate: month of study). P values ⬍ .05 were considered significant. After the fifth month of the study, the access methods for the 2 groups were crossed over and followed for another 5 months. Since this crossover was not amenable to a repeated analysis of covariance because of the substantial carryover effects, 2 additional hypotheses were tested: (1) Does use decrease when PACS-integrated access is withdrawn? (2) Does use improve when PACS-integrated access is granted to users who were already using a nonintegrated tool? To determine whether use had changed after the change in PACS-integrated access, a 95% confidence interval for the average monthly use after crossover was compared with the peak use at the end of the initial 5-month study period for each group. All results were calculated using SPSS version 16.0 (SPSS, Inc, Chicago, Illinois). Our institutional review board approved this study. RESULTS
METHODS
The study was conducted at a large, university-based radiology department using all 50 radiology residents as subjects. One resident on medical leave and one coauthor of the study (M.B.M.) were excluded, for a total of 48 residents. The residents were randomly assigned to either the control or the experimental group, such that each study group had the same number of residents from each postgraduate year. The control group was provided access to an online radiology decision support tool (STATdx; Amirsys, Inc, Salt Lake City, Utah) via Web access, which required residents to minimize the PACS interface, launch a browser from a desktop icon, and then log in to the decision support Web page on the 3-monitor workstations used at our institution. The experimental group was provided access to the same tool but was given a PACS-integrated portal that launched the application directly from within the PACS environment with automated login and authentication. The experimental group was also allowed to use the more cumbersome login, which might be preferable when the PACS is not active or the resident is not on the hospital campus. A learning session for an individual was defined as a successful login, via either interface, that occurred ⱖ10 minutes after the previous login for that individual (logins within 10 minutes of each other were considered to be a single session). Monthly use was defined as the total
The numbers of sessions recorded in each of the 10 months of the study for each group are listed in Table 1. During the first 5-month study period, use of the decision support tool increased over time in both groups (Figure 1). Use of the integrated decision support showed a jump by an order of magnitude starting after the second month. In the fifth month, use reached 604 sessions in the experimental group, compared with 229 sessions in the control group. The experimental group averaged 279 sessions/month, compared with 92 sessions/month for the control group (P ⫽ .02). In the second 5-month study period, when PACSintegrated access was switched between the groups, average use dropped from 604 to 289 sessions/month (⫺52%; 95% confidence interval, ⫺50% to ⫺55%) in the group that lost PACS-integrated access and rose from 229 to 274 sessions/month (⫹20%; 95% confidence interval, ⫹14% to ⫹25%) in the group that gained integrated access (Figure 2) The change was statistically significant for both groups. DISCUSSION
Our results indicate that PACS-integrated decision support software significantly increased use compared with nonintegrated software. Although both types of decision support showed steady increases in use over the first 5 months of the study, the group with integrated access logged 3 times the number of sessions as the noninte-
Morgan et al/Radiologist Decision Support 499
Table 1. Number of logins per month to a Web-based clinical decision support tool, for PACS-integrated and nonintegrated users PACS Month Integrated Nonintegrated 1 2 3 4 5 6 7 8 9 10
50 164 169 410 604 229 279 315 303 321
0 97 55 79 229 308 272 309 248 231
Note: PACS ⫽ picture archiving and communication system. In the second half of the study (months 6-10), access was withdrawn from the PACS-integrated group and granted to the nonintegrated group to determine the effect of postdeployment PACS integration. There were 24 radiologists in each group.
grated group, confirming a strong initial association between integration and use. However, it is not known if this difference would have persisted, partially resolved, or equalized over time. The order of magnitude jump in use starting after month 2 likely follows the “technology adoption s curve” seen in diffusion of innovations theory [5]. Users tend to adopt technologies in a typical pattern moving from initial contact and awareness to trial use and adoption. In the second study period, the group that lost PACSintegrated access fell to about half of its prior use, whereas the group that gained integrated access increased its use by about 35% initially but then fell back to its preintegration baseline. Reasons for this behavior are not completely clear but may have to do with the difficulty of overcoming formed behaviors. Once introduced to one method of accomplishing a task, individuals tend to repeat that method because it is familiar or habitual. In fact, physicians may be
Fig 1. Results of the first 5-month period. There was a significant increase in the use of the decision support tool by the group with PACS-integrated access over the group without integrated access.
Fig 2. Analysis of percentage change in the use of decision support when the groups were switched. The users who lost PACS-integrated access dropped their use by 52%. The users who gained PACS-integrated access increased their use by an average of 19%, but after 5 months, use had returned to the previous baseline.
more prone to this tendency given the importance of methodic analysis in their daily work. There are many potential barriers to accessing decision support. Small technical factors can be significant deterrents to the use of decision support. Each step required to access a given support tool is a potential point of failure. For example, in a nonintegrated system, getting to the tool can be a problem if the link to launch the tool is lost amid numerous icons on a cluttered PC desktop, which is not uncommon on the shared workstations at our institution. If the link cannot be found, or if the link is inadvertently deleted from the desktop, one would have to remember the address of the system and manually type it in to the browser’s address bar. Once the tool is launched, the authentication process is another possible point of failure. With all the systems radiologists use, it can be difficult to recall all the unique usernames and passwords. If after several attempts the login to the decision support tool is unsuccessful, the radiologist may abandon the system altogether rather than take the time to track down the correct login information. Finally, even the number of clicks to get to the decision support tool can also be a significant factor. If access to the system requires exiting the PACS environment to launch another application that requires a distinct login, the user may decide that the extra hassle is not worth the benefit. Any risk for wasted time without successful payoff is a deterrent in a busy practice environment. As with all computerized systems, the designers of PACS and decision support systems should “make it easy to do it right” [6]. Embedding the decision support tool within the PACS is a convenient and effective way to inject learning into the workflow, and the closer the information access is to the workflow, the better chance it has to improve outcomes and affect clinical care. Future
500 Journal of the American College of Radiology/ Vol. 8 No. 7 July 2011
innovations in decision support will move beyond integration and explore ways for decision support to become context and patient specific. There were several limitations of this study. At the termination of the first phase, the numbers for both groups were trending up. Therefore, it is not known if either group would start trending down, continue to rise, or eventually plateau over time. The study population consisted of radiology residents, not practicing radiologists. Patterns of use by residents who are in training may vary from those of established practicing radiologists, but we believe that the beneficial effects of integrated workflow are generally applicable. In addition, other institutions and practices may have different technology environments that could limit the broad applicability of the study’s conclusions. For example, some practices may not have Internet access through PACS workstations. Finally, we recognize that use of a tool is only a surrogate indicator. Ultimately, the indicator of success is not just the use of modern technology but its impact on patient care. We did not measure outcomes on the basis of actual improvement of patient care or resident education. CONCLUSIONS
With the transition to digital workflows, radiologists are in a perfect position to take advantage of unprecedented
access to online information. Integration with PACS can increase the use of clinical decision support tools by embedding the tools in the clinical workflow. However, adding integration after users have learned an alternate means of access provides less benefit. Decision support tools that are embedded into the clinical workflow have the best chance of improving quality of care and, in the current climate of radiology outsourcing, may also help preserve radiologists’ added value.
REFERENCES 1. Kahn CE Jr, Ehlers KC, Wood BP. Radiologists’ preferences for justin-time learning. J Digit Imaging 2006;19:202-6. 2. Kahn CE Jr. Improving outcomes in radiology: bringing computer-based decision support and education to the point of care. Acad Radiol 2005;12: 409-14. 3. Galvin JR, D’Alessandro MP, Erkonen WE, Smith WL, el-Khoury GY, Weinstein JN. The virtual hospital. Providing multimedia decision support tools via the Internet. Spine 1995;20:1735-8. 4. D’Alessandro MP, Galvin JR, Santer DM, Erkonen WE. Hand-held digital books in radiology: convenient access to information. AJR Am J Roentgenol 1995;164:485-8. 5. Rogers EM. Diffusion of innovations. New York: Free Press; 1983. 6. James BC. Making it easy to do it right. N Engl J Med 2001;345:991-3.