- Email: [email protected]
Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial
+
MODEL
Journal of Pediatric Urology (2015) xx, 1.e1e1.e8
a
Division of Urology, Ann & Robert H. Lurie Children’s Hospital of Chicago, Northwestern University Feinberg School of Medicine, 225 E. Chicago Ave., Box 24, Chicago, IL 60611, USA
b
Department of Urology, University of Oklahoma Health Science Center, 920 Stanton L. Young Blvd, W.P. 3150, Oklahoma City, OK 73104, USA
c Division of Pediatric Urology, University of Virginia School of Medicine, P.O. Box 800422 Charlottesville, VA 22908, USA
Correspondence to: Dennis B. Liu, Ann & Robert H. Lurie Children’s Hospital of Chicago, 225 E. Chicago Ave., Box 24, Chicago IL 60611, USA, Tel.: þ1 312 227 6340; fax: þ1 312 227 9412 [email protected] (D.B. Liu) [email protected] (B. Palmer) [email protected] (C.D.A. Herndon) [email protected] (M. Maizels) Keywords Hydronephrosis; Society for Fetal Urology; CEVL; eLearning; Ultrasound; Education Received 9 January 2015 Accepted 6 May 2015 Available online xxx
Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial Dennis B. Liu a, Blake Palmer b, C.D. Anthony Herndon c, Max Maizels a Summary Introduction It is unclear how clinicians learn to grade pediatric hydronephrosis (HN) and how effective their training has been. We sought to:
1. Assess how clinicians learn to grade HN and their confidence in their training and abilities and 2. To assess Computer Enhanced Visual Learning (CEVL) e-Learning to learn the Society for Fetal Urology (SFU) grading system for pediatric HN. Methods and materials A multi-institutional online survey was distributed to pediatric urologists, nephrologists, and radiologists. Respondents used a 6-point Likert scale (0 Z not confident to 5 Z very confident) to assess their confidence in knowledge of the criteria, indications, and ability to grade HN, and how they learned to grade. Participants assigned SFU grades to 15 neonatal ultrasounds (US). A CEVL module on the SFU grading system was accessed and a post-CEVL survey completed. Changes in confidence and accuracy of grading were compared before and after CEVL e-Learning. Results The most common method of learning was “casually during training” (44.5%). Significant increases in confidence in knowledge of criteria, indications, and ability to grade, as well as the accuracy of grading
were seen following CEVL e-Learning (Figure A and B). Discussion Although the SFU grading system is considered the predominant grading system for HN, its application in clinical practice has been inconsistent. While this may be due to the grading system itself, it is possible that deficient training and confidence are the root causes. Our data supports this by demonstrating that most clinicians receive only casual training and accordingly, report low confidence in their knowledge and ability to grade HN. Therefore, we conclude that there exists a strong need to improve the teaching of the SFU grading system. e-Learning has been shown to be effective in teaching difficult topics and skills. We demonstrate that e-Learning with CEVL is effective in increasing both the confidence and accuracy of SFU grading of pediatric HN. Limitations of our study include a small sample size, low response rate, and discrepant participation. Furthermore, we did not assess the extent to which the CEVL module was used or include a control group learning through traditional means. Therefore, we were unable to evaluate the efficiency of learning or be certain that the improvements seen were derived exclusively from CEVL. Conclusion Current training in SFU grading of HN is mostly unstructured and inaccurate grading is common. Learners who use CEVL show improvements in their confidence and ability to SFU grade HN.
http://dx.doi.org/10.1016/j.jpurol.2015.05.008 1477-5131/ª 2015 Journal of Pediatric Urology Company. Published by Elsevier Ltd. All rights reserved.
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
1.e2
D.B. Liu et al.
Figure Confidence (A) and accuracy of grading (B) before and after CEVL e-Learning. a Confidences are expressed on a 6-point Likert scale (0 Z no confidence to 5 Z very confident) b Number of correctly graded US are out of 15. * p-value <0.05 denotes statistical significance. CEVL-Computer enhanced visual learning.
Introduction
Methodology
Antenatal hydronephrosis (HN) affects up to 4.5% of all pregnancies [1e4]. However, up to 88% of affected fetuses will ultimately not require urologic intervention [5]. Predictors of the need for urologic intervention are predicated on the grade of the initial postnatal ultrasound (US). In an effort to standardize grading, the Society for Fetal Urology (SFU) introduced a 5-point grading system (0e4) based upon the extent of caliceal dilatation and the thickness of the parenchyma overlying the calices [6,7]. Since its introduction, the SFU system has become the most widely used method to grade pediatric HN [8e13]. The ability to accurately grade HN is an essential skill for pediatric urologists (PU), nephrologists (PN), and radiologists (PR). However, we believe that many misconceptions regarding the grading system exist, which decrease the ability to accurately grade HN and in turn, reduce the utility of the system. Research on practices to learn SFU grading may ultimately lead to improvements in the accuracy of this skill. It is currently unclear how the training of clinicians has been accomplished and how effective this training has been. Computer Enhanced Visual Learning (CEVL) was created as web-based educational tool to deliver content in an easily accessible, interactive format. The CEVL platform was first shown effective for teaching surgical skills [14e16]. It was then adapted to teach grading of HN and has been shown to be effective for teaching urology residents at a single institution [17]. The primary objective of this study is to examine current methods of learning to grade HN amongst PU, PN, and PR and assess their confidence in their training. Secondly, we aim to assess the effectiveness of CEVL as an eLearning tool to improve the confidence and ability to SFU grade HN at multiple institutions and amongst PU, PN, and PR.
Intake survey After institutional review board determination of exemption, email invitations to participate in an anonymous, online survey were sent to physician extenders, residents, fellows, and attending physicians in the divisions of PU, PN, and PR at 6 institutions (listed in Table 1) as well as to members of the SFU, with a follow up email sent 2e4 weeks after the initial invitation. Participants were queried on how they learned the SFU grading system and asked to assess their confidence in their knowledge of the criteria to grade, indications to grade, and their ability to grade pediatric HN according to a 6-point Likert scale (where 0 Z not confident and 5 Z very confident). Respondents were then asked to assign SFU grades to 15 representative newborn US images that had been selected and graded by consensus of the authors. None of the participants had previous access to the CEVL module.
CEVL module on SFU grading of hydronephrosis After completion of the intake survey, participants were directed to an educational module developed using the CEVL platform on how to SFU grade HN as previously described (17). Briefly, an online tutorial was developed at our institution in collaboration with co-authors from two other institutions to present the key concepts and skills of SFU grading, including background knowledge, relevant anatomy, criteria to grade, and indications for SFU grading in accordance with the original description of the grading system [6,7]. Improvements to the module’s layout were made to enhance user interactivity. Specifically, sections providing self-assessment and immediate feedback including explanatory texts and graphic overlays were built (Fig. 1aec). The CEVL module was also made accessible via
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
CEVL e-learning Improve SFU Grading of Pediatric Hydronephrosis Table 1
1.e3
Study demographics.
Total Institution Lurie Children’s Hospital University of Oklahoma University of Virginia Loyola University Rush University Long Island Jewish/Cohen’s Society for Fetal Urology Department Urology Nephrology Radiology Professional role Resident Fellow Attending APN/PA Level of training1 PGY 1 PGY 2 PGY 3 PGY 4 PGY 5 PGY 6 How did you learn SFU grading Casually during training No one Textbooks or journal articles Formal lectures Web learning No answer Confidence in knowledge of: Criteria to grade Indications to grade Ability to grade Number of correctly graded US
Completed the study n (%)
Did not complete the study n (%)
24
30
1 (4.2) 11 (45.8) 2 (8.3) 2 (8.3) 1 (4.2) 2 (8.3) 5 (20.8)
8 (26.7) 8 (26.7) 0 (0.0) 0 (0.0) 3 (10.0) 1 (3.3) 10 (33.3)
21 (87.5) 2 (8.3) 1 (4.2)
23 (76.7) 4 (13.3) 3 (10.0)
13 (54.2) 5 (20.8) 5 (20.8) 1 (4.2)
9 7 9 5
p-value 0.035a,*
0.864a
0.264a (30.0) (23.3) (30.0) (16.7) 0.804a 3 2 2 4 2 0
(23.0) (15.4) (15.4) (30.8) (15.4) (0)
1 2 2 1 2 1
(11.1) (22.2) (22.2) (11.1) (22.2) (11.1)
15 (62.5) 3 (12.5) 3 (12.5) 2 (8.3) 1 (4.2) 0 (0.0)
9 7 5 2 3 4
(30.0) (23.3) (16.7) (6.7) (10.0) (13.3)
0.382a
3.5 3.0 3.0 9.0
(IQR (IQR (IQR (IQR
Z Z Z Z
3.0e4.0) 3.0e4.0) 2.25e4.0) 7.0e9.0)
3.5 3.0 3.0 9.0
(IQR (IQR (IQR (IQR
Z Z Z Z
2.0e4.0) 2.0e4.0) 2.0e4.0) 6.0e10.0)
0.566a 0.585a 0.446a 0.909b
* Statistical significance at p < 0.05. 1 Applicable only for residents where PGY Z Post-graduate year, APN Z Advanced Practice Nurse, PA Z Physician Assistant US Z Ultrasound, IQR Z Interquartile Range (25th to 75th Quartile). a Fisher exact test. b ManneWhitney U test.
smartphones and tablet computers. Experts in the field from multiple institutions reviewed and approved the completed module for content validity.
CEVL to their peers and whether they thought CEVL was superior to current learning methods.
Statistical analysis Exit survey After completion of the module, participants were asked to complete an exit survey that re-assessed their confidence in their knowledge of the criteria and indications for SFU grading, and their ability to grade HN. Participants were also asked to re-grade the same 15 US images. Using a 6point Likert scale (0-strongly disagree to 5-strongly agree), respondents were asked whether they would recommend
All data were collected and tabulated electronically and analyzed using IBM SPSS Statistics (version 22, Armonk, NY.). While demographic and methods of learning information were obtained from all intake surveys, only participants who completed both the intake and exit surveys were included in the pre and post CEVL analysis. Primary outcomes assessed were changes in confidence in knowledge of the criteria to grade, indications for grading, and the ability to grade HN
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
1.e4
D.B. Liu et al.
Figure 1 Screenshots from the Computer Enhanced Visual Learning (CEVL) module on the Society for Fetal Urology (SFU) grading system of pediatric hydronephrosis. Self-assessment and feedback increases interactivity. (a.) A gallery of available US for selfassessment. (b.) A sample ultrasound to grade for self-assessment. (c.) An example of the feedback available. Following assignment of an SFU grade, learners have the opportunity to receive immediate feedback. To better illustrate the anatomy, interactive buttons (arrow) allow color overlays (light blue) of the anatomical structures of the kidney (minor calyces) as seen on renal ultrasound. This interactivity clearly demonstrates the ultrasound correlation with renal anatomy and enhances the educational experience of the learner. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
and the accuracy of grading. Categorical data were compared by Fisher’s exact test or Chi-squared as appropriate. Nonparametric data were compared by the Wilcoxon Signed Rank test for paired samples and ManneWhitney U test for independent samples. A p-value < 0.05 was considered to be statistically significant.
Results A total of 54 intake surveys were completed. Study participants’ demographics are shown in Table 1. Participants were mostly from the field of PU (44, 81.5%) and a large number of responders were residents (22, 40.7%). The most common method of learning the SFU grading system was reported as “casually during training or medical school” (24, 44.5%), with the second most common reported as being taught by “no one” (10, 18.5%). Four (7.4%) respondents reported receiving formal lectures. The median participants’ confidence in knowledge of the criteria to grade, indication to grade, and ability to grade were 3.5 (IQR Z 2e4), 3.0 (IQR Z 2e4), and 3.0 (IQR Z 2e4) respectively. The median number of correctly graded ultrasounds was 9 (IQR Z 6e10) out of 15 (60%). A total of 13 residents (Urology), 5 fellows (4 PU, 1 PN), 5 attendings (3 PU, 1 PN, 1 PR), and 1 physician assistant (PU) (n Z 24, 44.4%) completed the exit surveys and were included in the comparative analysis. 30 participants did not complete the exit survey and were included only in the assessment of pre-CEVL confidence in knowledge and ability to grade, accuracy of grading, and how they learned to grade. The only significant difference between those who completed the study and those who did not was seen in the participants’ institutions (p Z 0.035) (Table 1).
CEVL learning increased median confidence in knowledge of the criteria to grade [3.5 (IQR Z 3.0e4.0) to 4 (IQR Z 3.25e5.0), p < 0.001], indication to grade [3.0 (IQR Z 3.0e4.0) to 4.0 (IQR Z 3.0e4.75), p Z 0.007], and ability to grade [3.0 (IQR Z 2.25e4.0) to 4.0 (IQR Z 3.0e4.0), p < 0.001]. Furthermore, the median number of correctly graded US increased from 9.0 (60%, IQR Z 7.0e9.0) to 10.0 (67%, IQR Z 9.0e12.0) (p Z 0.006) (Table 2). A total of 17 (70.8%) participants improved the number of correctly graded US after CEVL learning. Increased accuracy of grading was seen for grades 1e4 after CEVL learning (Table 3). Greatest improvement in grading after CEVL learning was seen in SFU grade 2 (15.48%, p Z 0.004). SFU grade 3 had the lowest accuracy of grading both before (44.4% correct) and after (51.39%) CEVL learning. The most common grade mistakenly assigned for SFU grade 3 was grade 4 (40.28%). The accuracy of identifying “clinically significant” HN (grades 3 and 4) increased from 83.3% to 93.3% after CEVL learning while “clinically insignificant” (grades 0,1, and 2) decreased from 92.9% to 85.4%. A total of 23 out of 24 (95.8%) respondents agreed or strongly agreed that they would recommend CEVL for learning the SFU grading system. Seventy-nine percent of respondents agreed or strongly agreed that CEVL learning was superior to other available methods of learning.
Discussion The skill to accurately SFU grade HN is critical for daily clinical care for PU, PN, and PR. Accurate grades allow for more objective communication amongst caregivers. This likely results in improved patient care. Although commonly used, little is known about how clinicians are learning to
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
CEVL e-learning Improve SFU Grading of Pediatric Hydronephrosis Table 2
1.e5
Results of CEVL learning on the accuracy and confidence in the SFU grading system. Pre-CEVL
Post-CEVL
p-value (95% CI)
All respondents (n Z 24) Confidence Criteria Indications Ability to grade Correctly Graded US Subgroups Residents (n Z 13) Confidence Criteria Indications Ability to grade Correctly graded US Fellows (n Z 5) Confidence Criteria Indications Ability to grade Correctly graded US Attendings (n Z 5) Confidence Criteria Indications Ability to grade Correctly graded US
Median
IQR
Median
IQR
3.50 3.00 3.00 9.0 (60%)
3.00e4.00 3.00e4.00 2.25e4.00 7.00e9.00
4.00 4.00 4.00 10.0 (67%)
3.25e5.00 3.00e4.75 3.00e4.00 9.0e12.0
3.00 3.00 3.00 8.00 (53%)
2.00e4.80 2.50e4.00 2.00e3.50 6.00e9.00
4.00 4.00 4.00 9.00 (60%)
3.00e4.00 3.00e4.00 3.00e4.00 8.50e12.00
5.00 3.00 3.00 9.00 (60%)
3.50e5.00 2.00e4.50 2.00e4.50 8.00e11.00
4.00 5.00 4.00 11.00 (73%)
3.25e4.50 3.50e5.00 3.50e4.50 9.50e12.50
4.00 4.00 4.00 9.00 (60%)
3.50e5.00 3.50e5.00 3.50e5.00 7.00e9.50
5.00 5.00 5.00 10.00 (67%)
4.00e5.00 4.00e5.00 4.00e5.00 7.00e11.50
<0.001* 0.007* <0001* 0.006*
*Wilcoxon Signed Rank test, statistical significance p < 0.05. IQR Z Interquartile range (25th to 75th Quartile). CEVL Z Computer Enhanced Visual Learning. SFU Z Society for Fetal Urology. US Z Ultrasound.
SFU grade HN and whether changes in delivery of education could improve their training. Our study demonstrates that most clinicians receive little to no formal training in SFU grading, leading to inaccuracies of HN grading.
Table 3 Grade
Furthermore, we demonstrate that structured, electronic learning (e-Learning) through CEVL increases both the clinicians’ confidence in the SFU grading system and improves their accuracy of grading.
Accuracy per SFU grade. Pre-CEVL
Most common incorrect grade
Post-CEVL
Most common incorrect grade
0 79.17% Equally 2 and 3 (8.3%) 66.67% 3 (25%) 1 50.00% 0 (43.75%) 64.58% 0 (22.92%) 2 55.95% 1 (32.74%) 71.43% 3 (14.88%) 3 44.44% 4 (36.11%) 51.39% 4 (40.28%) 4 66.67% 3 (20.83%) 81.25% 3 (14.58%) Clinically significant HN correctly graded as clinically significant 83.30% 93.30% Clinically insignificant HN correctly graded as clinically insignificant 92.9% 85.40%
Percent change
p-valuea
12.50% 14.58% 15.48% 6.95% 14.58%
0.517 0.216 0.004a,* 0.505 0.162 0.016b,* 0.005b,*
*Statistical significance p < 0.05. Abbreviations: SFU Z Society for Fetal Urology, CEVL Z Computer Enhanced Visual Learning, HN Z Hydronephrosis. Please note: Clinically significant HN includes SFU grades 3 and 4; Clinically Insignificant HN includes SFU grades 0, 1, and 2. a Fisher exact test. b Chi-squared test.
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
1.e6 With the proliferation of technology and the increasing constraints of time and resources in modern medical education, e-Learning has gained increased popularity [18]. It has been found to be effective in teaching a wide variety of clinical skills, such as ECG interpretation and pediatric resuscitation [19,20]. Advantages of e-Learning include accessibility, flexibility, interactivity and the ability for self-assessment, shifting the focus from teacher to learner [21]. e-Learning increases accessibility by freeing education from scheduling and geographic constraints [22]. It allows learners to individualize learning by focusing on problematic concepts, thereby increasing flexibility. Interactivity is increased by the incorporation of multimedia to enhance the delivery of educational materials and by providing the opportunity for selfassessment, leading to increased efficacy of learning [23,24]. The CEVL method was developed as a mechanism for eLearning. CEVL is a web-based program that provides a structured learning platform that is modifiable, interactive, and readily available. The principles of CEVL are an adaptation of the Ericsson method of deliberate practice and are rooted in the fundamentals of: 1. preparedness, 2. performance, and 3. feedback with remediation [25]. CEVL was originally designed to teach surgical skills to residents and has been previously demonstrated to improve teaching of pediatric orchiopexy [14e16]. Previously, Marks et al. demonstrated in a pilot study that CEVL learning was also adaptable to teach diagnostic skills such as how to SFU grade pediatric HN [17]. In this study, 21 urology residents and 8 medical students were asked to grade 16 pediatric US. The entire group then viewed the CEVL module during a conference and they were asked to re-grade the same 16 ultrasounds. An overall 21% mean improvement in accuracy of SFU HN grade was seen after CEVL learning. While demonstrating that CEVL learning is effective when used in a traditional lecture setting, Marks’s study did not assess CEVL’s e-Learning attributes of increased accessibility, interactivity, and ability for self-assessment. Our study more accurately assesses CEVL’s e-Learning attributes, demonstrating its effectiveness in improving the learning of SFU grading remotely at multiple institutions through self-study. By providing unlimited electronic access and expanding access to mobile devices, we sought to increase CEVL’s accessibility. Furthermore, by incorporating self-assessment with feedback, the interactivity of CEVL was improved (Fig. 1aec). Improved confidence and accuracy to grade seen in our study are testaments to the effectiveness of e-Learning through CEVL. One of the concerns raised about the SFU grading system has been its reliability. Keays et al. found that the inter-rater reliability of the SFU grading system was substantial for grade 0, moderate for grades 1, 2, and 4, and only slight to fair for SFU grade 3 [26]. The authors attributed this finding to a lack of precision of the SFU system, especially for higher grades. As such, the authors advocated for the need to refine the SFU grading system to better differentiate grades 3 and 4. This has led to a modification to the SFU grading system by Silbai et al., who subclassified SFU grade 4 into segmental (4a) and diffuse (4b) [27].
D.B. Liu et al. However, an alternative explanation may lie in the training clinicians received in SFU grading. In Keays’s study, raters were provided only written instructions on SFU grading without further training. Furthermore, the provided instructions categorized any degree of parenchymal loss as SFU grade 4. In contrast, the SFU grading system does in fact restrict grade 4 to parenchymal loss of greater than 50% [6]. In our study, we demonstrated that most clinicians receive little to no formal training in SFU grading and that given structured, web-based training, the accuracy of grading improved. Thus, it is conceivable that the modest inter-rater reliability of the SFU grading system seen in Keays’ study reflects not deficiencies in the system itself but rather deficiencies in knowledge. Future study of inter-rater reliability after structured learning, such as CEVL, may be warranted. Interestingly, our study demonstrated that accuracy of grading HN as “clinically significant” versus “insignificant” increased after CEVL learning while accurate grading of “clinically insignificant” HN decreased after CEVL eLearning. This finding may be due to increased awareness of the grading system and a possible bias of learners to over-grade HN when undecided in-between grades. Furthermore, grading of SFU grade 3 was most inaccurate and showed the least improvement with CEVL learning. While a lack of precision for SFU grades 3 and 4 may exist within the SFU grading system, it is also possible that our CEVL module did not emphasize enough the distinction between these two grades. Further refinements to the contents of the CEVL module may be beneficial to improve the accuracy. Recently, a new multidisciplinary classification system was introduced for prenatal and postnatal urinary tract dilation [28]. As acceptance of this new classification system becomes more widespread, the need for educational tools to help familiarize clinicians with this new system will be needed. We believe that our study shows the effectiveness of CEVL to deliver such educational content and that such a platform may be readily adapted to teach the new classification system of UTD. Our study does have its limitations. First, the sample size of our study was small as only 24 of the original 54 subjects completed the study. The small sample size, low participation rate, discrepant resident and institutional participation raise the possibility of selection bias, with individuals who felt less confident in their knowledge more likely to participate. Furthermore, the study population was skewed towards PU and residents. The small numbers of fellows, attending physicians, PN, and PR limit meaningful analyses on the effectiveness of CEVL learning for more experienced clinicians and for different pediatric specialists. While a number of factors may be responsible for the low study completion rate, including a lack of interest or technical difficulties, the comprehensive design of the module as an educational tool may have required a time-commitment greater than desired for many potential participants. However, among the 24 participants who completed the module, 23 (96%) agreed or strongly agreed that the module length was appropriate. Secondly, our study was not designed to monitor the number of times CEVL was accessed, the time spent on the module, and whether the CEVL module was
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
CEVL e-learning Improve SFU Grading of Pediatric Hydronephrosis completed. Therefore, we were unable to determine the efficiency of learning through CEVL and to ascertain that the improvements seen were exclusively derived from CEVL. Third, the lack of a control group learning through traditional methods, such as didactic lectures, prevented comparison of CEVL learning to other forms of education. However, the goal of this study was to demonstrate that CEVL does improve learning, rather than examining if it was better than other methods of learning. Lastly, we did not assess the impact of CEVL learning on long-term knowledge retention.
Conclusion In this study, we demonstrate that current training to SFU grading of HN is mostly unstructured and inaccurate grading is common. The CEVL platform provides a structured eLearning approach to learning the SFU grading system. Learners who use CEVL show improvements in their confidence in their knowledge of and ability to SFU grade HN.
Ethical approval This study was designated exempt by the Institutional Review Board at our institution.
Funding None.
Conflict of interest Max Maizels, MD is the Co-founder and co-Executive Director of CEVL for Healthcare, Inc.
Acknowledgement We would like to thank Karen Rychlik, MS, for providing statistical assistance with this project.
References [1] Ismaili K, Hall M, Donner C, Thomas D, Vermeylen D, Avni FE. Results of systematic screening for minor degrees of renal pelvis dilatation in an unselected population. Am J Obstet Gynecol 2003;188:242e6. [2] Coplen DE, Austin PF, Yan Y, Blanco VM, Dicke JM. The magnitude of fetal renal pelvic dilatation can identify obstructive postnatal hydronephrosis, and direct postnatal evaluation and management. J Urol 2006;176:724e7. [3] Morin L, Cendron M, Crombleholme TM, Garmel SH, Klauber GT, D’Alton ME. Minimal hydronephrosis in the fetus: clinical significance and implications for management. J Urol 1996;155:2047e9. [4] Hothi DK, Wade AS, Gilbert R, Winyard PJD. Mild fetal renal pelvis dilatation-much ado about nothing? Clin J Am Soc Nephrol 2009;4:168e77.
1.e7
[5] Lee RS, Cendron M, Kinnamon DD, Nguyen HT. Antenatal hydronephrosis as a predictor of postnatal outcome: a metaanalysis. Pediatrics 2006;118:586e96. [6] Maizels M, Reisman ME, Flom LS, Nelson J, Fernbach S, Firlit CF, et al. Grading nephroureteral dilatation detected in the first year of life: correlation with obstruction. J Urol 1992; 148:609e14. [7] Fernbach SK, Maizels M, Conway JJ. Ultrasound grading of hydronephrosis: introduction to the system used by the Society for Fetal Urology. Pediatr Radiol 1993;23:478e80. [8] Nguyen HT, Herndon CD, Cooper C, Gatti H, Kirsch A, Kokorowski P, et al. The Society for Fetal urology consensus statement on the evaluation and management of antenatal hydronephrosis. J Pediatr Urol 2010;6:212e31. [9] Sidhu G, Beyene J, Rosenblum ND. Outcome of isolated antenatal hydronephrosis: a systemic review and meta-analysis. Pediatr Nephrol 2006;21:218e24. [10] Chi T, Feldstein VA, Nguyen HT. Increased echogenicity as a predictor of poor renal function in children with grade 3 to 4 hydronephrosis. J Urol 2006;175:1898e901. [11] Hafez AT, McLorie G, Bagli D, Khoury A. Analysis of trends on serial ultrasounds for high grade neonatal hydronephrosis. J Urol 2002;168:1518e21. [12] Erickson BA, Maizels M, Shore RM, Pazona JF, Hagerty JA, Yerkes EB, et al. Newborn society of fetal urology grade 3 hydronephrosis is equivalent to preserved percentage differential function. J Pediatr Urol 2007;3:382e6. [13] Onen A, Jayanthi VR, Koff SA. Long-term followup of prenatallydetected severe bilateral newborn hydronephrosis initially managed nonoperatively. J Urol 2002;168:1118e20. [14] Maizels M, Yerkes E, Macejko A, Hagerty J, Chaviano AC, Cheng EY, et al. A new computer enhanced visual learning method to train urology residents in pediatric orchiopexy: a prototype for accreditation council for graduate medical education. J Urol 2008;180:1814e8. [15] McQuiston L, MacNeily A, Liu D, Mickelson J, Yerkes E, Chaviano A, et al. Computer enhanced visual learning method to train urology residents in pediatric orchiopexy provided a consistent learning experience in a multi-institutional trial. J Urol 2010;184:1748e53. [16] Matoka D, Marks A, Stoltz R, Maizels M. Utilization of computer enhanced visual learning (CEVL) method improves endoscopic diagnosis of posterior urethral valves. J Ped Urol 2013;9: 498e502. [17] Marks A, Maizels M, Mickelson J, Yerkes E, Herndon CDA, Lane J, et al. Effectiveness of the computer enhanced learning method in teaching the society for fetal urology hydronephrosis grading system for urology trainees. J Ped Urol 2011;7:113e7. [18] Evgeniou E, Loizou P. The theoretical base of e-learning and its role in surgical education. J Surg Educ 2012:665e9. [19] Montassier E, Hardouin J, Segard J, Batard E, Potel G, Planchon B, et al. e-Learning versus lecture-based courses in ECG interpretation for undergraduate medical students: a randomized noninferiority study. Eur J Emerg Med 2014:00. [20] O’Leary F. Paediatric resuscitation training: is e-Learning the answer? A before and after pilot study. J Paediatr Child Health 2012;48:529e33. [21] Ellaway R, Masters K. AMEE guide 32: e-Learning in medical education. Part 1: learning, teaching, and assessment. Med Teach 2008;30:455e73. [22] Segal G, Balik C, Hovav B, Mayer A, Rozani V, Damary I, et al. Online nephrology course replacing a face to face course in nursing schools’ bachelor’s program: a prospective, controlled trial in four Israeli nursing schools. Nurse Educ Today 2013;33: 1587e91. [23] Issa N, Schuller M, Santacaterina S, Shapiro M, Wang E, Mayer RE, et al. Applying multimedia design principles
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
1.e8 enhances learning in medical education. Med Educ 2011;45: 818e26. [24] Cook DA, Thompson WG, Thomas KG, Thomas MR, Pankratz VS. Impact of self-assessment questions and learning styles in web-based learning: a randomized, controlled, crossover trial. Acad Med 2006;81:231e8. [25] Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 2004;79(10 suppl):S70. [26] Keays MA, Guerra JM, Mihill J, Raju G, Al-Asheeri N, Geier P, et al. Reliability assessment of Society for Fetal Urology
D.B. Liu et al. ultrasound grading system for hydronephrosis. J Urol 2008; 180:1680e3. [27] Sibai H, Pippi Salle JL, Houle AM, Lambert R. Hydronephrosis with diffuse and segmental cortical thinning: impact on renal function. J Urol 2001;165:2293e5. [28] Nguyen HT, Benson CB, Bromley B, Campbell JB, Chow J, Coleman B, et al. Multidisciplinary consensus on the classification of prenatal and postnatal urinary tract dilation (UTD classification system). J Ped Urol 2014;10(6):982e98.
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
MODEL
Journal of Pediatric Urology (2015) xx, 1.e1e1.e8
a
Division of Urology, Ann & Robert H. Lurie Children’s Hospital of Chicago, Northwestern University Feinberg School of Medicine, 225 E. Chicago Ave., Box 24, Chicago, IL 60611, USA
b
Department of Urology, University of Oklahoma Health Science Center, 920 Stanton L. Young Blvd, W.P. 3150, Oklahoma City, OK 73104, USA
c Division of Pediatric Urology, University of Virginia School of Medicine, P.O. Box 800422 Charlottesville, VA 22908, USA
Correspondence to: Dennis B. Liu, Ann & Robert H. Lurie Children’s Hospital of Chicago, 225 E. Chicago Ave., Box 24, Chicago IL 60611, USA, Tel.: þ1 312 227 6340; fax: þ1 312 227 9412 [email protected] (D.B. Liu) [email protected] (B. Palmer) [email protected] (C.D.A. Herndon) [email protected] (M. Maizels) Keywords Hydronephrosis; Society for Fetal Urology; CEVL; eLearning; Ultrasound; Education Received 9 January 2015 Accepted 6 May 2015 Available online xxx
Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial Dennis B. Liu a, Blake Palmer b, C.D. Anthony Herndon c, Max Maizels a Summary Introduction It is unclear how clinicians learn to grade pediatric hydronephrosis (HN) and how effective their training has been. We sought to:
1. Assess how clinicians learn to grade HN and their confidence in their training and abilities and 2. To assess Computer Enhanced Visual Learning (CEVL) e-Learning to learn the Society for Fetal Urology (SFU) grading system for pediatric HN. Methods and materials A multi-institutional online survey was distributed to pediatric urologists, nephrologists, and radiologists. Respondents used a 6-point Likert scale (0 Z not confident to 5 Z very confident) to assess their confidence in knowledge of the criteria, indications, and ability to grade HN, and how they learned to grade. Participants assigned SFU grades to 15 neonatal ultrasounds (US). A CEVL module on the SFU grading system was accessed and a post-CEVL survey completed. Changes in confidence and accuracy of grading were compared before and after CEVL e-Learning. Results The most common method of learning was “casually during training” (44.5%). Significant increases in confidence in knowledge of criteria, indications, and ability to grade, as well as the accuracy of grading
were seen following CEVL e-Learning (Figure A and B). Discussion Although the SFU grading system is considered the predominant grading system for HN, its application in clinical practice has been inconsistent. While this may be due to the grading system itself, it is possible that deficient training and confidence are the root causes. Our data supports this by demonstrating that most clinicians receive only casual training and accordingly, report low confidence in their knowledge and ability to grade HN. Therefore, we conclude that there exists a strong need to improve the teaching of the SFU grading system. e-Learning has been shown to be effective in teaching difficult topics and skills. We demonstrate that e-Learning with CEVL is effective in increasing both the confidence and accuracy of SFU grading of pediatric HN. Limitations of our study include a small sample size, low response rate, and discrepant participation. Furthermore, we did not assess the extent to which the CEVL module was used or include a control group learning through traditional means. Therefore, we were unable to evaluate the efficiency of learning or be certain that the improvements seen were derived exclusively from CEVL. Conclusion Current training in SFU grading of HN is mostly unstructured and inaccurate grading is common. Learners who use CEVL show improvements in their confidence and ability to SFU grade HN.
http://dx.doi.org/10.1016/j.jpurol.2015.05.008 1477-5131/ª 2015 Journal of Pediatric Urology Company. Published by Elsevier Ltd. All rights reserved.
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
1.e2
D.B. Liu et al.
Figure Confidence (A) and accuracy of grading (B) before and after CEVL e-Learning. a Confidences are expressed on a 6-point Likert scale (0 Z no confidence to 5 Z very confident) b Number of correctly graded US are out of 15. * p-value <0.05 denotes statistical significance. CEVL-Computer enhanced visual learning.
Introduction
Methodology
Antenatal hydronephrosis (HN) affects up to 4.5% of all pregnancies [1e4]. However, up to 88% of affected fetuses will ultimately not require urologic intervention [5]. Predictors of the need for urologic intervention are predicated on the grade of the initial postnatal ultrasound (US). In an effort to standardize grading, the Society for Fetal Urology (SFU) introduced a 5-point grading system (0e4) based upon the extent of caliceal dilatation and the thickness of the parenchyma overlying the calices [6,7]. Since its introduction, the SFU system has become the most widely used method to grade pediatric HN [8e13]. The ability to accurately grade HN is an essential skill for pediatric urologists (PU), nephrologists (PN), and radiologists (PR). However, we believe that many misconceptions regarding the grading system exist, which decrease the ability to accurately grade HN and in turn, reduce the utility of the system. Research on practices to learn SFU grading may ultimately lead to improvements in the accuracy of this skill. It is currently unclear how the training of clinicians has been accomplished and how effective this training has been. Computer Enhanced Visual Learning (CEVL) was created as web-based educational tool to deliver content in an easily accessible, interactive format. The CEVL platform was first shown effective for teaching surgical skills [14e16]. It was then adapted to teach grading of HN and has been shown to be effective for teaching urology residents at a single institution [17]. The primary objective of this study is to examine current methods of learning to grade HN amongst PU, PN, and PR and assess their confidence in their training. Secondly, we aim to assess the effectiveness of CEVL as an eLearning tool to improve the confidence and ability to SFU grade HN at multiple institutions and amongst PU, PN, and PR.
Intake survey After institutional review board determination of exemption, email invitations to participate in an anonymous, online survey were sent to physician extenders, residents, fellows, and attending physicians in the divisions of PU, PN, and PR at 6 institutions (listed in Table 1) as well as to members of the SFU, with a follow up email sent 2e4 weeks after the initial invitation. Participants were queried on how they learned the SFU grading system and asked to assess their confidence in their knowledge of the criteria to grade, indications to grade, and their ability to grade pediatric HN according to a 6-point Likert scale (where 0 Z not confident and 5 Z very confident). Respondents were then asked to assign SFU grades to 15 representative newborn US images that had been selected and graded by consensus of the authors. None of the participants had previous access to the CEVL module.
CEVL module on SFU grading of hydronephrosis After completion of the intake survey, participants were directed to an educational module developed using the CEVL platform on how to SFU grade HN as previously described (17). Briefly, an online tutorial was developed at our institution in collaboration with co-authors from two other institutions to present the key concepts and skills of SFU grading, including background knowledge, relevant anatomy, criteria to grade, and indications for SFU grading in accordance with the original description of the grading system [6,7]. Improvements to the module’s layout were made to enhance user interactivity. Specifically, sections providing self-assessment and immediate feedback including explanatory texts and graphic overlays were built (Fig. 1aec). The CEVL module was also made accessible via
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
CEVL e-learning Improve SFU Grading of Pediatric Hydronephrosis Table 1
1.e3
Study demographics.
Total Institution Lurie Children’s Hospital University of Oklahoma University of Virginia Loyola University Rush University Long Island Jewish/Cohen’s Society for Fetal Urology Department Urology Nephrology Radiology Professional role Resident Fellow Attending APN/PA Level of training1 PGY 1 PGY 2 PGY 3 PGY 4 PGY 5 PGY 6 How did you learn SFU grading Casually during training No one Textbooks or journal articles Formal lectures Web learning No answer Confidence in knowledge of: Criteria to grade Indications to grade Ability to grade Number of correctly graded US
Completed the study n (%)
Did not complete the study n (%)
24
30
1 (4.2) 11 (45.8) 2 (8.3) 2 (8.3) 1 (4.2) 2 (8.3) 5 (20.8)
8 (26.7) 8 (26.7) 0 (0.0) 0 (0.0) 3 (10.0) 1 (3.3) 10 (33.3)
21 (87.5) 2 (8.3) 1 (4.2)
23 (76.7) 4 (13.3) 3 (10.0)
13 (54.2) 5 (20.8) 5 (20.8) 1 (4.2)
9 7 9 5
p-value 0.035a,*
0.864a
0.264a (30.0) (23.3) (30.0) (16.7) 0.804a 3 2 2 4 2 0
(23.0) (15.4) (15.4) (30.8) (15.4) (0)
1 2 2 1 2 1
(11.1) (22.2) (22.2) (11.1) (22.2) (11.1)
15 (62.5) 3 (12.5) 3 (12.5) 2 (8.3) 1 (4.2) 0 (0.0)
9 7 5 2 3 4
(30.0) (23.3) (16.7) (6.7) (10.0) (13.3)
0.382a
3.5 3.0 3.0 9.0
(IQR (IQR (IQR (IQR
Z Z Z Z
3.0e4.0) 3.0e4.0) 2.25e4.0) 7.0e9.0)
3.5 3.0 3.0 9.0
(IQR (IQR (IQR (IQR
Z Z Z Z
2.0e4.0) 2.0e4.0) 2.0e4.0) 6.0e10.0)
0.566a 0.585a 0.446a 0.909b
* Statistical significance at p < 0.05. 1 Applicable only for residents where PGY Z Post-graduate year, APN Z Advanced Practice Nurse, PA Z Physician Assistant US Z Ultrasound, IQR Z Interquartile Range (25th to 75th Quartile). a Fisher exact test. b ManneWhitney U test.
smartphones and tablet computers. Experts in the field from multiple institutions reviewed and approved the completed module for content validity.
CEVL to their peers and whether they thought CEVL was superior to current learning methods.
Statistical analysis Exit survey After completion of the module, participants were asked to complete an exit survey that re-assessed their confidence in their knowledge of the criteria and indications for SFU grading, and their ability to grade HN. Participants were also asked to re-grade the same 15 US images. Using a 6point Likert scale (0-strongly disagree to 5-strongly agree), respondents were asked whether they would recommend
All data were collected and tabulated electronically and analyzed using IBM SPSS Statistics (version 22, Armonk, NY.). While demographic and methods of learning information were obtained from all intake surveys, only participants who completed both the intake and exit surveys were included in the pre and post CEVL analysis. Primary outcomes assessed were changes in confidence in knowledge of the criteria to grade, indications for grading, and the ability to grade HN
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
1.e4
D.B. Liu et al.
Figure 1 Screenshots from the Computer Enhanced Visual Learning (CEVL) module on the Society for Fetal Urology (SFU) grading system of pediatric hydronephrosis. Self-assessment and feedback increases interactivity. (a.) A gallery of available US for selfassessment. (b.) A sample ultrasound to grade for self-assessment. (c.) An example of the feedback available. Following assignment of an SFU grade, learners have the opportunity to receive immediate feedback. To better illustrate the anatomy, interactive buttons (arrow) allow color overlays (light blue) of the anatomical structures of the kidney (minor calyces) as seen on renal ultrasound. This interactivity clearly demonstrates the ultrasound correlation with renal anatomy and enhances the educational experience of the learner. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
and the accuracy of grading. Categorical data were compared by Fisher’s exact test or Chi-squared as appropriate. Nonparametric data were compared by the Wilcoxon Signed Rank test for paired samples and ManneWhitney U test for independent samples. A p-value < 0.05 was considered to be statistically significant.
Results A total of 54 intake surveys were completed. Study participants’ demographics are shown in Table 1. Participants were mostly from the field of PU (44, 81.5%) and a large number of responders were residents (22, 40.7%). The most common method of learning the SFU grading system was reported as “casually during training or medical school” (24, 44.5%), with the second most common reported as being taught by “no one” (10, 18.5%). Four (7.4%) respondents reported receiving formal lectures. The median participants’ confidence in knowledge of the criteria to grade, indication to grade, and ability to grade were 3.5 (IQR Z 2e4), 3.0 (IQR Z 2e4), and 3.0 (IQR Z 2e4) respectively. The median number of correctly graded ultrasounds was 9 (IQR Z 6e10) out of 15 (60%). A total of 13 residents (Urology), 5 fellows (4 PU, 1 PN), 5 attendings (3 PU, 1 PN, 1 PR), and 1 physician assistant (PU) (n Z 24, 44.4%) completed the exit surveys and were included in the comparative analysis. 30 participants did not complete the exit survey and were included only in the assessment of pre-CEVL confidence in knowledge and ability to grade, accuracy of grading, and how they learned to grade. The only significant difference between those who completed the study and those who did not was seen in the participants’ institutions (p Z 0.035) (Table 1).
CEVL learning increased median confidence in knowledge of the criteria to grade [3.5 (IQR Z 3.0e4.0) to 4 (IQR Z 3.25e5.0), p < 0.001], indication to grade [3.0 (IQR Z 3.0e4.0) to 4.0 (IQR Z 3.0e4.75), p Z 0.007], and ability to grade [3.0 (IQR Z 2.25e4.0) to 4.0 (IQR Z 3.0e4.0), p < 0.001]. Furthermore, the median number of correctly graded US increased from 9.0 (60%, IQR Z 7.0e9.0) to 10.0 (67%, IQR Z 9.0e12.0) (p Z 0.006) (Table 2). A total of 17 (70.8%) participants improved the number of correctly graded US after CEVL learning. Increased accuracy of grading was seen for grades 1e4 after CEVL learning (Table 3). Greatest improvement in grading after CEVL learning was seen in SFU grade 2 (15.48%, p Z 0.004). SFU grade 3 had the lowest accuracy of grading both before (44.4% correct) and after (51.39%) CEVL learning. The most common grade mistakenly assigned for SFU grade 3 was grade 4 (40.28%). The accuracy of identifying “clinically significant” HN (grades 3 and 4) increased from 83.3% to 93.3% after CEVL learning while “clinically insignificant” (grades 0,1, and 2) decreased from 92.9% to 85.4%. A total of 23 out of 24 (95.8%) respondents agreed or strongly agreed that they would recommend CEVL for learning the SFU grading system. Seventy-nine percent of respondents agreed or strongly agreed that CEVL learning was superior to other available methods of learning.
Discussion The skill to accurately SFU grade HN is critical for daily clinical care for PU, PN, and PR. Accurate grades allow for more objective communication amongst caregivers. This likely results in improved patient care. Although commonly used, little is known about how clinicians are learning to
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
CEVL e-learning Improve SFU Grading of Pediatric Hydronephrosis Table 2
1.e5
Results of CEVL learning on the accuracy and confidence in the SFU grading system. Pre-CEVL
Post-CEVL
p-value (95% CI)
All respondents (n Z 24) Confidence Criteria Indications Ability to grade Correctly Graded US Subgroups Residents (n Z 13) Confidence Criteria Indications Ability to grade Correctly graded US Fellows (n Z 5) Confidence Criteria Indications Ability to grade Correctly graded US Attendings (n Z 5) Confidence Criteria Indications Ability to grade Correctly graded US
Median
IQR
Median
IQR
3.50 3.00 3.00 9.0 (60%)
3.00e4.00 3.00e4.00 2.25e4.00 7.00e9.00
4.00 4.00 4.00 10.0 (67%)
3.25e5.00 3.00e4.75 3.00e4.00 9.0e12.0
3.00 3.00 3.00 8.00 (53%)
2.00e4.80 2.50e4.00 2.00e3.50 6.00e9.00
4.00 4.00 4.00 9.00 (60%)
3.00e4.00 3.00e4.00 3.00e4.00 8.50e12.00
5.00 3.00 3.00 9.00 (60%)
3.50e5.00 2.00e4.50 2.00e4.50 8.00e11.00
4.00 5.00 4.00 11.00 (73%)
3.25e4.50 3.50e5.00 3.50e4.50 9.50e12.50
4.00 4.00 4.00 9.00 (60%)
3.50e5.00 3.50e5.00 3.50e5.00 7.00e9.50
5.00 5.00 5.00 10.00 (67%)
4.00e5.00 4.00e5.00 4.00e5.00 7.00e11.50
<0.001* 0.007* <0001* 0.006*
*Wilcoxon Signed Rank test, statistical significance p < 0.05. IQR Z Interquartile range (25th to 75th Quartile). CEVL Z Computer Enhanced Visual Learning. SFU Z Society for Fetal Urology. US Z Ultrasound.
SFU grade HN and whether changes in delivery of education could improve their training. Our study demonstrates that most clinicians receive little to no formal training in SFU grading, leading to inaccuracies of HN grading.
Table 3 Grade
Furthermore, we demonstrate that structured, electronic learning (e-Learning) through CEVL increases both the clinicians’ confidence in the SFU grading system and improves their accuracy of grading.
Accuracy per SFU grade. Pre-CEVL
Most common incorrect grade
Post-CEVL
Most common incorrect grade
0 79.17% Equally 2 and 3 (8.3%) 66.67% 3 (25%) 1 50.00% 0 (43.75%) 64.58% 0 (22.92%) 2 55.95% 1 (32.74%) 71.43% 3 (14.88%) 3 44.44% 4 (36.11%) 51.39% 4 (40.28%) 4 66.67% 3 (20.83%) 81.25% 3 (14.58%) Clinically significant HN correctly graded as clinically significant 83.30% 93.30% Clinically insignificant HN correctly graded as clinically insignificant 92.9% 85.40%
Percent change
p-valuea
12.50% 14.58% 15.48% 6.95% 14.58%
0.517 0.216 0.004a,* 0.505 0.162 0.016b,* 0.005b,*
*Statistical significance p < 0.05. Abbreviations: SFU Z Society for Fetal Urology, CEVL Z Computer Enhanced Visual Learning, HN Z Hydronephrosis. Please note: Clinically significant HN includes SFU grades 3 and 4; Clinically Insignificant HN includes SFU grades 0, 1, and 2. a Fisher exact test. b Chi-squared test.
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
1.e6 With the proliferation of technology and the increasing constraints of time and resources in modern medical education, e-Learning has gained increased popularity [18]. It has been found to be effective in teaching a wide variety of clinical skills, such as ECG interpretation and pediatric resuscitation [19,20]. Advantages of e-Learning include accessibility, flexibility, interactivity and the ability for self-assessment, shifting the focus from teacher to learner [21]. e-Learning increases accessibility by freeing education from scheduling and geographic constraints [22]. It allows learners to individualize learning by focusing on problematic concepts, thereby increasing flexibility. Interactivity is increased by the incorporation of multimedia to enhance the delivery of educational materials and by providing the opportunity for selfassessment, leading to increased efficacy of learning [23,24]. The CEVL method was developed as a mechanism for eLearning. CEVL is a web-based program that provides a structured learning platform that is modifiable, interactive, and readily available. The principles of CEVL are an adaptation of the Ericsson method of deliberate practice and are rooted in the fundamentals of: 1. preparedness, 2. performance, and 3. feedback with remediation [25]. CEVL was originally designed to teach surgical skills to residents and has been previously demonstrated to improve teaching of pediatric orchiopexy [14e16]. Previously, Marks et al. demonstrated in a pilot study that CEVL learning was also adaptable to teach diagnostic skills such as how to SFU grade pediatric HN [17]. In this study, 21 urology residents and 8 medical students were asked to grade 16 pediatric US. The entire group then viewed the CEVL module during a conference and they were asked to re-grade the same 16 ultrasounds. An overall 21% mean improvement in accuracy of SFU HN grade was seen after CEVL learning. While demonstrating that CEVL learning is effective when used in a traditional lecture setting, Marks’s study did not assess CEVL’s e-Learning attributes of increased accessibility, interactivity, and ability for self-assessment. Our study more accurately assesses CEVL’s e-Learning attributes, demonstrating its effectiveness in improving the learning of SFU grading remotely at multiple institutions through self-study. By providing unlimited electronic access and expanding access to mobile devices, we sought to increase CEVL’s accessibility. Furthermore, by incorporating self-assessment with feedback, the interactivity of CEVL was improved (Fig. 1aec). Improved confidence and accuracy to grade seen in our study are testaments to the effectiveness of e-Learning through CEVL. One of the concerns raised about the SFU grading system has been its reliability. Keays et al. found that the inter-rater reliability of the SFU grading system was substantial for grade 0, moderate for grades 1, 2, and 4, and only slight to fair for SFU grade 3 [26]. The authors attributed this finding to a lack of precision of the SFU system, especially for higher grades. As such, the authors advocated for the need to refine the SFU grading system to better differentiate grades 3 and 4. This has led to a modification to the SFU grading system by Silbai et al., who subclassified SFU grade 4 into segmental (4a) and diffuse (4b) [27].
D.B. Liu et al. However, an alternative explanation may lie in the training clinicians received in SFU grading. In Keays’s study, raters were provided only written instructions on SFU grading without further training. Furthermore, the provided instructions categorized any degree of parenchymal loss as SFU grade 4. In contrast, the SFU grading system does in fact restrict grade 4 to parenchymal loss of greater than 50% [6]. In our study, we demonstrated that most clinicians receive little to no formal training in SFU grading and that given structured, web-based training, the accuracy of grading improved. Thus, it is conceivable that the modest inter-rater reliability of the SFU grading system seen in Keays’ study reflects not deficiencies in the system itself but rather deficiencies in knowledge. Future study of inter-rater reliability after structured learning, such as CEVL, may be warranted. Interestingly, our study demonstrated that accuracy of grading HN as “clinically significant” versus “insignificant” increased after CEVL learning while accurate grading of “clinically insignificant” HN decreased after CEVL eLearning. This finding may be due to increased awareness of the grading system and a possible bias of learners to over-grade HN when undecided in-between grades. Furthermore, grading of SFU grade 3 was most inaccurate and showed the least improvement with CEVL learning. While a lack of precision for SFU grades 3 and 4 may exist within the SFU grading system, it is also possible that our CEVL module did not emphasize enough the distinction between these two grades. Further refinements to the contents of the CEVL module may be beneficial to improve the accuracy. Recently, a new multidisciplinary classification system was introduced for prenatal and postnatal urinary tract dilation [28]. As acceptance of this new classification system becomes more widespread, the need for educational tools to help familiarize clinicians with this new system will be needed. We believe that our study shows the effectiveness of CEVL to deliver such educational content and that such a platform may be readily adapted to teach the new classification system of UTD. Our study does have its limitations. First, the sample size of our study was small as only 24 of the original 54 subjects completed the study. The small sample size, low participation rate, discrepant resident and institutional participation raise the possibility of selection bias, with individuals who felt less confident in their knowledge more likely to participate. Furthermore, the study population was skewed towards PU and residents. The small numbers of fellows, attending physicians, PN, and PR limit meaningful analyses on the effectiveness of CEVL learning for more experienced clinicians and for different pediatric specialists. While a number of factors may be responsible for the low study completion rate, including a lack of interest or technical difficulties, the comprehensive design of the module as an educational tool may have required a time-commitment greater than desired for many potential participants. However, among the 24 participants who completed the module, 23 (96%) agreed or strongly agreed that the module length was appropriate. Secondly, our study was not designed to monitor the number of times CEVL was accessed, the time spent on the module, and whether the CEVL module was
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
CEVL e-learning Improve SFU Grading of Pediatric Hydronephrosis completed. Therefore, we were unable to determine the efficiency of learning through CEVL and to ascertain that the improvements seen were exclusively derived from CEVL. Third, the lack of a control group learning through traditional methods, such as didactic lectures, prevented comparison of CEVL learning to other forms of education. However, the goal of this study was to demonstrate that CEVL does improve learning, rather than examining if it was better than other methods of learning. Lastly, we did not assess the impact of CEVL learning on long-term knowledge retention.
Conclusion In this study, we demonstrate that current training to SFU grading of HN is mostly unstructured and inaccurate grading is common. The CEVL platform provides a structured eLearning approach to learning the SFU grading system. Learners who use CEVL show improvements in their confidence in their knowledge of and ability to SFU grade HN.
Ethical approval This study was designated exempt by the Institutional Review Board at our institution.
Funding None.
Conflict of interest Max Maizels, MD is the Co-founder and co-Executive Director of CEVL for Healthcare, Inc.
Acknowledgement We would like to thank Karen Rychlik, MS, for providing statistical assistance with this project.
References [1] Ismaili K, Hall M, Donner C, Thomas D, Vermeylen D, Avni FE. Results of systematic screening for minor degrees of renal pelvis dilatation in an unselected population. Am J Obstet Gynecol 2003;188:242e6. [2] Coplen DE, Austin PF, Yan Y, Blanco VM, Dicke JM. The magnitude of fetal renal pelvic dilatation can identify obstructive postnatal hydronephrosis, and direct postnatal evaluation and management. J Urol 2006;176:724e7. [3] Morin L, Cendron M, Crombleholme TM, Garmel SH, Klauber GT, D’Alton ME. Minimal hydronephrosis in the fetus: clinical significance and implications for management. J Urol 1996;155:2047e9. [4] Hothi DK, Wade AS, Gilbert R, Winyard PJD. Mild fetal renal pelvis dilatation-much ado about nothing? Clin J Am Soc Nephrol 2009;4:168e77.
1.e7
[5] Lee RS, Cendron M, Kinnamon DD, Nguyen HT. Antenatal hydronephrosis as a predictor of postnatal outcome: a metaanalysis. Pediatrics 2006;118:586e96. [6] Maizels M, Reisman ME, Flom LS, Nelson J, Fernbach S, Firlit CF, et al. Grading nephroureteral dilatation detected in the first year of life: correlation with obstruction. J Urol 1992; 148:609e14. [7] Fernbach SK, Maizels M, Conway JJ. Ultrasound grading of hydronephrosis: introduction to the system used by the Society for Fetal Urology. Pediatr Radiol 1993;23:478e80. [8] Nguyen HT, Herndon CD, Cooper C, Gatti H, Kirsch A, Kokorowski P, et al. The Society for Fetal urology consensus statement on the evaluation and management of antenatal hydronephrosis. J Pediatr Urol 2010;6:212e31. [9] Sidhu G, Beyene J, Rosenblum ND. Outcome of isolated antenatal hydronephrosis: a systemic review and meta-analysis. Pediatr Nephrol 2006;21:218e24. [10] Chi T, Feldstein VA, Nguyen HT. Increased echogenicity as a predictor of poor renal function in children with grade 3 to 4 hydronephrosis. J Urol 2006;175:1898e901. [11] Hafez AT, McLorie G, Bagli D, Khoury A. Analysis of trends on serial ultrasounds for high grade neonatal hydronephrosis. J Urol 2002;168:1518e21. [12] Erickson BA, Maizels M, Shore RM, Pazona JF, Hagerty JA, Yerkes EB, et al. Newborn society of fetal urology grade 3 hydronephrosis is equivalent to preserved percentage differential function. J Pediatr Urol 2007;3:382e6. [13] Onen A, Jayanthi VR, Koff SA. Long-term followup of prenatallydetected severe bilateral newborn hydronephrosis initially managed nonoperatively. J Urol 2002;168:1118e20. [14] Maizels M, Yerkes E, Macejko A, Hagerty J, Chaviano AC, Cheng EY, et al. A new computer enhanced visual learning method to train urology residents in pediatric orchiopexy: a prototype for accreditation council for graduate medical education. J Urol 2008;180:1814e8. [15] McQuiston L, MacNeily A, Liu D, Mickelson J, Yerkes E, Chaviano A, et al. Computer enhanced visual learning method to train urology residents in pediatric orchiopexy provided a consistent learning experience in a multi-institutional trial. J Urol 2010;184:1748e53. [16] Matoka D, Marks A, Stoltz R, Maizels M. Utilization of computer enhanced visual learning (CEVL) method improves endoscopic diagnosis of posterior urethral valves. J Ped Urol 2013;9: 498e502. [17] Marks A, Maizels M, Mickelson J, Yerkes E, Herndon CDA, Lane J, et al. Effectiveness of the computer enhanced learning method in teaching the society for fetal urology hydronephrosis grading system for urology trainees. J Ped Urol 2011;7:113e7. [18] Evgeniou E, Loizou P. The theoretical base of e-learning and its role in surgical education. J Surg Educ 2012:665e9. [19] Montassier E, Hardouin J, Segard J, Batard E, Potel G, Planchon B, et al. e-Learning versus lecture-based courses in ECG interpretation for undergraduate medical students: a randomized noninferiority study. Eur J Emerg Med 2014:00. [20] O’Leary F. Paediatric resuscitation training: is e-Learning the answer? A before and after pilot study. J Paediatr Child Health 2012;48:529e33. [21] Ellaway R, Masters K. AMEE guide 32: e-Learning in medical education. Part 1: learning, teaching, and assessment. Med Teach 2008;30:455e73. [22] Segal G, Balik C, Hovav B, Mayer A, Rozani V, Damary I, et al. Online nephrology course replacing a face to face course in nursing schools’ bachelor’s program: a prospective, controlled trial in four Israeli nursing schools. Nurse Educ Today 2013;33: 1587e91. [23] Issa N, Schuller M, Santacaterina S, Shapiro M, Wang E, Mayer RE, et al. Applying multimedia design principles
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008
+
MODEL
1.e8 enhances learning in medical education. Med Educ 2011;45: 818e26. [24] Cook DA, Thompson WG, Thomas KG, Thomas MR, Pankratz VS. Impact of self-assessment questions and learning styles in web-based learning: a randomized, controlled, crossover trial. Acad Med 2006;81:231e8. [25] Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 2004;79(10 suppl):S70. [26] Keays MA, Guerra JM, Mihill J, Raju G, Al-Asheeri N, Geier P, et al. Reliability assessment of Society for Fetal Urology
D.B. Liu et al. ultrasound grading system for hydronephrosis. J Urol 2008; 180:1680e3. [27] Sibai H, Pippi Salle JL, Houle AM, Lambert R. Hydronephrosis with diffuse and segmental cortical thinning: impact on renal function. J Urol 2001;165:2293e5. [28] Nguyen HT, Benson CB, Bromley B, Campbell JB, Chow J, Coleman B, et al. Multidisciplinary consensus on the classification of prenatal and postnatal urinary tract dilation (UTD classification system). J Ped Urol 2014;10(6):982e98.
Please cite this article in press as: Liu DB, et al., Teaching of the Society for Fetal Urology grading system for pediatric hydronephrosis is improved by e-Learning using Computer Enhanced Visual Learning (CEVL): A multi-institutional trial, Journal of Pediatric Urology (2015), http://dx.doi.org/10.1016/j.jpurol.2015.05.008