- Email: [email protected]
Feasibility of a novel digital tool in automatic scoring of an online ECG examination
International Journal of Cardiology 185 (2015) 88–89
Contents lists available at ScienceDirect
International Journal of Cardiology journal homepage: www.elsevier.com/locate/ijcard
Letter to the Editor
Feasibility of a novel digital tool in automatic scoring of an online ECG examination Kieran L. Quinn a,b, Adrian Baranchuk b,⁎ a b
Keenan Research Centre for Biomedical Science and Department of Medicine, St. Michael's Hospital, University of Toronto, Toronto, ON, Canada Arrhythmia Service, Kingston General Hospital, Queen's University, Kingston, ON, Canada
a r t i c l e
i n f o
Article history: Received 1 February 2015 Accepted 7 March 2015 Available online 10 March 2015 Keywords: ECG interpretation Medical education Technology
Electrocardiography (ECG) is of critical importance in diagnosing heart disease, yet mounting evidence demonstrates that there is a concerning lack of competency in ECG diagnosis among medical trainees [1–3]. Memorizing the patterns of ECGs and associating them with disease processes is prone to error [4], and a patient's clinical history significantly improves a physician's diagnostic accuracy when reading ECGs [5]. There is a lack of research addressing the most accurate diagnostic approach to ECG interpretation, which precludes the ability to make educational recommendations for improvement in this key area of diagnostic test interpretation. A major challenge in assessing competency in ECG interpretation is the labor-intensive effort involved in administering these tests through manual review of each accompanying diagnosis provided by trainees. Consequently, studies in this area of education research have been limited to one or two-center cohorts focusing on an individual specialty [6]. Herein we describe the development of a digital tool capable of automatically recognizing free-text responses for the purposes of marking online examinations; and characterize its performance when applied to an exam comparing a pattern recognition strategy to an inductive–deductive reasoning strategy in ECG interpretation. The study was conducted over the academic year from July 1st, 2013 to June 30th, 2014 in medical residents training in Internal Medicine, Emergency Medicine and Family Medicine from 5 accredited urban residency programs across Canada. Study participants were sent an invitation to participate ⁎ Corresponding author at: Arrhythmia Service, Kingston General Hospital, Queen's University, 76 Stuart Street, Kingston, ON, Canada. E-mail address: [email protected] (A. Baranchuk).
http://dx.doi.org/10.1016/j.ijcard.2015.03.135 0167-5273/© 2015 Elsevier Ireland Ltd. All rights reserved.
in the study containing a unique secure link through their institutional email address that was provided by their respective program directors. The ECG Study web application was developed by the Medical Education Technology Unit at Queen's University and consisted of an online exam performed in a stepwise approach. Step 1 consisted of 10 ECG cases: Anterior STEMI, Atrial Fibrillation with Rapid Ventricular Response, Complete Heart Block, Monomorphic VT, pseudo VT (artifact), Wolff– Parkinson–White Syndrome, Left Bundle Branch Block, Hyperkalemia, Long QT Syndrome, and Type-1 Brugada pattern presented in random order without any additional information provided. Participants had 90 s to provide a single critical diagnosis for each ECG. This tested the accuracy of a pattern recognition strategy. Once completed, participants entered step 2 of the exam, consisting of the same 10 ECGs, presented in random order and accompanied by a 1–2 sentence clinical vignette. This tested the impact of the clinical history on the accuracy of ECG diagnoses. Participants were not permitted to change previously entered responses (Table 2). The application securely saved participants' responses to an online repository and allowed study administrators to add and remove participants, as well as score and analyze the response data saved to the database. This application was developed using the PHP server-side programming language. The web interface itself utilizes Twitter Bootstrap, and the jQuery JavaScript framework is used as a base for the code that automatically submits and advances study participants at 90 second intervals. The study questions, participants, and responses are stored in a normalized manner throughout a number of tables in a dedicated MySQL database. MySQL is a widely adopted relational database management system that is based on the Structured Query Language ANSI/ISO standard. When a participant accessed the study online using their unique one-time use 32 character hash code, they were asked to provide basic demographic information (i.e. age, gender, country, program, year of study, and institution). Stored responses provided by study participants can then be exported in CSV format for processing and analysis in Microsoft Excel. The scoring mechanism is unique in that it provides study administrators with the ability to automatically score text-based responses. It accomplishes this by using an acceptable variations list that can be dynamically managed as study administrators review “incorrect” responses. When an acceptable variation is encountered as an “incorrect” response
K.L. Quinn, A. Baranchuk / International Journal of Cardiology 185 (2015) 88–89
89
Table 1 Performance of the online tool in automatic marking. Total responses
Correct responses identified n, (%)
Responses requiring manual review n, (%)
Correct responses identified from manual review n, (%)
Incorrect responses identified n, (%)
2500
1150 (46)
1350 (54)
400 (16)
950 (38)
Table 2 Baseline characteristics of study participants and accuracy of ECG diagnosis using pattern recognition versus clinical reasoning strategies. Gendera
Accuracy of diagnosis (% ± SEM) (Pattern)
Accuracy of diagnosis (% ± SEM) (Clinical)
60 (48.0)
53.0 ± 2.0
68.8 ± 1.9
36 (52.9)
31 (45.6)
61.9 ± 4.8
75.9 ± 2.4
29.9 ± 1.1
7 (53.8)
6 (46.2)
57.0 ± 5.0
70.0 ± 3.4
28.5 ± 0.4
20 (45.5)
23 (52.3)
40.0 ± 4.6
56.7 ± 3.1
Program of study (n, %)
Year of study (Mean ± SEM)
Age (yrs ± SEM)
All residents (125) Internal medicine (68, 54.8) FRPC-EM (13, 10.5) CFPC (44, 33.9)
1.8 ± 0.1
Male n (%)
Female n (%)
29.0 ± 0.4
63 (50.4)
1.9 ± 0.1
28.4 ± 0.4
2.3 ± 0.4 1.4 ± 0.1
FRPC — Fellow of the Royal College of Physicians; EM — Emergency Medicine; CFPC — The College of Family Physicians of Canada. a 2 participants did not disclose gender.
(e.g. hypokalaemia vs. hypokalemia), study administrators can correct the variation or add the variation to the acceptable variations list so future participant responses will automatically be marked as correct. The tool becomes more powerful and efficient in reducing the need for manual review and correction of acceptable variations as administrators build the acceptable variations response list. There were a total of 2500 responses provided by the 125 residents who participated in the study (1250 for part 1 and part 2 each). 1550 responses were correct in the diagnosis and of these, the online tool automatically identified 1150. The remaining 1350 responses required secondary manual review. 400 of these 1350 responses were accurate in the diagnosis, but required correction of typographical and other errors. There were 950 incorrect responses remaining that were appropriately identified as incorrect. The average time to review a response took 10 s. If the same exam were administered to 125 participants without the use of our computer-marking tool, the total marking time would equate to 7 h. By utilizing the automatic scoring tool, our total marking time was reduced by 46%. The overall accuracy of correct ECG diagnosis for all residents was 53% and 68% when using pattern recognition or inductive–deductive strategies, respectively (p b 0.0001) (Table 1). To the best of our knowledge, our study is the first to employ the use of an automatic online marking tool to carry out a multicenter evaluation of ECG competency in residents across different training programs. It is a widely accessible, simple, flexible and low-cost method that can be implemented into curricular programs to provide rapid assessment of trainees' accuracy in ECG diagnosis. We believe the greatest strength our online tool comes from its ability to automatically recognize free-text responses, which makes its
utility applicable to many forms of exam administration at educational institutions. We intend to further evaluate the online tool's ability to mark short answer examinations in future studies. One limitation of the tool was its inability to identify responses that were provided as a pathophysiologic interpretation or description of the ECG abnormality. However, as the common combinations of these are identified through analysis of the types of responses given, the tools performance will improve to capture these and automatically mark them as correct. The authors wish to thank Matt Simpson and Ryan Warner for their valued input. Competing interests None declared. References [1] G. Fent, J. Gosai, M. Purva, Teaching the interpretation of electrocardiograms: which method is best? J. Electrocardiol. 48 (2015) 190–193. [2] M.G. Sibbald, E.G. Davies, P. Dorian, H.C. Eric, Electrocardiographic interpretation skills of cardiology residents: are they competent? Can. J. Cardiol. 30 (2014) 1721–1724. [3] N.A. Lever, P.D. Larsen, M. Dawes, A. Wong, S.A. Harding, Are our medical graduates in New Zealand safe and accurate in ECG interpretation? N. Z. Med. J. 122 (2009) 9–15. [4] J.W. Hurst, Methods used to interpret the 12-lead electrocardiogram: pattern memorization versus the use of vector concepts, Clin. Cardiol. 23 (2000) 4–13. [5] R. Hatala, G.R. Norman, L.R. Brooks, Impact of a clinical scenario on accuracy of electrocardiogram interpretation, J. Gen. Intern. Med. 14 (1999) 126–129. [6] A. Baranchuk, G. Dagnone, P. Fowler, N.E. Harrison MN, L. Lisnevskaia, et al., Education at distance: broadcasting ECG rounds to Southeastern Ontario (BESO Project). An innovative approach for teaching electrocardiography, Clin. Invest. Med. 30 (2007) S: 51–S:52.
Contents lists available at ScienceDirect
International Journal of Cardiology journal homepage: www.elsevier.com/locate/ijcard
Letter to the Editor
Feasibility of a novel digital tool in automatic scoring of an online ECG examination Kieran L. Quinn a,b, Adrian Baranchuk b,⁎ a b
Keenan Research Centre for Biomedical Science and Department of Medicine, St. Michael's Hospital, University of Toronto, Toronto, ON, Canada Arrhythmia Service, Kingston General Hospital, Queen's University, Kingston, ON, Canada
a r t i c l e
i n f o
Article history: Received 1 February 2015 Accepted 7 March 2015 Available online 10 March 2015 Keywords: ECG interpretation Medical education Technology
Electrocardiography (ECG) is of critical importance in diagnosing heart disease, yet mounting evidence demonstrates that there is a concerning lack of competency in ECG diagnosis among medical trainees [1–3]. Memorizing the patterns of ECGs and associating them with disease processes is prone to error [4], and a patient's clinical history significantly improves a physician's diagnostic accuracy when reading ECGs [5]. There is a lack of research addressing the most accurate diagnostic approach to ECG interpretation, which precludes the ability to make educational recommendations for improvement in this key area of diagnostic test interpretation. A major challenge in assessing competency in ECG interpretation is the labor-intensive effort involved in administering these tests through manual review of each accompanying diagnosis provided by trainees. Consequently, studies in this area of education research have been limited to one or two-center cohorts focusing on an individual specialty [6]. Herein we describe the development of a digital tool capable of automatically recognizing free-text responses for the purposes of marking online examinations; and characterize its performance when applied to an exam comparing a pattern recognition strategy to an inductive–deductive reasoning strategy in ECG interpretation. The study was conducted over the academic year from July 1st, 2013 to June 30th, 2014 in medical residents training in Internal Medicine, Emergency Medicine and Family Medicine from 5 accredited urban residency programs across Canada. Study participants were sent an invitation to participate ⁎ Corresponding author at: Arrhythmia Service, Kingston General Hospital, Queen's University, 76 Stuart Street, Kingston, ON, Canada. E-mail address: [email protected] (A. Baranchuk).
http://dx.doi.org/10.1016/j.ijcard.2015.03.135 0167-5273/© 2015 Elsevier Ireland Ltd. All rights reserved.
in the study containing a unique secure link through their institutional email address that was provided by their respective program directors. The ECG Study web application was developed by the Medical Education Technology Unit at Queen's University and consisted of an online exam performed in a stepwise approach. Step 1 consisted of 10 ECG cases: Anterior STEMI, Atrial Fibrillation with Rapid Ventricular Response, Complete Heart Block, Monomorphic VT, pseudo VT (artifact), Wolff– Parkinson–White Syndrome, Left Bundle Branch Block, Hyperkalemia, Long QT Syndrome, and Type-1 Brugada pattern presented in random order without any additional information provided. Participants had 90 s to provide a single critical diagnosis for each ECG. This tested the accuracy of a pattern recognition strategy. Once completed, participants entered step 2 of the exam, consisting of the same 10 ECGs, presented in random order and accompanied by a 1–2 sentence clinical vignette. This tested the impact of the clinical history on the accuracy of ECG diagnoses. Participants were not permitted to change previously entered responses (Table 2). The application securely saved participants' responses to an online repository and allowed study administrators to add and remove participants, as well as score and analyze the response data saved to the database. This application was developed using the PHP server-side programming language. The web interface itself utilizes Twitter Bootstrap, and the jQuery JavaScript framework is used as a base for the code that automatically submits and advances study participants at 90 second intervals. The study questions, participants, and responses are stored in a normalized manner throughout a number of tables in a dedicated MySQL database. MySQL is a widely adopted relational database management system that is based on the Structured Query Language ANSI/ISO standard. When a participant accessed the study online using their unique one-time use 32 character hash code, they were asked to provide basic demographic information (i.e. age, gender, country, program, year of study, and institution). Stored responses provided by study participants can then be exported in CSV format for processing and analysis in Microsoft Excel. The scoring mechanism is unique in that it provides study administrators with the ability to automatically score text-based responses. It accomplishes this by using an acceptable variations list that can be dynamically managed as study administrators review “incorrect” responses. When an acceptable variation is encountered as an “incorrect” response
K.L. Quinn, A. Baranchuk / International Journal of Cardiology 185 (2015) 88–89
89
Table 1 Performance of the online tool in automatic marking. Total responses
Correct responses identified n, (%)
Responses requiring manual review n, (%)
Correct responses identified from manual review n, (%)
Incorrect responses identified n, (%)
2500
1150 (46)
1350 (54)
400 (16)
950 (38)
Table 2 Baseline characteristics of study participants and accuracy of ECG diagnosis using pattern recognition versus clinical reasoning strategies. Gendera
Accuracy of diagnosis (% ± SEM) (Pattern)
Accuracy of diagnosis (% ± SEM) (Clinical)
60 (48.0)
53.0 ± 2.0
68.8 ± 1.9
36 (52.9)
31 (45.6)
61.9 ± 4.8
75.9 ± 2.4
29.9 ± 1.1
7 (53.8)
6 (46.2)
57.0 ± 5.0
70.0 ± 3.4
28.5 ± 0.4
20 (45.5)
23 (52.3)
40.0 ± 4.6
56.7 ± 3.1
Program of study (n, %)
Year of study (Mean ± SEM)
Age (yrs ± SEM)
All residents (125) Internal medicine (68, 54.8) FRPC-EM (13, 10.5) CFPC (44, 33.9)
1.8 ± 0.1
Male n (%)
Female n (%)
29.0 ± 0.4
63 (50.4)
1.9 ± 0.1
28.4 ± 0.4
2.3 ± 0.4 1.4 ± 0.1
FRPC — Fellow of the Royal College of Physicians; EM — Emergency Medicine; CFPC — The College of Family Physicians of Canada. a 2 participants did not disclose gender.
(e.g. hypokalaemia vs. hypokalemia), study administrators can correct the variation or add the variation to the acceptable variations list so future participant responses will automatically be marked as correct. The tool becomes more powerful and efficient in reducing the need for manual review and correction of acceptable variations as administrators build the acceptable variations response list. There were a total of 2500 responses provided by the 125 residents who participated in the study (1250 for part 1 and part 2 each). 1550 responses were correct in the diagnosis and of these, the online tool automatically identified 1150. The remaining 1350 responses required secondary manual review. 400 of these 1350 responses were accurate in the diagnosis, but required correction of typographical and other errors. There were 950 incorrect responses remaining that were appropriately identified as incorrect. The average time to review a response took 10 s. If the same exam were administered to 125 participants without the use of our computer-marking tool, the total marking time would equate to 7 h. By utilizing the automatic scoring tool, our total marking time was reduced by 46%. The overall accuracy of correct ECG diagnosis for all residents was 53% and 68% when using pattern recognition or inductive–deductive strategies, respectively (p b 0.0001) (Table 1). To the best of our knowledge, our study is the first to employ the use of an automatic online marking tool to carry out a multicenter evaluation of ECG competency in residents across different training programs. It is a widely accessible, simple, flexible and low-cost method that can be implemented into curricular programs to provide rapid assessment of trainees' accuracy in ECG diagnosis. We believe the greatest strength our online tool comes from its ability to automatically recognize free-text responses, which makes its
utility applicable to many forms of exam administration at educational institutions. We intend to further evaluate the online tool's ability to mark short answer examinations in future studies. One limitation of the tool was its inability to identify responses that were provided as a pathophysiologic interpretation or description of the ECG abnormality. However, as the common combinations of these are identified through analysis of the types of responses given, the tools performance will improve to capture these and automatically mark them as correct. The authors wish to thank Matt Simpson and Ryan Warner for their valued input. Competing interests None declared. References [1] G. Fent, J. Gosai, M. Purva, Teaching the interpretation of electrocardiograms: which method is best? J. Electrocardiol. 48 (2015) 190–193. [2] M.G. Sibbald, E.G. Davies, P. Dorian, H.C. Eric, Electrocardiographic interpretation skills of cardiology residents: are they competent? Can. J. Cardiol. 30 (2014) 1721–1724. [3] N.A. Lever, P.D. Larsen, M. Dawes, A. Wong, S.A. Harding, Are our medical graduates in New Zealand safe and accurate in ECG interpretation? N. Z. Med. J. 122 (2009) 9–15. [4] J.W. Hurst, Methods used to interpret the 12-lead electrocardiogram: pattern memorization versus the use of vector concepts, Clin. Cardiol. 23 (2000) 4–13. [5] R. Hatala, G.R. Norman, L.R. Brooks, Impact of a clinical scenario on accuracy of electrocardiogram interpretation, J. Gen. Intern. Med. 14 (1999) 126–129. [6] A. Baranchuk, G. Dagnone, P. Fowler, N.E. Harrison MN, L. Lisnevskaia, et al., Education at distance: broadcasting ECG rounds to Southeastern Ontario (BESO Project). An innovative approach for teaching electrocardiography, Clin. Invest. Med. 30 (2007) S: 51–S:52.