- Email: [email protected]
Recall after cardiac arrest scenario testing
Resuscitation (2006) 68, 239—242
TRAINING AND EDUCATION
Recall after cardiac arrest scenario testing夽 Robin P. Davies a, Gavin D. Perkins b,∗ a
Birmingham Heartlands Hospital, Birmingham B9 5SS, UK Department of Respiratory and Critical Care Medicine, Division of Medical Sciences, University of Birmingham, Birmingham B15 2TT, UK
b
Received 9 May 2005; accepted 15 July 2005 KEYWORDS Advanced Life Support (ALS); Education; Training
Summary The Resuscitation Council (UK) Advanced Life Support (ALS) Course is a multidisciplinary training course which teaches participants how to manage the resuscitation of a patient at risk of or in cardiac arrest. To reduce variability in assessments, four standardised patient scenarios have been developed with common performance criteria. The aim of the study was to establish how much candidates remembered about their test in order to assess the potential for collusion. Eighty-nine candidates were asked immediately after testing what they remembered about their ALS scenario. Recall of the underlying problem with the simulated patient was good (85 [96%]). Forty-two [47%] correctly remembered the initial cardiac arrest rhythm and 55 [61%] the subsequent cardiac arrest rhythm. Fifty-nine [60%] candidates passed the assessment. Candidates who passed the assessment were significantly more likely to correctly recall the initial and subsequent cardiac arrest rhythms than those who did not. However, even in this group, the overall recall of all elements of the scenario was correct in only 49% of instances. This study demonstrated that immediately after testing candidates had good recall of the initial clinical scenario with which they were presented, but poor recall of cardiac arrest rhythms during the simulated resuscitation attempt. These findings provide some reassurance that the likelihood of successful collusion improving subsequent candidates performance is likely to be small as recall of the scenario progression is limited. © 2005 Elsevier Ireland Ltd. All rights reserved.
Introduction The Resuscitation Council (UK) Advanced Life Support (ALS) Course is a multidisciplinary training 夽 A Spanish translated version of the summary of this arti-
cle appears as Appendix in the online version at 10.1016/ j.resuscitation.2005.07.006. ∗ Corresponding author. Fax: +44 121 443 2494. E-mail address: [email protected] (G.D. Perkins).
course which aims to teach participants how to manage the resuscitation of a patient at risk of or in cardiac arrest.1,2 The course has been adopted by the European Resuscitation Council as the ERC ALS course. The course uses role play during simulated clinical scenarios to develop patient assessment, resuscitation and cardiac arrest team leadership skills. These skills are tested using a similar approach. Each candidate is assessed leading a resuscitation team (comprised of fac-
0300-9572/$ — see front matter © 2005 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.resuscitation.2005.07.006
240
R.P. Davies, G.D. Perkins
ulty members) in the systematic assessment of a critically ill patient and subsequently resuscitation from cardiopulmonary arrest. To reduce variability in assessments,3 four standardised patient scenarios have been developed with common performance criteria.4 Course Directors are tasked with selecting one scenario for testing and one for re-testing on each course. Whilst this reduces variability in the test delivered to candidates on a particular course, a potential limitation of this approach is loss of security of the examination. The aim of the study was to establish how much candidates remembered about the clinical scenario, initial and subsequent cardiac arrest rhythms encountered during ALS testing.
was calculated. Chi-squared or Fishers exact test were used to see if candidates that passed the assessment had better recall than those that did not. A P-value of 0.05 was considered statistically significant.
Results Eighty-nine candidates were recruited into the study. Overall candidate recall of the underlying problem with the simulated patient was good (85 [96%] correctly remembered the patient condition). However, recall of the initial cardiac arrest rhythm (42 [47%] correct) and subsequent cardiac arrest rhythm (55 [61%] correct) was poor. Few candidates correctly recalled all three elements of the scenario accurately (33 [37%] correct). The results for each different scenario are presented in Table 1. Fifty-nine [60%] candidates passed the assessment. Candidates who passed the assessment were significantly more likely to correctly recall the initial and subsequent cardiac arrest rhythms than those that did not Table 2. Even candidates who passed remembered the scenario progression incorrectly in 26—43% of instances.
Materials and methods Candidates at four different ALS centres in the UK completed a short questionnaire immediately following testing. They were asked: (1) what was underlying problem with their patient, (2) what was the initial cardiac arrest rhythm and (3) what was the subsequent cardiac arrest rhythm. Answers were provided using free text. Responses were then compared to the clinical scenario with which the candidate was tested. Whether the candidate passed or failed the assessment was also recorded. Data were collated using Microsoft Excel. The proportion of candidates who correctly recalled the clinical scenario, initial and subsequent rhythms
Table 1
Scenario Scenario Scenario Scenario
Discussion This study shows that immediately after testing candidates have good recall of the initial clinical scenario with which they are presented, but poor
Candidate recall of initial and subsequent scenario contents
1 2 3 4
N
Diagnosis Correct
Incorrect
Correct
23 22 30 14
23 19 30 13
0 3 0 1
9 (39%) 10 (45%) 19 (63%) 4 (28%)
(100%) (86%) (100%) (92%)
Initial rhythm
(0%) (14%) (0%) (8%)
Subsequent rhythm
All responses
Incorrect
Correct
Incorrect
Correct
Incorrect
14 12 11 10
8 (35%) 15 (68%) 23 (76%) 9 (64%)
15 (65%) 7 (32%) 7 (24%) 5 (36%)
3 (13%) 9 (41%) 17 (56%) 4 (28%)
20 13 13 10
(61%) (55%) (37%) (72%)
(87%) (59%) (44%) (72%)
Table 2 Candidate recall of initial and subsequent scenario contents according to whether candidates passed or failed the assessment Diagnosis Correct Pass Fail P-value
Incorrect
52 (98%) 1 (2%) 33 (92%) 3 (8%) 0.299
Initial rhythm
Subsequent rhythm
All responses
Correct
Correct
Correct
Incorrect
30 (57%) 23 (43%) 12 (33%) 24 (67%) 0.031
39 (74%) 16 (44%)
Incorrect 14 (26%) 20 (56%) 0.005
Incorrect
26 (49%) 27 (51%) 7 (19%) 29 (81%) 0.005
Recall after cardiac arrest scenario testing recall of cardiac arrest rhythms during the simulated resuscitation attempt. Candidates who passed the assessment were significantly more likely to remember the sequence of cardiac arrest rhythms. However, even in this group, the overall recall of all elements of the scenario was only correct in 49% of instances. The ALS course is a high stakes examination. This makes the reliability of the assessments important. We have demonstrated previously a significant variability between assessors, not only in judging performance, but also in the difficulty of the test itself.3 By creating a set of standardised testing scenarios with common performance criteria between them, one source of variability between assessments is reduced. A potential limitation of this approach is that the security of the exam and a fair test of performance may be lost if candidates collude by telling each other the contents of the test. Collusion or cheating in examinations is a concern in medical education, particularly with the growth of electronic media that make communication and flow of information easier.5 Remote electronic re-certification is already available as part of the American Heart Association Advanced Cardiac Life Support Programme.6 Whilst this offers advantages in allowing learning and re-certification to occur at the learners pace and convenience, direct supervision of participants during this process is impossible. Un-authorised duplication and sale of test papers (with or without answers) is another problem. Websites selling examination papers or model essays are a growing problem. ACLS Preparation (http://www.aclspreparation.com) offers to sell multiple choice papers for the AHA ACLS 2005 certification course. Many systems rely on the integrity and professionalism of the individuals involved. However, an awareness that cheating may occur is sensible and steps to assess the potential opportunities for cheating within an examination system should be encouraged. The findings of the present study provide some reassurance that the likelihood of successful collusion between candidates improving subsequent candidates performance is likely to be small as recall of the scenario progression is limited. The influence of two or more candidates discussing scenario content and the effect that this has on recall was not evaluated. It is possible that discussing the scenarios in groups may enhance the potential for benefit from collusion. Summative assessments are not primarily designed to provide feedback on performance, however, the opportunity to enhance learning should be seized at every opportunity.7 Although
241 the present study did not address what candidates have learned from their test, the fact that few could remember the details of their scenario suggests that learning may be similarly limited. An alternative approach which may overcome the issue of opportunities for collusion and possibly enhance learning opportunities might be to issue candidates with a template of the assessment defining clearly what is going to be tested. If the assessment truly matches the learning outcomes of the course then having prior knowledge of assessment should help candidates to achieve the learning outcomes.
Conclusion This study demonstrated that immediately after testing candidates on the ALS course had good recall of the initial clinical scenario with which they were presented, but poor recall of cardiac arrest rhythms during the simulated resuscitation attempt. These findings provide some reassurance that the likelihood of successful collusion between candidates improving subsequent candidates performance is likely to be small, as recall of the scenario progression is limited.
Conflict of interest GDP is a member of the Resuscitation Council (UK) Advanced Life Support Working group and was involved in the development of the present ALS cardiac arrest scenario tests.
Acknowledgements Advanced Life Support training centres at Alexandra Hospital, Redditch; Blackpool Victoria Hospital, Blackpool; New Cross Hospital, Wolverhampton; Grantham Hospital, Lincoln. This study was supported by the Resuscitation Council (UK).
References 1. Perkins G, Lockey A. The advanced life support provider course. BMJ 2002;325:S81. 2. Nolan JP. Advanced life support training. Resuscitation 2001;50:9—11. 3. Perkins GD, Hulme J, Tweed MJ. Variability in the assessment of advanced life support skills. Resuscitation 2001;50: 281—6.
242 4. Perkins GD. Cardiac arrest scenario testing. In: Bullock I, Colquhoun M, Coleman A, editors. Advanced life support instructor manual. London: Resuscitation Council (UK); 2004. p. 198—208. 5. Chaput de Saintonge DM, Pavlovic A. Cheating. Med Educ 2004;38:8—9.
R.P. Davies, G.D. Perkins 6. American Heart Association. HeartCode ACLS Anywhere. http://www.americanheart.org/presenter.jhtml?identifier= 3026627. 7. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001;357:945—9.
TRAINING AND EDUCATION
Recall after cardiac arrest scenario testing夽 Robin P. Davies a, Gavin D. Perkins b,∗ a
Birmingham Heartlands Hospital, Birmingham B9 5SS, UK Department of Respiratory and Critical Care Medicine, Division of Medical Sciences, University of Birmingham, Birmingham B15 2TT, UK
b
Received 9 May 2005; accepted 15 July 2005 KEYWORDS Advanced Life Support (ALS); Education; Training
Summary The Resuscitation Council (UK) Advanced Life Support (ALS) Course is a multidisciplinary training course which teaches participants how to manage the resuscitation of a patient at risk of or in cardiac arrest. To reduce variability in assessments, four standardised patient scenarios have been developed with common performance criteria. The aim of the study was to establish how much candidates remembered about their test in order to assess the potential for collusion. Eighty-nine candidates were asked immediately after testing what they remembered about their ALS scenario. Recall of the underlying problem with the simulated patient was good (85 [96%]). Forty-two [47%] correctly remembered the initial cardiac arrest rhythm and 55 [61%] the subsequent cardiac arrest rhythm. Fifty-nine [60%] candidates passed the assessment. Candidates who passed the assessment were significantly more likely to correctly recall the initial and subsequent cardiac arrest rhythms than those who did not. However, even in this group, the overall recall of all elements of the scenario was correct in only 49% of instances. This study demonstrated that immediately after testing candidates had good recall of the initial clinical scenario with which they were presented, but poor recall of cardiac arrest rhythms during the simulated resuscitation attempt. These findings provide some reassurance that the likelihood of successful collusion improving subsequent candidates performance is likely to be small as recall of the scenario progression is limited. © 2005 Elsevier Ireland Ltd. All rights reserved.
Introduction The Resuscitation Council (UK) Advanced Life Support (ALS) Course is a multidisciplinary training 夽 A Spanish translated version of the summary of this arti-
cle appears as Appendix in the online version at 10.1016/ j.resuscitation.2005.07.006. ∗ Corresponding author. Fax: +44 121 443 2494. E-mail address: [email protected] (G.D. Perkins).
course which aims to teach participants how to manage the resuscitation of a patient at risk of or in cardiac arrest.1,2 The course has been adopted by the European Resuscitation Council as the ERC ALS course. The course uses role play during simulated clinical scenarios to develop patient assessment, resuscitation and cardiac arrest team leadership skills. These skills are tested using a similar approach. Each candidate is assessed leading a resuscitation team (comprised of fac-
0300-9572/$ — see front matter © 2005 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.resuscitation.2005.07.006
240
R.P. Davies, G.D. Perkins
ulty members) in the systematic assessment of a critically ill patient and subsequently resuscitation from cardiopulmonary arrest. To reduce variability in assessments,3 four standardised patient scenarios have been developed with common performance criteria.4 Course Directors are tasked with selecting one scenario for testing and one for re-testing on each course. Whilst this reduces variability in the test delivered to candidates on a particular course, a potential limitation of this approach is loss of security of the examination. The aim of the study was to establish how much candidates remembered about the clinical scenario, initial and subsequent cardiac arrest rhythms encountered during ALS testing.
was calculated. Chi-squared or Fishers exact test were used to see if candidates that passed the assessment had better recall than those that did not. A P-value of 0.05 was considered statistically significant.
Results Eighty-nine candidates were recruited into the study. Overall candidate recall of the underlying problem with the simulated patient was good (85 [96%] correctly remembered the patient condition). However, recall of the initial cardiac arrest rhythm (42 [47%] correct) and subsequent cardiac arrest rhythm (55 [61%] correct) was poor. Few candidates correctly recalled all three elements of the scenario accurately (33 [37%] correct). The results for each different scenario are presented in Table 1. Fifty-nine [60%] candidates passed the assessment. Candidates who passed the assessment were significantly more likely to correctly recall the initial and subsequent cardiac arrest rhythms than those that did not Table 2. Even candidates who passed remembered the scenario progression incorrectly in 26—43% of instances.
Materials and methods Candidates at four different ALS centres in the UK completed a short questionnaire immediately following testing. They were asked: (1) what was underlying problem with their patient, (2) what was the initial cardiac arrest rhythm and (3) what was the subsequent cardiac arrest rhythm. Answers were provided using free text. Responses were then compared to the clinical scenario with which the candidate was tested. Whether the candidate passed or failed the assessment was also recorded. Data were collated using Microsoft Excel. The proportion of candidates who correctly recalled the clinical scenario, initial and subsequent rhythms
Table 1
Scenario Scenario Scenario Scenario
Discussion This study shows that immediately after testing candidates have good recall of the initial clinical scenario with which they are presented, but poor
Candidate recall of initial and subsequent scenario contents
1 2 3 4
N
Diagnosis Correct
Incorrect
Correct
23 22 30 14
23 19 30 13
0 3 0 1
9 (39%) 10 (45%) 19 (63%) 4 (28%)
(100%) (86%) (100%) (92%)
Initial rhythm
(0%) (14%) (0%) (8%)
Subsequent rhythm
All responses
Incorrect
Correct
Incorrect
Correct
Incorrect
14 12 11 10
8 (35%) 15 (68%) 23 (76%) 9 (64%)
15 (65%) 7 (32%) 7 (24%) 5 (36%)
3 (13%) 9 (41%) 17 (56%) 4 (28%)
20 13 13 10
(61%) (55%) (37%) (72%)
(87%) (59%) (44%) (72%)
Table 2 Candidate recall of initial and subsequent scenario contents according to whether candidates passed or failed the assessment Diagnosis Correct Pass Fail P-value
Incorrect
52 (98%) 1 (2%) 33 (92%) 3 (8%) 0.299
Initial rhythm
Subsequent rhythm
All responses
Correct
Correct
Correct
Incorrect
30 (57%) 23 (43%) 12 (33%) 24 (67%) 0.031
39 (74%) 16 (44%)
Incorrect 14 (26%) 20 (56%) 0.005
Incorrect
26 (49%) 27 (51%) 7 (19%) 29 (81%) 0.005
Recall after cardiac arrest scenario testing recall of cardiac arrest rhythms during the simulated resuscitation attempt. Candidates who passed the assessment were significantly more likely to remember the sequence of cardiac arrest rhythms. However, even in this group, the overall recall of all elements of the scenario was only correct in 49% of instances. The ALS course is a high stakes examination. This makes the reliability of the assessments important. We have demonstrated previously a significant variability between assessors, not only in judging performance, but also in the difficulty of the test itself.3 By creating a set of standardised testing scenarios with common performance criteria between them, one source of variability between assessments is reduced. A potential limitation of this approach is that the security of the exam and a fair test of performance may be lost if candidates collude by telling each other the contents of the test. Collusion or cheating in examinations is a concern in medical education, particularly with the growth of electronic media that make communication and flow of information easier.5 Remote electronic re-certification is already available as part of the American Heart Association Advanced Cardiac Life Support Programme.6 Whilst this offers advantages in allowing learning and re-certification to occur at the learners pace and convenience, direct supervision of participants during this process is impossible. Un-authorised duplication and sale of test papers (with or without answers) is another problem. Websites selling examination papers or model essays are a growing problem. ACLS Preparation (http://www.aclspreparation.com) offers to sell multiple choice papers for the AHA ACLS 2005 certification course. Many systems rely on the integrity and professionalism of the individuals involved. However, an awareness that cheating may occur is sensible and steps to assess the potential opportunities for cheating within an examination system should be encouraged. The findings of the present study provide some reassurance that the likelihood of successful collusion between candidates improving subsequent candidates performance is likely to be small as recall of the scenario progression is limited. The influence of two or more candidates discussing scenario content and the effect that this has on recall was not evaluated. It is possible that discussing the scenarios in groups may enhance the potential for benefit from collusion. Summative assessments are not primarily designed to provide feedback on performance, however, the opportunity to enhance learning should be seized at every opportunity.7 Although
241 the present study did not address what candidates have learned from their test, the fact that few could remember the details of their scenario suggests that learning may be similarly limited. An alternative approach which may overcome the issue of opportunities for collusion and possibly enhance learning opportunities might be to issue candidates with a template of the assessment defining clearly what is going to be tested. If the assessment truly matches the learning outcomes of the course then having prior knowledge of assessment should help candidates to achieve the learning outcomes.
Conclusion This study demonstrated that immediately after testing candidates on the ALS course had good recall of the initial clinical scenario with which they were presented, but poor recall of cardiac arrest rhythms during the simulated resuscitation attempt. These findings provide some reassurance that the likelihood of successful collusion between candidates improving subsequent candidates performance is likely to be small, as recall of the scenario progression is limited.
Conflict of interest GDP is a member of the Resuscitation Council (UK) Advanced Life Support Working group and was involved in the development of the present ALS cardiac arrest scenario tests.
Acknowledgements Advanced Life Support training centres at Alexandra Hospital, Redditch; Blackpool Victoria Hospital, Blackpool; New Cross Hospital, Wolverhampton; Grantham Hospital, Lincoln. This study was supported by the Resuscitation Council (UK).
References 1. Perkins G, Lockey A. The advanced life support provider course. BMJ 2002;325:S81. 2. Nolan JP. Advanced life support training. Resuscitation 2001;50:9—11. 3. Perkins GD, Hulme J, Tweed MJ. Variability in the assessment of advanced life support skills. Resuscitation 2001;50: 281—6.
242 4. Perkins GD. Cardiac arrest scenario testing. In: Bullock I, Colquhoun M, Coleman A, editors. Advanced life support instructor manual. London: Resuscitation Council (UK); 2004. p. 198—208. 5. Chaput de Saintonge DM, Pavlovic A. Cheating. Med Educ 2004;38:8—9.
R.P. Davies, G.D. Perkins 6. American Heart Association. HeartCode ACLS Anywhere. http://www.americanheart.org/presenter.jhtml?identifier= 3026627. 7. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001;357:945—9.