- Email: [email protected]
Workplace-based assessment: The need for continued evaluation and refinement
t h e s u r g e o n 9 ( 2 0 1 1 ) S 1 2 eS 1 3
available at www.sciencedirect.com
The Surgeon, Journal of the Royal Colleges of Surgeons of Edinburgh and Ireland www.thesurgeon.net
Review
Workplace-based assessment: The need for continued evaluation and refinement Jonathan Beard* University of Sheffield, UK
article info
abstract
Article history:
Workplace-Based Assessment (WBA) has been an integral part of the UK Intercollegiate
Received 22 October 2010
Surgical Curriculum Programme (ISCP) since 2007 (www.iscp.ac.uk). The UK Postgraduate
Accepted 3 November 2010
Medical Education and Training Board (now part of the General Medical Council) has defined WBA as ‘the assessment of working practices based on what trainees actually do in the workplace, and predominantly carried out in the workplace itself’ (www.gmc.org.uk).
Keywords:
This article reviews the purpose of WBA and the methods in current use. It also discusses
Workplace-based assessment
the misuse of WBA and possible solutions, including redesign of the rating scales.
Surgical training Competence
ª 2010 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved.
Performance
Introduction Workplace-Based Assessment (WBA) has been an integral part of the UK Intercollegiate Surgical Curriculum Programme (ISCP) since 2007 (www.iscp.ac.uk). The UK Postgraduate Medical Education and Training Board (now part of the General Medical Council) has defined WBA as ‘the assessment of working practices based on what trainees actually do in the workplace, and predominantly carried out in the workplace itself’ (www.gmc.org.uk). WBA integrates teaching, learning, assessment and feedback, and is designed to assess skills and behaviour rather than knowledge. Like most modern curricula, the ISCP is competence-based. Progress and certification are based on the achievement of defined competencies, rather than a fixed duration of training. The reduction in the clinical experience of trainees caused by shifts and hours of work regulations means that new methods are needed to
ensure that the requisite competencies are achieved within a reasonable timeframe. The ISCP uses WBA to address this need. It is important to understand that WBA is not a substitute for experience, as competence cannot be acquired without experience. The aim of WBA is to make the most of that experience.
The purpose of WBA The primary purpose of WBA is to aid learning by providing trainees with constructive feedback, based on objective, structured assessment (Assessment for Learning). It has been suggested that renaming WBA as Workplace-Based Assessment for Learning (WOBAL) might better explain this primary purpose! Although the principal role of each assessment is to aid learning, a collection of assessments can be used to inform
* Sheffield Vascular Institute, Northern General Hospital, Sheffield S5 7AU, UK. E-mail addresses: [email protected], [email protected]. 1479-666X/$ e see front matter ª 2010 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.surge.2010.11.014
t h e s u r g e o n 9 ( 2 0 1 1 ) S 1 2 eS 1 3
the Annual Review of Competency Progress (ARCP), provided that the trainee has requested enough assessments to provide sufficient evidence. This evidence, collated in the trainee’s portfolio, thus becomes a summative Assessment of Learning. It is important that evidence of competence is viewed together with the logbook of experience as the two are complementary. Like all assessments, WBA requires good utility, i.e. a combination of validity, reliability, acceptability, educational impact and cost-effectiveness. Whilst slightly lower reliability can be accepted for WBA than high-stakes (summative) exams, other components such as acceptability must be higher.
WBA methods WBA can be classified as opportunistic (done whenever the opportunity arises) and scheduled (because they require more planning). Opportunistic methods adopted by the ISCP included the mini-Clinical Evaluation Exercise (mini-CEX), Direct Observation of Procedural Skills in Surgery (S-DOPS) and Procedure-Based Assessment (PBA). The scheduled methods include Case-Based Discussion (CBD) and mini-Peer Assessment Tool (mini-PAT). Mini-PAT is a Multi-Source Feedback (MSF) method which is done once towards the end of a placement, and therefore has a more summative feel than the other methods. It is also the only method that does not require assessor training. Except for PBA, these methods are also used in the Foundation Programme (FP). Full descriptions, together with guidance notes, can be found on the ISCP website. The reason for choosing the same methods was because they had been tried and tested, and trainees coming from the FP would be familiar with them. However, except for PBA, they have turned out to have poor utility, for the reasons outlined below.
Misuse of WBA When WBA was first introduced, minimum numbers of assessments were set as a guide for trainees and assessors, as for the FP. In retrospect, setting minimum numbers was a mistake. Most trainees undertook the minimum, and infrequent assessments led to them being regarded as ‘miniexams’. Trainees practised informally and only asked for an assessment when they felt confident of achieving a good ‘score’. This also put pressure on the assessors to give a good ‘score’, which they usually did. Thus, the main purpose of WBA as an assessment for learning was lost and it was often regarded as being little more than a tick-box exercise. The guidance for WBA has now been amended to encourage early and frequent assessment. At the start of a placement, the educational supervisor and the trainee should determine the learning objectives and how they will be assessed. Early assessment allows the supervisor(s) to judge the level of the trainee, and subsequent assessments chart progression towards competence. The trainee should repeat assessments as often as necessary, in order to achieve the required standard before the end of the placement. The guidance has also been
S13
amended to state that a trainee must ask for an assessment beforehand, that verbal feedback is mandatory, the assessor must document the feedback given, and the assessment must be entered into the trainee’s portfolio, whatever the outcome. Trainees should be encouraged to use as many different assessors as possible, as this improves reliability and spreads the workload. It is vital that assessors are senior healthcare professionals who have been trained, especially with regard to giving feedback, and courses such as Training and Assessment in Practice (TAIP) provide such training (www.rcseng.ac.uk/ education/courses). Valuing and rewarding trainers and changing the culture towards assessment for learning will require a radical overhaul of postgraduate medical training in the UK, as recommended in Sir John Temple’s report for Medical Education England (www.mee.nhs.uk).
Design of WBA Most existing methods use a 6-point rating scale to rate each competency and the overall performance. The scale extends from ‘below expectations’ (1e2) to ‘above expectations’ (5e6), with 4 being defined as ‘meets expectations for completion of that level of training’. This norm-referenced scale is difficult for assessors to use as it requires a detailed knowledge of the competencies required for each level of the syllabus. Assessors therefore tend to give trainees the benefit of the doubt and award them a 4 or 5, as shown in a recent evaluation of SDOPS by the ISCP. The Royal College of Physicians (RCP) has also found the same problem with mini-CEX and CBD. The ISCP has addressed this problem by adopting the criterionreferenced scale for S-DOPS which was originally developed for PBA. This rates the ability of the trainee to perform the procedure on that occasion without help. The RCP has opted for a different solution for mini-CEX and CBD, using an educationally-referenced scale (e.g. ‘performed at the level expected for Certificate of Completion of Training’). Use of such scales has improved acceptability, validity and reliability, as they are more easily understood by assessors and trainees.
Conclusion Workplace-Based Assessments are a vital part of any competence-based postgraduate curriculum. Whilst it might seem tempting to pick an existing method ‘off the shelf’, utility cannot be assumed as it will depend upon the purpose of the assessment and the culture of the training programme. The purpose, timing and frequency of each assessment require detailed guidance. Continued evaluation and refinement are necessary to ensure that they have good utility and are fit for their intended purpose.
Conflict of interest None declared.
available at www.sciencedirect.com
The Surgeon, Journal of the Royal Colleges of Surgeons of Edinburgh and Ireland www.thesurgeon.net
Review
Workplace-based assessment: The need for continued evaluation and refinement Jonathan Beard* University of Sheffield, UK
article info
abstract
Article history:
Workplace-Based Assessment (WBA) has been an integral part of the UK Intercollegiate
Received 22 October 2010
Surgical Curriculum Programme (ISCP) since 2007 (www.iscp.ac.uk). The UK Postgraduate
Accepted 3 November 2010
Medical Education and Training Board (now part of the General Medical Council) has defined WBA as ‘the assessment of working practices based on what trainees actually do in the workplace, and predominantly carried out in the workplace itself’ (www.gmc.org.uk).
Keywords:
This article reviews the purpose of WBA and the methods in current use. It also discusses
Workplace-based assessment
the misuse of WBA and possible solutions, including redesign of the rating scales.
Surgical training Competence
ª 2010 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved.
Performance
Introduction Workplace-Based Assessment (WBA) has been an integral part of the UK Intercollegiate Surgical Curriculum Programme (ISCP) since 2007 (www.iscp.ac.uk). The UK Postgraduate Medical Education and Training Board (now part of the General Medical Council) has defined WBA as ‘the assessment of working practices based on what trainees actually do in the workplace, and predominantly carried out in the workplace itself’ (www.gmc.org.uk). WBA integrates teaching, learning, assessment and feedback, and is designed to assess skills and behaviour rather than knowledge. Like most modern curricula, the ISCP is competence-based. Progress and certification are based on the achievement of defined competencies, rather than a fixed duration of training. The reduction in the clinical experience of trainees caused by shifts and hours of work regulations means that new methods are needed to
ensure that the requisite competencies are achieved within a reasonable timeframe. The ISCP uses WBA to address this need. It is important to understand that WBA is not a substitute for experience, as competence cannot be acquired without experience. The aim of WBA is to make the most of that experience.
The purpose of WBA The primary purpose of WBA is to aid learning by providing trainees with constructive feedback, based on objective, structured assessment (Assessment for Learning). It has been suggested that renaming WBA as Workplace-Based Assessment for Learning (WOBAL) might better explain this primary purpose! Although the principal role of each assessment is to aid learning, a collection of assessments can be used to inform
* Sheffield Vascular Institute, Northern General Hospital, Sheffield S5 7AU, UK. E-mail addresses: [email protected], [email protected]. 1479-666X/$ e see front matter ª 2010 Royal College of Surgeons of Edinburgh (Scottish charity number SC005317) and Royal College of Surgeons in Ireland. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.surge.2010.11.014
t h e s u r g e o n 9 ( 2 0 1 1 ) S 1 2 eS 1 3
the Annual Review of Competency Progress (ARCP), provided that the trainee has requested enough assessments to provide sufficient evidence. This evidence, collated in the trainee’s portfolio, thus becomes a summative Assessment of Learning. It is important that evidence of competence is viewed together with the logbook of experience as the two are complementary. Like all assessments, WBA requires good utility, i.e. a combination of validity, reliability, acceptability, educational impact and cost-effectiveness. Whilst slightly lower reliability can be accepted for WBA than high-stakes (summative) exams, other components such as acceptability must be higher.
WBA methods WBA can be classified as opportunistic (done whenever the opportunity arises) and scheduled (because they require more planning). Opportunistic methods adopted by the ISCP included the mini-Clinical Evaluation Exercise (mini-CEX), Direct Observation of Procedural Skills in Surgery (S-DOPS) and Procedure-Based Assessment (PBA). The scheduled methods include Case-Based Discussion (CBD) and mini-Peer Assessment Tool (mini-PAT). Mini-PAT is a Multi-Source Feedback (MSF) method which is done once towards the end of a placement, and therefore has a more summative feel than the other methods. It is also the only method that does not require assessor training. Except for PBA, these methods are also used in the Foundation Programme (FP). Full descriptions, together with guidance notes, can be found on the ISCP website. The reason for choosing the same methods was because they had been tried and tested, and trainees coming from the FP would be familiar with them. However, except for PBA, they have turned out to have poor utility, for the reasons outlined below.
Misuse of WBA When WBA was first introduced, minimum numbers of assessments were set as a guide for trainees and assessors, as for the FP. In retrospect, setting minimum numbers was a mistake. Most trainees undertook the minimum, and infrequent assessments led to them being regarded as ‘miniexams’. Trainees practised informally and only asked for an assessment when they felt confident of achieving a good ‘score’. This also put pressure on the assessors to give a good ‘score’, which they usually did. Thus, the main purpose of WBA as an assessment for learning was lost and it was often regarded as being little more than a tick-box exercise. The guidance for WBA has now been amended to encourage early and frequent assessment. At the start of a placement, the educational supervisor and the trainee should determine the learning objectives and how they will be assessed. Early assessment allows the supervisor(s) to judge the level of the trainee, and subsequent assessments chart progression towards competence. The trainee should repeat assessments as often as necessary, in order to achieve the required standard before the end of the placement. The guidance has also been
S13
amended to state that a trainee must ask for an assessment beforehand, that verbal feedback is mandatory, the assessor must document the feedback given, and the assessment must be entered into the trainee’s portfolio, whatever the outcome. Trainees should be encouraged to use as many different assessors as possible, as this improves reliability and spreads the workload. It is vital that assessors are senior healthcare professionals who have been trained, especially with regard to giving feedback, and courses such as Training and Assessment in Practice (TAIP) provide such training (www.rcseng.ac.uk/ education/courses). Valuing and rewarding trainers and changing the culture towards assessment for learning will require a radical overhaul of postgraduate medical training in the UK, as recommended in Sir John Temple’s report for Medical Education England (www.mee.nhs.uk).
Design of WBA Most existing methods use a 6-point rating scale to rate each competency and the overall performance. The scale extends from ‘below expectations’ (1e2) to ‘above expectations’ (5e6), with 4 being defined as ‘meets expectations for completion of that level of training’. This norm-referenced scale is difficult for assessors to use as it requires a detailed knowledge of the competencies required for each level of the syllabus. Assessors therefore tend to give trainees the benefit of the doubt and award them a 4 or 5, as shown in a recent evaluation of SDOPS by the ISCP. The Royal College of Physicians (RCP) has also found the same problem with mini-CEX and CBD. The ISCP has addressed this problem by adopting the criterionreferenced scale for S-DOPS which was originally developed for PBA. This rates the ability of the trainee to perform the procedure on that occasion without help. The RCP has opted for a different solution for mini-CEX and CBD, using an educationally-referenced scale (e.g. ‘performed at the level expected for Certificate of Completion of Training’). Use of such scales has improved acceptability, validity and reliability, as they are more easily understood by assessors and trainees.
Conclusion Workplace-Based Assessments are a vital part of any competence-based postgraduate curriculum. Whilst it might seem tempting to pick an existing method ‘off the shelf’, utility cannot be assumed as it will depend upon the purpose of the assessment and the culture of the training programme. The purpose, timing and frequency of each assessment require detailed guidance. Continued evaluation and refinement are necessary to ensure that they have good utility and are fit for their intended purpose.
Conflict of interest None declared.