- Email: [email protected]
Efficacy and Feasibility of Objective Structured Clinical Examination in the Internal Assessment for Surgery Postgraduates
ORIGINAL REPORTS
Efficacy and Feasibility of Objective Structured Clinical Examination in the Internal Assessment for Surgery Postgraduates Anitha Muthusami, MS, MRCS (Ed),* Subair Mohsina, MBBS,* Sathasivam Sureshkumar, MS, DNB, MNAMS, FMAS,* Amaranathan Anandhi, MS, FMAS, FIAGES,* Thirthar Palanivelu Elamurugan, MS, DNB, FMAS,* Krishnamachari Srinivasan, MS,* Thulasingam Mahalakshmy, MD,† and Vikram Kate, MS, FRCS (Eng), FRCS (Ed), FRCS (Glasg), PhD (Surg Gastro), MAMS, FIMSA, MASCRS, FACS, FACG, MFSTEd* *
Department of Surgery, Pondicherry, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry, India; and †Department of Preventive and Social Medicine, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry, India
INTRODUCTION: Traditionally assessment in medical
training programs has been through subjective faculty evaluations or multiple choice questions. Conventional examinations provide assessment of the global performance rather than individual competencies thus making the final feedback less meaningful. The objective structured clinical examination (OSCE) is a relatively new multidimensional tool for evaluating training. This study was carried out to determine the efficacy and feasibility of OSCE as a tool for the internal assessment of surgery residents. METHODS: This study was carried out on the marks
obtained by surgery residents at different levels of training in a single tertiary center in India over the 4 OSCEs conducted in the years 2015 and 2016. The marks of the OSCE were collected from the departmental records and analyzed. Reliability was calculated using internal consistency using Cronbach’s α. Validity was calculated by item total correlation. Content validation was done by obtaining expert reviews from 5 experts using a proforma, to assess the content and checklist of each station of the OSCE. RESULTS: A total of 49 surgery residents were assessed in small batches during the above mentioned period. Of the 4 OSCEs conducted by us, 3 had a high value of Cronbach’s α of greater than 0.9, as opposed to the set standard of 0.7. Out of 23 stations used in the
Correspondence: Inquiries to Vikram Kate, MS, FRCS (Eng), FRCS (Ed), FRCS (Glasg), PhD (Surg Gastro), MAMS, FIMSA, MASCRS, FACS, FACG, MFSTEd, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry, India; fax: (413) 227-2066; e-mail: [email protected]
4 examinations separately, only 3 stations were found to have a low correlation coefficient (item total correlation), and hence, a low validity. The remaining 20 stations were found to have a high validity. Expert review showed unanimous validation of the content of 17 out of the 23 stations, with few suggestions for change in the remaining 6 stations. The material and manpower used was minimal and easy to obtain, thus making the OSCE feasible to conduct. CONCLUSION: OSCE is a reliable, valid. and feasible method for evaluating surgery residents at various levels of C training. ( J Surg Ed ]:]]]-]]]. J 2016 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: OSCE, objective assessment, reliability,
validity, surgical residents COMPETENCIES: Surgical Teaching, Professionalism, Res-
ident Evaluation, Interpersonal and Communication Skills, General Surgery
INTRODUCTION Traditionally assessment in medical training programs has been through subjective faculty evaluations or multiple choice questions. The former tend to inflate or diminish resident performance and the latter is a one-dimensional assessment tool. The objective structured clinical examination (OSCE) is a relatively new tool for evaluating physicians in training. The examination was introduced by Harden et al.1
Journal of Surgical Education & 2016 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2016.11.004
1
at the University of Dundee (Dundee, Scotland) in an effort to improve the evaluation of medical students’ clinical performance. In this study, OSCE consisted of a number of stations through which medical students rotated, spending predetermined amounts of time at each station.1 Different clinical skills were assessed at the individual stations, and item checklists were used to objectively grade students’ skills in performing clinical tasks. Evolution of OSCE as a teaching methodology is becoming widespread in use because of the feedback provided which is lacking in the conventional methods of examination. In an OSCE, the variables and complexity of examination are more easily controlled, its aims more clearly defined and the students’ knowledge can be tested. Hence, the patient and examiner variables are controlled, and the test becomes more objective and thereby easily repeatable. Moreover, communication skills, attitude, and psychomotor skills are not assessed objectively in our current assessment system. OSCE was described as the “gold standard for clinical assessment” by Norman.2 Though it quickly caught up as a gold standard assessment tool in Europe and North America, it was studied in Asia only after 1993.3 Yet even after 2 decades, there is not enough data from India available about the usage of OSCE as a tool for assessment as its use is limited to a few institutes. Moreover it has been used more for undergraduate assessment. OSCE, being a multidimensional assessment tool, can help assess individual competencies such as cognitive skills (including problem solving, decision making, and treatment skills) and psychomotor skills (including technical, communication, and physical examination skills) and, hence, can be of immense use in the assessment of students at postgraduate level of training.4 However, the experience with OSCE in postgraduate training programs, particularly surgical programs is limited. The utility of OSCE becomes more pronounced in a surgical residency training, where the conventional examination continues to evaluate the technical skills by the residents’ “know-how” regarding a particular technique/ procedure. In an OSCE examination, evaluation of how the residents perform the task on a simulators or simulated situations can be observed and evaluated thus making it more objective when compared to traditional examination, albeit not as effective as an ideal examination carried out within the operating room.5 OSCE can thus help in the improvising the assessment of a resident, particularly surgical trainee. Reliability and validity of OSCE as a tool for assessment has been previously determined in several studies carried out in the Western setup.6 OSCE can be used as a part of final assessment (summative OSCE) or as a continuous or sequential assessment tool (formative assessment).7 Pell et al.8 have shown that sequential testing improves the reliability, especially for borderline students. Feedbacks from students and student perception have been evaluated in several studies. OSCE has been found to
be highly accepted in previously described literature.9 Most of the student perception has been that OSCEs are easier than traditional examinations, and this perception has been seen to increase with the year of training of the resident. The recent rising trend and shift to the need for “competency-based medical education” has highlighted the need for direct observation and effective feedback. OSCE has thus been found to be a multidimensional and wholesome assessment tool for testing the competency of residents. Hence, this study was carried out to assess the reliability, validity, and feasibility of OSCE in the formative assessment of surgical residents.
2
Journal of Surgical Education Volume ]/Number ] ] 2016
METHODOLOGY Total 4 OSCEs were conducted in the year 2015 and 2016 in the department of surgery of a tertiary care institute in South India. The examinations conducted in the year 2015 had 5 comprehensive OSCE stations administered to 10 junior residents at the end of their second year of training (PGY2 2015), and 7 stations administered to 13 junior residents at the end of their first year of training (PGY1 2015). Similarly, the examinations conducted in the year 2016 had 5 OSCE stations administered to 13 (PGY2 2016) residents and 6 stations administered to thirteen (PGY1 2016) residents. All residents were from the same surgical training program in the above mentioned institute. The examinations were conducted at the end of each academic year. The examination for the PGY1 and PGY2 residents were conducted separately with a different set of questions. The various skills assessed were the residents problem solving ability, technical skills, communication skills, physical examination skills, and knowledge as given in Table 1. Each station was assigned a total score of 10 with the minimum required mark per station set as 50%. Each candidate was given 10 minutes at each station. Some of the stations were manned stations and some stations were unmanned. At each station the candidate had to carry out a different task based on the set of instructions provided. At some stations, they were asked to examine a patient or get consent for a procedure or demonstrate their suturing skills. The stations which required physical examination had actual patients whereas some other stations had someone playing the role of the patient. These people were given instructions before the beginning of the examination. All manned stations had a faculty member grading each resident, according to a given set of predetermined criteria presented in the form of a checklist. The items on the checklist were deemed critical for a competent examination. The faculty at the manned stations were given instructions to be passive evaluators, and not to guide or prompt the residents. All faculties belonged to the department of
TABLE 1. Content of OSCE Stations OSCE Problem Surgical anatomy of thyroid Breaking bad news Filling a death form
Description To test knowledge of surgical anatomy of thyroid which included labeling parts in a diagram. Communication of bad news to an actor simulating a sick patient. To test knowledge of filling a medicolegal form completely.
Surgical skills station
Demonstration of suturing techniques on a suturing board, testing suturing skills. Postoperative fluid To test knowledge of writing fluid and electrolyte orders for a management postoperative patient. Abdomen examination Examination of the abdomen of an actual patient with hepatomegaly. Knowledge of sutures To test the knowledge of various suturing materials in various scenarios. Thyroid examination Examination of an actual patient with a thyroid swelling and demonstration of various tests. Surgical skills station Demonstration of suturing techniques and hand knots. Getting consent for a major Getting consent for a major surgical procedure— elective procedure abdominoperineal resection, including stoma consent from a simulated patient. Interpretation of investigations To test knowledge of radiological investigations— identification of various pathologies in CT/barium/MRI. Knowledge of instruments Identification and testing of knowledge of indications, use and complications of a Sengstaken—Blakemore tube. Surgical skills station Demonstration of suturing techniques and omental patch closure. Inguinoscrotal swelling Examination of an actual patient with an inguinal hernia. examination Interpretation of biochemical To test knowledge of arterial blood gases (ABG) and its investigations interpretation. Interpretation of radiological To test knowledge of radiological investigations— investigations, specimens, identification of various pathologies in CT/barium/MRI, of instruments specimens—melanoma foot, and instruments—endoscopic band applicator. Getting consent for a major Getting consent for a major emergency procedure— emergency procedure exploratory laparotomy form a simulated patient.
Manned/ Unmanned
PGY LevelYear
Unmanned
PGY-1 2015
Manned
PGY-1 2015
Unmanned
PGY-1 2015, 2016 PGY-1 2015
Manned Unmanned
Manned
PGY-1 2015, 2016 PGY-1 2015, 2016 PGY-1 2015, 2016 PGY-2 2015
Manned Manned
PGY-2 2015 PGY-2 2015
Unmanned
PGY-2 2015
Unmanned
PGY-2 2015
Manned
PGY-2 2016
Manned
PGY-2 2016
Unmanned
PGY-2 2016
Unmanned
PGY-2 2016
Manned
PGY-1 2016
Manned Unmanned
CT, computed tomography; MRI, magnetic resonance imaging.
surgery. OSCE enables the faculty to evaluate how a resident elicits a clinical sign/performs the assigned task such as suturing or a particular procedure. This evaluation is not feasible in the conventional examination where the technical skill evaluation is confined to the residents’ description of how it is carried out rather than demonstration of the same. At the unmanned stations, the residents were tested on problem solving skills like interpretation of investigations— biochemical and radiological, knowledge of instruments, knowledge of surgical anatomy, and others (Table 1). Examination answer sheets were placed at the stations and residents were instructed beforehand to write their answers on the answer sheet and submit the paper to the faculty at the next station. The questions and checklists for each OSCE station were set by the faculty of the department of surgery. All physical props needed for the station were prepared by the faculty before the examination. The
questions, checklist, answer sheets, and the mark sheets of all 4 examinations were collected from the records in the department of surgery. The marks were tabulated for data analysis in de-identified format and analyzed using SPSS version 17. OSCE scores of each station and total score of each student were summarized using mean score with standard deviation. Validity was tested by expert content validity and item total correlation to assess “content” and “internal structure,” respectively.10 Validity was assessed by expert review of the content. A group of 5 experts were asked to review OSCE stations and checklists with a proforma using the Likert scale.11 The factors reviewed were, with reference to the OSCE station, whether the OSCE station was evaluating an essential skill for a surgical trainee, whether the subject was in line with the curriculum, whether the setting and context at each station was authentic and whether the instructions to the candidate were clear and unambiguous. The factors
Journal of Surgical Education Volume ]/Number ] ] 2016
3
TABLE 2. Reliability and Validity of OSCE for PGY-1 2015 Sl No. 1 2 3 4 5 6 7
Station Thyroid anatomy Filling death form Fluid orders Suture knowledge Surgical skills Abdomen examination Breaking bad news
Mean % OSCE Score
36
Average Score (SD) 3.8 3.4 3.7 3.8 3.9 3.6
Cronbach’s α (40.7)
(0.8) (0.7) (0.7) (0.6) (1.0) (0.9)
Item Total Correlation (40.7) 0.92 0.81 0.88 0.71 0.82 0.82
0.92
3.1 (0.8)
0.84
SD, standard deviation.
reviewed in reference with the checklist were, whether the checklist included the most essential steps for that particular skill and whether the checklist was clear, objective and unambiguous. In addition, item total correlation—Pearson’s correlation between the scores obtained by student in each station to that of total scored by the same student in the examination was calculated and a station with a correlation coefficient of 40.7 was considered to have a good validity. Competent performance was operationally defined as 50% individually in each station and in the overall score. Reliability (internal consistency) of the examination was assessed using Cronbach’s α to measure the stability of different stations.12 Cronbach’s α was calculated on the total scores from all stations for each examination per year. The unit of analysis in calculating Cronbach’s α was the total score of each examination. A value 40.7 was considered as good reliability.
unmanned stations. It was observed that the residents at the end of the second year of training performed better than those at the end of their first year of training, as expected in a valid examination where the performance improves with the level of training. OSCE-PGY1 2015 This examination comprised a total of 7 stations of which, 4 were unmanned and 3 were manned stations (Table 2). The manned stations were meant for evaluating the communication skills such as breaking bad news to an actor simulating a patient diagnosed with malignancy and technical skills such as demonstrating the suturing skills on a simulated tissue. Unmanned station was for assessing the problem solving ability such as writing the postoperative fluid orders. The overall mean percentage score for this examination was found to be 36%. The average score for all stations in this examination was found to be low (o5). The internal consistency calculated using Cronbach’s α was 0.92, showing that this examination had a high reliability. The validity of each station in this OSCE was also high, which is seen in the high individual item total correlation
RESULTS A total of 23 stations were prepared for a total of 49 candidates. Of these 12 were manned stations and 11 were
% of residents scoring <50% ment knowledgge PGYY1-2016Instrum men examinao PGYY1-2016Abdom on PG on GY1-2016Surgical Skills Stao PGY1-2 2016Fluid orde nt ers for a post operave paen PGY1 on 1-2016Consentt for Perforao PGY 1-2016 Filling Death form ws PGY1-2015Breeaking bad new men examinao PGYY1-2015Abdom on PG on GY1-2015Surgiical Skills stao ment knowledgge PGYY1-2015Instrum PGY1-2 2015Fluid Orde nt ers for a post operave paen ng a Death Form PG GY 1-2015 Fillin PGY 1-201 15 Surgical Anaatomy of thyroid on PGY2-2015 CT C interpetao PG on GY2-2015 Surgiical Skills stao Consent for AP PR PGY 2-2015C ment knowledgge PGY 2-2015Instrum naon of thyroid PGY 2-2015 2 Examin PGY 2-20 men/ instrumen 016 CT interpre nt etaon/ specim for AP PR PGY 2-2016 Consent C PGY 2-2016 Surgiical Skills stao on ood gas Analyssis PGY 2-20 016 Arterial Blo PGY 2-2016 Examinaon E off Inguinal Hernia 0
200
40
60
80
100
120
FIGURE 1. Distribution of the unsatisfactory performance of residents scoring less than 50% across the OSCE stations. 4
Journal of Surgical Education Volume ]/Number ] ] 2016
TABLE 3. Reliability and Validity of OSCE-PGY2 2015 Sl No. 1 2 3 4 5
Station Examination of thyroid SB tube knowledge Consent for APR Surgical skills CT interpretation
Mean % OSCE Score 75.9
Average Score (SD) 7.8 7.5 7.75 7.4 7.5
(1.0) (1.2) (0.7) (1.9) (0.9)
Cronbach’s α (40.7) 0.93
Item Total Correlation (40.7) 0.96 0.90 0.91 0.96 0.95
APR, abdominoperineal resection; CT, computed tomography; SB, Sengstaken–Blakemore; SD, standard deviation.
(40.7). Hence, this examination was found to have a high reliability and validity. The overall performance in this OSCE, however, was poor as seen by the percentage of unsatisfactory performance in each station (Fig. 1). All the 13 students failed to score the optimum marks in this examination. OSCE-PGY2 2015 This examination consisted of a total of 5 stations of which, 2 were unmanned and 3 were manned stations (Table 3). The manned stations included demonstration of clinical skills such as how to carry out thyroid examination in an actual patient and suturing skills and communication skill and aptitude of the candidate. The overall mean percentage score for this examination was found to be 75.9%. The average score for all stations in this examination was found to be 47. The internal consistency calculated using Cronbach’s α was 0.93, showing that this examination also had a high reliability. The validity of each station in this OSCE was also high, which is seen in the very high individual item total correlation of more than 0.9. Hence, this examination was found to have a high reliability and validity. The overall performance in this OSCE, however, was generally good with only 1 out of the 10 candidates (10%) failing to achieve the minimum required mark (50%) in 1 station (Fig. 1). OSCE-PGY1 2016 This examination consisted of a total of 6 stations, of which 3 were unmanned and 3 were manned stations (Table 4). The broad areas in this examination was devised similar to that of OSCE-PGY1 2015 to assess the aptitude,
communication, procedural skills, and problem solving ability of the candidate. The overall mean percentage score for this examination was found to be 74.3%. The average score for all stations in this examination was found to be more than 7. The internal consistency calculated using Cronbach’s α was 0.96, showing that this examination too had a high reliability. The validity of each station in this OSCE was also high, which is seen in the high individual item total correlation of more than 0.9. Hence, this examination was found to have a high reliability and validity. The overall performance in this OSCE showed that 2 out of 13 candidates (15.3%) failed to achieve the minimal required marks in 2 individual stations (Fig. 1). OSCE-PGY2 2016 This examination consisted of a total of 5 stations of which, 2 were unmanned and 3 were manned stations (Table 5). In this examination also manned stations assessed the communication and surgical skills. The residents were asked to perform an omental patch closure on simulated tissue. Unmanned stations evaluated the problem solving skills such as interpreting arterial blood gas and computed tomography (CT) images. The overall mean percentage score for this examination was found to be 79.8%. The average score for all stations in this examination was found to be more than 7, with 1 station having an average as high as 9. The internal consistency calculated using Cronbach’s α was 0.67, showing that this examination had a lower score for reliability. The validity of each station in this OSCE was variable, which is seen in individual item total correlation, with only 2 of the 5 stations having high correlation coefficient (40.7), whereas the remaining 3 stations had a low score for validity. Hence, this examination was found
TABLE 4. Reliability and Validity of OSCE-PGY1 2016 Sl No. 1 2 3 4 5 6
Station Filling death form Consent for perforation Fluid orders Surgical skills Abdomen examination Suture knowledge
Mean % OSCE Score
74.3
Average Score (SD) 7.6 7.3 7.6 7.1 7.6 7.4
(1.2) (1.3) (1.5) (1.0) (1.3) (1.5)
Cronbach’s α (40.7)
0.96
Item Total Correlation (40.7) 0.96 0.91 0.91 0.91 0.92 0.96
SD, standard deviation. Journal of Surgical Education Volume ]/Number ] ] 2016
5
TABLE 5. Reliability and Validity of OSCE-PGY2 2016 Sl No. 1 2 3 4 5
Station Examination of Inguinal hernia ABG interpretation Consent for APR Surgical skills CT/specimen, instrument knowledge
Mean % OSCE Score 79.8
Average Score (SD) 9 7.2 8.5 7.3 8.5
(1.5) (1.9) (1.0) (1.3) (1.6)
Cronbach's α (40.7) 0.67*
Item Total Correlation (40.7) 0.83 0.64† 0.41† 0.84 0.57†
ABG, arterial blood gas; APR, abdominoperineal resection; CT, computed tomography; SD, standard deviation. *The examination had a low score for reliability. † 3 of the 5 stations had a low score for validity.
FIGURE 2. Content validation of OSCE stations by 5 designated experts.
assessment system does not cater to the objective assessment of these skills. Stillman et al.13 reported that in many cases, internal medicine residents eliciting a history or performing a physical examination were never observed by faculty members. Attitude and communication skills are not at all assessed in the conventional examination. Sloan et al.14 reported that faculty members tend to inflate or deflate resident performance and are generally reluctant to underscore the clinical performance of the candidates. The assessment is confined to the candidates’ global performance rather than individual competency.15 This makes the final feedback to the students regarding the lacunae less meaningful as it does not provide information regarding specific ways to improvise themselves. After its introduction in 1975 by Harden et al., several subsequent authors have shown that OSCE is a reliable and valid examination not only for medical students but also for residents in a variety of disciplines, including internal medicine and surgery.16,17 These studies reported OSCE to be a valid and reliable examination in assessing technical skills especially in internal medicine and surgery when used as an in-training examination. It was found to be useful for assessing the multiple competencies related to procedural skills. In Canada, confidence in the OSCE as a superior method of performance evaluation has led the Medical Council of Canada and several specialty boards to include the OSCE in the licensing and certifying examination process.18 There are a few reports from India in the literature which demonstrated the utility of OSCE as an assessment tool for the orientation program for first-year surgical trainees and to assess their preparedness.19,20 Although OSCE is being practised in limited centers in India, it has not been widely accepted as a method for formative or summative assessment of surgical trainees in our country. A wide range of reliabilities for OSCE have been reported in literature, ranging from 0.19 to 0.89.21,22 In the present study, of the 4 OSCE examinations evaluated, 3 had high reliability and 1 had a relatively low reliability with a Cronbach’s α score of 0.67. The wide variation in the reliabilities in the literature was attributed to the shorter duration of examination. This led some authors to suggest that the test is only reliable if it lasts for 6 to 10 hours,
6
Journal of Surgical Education Volume ]/Number ] ] 2016
to have a low score for both reliability and validity. The overall performance in this OSCE was quite high with only 2 out of 13 candidates (15.3%) failing to acquire the adequate score in 1 station (Fig. 1). Expert review: The expert review obtained from 5 experts for content validation showed that 17 stations were unanimously validated by all the experts (Fig. 2). The remaining stations content validation was borderline, with none of the stations having poor or invalid contents. Of the stations with borderline validation, some were filling a death form, which some experts felt were not a part of postgraduate training, but a part of undergraduate training. In the station which tested the examination of thyroid, some experts opined that some of the signs such as Kocher’s sign elicited may not be necessary. Some experts felt that knowledge of computed tomography was “good to know” and not in the “need to know” category. Among the 3 stations which derived a poor correlation coefficient (o0.7), the expert review for content validation was satisfactory. Surprisingly, 2 of these stations had been previously found to have a high item total correlation.
DISCUSSION Clinical competence comprises not only of sound knowledge but also of various other aspects like interviewing and interpersonal skills, physical diagnosis skills, problem solving skills, and technical skills. Unfortunately, our present
which would clearly produce an unmanageable and impractical examination.21 As OSCE of more than 3 hours become cumbersome and less than half an hour might be inadequate for assessment, Idris et al.9 used 15 stations of 5 minutes each in their assessment of undergraduate students. Selby et al.,23 suggested 10–25 stations. However, in the present study, authors reported a high Cronbach’s α of 40.9 in 3 examinations with 5 to 7 stations of 10 minutes each. The OSCE fourth examination of 5 stations had Cronbach’s α less than the optimum value of 0.7. Although reports in the literature demonstrated optimum reliability with 12 to 16 stations using the Spearman-Brown prophecy formula, the present study was able to achieve optimum reliability with even lesser number of stations.6,14 The authors suggest that the number of stations should be decided based on the objectives of the course and the level of competence to be assessed. In the present study, the use of Cronbach’s α for measure of reliability is not without limitations because of the heterogenous constructs assessed in each examinations.24 However, it has been reported that a multidimensional test does not necessarily have a lower α than a unidimensional test.24 Although the unidimensionality of the constructs is not well represented by the use of Cronbach’s α on the average score of each examination, the α score was relatively higher in the 3 of the 4 examinations conducted indicating the stability of the stations tested.12 The high values of α in these examinations should be judged keeping in mind the fact that Interrater reliability and test retest reliability could not be assessed in these examinations. The higher values obtained in the item total correlation of each of the stations, however, helps to further emphasize the validity of the tests. The reliability for OSCEs reported in the literature for residents were found to be better than those for medical students. Sloan et al.14 administered OSCE examination to surgical trainees at multiple levels to assess clinical skills and reported high reliability which was attributed to the population tested. They have also found that the reliability was also improved by the inclusion of a larger number of clinical problems. Similarly in the present study, the high reliability achieved with lesser number of stations may be because of the inclusion of larger number of clinical problems which are addressed by the residents on day-today basis. The validity of the OSCE examination can be established by testing residents at multiple levels of training. More experienced residents are expected to perform better when compared to less experienced residents in a valid study.25 Sloan et al.14 reported a significant difference in the level of performance between senior residents, surgical trainees, and interns. Similar results were found in the present study where residents at end of second year of training performed better than residents at the end of their first year. In the present study, the validity of the examination was further emphasized by item total correlation and expert validation.
OSCE has several advantages. The trainees can be placed in a variety of controlled clinical situations that cannot be reproduced by an objective or subjective question on a clinical scenario. This allows for a holistic assessment of the candidate, especially, interpersonal and clinical skills.26 Most of the previously reported studies have found use of “simulated patients” reliable. In the present study, authors have relied on real patients wherever possible. Real patients are far superior to simulated patients especially in stations that require demonstration of clinical skills/physical signs. OSCE helps in identifying the problem resident and in giving effective feedback for addressing the lacunae in the training. Another advantage of the OSCE is that it emphasizes the importance of basic clinical skills to the postgraduates. The greatest benefit of the OSCE is that it helps in identifying the lacunae in the training programs in individual areas of competency rather than the global performance, which can in turn be used for modifying the teaching curriculum. In the present study, OSCE has been demonstrated as a reliable and valid tool for the assessment of surgical postgraduates. It helps to assess areas like surgical skills and communication skills, which are not normally assessed as a part of our traditional assessment examinations. The present study demonstrates the feasibility of conducting a valid OSCE examination for assessing surgical trainees in our set up. The authors intend to improve the current OSCE system attempted by providing a more structured feedback for individual residents.
Journal of Surgical Education Volume ]/Number ] ] 2016
7
CONCLUSION OSCE is a reliable, valid and feasible method for evaluating surgical residents at various levels of training. As OSCE overcomes some of the lacunae in the traditional assessment pattern and gives a unique insight into the residents’ clinical competence, the authors recommend that it can become a standard part of resident evaluation.
ACKNOWLEDGMENTS The authors would like to acknowledge Dr. S. Manwar Ali, Associate Professor of Surgery and Dr. Gomathi Shankar, Assistant Professor of Surgery for their contributions toward the expert review of the contents of OSCE.
REFERENCES 1. Harden RM, Stevenson M, Downie WW, Wilson
GM. Assessment of clinical competence using objective structured examination. Br Med J. 1975;1(5955): 447-451.
2. Norman G. Research in medical education: three
14. Sloan DA, Donnelly MB, Schwartz RW, Strode WE.
decades of progress. Br Med J. 2002;324(7353): 1560-1562.
The objective structured clinical examination—the new gold standard for evaluating postgraduate clinical performance. Ann Surg. 1995;222(6):735-742.
3. Nasir AA, Yusuf AS, Abdur-Rahman LO, Babalola
OM, Adeyeye A. A medical Students’ Perception of objective structured clinical examination: a feedback for process improvement. J Surg Educ. 2014;71(5): 701-706.
15. Ananthakrishnan N. Objective structured clinical/prac-
4. Patricio MF, Juliao M, Fareleira F, Carneiro AV. Is the
A. Reliability and validity of the objective structured clinical examination in assessing surgical residents. Am J Surg. 1990;160(6):302-305.
OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach J. 2013;35(6): 503-514. 5. Nguyen LN, Tardioli K, Roberts M, Watterson J.
Development and incorporation of hybrid simulation OSCE into in-training examinations to assess multiple CanMEDS competencies in urologic trainees. Can Urol Assoc J. 2015;9(1-2):32-36. 6. Jalilian N. Reliability and validity of objective struc-
tured clinical examination for residents of obstetrics and gynecology at Kermanshah University of Medical Sciences. Edu Res Med Sci. 2012;1:23-27. 7. Brannick
MT, Erol-Korkmaz HT, Prewett M. A systematic review of the reliability of objective structured clinical examination. Med Educ. 2011;45(12): 1181-1189.
tical examination (OSCE/OSPE). J Postgrad Med. 1993;39(2):82-84. 16. Cohen R, Reznick RK, Taylor BR, Provan J, Rothman
17. Pugh D, Hamstra SJ, Wood TJ, et al. A Procedural
skills OSCE: assessing technical and non-technical skills of internal medicine residents. Adv Health Sci Educ Theory Pract. 2015;20(1):85-100. 18. Reznick R, Smee S, Rothman A. An objective struc-
tured clinical examination for the licentiate: report of the Pilot Project of the Medical Council of Canada. Acad Med. 1992;67(8):487-494. 19. Pandya JS, Bhagwat SM, Kini SL. Evaluation of
clinical skills for first-year surgical residents using orientation programme and objective structured clinical evaluation as a tool of assessment. J Postgrad Med. 2010;56(4):297-300.
20. Bansal PK1, Saoji VA, Gruppen LD. From a “general-
objective structured clinical examination: sequential testing in theory and practice. Med Educ. 2013;47 (6):569-577.
ist” medical graduate to a “specialty” resident: can an entry-level assessment facilitate the transition? assessing the preparedness level of new surgical trainees Ann Acad Med Singapore. 2007;36(9):719-724.
9. Idris SA, Hamza AA, Elhaj MAB, et al. Students’
21. Newble DI, Swanson DB. Psychometric characteristics
perception of surgical objective structured clinical examination (OSCE) at final year MBBS, University of Khartoum, Sudan. Med J. 2014;1:17-20.
22. Roberts J, Norman G. Reliability and learning from
8. Pell G, Fuller R, Homer M, Roberts T. Advancing the
10. Ghaderi I, Manji F, Park YS, et al. Technical
Skills assessment toolbox a review using the unitary framework of validity. Ann Surg. 2015;261(2): 251-262. 11. Likert R. A technique for the measurement of atti-
tudes. Arch Psychol. 1932;22(140):55. 12. Al-Osail AM, Al-Sheikh MH, Al-Osail EM, et al. Is
of the objective structured clinical examination. Med Educ. 1988;22(4):325-334. the objective structured clinical examination. Med Educ. 1990;24(3):219-223. 23. Selby C, Osman L, Davis M, Lee M. Set up and run
an objective structured clinical exam. Br Med J. 1995;310(6988):1187-1190. 24. Tavakol M, Dennick R. Making sense of Cronbach’s
alpha. Int J Med Educ. 2011;2:53-55.
Cronbach’s alpha sufficient for assessing the reliability of the OSCE for an internal medicine course? BMC Res Notes. 2015;8:582. http://dx.doi.org/10.1016/ s13104-015-1533-x.
25. Joorabchi B. Objective structured clinical examination
13. Stillman PL, Swanson DB, Smee S. Assessing clinical
objective structured clinical examination (OSCE) to measure improvement in clinical competence during the surgical internship. Surgery. 1993;114(2):343-351.
skills of residents with standardized patients. Ann Intern Med. 1986;105(5):762-771.
8
in a pediatric residency program. Am J Dis Child. 1991;145(7):757-762. 26. Sloan DA, Donnelly MB, Johnson SB. Use of an
Journal of Surgical Education Volume ]/Number ] ] 2016
Efficacy and Feasibility of Objective Structured Clinical Examination in the Internal Assessment for Surgery Postgraduates Anitha Muthusami, MS, MRCS (Ed),* Subair Mohsina, MBBS,* Sathasivam Sureshkumar, MS, DNB, MNAMS, FMAS,* Amaranathan Anandhi, MS, FMAS, FIAGES,* Thirthar Palanivelu Elamurugan, MS, DNB, FMAS,* Krishnamachari Srinivasan, MS,* Thulasingam Mahalakshmy, MD,† and Vikram Kate, MS, FRCS (Eng), FRCS (Ed), FRCS (Glasg), PhD (Surg Gastro), MAMS, FIMSA, MASCRS, FACS, FACG, MFSTEd* *
Department of Surgery, Pondicherry, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry, India; and †Department of Preventive and Social Medicine, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry, India
INTRODUCTION: Traditionally assessment in medical
training programs has been through subjective faculty evaluations or multiple choice questions. Conventional examinations provide assessment of the global performance rather than individual competencies thus making the final feedback less meaningful. The objective structured clinical examination (OSCE) is a relatively new multidimensional tool for evaluating training. This study was carried out to determine the efficacy and feasibility of OSCE as a tool for the internal assessment of surgery residents. METHODS: This study was carried out on the marks
obtained by surgery residents at different levels of training in a single tertiary center in India over the 4 OSCEs conducted in the years 2015 and 2016. The marks of the OSCE were collected from the departmental records and analyzed. Reliability was calculated using internal consistency using Cronbach’s α. Validity was calculated by item total correlation. Content validation was done by obtaining expert reviews from 5 experts using a proforma, to assess the content and checklist of each station of the OSCE. RESULTS: A total of 49 surgery residents were assessed in small batches during the above mentioned period. Of the 4 OSCEs conducted by us, 3 had a high value of Cronbach’s α of greater than 0.9, as opposed to the set standard of 0.7. Out of 23 stations used in the
Correspondence: Inquiries to Vikram Kate, MS, FRCS (Eng), FRCS (Ed), FRCS (Glasg), PhD (Surg Gastro), MAMS, FIMSA, MASCRS, FACS, FACG, MFSTEd, Jawaharlal Institute of Postgraduate Medical Education and Research, Pondicherry, India; fax: (413) 227-2066; e-mail: [email protected]
4 examinations separately, only 3 stations were found to have a low correlation coefficient (item total correlation), and hence, a low validity. The remaining 20 stations were found to have a high validity. Expert review showed unanimous validation of the content of 17 out of the 23 stations, with few suggestions for change in the remaining 6 stations. The material and manpower used was minimal and easy to obtain, thus making the OSCE feasible to conduct. CONCLUSION: OSCE is a reliable, valid. and feasible method for evaluating surgery residents at various levels of C training. ( J Surg Ed ]:]]]-]]]. J 2016 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: OSCE, objective assessment, reliability,
validity, surgical residents COMPETENCIES: Surgical Teaching, Professionalism, Res-
ident Evaluation, Interpersonal and Communication Skills, General Surgery
INTRODUCTION Traditionally assessment in medical training programs has been through subjective faculty evaluations or multiple choice questions. The former tend to inflate or diminish resident performance and the latter is a one-dimensional assessment tool. The objective structured clinical examination (OSCE) is a relatively new tool for evaluating physicians in training. The examination was introduced by Harden et al.1
Journal of Surgical Education & 2016 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2016.11.004
1
at the University of Dundee (Dundee, Scotland) in an effort to improve the evaluation of medical students’ clinical performance. In this study, OSCE consisted of a number of stations through which medical students rotated, spending predetermined amounts of time at each station.1 Different clinical skills were assessed at the individual stations, and item checklists were used to objectively grade students’ skills in performing clinical tasks. Evolution of OSCE as a teaching methodology is becoming widespread in use because of the feedback provided which is lacking in the conventional methods of examination. In an OSCE, the variables and complexity of examination are more easily controlled, its aims more clearly defined and the students’ knowledge can be tested. Hence, the patient and examiner variables are controlled, and the test becomes more objective and thereby easily repeatable. Moreover, communication skills, attitude, and psychomotor skills are not assessed objectively in our current assessment system. OSCE was described as the “gold standard for clinical assessment” by Norman.2 Though it quickly caught up as a gold standard assessment tool in Europe and North America, it was studied in Asia only after 1993.3 Yet even after 2 decades, there is not enough data from India available about the usage of OSCE as a tool for assessment as its use is limited to a few institutes. Moreover it has been used more for undergraduate assessment. OSCE, being a multidimensional assessment tool, can help assess individual competencies such as cognitive skills (including problem solving, decision making, and treatment skills) and psychomotor skills (including technical, communication, and physical examination skills) and, hence, can be of immense use in the assessment of students at postgraduate level of training.4 However, the experience with OSCE in postgraduate training programs, particularly surgical programs is limited. The utility of OSCE becomes more pronounced in a surgical residency training, where the conventional examination continues to evaluate the technical skills by the residents’ “know-how” regarding a particular technique/ procedure. In an OSCE examination, evaluation of how the residents perform the task on a simulators or simulated situations can be observed and evaluated thus making it more objective when compared to traditional examination, albeit not as effective as an ideal examination carried out within the operating room.5 OSCE can thus help in the improvising the assessment of a resident, particularly surgical trainee. Reliability and validity of OSCE as a tool for assessment has been previously determined in several studies carried out in the Western setup.6 OSCE can be used as a part of final assessment (summative OSCE) or as a continuous or sequential assessment tool (formative assessment).7 Pell et al.8 have shown that sequential testing improves the reliability, especially for borderline students. Feedbacks from students and student perception have been evaluated in several studies. OSCE has been found to
be highly accepted in previously described literature.9 Most of the student perception has been that OSCEs are easier than traditional examinations, and this perception has been seen to increase with the year of training of the resident. The recent rising trend and shift to the need for “competency-based medical education” has highlighted the need for direct observation and effective feedback. OSCE has thus been found to be a multidimensional and wholesome assessment tool for testing the competency of residents. Hence, this study was carried out to assess the reliability, validity, and feasibility of OSCE in the formative assessment of surgical residents.
2
Journal of Surgical Education Volume ]/Number ] ] 2016
METHODOLOGY Total 4 OSCEs were conducted in the year 2015 and 2016 in the department of surgery of a tertiary care institute in South India. The examinations conducted in the year 2015 had 5 comprehensive OSCE stations administered to 10 junior residents at the end of their second year of training (PGY2 2015), and 7 stations administered to 13 junior residents at the end of their first year of training (PGY1 2015). Similarly, the examinations conducted in the year 2016 had 5 OSCE stations administered to 13 (PGY2 2016) residents and 6 stations administered to thirteen (PGY1 2016) residents. All residents were from the same surgical training program in the above mentioned institute. The examinations were conducted at the end of each academic year. The examination for the PGY1 and PGY2 residents were conducted separately with a different set of questions. The various skills assessed were the residents problem solving ability, technical skills, communication skills, physical examination skills, and knowledge as given in Table 1. Each station was assigned a total score of 10 with the minimum required mark per station set as 50%. Each candidate was given 10 minutes at each station. Some of the stations were manned stations and some stations were unmanned. At each station the candidate had to carry out a different task based on the set of instructions provided. At some stations, they were asked to examine a patient or get consent for a procedure or demonstrate their suturing skills. The stations which required physical examination had actual patients whereas some other stations had someone playing the role of the patient. These people were given instructions before the beginning of the examination. All manned stations had a faculty member grading each resident, according to a given set of predetermined criteria presented in the form of a checklist. The items on the checklist were deemed critical for a competent examination. The faculty at the manned stations were given instructions to be passive evaluators, and not to guide or prompt the residents. All faculties belonged to the department of
TABLE 1. Content of OSCE Stations OSCE Problem Surgical anatomy of thyroid Breaking bad news Filling a death form
Description To test knowledge of surgical anatomy of thyroid which included labeling parts in a diagram. Communication of bad news to an actor simulating a sick patient. To test knowledge of filling a medicolegal form completely.
Surgical skills station
Demonstration of suturing techniques on a suturing board, testing suturing skills. Postoperative fluid To test knowledge of writing fluid and electrolyte orders for a management postoperative patient. Abdomen examination Examination of the abdomen of an actual patient with hepatomegaly. Knowledge of sutures To test the knowledge of various suturing materials in various scenarios. Thyroid examination Examination of an actual patient with a thyroid swelling and demonstration of various tests. Surgical skills station Demonstration of suturing techniques and hand knots. Getting consent for a major Getting consent for a major surgical procedure— elective procedure abdominoperineal resection, including stoma consent from a simulated patient. Interpretation of investigations To test knowledge of radiological investigations— identification of various pathologies in CT/barium/MRI. Knowledge of instruments Identification and testing of knowledge of indications, use and complications of a Sengstaken—Blakemore tube. Surgical skills station Demonstration of suturing techniques and omental patch closure. Inguinoscrotal swelling Examination of an actual patient with an inguinal hernia. examination Interpretation of biochemical To test knowledge of arterial blood gases (ABG) and its investigations interpretation. Interpretation of radiological To test knowledge of radiological investigations— investigations, specimens, identification of various pathologies in CT/barium/MRI, of instruments specimens—melanoma foot, and instruments—endoscopic band applicator. Getting consent for a major Getting consent for a major emergency procedure— emergency procedure exploratory laparotomy form a simulated patient.
Manned/ Unmanned
PGY LevelYear
Unmanned
PGY-1 2015
Manned
PGY-1 2015
Unmanned
PGY-1 2015, 2016 PGY-1 2015
Manned Unmanned
Manned
PGY-1 2015, 2016 PGY-1 2015, 2016 PGY-1 2015, 2016 PGY-2 2015
Manned Manned
PGY-2 2015 PGY-2 2015
Unmanned
PGY-2 2015
Unmanned
PGY-2 2015
Manned
PGY-2 2016
Manned
PGY-2 2016
Unmanned
PGY-2 2016
Unmanned
PGY-2 2016
Manned
PGY-1 2016
Manned Unmanned
CT, computed tomography; MRI, magnetic resonance imaging.
surgery. OSCE enables the faculty to evaluate how a resident elicits a clinical sign/performs the assigned task such as suturing or a particular procedure. This evaluation is not feasible in the conventional examination where the technical skill evaluation is confined to the residents’ description of how it is carried out rather than demonstration of the same. At the unmanned stations, the residents were tested on problem solving skills like interpretation of investigations— biochemical and radiological, knowledge of instruments, knowledge of surgical anatomy, and others (Table 1). Examination answer sheets were placed at the stations and residents were instructed beforehand to write their answers on the answer sheet and submit the paper to the faculty at the next station. The questions and checklists for each OSCE station were set by the faculty of the department of surgery. All physical props needed for the station were prepared by the faculty before the examination. The
questions, checklist, answer sheets, and the mark sheets of all 4 examinations were collected from the records in the department of surgery. The marks were tabulated for data analysis in de-identified format and analyzed using SPSS version 17. OSCE scores of each station and total score of each student were summarized using mean score with standard deviation. Validity was tested by expert content validity and item total correlation to assess “content” and “internal structure,” respectively.10 Validity was assessed by expert review of the content. A group of 5 experts were asked to review OSCE stations and checklists with a proforma using the Likert scale.11 The factors reviewed were, with reference to the OSCE station, whether the OSCE station was evaluating an essential skill for a surgical trainee, whether the subject was in line with the curriculum, whether the setting and context at each station was authentic and whether the instructions to the candidate were clear and unambiguous. The factors
Journal of Surgical Education Volume ]/Number ] ] 2016
3
TABLE 2. Reliability and Validity of OSCE for PGY-1 2015 Sl No. 1 2 3 4 5 6 7
Station Thyroid anatomy Filling death form Fluid orders Suture knowledge Surgical skills Abdomen examination Breaking bad news
Mean % OSCE Score
36
Average Score (SD) 3.8 3.4 3.7 3.8 3.9 3.6
Cronbach’s α (40.7)
(0.8) (0.7) (0.7) (0.6) (1.0) (0.9)
Item Total Correlation (40.7) 0.92 0.81 0.88 0.71 0.82 0.82
0.92
3.1 (0.8)
0.84
SD, standard deviation.
reviewed in reference with the checklist were, whether the checklist included the most essential steps for that particular skill and whether the checklist was clear, objective and unambiguous. In addition, item total correlation—Pearson’s correlation between the scores obtained by student in each station to that of total scored by the same student in the examination was calculated and a station with a correlation coefficient of 40.7 was considered to have a good validity. Competent performance was operationally defined as 50% individually in each station and in the overall score. Reliability (internal consistency) of the examination was assessed using Cronbach’s α to measure the stability of different stations.12 Cronbach’s α was calculated on the total scores from all stations for each examination per year. The unit of analysis in calculating Cronbach’s α was the total score of each examination. A value 40.7 was considered as good reliability.
unmanned stations. It was observed that the residents at the end of the second year of training performed better than those at the end of their first year of training, as expected in a valid examination where the performance improves with the level of training. OSCE-PGY1 2015 This examination comprised a total of 7 stations of which, 4 were unmanned and 3 were manned stations (Table 2). The manned stations were meant for evaluating the communication skills such as breaking bad news to an actor simulating a patient diagnosed with malignancy and technical skills such as demonstrating the suturing skills on a simulated tissue. Unmanned station was for assessing the problem solving ability such as writing the postoperative fluid orders. The overall mean percentage score for this examination was found to be 36%. The average score for all stations in this examination was found to be low (o5). The internal consistency calculated using Cronbach’s α was 0.92, showing that this examination had a high reliability. The validity of each station in this OSCE was also high, which is seen in the high individual item total correlation
RESULTS A total of 23 stations were prepared for a total of 49 candidates. Of these 12 were manned stations and 11 were
% of residents scoring <50% ment knowledgge PGYY1-2016Instrum men examinao PGYY1-2016Abdom on PG on GY1-2016Surgical Skills Stao PGY1-2 2016Fluid orde nt ers for a post operave paen PGY1 on 1-2016Consentt for Perforao PGY 1-2016 Filling Death form ws PGY1-2015Breeaking bad new men examinao PGYY1-2015Abdom on PG on GY1-2015Surgiical Skills stao ment knowledgge PGYY1-2015Instrum PGY1-2 2015Fluid Orde nt ers for a post operave paen ng a Death Form PG GY 1-2015 Fillin PGY 1-201 15 Surgical Anaatomy of thyroid on PGY2-2015 CT C interpetao PG on GY2-2015 Surgiical Skills stao Consent for AP PR PGY 2-2015C ment knowledgge PGY 2-2015Instrum naon of thyroid PGY 2-2015 2 Examin PGY 2-20 men/ instrumen 016 CT interpre nt etaon/ specim for AP PR PGY 2-2016 Consent C PGY 2-2016 Surgiical Skills stao on ood gas Analyssis PGY 2-20 016 Arterial Blo PGY 2-2016 Examinaon E off Inguinal Hernia 0
200
40
60
80
100
120
FIGURE 1. Distribution of the unsatisfactory performance of residents scoring less than 50% across the OSCE stations. 4
Journal of Surgical Education Volume ]/Number ] ] 2016
TABLE 3. Reliability and Validity of OSCE-PGY2 2015 Sl No. 1 2 3 4 5
Station Examination of thyroid SB tube knowledge Consent for APR Surgical skills CT interpretation
Mean % OSCE Score 75.9
Average Score (SD) 7.8 7.5 7.75 7.4 7.5
(1.0) (1.2) (0.7) (1.9) (0.9)
Cronbach’s α (40.7) 0.93
Item Total Correlation (40.7) 0.96 0.90 0.91 0.96 0.95
APR, abdominoperineal resection; CT, computed tomography; SB, Sengstaken–Blakemore; SD, standard deviation.
(40.7). Hence, this examination was found to have a high reliability and validity. The overall performance in this OSCE, however, was poor as seen by the percentage of unsatisfactory performance in each station (Fig. 1). All the 13 students failed to score the optimum marks in this examination. OSCE-PGY2 2015 This examination consisted of a total of 5 stations of which, 2 were unmanned and 3 were manned stations (Table 3). The manned stations included demonstration of clinical skills such as how to carry out thyroid examination in an actual patient and suturing skills and communication skill and aptitude of the candidate. The overall mean percentage score for this examination was found to be 75.9%. The average score for all stations in this examination was found to be 47. The internal consistency calculated using Cronbach’s α was 0.93, showing that this examination also had a high reliability. The validity of each station in this OSCE was also high, which is seen in the very high individual item total correlation of more than 0.9. Hence, this examination was found to have a high reliability and validity. The overall performance in this OSCE, however, was generally good with only 1 out of the 10 candidates (10%) failing to achieve the minimum required mark (50%) in 1 station (Fig. 1). OSCE-PGY1 2016 This examination consisted of a total of 6 stations, of which 3 were unmanned and 3 were manned stations (Table 4). The broad areas in this examination was devised similar to that of OSCE-PGY1 2015 to assess the aptitude,
communication, procedural skills, and problem solving ability of the candidate. The overall mean percentage score for this examination was found to be 74.3%. The average score for all stations in this examination was found to be more than 7. The internal consistency calculated using Cronbach’s α was 0.96, showing that this examination too had a high reliability. The validity of each station in this OSCE was also high, which is seen in the high individual item total correlation of more than 0.9. Hence, this examination was found to have a high reliability and validity. The overall performance in this OSCE showed that 2 out of 13 candidates (15.3%) failed to achieve the minimal required marks in 2 individual stations (Fig. 1). OSCE-PGY2 2016 This examination consisted of a total of 5 stations of which, 2 were unmanned and 3 were manned stations (Table 5). In this examination also manned stations assessed the communication and surgical skills. The residents were asked to perform an omental patch closure on simulated tissue. Unmanned stations evaluated the problem solving skills such as interpreting arterial blood gas and computed tomography (CT) images. The overall mean percentage score for this examination was found to be 79.8%. The average score for all stations in this examination was found to be more than 7, with 1 station having an average as high as 9. The internal consistency calculated using Cronbach’s α was 0.67, showing that this examination had a lower score for reliability. The validity of each station in this OSCE was variable, which is seen in individual item total correlation, with only 2 of the 5 stations having high correlation coefficient (40.7), whereas the remaining 3 stations had a low score for validity. Hence, this examination was found
TABLE 4. Reliability and Validity of OSCE-PGY1 2016 Sl No. 1 2 3 4 5 6
Station Filling death form Consent for perforation Fluid orders Surgical skills Abdomen examination Suture knowledge
Mean % OSCE Score
74.3
Average Score (SD) 7.6 7.3 7.6 7.1 7.6 7.4
(1.2) (1.3) (1.5) (1.0) (1.3) (1.5)
Cronbach’s α (40.7)
0.96
Item Total Correlation (40.7) 0.96 0.91 0.91 0.91 0.92 0.96
SD, standard deviation. Journal of Surgical Education Volume ]/Number ] ] 2016
5
TABLE 5. Reliability and Validity of OSCE-PGY2 2016 Sl No. 1 2 3 4 5
Station Examination of Inguinal hernia ABG interpretation Consent for APR Surgical skills CT/specimen, instrument knowledge
Mean % OSCE Score 79.8
Average Score (SD) 9 7.2 8.5 7.3 8.5
(1.5) (1.9) (1.0) (1.3) (1.6)
Cronbach's α (40.7) 0.67*
Item Total Correlation (40.7) 0.83 0.64† 0.41† 0.84 0.57†
ABG, arterial blood gas; APR, abdominoperineal resection; CT, computed tomography; SD, standard deviation. *The examination had a low score for reliability. † 3 of the 5 stations had a low score for validity.
FIGURE 2. Content validation of OSCE stations by 5 designated experts.
assessment system does not cater to the objective assessment of these skills. Stillman et al.13 reported that in many cases, internal medicine residents eliciting a history or performing a physical examination were never observed by faculty members. Attitude and communication skills are not at all assessed in the conventional examination. Sloan et al.14 reported that faculty members tend to inflate or deflate resident performance and are generally reluctant to underscore the clinical performance of the candidates. The assessment is confined to the candidates’ global performance rather than individual competency.15 This makes the final feedback to the students regarding the lacunae less meaningful as it does not provide information regarding specific ways to improvise themselves. After its introduction in 1975 by Harden et al., several subsequent authors have shown that OSCE is a reliable and valid examination not only for medical students but also for residents in a variety of disciplines, including internal medicine and surgery.16,17 These studies reported OSCE to be a valid and reliable examination in assessing technical skills especially in internal medicine and surgery when used as an in-training examination. It was found to be useful for assessing the multiple competencies related to procedural skills. In Canada, confidence in the OSCE as a superior method of performance evaluation has led the Medical Council of Canada and several specialty boards to include the OSCE in the licensing and certifying examination process.18 There are a few reports from India in the literature which demonstrated the utility of OSCE as an assessment tool for the orientation program for first-year surgical trainees and to assess their preparedness.19,20 Although OSCE is being practised in limited centers in India, it has not been widely accepted as a method for formative or summative assessment of surgical trainees in our country. A wide range of reliabilities for OSCE have been reported in literature, ranging from 0.19 to 0.89.21,22 In the present study, of the 4 OSCE examinations evaluated, 3 had high reliability and 1 had a relatively low reliability with a Cronbach’s α score of 0.67. The wide variation in the reliabilities in the literature was attributed to the shorter duration of examination. This led some authors to suggest that the test is only reliable if it lasts for 6 to 10 hours,
6
Journal of Surgical Education Volume ]/Number ] ] 2016
to have a low score for both reliability and validity. The overall performance in this OSCE was quite high with only 2 out of 13 candidates (15.3%) failing to acquire the adequate score in 1 station (Fig. 1). Expert review: The expert review obtained from 5 experts for content validation showed that 17 stations were unanimously validated by all the experts (Fig. 2). The remaining stations content validation was borderline, with none of the stations having poor or invalid contents. Of the stations with borderline validation, some were filling a death form, which some experts felt were not a part of postgraduate training, but a part of undergraduate training. In the station which tested the examination of thyroid, some experts opined that some of the signs such as Kocher’s sign elicited may not be necessary. Some experts felt that knowledge of computed tomography was “good to know” and not in the “need to know” category. Among the 3 stations which derived a poor correlation coefficient (o0.7), the expert review for content validation was satisfactory. Surprisingly, 2 of these stations had been previously found to have a high item total correlation.
DISCUSSION Clinical competence comprises not only of sound knowledge but also of various other aspects like interviewing and interpersonal skills, physical diagnosis skills, problem solving skills, and technical skills. Unfortunately, our present
which would clearly produce an unmanageable and impractical examination.21 As OSCE of more than 3 hours become cumbersome and less than half an hour might be inadequate for assessment, Idris et al.9 used 15 stations of 5 minutes each in their assessment of undergraduate students. Selby et al.,23 suggested 10–25 stations. However, in the present study, authors reported a high Cronbach’s α of 40.9 in 3 examinations with 5 to 7 stations of 10 minutes each. The OSCE fourth examination of 5 stations had Cronbach’s α less than the optimum value of 0.7. Although reports in the literature demonstrated optimum reliability with 12 to 16 stations using the Spearman-Brown prophecy formula, the present study was able to achieve optimum reliability with even lesser number of stations.6,14 The authors suggest that the number of stations should be decided based on the objectives of the course and the level of competence to be assessed. In the present study, the use of Cronbach’s α for measure of reliability is not without limitations because of the heterogenous constructs assessed in each examinations.24 However, it has been reported that a multidimensional test does not necessarily have a lower α than a unidimensional test.24 Although the unidimensionality of the constructs is not well represented by the use of Cronbach’s α on the average score of each examination, the α score was relatively higher in the 3 of the 4 examinations conducted indicating the stability of the stations tested.12 The high values of α in these examinations should be judged keeping in mind the fact that Interrater reliability and test retest reliability could not be assessed in these examinations. The higher values obtained in the item total correlation of each of the stations, however, helps to further emphasize the validity of the tests. The reliability for OSCEs reported in the literature for residents were found to be better than those for medical students. Sloan et al.14 administered OSCE examination to surgical trainees at multiple levels to assess clinical skills and reported high reliability which was attributed to the population tested. They have also found that the reliability was also improved by the inclusion of a larger number of clinical problems. Similarly in the present study, the high reliability achieved with lesser number of stations may be because of the inclusion of larger number of clinical problems which are addressed by the residents on day-today basis. The validity of the OSCE examination can be established by testing residents at multiple levels of training. More experienced residents are expected to perform better when compared to less experienced residents in a valid study.25 Sloan et al.14 reported a significant difference in the level of performance between senior residents, surgical trainees, and interns. Similar results were found in the present study where residents at end of second year of training performed better than residents at the end of their first year. In the present study, the validity of the examination was further emphasized by item total correlation and expert validation.
OSCE has several advantages. The trainees can be placed in a variety of controlled clinical situations that cannot be reproduced by an objective or subjective question on a clinical scenario. This allows for a holistic assessment of the candidate, especially, interpersonal and clinical skills.26 Most of the previously reported studies have found use of “simulated patients” reliable. In the present study, authors have relied on real patients wherever possible. Real patients are far superior to simulated patients especially in stations that require demonstration of clinical skills/physical signs. OSCE helps in identifying the problem resident and in giving effective feedback for addressing the lacunae in the training. Another advantage of the OSCE is that it emphasizes the importance of basic clinical skills to the postgraduates. The greatest benefit of the OSCE is that it helps in identifying the lacunae in the training programs in individual areas of competency rather than the global performance, which can in turn be used for modifying the teaching curriculum. In the present study, OSCE has been demonstrated as a reliable and valid tool for the assessment of surgical postgraduates. It helps to assess areas like surgical skills and communication skills, which are not normally assessed as a part of our traditional assessment examinations. The present study demonstrates the feasibility of conducting a valid OSCE examination for assessing surgical trainees in our set up. The authors intend to improve the current OSCE system attempted by providing a more structured feedback for individual residents.
Journal of Surgical Education Volume ]/Number ] ] 2016
7
CONCLUSION OSCE is a reliable, valid and feasible method for evaluating surgical residents at various levels of training. As OSCE overcomes some of the lacunae in the traditional assessment pattern and gives a unique insight into the residents’ clinical competence, the authors recommend that it can become a standard part of resident evaluation.
ACKNOWLEDGMENTS The authors would like to acknowledge Dr. S. Manwar Ali, Associate Professor of Surgery and Dr. Gomathi Shankar, Assistant Professor of Surgery for their contributions toward the expert review of the contents of OSCE.
REFERENCES 1. Harden RM, Stevenson M, Downie WW, Wilson
GM. Assessment of clinical competence using objective structured examination. Br Med J. 1975;1(5955): 447-451.
2. Norman G. Research in medical education: three
14. Sloan DA, Donnelly MB, Schwartz RW, Strode WE.
decades of progress. Br Med J. 2002;324(7353): 1560-1562.
The objective structured clinical examination—the new gold standard for evaluating postgraduate clinical performance. Ann Surg. 1995;222(6):735-742.
3. Nasir AA, Yusuf AS, Abdur-Rahman LO, Babalola
OM, Adeyeye A. A medical Students’ Perception of objective structured clinical examination: a feedback for process improvement. J Surg Educ. 2014;71(5): 701-706.
15. Ananthakrishnan N. Objective structured clinical/prac-
4. Patricio MF, Juliao M, Fareleira F, Carneiro AV. Is the
A. Reliability and validity of the objective structured clinical examination in assessing surgical residents. Am J Surg. 1990;160(6):302-305.
OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach J. 2013;35(6): 503-514. 5. Nguyen LN, Tardioli K, Roberts M, Watterson J.
Development and incorporation of hybrid simulation OSCE into in-training examinations to assess multiple CanMEDS competencies in urologic trainees. Can Urol Assoc J. 2015;9(1-2):32-36. 6. Jalilian N. Reliability and validity of objective struc-
tured clinical examination for residents of obstetrics and gynecology at Kermanshah University of Medical Sciences. Edu Res Med Sci. 2012;1:23-27. 7. Brannick
MT, Erol-Korkmaz HT, Prewett M. A systematic review of the reliability of objective structured clinical examination. Med Educ. 2011;45(12): 1181-1189.
tical examination (OSCE/OSPE). J Postgrad Med. 1993;39(2):82-84. 16. Cohen R, Reznick RK, Taylor BR, Provan J, Rothman
17. Pugh D, Hamstra SJ, Wood TJ, et al. A Procedural
skills OSCE: assessing technical and non-technical skills of internal medicine residents. Adv Health Sci Educ Theory Pract. 2015;20(1):85-100. 18. Reznick R, Smee S, Rothman A. An objective struc-
tured clinical examination for the licentiate: report of the Pilot Project of the Medical Council of Canada. Acad Med. 1992;67(8):487-494. 19. Pandya JS, Bhagwat SM, Kini SL. Evaluation of
clinical skills for first-year surgical residents using orientation programme and objective structured clinical evaluation as a tool of assessment. J Postgrad Med. 2010;56(4):297-300.
20. Bansal PK1, Saoji VA, Gruppen LD. From a “general-
objective structured clinical examination: sequential testing in theory and practice. Med Educ. 2013;47 (6):569-577.
ist” medical graduate to a “specialty” resident: can an entry-level assessment facilitate the transition? assessing the preparedness level of new surgical trainees Ann Acad Med Singapore. 2007;36(9):719-724.
9. Idris SA, Hamza AA, Elhaj MAB, et al. Students’
21. Newble DI, Swanson DB. Psychometric characteristics
perception of surgical objective structured clinical examination (OSCE) at final year MBBS, University of Khartoum, Sudan. Med J. 2014;1:17-20.
22. Roberts J, Norman G. Reliability and learning from
8. Pell G, Fuller R, Homer M, Roberts T. Advancing the
10. Ghaderi I, Manji F, Park YS, et al. Technical
Skills assessment toolbox a review using the unitary framework of validity. Ann Surg. 2015;261(2): 251-262. 11. Likert R. A technique for the measurement of atti-
tudes. Arch Psychol. 1932;22(140):55. 12. Al-Osail AM, Al-Sheikh MH, Al-Osail EM, et al. Is
of the objective structured clinical examination. Med Educ. 1988;22(4):325-334. the objective structured clinical examination. Med Educ. 1990;24(3):219-223. 23. Selby C, Osman L, Davis M, Lee M. Set up and run
an objective structured clinical exam. Br Med J. 1995;310(6988):1187-1190. 24. Tavakol M, Dennick R. Making sense of Cronbach’s
alpha. Int J Med Educ. 2011;2:53-55.
Cronbach’s alpha sufficient for assessing the reliability of the OSCE for an internal medicine course? BMC Res Notes. 2015;8:582. http://dx.doi.org/10.1016/ s13104-015-1533-x.
25. Joorabchi B. Objective structured clinical examination
13. Stillman PL, Swanson DB, Smee S. Assessing clinical
objective structured clinical examination (OSCE) to measure improvement in clinical competence during the surgical internship. Surgery. 1993;114(2):343-351.
skills of residents with standardized patients. Ann Intern Med. 1986;105(5):762-771.
8
in a pediatric residency program. Am J Dis Child. 1991;145(7):757-762. 26. Sloan DA, Donnelly MB, Johnson SB. Use of an
Journal of Surgical Education Volume ]/Number ] ] 2016