- Email: [email protected]
Evaluation of a clinical skills orientation program for residents
GENERAL OBSTETRICS AND GYNECOLOGY Education
Evaluation of a clinical skills orientation program for residents Peter E. Nielsen, MD, Robert H. B. Holland, MD, and Lisa M. Foglia, MD Tacoma, Wash
OBJECTIVE: Our purpose was to implement and evaluate an orientation program for residents, focusing on outpatient clinical skills. STUDY DESIGN: Eleven of 12 residents participated in a clinical skills orientation program immediately preceding the academic year. The skill stations included evaluation of abnormal uterine bleeding, intrauterine device insertion, basic infertility evaluation, endometrial and vulvar biopsies, pelvic organ prolapse quantification examination, hysterosalpingography and office hysteroscopy, ultrasound scanning, labor and delivery triage, and clinic administrative responsibilities. Before test, after test, and anonymous resident evaluations were used to evaluate the program. RESULTS: First-year residents demonstrated a statistically significant increase in posttest scores compared to pretest scores (42.5% vs 71.3%, P = .003). Only first-year resident posttest scores for the labor and delivery triage and basic infertility evaluation stations demonstrated statistically significant increases over pretest scores (14.3% vs 46.4%, P = .009; and 41.7% vs 83.3%, P = .049, respectively). Sixty-four percent of the residents rated the program as ‘‘very helpful.’’ Most residents felt that the program was well organized and that the facilities were conducive to learning; all of the participants recommended an annual clinical orientation program. CONCLUSION: A clinical skills orientation program was well received and strongly desired by residents. Firstyear residents appeared to benefit the most from this orientation. (Am J Obstet Gynecol 2003;189:858-60.)
Key words: Resident orientation, clinical skills orientation
Transition from one level of medical training to another always invokes anxiety because of a host of new tasks, responsibilities, and expectations. Orientation to clinical skills is often ‘‘on the job’’ and requires a steep learning curve in the first few weeks or months of each new training year. Published reports that estimate the frequency of orientation programs from surveys of family medicine and emergency medicine residency program directors found that 50% to 90% of programs in these specialties have resident orientation.1,2 Few data exist to
From the Madigan Army Medical Center. The views expressed in this article are those of the authors and do not reflect the official policy or position of the Department of the Army, the Department of Defense, or the United States Government. Received for publication March 17, 2003; revised May 3, 2003; accepted May 8, 2003. Reprint requests: Peter E. Nielsen, MD, Department of Obstetrics and Gynecology, Madigan Army Medical Center, MCHJ-OG (ATTN: LTC Nielsen), Tacoma, WA 98431. E-mail: [email protected]. army.mil doi:10.1067/S0002-9378(03)00624-0
858
estimate the number, type, or effect of orientation programs that are offered in obstetric and gynecology residency programs. Grover and Puczynski3 surveyed 100 family medicine residency programs and evaluated 69 returned surveys for the most frequently presented activities at orientation. A social event with faculty was the most frequent; an assessment of cognitive knowledge was least frequent. This study demonstrated that residents desired orientation to clinical programs; however, the greatest number of orientation activities provided introduction to hospital services and administration, not cognitive knowledge or clinical skills. In addition, residents from military programs that were surveyed were more likely not to be satisfied with their orientations. Therefore, in an attempt to improve orientation for the transition in resident training years, a clinical skills orientation program was developed to provide practical advice and ‘‘hands-on’’ experience for residents who were progressing to the next level of training. The purpose of this project was to implement and evaluate an orientation program for residents that focused on clinical skills.
Nielsen, Holland, and Foglia 859
Volume 189, Number 3 Am J Obstet Gynecol
Fig 1. Pretest (light gray bars) and posttest (medium gray bars) scores by resident year level and for all year groups combined. PGY, Postgraduate year; NS, not significant.
Material and methods Eleven of 12 postgraduate-year level residents 1, 2, and 3 were administered a one-half–day orientation program immediately preceding the academic year. The study was reviewed and approved by the Madigan Army Medical Center Institutional Review Board. Faculty, fellows, and chief residents provided instruction in 10 stations. Each station emphasized a specific clinical skill and included 30 minutes of instruction, both didactic and ‘‘hands on’’ when appropriate. The stations included an evaluation of abnormal uterine bleeding, intrauterine device insertion, basic infertility evaluation, endometrial and vulvar biopsies, pelvic organ prolapse quantification examination, hysterosalpingography and office hysteroscopy, ultrasound scanning, labor and delivery triage, and clinic administrative responsibilities. Residents were assigned to stations on the basis of their specific level of training and the need for orientation to that specific clinical skill for that particular year of training. For example, all residents were oriented to the mechanics of ultrasound machine operation; however, only second- and third-year residents were introduced to techniques for basic fetal biometric measurements. First-, second-, and third-year residents rotated through the following stations: basic infertility evaluation, intrauterine device insertion, endometrial and vulvar biopsies, pelvic organ prolapse quantification evaluation, hysterosalpingogram and office hysteroscopy, ultrasound scans, and an administrative station. In addition, first-year residents rotated through vaginal delivery rounds and a labor and delivery triage station; second-year residents rotated through cesarean delivery rounds and colposcopy stations. Instruction at each station was conducted in small groups, with four or five resident students at each station. Faculty used lectures and handouts to aid instruction in the following stations: evaluation of abnormal uterine bleeding, basic infertility evaluation, pelvic organ prolapse quantification, labor and delivery triage, vaginal delivery and cesarean delivery rounds, and clinic administrative responsibilities. The remainder of
Fig 2. Pretest (light gray bars) and posttest (medium gray bars) scores for interns (n = 4) that were based on assigned orientation stations. NS, Not significant; L&D, labor and delivery; US, ultrasound scan; IUD, intrauterine device insertion; HSG, hysterosalpingography; INF, infertility evaluation; Admin, administrative duties.
the stations had training aids. Intrauterine device insertion was demonstrated with the training model from Paragard (Ortho-McNeil Pharmaceutical, Inc, Raritan, NJ). The endometrial and vulvar biopsy and colposcopy stations were performed in our colposcopy room with endometrial biopsy instruments, Kevorkian curettes, Kevorkian biopsy instruments, and minor surgical trays used as training aids and for demonstration. The office hysteroscopy and hysterosalpingography station was performed in our hysteroscopy room, with all equipment demonstrated. Pretest, posttest, and anonymous resident evaluations of each station and the overall program were used to evaluate the orientation. The Student t test (paired, two tailed) was used to compare continuous data; Fisher exact and v2 tests were used to compare proportions, and the Mann-Whitney U test was used to compare ordinal data. A probability value of <.05 was considered significant. Results Fig 1 presents pretest and posttest results for each resident group by year level and the entire cohort. Firstyear residents demonstrated a statistically significant increase in posttest scores compared with pretest scores (42.5% vs 71.3%, P = .003). All other resident groups demonstrated an increase in posttest scores that did not reach statistical significance. The entire cohort also demonstrated a statistically significant increase in posttest scores compared with pretest scores. The first-year resident group data were analyzed to determine whether any particular station was responsible for the increased posttest scores. Fig 2 shows that the first-year residents’ posttest scores for all stations increased; however, only the labor and delivery triage and basic infertility evaluation stations demonstrated statistically significant increases over pretest scores (14.3% vs 46.4%, P = .009; and 41.7% vs 83.3%, P = .049, respectively). Resident evaluations of the orientation were based on a 5-point scale, with 1 that
860 Nielsen, Holland, and Foglia
represented ‘‘not at all helpful’’ to 5 that represented ‘‘very helpful.’’ Sixty-four percent (7/11 residents) rated the program as ‘‘very helpful.’’ Most participants thought it was well organized (mean score, 4.5) and that the facilities were conducive to learning (mean score, 4.2). However, the residents did not feel that the program allowed sufficient time for each station (mean score, 3.4). Specific comments included the need for more discussion and ‘‘hands on’’ time, and all participants recommended that the orientation be provided annually. Because the labor and delivery and infertility station posttest scores for the first-year resident group were significantly increased compared to pretest scores, the evaluation scores for these stations were compared with those for all other stations. Results of this comparison demonstrated no significant differences in mean evaluation scores among these stations. Comment In 1994, Duff4 described a three-phase orientation program for obstetric and gynecologic first-year residents that would be administered over the 5 days before the start of the residency training. This training included certification in basic life support and neonatal resuscitation during the first phase. The second phase included a minicurriculum in obstetrics and gynecology and included prenatal care, fetal heart rate monitoring, labor and vaginal delivery, cesarean delivery, intrapartum and puerperal infection, antibiotic selection, pelvic inflammatory disease, and ectopic pregnancy. The final phase included a series of practicums that reviewed clinical skills. These skills included pelvic examination, gowning and gloving in the operating room, universal precautions, instruments, suture types, review of pelvic anatomy, and an animal laboratory for instruction in a variety of operative procedures. The residents in this cohort all strongly recommended that this orientation program be made a permanent part of the department’s education program. However, there was no pretest or posttest examination of these first-year residents to evaluate this orientation program objectively.
September 2003 Am J Obstet Gynecol
Our data confirm the findings of both Duff4 and Grover and Puczynski3 that residents desire a clinical orientation program as part of the new academic year. In our study, first-year residents appeared to benefit the most from the program, as demonstrated by the improvement in the posttest scores. Second- and third-year residents may not have benefited as much from the didactic portion of the orientation and therefore showed little improvement in posttest examination scores, which emphasized academic knowledge as opposed to the demonstration of practical skills. It is possible that these residents may have demonstrated statistically significant improvement in posttest scores if administered a ‘‘hands on’’ examination to evaluate these clinical skills. Despite this finding, most residents found the orientation very helpful and requested inclusion annually at the beginning of the academic year for first-, second-, and third-year residents. This orientation program could be administered in all residency programs with little cost and effort. The most significant component is the time that is dedicated by the staff and senior residents to organize and administer the stations. The program required one-half day for the residents and another one-half day of faculty preparation, including the setting up of each clinical station. Further work is needed to evaluate the effect of lengthening the duration of some stations and increasing the opportunity for additional ‘‘hands-on’’ exposure to these skills. In addition, more extensive use of models for both pretest and posttest evaluation may improve the resident’s experience and provide more accurate data with which to assess improvements in clinical skills.
REFERENCES 1. Merenstein JH, Preisach P. Orienting interns to being second-year residents. Fam Med 2002;34:101-3. 2. Brillman JC, Sklar DP, Viccellio P. Characteristics of emergency medicine resident orientation programs. Acad Emerg Med 1995; 2:25-31. 3. Grover M, Puczynski S. Residency orientation: what we present and its effect on our residents. Fam Med 1999;31:697-702. 4. Duff P. An orientation program for new residents in obstetrics and gynecology. Obstet Gynecol 1994;83:473-5.
Evaluation of a clinical skills orientation program for residents Peter E. Nielsen, MD, Robert H. B. Holland, MD, and Lisa M. Foglia, MD Tacoma, Wash
OBJECTIVE: Our purpose was to implement and evaluate an orientation program for residents, focusing on outpatient clinical skills. STUDY DESIGN: Eleven of 12 residents participated in a clinical skills orientation program immediately preceding the academic year. The skill stations included evaluation of abnormal uterine bleeding, intrauterine device insertion, basic infertility evaluation, endometrial and vulvar biopsies, pelvic organ prolapse quantification examination, hysterosalpingography and office hysteroscopy, ultrasound scanning, labor and delivery triage, and clinic administrative responsibilities. Before test, after test, and anonymous resident evaluations were used to evaluate the program. RESULTS: First-year residents demonstrated a statistically significant increase in posttest scores compared to pretest scores (42.5% vs 71.3%, P = .003). Only first-year resident posttest scores for the labor and delivery triage and basic infertility evaluation stations demonstrated statistically significant increases over pretest scores (14.3% vs 46.4%, P = .009; and 41.7% vs 83.3%, P = .049, respectively). Sixty-four percent of the residents rated the program as ‘‘very helpful.’’ Most residents felt that the program was well organized and that the facilities were conducive to learning; all of the participants recommended an annual clinical orientation program. CONCLUSION: A clinical skills orientation program was well received and strongly desired by residents. Firstyear residents appeared to benefit the most from this orientation. (Am J Obstet Gynecol 2003;189:858-60.)
Key words: Resident orientation, clinical skills orientation
Transition from one level of medical training to another always invokes anxiety because of a host of new tasks, responsibilities, and expectations. Orientation to clinical skills is often ‘‘on the job’’ and requires a steep learning curve in the first few weeks or months of each new training year. Published reports that estimate the frequency of orientation programs from surveys of family medicine and emergency medicine residency program directors found that 50% to 90% of programs in these specialties have resident orientation.1,2 Few data exist to
From the Madigan Army Medical Center. The views expressed in this article are those of the authors and do not reflect the official policy or position of the Department of the Army, the Department of Defense, or the United States Government. Received for publication March 17, 2003; revised May 3, 2003; accepted May 8, 2003. Reprint requests: Peter E. Nielsen, MD, Department of Obstetrics and Gynecology, Madigan Army Medical Center, MCHJ-OG (ATTN: LTC Nielsen), Tacoma, WA 98431. E-mail: [email protected]. army.mil doi:10.1067/S0002-9378(03)00624-0
858
estimate the number, type, or effect of orientation programs that are offered in obstetric and gynecology residency programs. Grover and Puczynski3 surveyed 100 family medicine residency programs and evaluated 69 returned surveys for the most frequently presented activities at orientation. A social event with faculty was the most frequent; an assessment of cognitive knowledge was least frequent. This study demonstrated that residents desired orientation to clinical programs; however, the greatest number of orientation activities provided introduction to hospital services and administration, not cognitive knowledge or clinical skills. In addition, residents from military programs that were surveyed were more likely not to be satisfied with their orientations. Therefore, in an attempt to improve orientation for the transition in resident training years, a clinical skills orientation program was developed to provide practical advice and ‘‘hands-on’’ experience for residents who were progressing to the next level of training. The purpose of this project was to implement and evaluate an orientation program for residents that focused on clinical skills.
Nielsen, Holland, and Foglia 859
Volume 189, Number 3 Am J Obstet Gynecol
Fig 1. Pretest (light gray bars) and posttest (medium gray bars) scores by resident year level and for all year groups combined. PGY, Postgraduate year; NS, not significant.
Material and methods Eleven of 12 postgraduate-year level residents 1, 2, and 3 were administered a one-half–day orientation program immediately preceding the academic year. The study was reviewed and approved by the Madigan Army Medical Center Institutional Review Board. Faculty, fellows, and chief residents provided instruction in 10 stations. Each station emphasized a specific clinical skill and included 30 minutes of instruction, both didactic and ‘‘hands on’’ when appropriate. The stations included an evaluation of abnormal uterine bleeding, intrauterine device insertion, basic infertility evaluation, endometrial and vulvar biopsies, pelvic organ prolapse quantification examination, hysterosalpingography and office hysteroscopy, ultrasound scanning, labor and delivery triage, and clinic administrative responsibilities. Residents were assigned to stations on the basis of their specific level of training and the need for orientation to that specific clinical skill for that particular year of training. For example, all residents were oriented to the mechanics of ultrasound machine operation; however, only second- and third-year residents were introduced to techniques for basic fetal biometric measurements. First-, second-, and third-year residents rotated through the following stations: basic infertility evaluation, intrauterine device insertion, endometrial and vulvar biopsies, pelvic organ prolapse quantification evaluation, hysterosalpingogram and office hysteroscopy, ultrasound scans, and an administrative station. In addition, first-year residents rotated through vaginal delivery rounds and a labor and delivery triage station; second-year residents rotated through cesarean delivery rounds and colposcopy stations. Instruction at each station was conducted in small groups, with four or five resident students at each station. Faculty used lectures and handouts to aid instruction in the following stations: evaluation of abnormal uterine bleeding, basic infertility evaluation, pelvic organ prolapse quantification, labor and delivery triage, vaginal delivery and cesarean delivery rounds, and clinic administrative responsibilities. The remainder of
Fig 2. Pretest (light gray bars) and posttest (medium gray bars) scores for interns (n = 4) that were based on assigned orientation stations. NS, Not significant; L&D, labor and delivery; US, ultrasound scan; IUD, intrauterine device insertion; HSG, hysterosalpingography; INF, infertility evaluation; Admin, administrative duties.
the stations had training aids. Intrauterine device insertion was demonstrated with the training model from Paragard (Ortho-McNeil Pharmaceutical, Inc, Raritan, NJ). The endometrial and vulvar biopsy and colposcopy stations were performed in our colposcopy room with endometrial biopsy instruments, Kevorkian curettes, Kevorkian biopsy instruments, and minor surgical trays used as training aids and for demonstration. The office hysteroscopy and hysterosalpingography station was performed in our hysteroscopy room, with all equipment demonstrated. Pretest, posttest, and anonymous resident evaluations of each station and the overall program were used to evaluate the orientation. The Student t test (paired, two tailed) was used to compare continuous data; Fisher exact and v2 tests were used to compare proportions, and the Mann-Whitney U test was used to compare ordinal data. A probability value of <.05 was considered significant. Results Fig 1 presents pretest and posttest results for each resident group by year level and the entire cohort. Firstyear residents demonstrated a statistically significant increase in posttest scores compared with pretest scores (42.5% vs 71.3%, P = .003). All other resident groups demonstrated an increase in posttest scores that did not reach statistical significance. The entire cohort also demonstrated a statistically significant increase in posttest scores compared with pretest scores. The first-year resident group data were analyzed to determine whether any particular station was responsible for the increased posttest scores. Fig 2 shows that the first-year residents’ posttest scores for all stations increased; however, only the labor and delivery triage and basic infertility evaluation stations demonstrated statistically significant increases over pretest scores (14.3% vs 46.4%, P = .009; and 41.7% vs 83.3%, P = .049, respectively). Resident evaluations of the orientation were based on a 5-point scale, with 1 that
860 Nielsen, Holland, and Foglia
represented ‘‘not at all helpful’’ to 5 that represented ‘‘very helpful.’’ Sixty-four percent (7/11 residents) rated the program as ‘‘very helpful.’’ Most participants thought it was well organized (mean score, 4.5) and that the facilities were conducive to learning (mean score, 4.2). However, the residents did not feel that the program allowed sufficient time for each station (mean score, 3.4). Specific comments included the need for more discussion and ‘‘hands on’’ time, and all participants recommended that the orientation be provided annually. Because the labor and delivery and infertility station posttest scores for the first-year resident group were significantly increased compared to pretest scores, the evaluation scores for these stations were compared with those for all other stations. Results of this comparison demonstrated no significant differences in mean evaluation scores among these stations. Comment In 1994, Duff4 described a three-phase orientation program for obstetric and gynecologic first-year residents that would be administered over the 5 days before the start of the residency training. This training included certification in basic life support and neonatal resuscitation during the first phase. The second phase included a minicurriculum in obstetrics and gynecology and included prenatal care, fetal heart rate monitoring, labor and vaginal delivery, cesarean delivery, intrapartum and puerperal infection, antibiotic selection, pelvic inflammatory disease, and ectopic pregnancy. The final phase included a series of practicums that reviewed clinical skills. These skills included pelvic examination, gowning and gloving in the operating room, universal precautions, instruments, suture types, review of pelvic anatomy, and an animal laboratory for instruction in a variety of operative procedures. The residents in this cohort all strongly recommended that this orientation program be made a permanent part of the department’s education program. However, there was no pretest or posttest examination of these first-year residents to evaluate this orientation program objectively.
September 2003 Am J Obstet Gynecol
Our data confirm the findings of both Duff4 and Grover and Puczynski3 that residents desire a clinical orientation program as part of the new academic year. In our study, first-year residents appeared to benefit the most from the program, as demonstrated by the improvement in the posttest scores. Second- and third-year residents may not have benefited as much from the didactic portion of the orientation and therefore showed little improvement in posttest examination scores, which emphasized academic knowledge as opposed to the demonstration of practical skills. It is possible that these residents may have demonstrated statistically significant improvement in posttest scores if administered a ‘‘hands on’’ examination to evaluate these clinical skills. Despite this finding, most residents found the orientation very helpful and requested inclusion annually at the beginning of the academic year for first-, second-, and third-year residents. This orientation program could be administered in all residency programs with little cost and effort. The most significant component is the time that is dedicated by the staff and senior residents to organize and administer the stations. The program required one-half day for the residents and another one-half day of faculty preparation, including the setting up of each clinical station. Further work is needed to evaluate the effect of lengthening the duration of some stations and increasing the opportunity for additional ‘‘hands-on’’ exposure to these skills. In addition, more extensive use of models for both pretest and posttest evaluation may improve the resident’s experience and provide more accurate data with which to assess improvements in clinical skills.
REFERENCES 1. Merenstein JH, Preisach P. Orienting interns to being second-year residents. Fam Med 2002;34:101-3. 2. Brillman JC, Sklar DP, Viccellio P. Characteristics of emergency medicine resident orientation programs. Acad Emerg Med 1995; 2:25-31. 3. Grover M, Puczynski S. Residency orientation: what we present and its effect on our residents. Fam Med 1999;31:697-702. 4. Duff P. An orientation program for new residents in obstetrics and gynecology. Obstet Gynecol 1994;83:473-5.