- Email: [email protected]
Needs assessment of neurosurgery trainees: a survey study of two large training programs in the developing and developed worlds
Surgical Neurology 66 (2006) 117 – 126 www.surgicalneurology-online.com
Education and Training
Needs assessment of neurosurgery trainees: a survey study of two large training programs in the developing and developed worldsB Mark Bernstein, MD, MHSc, FRCSCa,4, Stanley J. Hamstra, PhDb, Sarah Woodrow, MD, MEda,b, Shaila Goldsmana, Richard K. Reznick, MD, MEd, FRCSCa,b, David Fairholm, MD, FRCSCc a
b
Department of Surgery, University of Toronto, Toronto, Canada M5G 1L5 Wilson Centre for Research in Education, University of Toronto, Toronto, Canada M5G 2C4 c Department of Surgery, University of British Columbia, Vancouver, Canada V5Z 1L7 Received 9 August 2005; accepted 9 December 2005
Abstract
Background: There are challenges facing surgical education in both the developing and the developed worlds. Few studies have examined trainee perceptions of their educational needs in a systematic way. We undertook a study to examine this issue, focusing on two large training programs, one in the developed world and one in the developing world. Methods: Neurosurgical trainees at the University of Toronto, Toronto, Canada, and at Hasan Sadikin Hospital in Bandung, Indonesia, were surveyed with a comprehensive questionnaire assessing both the content and the methods of their training. The questionnaire had 37 quantitative questions requesting responses on a 7-point Likert scale and three open-ended questions to give more qualitative data. Results: Sixty-four percent of all trainees responded. A number of interesting findings about the strengths and weaknesses of training emerged. For example, Bandung trainees felt they had excellent training in trauma but not in specialty areas, especially spine and vascular, with ample opportunity to operate as the primary surgeon. Toronto trainees felt that the volume and the variety of cases were excellent but they did not have enough ambulatory experience, and that they had suboptimal experience as the primary surgeon. Trainees in both centers agree that they will feel competent to practice neurosurgery upon completion of their training. Conclusion: This study defined different educational needs for neurosurgical trainees in two centers that reflect both their individual training environments and the local culture of medicine. As such, trainees’ perceptions of these needs represent an important adjunct to program evaluation. D 2006 Elsevier Inc. All rights reserved.
Keywords:
Developed world; Developing world; Needs assessment; Surgical education
1. Introduction Challenges to training surgeons exist in the developing and the developed worlds. To focus initially on the developing world, it is relatively underserviced regarding specialty surgical care. For example, Indonesia has 240 000 000 people and only about 90 neurosurgeons,
B
Dr Woodrow was a recipient of the Royal College of Physicians and Surgeons of Canada’s Fellowship in Medical Education. 4 Corresponding author. Toronto Western Hospital, Toronto, ON, Canada M5T 2S8. Tel.: +1 416 603 6499; fax: +1 416 603 5298. E-mail address: [email protected] (M. Bernstein). 0090-3019/$ – see front matter D 2006 Elsevier Inc. All rights reserved. doi:10.1016/j.surneu.2005.12.033
compared with Canada, which has about 180 neurosurgeons for just more than 30 000 000 people. Furthermore, in general, these specialist surgeons are not as well supported with resources as those in the developed world. Bandung has the largest neurosurgery training program in Indonesia with 42 residents. Smaller training programs are in Jakarta and Surabaya. In Bandung, there are 5 staff to train the 42 residents, whereas in Toronto, there are 24 staff and 32 residents. This discrepancy in educatortrainee ratio would be expected to produce significant challenges in the Bandung training program. One way Bandung has tried to help cope with resident training is to welcome temporary visitors from the developed world
118
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
under the auspices of The Foundation for International Education in Neurological Surgery (FIENS) [1]. Between two of the authors (DF and MB), about 8 person-months of on-site training has been committed to the residents in Bandung over the last 3 years [2,3]. After these visits, we became curious about the perception that residents in these two diverse training programs had of their respective training programs. We speculated that these questions might be addressed with a formal needs assessment survey. Little rigorous study of training of surgical residents in the developing world is available, and any literature is observational and anecdotal [8,12,14,20,24 -27]. Likewise, in the developed world, there still exist numerous challenges to the successful training of future neurosurgeons. Limited operating room resources, restricted resident work hours, a transition toward outpatient care, and increasing medicolegal concerns are just some of the many issues that surgical educators must try and balance with the educational needs of their trainees. Despite a constantly evolving healthcare environment, there is similarly a paucity of literature on needs assessment in postgraduate training in the developed world. An extensive search using MEDLINE, EMBASE, and ERIC turned up few relevant articles, including a brief review of the methodology of study of needs assessment [22] and a needs assessment of interns regarding their medical school training [16]. In the surgical literature, there is a description of a needs assessment of practicing surgeons regarding the adequacy of their training in communicating with patients [13] and of a needs assessment for surgeons in their role as educators [7]. The challenges of surgical training in the developing world might be considered self-evident to surgical educators in the developing and developed worlds. The purpose of doing this study was to investigate perceptions of training in hopes of providing useful data that could help neurosurgery programs in the developing and the developed worlds to improve the program, the training experience, and ultimately the quality of graduating neurosurgeons and hence patient care. Formal documentation of this information could also serve to enhance awareness of the main issues of importance to trainees in this and other surgical disciplines in the developing world and in the developed world.
2. Methods 2.1. Study design The study was a mixed-methods (ie, both quantitative and qualitative data), cross-sectional (ie, one point in time examined) study. It consisted of a self-administered structured questionnaire filled out anonymously by neurosurgery residents and recent graduates (Appendix A). Thirty-seven questions requested responses using a Likert scale to record the score of each response. Three additional questions were open-ended to obtain more qualitative data. Interviews were felt not to be feasible in this setting for a number of reasons,
including the perceived difficulty in recruiting a skilled interviewer (especially in Bandung) whom the trainees could trust completely with respect to confidentiality. We attempted to design a comprehensive and feasible questionnaire to ensure high compliance and to obtain the maximum amount of information possible [19]. Input on questionnaire content was received from several University of Toronto neurosurgery residents, Canadian-trained clinical fellows, and several clinical fellows (currently training at the Toronto Western Hospital) who received their postgraduate training in the developing world. 2.2. Setting and selection of participants All 42 residents currently in the neurosurgery training program in Bandung, Indonesia, located in Hasan Sadikin Hospital, Medical School of Padjadjaran University, were invited to participate, as were 7 recent graduates of this program now practicing in Indonesia. Potential participants in Toronto included all 32 current residents in the program and 6 recent graduates now practicing in Toronto. 2.3. Data collection and analysis Surveys were given to the participants with explicit verbal and written instructions by the principal author (MB) on the first day of a month-long sabbatical in Bandung, November 22, 2004, and collected before he left. Simultaneously, the questionnaire was sent by e-mail to the University of Toronto residents and recent graduates and collected over the same period. Mean responses for each question constructed on a 7-point Likert scale were calculated. Data were analyzed using commercially available software (SPSS version 11.5, SPSS Inc, Chicago, Ill). Because the data were found to be nonnormal in their distribution, Likert responses were compared between groups using the Mann-Whitney U test. Statistical significance was defined as a probability value of less than .05. A modified thematic analysis of responses to the qualitative questions was also performed [29]. 2.4. Research ethics Participation was voluntary and informed consent was implied from each participant who returned a completed questionnaire. The protocol was approved by the Research Ethics Board at The University Health Network. The Research Ethics Board at Hasan Sadikin Hospital in Bandung gave verbal approval, stating it did not need to formally review the protocol because there was no research on patients. As a courtesy, permission to do the study was also obtained from the Chairmen of Neurosurgery in Bandung and in Toronto. All surveys and survey data were stored anonymously and confidentially. 3. Results Of the 87 residents and recent graduates (hereafter referred to as trainees) invited to participate, 56 participants
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
119
Table 1 Neurosurgical and nonneurosurgical training The The The The The The The The The The The The The The
amount amount amount amount amount amount amount amount amount amount amount amount amount amount
of of of of of of of of of of of of of of
training training training training training training training training training training training training training training
in in in in in in in in in in in in in in
trauma surgery vascular neurosurgery benign spinal disorders surgery of functional disorders peripheral nerve surgery pediatric neurosurgery spinal column and spinal cord tumors brain tumor surgery neurology neuropathology neuroradiology general surgery the basic sciences intensive care
Bandung
Toronto
P
5.93 2.14 3.36 1.82 1.68 4.68 3.11 4.25 3.36 2.79 3.25 4.36 4.07 3.75
3.96 3.50 4.54 3.04 2.59 4.11 3.71 4.50 4.43 4.43 4.04 5.07 3.64 4.00
b .000 .001 .001 .001 .001 .053 .104 .958 b .000 b .000 .015 .043 .221 .329
Respondents’ ratings of the amount of training received in each of the following subspecialties on a 7-point Likert scale (1 = too little, 7 = too much). Mean scores are reported, with P values indicating differences between the groups.
completed the survey (overall response rate 64%). Of the Bandung study group, 28 of 49 responded (response rate 57%), whereas 28 of 38 potential respondents from Toronto participated (response rate 74%). Level of training was broken down as follows. In Bandung, there were 5 junior residents (postgraduate year [PGY] 1 and 2), 9 mid-level residents (PGY 3 and 4), 8 senior residents (PGY 5 and 6), and 6 recent graduates. In Toronto, there were 9 junior residents (PGY 1 and 2), 10 mid-level residents (PGY 3 and 4), 6 senior residents (PGY 5 and 6), and 3 recent graduates. Overall, the demographics of the respondents were representative of the trainee pool in each center in terms of both level of training and sex. 3.1. Quantitative data Trainees were asked to rate their exposure to different aspects of both neurosurgical and related training. The mean values for the Likert scale responses for each group are reported in Table 1, along with P values indicating differences between the groups. In general, Toronto trainees perceived receiving appropriate amounts of training in most of the 14 specialty areas examined, although deficits were most strongly noted in peripheral nerve and functional neurosurgery with an excess of time devoted to general
surgical training. In contrast, trainees in Bandung were more likely to perceive gaps in their training with excessive exposure in trauma and pediatric neurosurgery noted. Statistically significant differences between the Bandung and Toronto groups were observed for 9 of 14 items. In examining more general learning opportunities such as operative experience, formal instruction received, and opportunity to see ambulatory patients, fewer differences were noted in the perception of the two groups (Table 2). Despite reporting limited amounts of training in Table 1, trainees in Bandung believe that they experience appropriate amounts of training as primary surgeons, but note limited supervision by staff in the operating room. Research opportunities are also perceived as being limited. In contrast, Toronto trainees perceive suboptimal experience as a primary surgeon. When asked about the availability of additional resources, Toronto trainees more strongly agreed that they had ample exposure to emerging technology, such as magnetic resonance imaging and image-guided surgery, than did their colleagues in Indonesia (Table 3). Bandung trainees in particular reported limited accessibility to a surgical skills laboratory. In addition, Bandung trainees agreed that FIENS volunteers represented a valuable teaching experience for
Table 2 Educational opportunities and training Opportunity to pursue research activities The amount of supervision in the operating room by staff The amount of operative experience as primary surgeon The opportunity to see ambulatory patients The amount of formal instruction Amount of time worked per week The amount of training in ethical issues The amount of operative experience as first assistant The opportunity to present cases/rounds The opportunity to develop leadership skills
Bandung
Toronto
P
3.32 3.07 4.11 4.14 4.57 4.54 4.21 4.39 4.57 4.11
4.75 4.38 3.18 3.57 4.36 4.82 3.71 4.14 4.57 4.07
b .000 b .000 .011 .127 .287 .347 .445 .829 .896 .937
Respondents’ ratings of training opportunities experienced during residency on a 7-point Likert scale (1 = too little, 7 = too much). Mean scores are reported, with P values indicating group differences.
120
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
Table 3 Access to resources I have adequate access to technology to become comfortable with ita My access to a surgical skills laboratory (eg. operating on cadavers, models etc) isb My access to information on the Internet isb My access to books and journals isb I have had/will have an adequate opportunity to pursue training in another centera
Bandung
Toronto
P
4.32 2.79 5.39 4.86 5.32
5.89 3.57 4.86 5.04 5.21
b .000 .055 .079 .779 .813
Respondents’ ratings of availability of training resources on a 7-point Likert scale. Mean scores are reported, with P values indicating group differences. a Seven-point Likert scale where 1 = disagree and 7 = agree. b Seven-point Likert scale where 1 = too little and 7 = too much.
them (mean score 5.82 F 0.94, where 1 = disagree and 7 = agree). Compared with trainees in Toronto, those in Bandung perceived themselves to be less adequately prepared for postgraduate neurosurgical training from medical school (Table 4). In addition, the Toronto trainees perceive more well-defined training goals and better preparation for board examinations. Both sets of trainees, however, believe they will be competent to practice after completing their training. 3.1.1. Effect of level of training Analysis of the data by training level revealed no levelspecific differences among residents within each of the two training programs. However, comparing recent graduates with all current residents showed some significant differences in both programs. In Bandung, there were statistically significant differences (with large effect size) in responses in the 6 graduates compared with the current residents on 11 of 37 items surveyed. To summarize, recent graduates felt that medical school prepared them better for residency training; training in benign surgical disorders and spinal tumors was better; training in general surgery was too little as opposed to too much; the amount of surgery as the primary surgeon was better; the opportunity to develop leadership skills was better; the program had a better fear-free environment; access to written and electronic information was better; and training prepared them better for practice than current residents report. In Toronto, there were statistically significant differences in responses in the recent graduates compared with the current residents in 6 of 37 questions. To summarize, recent graduates felt they had better access to technology; better
defined objectives of training and feedback; better preparation for board examinations; a more appropriate opportunity to do research; and more appropriate access to internet information than current residents (data not shown). 3.2. Qualitative data The responses to the short answer questions were examined and overarching themes collected. In Toronto, the strengths of the program were by and large felt to be the following: the large volume and diversity of clinical case material; the large amount and good quality of didactic teaching; and the opportunity to pursue basic research. The main weaknesses were the following: the workload was felt to be too high; the presence of too many clinical fellows diluted the clinical experience of the residents; and there was not enough exposure to ambulatory care. In Bandung, the strengths of the program were by and large felt to be the following: good exposure to trauma and pediatric cases; ample opportunity to operate as the primary surgeon; and good opportunities to study abroad. The main weaknesses were the following: too few staff and too many residents; lack of subspecialty training especially in vascular and spine cases; lack of modern resources and equipment; and lack of a clear-cut curriculum and feedback.
4. Discussion 4.1. Interpretation of findings Some interesting insights come out of this study. The trainees in Bandung perceive that they receive insufficient education in neuropathology, neurology, and neuroradiology, as well as insufficient exposure to clinical cases other
Table 4 Training goals and expectations Medical school prepared me well for neurosurgery training My program has clear training goals for each year/level of training My program has pre-set objectives in each subspecialty area of neurosurgery My curriculum prepares me adequately for my board exams My progress is tracked and I am provided with feedback about my strengths and weaknesses by my supervisors When I complete my training I will feel competent to practice neurosurgery My program has a fear-free environment so that I may report problems without fear of reprisal
Bandung
Toronto
P
3.46 3.57 3.68 4.54 4.21 5.25 3.96
5.29 4.82 4.89 5.21 4.86 5.54 4.21
b .000 .003 .003 .030 .144 .267 .634
Respondents’ rating of agreement with the following statements on a 7-point Likert scale (1 = disagree and 7 = agree). Mean scores are reported, with P values indicating group differences.
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
than trauma, brain tumors, and pediatrics. One contributor to this perception may be that many patients who present to hospital with these complex conditions are not operated on for a number of reasons. Many patients live in subsistence poverty, have no resources or health insurance, and cannot afford to have elective or even urgent conditions treated. Operative backlogs and inefficiencies are often responsible for patients deteriorating or leaving hospital untreated. Others are not treated because the staff does not have the confidence or resources to tackle complex problems. Bandung residents also perceive that they receive too little operative supervision, a finding quite different from the experience reported by the Toronto trainees. The Indonesian experience may be partly explained by a higher resident-tofaculty ratio of more than 8:1 present in Bandung. In addition, this finding is consistent with the local customs and culture of medical education where close supervision of learners is not the common practice. One would think that the factors described above might result in a perceived lack of confidence in the Bandung trainees when they graduate; however, this feeling was not articulated. In fact, trainee responses in this survey suggest that these trainees, like their counterparts in Toronto, are reasonably confident they will be well trained by the time they graduate. Such confidence in clinical abilities, however, must be interpreted with caution as multiple reports in the literature have highlighted the inaccuracy of selfassessment in students and its inability to correlate with other performance measures [9,10,15,28]. In one metaanalysis, for example, Falchikov and Boud [10] showed correlation coefficients of 0.39 indicating on average, poor self-judgment of performance by students. Arguably, much of a surgical trainee’s perception of being satisfied with his/her education may be predicated on his/her expectations of what he/she will be required to do as a practicing surgeon after graduation. In fact, most graduates from Indonesian programs practice their careers in regional or general hospitals where equipment and resources are unavailable to perform microsurgical or complex procedures. As a result, the discrepancy between the perceived experience gained by these two groups of individuals during their residency and their similar levels of perceived competency are likely accounted for, at least in part, by the different nature of neurosurgical practice in their respective countries. Such findings highlight the necessity to develop an appropriate understanding of the local needs and expectations, not to mention availability of resources, when discussing surgical training in an international forum. Likewise, they demonstrate the limitations of applying western standards of training to other countries. Although different standards of training may exist, each can be adequately maintained through the setting of training objectives. As a result, formal detailed objectives and expected levels of competencies for trainees in Bandung may be set to the level of practice expected in that environment. These objectives may be quite different
121
and not comparable with those in Toronto or other international training centers. Although in both programs trainees felt that they would meet the objectives designed for that program, trainees in Toronto appeared to have greater knowledge of these benchmarks. Admittedly, we did not specifically examine either the degree to which trainees did in fact meet the objectives of their respective training programs or the appropriateness of those objectives. However, opinion from recent graduates from both countries, now engaged in practice, regarding their perceived level of competency concurred with those of current residents suggesting that indeed they were well prepared to engage in independent practice. In Toronto, trainees cited the large number of fellows as a weakness in their educational experience. Because of the weaknesses of training in the developing world, many young developing world surgeons seek to improve themselves by studying as clinical fellows in centers such as Toronto. The additional layer of trainees seems to impact basic training of residents, where competition for finite educational resources can be formidable in this and other developed world surgical training programs [4]. Even within the developed world, there is an emerging trend favoring fellowship training and subspecialization [6,23]. Recent resident work-hour limitations instituted throughout many developed countries will only propagate this trend further as additional training may be seen as one way of compensating for the reported reduction in experiential learning that has resulted [5,17]. Greater energy will be needed to develop strategies to optimize residency training in both the developing and the developed worlds to ensure that these, and other competing interests, do not impact the quality of future graduates. One way of measuring the quality of any surgical curriculum is to gather input from the residents undergoing training. Residents have traditionally not had a primary role in this process. However, according to one of the principles of Knowles’ adult learning theory, adults need to be involved in the planning and evaluation of their instruction to optimize their own learning [18]. Although this theory has been challenged recently in its applicability to both undergraduate and continuing medical education [11,21], postgraduate training may represent a unique learning environment in which individuals are aware of the clinical relevance of their learning as well as of the importance of accumulating this mass of knowledge to successfully engage in independent practice. As primary stakeholders in the postgraduate education process, regular and systematic surveying of this group, as was done here, would provide important data tracking the impact of the many forces of modern-day health care on the surgical training process. 4.2. Weaknesses of this study Survey-based studies have inherent flaws that make interpretation difficult. The tripolarity of the Likert scale (ie, too little, just right, too much) that was used for most
122
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
questions can introduce complexity in analyzing the responses. In addition, time constraints on respondents may limit their ability to respond as accurately as possible. Although the results are anonymized, asymmetry of power between trainees and staff still exists in surgical residency programs in all parts of the world, and fear of consequence may impact the complete honesty of the responses by trainees. The results may also be biased by the absence of data from those who chose not to participate (approximately one third). Our analysis of results included junior residents at the PGY 1 and 2 levels. This may influence the results of the survey in that they have not yet had enough experience to reach any conclusions whether their training can meet objectives or meet desired competencies. Finally, we are limited in our ability to substantiate the survey’s results. Although day-to-day experience of several of the study’s authors as educators in either Toronto (MB, RR, SH) or Bandung (DF, MB) does support many of the findings presented here, formal evaluation of the instrument’s reliability and validity is needed before more generalized use. 5. Conclusions This study suggests the areas of need expressed by the residents and graduates from Bandung are more exposure to neuropathology, neuroradiology, peripheral nerve surgery, functional neurosurgery, vascular neurosurgery, and spinal
surgery and an increase in operative supervision. The areas of need expressed by trainees from Toronto include training in peripheral nerve surgery, operative experience as primary surgeon, and addressing or mitigating the impact of specialty training by fellows on the clinical service. Although the developed world does not have all of the problems of the developing world, there still exist numerous challenges in the developed world to the successful training of future neurosurgeons. Suffice it to say that wellintentioned residency training programs do their best, but fall short of providing the ideal pedagogical experience for their trainees, and this situation occurs across the globe. The importance of examining such programs in the context of their local environment (both medical and cultural) cannot be overemphasized. This type of study has not previously been done to our knowledge and we do not presume to suggest our findings are conclusive. This study attempted to address an important issue in surgical education and in global inequity in medicine. The findings probably do not apply universally to all residency programs and may only partially reflect the true educational challenges in the two programs studied. Hopefully, other similar studies done in the future will improve upon our methods. Acknowledgment The authors thank all of the trainees in both Bandung and Toronto who participated in this study.
Appendix A. The Questionnaire (This is the version for Bandung residents. The questionnaires for Toronto residents and for the graduates had different questions 1, 2, 3, 4, and 41.) Please do not put your name on this form. This will ensure your anonymity. 1. 2. 3. 4.
What is your level of training (or for graduates—when did you finish training)? Where did you go to medical school? What training have you had outside Bandung (or Toronto)? What training are you planning to have outside Bandung (or Toronto)?
For these questions please circle the number that best reflects your score. Please feel free to add comments in the margin. Disagree. . .1 2 3 4 5 6 7. . .Agree 5. Medical school prepared me well for neurosurgery training. z The amount of training in . . .. . . is: 6. basic sciences (eg neuroanatomy, neurophysiology) 7. neurology 8. neuroradiology 9. neuropathology 10. general surgery 11. intensive care 12. trauma surgery 13. brain tumor surgery
Too little. . .1 2 3 4 5 6 7. . .Too much
z
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
123
Too little. . .1 2 3 4 5 6 7. . .Too much 14. 15. 16. 17. 18. 19.
pediatric neurosurgery vascular neurosurgery (aneurysms, AVM’s) surgery of benign spinal disorders (eg. disc disease) spinal column and spinal cord tumors surgery of functional disorders (eg. epilepsy) peripheral nerve surgery
z Too little. . .1 2 3 4 5 6 7. . .Too much
20. The amount of operative experience as a first assistant is: 21. The amount of operative experience as the primary surgeon is: 22. The amount of supervision in the operating room by staff neurosurgeons is: 23. The opportunity I get to develop my leadership skills is: 24. The amount of class room time (ie. formal instruction) is: 25. The opportunity to present cases/rounds is: 26. The opportunity to see ambulatory patients in a clinic setting is: 27. The amount of training in ethical issues including how to communicate properly with patients is:
z Disagree. . .1 2 3 4 5 6 7. . .Agree
28. My curriculum prepares me adequately for my board exams 29. My progress is tracked and I am provided with feedback about my strengths and weaknesses by my supervisors. 30. My program has pre-set objectives in each subspecialty area of neurosurgery. 31. My program has clear training goals for each year/level of training. 32. My program has a fear-free environment so that I may report problems without fear of reprisal. 33. I have adequate access to technology to become comfortable with it (MRI, stereotaxy, image-guidance, spinal instrumentation, good microscopes, etc)
z Too little. . .1 2 3 4 5 6 7. . .Too much
34. My access to books and journals is: 35. My access to information on the Internet is: 36. I have had/will have an adequate opportunity to pursue training in another center. 37. My access to a surgical skills laboratory (eg. operating on cadavers, models etc) is: 38. The amount of time I work per week (regular + on call) is: 39. My opportunity to pursue clinical and/or laboratory research activities is: 40. 41. 42. 43. 44.
When I complete my training I will feel competent to practice neurosurgery. Teaching from FIENS volunteers is a valuable educational experience. List the 3 best strengths of your training Program. List the 3 biggest weaknesses of your training Program. List anything you would like to see improved in your training Program.
References [1] Ablin G, Fairholm DJ, Kelly DF. Report of FIENS activities. Foundation for International Education in Neurological Surgery. J Neurosurg 1999;90:986 - 7. [2] Bernstein M. Ethical dilemmas encountered while operating and teaching in a developing country. Can J Surg 2004;47:170 - 2. [3] Bernstein M. Stepping in. A Canadian neurosurgeon gets his feet wet training residents during the rainy season in Indonesia. Dr Rev 2004;22:64 - 70. [4] Bernstein M, Rutka J. Neuro-oncology fellowships in North America. J Neurooncol 1994;18:61 - 8.
z Disagree. . .1 2 3 4 5 6 7. . .Agree z Too little. . .1 2 3 4 5 6 7. . .Too much
z Disagree. . .1 2 3 4 5 6 7. . .Agree z
[5] Cohen-Gadol AA, Piepgras DG, Krishnamurthy S, Fessler RD. Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery 2005;56:398 - 403. [6] Couldwell WT, Rovit RL. Rethinking neurosurgical subspecialization. Surg Neurol 2002;58:359 - 63. [7] DaRosa DA, Folse JR, Sachdeva AK, Dunnington GL, Reznick R. Description and results of a needs assessment in preparation for the bSurgeons as educatorsQ course. Am J Surg 1995;169:410 - 3. [8] Dormans JP. Orthopaedic surgery in the developing world—can orthopaedic residents help? J Bone Joint Surg Am 2002;84-A: 1086 - 94.
124
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
[9] Eva KW, Cunnington JP, Reiter HI, Keane DR, Norman GR. How can I know what I don’t know? Poor self assessment in a well-defined domain. Adv Health Sci Educ Theory Pract 2004;9:211 - 24. [10] Falchikov N, Boud D. Student self assessment in higher education. Rev Educ Res 1989;59:395 - 430. [11] Fox RD, Harvill LM. Self-assessment of need, relevance and motivation to learn as indicators of participation in continuing medical education. Med Educ 1984;18:275 - 81. [12] Ghani KR, Cotton MH, Paterson-Brown S. Surgical training in the developing world. BMJ 2003;326:S1. [13] Girgis A, Sanson-Fisher RW, McCarthy WH. Communicating with patients: surgeons’ perceptions of their skills and need for training. Aust N Z J Surg 1997;67:775 - 80. [14] Goodacre TE. Plastic surgery in a rural African hospital: spectrum and implications. Ann R Coll Surg Engl 1986;68:42 - 4. [15] Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med 1991;66:762 - 9. [16] Hannon FB. A national medical education needs’ assessment of interns and the development of an intern education and training programme. Med Educ 2000;34:275 - 84. [17] Jarman BT, Miller MR, Brown RS, Armen SB, Bozaan AG, Ho GT, et al. The 80-hour work week: will we have less-experienced graduating surgeons? Curr Surg 2004;61:612 - 5. [18] Knowles MS. The modern practice of adult education; from andragogy to pedagogy. Englewood Cliffs (NJ)7 Cambridge Adult Education; 1980. [19] Lockyer J. Getting started with needs assessment: Part 1—The questionnaire. J Contin Educ Health Prof 1998;18:58 - 61. [20] Manktelow A. Burn injury and management in Liberia. Burns 1990;16:432 - 6. [21] Norman GR. The adult learner: a mythical species. Acad Med 1999;74:886 - 9. [22] Ratnapalan S, Hilliard RI. Needs assessment in postgraduate medical education: a review. Med Educ Online [serial online] 2002;7:8 [accessed February 23, 2005 (Available at: http://www.med-edonline.org/pdf/f0000040.pdf)]. [23] Sarmiento A. Subspecialization in orthopaedics. Has it been all for the better? J Bone Joint Surg Am 2003;85-A:369 - 73. [24] Silva JF. The urgent need to train orthopaedic surgeons in Third World countries. Med Educ 1979;13:28 - 30. [25] Steiner AK. Surgery and training in surgery in remote rural hospitals. East Afr J Med 1996;73:830 - 1. [26] Tanaka D, Kobayashi M, Chiyokura H, Nakajima T, Fujino T. Webbased educational tool for cleft lip repair using XVL. Stud Health Technol Inform 2001;81:485 - 91. [27] Udwadia TE, Udwadia RT, Menon K, Kaul P, Kukreja L, Jain R, et al. Laparoscopic surgery in the developing world. An overview of the Indian scene. Int Surg 1995;80:371 - 5. [28] Ward M, MacRae H, Schlachta C, Mamazza J, Poulin E, Reznick R, et al. Resident self-assessment of operative performance. Am J Surg 2003;185:521 - 4. [29] Yin RK. Case study research. Design and methods. 2nd ed. Thousand Oaks7 SAGE Publications; 1994.
Commentary The authors should be congratulated for having attempted to compare two different training programs, the first from Canada and the second from Indonesia. In many aspects, the conclusion of the study could be anticipated due to some bias of the study, which the authors also recognize. In fact, the study is based on the self-perception of the trainees, which probably is not the best criterion to evaluate the effectiveness of a training system. Beside the known
limits of any autoevaluation, the objective analysis of the trainees in this study was hampered—at least for what concerns the Indonesian group—by the fact that the Western type of education is more appealing for its technological advances than an education less sophisticated (but maybe more adequate for the social environment in which the neurosurgeon is called to operate). A typical example of such a fact is the complaint of the Indonesian trainees of an excessive exposure to pediatric problems. This is a surprising comment as the percentage of the population under 16 years of life in Indonesia is significantly high, and the future of this country is more based on an adequate care of this population subgroup than of the older counterpart. In other words, the training system in Bandung is just offering what an Indonesian trainee should ask for. Such an observation introduces the only criticism that I can make to the authors, that is, the mere assumption of the limited value of the autoevaluation without further discussing its implications. Maybe the study could have been more exhaustive if even the goals of the people responsible for the teaching program would have been taken into account. Regardless, the paper has value as it gives some insight on the education of the neurosurgeon in fieri. Concezio Di Rocco, MD Istituto di Neurochirurgia Universita` Cattolica del Sacro Cuore 00168 Rome, Italy
This is a well-presented and, within its limits, welldesigned study assessing differences in training, from the trainee’s point of view, in two very different neurosurgical centers: one in the developed and one in the developing world, the latter in an area with a desperate shortage of neurosurgeons where commendable efforts are being made to train as many as possible. Interesting differences were shown, with trainees in Toronto noting insufficient experience as the operating surgeon and in seeing ambulatory patients, whereas those in Bandung suffered from less hands on supervision and less experience in vascular and spinal surgery. As noted, some of these differences may be related to cultural factors and to the reality of neurosurgical practice in the developing world, where none but the large training centers or large private hospitals have facilities for complex neurovascular and spinal procedures. Both groups felt that at the end of their training, they had sufficient knowledge and expertise to carry on in practice. It would have been interesting to see separately the opinion of those Bandung residents who had been able, during or after their training, to do a fellowship in the Toronto unit or elsewhere in the developed world. They may not have shared this opinion, but again, the training in Bandung is within limits sufficient for the environment in which they will practice. It is interesting that the graduates from
Education and Training
Needs assessment of neurosurgery trainees: a survey study of two large training programs in the developing and developed worldsB Mark Bernstein, MD, MHSc, FRCSCa,4, Stanley J. Hamstra, PhDb, Sarah Woodrow, MD, MEda,b, Shaila Goldsmana, Richard K. Reznick, MD, MEd, FRCSCa,b, David Fairholm, MD, FRCSCc a
b
Department of Surgery, University of Toronto, Toronto, Canada M5G 1L5 Wilson Centre for Research in Education, University of Toronto, Toronto, Canada M5G 2C4 c Department of Surgery, University of British Columbia, Vancouver, Canada V5Z 1L7 Received 9 August 2005; accepted 9 December 2005
Abstract
Background: There are challenges facing surgical education in both the developing and the developed worlds. Few studies have examined trainee perceptions of their educational needs in a systematic way. We undertook a study to examine this issue, focusing on two large training programs, one in the developed world and one in the developing world. Methods: Neurosurgical trainees at the University of Toronto, Toronto, Canada, and at Hasan Sadikin Hospital in Bandung, Indonesia, were surveyed with a comprehensive questionnaire assessing both the content and the methods of their training. The questionnaire had 37 quantitative questions requesting responses on a 7-point Likert scale and three open-ended questions to give more qualitative data. Results: Sixty-four percent of all trainees responded. A number of interesting findings about the strengths and weaknesses of training emerged. For example, Bandung trainees felt they had excellent training in trauma but not in specialty areas, especially spine and vascular, with ample opportunity to operate as the primary surgeon. Toronto trainees felt that the volume and the variety of cases were excellent but they did not have enough ambulatory experience, and that they had suboptimal experience as the primary surgeon. Trainees in both centers agree that they will feel competent to practice neurosurgery upon completion of their training. Conclusion: This study defined different educational needs for neurosurgical trainees in two centers that reflect both their individual training environments and the local culture of medicine. As such, trainees’ perceptions of these needs represent an important adjunct to program evaluation. D 2006 Elsevier Inc. All rights reserved.
Keywords:
Developed world; Developing world; Needs assessment; Surgical education
1. Introduction Challenges to training surgeons exist in the developing and the developed worlds. To focus initially on the developing world, it is relatively underserviced regarding specialty surgical care. For example, Indonesia has 240 000 000 people and only about 90 neurosurgeons,
B
Dr Woodrow was a recipient of the Royal College of Physicians and Surgeons of Canada’s Fellowship in Medical Education. 4 Corresponding author. Toronto Western Hospital, Toronto, ON, Canada M5T 2S8. Tel.: +1 416 603 6499; fax: +1 416 603 5298. E-mail address: [email protected] (M. Bernstein). 0090-3019/$ – see front matter D 2006 Elsevier Inc. All rights reserved. doi:10.1016/j.surneu.2005.12.033
compared with Canada, which has about 180 neurosurgeons for just more than 30 000 000 people. Furthermore, in general, these specialist surgeons are not as well supported with resources as those in the developed world. Bandung has the largest neurosurgery training program in Indonesia with 42 residents. Smaller training programs are in Jakarta and Surabaya. In Bandung, there are 5 staff to train the 42 residents, whereas in Toronto, there are 24 staff and 32 residents. This discrepancy in educatortrainee ratio would be expected to produce significant challenges in the Bandung training program. One way Bandung has tried to help cope with resident training is to welcome temporary visitors from the developed world
118
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
under the auspices of The Foundation for International Education in Neurological Surgery (FIENS) [1]. Between two of the authors (DF and MB), about 8 person-months of on-site training has been committed to the residents in Bandung over the last 3 years [2,3]. After these visits, we became curious about the perception that residents in these two diverse training programs had of their respective training programs. We speculated that these questions might be addressed with a formal needs assessment survey. Little rigorous study of training of surgical residents in the developing world is available, and any literature is observational and anecdotal [8,12,14,20,24 -27]. Likewise, in the developed world, there still exist numerous challenges to the successful training of future neurosurgeons. Limited operating room resources, restricted resident work hours, a transition toward outpatient care, and increasing medicolegal concerns are just some of the many issues that surgical educators must try and balance with the educational needs of their trainees. Despite a constantly evolving healthcare environment, there is similarly a paucity of literature on needs assessment in postgraduate training in the developed world. An extensive search using MEDLINE, EMBASE, and ERIC turned up few relevant articles, including a brief review of the methodology of study of needs assessment [22] and a needs assessment of interns regarding their medical school training [16]. In the surgical literature, there is a description of a needs assessment of practicing surgeons regarding the adequacy of their training in communicating with patients [13] and of a needs assessment for surgeons in their role as educators [7]. The challenges of surgical training in the developing world might be considered self-evident to surgical educators in the developing and developed worlds. The purpose of doing this study was to investigate perceptions of training in hopes of providing useful data that could help neurosurgery programs in the developing and the developed worlds to improve the program, the training experience, and ultimately the quality of graduating neurosurgeons and hence patient care. Formal documentation of this information could also serve to enhance awareness of the main issues of importance to trainees in this and other surgical disciplines in the developing world and in the developed world.
2. Methods 2.1. Study design The study was a mixed-methods (ie, both quantitative and qualitative data), cross-sectional (ie, one point in time examined) study. It consisted of a self-administered structured questionnaire filled out anonymously by neurosurgery residents and recent graduates (Appendix A). Thirty-seven questions requested responses using a Likert scale to record the score of each response. Three additional questions were open-ended to obtain more qualitative data. Interviews were felt not to be feasible in this setting for a number of reasons,
including the perceived difficulty in recruiting a skilled interviewer (especially in Bandung) whom the trainees could trust completely with respect to confidentiality. We attempted to design a comprehensive and feasible questionnaire to ensure high compliance and to obtain the maximum amount of information possible [19]. Input on questionnaire content was received from several University of Toronto neurosurgery residents, Canadian-trained clinical fellows, and several clinical fellows (currently training at the Toronto Western Hospital) who received their postgraduate training in the developing world. 2.2. Setting and selection of participants All 42 residents currently in the neurosurgery training program in Bandung, Indonesia, located in Hasan Sadikin Hospital, Medical School of Padjadjaran University, were invited to participate, as were 7 recent graduates of this program now practicing in Indonesia. Potential participants in Toronto included all 32 current residents in the program and 6 recent graduates now practicing in Toronto. 2.3. Data collection and analysis Surveys were given to the participants with explicit verbal and written instructions by the principal author (MB) on the first day of a month-long sabbatical in Bandung, November 22, 2004, and collected before he left. Simultaneously, the questionnaire was sent by e-mail to the University of Toronto residents and recent graduates and collected over the same period. Mean responses for each question constructed on a 7-point Likert scale were calculated. Data were analyzed using commercially available software (SPSS version 11.5, SPSS Inc, Chicago, Ill). Because the data were found to be nonnormal in their distribution, Likert responses were compared between groups using the Mann-Whitney U test. Statistical significance was defined as a probability value of less than .05. A modified thematic analysis of responses to the qualitative questions was also performed [29]. 2.4. Research ethics Participation was voluntary and informed consent was implied from each participant who returned a completed questionnaire. The protocol was approved by the Research Ethics Board at The University Health Network. The Research Ethics Board at Hasan Sadikin Hospital in Bandung gave verbal approval, stating it did not need to formally review the protocol because there was no research on patients. As a courtesy, permission to do the study was also obtained from the Chairmen of Neurosurgery in Bandung and in Toronto. All surveys and survey data were stored anonymously and confidentially. 3. Results Of the 87 residents and recent graduates (hereafter referred to as trainees) invited to participate, 56 participants
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
119
Table 1 Neurosurgical and nonneurosurgical training The The The The The The The The The The The The The The
amount amount amount amount amount amount amount amount amount amount amount amount amount amount
of of of of of of of of of of of of of of
training training training training training training training training training training training training training training
in in in in in in in in in in in in in in
trauma surgery vascular neurosurgery benign spinal disorders surgery of functional disorders peripheral nerve surgery pediatric neurosurgery spinal column and spinal cord tumors brain tumor surgery neurology neuropathology neuroradiology general surgery the basic sciences intensive care
Bandung
Toronto
P
5.93 2.14 3.36 1.82 1.68 4.68 3.11 4.25 3.36 2.79 3.25 4.36 4.07 3.75
3.96 3.50 4.54 3.04 2.59 4.11 3.71 4.50 4.43 4.43 4.04 5.07 3.64 4.00
b .000 .001 .001 .001 .001 .053 .104 .958 b .000 b .000 .015 .043 .221 .329
Respondents’ ratings of the amount of training received in each of the following subspecialties on a 7-point Likert scale (1 = too little, 7 = too much). Mean scores are reported, with P values indicating differences between the groups.
completed the survey (overall response rate 64%). Of the Bandung study group, 28 of 49 responded (response rate 57%), whereas 28 of 38 potential respondents from Toronto participated (response rate 74%). Level of training was broken down as follows. In Bandung, there were 5 junior residents (postgraduate year [PGY] 1 and 2), 9 mid-level residents (PGY 3 and 4), 8 senior residents (PGY 5 and 6), and 6 recent graduates. In Toronto, there were 9 junior residents (PGY 1 and 2), 10 mid-level residents (PGY 3 and 4), 6 senior residents (PGY 5 and 6), and 3 recent graduates. Overall, the demographics of the respondents were representative of the trainee pool in each center in terms of both level of training and sex. 3.1. Quantitative data Trainees were asked to rate their exposure to different aspects of both neurosurgical and related training. The mean values for the Likert scale responses for each group are reported in Table 1, along with P values indicating differences between the groups. In general, Toronto trainees perceived receiving appropriate amounts of training in most of the 14 specialty areas examined, although deficits were most strongly noted in peripheral nerve and functional neurosurgery with an excess of time devoted to general
surgical training. In contrast, trainees in Bandung were more likely to perceive gaps in their training with excessive exposure in trauma and pediatric neurosurgery noted. Statistically significant differences between the Bandung and Toronto groups were observed for 9 of 14 items. In examining more general learning opportunities such as operative experience, formal instruction received, and opportunity to see ambulatory patients, fewer differences were noted in the perception of the two groups (Table 2). Despite reporting limited amounts of training in Table 1, trainees in Bandung believe that they experience appropriate amounts of training as primary surgeons, but note limited supervision by staff in the operating room. Research opportunities are also perceived as being limited. In contrast, Toronto trainees perceive suboptimal experience as a primary surgeon. When asked about the availability of additional resources, Toronto trainees more strongly agreed that they had ample exposure to emerging technology, such as magnetic resonance imaging and image-guided surgery, than did their colleagues in Indonesia (Table 3). Bandung trainees in particular reported limited accessibility to a surgical skills laboratory. In addition, Bandung trainees agreed that FIENS volunteers represented a valuable teaching experience for
Table 2 Educational opportunities and training Opportunity to pursue research activities The amount of supervision in the operating room by staff The amount of operative experience as primary surgeon The opportunity to see ambulatory patients The amount of formal instruction Amount of time worked per week The amount of training in ethical issues The amount of operative experience as first assistant The opportunity to present cases/rounds The opportunity to develop leadership skills
Bandung
Toronto
P
3.32 3.07 4.11 4.14 4.57 4.54 4.21 4.39 4.57 4.11
4.75 4.38 3.18 3.57 4.36 4.82 3.71 4.14 4.57 4.07
b .000 b .000 .011 .127 .287 .347 .445 .829 .896 .937
Respondents’ ratings of training opportunities experienced during residency on a 7-point Likert scale (1 = too little, 7 = too much). Mean scores are reported, with P values indicating group differences.
120
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
Table 3 Access to resources I have adequate access to technology to become comfortable with ita My access to a surgical skills laboratory (eg. operating on cadavers, models etc) isb My access to information on the Internet isb My access to books and journals isb I have had/will have an adequate opportunity to pursue training in another centera
Bandung
Toronto
P
4.32 2.79 5.39 4.86 5.32
5.89 3.57 4.86 5.04 5.21
b .000 .055 .079 .779 .813
Respondents’ ratings of availability of training resources on a 7-point Likert scale. Mean scores are reported, with P values indicating group differences. a Seven-point Likert scale where 1 = disagree and 7 = agree. b Seven-point Likert scale where 1 = too little and 7 = too much.
them (mean score 5.82 F 0.94, where 1 = disagree and 7 = agree). Compared with trainees in Toronto, those in Bandung perceived themselves to be less adequately prepared for postgraduate neurosurgical training from medical school (Table 4). In addition, the Toronto trainees perceive more well-defined training goals and better preparation for board examinations. Both sets of trainees, however, believe they will be competent to practice after completing their training. 3.1.1. Effect of level of training Analysis of the data by training level revealed no levelspecific differences among residents within each of the two training programs. However, comparing recent graduates with all current residents showed some significant differences in both programs. In Bandung, there were statistically significant differences (with large effect size) in responses in the 6 graduates compared with the current residents on 11 of 37 items surveyed. To summarize, recent graduates felt that medical school prepared them better for residency training; training in benign surgical disorders and spinal tumors was better; training in general surgery was too little as opposed to too much; the amount of surgery as the primary surgeon was better; the opportunity to develop leadership skills was better; the program had a better fear-free environment; access to written and electronic information was better; and training prepared them better for practice than current residents report. In Toronto, there were statistically significant differences in responses in the recent graduates compared with the current residents in 6 of 37 questions. To summarize, recent graduates felt they had better access to technology; better
defined objectives of training and feedback; better preparation for board examinations; a more appropriate opportunity to do research; and more appropriate access to internet information than current residents (data not shown). 3.2. Qualitative data The responses to the short answer questions were examined and overarching themes collected. In Toronto, the strengths of the program were by and large felt to be the following: the large volume and diversity of clinical case material; the large amount and good quality of didactic teaching; and the opportunity to pursue basic research. The main weaknesses were the following: the workload was felt to be too high; the presence of too many clinical fellows diluted the clinical experience of the residents; and there was not enough exposure to ambulatory care. In Bandung, the strengths of the program were by and large felt to be the following: good exposure to trauma and pediatric cases; ample opportunity to operate as the primary surgeon; and good opportunities to study abroad. The main weaknesses were the following: too few staff and too many residents; lack of subspecialty training especially in vascular and spine cases; lack of modern resources and equipment; and lack of a clear-cut curriculum and feedback.
4. Discussion 4.1. Interpretation of findings Some interesting insights come out of this study. The trainees in Bandung perceive that they receive insufficient education in neuropathology, neurology, and neuroradiology, as well as insufficient exposure to clinical cases other
Table 4 Training goals and expectations Medical school prepared me well for neurosurgery training My program has clear training goals for each year/level of training My program has pre-set objectives in each subspecialty area of neurosurgery My curriculum prepares me adequately for my board exams My progress is tracked and I am provided with feedback about my strengths and weaknesses by my supervisors When I complete my training I will feel competent to practice neurosurgery My program has a fear-free environment so that I may report problems without fear of reprisal
Bandung
Toronto
P
3.46 3.57 3.68 4.54 4.21 5.25 3.96
5.29 4.82 4.89 5.21 4.86 5.54 4.21
b .000 .003 .003 .030 .144 .267 .634
Respondents’ rating of agreement with the following statements on a 7-point Likert scale (1 = disagree and 7 = agree). Mean scores are reported, with P values indicating group differences.
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
than trauma, brain tumors, and pediatrics. One contributor to this perception may be that many patients who present to hospital with these complex conditions are not operated on for a number of reasons. Many patients live in subsistence poverty, have no resources or health insurance, and cannot afford to have elective or even urgent conditions treated. Operative backlogs and inefficiencies are often responsible for patients deteriorating or leaving hospital untreated. Others are not treated because the staff does not have the confidence or resources to tackle complex problems. Bandung residents also perceive that they receive too little operative supervision, a finding quite different from the experience reported by the Toronto trainees. The Indonesian experience may be partly explained by a higher resident-tofaculty ratio of more than 8:1 present in Bandung. In addition, this finding is consistent with the local customs and culture of medical education where close supervision of learners is not the common practice. One would think that the factors described above might result in a perceived lack of confidence in the Bandung trainees when they graduate; however, this feeling was not articulated. In fact, trainee responses in this survey suggest that these trainees, like their counterparts in Toronto, are reasonably confident they will be well trained by the time they graduate. Such confidence in clinical abilities, however, must be interpreted with caution as multiple reports in the literature have highlighted the inaccuracy of selfassessment in students and its inability to correlate with other performance measures [9,10,15,28]. In one metaanalysis, for example, Falchikov and Boud [10] showed correlation coefficients of 0.39 indicating on average, poor self-judgment of performance by students. Arguably, much of a surgical trainee’s perception of being satisfied with his/her education may be predicated on his/her expectations of what he/she will be required to do as a practicing surgeon after graduation. In fact, most graduates from Indonesian programs practice their careers in regional or general hospitals where equipment and resources are unavailable to perform microsurgical or complex procedures. As a result, the discrepancy between the perceived experience gained by these two groups of individuals during their residency and their similar levels of perceived competency are likely accounted for, at least in part, by the different nature of neurosurgical practice in their respective countries. Such findings highlight the necessity to develop an appropriate understanding of the local needs and expectations, not to mention availability of resources, when discussing surgical training in an international forum. Likewise, they demonstrate the limitations of applying western standards of training to other countries. Although different standards of training may exist, each can be adequately maintained through the setting of training objectives. As a result, formal detailed objectives and expected levels of competencies for trainees in Bandung may be set to the level of practice expected in that environment. These objectives may be quite different
121
and not comparable with those in Toronto or other international training centers. Although in both programs trainees felt that they would meet the objectives designed for that program, trainees in Toronto appeared to have greater knowledge of these benchmarks. Admittedly, we did not specifically examine either the degree to which trainees did in fact meet the objectives of their respective training programs or the appropriateness of those objectives. However, opinion from recent graduates from both countries, now engaged in practice, regarding their perceived level of competency concurred with those of current residents suggesting that indeed they were well prepared to engage in independent practice. In Toronto, trainees cited the large number of fellows as a weakness in their educational experience. Because of the weaknesses of training in the developing world, many young developing world surgeons seek to improve themselves by studying as clinical fellows in centers such as Toronto. The additional layer of trainees seems to impact basic training of residents, where competition for finite educational resources can be formidable in this and other developed world surgical training programs [4]. Even within the developed world, there is an emerging trend favoring fellowship training and subspecialization [6,23]. Recent resident work-hour limitations instituted throughout many developed countries will only propagate this trend further as additional training may be seen as one way of compensating for the reported reduction in experiential learning that has resulted [5,17]. Greater energy will be needed to develop strategies to optimize residency training in both the developing and the developed worlds to ensure that these, and other competing interests, do not impact the quality of future graduates. One way of measuring the quality of any surgical curriculum is to gather input from the residents undergoing training. Residents have traditionally not had a primary role in this process. However, according to one of the principles of Knowles’ adult learning theory, adults need to be involved in the planning and evaluation of their instruction to optimize their own learning [18]. Although this theory has been challenged recently in its applicability to both undergraduate and continuing medical education [11,21], postgraduate training may represent a unique learning environment in which individuals are aware of the clinical relevance of their learning as well as of the importance of accumulating this mass of knowledge to successfully engage in independent practice. As primary stakeholders in the postgraduate education process, regular and systematic surveying of this group, as was done here, would provide important data tracking the impact of the many forces of modern-day health care on the surgical training process. 4.2. Weaknesses of this study Survey-based studies have inherent flaws that make interpretation difficult. The tripolarity of the Likert scale (ie, too little, just right, too much) that was used for most
122
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
questions can introduce complexity in analyzing the responses. In addition, time constraints on respondents may limit their ability to respond as accurately as possible. Although the results are anonymized, asymmetry of power between trainees and staff still exists in surgical residency programs in all parts of the world, and fear of consequence may impact the complete honesty of the responses by trainees. The results may also be biased by the absence of data from those who chose not to participate (approximately one third). Our analysis of results included junior residents at the PGY 1 and 2 levels. This may influence the results of the survey in that they have not yet had enough experience to reach any conclusions whether their training can meet objectives or meet desired competencies. Finally, we are limited in our ability to substantiate the survey’s results. Although day-to-day experience of several of the study’s authors as educators in either Toronto (MB, RR, SH) or Bandung (DF, MB) does support many of the findings presented here, formal evaluation of the instrument’s reliability and validity is needed before more generalized use. 5. Conclusions This study suggests the areas of need expressed by the residents and graduates from Bandung are more exposure to neuropathology, neuroradiology, peripheral nerve surgery, functional neurosurgery, vascular neurosurgery, and spinal
surgery and an increase in operative supervision. The areas of need expressed by trainees from Toronto include training in peripheral nerve surgery, operative experience as primary surgeon, and addressing or mitigating the impact of specialty training by fellows on the clinical service. Although the developed world does not have all of the problems of the developing world, there still exist numerous challenges in the developed world to the successful training of future neurosurgeons. Suffice it to say that wellintentioned residency training programs do their best, but fall short of providing the ideal pedagogical experience for their trainees, and this situation occurs across the globe. The importance of examining such programs in the context of their local environment (both medical and cultural) cannot be overemphasized. This type of study has not previously been done to our knowledge and we do not presume to suggest our findings are conclusive. This study attempted to address an important issue in surgical education and in global inequity in medicine. The findings probably do not apply universally to all residency programs and may only partially reflect the true educational challenges in the two programs studied. Hopefully, other similar studies done in the future will improve upon our methods. Acknowledgment The authors thank all of the trainees in both Bandung and Toronto who participated in this study.
Appendix A. The Questionnaire (This is the version for Bandung residents. The questionnaires for Toronto residents and for the graduates had different questions 1, 2, 3, 4, and 41.) Please do not put your name on this form. This will ensure your anonymity. 1. 2. 3. 4.
What is your level of training (or for graduates—when did you finish training)? Where did you go to medical school? What training have you had outside Bandung (or Toronto)? What training are you planning to have outside Bandung (or Toronto)?
For these questions please circle the number that best reflects your score. Please feel free to add comments in the margin. Disagree. . .1 2 3 4 5 6 7. . .Agree 5. Medical school prepared me well for neurosurgery training. z The amount of training in . . .. . . is: 6. basic sciences (eg neuroanatomy, neurophysiology) 7. neurology 8. neuroradiology 9. neuropathology 10. general surgery 11. intensive care 12. trauma surgery 13. brain tumor surgery
Too little. . .1 2 3 4 5 6 7. . .Too much
z
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
123
Too little. . .1 2 3 4 5 6 7. . .Too much 14. 15. 16. 17. 18. 19.
pediatric neurosurgery vascular neurosurgery (aneurysms, AVM’s) surgery of benign spinal disorders (eg. disc disease) spinal column and spinal cord tumors surgery of functional disorders (eg. epilepsy) peripheral nerve surgery
z Too little. . .1 2 3 4 5 6 7. . .Too much
20. The amount of operative experience as a first assistant is: 21. The amount of operative experience as the primary surgeon is: 22. The amount of supervision in the operating room by staff neurosurgeons is: 23. The opportunity I get to develop my leadership skills is: 24. The amount of class room time (ie. formal instruction) is: 25. The opportunity to present cases/rounds is: 26. The opportunity to see ambulatory patients in a clinic setting is: 27. The amount of training in ethical issues including how to communicate properly with patients is:
z Disagree. . .1 2 3 4 5 6 7. . .Agree
28. My curriculum prepares me adequately for my board exams 29. My progress is tracked and I am provided with feedback about my strengths and weaknesses by my supervisors. 30. My program has pre-set objectives in each subspecialty area of neurosurgery. 31. My program has clear training goals for each year/level of training. 32. My program has a fear-free environment so that I may report problems without fear of reprisal. 33. I have adequate access to technology to become comfortable with it (MRI, stereotaxy, image-guidance, spinal instrumentation, good microscopes, etc)
z Too little. . .1 2 3 4 5 6 7. . .Too much
34. My access to books and journals is: 35. My access to information on the Internet is: 36. I have had/will have an adequate opportunity to pursue training in another center. 37. My access to a surgical skills laboratory (eg. operating on cadavers, models etc) is: 38. The amount of time I work per week (regular + on call) is: 39. My opportunity to pursue clinical and/or laboratory research activities is: 40. 41. 42. 43. 44.
When I complete my training I will feel competent to practice neurosurgery. Teaching from FIENS volunteers is a valuable educational experience. List the 3 best strengths of your training Program. List the 3 biggest weaknesses of your training Program. List anything you would like to see improved in your training Program.
References [1] Ablin G, Fairholm DJ, Kelly DF. Report of FIENS activities. Foundation for International Education in Neurological Surgery. J Neurosurg 1999;90:986 - 7. [2] Bernstein M. Ethical dilemmas encountered while operating and teaching in a developing country. Can J Surg 2004;47:170 - 2. [3] Bernstein M. Stepping in. A Canadian neurosurgeon gets his feet wet training residents during the rainy season in Indonesia. Dr Rev 2004;22:64 - 70. [4] Bernstein M, Rutka J. Neuro-oncology fellowships in North America. J Neurooncol 1994;18:61 - 8.
z Disagree. . .1 2 3 4 5 6 7. . .Agree z Too little. . .1 2 3 4 5 6 7. . .Too much
z Disagree. . .1 2 3 4 5 6 7. . .Agree z
[5] Cohen-Gadol AA, Piepgras DG, Krishnamurthy S, Fessler RD. Resident duty hours reform: results of a national survey of the program directors and residents in neurosurgery training programs. Neurosurgery 2005;56:398 - 403. [6] Couldwell WT, Rovit RL. Rethinking neurosurgical subspecialization. Surg Neurol 2002;58:359 - 63. [7] DaRosa DA, Folse JR, Sachdeva AK, Dunnington GL, Reznick R. Description and results of a needs assessment in preparation for the bSurgeons as educatorsQ course. Am J Surg 1995;169:410 - 3. [8] Dormans JP. Orthopaedic surgery in the developing world—can orthopaedic residents help? J Bone Joint Surg Am 2002;84-A: 1086 - 94.
124
M. Bernstein et al. / Surgical Neurology 66 (2006) 117–126
[9] Eva KW, Cunnington JP, Reiter HI, Keane DR, Norman GR. How can I know what I don’t know? Poor self assessment in a well-defined domain. Adv Health Sci Educ Theory Pract 2004;9:211 - 24. [10] Falchikov N, Boud D. Student self assessment in higher education. Rev Educ Res 1989;59:395 - 430. [11] Fox RD, Harvill LM. Self-assessment of need, relevance and motivation to learn as indicators of participation in continuing medical education. Med Educ 1984;18:275 - 81. [12] Ghani KR, Cotton MH, Paterson-Brown S. Surgical training in the developing world. BMJ 2003;326:S1. [13] Girgis A, Sanson-Fisher RW, McCarthy WH. Communicating with patients: surgeons’ perceptions of their skills and need for training. Aust N Z J Surg 1997;67:775 - 80. [14] Goodacre TE. Plastic surgery in a rural African hospital: spectrum and implications. Ann R Coll Surg Engl 1986;68:42 - 4. [15] Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med 1991;66:762 - 9. [16] Hannon FB. A national medical education needs’ assessment of interns and the development of an intern education and training programme. Med Educ 2000;34:275 - 84. [17] Jarman BT, Miller MR, Brown RS, Armen SB, Bozaan AG, Ho GT, et al. The 80-hour work week: will we have less-experienced graduating surgeons? Curr Surg 2004;61:612 - 5. [18] Knowles MS. The modern practice of adult education; from andragogy to pedagogy. Englewood Cliffs (NJ)7 Cambridge Adult Education; 1980. [19] Lockyer J. Getting started with needs assessment: Part 1—The questionnaire. J Contin Educ Health Prof 1998;18:58 - 61. [20] Manktelow A. Burn injury and management in Liberia. Burns 1990;16:432 - 6. [21] Norman GR. The adult learner: a mythical species. Acad Med 1999;74:886 - 9. [22] Ratnapalan S, Hilliard RI. Needs assessment in postgraduate medical education: a review. Med Educ Online [serial online] 2002;7:8 [accessed February 23, 2005 (Available at: http://www.med-edonline.org/pdf/f0000040.pdf)]. [23] Sarmiento A. Subspecialization in orthopaedics. Has it been all for the better? J Bone Joint Surg Am 2003;85-A:369 - 73. [24] Silva JF. The urgent need to train orthopaedic surgeons in Third World countries. Med Educ 1979;13:28 - 30. [25] Steiner AK. Surgery and training in surgery in remote rural hospitals. East Afr J Med 1996;73:830 - 1. [26] Tanaka D, Kobayashi M, Chiyokura H, Nakajima T, Fujino T. Webbased educational tool for cleft lip repair using XVL. Stud Health Technol Inform 2001;81:485 - 91. [27] Udwadia TE, Udwadia RT, Menon K, Kaul P, Kukreja L, Jain R, et al. Laparoscopic surgery in the developing world. An overview of the Indian scene. Int Surg 1995;80:371 - 5. [28] Ward M, MacRae H, Schlachta C, Mamazza J, Poulin E, Reznick R, et al. Resident self-assessment of operative performance. Am J Surg 2003;185:521 - 4. [29] Yin RK. Case study research. Design and methods. 2nd ed. Thousand Oaks7 SAGE Publications; 1994.
Commentary The authors should be congratulated for having attempted to compare two different training programs, the first from Canada and the second from Indonesia. In many aspects, the conclusion of the study could be anticipated due to some bias of the study, which the authors also recognize. In fact, the study is based on the self-perception of the trainees, which probably is not the best criterion to evaluate the effectiveness of a training system. Beside the known
limits of any autoevaluation, the objective analysis of the trainees in this study was hampered—at least for what concerns the Indonesian group—by the fact that the Western type of education is more appealing for its technological advances than an education less sophisticated (but maybe more adequate for the social environment in which the neurosurgeon is called to operate). A typical example of such a fact is the complaint of the Indonesian trainees of an excessive exposure to pediatric problems. This is a surprising comment as the percentage of the population under 16 years of life in Indonesia is significantly high, and the future of this country is more based on an adequate care of this population subgroup than of the older counterpart. In other words, the training system in Bandung is just offering what an Indonesian trainee should ask for. Such an observation introduces the only criticism that I can make to the authors, that is, the mere assumption of the limited value of the autoevaluation without further discussing its implications. Maybe the study could have been more exhaustive if even the goals of the people responsible for the teaching program would have been taken into account. Regardless, the paper has value as it gives some insight on the education of the neurosurgeon in fieri. Concezio Di Rocco, MD Istituto di Neurochirurgia Universita` Cattolica del Sacro Cuore 00168 Rome, Italy
This is a well-presented and, within its limits, welldesigned study assessing differences in training, from the trainee’s point of view, in two very different neurosurgical centers: one in the developed and one in the developing world, the latter in an area with a desperate shortage of neurosurgeons where commendable efforts are being made to train as many as possible. Interesting differences were shown, with trainees in Toronto noting insufficient experience as the operating surgeon and in seeing ambulatory patients, whereas those in Bandung suffered from less hands on supervision and less experience in vascular and spinal surgery. As noted, some of these differences may be related to cultural factors and to the reality of neurosurgical practice in the developing world, where none but the large training centers or large private hospitals have facilities for complex neurovascular and spinal procedures. Both groups felt that at the end of their training, they had sufficient knowledge and expertise to carry on in practice. It would have been interesting to see separately the opinion of those Bandung residents who had been able, during or after their training, to do a fellowship in the Toronto unit or elsewhere in the developed world. They may not have shared this opinion, but again, the training in Bandung is within limits sufficient for the environment in which they will practice. It is interesting that the graduates from