- Email: [email protected]
Neurological surgery and the assessment of accomplishment
176
Surg Neurol 1992;37:175-81
nite role in graduate medical education than it has in the past if some of the advocated reforms are to be realized. This conclusion, in turn, demands that efforts be instituted to define that role so that the first steps will be on as firm a basis as possible. In February 1968, the Council of Academic Societies voted to sponsor a national conference on The Role of the University in Graduate Medical Education. This report is an account of that conference, which was held October 2-5, 1968.
References 1. CoggeshallLT. Planning for medical progress through education. Evanston, Illinois: Association of American Medical Colleges, 1965. 2. MillisJS (chairman).The graduate education of physicians. Report of the Citizens Commissionon Graduate Medical Education. Chicago: American Medical Association, 1966.
Neurological Surgery and the Assessment of Accomplishment
Guy O. Odom, M.D.* Recognizing the need for detailed training and special qualifications for the practice of neurological surgery, representatives of the Society of Neurological Surgeons and the Harvey Cushing Society held an informal meeting on March 27, 1939. Later, this group was enlarged to include representatives from the Section on Nervous and Mental Diseases of the American Medical Association (AMA), the Section on Surgery of the AMA, the American Neurological Association, and the American College of Surgeons. The enlarged group unanimously resolved that a special board be formed for certification in neurological surgery. The American Board of Neurological Surgery was then created in accordance with the actions of the Advisory Board for Medical Specialists as approved by the AMA Council on Medical Education and Hospitals with the announced broad aims, "To encourage the study, improve the practice, elevate the standards, and advance the science of neurological surgery and thereby to serve the cause of public health." The purposes of the American Board of Neurological Surgery may be listed as follows: (a) to conduct examinations of eligible candidates who seek certification; (b) to issue certificates of qualification to all those candidates meeting the board's requirements and satisfactorily completing its examinations; and (c) to improve the opportu-
* Chairman, Division of Neurological Surgery, Duke University Medical Center, Durham, North Carolina, and Secretary, American Board of Neurological Surgery.
Smythe et al
nities, standards, and training in neurological surgery throughout the United States. The board established certain minimal requirements for an approved neurological surgery training program and the amount of time that a trainee must spend on an approved program. To be eligible for certification by the American Board o f Neurological Surgery, a candidate must have completed a 1-year training in general surgery, 4 years of training in neurosurgery, including 30 months of clinical neurosurgery of which 24 months must be in one institution, and not less than 2 years of satisfactory practice of neurological surgery in one community. At the present time there are 93 approved neurosurgical training programs in the United States and Canada. The number of hospitals offering these approved programs are as follows: 26 programs are conducted in one hospital; 33 in two hospitals; 21 in three hospitals; 12 in four hospitals; and one is based in five hospitals. O f these 93 programs, 73 have university affiliations, two are in military hospitals, one in a Veterans Administration hospital, and 17 in general hospitals. It was estimated that in 1968 there were 487 trainees on approved programs. O f these, 109 were foreign medical graduates (FMGs). Approximately 125 residents completed their training in 1967. From 1954 to 1964, an average of 55 candidates were certified each year, but for the years 1965 through 1967, the average was 78. The failure rate on the oral examination ranged from 14% to 42% per year since 1950, with an average of approximately 33% a year. Only five times in the 32 examinations given over the last 16 years was the failure rate below 20%. Members of the American Board of Neurological Surgery have become increasingly concerned about this high failure rate. It was thought that it could be due to: (a) poor examinations by the examiners, (b) difficulty of some trainees with oral examinations ("freeze-up"), (c) deficiencies in the training program, or (d) selection of inferior candidates or retention of poor trainees.
In-Training Written Examinations Everyone agreed that something should be done during the training period rather than to have the trainee rely upon a cram session before the oral board examination if the overall performance of the trainee and, therefore, his general knowledge and qualifications to practice the specialty were to be improved. After various suggestions had been considered, the board decided to conduct a written examination during the formal training period rather than at the end of the training period. Such examinations were to be designed to determine the trainee's weaknesses and the deficiencies in the program. By 1963 it was further decided that a written examination given
Graduate Medical Education
prior to the trainee's chief residency was the best instrument to determine the quality of the candidate's training as well as o f his training program. The board then made it compulsory that every trainee within 2 years o f completing his training take the in-training written examination. The board hoped that the men who already had completed their training and were in practice would take the examination also if they had not been certified. This requirement was instituted to shed some light on such questions as the following: 1. Were mediocre men being kept on training programs in order to take care of large numbers of private patients on a service or to man research projects or other extracurricular activities of the program chief but without ample opportunity for them to receive adequate, supervised clinical training or organized exposure to ancillary disciplines and basic sciences as they apply to the specialty? 2. H o w could areas of weakness in training programs be pointed out to the directors, as well as to the trainees in their programs, in time for something to be done to repair any weakness in the knowledge and skills of individual trainees before their periods of formal training were completed? To further these ends, the American Board o f Neurological Surgery created a special commission since it was felt that research into the method o f the board's examinations and other reviewers could be studied best independently of the normal board activity. The American Association of Neurological Surgeons agreed to finance the commission's activity. Rather extensive discussions ensued regarding the purpose, content, and format of the proposed in-training examination. It was decided to develop a full-day examination of written, multiplechoice questions, covering the following areas: neuroanatomy, neuropathology, neurophysiology, neuroradiology, medical neurology, general surgery, and neurosurgery. It was decided that the results of a candidate's in-training examination should not influence his acceptance at a later date for oral certifying examinations of the board. The results of these in-training examinations, therefore, have been made available only to the candidates and to their program directors. The members of the American Board of Neurological Surgery have never been informed of the trainee's score before he has taken the oral examination for certification. Since December 12, 1964, the in-training examination has been given annually. A total o f 781 trainees have taken the examination: 397 have taken two examinations, 109 have taken at least three examinations, and 23 have taken four examinations. This number is a total of 1310 written examination scores that have been accumulated and used in the study group.
Surg Neurol 1992;37:175-81
177
The National Board of Medical Examiners has processed and analyzed the results of each examination. Each trainee has been given the mean national score and his own score to compare against the national norm. H e is informed of his percentile ranking, and if his performance on any particular subtest is deficient, he is so advised. The number o f trainees taking the in-training examination each year has varied from 302 in 1966 to 362 in 1965. An average of 484 questions were included in each examination. The examination contained five subtests: neuroanatomy and neurophysiology, neuropathology, neurosurgery, clinical neurology, and neuroradiology. It was decided that at least 80 questions should be included in each subtest in order to obtain a reliable subscore for the various subjects. Only 49 items were included under neuroradiology the first year, and, therefore, this was not considered a reliable subscore. This was corrected in the following year. The total average of correct answers for all subtests has ranged from 70.5 in 1965 to 76.1 in 1967. The reliability of the examination has been excellent, varying from 0.93 in 1964 to 0.96 in 1967. The data reveal that the examination consistently and accurately has been measuring what it was intended to measure. The mean r or discriminating index of the subtests also has improved and has been a satisfactory value. These analyses indicate that the examination has been reliable and that its quality has increased from 1964 to 1967. The performance of the first-time examinees from US programs who were within 12 to 24 months of completion of their training was analyzed. The mean test scores were higher in 1966 and 1967 than they were for 1964 and 1965. It should be stressed that these examinees were first-time takers each year and not the same men repeating the examination. These results raise several questions that must be answered by future investigation: 1. Were the 1966 and 1967 examinations easier than those in 1964 and 1965? 2. Did the first two examinations have an educational effect on the training program? 3. Were the trainees of 1966 and 1967 of a better caliber? When the performance of the trainees were analyzed in relation to their level of training, it was found that with the first examination there was a significant difference in the mean scores at different levels of training. The highest score (55.5) was made during the same year as the completion of residency. Two or more years after completion of residency, the mean score dropped to 51.3, just ahead of the mean score of 50.9 for 307 US graduates taken 2 years prior to the completion of their residency.
178
Surg Neurol 1992;37:175-81
A comparison of the performance of US graduates with that of foreign graduates revealed that the mean score of the US graduates was significantly higher than the mean score for foreign graduates. The level of training of FMGs, as was the case with US graduates, appreciably affected their scores, which ranged from a low of 41.0 one year prior to completion of residency to a high of 47.7 one year after completion of residency. It must be pointed out that the size of this sample was too small to establish statistical reliability. The performance of examinees who were tested more than once revealed a significantly progressive higher performance. Fifty-six examinees took the examination three times with progressive mean scores of 47.5, 51.8, and 53.1. Another group of 224 examinees took the examination twice with scores of 49.3 and 52.8. There has been a definite correlation between the National Board examinations and the first in-training written examination. National Board examination scores were available on 126 of the trainees who took the first in-training written examination. It was found that the mean performance of these trainees was above that of the scores of the National Board candidates over the past 5 years, from 1964 to 1968. The mean score of the 126 trainees on part I was 82.4 and that of the National Board candidates was 80.6. On part II, the mean score for the trainees was 83 and that of the National Board candidates was 82.3. There was a correlation of 0.64 and 0.63, respectively, which indicates a moderately high positive relationship between these two measurements.
Smythe et al
1. The individual who does well on the written has no difficulty with the oral. Only three candidates who have been above the 70 percentile group on the written examination have failed the oral examination. Only eight candidates who were in the 5 0 - 7 0 percentile group failed the oral examination. 2. A large number of candidates who scored poorly on the written passed the oral six months later. Thirtytwo candidates who passed the oral examination were in the 50 percentile group or below on the written examination. This result seems to indicate that the candidates who are informed of their weaknesses on the written examination can take advantage of the opportunity to review these particular subjects. 3. The majority of candidates who failed the oral examination did poorly on the written. O f 32 who failed the oral examination, 21 were below 50%; eight were between 50% and 70%; and three, as previously mentioned, were between 70% and 80%. We have not as yet had sufficient time to correlate the results of the examination with the various training programs, but this study is under way. In the study group approximately one third of the trainees have had straight surgical internships, and about 30% of these have continued in the same institution for residency training. It is now a board requirement that an individual have a full year of surgical training. Until 1964 a trainee could be accepted in another surgical training program with 1 year of a rotating internship and 6 months of general surgery.
Oral Examinations
Summary and Conclusion
In the opinion of Dr. Edithe J. Levit and the staff of the National Board of Medical Examiners, current data of the American Board of Neurological Surgery do not permit a correlation study between the performance on the in-training written examination and that on the oral examination. O f the 781 trainees in the study group, 134 have taken their first oral examination within 3 years of the completion of their residency. O f these 134, only 86 who took their first written examination during residency can be included for purposes of this correlation study. It is estimated that by N o v e m b e r 1969, this group will number approximately 200 individuals and should constitute a sufficiently large sample for this correlation study. Although it cannot be supported statistically at this time, there is general consensus that the in-training written examination already has ruled out the so-called freeze-up by examinees as being responsible for the failure rate on the oral examinations. An analysis of the scores of all the trainees who have taken the written and oral examinations demonstrates several facts:
The reliability of the in-training examination has been excellent. There is a significant difference in the mean score at different levels of training, the highest being achieved during the year of completion of residency. Performance of US graduates has been consistently higher than that of foreign graduates. Progressively higher scores were obtained with repeated examinations. There was a definite correlation between the National Board scores and scores on the in-training written examination. There seems to be a correlation between the intraining written examination and the oral examination, although this result has not been verified by the National Board of Medical Examiners. It is my impression that the in-training written examination also has proved to be highly valuable to the trainee and to the program directors, and there is no question that the examination has decreased the failure rate on the oral examination.
Acknowledgments Grateful appreciation is due Dr. EditheJ. Levit, Dr. John P. Hubbard, Dr. Charles F. Schumacher, and the staff
Graduate Medical Education
of the National Board o f Medical Examiners for their very excellent cooperation and assistance.
Discussion Dr. Rhoades. There is the possibility that the people of the United States might join the people of Iran and certain other less developed countries and decide not to license doctors for the practice of medicine and surgery, blanketing the whole field, but to license them only to do what each was expert in. They would require certification as a license to practice in the specialty beyond handling an emergency situation in which nobody else was available. If we should find ourselves later in this situation, it would put the boards in a different posture. We can at present countenance utilizing 3 to 5 years of the best years of a man's life with the prospect that he has a 75 % or better chance of being successful in getting certified at the end of it. We can accept this because those who do not pass still have the opportunity to lead useful lives, applying what they have learned in any number of useful ways. But what if a person cannot practice at all unless he passes? Then one can hardly justify a system that places so much discriminatory power at the end of the process. There will be increasing pressures, and there have been for the 4 years of medical school, to make the point of entry the narrow place where few people can get through and make it very likely that most of those who do get through this point of entry can be successful. An examination at the end of 1 year that would effectively eliminate those candidates who were not going to make it eventually would suffice. It could be even worse with these national examinations than with the state boards. In the first place, the state boards do not fail very many people except in some of the vacation states. In the second place, if you fail in one jurisdiction, you have 49 other jurisdictions in which you can try. But if the examining boards in effect become national licensing boards, then a candidate is really in trouble if he fails.
Mr, Nattress. There are two fields or two areas of thought within the field o f evaluation today. One is the content orientation, and the other is the process orientation. The first is related more closely to the statistical analysis. The process group is looking more at the application of factual information in the actual problemsolving. It is only fair to point out that a reliability coefficient as high as you have reported here is directly related to the number of items that have been included in the examination, and also to the discrimination of these items between those people who have the answers and
Surg Neurol 1992;37:175-81
179
those people who do not have the answers. We are finding in orthopedics that to gain high reliability, we must sacrifice the validity of the examination in terms of the practice and the application of this knowledge, because to get high discrimination, we tend to test more in the periphery of the information in the field. These two philosophical areas of approach need to be kept in mind in talking about evaluation. We are using very similar terms here to connote quite different things. And we are talking from an educational or evaluation point of view. P a n e l L e d b y H e n r y G. S c h w a r t z , M . D . , Representing Neurosurgery on the Council of Academic Societies of the AAMC
Can the Desired Products of Graduate Training Programs be Defined?* Such definitions should include at least attention to such components as generalized and specialized clinical scientific competence, skill, and role in interpersonal relationships with patients and roles in society. The graduates of all advanced medical training programs need definition in attitudinal, behavioral, and core knowledge terms. Seven characteristics in common should be sought.
A medical conscience, The graduate must have a primary concern for the best interests of his patients. H e must exercise the integrity for responsible judgment and skills for the benefit of the patient and his family while recognizing his own limitations. An historicalperspective. An understanding of the historical background and evolution upon which medicine as a profession has been built and advanced provides a basis for his professional attitudes. It should be emphasized that rather than being bound by the past, awareness of past events provides him with the facility for effective development in the future. A set of skills. The graduate should possess the knowledge and skills pertinent to his specific field. In addition, he should possess scientific background of more general nature that will permit acquisition of new tools that may apply to his specialty.
Scholarship. A graduate should possess that degree of scholarship that permits continued familiarity with the * Report submitted by Henry G. Schwartz, M.D., Chairman, and Richard F. Manegold, M.D., Recorder.
180
Surg Neurol 1992;37:175-81
language of his profession, ability to analyze critically his own works and the works of others, and competence to communicate his own ideas.
Ability to solveproblems. The graduate should have the ability to apply his own knowledge for the solution of problems presented by patients, rather than relying on the rote application of familiar technical procedures. Motivation. The graduate should be motivated to continue learning and remain productive his entire career.
Social commitment. The physician has a civic and professional responsibility to society as a whole. He should be particularly cognizant of those social and environmental conditions related to the health of the community. It is recognized that many of the determinants of the foregoing qualities have their origins in childhood, in the early stages of the educational process, or in the general mores of society. Therefore, ensuring that the graduates of medical education programs possess these qualities will be as much a result of identification and selection of candidates as of the graduate training curriculum. Selecting candidates and testing for achievement in skills and knowledge are relatively easy, but the lack of appropriate methods for identifying the essential qualities of attitude and motivation is a serious problem in medical education. The Following Panel Was Led by Eben A l e x a n d e r , Jr., M . D . , R e p r e s e n t i n g Neurosurgery on the Council of Academic Societies of the AANS
How Should Graduate Medical Education be Financed?* Today, salaries are derived from hospital general funds, and no effort is made to identify or compensate for either the education or service or components of the residents' work. What are the consequences of this policy? Is this the best method for financing such educational programs? As an alternate, graduate medical education could be financed as determined by the major activity of the residents--the provision of medical service. The residents would be made members of the hospital staff, their services to patients billed for, and their salaries paid from the proceeds. A portion of the salaries of those who supervise their work would also be provided from this source. Third parties would be led to recognize the
* Report submitted by Eben Alexander,Jr., M.D., Chairman,and Philip C. Anderson, M.D., Recorder.
Smythe et al
validity of this pattern of financing. Is this the optimal method by which the university can finance graduate medical education? Is it feasible, and what would be its consequences, advantages, and disadvantages?
Disadvantages of Present Policy The disadvantages of present policy that stress provision of service include: 1. If residents are vendors of service, they may properly negotiate for better salaries. 2. If they are primarly vendors of service, the educational component of the residency must be secondary. 3. If their services bring in many dollars, there will be resistance to changing the system because of educational considerations, especially if such changes are threatening to the generation of funds by the residents. . Because of the value of the services they provide at low cost, there is little incentive to hospitals to shorten the duration of training.
Advantages of Present Policy There are also advantages that include: 1. The current system has a splendid record of providing highly skilled physicians in the various specialty fields. 2. By generating its own funds, it has placed no financial demand on the universities, and that on the hospitals is being passed on to the patients.
Other Conclusions Other conclusions reached include: 1. Changes need to be introduced in the current system. 2. It is unlikely that third parties will ever be willing to bear educational costs of residencies. 3. Any method of financing that ties education to deliw ery of service is not optimal. 4. Poorly supervised residencies in which men are "trained" by providing services to patients in whom no one else is interested remain too common. More effective quality control in graduate education is needed. 5. The possibility of federal support of the educational component of the cost of residencies was advocated by some, but there were many objections to this proposal. 6. Since the provision of a supply of physicians is in the interest of all third parties, especially the government plans, the inclusion of educational costs in the calcula-
Graduate Medical Education
tion of per diems becomes possible if the whole health insurance industry is brought into conformity with the practice. Three components of house officer effort should be so identified in order to permit cost allocation to them and provide a basis for insistence that those parties that benefit from these efforts will directly or indirectly bear their costs.
The service component. Residents clearly provide a service. In addition to those things they do directly for patients, their presence in a hospital results in more effective patient care. As such, that component of their cost that cannot be offset by fees derived from patients either directly or through third parties should be levied against all carriers of hospital insurance and other users of the hospital as a legitimate part of the general services the hospital offers.
Surg Neurol 1992;37:175-81
181
The self-education component. This is an educational cost to be borne either by the resident himself as the difference between his salary and what he might earn in practice (deferred income) or by the university and paid as a graduate training stipend. The instructional component. Students for whose education the university is responsible are the beneficiaries of the very large amount of teaching residents provide. As such, this cost should be borne directly by the university and theoretically could be passed along to the students. If the universities will assume responsibility for organizing the financing of residency training, they will be in a much better position to extend their influence over the educational component of this portion of medical education. EBEN ALEXANDER, JR., M.D.
Surg Neurol 1992;37:175-81
nite role in graduate medical education than it has in the past if some of the advocated reforms are to be realized. This conclusion, in turn, demands that efforts be instituted to define that role so that the first steps will be on as firm a basis as possible. In February 1968, the Council of Academic Societies voted to sponsor a national conference on The Role of the University in Graduate Medical Education. This report is an account of that conference, which was held October 2-5, 1968.
References 1. CoggeshallLT. Planning for medical progress through education. Evanston, Illinois: Association of American Medical Colleges, 1965. 2. MillisJS (chairman).The graduate education of physicians. Report of the Citizens Commissionon Graduate Medical Education. Chicago: American Medical Association, 1966.
Neurological Surgery and the Assessment of Accomplishment
Guy O. Odom, M.D.* Recognizing the need for detailed training and special qualifications for the practice of neurological surgery, representatives of the Society of Neurological Surgeons and the Harvey Cushing Society held an informal meeting on March 27, 1939. Later, this group was enlarged to include representatives from the Section on Nervous and Mental Diseases of the American Medical Association (AMA), the Section on Surgery of the AMA, the American Neurological Association, and the American College of Surgeons. The enlarged group unanimously resolved that a special board be formed for certification in neurological surgery. The American Board of Neurological Surgery was then created in accordance with the actions of the Advisory Board for Medical Specialists as approved by the AMA Council on Medical Education and Hospitals with the announced broad aims, "To encourage the study, improve the practice, elevate the standards, and advance the science of neurological surgery and thereby to serve the cause of public health." The purposes of the American Board of Neurological Surgery may be listed as follows: (a) to conduct examinations of eligible candidates who seek certification; (b) to issue certificates of qualification to all those candidates meeting the board's requirements and satisfactorily completing its examinations; and (c) to improve the opportu-
* Chairman, Division of Neurological Surgery, Duke University Medical Center, Durham, North Carolina, and Secretary, American Board of Neurological Surgery.
Smythe et al
nities, standards, and training in neurological surgery throughout the United States. The board established certain minimal requirements for an approved neurological surgery training program and the amount of time that a trainee must spend on an approved program. To be eligible for certification by the American Board o f Neurological Surgery, a candidate must have completed a 1-year training in general surgery, 4 years of training in neurosurgery, including 30 months of clinical neurosurgery of which 24 months must be in one institution, and not less than 2 years of satisfactory practice of neurological surgery in one community. At the present time there are 93 approved neurosurgical training programs in the United States and Canada. The number of hospitals offering these approved programs are as follows: 26 programs are conducted in one hospital; 33 in two hospitals; 21 in three hospitals; 12 in four hospitals; and one is based in five hospitals. O f these 93 programs, 73 have university affiliations, two are in military hospitals, one in a Veterans Administration hospital, and 17 in general hospitals. It was estimated that in 1968 there were 487 trainees on approved programs. O f these, 109 were foreign medical graduates (FMGs). Approximately 125 residents completed their training in 1967. From 1954 to 1964, an average of 55 candidates were certified each year, but for the years 1965 through 1967, the average was 78. The failure rate on the oral examination ranged from 14% to 42% per year since 1950, with an average of approximately 33% a year. Only five times in the 32 examinations given over the last 16 years was the failure rate below 20%. Members of the American Board of Neurological Surgery have become increasingly concerned about this high failure rate. It was thought that it could be due to: (a) poor examinations by the examiners, (b) difficulty of some trainees with oral examinations ("freeze-up"), (c) deficiencies in the training program, or (d) selection of inferior candidates or retention of poor trainees.
In-Training Written Examinations Everyone agreed that something should be done during the training period rather than to have the trainee rely upon a cram session before the oral board examination if the overall performance of the trainee and, therefore, his general knowledge and qualifications to practice the specialty were to be improved. After various suggestions had been considered, the board decided to conduct a written examination during the formal training period rather than at the end of the training period. Such examinations were to be designed to determine the trainee's weaknesses and the deficiencies in the program. By 1963 it was further decided that a written examination given
Graduate Medical Education
prior to the trainee's chief residency was the best instrument to determine the quality of the candidate's training as well as o f his training program. The board then made it compulsory that every trainee within 2 years o f completing his training take the in-training written examination. The board hoped that the men who already had completed their training and were in practice would take the examination also if they had not been certified. This requirement was instituted to shed some light on such questions as the following: 1. Were mediocre men being kept on training programs in order to take care of large numbers of private patients on a service or to man research projects or other extracurricular activities of the program chief but without ample opportunity for them to receive adequate, supervised clinical training or organized exposure to ancillary disciplines and basic sciences as they apply to the specialty? 2. H o w could areas of weakness in training programs be pointed out to the directors, as well as to the trainees in their programs, in time for something to be done to repair any weakness in the knowledge and skills of individual trainees before their periods of formal training were completed? To further these ends, the American Board o f Neurological Surgery created a special commission since it was felt that research into the method o f the board's examinations and other reviewers could be studied best independently of the normal board activity. The American Association of Neurological Surgeons agreed to finance the commission's activity. Rather extensive discussions ensued regarding the purpose, content, and format of the proposed in-training examination. It was decided to develop a full-day examination of written, multiplechoice questions, covering the following areas: neuroanatomy, neuropathology, neurophysiology, neuroradiology, medical neurology, general surgery, and neurosurgery. It was decided that the results of a candidate's in-training examination should not influence his acceptance at a later date for oral certifying examinations of the board. The results of these in-training examinations, therefore, have been made available only to the candidates and to their program directors. The members of the American Board of Neurological Surgery have never been informed of the trainee's score before he has taken the oral examination for certification. Since December 12, 1964, the in-training examination has been given annually. A total o f 781 trainees have taken the examination: 397 have taken two examinations, 109 have taken at least three examinations, and 23 have taken four examinations. This number is a total of 1310 written examination scores that have been accumulated and used in the study group.
Surg Neurol 1992;37:175-81
177
The National Board of Medical Examiners has processed and analyzed the results of each examination. Each trainee has been given the mean national score and his own score to compare against the national norm. H e is informed of his percentile ranking, and if his performance on any particular subtest is deficient, he is so advised. The number o f trainees taking the in-training examination each year has varied from 302 in 1966 to 362 in 1965. An average of 484 questions were included in each examination. The examination contained five subtests: neuroanatomy and neurophysiology, neuropathology, neurosurgery, clinical neurology, and neuroradiology. It was decided that at least 80 questions should be included in each subtest in order to obtain a reliable subscore for the various subjects. Only 49 items were included under neuroradiology the first year, and, therefore, this was not considered a reliable subscore. This was corrected in the following year. The total average of correct answers for all subtests has ranged from 70.5 in 1965 to 76.1 in 1967. The reliability of the examination has been excellent, varying from 0.93 in 1964 to 0.96 in 1967. The data reveal that the examination consistently and accurately has been measuring what it was intended to measure. The mean r or discriminating index of the subtests also has improved and has been a satisfactory value. These analyses indicate that the examination has been reliable and that its quality has increased from 1964 to 1967. The performance of the first-time examinees from US programs who were within 12 to 24 months of completion of their training was analyzed. The mean test scores were higher in 1966 and 1967 than they were for 1964 and 1965. It should be stressed that these examinees were first-time takers each year and not the same men repeating the examination. These results raise several questions that must be answered by future investigation: 1. Were the 1966 and 1967 examinations easier than those in 1964 and 1965? 2. Did the first two examinations have an educational effect on the training program? 3. Were the trainees of 1966 and 1967 of a better caliber? When the performance of the trainees were analyzed in relation to their level of training, it was found that with the first examination there was a significant difference in the mean scores at different levels of training. The highest score (55.5) was made during the same year as the completion of residency. Two or more years after completion of residency, the mean score dropped to 51.3, just ahead of the mean score of 50.9 for 307 US graduates taken 2 years prior to the completion of their residency.
178
Surg Neurol 1992;37:175-81
A comparison of the performance of US graduates with that of foreign graduates revealed that the mean score of the US graduates was significantly higher than the mean score for foreign graduates. The level of training of FMGs, as was the case with US graduates, appreciably affected their scores, which ranged from a low of 41.0 one year prior to completion of residency to a high of 47.7 one year after completion of residency. It must be pointed out that the size of this sample was too small to establish statistical reliability. The performance of examinees who were tested more than once revealed a significantly progressive higher performance. Fifty-six examinees took the examination three times with progressive mean scores of 47.5, 51.8, and 53.1. Another group of 224 examinees took the examination twice with scores of 49.3 and 52.8. There has been a definite correlation between the National Board examinations and the first in-training written examination. National Board examination scores were available on 126 of the trainees who took the first in-training written examination. It was found that the mean performance of these trainees was above that of the scores of the National Board candidates over the past 5 years, from 1964 to 1968. The mean score of the 126 trainees on part I was 82.4 and that of the National Board candidates was 80.6. On part II, the mean score for the trainees was 83 and that of the National Board candidates was 82.3. There was a correlation of 0.64 and 0.63, respectively, which indicates a moderately high positive relationship between these two measurements.
Smythe et al
1. The individual who does well on the written has no difficulty with the oral. Only three candidates who have been above the 70 percentile group on the written examination have failed the oral examination. Only eight candidates who were in the 5 0 - 7 0 percentile group failed the oral examination. 2. A large number of candidates who scored poorly on the written passed the oral six months later. Thirtytwo candidates who passed the oral examination were in the 50 percentile group or below on the written examination. This result seems to indicate that the candidates who are informed of their weaknesses on the written examination can take advantage of the opportunity to review these particular subjects. 3. The majority of candidates who failed the oral examination did poorly on the written. O f 32 who failed the oral examination, 21 were below 50%; eight were between 50% and 70%; and three, as previously mentioned, were between 70% and 80%. We have not as yet had sufficient time to correlate the results of the examination with the various training programs, but this study is under way. In the study group approximately one third of the trainees have had straight surgical internships, and about 30% of these have continued in the same institution for residency training. It is now a board requirement that an individual have a full year of surgical training. Until 1964 a trainee could be accepted in another surgical training program with 1 year of a rotating internship and 6 months of general surgery.
Oral Examinations
Summary and Conclusion
In the opinion of Dr. Edithe J. Levit and the staff of the National Board of Medical Examiners, current data of the American Board of Neurological Surgery do not permit a correlation study between the performance on the in-training written examination and that on the oral examination. O f the 781 trainees in the study group, 134 have taken their first oral examination within 3 years of the completion of their residency. O f these 134, only 86 who took their first written examination during residency can be included for purposes of this correlation study. It is estimated that by N o v e m b e r 1969, this group will number approximately 200 individuals and should constitute a sufficiently large sample for this correlation study. Although it cannot be supported statistically at this time, there is general consensus that the in-training written examination already has ruled out the so-called freeze-up by examinees as being responsible for the failure rate on the oral examinations. An analysis of the scores of all the trainees who have taken the written and oral examinations demonstrates several facts:
The reliability of the in-training examination has been excellent. There is a significant difference in the mean score at different levels of training, the highest being achieved during the year of completion of residency. Performance of US graduates has been consistently higher than that of foreign graduates. Progressively higher scores were obtained with repeated examinations. There was a definite correlation between the National Board scores and scores on the in-training written examination. There seems to be a correlation between the intraining written examination and the oral examination, although this result has not been verified by the National Board of Medical Examiners. It is my impression that the in-training written examination also has proved to be highly valuable to the trainee and to the program directors, and there is no question that the examination has decreased the failure rate on the oral examination.
Acknowledgments Grateful appreciation is due Dr. EditheJ. Levit, Dr. John P. Hubbard, Dr. Charles F. Schumacher, and the staff
Graduate Medical Education
of the National Board o f Medical Examiners for their very excellent cooperation and assistance.
Discussion Dr. Rhoades. There is the possibility that the people of the United States might join the people of Iran and certain other less developed countries and decide not to license doctors for the practice of medicine and surgery, blanketing the whole field, but to license them only to do what each was expert in. They would require certification as a license to practice in the specialty beyond handling an emergency situation in which nobody else was available. If we should find ourselves later in this situation, it would put the boards in a different posture. We can at present countenance utilizing 3 to 5 years of the best years of a man's life with the prospect that he has a 75 % or better chance of being successful in getting certified at the end of it. We can accept this because those who do not pass still have the opportunity to lead useful lives, applying what they have learned in any number of useful ways. But what if a person cannot practice at all unless he passes? Then one can hardly justify a system that places so much discriminatory power at the end of the process. There will be increasing pressures, and there have been for the 4 years of medical school, to make the point of entry the narrow place where few people can get through and make it very likely that most of those who do get through this point of entry can be successful. An examination at the end of 1 year that would effectively eliminate those candidates who were not going to make it eventually would suffice. It could be even worse with these national examinations than with the state boards. In the first place, the state boards do not fail very many people except in some of the vacation states. In the second place, if you fail in one jurisdiction, you have 49 other jurisdictions in which you can try. But if the examining boards in effect become national licensing boards, then a candidate is really in trouble if he fails.
Mr, Nattress. There are two fields or two areas of thought within the field o f evaluation today. One is the content orientation, and the other is the process orientation. The first is related more closely to the statistical analysis. The process group is looking more at the application of factual information in the actual problemsolving. It is only fair to point out that a reliability coefficient as high as you have reported here is directly related to the number of items that have been included in the examination, and also to the discrimination of these items between those people who have the answers and
Surg Neurol 1992;37:175-81
179
those people who do not have the answers. We are finding in orthopedics that to gain high reliability, we must sacrifice the validity of the examination in terms of the practice and the application of this knowledge, because to get high discrimination, we tend to test more in the periphery of the information in the field. These two philosophical areas of approach need to be kept in mind in talking about evaluation. We are using very similar terms here to connote quite different things. And we are talking from an educational or evaluation point of view. P a n e l L e d b y H e n r y G. S c h w a r t z , M . D . , Representing Neurosurgery on the Council of Academic Societies of the AAMC
Can the Desired Products of Graduate Training Programs be Defined?* Such definitions should include at least attention to such components as generalized and specialized clinical scientific competence, skill, and role in interpersonal relationships with patients and roles in society. The graduates of all advanced medical training programs need definition in attitudinal, behavioral, and core knowledge terms. Seven characteristics in common should be sought.
A medical conscience, The graduate must have a primary concern for the best interests of his patients. H e must exercise the integrity for responsible judgment and skills for the benefit of the patient and his family while recognizing his own limitations. An historicalperspective. An understanding of the historical background and evolution upon which medicine as a profession has been built and advanced provides a basis for his professional attitudes. It should be emphasized that rather than being bound by the past, awareness of past events provides him with the facility for effective development in the future. A set of skills. The graduate should possess the knowledge and skills pertinent to his specific field. In addition, he should possess scientific background of more general nature that will permit acquisition of new tools that may apply to his specialty.
Scholarship. A graduate should possess that degree of scholarship that permits continued familiarity with the * Report submitted by Henry G. Schwartz, M.D., Chairman, and Richard F. Manegold, M.D., Recorder.
180
Surg Neurol 1992;37:175-81
language of his profession, ability to analyze critically his own works and the works of others, and competence to communicate his own ideas.
Ability to solveproblems. The graduate should have the ability to apply his own knowledge for the solution of problems presented by patients, rather than relying on the rote application of familiar technical procedures. Motivation. The graduate should be motivated to continue learning and remain productive his entire career.
Social commitment. The physician has a civic and professional responsibility to society as a whole. He should be particularly cognizant of those social and environmental conditions related to the health of the community. It is recognized that many of the determinants of the foregoing qualities have their origins in childhood, in the early stages of the educational process, or in the general mores of society. Therefore, ensuring that the graduates of medical education programs possess these qualities will be as much a result of identification and selection of candidates as of the graduate training curriculum. Selecting candidates and testing for achievement in skills and knowledge are relatively easy, but the lack of appropriate methods for identifying the essential qualities of attitude and motivation is a serious problem in medical education. The Following Panel Was Led by Eben A l e x a n d e r , Jr., M . D . , R e p r e s e n t i n g Neurosurgery on the Council of Academic Societies of the AANS
How Should Graduate Medical Education be Financed?* Today, salaries are derived from hospital general funds, and no effort is made to identify or compensate for either the education or service or components of the residents' work. What are the consequences of this policy? Is this the best method for financing such educational programs? As an alternate, graduate medical education could be financed as determined by the major activity of the residents--the provision of medical service. The residents would be made members of the hospital staff, their services to patients billed for, and their salaries paid from the proceeds. A portion of the salaries of those who supervise their work would also be provided from this source. Third parties would be led to recognize the
* Report submitted by Eben Alexander,Jr., M.D., Chairman,and Philip C. Anderson, M.D., Recorder.
Smythe et al
validity of this pattern of financing. Is this the optimal method by which the university can finance graduate medical education? Is it feasible, and what would be its consequences, advantages, and disadvantages?
Disadvantages of Present Policy The disadvantages of present policy that stress provision of service include: 1. If residents are vendors of service, they may properly negotiate for better salaries. 2. If they are primarly vendors of service, the educational component of the residency must be secondary. 3. If their services bring in many dollars, there will be resistance to changing the system because of educational considerations, especially if such changes are threatening to the generation of funds by the residents. . Because of the value of the services they provide at low cost, there is little incentive to hospitals to shorten the duration of training.
Advantages of Present Policy There are also advantages that include: 1. The current system has a splendid record of providing highly skilled physicians in the various specialty fields. 2. By generating its own funds, it has placed no financial demand on the universities, and that on the hospitals is being passed on to the patients.
Other Conclusions Other conclusions reached include: 1. Changes need to be introduced in the current system. 2. It is unlikely that third parties will ever be willing to bear educational costs of residencies. 3. Any method of financing that ties education to deliw ery of service is not optimal. 4. Poorly supervised residencies in which men are "trained" by providing services to patients in whom no one else is interested remain too common. More effective quality control in graduate education is needed. 5. The possibility of federal support of the educational component of the cost of residencies was advocated by some, but there were many objections to this proposal. 6. Since the provision of a supply of physicians is in the interest of all third parties, especially the government plans, the inclusion of educational costs in the calcula-
Graduate Medical Education
tion of per diems becomes possible if the whole health insurance industry is brought into conformity with the practice. Three components of house officer effort should be so identified in order to permit cost allocation to them and provide a basis for insistence that those parties that benefit from these efforts will directly or indirectly bear their costs.
The service component. Residents clearly provide a service. In addition to those things they do directly for patients, their presence in a hospital results in more effective patient care. As such, that component of their cost that cannot be offset by fees derived from patients either directly or through third parties should be levied against all carriers of hospital insurance and other users of the hospital as a legitimate part of the general services the hospital offers.
Surg Neurol 1992;37:175-81
181
The self-education component. This is an educational cost to be borne either by the resident himself as the difference between his salary and what he might earn in practice (deferred income) or by the university and paid as a graduate training stipend. The instructional component. Students for whose education the university is responsible are the beneficiaries of the very large amount of teaching residents provide. As such, this cost should be borne directly by the university and theoretically could be passed along to the students. If the universities will assume responsibility for organizing the financing of residency training, they will be in a much better position to extend their influence over the educational component of this portion of medical education. EBEN ALEXANDER, JR., M.D.