- Email: [email protected]
American College of Radiology In-Training Examination for Residents in Radiation Oncology (2004–2007)
Int. J. Radiation Oncology Biol. Phys., Vol. 70, No. 3, pp. 666–670, 2008 Copyright Ó 2008 Elsevier Inc. Printed in the USA. All rights reserved 0360-3016/08/$–see front matter
doi:10.1016/j.ijrobp.2007.09.049
REPORT
AMERICAN COLLEGE OF RADIOLOGY IN-TRAINING EXAMINATION FOR RESIDENTS IN RADIATION ONCOLOGY (2004–2007) ARNOLD C. PAULINO, M.D.,* AND ELIZABETH KURTZy * Department of Radiology, Division of Radiation Oncology, Baylor College of Medicine and Methodist Hospital, Houston, TX; and y Educational Services Division, American College of Radiology, Reston, VA Purpose: To review the results of the recent American College of Radiology (ACR) in-training examinations in radiation oncology and to provide information regarding the examination changes in recent years. Methods and Materials: A retrospective review of the 2004 to 2007 ACR in-training examination was undertaken. Results: The number of residents taking the in-training examination increased from 2004 to 2007, compatible with the increase in the number of radiation oncology residents in the United States and Canada. The number of questions decreased from approximately 510 in 2004 and 2005, to 405 in 2006 and 360 in 2007, most of these changes were in the clinical oncology section. Although the in-training examination showed construct validity with resident performance improving with each year of additional clinical oncology training, it did so only until Level 3 for biology and physics. Several changes have been made to the examination process, including allowing residents to keep the examination booklet for self-study, posting of the answer key and rationales to questions on the ACR Website, and providing hard copies to residency training directors. In addition, all questions are now A type or multiple choice questions with one best answer, similar to the American Board of Radiology written examination for radiation oncology. Conclusion: Several efforts by the ACR have been made in recent years to make the examination an educational tool for radiation oncology residents and residency directors. Ó 2008 Elsevier Inc. American College of Radiology, In-training examination, Resident training, Radiation oncology residency.
an educational experience for residents and program directors. This report reviewed the past 4 years of the in-training examination and discusses some of these changes and rationale leading to the current examination.
INTRODUCTION The American College of Radiology (ACR) in-training examination for residents in radiation oncology started in 1984 and reached its 24th year of testing in 2007. The Committee on Residency Training in Radiation Oncology was given the mandate by the Commission on Education to develop the yearly examination with questions written by residency program directors, academic radiation oncologists, biologists, and physicists. The in-training examination is intended to be a tool for residency program directors to evaluate their residents’ performance in radiobiology, physics, and clinical oncology and to help analyze the strengths and weaknesses of the residency program. It was also developed to help individual residents assess their knowledge base compared with their peers of the same postgraduate year level. It can be used as a method of determining areas of deficiency in training but is not intended to be the sole method for qualification to any postgraduate program or certification (1). During the past few years, the ACR in-training examination for radiation oncology has undergone changes to make the test
METHODS AND MATERIALS The ACR in-training examination for radiation oncology is offered to residents in the United States and Canada on a voluntary basis and is given yearly in the spring. The items tested include radiation and cancer biology, medical physics, statistics, and clinical oncology. Under clinical oncology, the subject areas tested include breast, bone/soft tissue, central nervous system and eye, gastrointestinal, genitourinary, gynecologic, head and neck/skin, lymphoma/ leukemia, pediatric and thoracic tumors. Each resident who registers for the examination must sign a consent form permitting the residency program director to have knowledge of the scores. The scores are sent to the residency program director approximately 6–8 weeks after the date of the examination, with the number of correct answers and percentile ranking of their residents compared with residents from other training programs at the same postgraduate year level.
Conflict of interest: none. Received Sept 10, 2007, and in revised form Sept 28, 2007. Accepted for publication Sept 28, 2007.
Reprint requests to: Arnold C. Paulino, M.D., Department of Radiation Oncology, Methodist Hospital, 6565 Fannin St., DB1-077, Houston, TX 77030. Tel: (713) 441-4844; Fax: (713) 441-4493; E-mail: [email protected] 666
ACR in-training examination in radiation oncology d A. C. PAULINO AND E. KURTZ
667
Table 2. Number of questions in in-training examination according to year and subject area
RESULTS Number of residents and year of training Table 1 lists the number of residents who participated in the in-training examination for the past 4 years. In general, the number of residents taking the examination has increased, from a total of 517 residents in 2004 to 572 in 2007. The number of Level 4 or senior residents taking the examination has declined from 251 (23.6%) of 1,063 residents in 2004 and 2005 to 216 (19.6%) of 1,103 residents in 2006 and 2007 (p = 0.026, chi-square). Number and subject area of questions Table 2 lists the number of questions according to year of testing and subject area being tested. The total number of questions decreased from 509–511 questions in 2004–2005 to 405 in 2006 and 360 in 2007. This was primarily secondary to a decrease in clinical oncology questions from 363 in 2004 to 335 in 2005, 270 in 2006, and 225 in 2007. All clinical subject areas had a decrease in the number of questions. In the 2007 examination, two subsections were eliminated and incorporated into another subsection. The eye questions were included in the central nervous system subsection, and the skin questions were included in the head-and-neck subsection. The physics questions have also decreased from 76–78 questions in 2004–2005 to 60 in 2006 and 2007. The radiation and cancer biology questions have remained at about 60–61 questions, with the exception of 2005 when there were 77 questions. Examination mean scores Tables 3 to 5 give the mean percentage of correct answers per level of training and year of testing in biology, physics, and clinical oncology. In general, the mean scores for Level 2 residents were better than those for Level 1 residents, and, likewise, the mean scores for Level 3 residents were better than those for Level 2 residents, with the exception of the 2004 biology section, for which the scores were about the same for each level of training. In general, the results for Level 4 residents were greater than those for Level 3 residents in clinical oncology. For the 2004 and 2005 examinations, the Level 4 mean scores were about the same as the Level 3 scores for the biology and physics sections. In contrast, in 2006 and 2007, the biology and physics mean scores were better for the Level 3 residents than for the Level 4 Table 1. Number of radiation oncology residents taking the in-training examination according to year and level of training
Year (n) Section Biology Physics Statistics Clinical Bone/soft tissue Breast Central nervous system Eye Gastrointestinal Genitourinary Gynecology Head and neck Lung Lymphoma Pediatrics Skin Total
2004
2005
2006
2007
61 76 9 363 35 38 28
77 78 21 335 20 35 27
60 60 15 270 15 25 25
60 60 15 225 15 24 21
20 35 37 30 21 32 49 27 11 509
18 32 28 29 33 32 36 31 14 511
15 25 25 25 25 25 25 25 15 405
—* 24 24 24 24 24 24 21 —y 360
* In 2007, eye questions were included in the central nervous system category. y In 2007, skin questions were included in the head-and-neck category.
residents. Table 6 shows the mean percentage of correct answers for the entire examination. The scores increased according to level of training until Level 3; the scores for Level 4 residents were approximately the same as (2004, 2005, and 2007 examinations), or worse than (2006 examination), those of the Level 3 residents. Figure 1 shows the resident percentile rank of residents in relation to the percentage of correct answers in the 2006 biology, physics, and clinical examinations, and Fig. 2 shows the 2007 results. In the clinical portion of the examination for both 2006 and 2007, Level 4 residents performed at a relative percentile rank that was superior to that of Level 3 residents. In the biology and physics sections, Level 3 residents performed at a relative percentile rank that was superior to that of Level 4 residents. Hence, Level 3 residents, as a group, answered more questions correctly than did the Level 4 residents in biology and physics for the 2006 and 2007 examinations. Changes in examination in past years On the basis of the input from the Association of Residents in Radiation Oncology, several changes have been Table 3. Mean percentage of correct answers to biology section according to level of training and year of testing
Year (n) Year (%) Level
2004
2005
2006
2007
1 2 3 4 Total
119 134 141 123 517
151 128 139 128 546
148 151 121 111 531
164 151 152 105 572
Level
2004
2005
2006
2007
1 2 3 4
78.8 79.5 79.4 79.3
60.9 65.1 70.0 69.7
61.0 65.5 69.3 41.2
60.5 64.8 71.3 65.0
I. J. Radiation Oncology d Biology d Physics
668
Table 4. Mean percentage of correct answers to physics section according to level of training and year of testing
Volume 70, Number 3, 2008
Table 6. Mean percentage of correct answers to entire examination according to level of training and year of testing
Year (%)
Year
Level
2004
2005
2006
2007
Level
2004
2005
2006
2007
1 2 3 4
78.5 82.7 83.9 85.7
55.2 58.7 63.2 63.1
45.2 51.2 57.0 34.2
49.8 53.5 58.8 52.0
Level 1 Level 2 Level 3 Level 4
73.3 75.4 78.2 77.8
59.1 63.9 68.4 68.7
54.5 59.9 63.2 56.9
55.7 60.5 64.1 63.7
instituted to make the in-training examination an educational experience for residents and their program directors. In 2002, the examination booklets with all the questions were given to the residents for self-study. In 2003, the answer key was also given to the program directors and posted on the ACR Website. Starting 2004, rationales to the answers for approximately 25% of the questions were posted on the ACR Website. In 2005, the biology and physics question writers were doubled, and the questions became more diverse, with multiple topics covered. In 2006, the B type (matching) and X type (true/false) questions were eliminated. The in-training examination consisted solely of A type questions (multiple choice with four possible answers). Before 2007, questions on the in-training examination were divided into sections with a heading of biology, physics, clinical oncology, or statistics. Beginning in 2007, the questions were mixed and were not divided according to section. In addition, rationales to the answers were available for each question and were given to the program directors and posted on the ACR Website.
DISCUSSION Compared with previous reports, the number of residents taking the ACR in-training examination in radiation oncology has increased, with 572 residents taking the examination in 2007. In 1984, only 365 took the examination vs. in 1991, when 497 did so (2, 3). It has been estimated that approximately 80% of residents currently in-training take the ACR in-training examination (4). The increase in the number taking the examination is in agreement with the increase in the number of radiation oncology residents currently training in North America. The total number of questions has also recently changed. In 2004 and 2005, approximately 500 questions were Table 5. Mean percentage of correct answers to clinical section according to level of training and year of testing Year Level
2004
2005
2006
2007
1 2 3 4
71.5 75.4 77.1 77.8
59.5 64.7 68.4 70.2
54.6 60.1 62.9 65.3
55.2 60.6 63.2 66.1
included, slightly greater than the 450 questions reported for 1984 to 1987. However, in 2006 and 2007, the number of questions decreased to 405 and 360, respectively. There are two explanations for why this occurred. First, the rationales for the answers to the questions were instituted in 2004, making it more time consuming for question writers to develop the test. Second, in 2006, the B type and X type questions were eliminated, leaving only A type questions. Because the matching type and true/false format was easier for question writers to construct compared with the multiple choice format, it was decided that the number of total questions would be lessened to compensate for this change. The move to the A-type format was partly because of the current format of the American Board of Radiology (ABR) written examination, which now consists of only A-type questions. Because many program directors and residents view the intraining examination as a practice for the ABR written board examination, the move to the A-type format for the intraining examination was a logical step for the committee. Other educators have also questioned the validity of X-type questions in which the examinee has a 50:50 chance of getting the answer correct (5, 6). Although the in-training examination shows construct validity, with resident performance improving with each year of additional clinical oncology training, it does so only until Level 3 of biology and physics training. In 2004 and 2005, the Level 3 and 4 mean scores were approximately the same for both biology and physics. In contrast, for the 2006 and 2007 examination, Level 3 residents had a greater mean score than did the Level 4 residents. The one change that occurred between 2005 and 2006 was the ability of the residents to take the ABR written boards in biology and physics during the beginning of their Level 4 of residency. If the residents had passed both the biology and physics examinations, they had little motivation to do well on the in-training examination in biology and physics, which was offered about 9 months after they had passed the ABR examination. Some residents skipped or answered the same key for the biology and physics examination. Even when we mixed the biology and physics questions with the clinical questions in the 2007 examination, the Level 4 scores were still not as great as the Level 3 scores in biology and physics (Fig. 2). Recent changes in the ACR in-training examination have made the process an educational tool for both radiation oncology residents and their program directors. Although
ACR in-training examination in radiation oncology d A. C. PAULINO AND E. KURTZ
669
Fig. 2. 2007 In-training examination for radiation oncology results by resident level. (a) Biology. (b) Physics. (c) Clinical section. Fig. 1. 2006 In-training examination for radiation oncology results by resident level. (a) Biology. (b) Physics. (c) Clinical section.
in the past, the examination booklets were collected after administration of the test, the booklets are now given to the residents. In addition, the answer key and rationales for all answers are now available either by hard copy or by the Web for residents and training directors to go over in their teaching sessions. In previous surveys, 77–88% of residents
thought that the in-training examination was helpful in assessing their knowledge (4, 7). It is our hope that the ACR in-training examination for radiation oncology will continue to be a useful tool in assessing resident knowledge but will also serve as an education tool for residents in-training and their program directors.
670
I. J. Radiation Oncology d Biology d Physics
Volume 70, Number 3, 2008
REFERENCES 1. In-training examination brochure. Reston, VA: American College of Radiology; 2007. 2. Wilson JF, Diamond JJ. Summary results of the ACR experience with an in-training examination for residents in radiation oncology. Int J Radiat Oncol Biol Phys 1988;15:1219–1221. 3. Coia LR, Wilson JF, Bresch JP, et al. Results of the in-training examination of the American College of Radiology for residents in radiation oncology. Int J Radiat Oncol Biol Phys 1992;24: 903–905.
4. Patel S, Jagsi R, Wilson J, et al. Results of the 2004 Association of Residents in Radiation Oncology (ARRO) survey. Int J Radiat Oncol Biol Phys 2006;66:1199–1203. 5. Available from: http://www.nbme.org/publications/item_ writing_manual_download.html. Accessed November 20, 2007. 6. Moss E. Multiple choice questions: Their value as an assessment tool. Curr Opin Anaesthesiol 2001;14:661–666. 7. Reddy S, Vijayakumar S. Evaluating clinical skills of radiation oncology residents: Parts I and II. Int J Cancer 2000;90:1–12.
doi:10.1016/j.ijrobp.2007.09.049
REPORT
AMERICAN COLLEGE OF RADIOLOGY IN-TRAINING EXAMINATION FOR RESIDENTS IN RADIATION ONCOLOGY (2004–2007) ARNOLD C. PAULINO, M.D.,* AND ELIZABETH KURTZy * Department of Radiology, Division of Radiation Oncology, Baylor College of Medicine and Methodist Hospital, Houston, TX; and y Educational Services Division, American College of Radiology, Reston, VA Purpose: To review the results of the recent American College of Radiology (ACR) in-training examinations in radiation oncology and to provide information regarding the examination changes in recent years. Methods and Materials: A retrospective review of the 2004 to 2007 ACR in-training examination was undertaken. Results: The number of residents taking the in-training examination increased from 2004 to 2007, compatible with the increase in the number of radiation oncology residents in the United States and Canada. The number of questions decreased from approximately 510 in 2004 and 2005, to 405 in 2006 and 360 in 2007, most of these changes were in the clinical oncology section. Although the in-training examination showed construct validity with resident performance improving with each year of additional clinical oncology training, it did so only until Level 3 for biology and physics. Several changes have been made to the examination process, including allowing residents to keep the examination booklet for self-study, posting of the answer key and rationales to questions on the ACR Website, and providing hard copies to residency training directors. In addition, all questions are now A type or multiple choice questions with one best answer, similar to the American Board of Radiology written examination for radiation oncology. Conclusion: Several efforts by the ACR have been made in recent years to make the examination an educational tool for radiation oncology residents and residency directors. Ó 2008 Elsevier Inc. American College of Radiology, In-training examination, Resident training, Radiation oncology residency.
an educational experience for residents and program directors. This report reviewed the past 4 years of the in-training examination and discusses some of these changes and rationale leading to the current examination.
INTRODUCTION The American College of Radiology (ACR) in-training examination for residents in radiation oncology started in 1984 and reached its 24th year of testing in 2007. The Committee on Residency Training in Radiation Oncology was given the mandate by the Commission on Education to develop the yearly examination with questions written by residency program directors, academic radiation oncologists, biologists, and physicists. The in-training examination is intended to be a tool for residency program directors to evaluate their residents’ performance in radiobiology, physics, and clinical oncology and to help analyze the strengths and weaknesses of the residency program. It was also developed to help individual residents assess their knowledge base compared with their peers of the same postgraduate year level. It can be used as a method of determining areas of deficiency in training but is not intended to be the sole method for qualification to any postgraduate program or certification (1). During the past few years, the ACR in-training examination for radiation oncology has undergone changes to make the test
METHODS AND MATERIALS The ACR in-training examination for radiation oncology is offered to residents in the United States and Canada on a voluntary basis and is given yearly in the spring. The items tested include radiation and cancer biology, medical physics, statistics, and clinical oncology. Under clinical oncology, the subject areas tested include breast, bone/soft tissue, central nervous system and eye, gastrointestinal, genitourinary, gynecologic, head and neck/skin, lymphoma/ leukemia, pediatric and thoracic tumors. Each resident who registers for the examination must sign a consent form permitting the residency program director to have knowledge of the scores. The scores are sent to the residency program director approximately 6–8 weeks after the date of the examination, with the number of correct answers and percentile ranking of their residents compared with residents from other training programs at the same postgraduate year level.
Conflict of interest: none. Received Sept 10, 2007, and in revised form Sept 28, 2007. Accepted for publication Sept 28, 2007.
Reprint requests to: Arnold C. Paulino, M.D., Department of Radiation Oncology, Methodist Hospital, 6565 Fannin St., DB1-077, Houston, TX 77030. Tel: (713) 441-4844; Fax: (713) 441-4493; E-mail: [email protected] 666
ACR in-training examination in radiation oncology d A. C. PAULINO AND E. KURTZ
667
Table 2. Number of questions in in-training examination according to year and subject area
RESULTS Number of residents and year of training Table 1 lists the number of residents who participated in the in-training examination for the past 4 years. In general, the number of residents taking the examination has increased, from a total of 517 residents in 2004 to 572 in 2007. The number of Level 4 or senior residents taking the examination has declined from 251 (23.6%) of 1,063 residents in 2004 and 2005 to 216 (19.6%) of 1,103 residents in 2006 and 2007 (p = 0.026, chi-square). Number and subject area of questions Table 2 lists the number of questions according to year of testing and subject area being tested. The total number of questions decreased from 509–511 questions in 2004–2005 to 405 in 2006 and 360 in 2007. This was primarily secondary to a decrease in clinical oncology questions from 363 in 2004 to 335 in 2005, 270 in 2006, and 225 in 2007. All clinical subject areas had a decrease in the number of questions. In the 2007 examination, two subsections were eliminated and incorporated into another subsection. The eye questions were included in the central nervous system subsection, and the skin questions were included in the head-and-neck subsection. The physics questions have also decreased from 76–78 questions in 2004–2005 to 60 in 2006 and 2007. The radiation and cancer biology questions have remained at about 60–61 questions, with the exception of 2005 when there were 77 questions. Examination mean scores Tables 3 to 5 give the mean percentage of correct answers per level of training and year of testing in biology, physics, and clinical oncology. In general, the mean scores for Level 2 residents were better than those for Level 1 residents, and, likewise, the mean scores for Level 3 residents were better than those for Level 2 residents, with the exception of the 2004 biology section, for which the scores were about the same for each level of training. In general, the results for Level 4 residents were greater than those for Level 3 residents in clinical oncology. For the 2004 and 2005 examinations, the Level 4 mean scores were about the same as the Level 3 scores for the biology and physics sections. In contrast, in 2006 and 2007, the biology and physics mean scores were better for the Level 3 residents than for the Level 4 Table 1. Number of radiation oncology residents taking the in-training examination according to year and level of training
Year (n) Section Biology Physics Statistics Clinical Bone/soft tissue Breast Central nervous system Eye Gastrointestinal Genitourinary Gynecology Head and neck Lung Lymphoma Pediatrics Skin Total
2004
2005
2006
2007
61 76 9 363 35 38 28
77 78 21 335 20 35 27
60 60 15 270 15 25 25
60 60 15 225 15 24 21
20 35 37 30 21 32 49 27 11 509
18 32 28 29 33 32 36 31 14 511
15 25 25 25 25 25 25 25 15 405
—* 24 24 24 24 24 24 21 —y 360
* In 2007, eye questions were included in the central nervous system category. y In 2007, skin questions were included in the head-and-neck category.
residents. Table 6 shows the mean percentage of correct answers for the entire examination. The scores increased according to level of training until Level 3; the scores for Level 4 residents were approximately the same as (2004, 2005, and 2007 examinations), or worse than (2006 examination), those of the Level 3 residents. Figure 1 shows the resident percentile rank of residents in relation to the percentage of correct answers in the 2006 biology, physics, and clinical examinations, and Fig. 2 shows the 2007 results. In the clinical portion of the examination for both 2006 and 2007, Level 4 residents performed at a relative percentile rank that was superior to that of Level 3 residents. In the biology and physics sections, Level 3 residents performed at a relative percentile rank that was superior to that of Level 4 residents. Hence, Level 3 residents, as a group, answered more questions correctly than did the Level 4 residents in biology and physics for the 2006 and 2007 examinations. Changes in examination in past years On the basis of the input from the Association of Residents in Radiation Oncology, several changes have been Table 3. Mean percentage of correct answers to biology section according to level of training and year of testing
Year (n) Year (%) Level
2004
2005
2006
2007
1 2 3 4 Total
119 134 141 123 517
151 128 139 128 546
148 151 121 111 531
164 151 152 105 572
Level
2004
2005
2006
2007
1 2 3 4
78.8 79.5 79.4 79.3
60.9 65.1 70.0 69.7
61.0 65.5 69.3 41.2
60.5 64.8 71.3 65.0
I. J. Radiation Oncology d Biology d Physics
668
Table 4. Mean percentage of correct answers to physics section according to level of training and year of testing
Volume 70, Number 3, 2008
Table 6. Mean percentage of correct answers to entire examination according to level of training and year of testing
Year (%)
Year
Level
2004
2005
2006
2007
Level
2004
2005
2006
2007
1 2 3 4
78.5 82.7 83.9 85.7
55.2 58.7 63.2 63.1
45.2 51.2 57.0 34.2
49.8 53.5 58.8 52.0
Level 1 Level 2 Level 3 Level 4
73.3 75.4 78.2 77.8
59.1 63.9 68.4 68.7
54.5 59.9 63.2 56.9
55.7 60.5 64.1 63.7
instituted to make the in-training examination an educational experience for residents and their program directors. In 2002, the examination booklets with all the questions were given to the residents for self-study. In 2003, the answer key was also given to the program directors and posted on the ACR Website. Starting 2004, rationales to the answers for approximately 25% of the questions were posted on the ACR Website. In 2005, the biology and physics question writers were doubled, and the questions became more diverse, with multiple topics covered. In 2006, the B type (matching) and X type (true/false) questions were eliminated. The in-training examination consisted solely of A type questions (multiple choice with four possible answers). Before 2007, questions on the in-training examination were divided into sections with a heading of biology, physics, clinical oncology, or statistics. Beginning in 2007, the questions were mixed and were not divided according to section. In addition, rationales to the answers were available for each question and were given to the program directors and posted on the ACR Website.
DISCUSSION Compared with previous reports, the number of residents taking the ACR in-training examination in radiation oncology has increased, with 572 residents taking the examination in 2007. In 1984, only 365 took the examination vs. in 1991, when 497 did so (2, 3). It has been estimated that approximately 80% of residents currently in-training take the ACR in-training examination (4). The increase in the number taking the examination is in agreement with the increase in the number of radiation oncology residents currently training in North America. The total number of questions has also recently changed. In 2004 and 2005, approximately 500 questions were Table 5. Mean percentage of correct answers to clinical section according to level of training and year of testing Year Level
2004
2005
2006
2007
1 2 3 4
71.5 75.4 77.1 77.8
59.5 64.7 68.4 70.2
54.6 60.1 62.9 65.3
55.2 60.6 63.2 66.1
included, slightly greater than the 450 questions reported for 1984 to 1987. However, in 2006 and 2007, the number of questions decreased to 405 and 360, respectively. There are two explanations for why this occurred. First, the rationales for the answers to the questions were instituted in 2004, making it more time consuming for question writers to develop the test. Second, in 2006, the B type and X type questions were eliminated, leaving only A type questions. Because the matching type and true/false format was easier for question writers to construct compared with the multiple choice format, it was decided that the number of total questions would be lessened to compensate for this change. The move to the A-type format was partly because of the current format of the American Board of Radiology (ABR) written examination, which now consists of only A-type questions. Because many program directors and residents view the intraining examination as a practice for the ABR written board examination, the move to the A-type format for the intraining examination was a logical step for the committee. Other educators have also questioned the validity of X-type questions in which the examinee has a 50:50 chance of getting the answer correct (5, 6). Although the in-training examination shows construct validity, with resident performance improving with each year of additional clinical oncology training, it does so only until Level 3 of biology and physics training. In 2004 and 2005, the Level 3 and 4 mean scores were approximately the same for both biology and physics. In contrast, for the 2006 and 2007 examination, Level 3 residents had a greater mean score than did the Level 4 residents. The one change that occurred between 2005 and 2006 was the ability of the residents to take the ABR written boards in biology and physics during the beginning of their Level 4 of residency. If the residents had passed both the biology and physics examinations, they had little motivation to do well on the in-training examination in biology and physics, which was offered about 9 months after they had passed the ABR examination. Some residents skipped or answered the same key for the biology and physics examination. Even when we mixed the biology and physics questions with the clinical questions in the 2007 examination, the Level 4 scores were still not as great as the Level 3 scores in biology and physics (Fig. 2). Recent changes in the ACR in-training examination have made the process an educational tool for both radiation oncology residents and their program directors. Although
ACR in-training examination in radiation oncology d A. C. PAULINO AND E. KURTZ
669
Fig. 2. 2007 In-training examination for radiation oncology results by resident level. (a) Biology. (b) Physics. (c) Clinical section. Fig. 1. 2006 In-training examination for radiation oncology results by resident level. (a) Biology. (b) Physics. (c) Clinical section.
in the past, the examination booklets were collected after administration of the test, the booklets are now given to the residents. In addition, the answer key and rationales for all answers are now available either by hard copy or by the Web for residents and training directors to go over in their teaching sessions. In previous surveys, 77–88% of residents
thought that the in-training examination was helpful in assessing their knowledge (4, 7). It is our hope that the ACR in-training examination for radiation oncology will continue to be a useful tool in assessing resident knowledge but will also serve as an education tool for residents in-training and their program directors.
670
I. J. Radiation Oncology d Biology d Physics
Volume 70, Number 3, 2008
REFERENCES 1. In-training examination brochure. Reston, VA: American College of Radiology; 2007. 2. Wilson JF, Diamond JJ. Summary results of the ACR experience with an in-training examination for residents in radiation oncology. Int J Radiat Oncol Biol Phys 1988;15:1219–1221. 3. Coia LR, Wilson JF, Bresch JP, et al. Results of the in-training examination of the American College of Radiology for residents in radiation oncology. Int J Radiat Oncol Biol Phys 1992;24: 903–905.
4. Patel S, Jagsi R, Wilson J, et al. Results of the 2004 Association of Residents in Radiation Oncology (ARRO) survey. Int J Radiat Oncol Biol Phys 2006;66:1199–1203. 5. Available from: http://www.nbme.org/publications/item_ writing_manual_download.html. Accessed November 20, 2007. 6. Moss E. Multiple choice questions: Their value as an assessment tool. Curr Opin Anaesthesiol 2001;14:661–666. 7. Reddy S, Vijayakumar S. Evaluating clinical skills of radiation oncology residents: Parts I and II. Int J Cancer 2000;90:1–12.