- Email: [email protected]
Factors affecting students performance on the National Ranking Examination in a French Medical School
Presse Med. 2010; 39: e134–e140 ß 2010 Elsevier Masson SAS. All rights reserved.
Original article
en ligne sur / on line on www.em-consulte.com/revue/lpm www.sciencedirect.com
Factors affecting students performance on the National Ranking Examination in a French Medical School Pascal Andujar1,2, Sylvie Bastuji-Garin3,4, Françoise Botterel2,5, Marc Prevel6, Jean-Pierre Farcet2,7, Pascal Claudepierre3,8
1. Hôpital intercommunal de Créteil, service de pneumologie et de pathologie professionnelle, 94010 Créteil, France 2. Université Paris 12, faculté de médecine, 94010 Créteil, France 3. Université Paris 12, LIC EA 4393, 94010, Créteil, France 4. AP–HP, groupe hospitalier Henri-Mondor–Albert-Chenevier, département de recherche clinique et santé publique, 94010 Créteil, France 5. AP–HP, groupe hospitalier Henri-Mondor–Albert-Chenevier, service de parasitologie-mycologie, 94010 Créteil, France 6. Hôpital de Saint-Denis, pôle URCP, service des urgences, 93200 Saint-Denis, France 7. AP–HP, groupe hospitalier Henri-Mondor–Albert-Chenevier, service d’immunologie biologique, 94010 Créteil, France 8. AP–HP, groupe hospitalier Henri-Mondor–Albert-Chenevier, service de rhumatologie, 94010 Créteil, France Received December 14, 2009 Accepted March 16, 2010 Available online: 5 May 2010
Correspondence: Sylvie Bastuji-Garin, Hôpital Henri-Mondor, département de recherche clinique et santé publique, 51, avenue Mal-de-Lattre-de-Tassigny, 94010 Créteil cedex, France. [email protected]
Résumé Facteurs associés à la performance des étudiants à l’Examen Classant National dans une faculté de médecine française
e134
Objectif > Le rang de classement à l’Examen Classant National (ECN) détermine le choix de la spécialité médicale et la région d’installation. Notre objectif était d’identifier les facteurs permettant de prédire le rang de classement à l’ECN afin de repérer les étudiants en difficulté potentielle. Méthode > Tous les étudiants ayant passé l’ECN entre 2004 et 2008 après un cursus complet à la faculté de médecine de Créteil (Université Paris 12) ont été sélectionnés à partir d’une base de données administrative secondairement anonymisée (n = 473). Les liens entre le rang de classement au 1er ECN et les caractéristiques des étudiants (sociodémographiques et performances antérieures)
Summary Objective > Results on the National Ranking Examination (NRE) taken at the end of 6 years of medical school determine how much choice students have about the medical specialty and geographic area where they will perform their residency. Our objective was to identify academic and non-academic factors predicting performance on the NRE. Methods > We conducted a database study of all medical students who completed the 6 years of medical studies at Creteil medical school (Paris 12 University) and who took the NRE between 2004 and 2008 (n = 473). Correlations between students’ characteristics and the NRE rank were analysed using multivariate linear regression models. The students were also divided into three categories based on whether their NRE rank was in the top quartile, bottom quartile, or middle two quartiles. Those 3 groups were compared using multivariate multinomial logistic regression models.
tome 39 > n86 > juin 2010 doi: 10.1016/j.lpm.2010.03.007
ont été analysés en régression linéaire multiple. Les étudiants classés dans les 25 % premiers, 25 % derniers et entre les deux ont été comparés en régression logistique multinomiale afin d’identifier les facteurs associés aux plus mauvaises performances. Résultats > Les facteurs indépendamment associés (p 0,05) au rang de classement à l’ECN étaient le redoublement en PCEM1 (coefficient : 11,92 ; IC95 % 8,69–15,15), le rang de classement aux examens facultaires en PCEM1 (0,14 ; 0,05–0,22), DCEM3 (0,19 ; 0,12–0,26), et DCEM4 (0,32 ; 0,22–0,42), le nombre d’années avec au moins un échec à un examen (3,94 ; 1,08–6,80), et le fait de ne pas avoir passé les ECN blancs (13,0 ; 12,39–13,61). Ces mêmes facteurs permettaient d’identifier les étudiants classés dans les 25 derniers. Les caractéristiques sociodémographiques très associées aux examens facultaires disparaissaient en analyse multivariée. Conclusion > La performance à l’ECN était très associée aux performances antérieures, et ceci dès la première année de médecine. Ces résultats ont été diffusés aux étudiants et vont être utilisés pour identifier précocement les étudiants les plus à risque de mauvais classement à l’ECN afin de leur proposer une aide individualisée.
Results > Factors independently associated (p 0.05) with rank on the NRE were repeating the first year of medical school (coefficient: 11.92 [95%IC 8.69–15.15]); rank on the first-, fifth-, and sixth-year examinations (0.14 [0.05–0.22]; 0.19 [0.12–0.26] and 0.32 [0.22– 0.42] respectively); number of years with at least one failed examination (3.94 [1.08–6.80]); and failure to attend a practice NRE session (13.0 [12.39–13.61]). Factors associated with the worst NRE performance were similar to those found when the NRE rank was handled as a continuous variable. Socio-economic characteristics of students were strongly associated with medical school performance and, therefore, were not independently associated with rank on the NRE. Conclusion > Performance on the NRE was strongly associated with previous performance on medical school examinations, ever since the first year of medical school. Students were informed of these results which will help us to identify high-risk students who require early remedial help.
A
questions each about which short answers must be given [1]. Thus, the rank on the NRE exerts a profound impact on the student’s career. An ability to detect students at risk for poor NRE performance early on, followed by information of these students and remedial help if needed, might exert a positive impact on these students’ careers. Therefore, we designed a study with the objective of identifying socio-demographic and academic characteristics associated with performance on the NRE.
t the end of 6 years of medical school, with daily clinical training during the last 3 years and qualifying examinations at the end of each year, French medical students take a local examination that meets national standards. Then, they take a national ranking examination (NRE), which determines how much choice they have regarding the field of medicine and geographic location where they will receive residency training. During the study period (2004-2008), the NRE consisted of a written examination including 9 clinical scenarios with 6 to 10
Original article
Factors affecting students performance on the National Ranking Examination in a French Medical School
Methods Design
Several factors are associated with test performance of medical students. What this study adds Strong correlations exist between rank on the national examination for medical students in France and previous academic performance, including rank on yearly medical school tests. Students at risk for low national rank could be identified as early as the first year of medical school. Socio-economic characteristics are strongly associated with medical school performance, but are not independently associated with rank on the national test.
tome 39 > n86 > juin 2010
Medical Training in France In France, all high-school graduates can enter medical school. At the end of the first year, students take a competitive written examination; about 15% of students pass this examination, which allows them to continue medical school. Students who fail are allowed one more try. Admitted students go on to the next 5 years of medical school, of which the first 2 years consist mainly of classroom work on basic sciences and the last 3 years of courses on specific specialties and of clinical rotations in
e135
What is already known on this topic
We conducted a database study of all medical students who completed the 6 years of medical studies at Creteil medical school (Paris 12 University) and who took the NRE between 2004 and 2008. Anonymity was guaranteed to participants. Given the database design, our ethics committee waived the need for informed consent, in accordance with French law.
P Andujar, S Bastuji-Garin, F Botterel, M Prevel, J-P Farcet, P Claudepierre
university hospitals. At the end of each of these 5 years, the students take written qualifying examinations on theoretical knowledge; oral examinations; and, during the last 3 years, standardized patient encounters. Failed examinations can be taken again in September, and students who fail then can repeat the year once. Therefore, the time needed to complete the medical school curriculum is at least 6 years, but may vary. Then, the students take the NRE, the results of which govern their choice of their residency posts.
Data Collection We used data from longitudinal surveys of students at a medical school in the Paris conurbation. Socio-demographic data and student performance from the surveys were extracted from the database and anonymised. For our study, we extracted the following demographic data: gender, age at entry in medical school, and age at the time of the NRE. To indirectly estimate socio-economic status, we used the place of residence, father’s occupation (mother’s one being not recorded in the database), and whether the student had a scholarship. Median pre-tax income per consumption unit (CU) [2–4] at the place of residence was computed using the corresponding postal code and data from the INSEE (French National Institute for Statistics and Economic Studies) and the French General Tax Directorate. We recorded the following information on academic performance of each student: whether the student repeated the first year, rank on each yearly qualifying test, number of failed yearly qualifying tests, and whether the student attended a practice NRE. Finally, we collected the rank achieved on the NRE. When a student attended twice the NRE, the rank of the first year was considered
Statistical analysis
e136
Continuous data are reported as median (range) and categorical data as number (percentage). To take into account the increasing number of students over the study period, ranks on the yearly qualifying examinations and NRE were converted to percentiles for statistical analyses. The students were also divided into three categories based on whether their NRE rank was in the top quartile, bottom quartile, or middle two quartiles. We compared the characteristics and performances of the students over the study period. We then conducted two analyses, one to look for correlations between student characteristics and the NRE rank and the other to compare characteristics among the NRE performance groups (top and bottom quartiles and intermediate two quartiles). Univariate and multivariate analyses with assessments of confounding and interactions were performed. Given that the students were independent across years but not necessarily within each year, all multivariate analyses used the NRE year as a cluster and were adjusted for that year.
We used linear regression models with robust standard errors to identify socio-demographic and academic characteristics potentially associated with NRE performance. To avoid multicollinearity, variables strongly associated with each other (e.g., age at first year of medical school and age at NRE) were not evaluated simultaneously, and the variance inflation factor was estimated. The omitted variable hypothesis was tested using the Ramsey test. To evaluate differences between student characteristics in the three NRE performance groups (top and bottom quartiles and intermediate two quartiles), we used the x2 test, Fisher exact test, or Kruskal-Wallis test, as appropriate. We then built multivariate multinomial logistic regression models in which the three NRE performance groups (top, bottom, and intermediate) were handled as possible values of the same nominal qualitative outcome variable, with the intermediate group serving as the reference. All statistical tests were done using Stata Statistical software (Stata Corp 2003. Release 8.0, College Station, TX, USA). All tests were two-tailed and values of p no greater than 0.05 were considered significant.
Results The characteristics of the 473 students who met the study criteria are reported in Table I. Socio-demographic characteristics did not vary significantly across the 5 study years (2004 to 2008) (data not shown). Performance on the NRE differed significantly across study years (p < 0.001) (Table II).
Factors associated with national ranking examination (NRE) performance Factors associated with lower NRE performance in the univariate analysis were age at NRE, being a scholarship recipient, lower performance on yearly qualifying tests, and failing to attend a practice NRE session (Table III, supplementary material). Better NRE performance was associated with having a father who was a senior executive and with living in a highincome area. The analysis of confidence intervals of the coefficients showed that NRE rank was stronger associated with ranks observed at the end of medical school (5th and 6th year) than with 1st year rank. No significant interactions were detected. However, several variables were strongly linked, such as father’s occupation and being a scholarship recipient, which were linked to each other and to performance on the yearly qualifying tests. In the multivariate analysis, only six parameters were independently associated with NRE performance (Table III): repeating the first year of medical school; performance on the first-, fifth-, and sixth-year qualifying tests, number of years with at least one failed test, and not attending a practice NRE session. For example, repeating the first year was associated with an 11.92 percentile increase in the NRE rank (95%CI, 8.69 to 15.15; tome 39 > n86 > juin 2010
Table I Characteristics of students at the study medical school who took the national ranking examination (NRE) between June 2004 and June 2008. Characteristics of students
Total (n = 473)
Initial characteristics Female
296 (62.6)
Age (years)
18.2 [15.9-26.6]
Father’s occupation Senior executive
263 (56.6)
Skilled tradesman/intermediate profession
88 (18.6)
Others
122 (25.8)
Repeated the first year of medical school
Factors associated with good or poor national ranking examination (NRE) performance Factors associated with the NRE performance group (top quartile, bottom quartile, or intermediate quartiles) in the univariate analysis (data not shown) and multivariate analysis (Table IV) were similar to those found when the NRE rank was handled as a continuous variable. Among the 73 students who performed poorly during the first medical school year, i.e., who had to repeat the first year and whose test results at the end of the repeated year were in the bottom quartile, 8 (11%) were in the top NRE quartile, 32 (43.8%) in the bottom NRE quartile, and 33 (45.2%) in the intermediate quartile.
Original article
Factors affecting students performance on the National Ranking Examination in a French Medical School
339 (71.7)
Discussion
General characteristics on the NRE year Age (years)
24.8 [21.6-30.3]
Scholarship recipient
72 (16.4)
Median pre-tax income per consumption unit at the place of residence (s)
21 253 [12 824-40 669]
Place of residence Administrative district where our medical school is located
The NRE is controversial in France, where there is no clear consensus about how best to evaluate medical students. Similarly, in the UK the fairness of national qualifying examinations is debated [5,6].
306 (69.7)
Main findings
Paris
51 (11.6)
Paris conurbation (except for our administrative district)
82 (18.7)
Continuous data are reported as median [range] and categorical data as number (%).
p = 0.001). These six variables explained 59% of the NRE rank variation (R2 = 0.59). The value of the variance inflation factor (mean, 1.52; range, 1.03 to 2.26) indicated that there was no multi-collinearity. The Ramsay test result rejected the omitted variable hypothesis (p = 0.44).
NRE performance was strongly associated with previous performance on the yearly qualifying tests. Socio-economic characteristics of students were strongly associated with yearly qualifying test performance and, therefore, were not independently associated with NRE performance in the multivariate analysis. Variables pertaining to the first year of medical school were associated with NRE performance, indicating that students at risk for poor NRE performance can be identified early on. However, at the individual level, some students with poor performance in medical school may obtain high ranks on the NRE. Thus, the NRE may serve as a last chance for motivated
Table II Results on the national ranking examination (NRE) obtained by the study students who took the test between June 2004 and June 2008. Year of the NRE Total All students in France, n Number of study students, n (%)
2004
2005
2006
2007
2008
23 280
3663
4232
4903
5500
5859
473(2.0)
73 (2.0)
90 (2.1)
95 (1.9)
88 (1.6)
127 (2.2)
p* 0.45
NRE rank Median rank** [range]
2368 [2–5776] 1285 [12–3580] 2474 [32–4223] 2062 [2–4903] 2288 [30–5157] 3577 [206–5776] < 0.001
Rank category, n (%) Top quartile
118 (25.0)
24 (32.9)
21 (23.3)
29 (30.5)
22 (25.0)
22 (17.3)
Intermediate two quartiles
243 (51.4)
39 (53.4)
48 (53.3)
50 (52.6)
46 (52.3)
60 (47.2)
Bottom quartile
113 (23.7)
10 (13.7)
21 (23.3)
16 (16.8)
20 (22.7)
45 (35.4)
0.015
NRE, National ranking examination. p value by Fisher exact test or non-parametric Kruskal–Wallis test as appropriate.
**
As the number of students differed across years, NRE ranks were converted to percentiles per year for the statistical analyses.
tome 39 > n86 > juin 2010
e137
*
P Andujar, S Bastuji-Garin, F Botterel, M Prevel, J-P Farcet, P Claudepierre
Table III Factors associated with national ranking examination (NRE) performance in the multivariate analysis (linear regression analysis). Coefficient [95%CI] * Repeated first year of medical school Rank on medical school tests
No
Reference
Yes
11.92 [8.69 ; 15.15]
0.001
1st year
0.14 [0.05 ; 0.22]**
0.011
5th year
0.19 [0.12 ; 0.26]
**
0.002
6th year
0.32 [0.22 ; 0.42]**
0.001
3.94 [1.08-6.80]***
0.02
Number of years with at least one failed test Attended practice NRE
p
Yes or unavailable
Reference
No
13.00 [12.39 ; 13.61]
< 0.001
* Coefficients with the 95% confidence interval were estimated using linear regression with robust standard errors adjusted for NRE year and for the variables listed in the table. Given that the number of students varied across years, NRE ranks and ranks on the yearly medical school tests were transformed into percentiles. **
For a 1-percentile increase in medical school test rank;.
***
For a 1-year increase in number of years with at least one failed test R2 = 0.59.
Table IV Factors associated with a national ranking examination result in the top quartile or bottom quartile in the multivariate analysis (multinomial logistic regression analysis) NRE performance category Intermediate quartiles
Top quartile OR [95%CI] Repeated first year Rank on medical school tests
*
p
0.39 [0.28–0.55]
Reference category
Bottom quartile OR [95 %IC] *
p
0.000
1
2.78 [1.07–7.26]
0.04
**
1st year
0.98 [0.97–0.99]
0.01
1
1.01 [1.01–1.02]
< 0.001
5th year
0.98 [0.97–0.98]
< 0.001
1
1.02 [1.01–1.04]
< 0.001
6th year
0.97 [0.96–0.99]
< 0.001
1
1.02 [1.01–1.03]
< 0.001
Failed at least one test between the 4th and 6th years
0.52 [0.34–0.81]
0.004
1
4.85 [3.01–7.82]
< 0.001
Did not attend a practice NRE
0.46 [0.10–2.01]
0.30
1
5.96 [1.78–19.91]
0.004
*
Odds-ratios with the 95% confidence interval were estimated using multinomial logistic regression adjusted for the NRE year and for the variables listed in the table.
**
Ranks on medical school tests were transformed into percentiles to take into account the differences in numbers of students across years. The odds-ratios are reported for a 1percentile increase in rank.
students to train in their preferred medical field and geographic location despite difficulties during medical school.
Strengths and weaknesses of our study
e138
Our study provides the first assessment of factors associated with NRE performance in France. Although we included five consecutive cohorts of medical students (2004–2008), the singlecentre design of our study may limit the extent to which our findings can be generalized to other schools. However, the results at our medical school are very close to the median in France: when we pooled the 5 years, we found that 25% of our students ranked in the top quartile and 24% in the bottom
quartile on the NRE. We are not aware of similar studies in other medical schools. Another weakness of our study is the paucity of data on the socio-economic and ethnic characteristics of the students. The mother’s occupation was not available in the database. A study using data from 30 countries found that the impact of the mother’s education and occupation on student performance was comparable to that of the father’s [7]. Several cohort studies demonstrated that belonging to an ethnic minority predicted poor performance in medical school [8–10]. However, French law does not allow the collection of data on ethnicity. Similarly, we had no data on substance abuse, or non-academic activities, both of which were associated with poor performance tome 39 > n86 > juin 2010
in earlier studies [8,11,12]. Neither did we have information on personal motivation or learning styles, which have been reported to predict success in medical school [13–16]. We had no data on high school performance. Strong points of our study include the use of reliable information on the entire medical school history of consecutive students admitted over 5 years. Furthermore, the factors identified by our multivariate analysis explained a large proportion (59%) of the variability in NRE performance. In addition, the omitted variable hypothesis was rejected.
Our findings in relation to other studies Our finding of a strong correlation between early medical school performance and NRE performance are consistent with those of a recent cohort study in the UK [17], in which poor performance on early written medical school tests predicted poor performance in subsequent written tests [17]. Earlier studies found that females performed better than males, a difference not found in our study. The impact of gender on performance remains debated [14,17]. The admission examination at the end of the first year, which has a 15% pass rate, and the predominance of women in our population (63%) may have affected our ability to detect an influence of gender. Age, a factor associated with medical school performance in earlier work [14], was not independently associated with NRE performance in our multivariate analysis that corrected for the strong associations between age and medical school test performance. Although no previous studies of NRE performance are available, our results are consistent with studies of factors associated with medical school test performance. Thus, they suggest that the NRE may constitute a fair way to evaluate medical students at the national level.
Potential implications for students and medical schools In France, the NRE is a strong focus of interest for medical schools, as it exerts a considerable influence on the career of
their students and enables a comparison of medical schools across the country. In the near future, French medical schools will be assessed based on their students’ NRE performance. Thus, the NRE may help to increase teacher motivation at medical schools. Furthermore, the independent association between NRE rank and both initial (1st year of medical school) and final medical school (5th and 6th year) ranks suggests that changes of ability of medical students to perform well occurs in the course of their medical school. A major finding from our study is that students at high risk for poor NRE performance can be identified early on, during the first few years of medical school. Students should be informed that repeating the first year is associated with poor subsequent performance, as this information may enable them to take corrective steps. Similarly, students should be informed of the statistical risk associated with failure on one or more yearly qualifying tests. Thus, our findings may help medical students to improve their performance and medical schools to identify high-risk students who require early remedial help.
Original article
Factors affecting students performance on the National Ranking Examination in a French Medical School
Conclusion Several factors are associated with NRE performance. Students should be informed of these factors, most notably those occurring early in medical school, so that they can take corrective action. Given the major influence of NRE performance on the careers of medical students, further studies of NRE performance in other medical schools should be performed. Our data suggest that the NRE may constitute a fair way to assess medical students. Conflicts of interest : None. Acknowledgments : The authors (all were members of the Creteil Medical School Committee for continuous education improvement) are indebted to the personnel at the Registrar’s Office who selected and retrieved the data from the database and to A Wolfe MD for reviewing the manuscript.
Supplementary data associated with this article can be found, in the online version, at doi:10.1016/ j.lpm.2010.03.007.
References
[2]
Segouin C, Jouquan J, Hodges B, Brechat PH, David S, Maillard D et al. Country report: medical education in France. Med Educ 2007;41:295-301. INSEE. Median household income per consumption unit before tax income in France in 2005. http://www.insee.fr/French National Institute for Statistics and Economic Studies (INSEE), 2005.
tome 39 > n86 > juin 2010
[3]
[4]
Hagenaars AJM, De Vos K., Zaidi MA. Poverty statistics in the late 1980 s: research based on micro-data. In: Communities OPotE, ed. Office for Official Publications of the European Communities ed. Luxembourg: Eurostat publication 1994. Forster M. What are Equivalence Scales? 2005 [cited 29/07/2009]; http://www.oecd.org/ dataoecd/61/52/35411111.pdf.
[5]
[6]
[7]
Ricketts C, Archer J. Are national qualifying examinations a fair way to rank medical students? Yes. BMJ 2008;337:a1282. Noble IS. Are national qualifying examinations a fair way to rank medical students? No. BMJ 2008;337:a1279. Marks G. Are father’s or mother’s socioeconomic characteristics more important influences on student performance? Recent
e139
[1]
P Andujar, S Bastuji-Garin, F Botterel, M Prevel, J-P Farcet, P Claudepierre
e140
international evidence. Soc Indic Res 2008;85:16. [8] Lumb AB, Vail A. Comparison of academic, application form and social factors in predicting early performance on the medical course. Med Educ 2004;38:1002-5. [9] Bessant R, Bessant D, Chesser A, Coakley G. Analysis of predictors of success in the MRCP (UK) PACES examination in candidates attending a revision course. Postgrad Med J 2006;82:145-9. [10] Yates J, James D. Risk factors for poor performance on the undergraduate medical course: cohort study at Nottingham University. Med Educ 2007;41:65-73.
[11] McAuliffe WE, Rohman M, Santangelo S, Feldman B, Magnuson E, Sobol A et al. Psychoactive drug use among practicing physicians and medical students. N Engl J Med 1986;315:805-10. [12] Baldisseri MR. Impaired healthcare professional. Crit Care Med 2007;35:S106-16. [13] Tanaka M, Mizuno K, Fukuda S, Tajima S, Watanabe Y. Personality traits associated with intrinsic academic motivation in medical students. Med Educ 2009;43: 384-7. [14] Ferguson E, James D, Madeley L. Factors associated with success in medical school:
systematic review of the literature. BMJ 2002;324:952-7. [15] Webb E. Academia: the view from below. Academic medicine does not fit in with motherhood. BMJ 1997;315:1469. [16] James D, Chilvers C. Academic and nonacademic predictors of success on the Nottingham undergraduate medical course 1970-1995. Med Educ 2001;35:1056-64. [17] Cleland JA, Milne A, Sinclair H, Lee AJ. Cohort study on predicting grades: is performance on early MBChB assessments predictive of later undergraduate grades? Med Educ 2008;42:676-83.
tome 39 > n86 > juin 2010
Original article
en ligne sur / on line on www.em-consulte.com/revue/lpm www.sciencedirect.com
Factors affecting students performance on the National Ranking Examination in a French Medical School Pascal Andujar1,2, Sylvie Bastuji-Garin3,4, Françoise Botterel2,5, Marc Prevel6, Jean-Pierre Farcet2,7, Pascal Claudepierre3,8
1. Hôpital intercommunal de Créteil, service de pneumologie et de pathologie professionnelle, 94010 Créteil, France 2. Université Paris 12, faculté de médecine, 94010 Créteil, France 3. Université Paris 12, LIC EA 4393, 94010, Créteil, France 4. AP–HP, groupe hospitalier Henri-Mondor–Albert-Chenevier, département de recherche clinique et santé publique, 94010 Créteil, France 5. AP–HP, groupe hospitalier Henri-Mondor–Albert-Chenevier, service de parasitologie-mycologie, 94010 Créteil, France 6. Hôpital de Saint-Denis, pôle URCP, service des urgences, 93200 Saint-Denis, France 7. AP–HP, groupe hospitalier Henri-Mondor–Albert-Chenevier, service d’immunologie biologique, 94010 Créteil, France 8. AP–HP, groupe hospitalier Henri-Mondor–Albert-Chenevier, service de rhumatologie, 94010 Créteil, France Received December 14, 2009 Accepted March 16, 2010 Available online: 5 May 2010
Correspondence: Sylvie Bastuji-Garin, Hôpital Henri-Mondor, département de recherche clinique et santé publique, 51, avenue Mal-de-Lattre-de-Tassigny, 94010 Créteil cedex, France. [email protected]
Résumé Facteurs associés à la performance des étudiants à l’Examen Classant National dans une faculté de médecine française
e134
Objectif > Le rang de classement à l’Examen Classant National (ECN) détermine le choix de la spécialité médicale et la région d’installation. Notre objectif était d’identifier les facteurs permettant de prédire le rang de classement à l’ECN afin de repérer les étudiants en difficulté potentielle. Méthode > Tous les étudiants ayant passé l’ECN entre 2004 et 2008 après un cursus complet à la faculté de médecine de Créteil (Université Paris 12) ont été sélectionnés à partir d’une base de données administrative secondairement anonymisée (n = 473). Les liens entre le rang de classement au 1er ECN et les caractéristiques des étudiants (sociodémographiques et performances antérieures)
Summary Objective > Results on the National Ranking Examination (NRE) taken at the end of 6 years of medical school determine how much choice students have about the medical specialty and geographic area where they will perform their residency. Our objective was to identify academic and non-academic factors predicting performance on the NRE. Methods > We conducted a database study of all medical students who completed the 6 years of medical studies at Creteil medical school (Paris 12 University) and who took the NRE between 2004 and 2008 (n = 473). Correlations between students’ characteristics and the NRE rank were analysed using multivariate linear regression models. The students were also divided into three categories based on whether their NRE rank was in the top quartile, bottom quartile, or middle two quartiles. Those 3 groups were compared using multivariate multinomial logistic regression models.
tome 39 > n86 > juin 2010 doi: 10.1016/j.lpm.2010.03.007
ont été analysés en régression linéaire multiple. Les étudiants classés dans les 25 % premiers, 25 % derniers et entre les deux ont été comparés en régression logistique multinomiale afin d’identifier les facteurs associés aux plus mauvaises performances. Résultats > Les facteurs indépendamment associés (p 0,05) au rang de classement à l’ECN étaient le redoublement en PCEM1 (coefficient : 11,92 ; IC95 % 8,69–15,15), le rang de classement aux examens facultaires en PCEM1 (0,14 ; 0,05–0,22), DCEM3 (0,19 ; 0,12–0,26), et DCEM4 (0,32 ; 0,22–0,42), le nombre d’années avec au moins un échec à un examen (3,94 ; 1,08–6,80), et le fait de ne pas avoir passé les ECN blancs (13,0 ; 12,39–13,61). Ces mêmes facteurs permettaient d’identifier les étudiants classés dans les 25 derniers. Les caractéristiques sociodémographiques très associées aux examens facultaires disparaissaient en analyse multivariée. Conclusion > La performance à l’ECN était très associée aux performances antérieures, et ceci dès la première année de médecine. Ces résultats ont été diffusés aux étudiants et vont être utilisés pour identifier précocement les étudiants les plus à risque de mauvais classement à l’ECN afin de leur proposer une aide individualisée.
Results > Factors independently associated (p 0.05) with rank on the NRE were repeating the first year of medical school (coefficient: 11.92 [95%IC 8.69–15.15]); rank on the first-, fifth-, and sixth-year examinations (0.14 [0.05–0.22]; 0.19 [0.12–0.26] and 0.32 [0.22– 0.42] respectively); number of years with at least one failed examination (3.94 [1.08–6.80]); and failure to attend a practice NRE session (13.0 [12.39–13.61]). Factors associated with the worst NRE performance were similar to those found when the NRE rank was handled as a continuous variable. Socio-economic characteristics of students were strongly associated with medical school performance and, therefore, were not independently associated with rank on the NRE. Conclusion > Performance on the NRE was strongly associated with previous performance on medical school examinations, ever since the first year of medical school. Students were informed of these results which will help us to identify high-risk students who require early remedial help.
A
questions each about which short answers must be given [1]. Thus, the rank on the NRE exerts a profound impact on the student’s career. An ability to detect students at risk for poor NRE performance early on, followed by information of these students and remedial help if needed, might exert a positive impact on these students’ careers. Therefore, we designed a study with the objective of identifying socio-demographic and academic characteristics associated with performance on the NRE.
t the end of 6 years of medical school, with daily clinical training during the last 3 years and qualifying examinations at the end of each year, French medical students take a local examination that meets national standards. Then, they take a national ranking examination (NRE), which determines how much choice they have regarding the field of medicine and geographic location where they will receive residency training. During the study period (2004-2008), the NRE consisted of a written examination including 9 clinical scenarios with 6 to 10
Original article
Factors affecting students performance on the National Ranking Examination in a French Medical School
Methods Design
Several factors are associated with test performance of medical students. What this study adds Strong correlations exist between rank on the national examination for medical students in France and previous academic performance, including rank on yearly medical school tests. Students at risk for low national rank could be identified as early as the first year of medical school. Socio-economic characteristics are strongly associated with medical school performance, but are not independently associated with rank on the national test.
tome 39 > n86 > juin 2010
Medical Training in France In France, all high-school graduates can enter medical school. At the end of the first year, students take a competitive written examination; about 15% of students pass this examination, which allows them to continue medical school. Students who fail are allowed one more try. Admitted students go on to the next 5 years of medical school, of which the first 2 years consist mainly of classroom work on basic sciences and the last 3 years of courses on specific specialties and of clinical rotations in
e135
What is already known on this topic
We conducted a database study of all medical students who completed the 6 years of medical studies at Creteil medical school (Paris 12 University) and who took the NRE between 2004 and 2008. Anonymity was guaranteed to participants. Given the database design, our ethics committee waived the need for informed consent, in accordance with French law.
P Andujar, S Bastuji-Garin, F Botterel, M Prevel, J-P Farcet, P Claudepierre
university hospitals. At the end of each of these 5 years, the students take written qualifying examinations on theoretical knowledge; oral examinations; and, during the last 3 years, standardized patient encounters. Failed examinations can be taken again in September, and students who fail then can repeat the year once. Therefore, the time needed to complete the medical school curriculum is at least 6 years, but may vary. Then, the students take the NRE, the results of which govern their choice of their residency posts.
Data Collection We used data from longitudinal surveys of students at a medical school in the Paris conurbation. Socio-demographic data and student performance from the surveys were extracted from the database and anonymised. For our study, we extracted the following demographic data: gender, age at entry in medical school, and age at the time of the NRE. To indirectly estimate socio-economic status, we used the place of residence, father’s occupation (mother’s one being not recorded in the database), and whether the student had a scholarship. Median pre-tax income per consumption unit (CU) [2–4] at the place of residence was computed using the corresponding postal code and data from the INSEE (French National Institute for Statistics and Economic Studies) and the French General Tax Directorate. We recorded the following information on academic performance of each student: whether the student repeated the first year, rank on each yearly qualifying test, number of failed yearly qualifying tests, and whether the student attended a practice NRE. Finally, we collected the rank achieved on the NRE. When a student attended twice the NRE, the rank of the first year was considered
Statistical analysis
e136
Continuous data are reported as median (range) and categorical data as number (percentage). To take into account the increasing number of students over the study period, ranks on the yearly qualifying examinations and NRE were converted to percentiles for statistical analyses. The students were also divided into three categories based on whether their NRE rank was in the top quartile, bottom quartile, or middle two quartiles. We compared the characteristics and performances of the students over the study period. We then conducted two analyses, one to look for correlations between student characteristics and the NRE rank and the other to compare characteristics among the NRE performance groups (top and bottom quartiles and intermediate two quartiles). Univariate and multivariate analyses with assessments of confounding and interactions were performed. Given that the students were independent across years but not necessarily within each year, all multivariate analyses used the NRE year as a cluster and were adjusted for that year.
We used linear regression models with robust standard errors to identify socio-demographic and academic characteristics potentially associated with NRE performance. To avoid multicollinearity, variables strongly associated with each other (e.g., age at first year of medical school and age at NRE) were not evaluated simultaneously, and the variance inflation factor was estimated. The omitted variable hypothesis was tested using the Ramsey test. To evaluate differences between student characteristics in the three NRE performance groups (top and bottom quartiles and intermediate two quartiles), we used the x2 test, Fisher exact test, or Kruskal-Wallis test, as appropriate. We then built multivariate multinomial logistic regression models in which the three NRE performance groups (top, bottom, and intermediate) were handled as possible values of the same nominal qualitative outcome variable, with the intermediate group serving as the reference. All statistical tests were done using Stata Statistical software (Stata Corp 2003. Release 8.0, College Station, TX, USA). All tests were two-tailed and values of p no greater than 0.05 were considered significant.
Results The characteristics of the 473 students who met the study criteria are reported in Table I. Socio-demographic characteristics did not vary significantly across the 5 study years (2004 to 2008) (data not shown). Performance on the NRE differed significantly across study years (p < 0.001) (Table II).
Factors associated with national ranking examination (NRE) performance Factors associated with lower NRE performance in the univariate analysis were age at NRE, being a scholarship recipient, lower performance on yearly qualifying tests, and failing to attend a practice NRE session (Table III, supplementary material). Better NRE performance was associated with having a father who was a senior executive and with living in a highincome area. The analysis of confidence intervals of the coefficients showed that NRE rank was stronger associated with ranks observed at the end of medical school (5th and 6th year) than with 1st year rank. No significant interactions were detected. However, several variables were strongly linked, such as father’s occupation and being a scholarship recipient, which were linked to each other and to performance on the yearly qualifying tests. In the multivariate analysis, only six parameters were independently associated with NRE performance (Table III): repeating the first year of medical school; performance on the first-, fifth-, and sixth-year qualifying tests, number of years with at least one failed test, and not attending a practice NRE session. For example, repeating the first year was associated with an 11.92 percentile increase in the NRE rank (95%CI, 8.69 to 15.15; tome 39 > n86 > juin 2010
Table I Characteristics of students at the study medical school who took the national ranking examination (NRE) between June 2004 and June 2008. Characteristics of students
Total (n = 473)
Initial characteristics Female
296 (62.6)
Age (years)
18.2 [15.9-26.6]
Father’s occupation Senior executive
263 (56.6)
Skilled tradesman/intermediate profession
88 (18.6)
Others
122 (25.8)
Repeated the first year of medical school
Factors associated with good or poor national ranking examination (NRE) performance Factors associated with the NRE performance group (top quartile, bottom quartile, or intermediate quartiles) in the univariate analysis (data not shown) and multivariate analysis (Table IV) were similar to those found when the NRE rank was handled as a continuous variable. Among the 73 students who performed poorly during the first medical school year, i.e., who had to repeat the first year and whose test results at the end of the repeated year were in the bottom quartile, 8 (11%) were in the top NRE quartile, 32 (43.8%) in the bottom NRE quartile, and 33 (45.2%) in the intermediate quartile.
Original article
Factors affecting students performance on the National Ranking Examination in a French Medical School
339 (71.7)
Discussion
General characteristics on the NRE year Age (years)
24.8 [21.6-30.3]
Scholarship recipient
72 (16.4)
Median pre-tax income per consumption unit at the place of residence (s)
21 253 [12 824-40 669]
Place of residence Administrative district where our medical school is located
The NRE is controversial in France, where there is no clear consensus about how best to evaluate medical students. Similarly, in the UK the fairness of national qualifying examinations is debated [5,6].
306 (69.7)
Main findings
Paris
51 (11.6)
Paris conurbation (except for our administrative district)
82 (18.7)
Continuous data are reported as median [range] and categorical data as number (%).
p = 0.001). These six variables explained 59% of the NRE rank variation (R2 = 0.59). The value of the variance inflation factor (mean, 1.52; range, 1.03 to 2.26) indicated that there was no multi-collinearity. The Ramsay test result rejected the omitted variable hypothesis (p = 0.44).
NRE performance was strongly associated with previous performance on the yearly qualifying tests. Socio-economic characteristics of students were strongly associated with yearly qualifying test performance and, therefore, were not independently associated with NRE performance in the multivariate analysis. Variables pertaining to the first year of medical school were associated with NRE performance, indicating that students at risk for poor NRE performance can be identified early on. However, at the individual level, some students with poor performance in medical school may obtain high ranks on the NRE. Thus, the NRE may serve as a last chance for motivated
Table II Results on the national ranking examination (NRE) obtained by the study students who took the test between June 2004 and June 2008. Year of the NRE Total All students in France, n Number of study students, n (%)
2004
2005
2006
2007
2008
23 280
3663
4232
4903
5500
5859
473(2.0)
73 (2.0)
90 (2.1)
95 (1.9)
88 (1.6)
127 (2.2)
p* 0.45
NRE rank Median rank** [range]
2368 [2–5776] 1285 [12–3580] 2474 [32–4223] 2062 [2–4903] 2288 [30–5157] 3577 [206–5776] < 0.001
Rank category, n (%) Top quartile
118 (25.0)
24 (32.9)
21 (23.3)
29 (30.5)
22 (25.0)
22 (17.3)
Intermediate two quartiles
243 (51.4)
39 (53.4)
48 (53.3)
50 (52.6)
46 (52.3)
60 (47.2)
Bottom quartile
113 (23.7)
10 (13.7)
21 (23.3)
16 (16.8)
20 (22.7)
45 (35.4)
0.015
NRE, National ranking examination. p value by Fisher exact test or non-parametric Kruskal–Wallis test as appropriate.
**
As the number of students differed across years, NRE ranks were converted to percentiles per year for the statistical analyses.
tome 39 > n86 > juin 2010
e137
*
P Andujar, S Bastuji-Garin, F Botterel, M Prevel, J-P Farcet, P Claudepierre
Table III Factors associated with national ranking examination (NRE) performance in the multivariate analysis (linear regression analysis). Coefficient [95%CI] * Repeated first year of medical school Rank on medical school tests
No
Reference
Yes
11.92 [8.69 ; 15.15]
0.001
1st year
0.14 [0.05 ; 0.22]**
0.011
5th year
0.19 [0.12 ; 0.26]
**
0.002
6th year
0.32 [0.22 ; 0.42]**
0.001
3.94 [1.08-6.80]***
0.02
Number of years with at least one failed test Attended practice NRE
p
Yes or unavailable
Reference
No
13.00 [12.39 ; 13.61]
< 0.001
* Coefficients with the 95% confidence interval were estimated using linear regression with robust standard errors adjusted for NRE year and for the variables listed in the table. Given that the number of students varied across years, NRE ranks and ranks on the yearly medical school tests were transformed into percentiles. **
For a 1-percentile increase in medical school test rank;.
***
For a 1-year increase in number of years with at least one failed test R2 = 0.59.
Table IV Factors associated with a national ranking examination result in the top quartile or bottom quartile in the multivariate analysis (multinomial logistic regression analysis) NRE performance category Intermediate quartiles
Top quartile OR [95%CI] Repeated first year Rank on medical school tests
*
p
0.39 [0.28–0.55]
Reference category
Bottom quartile OR [95 %IC] *
p
0.000
1
2.78 [1.07–7.26]
0.04
**
1st year
0.98 [0.97–0.99]
0.01
1
1.01 [1.01–1.02]
< 0.001
5th year
0.98 [0.97–0.98]
< 0.001
1
1.02 [1.01–1.04]
< 0.001
6th year
0.97 [0.96–0.99]
< 0.001
1
1.02 [1.01–1.03]
< 0.001
Failed at least one test between the 4th and 6th years
0.52 [0.34–0.81]
0.004
1
4.85 [3.01–7.82]
< 0.001
Did not attend a practice NRE
0.46 [0.10–2.01]
0.30
1
5.96 [1.78–19.91]
0.004
*
Odds-ratios with the 95% confidence interval were estimated using multinomial logistic regression adjusted for the NRE year and for the variables listed in the table.
**
Ranks on medical school tests were transformed into percentiles to take into account the differences in numbers of students across years. The odds-ratios are reported for a 1percentile increase in rank.
students to train in their preferred medical field and geographic location despite difficulties during medical school.
Strengths and weaknesses of our study
e138
Our study provides the first assessment of factors associated with NRE performance in France. Although we included five consecutive cohorts of medical students (2004–2008), the singlecentre design of our study may limit the extent to which our findings can be generalized to other schools. However, the results at our medical school are very close to the median in France: when we pooled the 5 years, we found that 25% of our students ranked in the top quartile and 24% in the bottom
quartile on the NRE. We are not aware of similar studies in other medical schools. Another weakness of our study is the paucity of data on the socio-economic and ethnic characteristics of the students. The mother’s occupation was not available in the database. A study using data from 30 countries found that the impact of the mother’s education and occupation on student performance was comparable to that of the father’s [7]. Several cohort studies demonstrated that belonging to an ethnic minority predicted poor performance in medical school [8–10]. However, French law does not allow the collection of data on ethnicity. Similarly, we had no data on substance abuse, or non-academic activities, both of which were associated with poor performance tome 39 > n86 > juin 2010
in earlier studies [8,11,12]. Neither did we have information on personal motivation or learning styles, which have been reported to predict success in medical school [13–16]. We had no data on high school performance. Strong points of our study include the use of reliable information on the entire medical school history of consecutive students admitted over 5 years. Furthermore, the factors identified by our multivariate analysis explained a large proportion (59%) of the variability in NRE performance. In addition, the omitted variable hypothesis was rejected.
Our findings in relation to other studies Our finding of a strong correlation between early medical school performance and NRE performance are consistent with those of a recent cohort study in the UK [17], in which poor performance on early written medical school tests predicted poor performance in subsequent written tests [17]. Earlier studies found that females performed better than males, a difference not found in our study. The impact of gender on performance remains debated [14,17]. The admission examination at the end of the first year, which has a 15% pass rate, and the predominance of women in our population (63%) may have affected our ability to detect an influence of gender. Age, a factor associated with medical school performance in earlier work [14], was not independently associated with NRE performance in our multivariate analysis that corrected for the strong associations between age and medical school test performance. Although no previous studies of NRE performance are available, our results are consistent with studies of factors associated with medical school test performance. Thus, they suggest that the NRE may constitute a fair way to evaluate medical students at the national level.
Potential implications for students and medical schools In France, the NRE is a strong focus of interest for medical schools, as it exerts a considerable influence on the career of
their students and enables a comparison of medical schools across the country. In the near future, French medical schools will be assessed based on their students’ NRE performance. Thus, the NRE may help to increase teacher motivation at medical schools. Furthermore, the independent association between NRE rank and both initial (1st year of medical school) and final medical school (5th and 6th year) ranks suggests that changes of ability of medical students to perform well occurs in the course of their medical school. A major finding from our study is that students at high risk for poor NRE performance can be identified early on, during the first few years of medical school. Students should be informed that repeating the first year is associated with poor subsequent performance, as this information may enable them to take corrective steps. Similarly, students should be informed of the statistical risk associated with failure on one or more yearly qualifying tests. Thus, our findings may help medical students to improve their performance and medical schools to identify high-risk students who require early remedial help.
Original article
Factors affecting students performance on the National Ranking Examination in a French Medical School
Conclusion Several factors are associated with NRE performance. Students should be informed of these factors, most notably those occurring early in medical school, so that they can take corrective action. Given the major influence of NRE performance on the careers of medical students, further studies of NRE performance in other medical schools should be performed. Our data suggest that the NRE may constitute a fair way to assess medical students. Conflicts of interest : None. Acknowledgments : The authors (all were members of the Creteil Medical School Committee for continuous education improvement) are indebted to the personnel at the Registrar’s Office who selected and retrieved the data from the database and to A Wolfe MD for reviewing the manuscript.
Supplementary data associated with this article can be found, in the online version, at doi:10.1016/ j.lpm.2010.03.007.
References
[2]
Segouin C, Jouquan J, Hodges B, Brechat PH, David S, Maillard D et al. Country report: medical education in France. Med Educ 2007;41:295-301. INSEE. Median household income per consumption unit before tax income in France in 2005. http://www.insee.fr/French National Institute for Statistics and Economic Studies (INSEE), 2005.
tome 39 > n86 > juin 2010
[3]
[4]
Hagenaars AJM, De Vos K., Zaidi MA. Poverty statistics in the late 1980 s: research based on micro-data. In: Communities OPotE, ed. Office for Official Publications of the European Communities ed. Luxembourg: Eurostat publication 1994. Forster M. What are Equivalence Scales? 2005 [cited 29/07/2009]; http://www.oecd.org/ dataoecd/61/52/35411111.pdf.
[5]
[6]
[7]
Ricketts C, Archer J. Are national qualifying examinations a fair way to rank medical students? Yes. BMJ 2008;337:a1282. Noble IS. Are national qualifying examinations a fair way to rank medical students? No. BMJ 2008;337:a1279. Marks G. Are father’s or mother’s socioeconomic characteristics more important influences on student performance? Recent
e139
[1]
P Andujar, S Bastuji-Garin, F Botterel, M Prevel, J-P Farcet, P Claudepierre
e140
international evidence. Soc Indic Res 2008;85:16. [8] Lumb AB, Vail A. Comparison of academic, application form and social factors in predicting early performance on the medical course. Med Educ 2004;38:1002-5. [9] Bessant R, Bessant D, Chesser A, Coakley G. Analysis of predictors of success in the MRCP (UK) PACES examination in candidates attending a revision course. Postgrad Med J 2006;82:145-9. [10] Yates J, James D. Risk factors for poor performance on the undergraduate medical course: cohort study at Nottingham University. Med Educ 2007;41:65-73.
[11] McAuliffe WE, Rohman M, Santangelo S, Feldman B, Magnuson E, Sobol A et al. Psychoactive drug use among practicing physicians and medical students. N Engl J Med 1986;315:805-10. [12] Baldisseri MR. Impaired healthcare professional. Crit Care Med 2007;35:S106-16. [13] Tanaka M, Mizuno K, Fukuda S, Tajima S, Watanabe Y. Personality traits associated with intrinsic academic motivation in medical students. Med Educ 2009;43: 384-7. [14] Ferguson E, James D, Madeley L. Factors associated with success in medical school:
systematic review of the literature. BMJ 2002;324:952-7. [15] Webb E. Academia: the view from below. Academic medicine does not fit in with motherhood. BMJ 1997;315:1469. [16] James D, Chilvers C. Academic and nonacademic predictors of success on the Nottingham undergraduate medical course 1970-1995. Med Educ 2001;35:1056-64. [17] Cleland JA, Milne A, Sinclair H, Lee AJ. Cohort study on predicting grades: is performance on early MBChB assessments predictive of later undergraduate grades? Med Educ 2008;42:676-83.
tome 39 > n86 > juin 2010