- Email: [email protected]
Objective examining — the cork experience
28
BIOCHEMICAL EDUCATION
April 1976 Vol. 4 No. 2
RERERENCES (References 1-19 were given in Biochemical Education, 1976, 4. 7-8) 20 D. T. Gibson. 1972. Initial reactions in the degradation of aromatic hydrocarbons. In Degradation of synthetic organic" molecules in the biosphere, p. 116. Printing and Publishing Office, National Aademy of Sciences, Washington, D.C. 21 D. M. Jerina, J. W. Daly, B. Witkop, P. Zaltman-Nirenberg and S. Udenfriend. 1968. Formation of 1,2-naphthalene oxide from naphthalene by liver microsomes, J.Amer. Chem. Soci. 90:6525. 22 D. M. Jerina, J. M. Daly, A. M. Jeffrey and D. T. Gibson. 1971. cis-l,2-Dihydroxy-l,2-dihydronaphthalene: a bacterial metabolite from naphthalene. Arch. Biochem. Biophys. 142:394. 23 W. C. Evans, H. N. Fernley and E. Griffiths. 1965. Oxidative metabolism of phenanthrene and anthracene by soil pseudomonads: the ring fission mechanism. Biochem. J. 95:819. 24 M. Blumer. 1961. Benzpyrenes in soil. Science 134:474. 25 M. Alexander. 1967. The breakdown of pesticides in soils. In Agriculture and the quality of our environment (N. C. Brady, ed.), p. 331. Amer. Assoc. for the Advancement of Science, Washington D.C.; P. C. Kearney and D. D. K a u f m a n n . 1972. Microbial degradation of some chlorinated pesticides. In Degradation of synthetic organic molecules in the biosphere, p. 116. Printing and Publishing Office, National Academy of Sciences, Washington, D.C. ;J-M. Bollag. 1972. Biochemical transformations of pesticides by soil fungi. CRC Critical reviews of microbiology 2:35. 26 L. Fowden. 1968. The occurrence and metabolism of carbonhalogen compounds. Proc. Roy. Soc. (Ixmd.) B, 171:5. z7 j. F. Siud and J. F. DeBernardis. 1973. Naturally occurring halogenated organic compounds. Lloydia, 36:107. za L. P. Hager, personal communication. 29 p. Goldman. 1972. Enzymology of carbon-halogen bonds. In Degradation of synthetic organic molecules in the biosphere,
OBJECTIVE E X A M I N I N G - THE CORK E X P E R I E N C E
p. 147. Printing and Publishing Office, National Academy of Sciences, Washington, D.C. 30 j. M. Tiedje, J. M. Duxbury, M. Alexander and J. E. Dawson. 1969. Pathway of degradation of chlorocatechols by Arthrobacter sp. J. Agric. Food Chem. 17:1021. Other relevant papers by this group: ibid, 17:1080 and 16:829. 31 W. C. Evans, B. S. W. Smith, H. N. Fernley and J. I. Davies. 1971. Bacterial metabolism of 2,4-dichlorophenoxyacetate. Biochem. J. 122:543. Other relevant papers: ibid, 122:519; 122:533; and 122:509. 32 H. W. Beam and J. J. Perry. 1974. Microbial degradation of cycloparaffinic hydrocarbons via cometabolism and commensalism. J. Gen. Mierobiol. 82:163. 33 Reort of the Committee on Agriculture and the Environment of the National Academy of Sciences. 1974. Productive agriculture and a quality environment, p. 17. Printing and Publishing Office, Washington, D.C. B. S. Hartley. 1974. Enzyme families. In Evolution in the microbial world. Symposium of the Society for General Microbiology no. 24, p. 151. See also P. H. Clarke: The evolution of enzymes for the utilization of novel substrates, in the same volume, p. 183. 35 G. D. Hegeman and S. L. Rosenberg. 1970. The evolution of bacterial enzyme systems. Annu. Rev. Microbiol. 24:429. 3e N. H. Horowitz. 1945. On the evolution of biochemical synthesis. Proc. Nat. Acad. Sci. USA 31:153; also, 1965. The evolution of biochemical synthesis - - retrospect and prospect. In Evolving genes and proteins (V. Bryson and H. J. Vogel, eds.), p. 15, Academic Press, Inc., New York. 3r W. L. Collinsworth, P. J. C h a p m a n and S. Dagley. 1973. Stereospecific enzymes in the degradation of aromatic compounds by Pseudomonas putida. J. Bacteriol. 133:922.
D. R. H E A D O N and C. A. ROSS D e p a r t m e n t of B i o c h e m i s t r y U n i v e r s i t y College Cork, Ireland
For two reasons the staff of the Biochemistry Department in U.C.C. took the decision in 1972 to introduce an objective form of examining. One was a dissatisfaction with the conventional 'essay' question paper which could only test the candidates' knowledge of a fraction of the course. The other reason was the sheer burden of examining an ever-increasing n u m b e r of students every summer. The size of the problem in 1972 was 165 candidates in the first year course and 40 in the second year course. By 1975, the candidature had risen to 210 and 76. The introduction of objective examining had the added advantage of permitting periodic testing to be held during the course.
Examples of the two types are: (i) The pKa of acetic acid is 4.7. How m u c h undissociated acid exists at pH 5.7? 1.0.1% 2 . 1 . 0 % 3 . 1 0 % 4 . 5 0 % 5. None of the above (if) Match the formulae of amino acids (1-3) with their correct names from the list:
Format of Examination
Questions in Section B are of the relationship analysis type (R.A.). Two statements are made, and the candidate selects the answer from the five possible combinations: 1. if both statements are correct and the second is the correct explanation of the first 2. if both statements are correct 3. if the first statement only is correct 4. if the second statement only is correct 5. if both statements are incorrect
After a period of four years, the form of the Cork objective test has reached a satisfactory stage of development. Papers consist of two types of question grouped into sections A and B; some sixty questions have to be answered and a time limit of ninety minutes is set which is more than adequate for the average student. The questions in section A may be classified into two sub-types: (i) multiple choice (M.C.) - - the selection of the correct answer from among a n u m b e r of plausible distractors. The test is extended by adding to the choice .the possibility that the correct answer has not been offered. (if) matching pairs (M.P.) from two lists and here the question is extended by having more choices in one list than in the other.
1.
alanine
2. 3.
phenylanine tyrosine
H0(~H2Cff.~0H20H 1_.
HS'CH2C
4. 5.
cysteine serine
-- NH2 _ H0CH2CH, cooH 3
~cNoH20 2 H --
Example: Methylation reactions are useful in determining the structure of polysaccharides. In such methylation reactions the carbons involved in forming the glycosidic linkages are methylated.
BIOCHEMICAL EDUCATION
April 1976 Vol. 4 No. 2
The answers to all questions are in the form of a single number (alternatives are ignored) and these are entered in numbered boxes on the answer sheet for ease of transfer to punch cards for correction and analysis by computer.
performance analysis on each occasion of use. Discarded questions are kept for reference purposes. During the period of development of this type of examination, care has been taken to collect up all question books since pre-knowledge of a question would invalidate the performance analysis described above. Once a sufficiently large bank of tried questions has been established, it is envisaged that they will be available to the students for study. New questions will always be subject to analysis before being adopted for future use.
Question Pedormance Analysis A computer programme written in Fortran 1V for use on College's IBM 370/135 computer, permits the recovery of performance of every student on each question and also performance of each question in terms of two parameters, facility value and the discriminatory power.
the the the the
1. Facility value measures the difficulty of a question and is the ratio of correct answers to the total number of answers. The most satisfactory questions have facility values within the range 0.4--0.7. 2. Discriminatory power measures the comparative difficulty of a question for the above average and below average candidates. It is the difference in the facility value of a question as determir,~d for the upper third of the class and for th, Jwer third. The greater toe difference, the more discriminating is the question. In addition, the performance of distractors in multiple choice and matching pair types questions are assessed. Where distractors are not being selected by candidates, they may be discarded in favour of more plausible ones.
Performance of various question types In general the average facility va:ues (a measure of the ease of answering the questions) are high~ for matching pair types questions while the relationship analys,~ type questions prove the most difficult (Fig. la). Having regard to the discriminatory power of the questions (a measure of how questions distinguis~ between the above and below average students), relationship analysis type que~"ms have the highest average discriminatory power and matching types have the lowest (as would be expected from the average facility values) (Fig. lb). Thus, in Cork, relationship analysis type questions tend to be more difficult and more discriminating than the other types used here. Students frequently complain of the difficulty in distinguishing between the choice of answer I or 2 in the relationship analysis type questions. Consideration of the facility values for these question indicates that these are higher for questions where the correct answer is 1 or 2 than for answers 3, 4 or 5. The problem thus appears to be a psychological one where the students apparently consider that knowing both statements were correct should be sufficient to obtain marks rather than to look further for a possible relationship between the correct statements. The questions are compiled by staff members on their particular sections of the course and are filed on cards with a record of their
29
Correlations The results from the first year class of 1974-5 have been analysed by Spearman correlation between the two objective tests held during the course and that held at the end. In addition, the final objective test has been compared with the written paper in Biochemistry (three answers from six questions in ninety minutes). The results are presented for the four groups of students of which the class is comprised (Table 1). A detailed study of these figures is largely of local interest only but some points are worthy of record. In all four groups, there is greater correlation between the second and third objective test than between the first and third which probably reflects an increasing familiarity with the form of the examination in addition to an increased knowledge of biochemistry. The medical group shows a higher correlation than the others which is no doubt due to the fact that these students have been selected on the basis of an above average ability at school. The science group, on the other hand, shows the smallest initial correlation and the greatest improvement which may reflect a greater motivation as the course progresses. The comparison of the third objective test (set on the entire course) with the essay paper which was written at the same examination shows a higher correlation for the two smaller groups, dental and dairy science students, than for the two major groups. The reason for this may be seen from a scatter diagram, Fig. 2, which shows the result from the dental and medical groups. The medical group contains more high scoring students but, because of the technique employed for assessing the essay paper, the scoring of the two types of examination diverges at higher marks in favour of the objective test. The pass mark for these students is 50% We are satisfied that the form of the examination in first year Biochemistry at U.C.C. is giving fair results and also enabling the Department to select its honours students with confidence. Table 1 Correlation o/examination results for First Year Class 1974-5
Group Dental Medical Dairy Science Science
No. 18 74 26 87
Obl/Ob 3 0.30 0.54 0.48 0.21
Obz/Ob3 0.54 0.70 0.63 0.57
Ob3/Essay 0.65 0.47 0.63 0.46
a M.¢.
O
._°¢ M . R
& O
R.A.
o.'2
0.4
o'.,
o'.a
1:0
°'.
O• O •
| "• •
•
$.
:~:..,
F a c i l i t y Value=
g @
O
M.C.
z
R.A
~ d2
01,4
Discriminatory
Figure I.
01.6
01,8
110
Power
The mean and range of (a) facility values and (b) discriminatory power for the three types of question -- multiple choice (M.C.). matchina pairs (M.P.) and relationshin analvsis (R.A.L
Objtctive
Test
Figure 2. Scatter diagram of relationship between the objective test and essay paper taken by medical students (e) and dental students (o) in the class of 1974-5.
BIOCHEMICAL EDUCATION
April 1976 Vol. 4 No. 2
RERERENCES (References 1-19 were given in Biochemical Education, 1976, 4. 7-8) 20 D. T. Gibson. 1972. Initial reactions in the degradation of aromatic hydrocarbons. In Degradation of synthetic organic" molecules in the biosphere, p. 116. Printing and Publishing Office, National Aademy of Sciences, Washington, D.C. 21 D. M. Jerina, J. W. Daly, B. Witkop, P. Zaltman-Nirenberg and S. Udenfriend. 1968. Formation of 1,2-naphthalene oxide from naphthalene by liver microsomes, J.Amer. Chem. Soci. 90:6525. 22 D. M. Jerina, J. M. Daly, A. M. Jeffrey and D. T. Gibson. 1971. cis-l,2-Dihydroxy-l,2-dihydronaphthalene: a bacterial metabolite from naphthalene. Arch. Biochem. Biophys. 142:394. 23 W. C. Evans, H. N. Fernley and E. Griffiths. 1965. Oxidative metabolism of phenanthrene and anthracene by soil pseudomonads: the ring fission mechanism. Biochem. J. 95:819. 24 M. Blumer. 1961. Benzpyrenes in soil. Science 134:474. 25 M. Alexander. 1967. The breakdown of pesticides in soils. In Agriculture and the quality of our environment (N. C. Brady, ed.), p. 331. Amer. Assoc. for the Advancement of Science, Washington D.C.; P. C. Kearney and D. D. K a u f m a n n . 1972. Microbial degradation of some chlorinated pesticides. In Degradation of synthetic organic molecules in the biosphere, p. 116. Printing and Publishing Office, National Academy of Sciences, Washington, D.C. ;J-M. Bollag. 1972. Biochemical transformations of pesticides by soil fungi. CRC Critical reviews of microbiology 2:35. 26 L. Fowden. 1968. The occurrence and metabolism of carbonhalogen compounds. Proc. Roy. Soc. (Ixmd.) B, 171:5. z7 j. F. Siud and J. F. DeBernardis. 1973. Naturally occurring halogenated organic compounds. Lloydia, 36:107. za L. P. Hager, personal communication. 29 p. Goldman. 1972. Enzymology of carbon-halogen bonds. In Degradation of synthetic organic molecules in the biosphere,
OBJECTIVE E X A M I N I N G - THE CORK E X P E R I E N C E
p. 147. Printing and Publishing Office, National Academy of Sciences, Washington, D.C. 30 j. M. Tiedje, J. M. Duxbury, M. Alexander and J. E. Dawson. 1969. Pathway of degradation of chlorocatechols by Arthrobacter sp. J. Agric. Food Chem. 17:1021. Other relevant papers by this group: ibid, 17:1080 and 16:829. 31 W. C. Evans, B. S. W. Smith, H. N. Fernley and J. I. Davies. 1971. Bacterial metabolism of 2,4-dichlorophenoxyacetate. Biochem. J. 122:543. Other relevant papers: ibid, 122:519; 122:533; and 122:509. 32 H. W. Beam and J. J. Perry. 1974. Microbial degradation of cycloparaffinic hydrocarbons via cometabolism and commensalism. J. Gen. Mierobiol. 82:163. 33 Reort of the Committee on Agriculture and the Environment of the National Academy of Sciences. 1974. Productive agriculture and a quality environment, p. 17. Printing and Publishing Office, Washington, D.C. B. S. Hartley. 1974. Enzyme families. In Evolution in the microbial world. Symposium of the Society for General Microbiology no. 24, p. 151. See also P. H. Clarke: The evolution of enzymes for the utilization of novel substrates, in the same volume, p. 183. 35 G. D. Hegeman and S. L. Rosenberg. 1970. The evolution of bacterial enzyme systems. Annu. Rev. Microbiol. 24:429. 3e N. H. Horowitz. 1945. On the evolution of biochemical synthesis. Proc. Nat. Acad. Sci. USA 31:153; also, 1965. The evolution of biochemical synthesis - - retrospect and prospect. In Evolving genes and proteins (V. Bryson and H. J. Vogel, eds.), p. 15, Academic Press, Inc., New York. 3r W. L. Collinsworth, P. J. C h a p m a n and S. Dagley. 1973. Stereospecific enzymes in the degradation of aromatic compounds by Pseudomonas putida. J. Bacteriol. 133:922.
D. R. H E A D O N and C. A. ROSS D e p a r t m e n t of B i o c h e m i s t r y U n i v e r s i t y College Cork, Ireland
For two reasons the staff of the Biochemistry Department in U.C.C. took the decision in 1972 to introduce an objective form of examining. One was a dissatisfaction with the conventional 'essay' question paper which could only test the candidates' knowledge of a fraction of the course. The other reason was the sheer burden of examining an ever-increasing n u m b e r of students every summer. The size of the problem in 1972 was 165 candidates in the first year course and 40 in the second year course. By 1975, the candidature had risen to 210 and 76. The introduction of objective examining had the added advantage of permitting periodic testing to be held during the course.
Examples of the two types are: (i) The pKa of acetic acid is 4.7. How m u c h undissociated acid exists at pH 5.7? 1.0.1% 2 . 1 . 0 % 3 . 1 0 % 4 . 5 0 % 5. None of the above (if) Match the formulae of amino acids (1-3) with their correct names from the list:
Format of Examination
Questions in Section B are of the relationship analysis type (R.A.). Two statements are made, and the candidate selects the answer from the five possible combinations: 1. if both statements are correct and the second is the correct explanation of the first 2. if both statements are correct 3. if the first statement only is correct 4. if the second statement only is correct 5. if both statements are incorrect
After a period of four years, the form of the Cork objective test has reached a satisfactory stage of development. Papers consist of two types of question grouped into sections A and B; some sixty questions have to be answered and a time limit of ninety minutes is set which is more than adequate for the average student. The questions in section A may be classified into two sub-types: (i) multiple choice (M.C.) - - the selection of the correct answer from among a n u m b e r of plausible distractors. The test is extended by adding to the choice .the possibility that the correct answer has not been offered. (if) matching pairs (M.P.) from two lists and here the question is extended by having more choices in one list than in the other.
1.
alanine
2. 3.
phenylanine tyrosine
H0(~H2Cff.~0H20H 1_.
HS'CH2C
4. 5.
cysteine serine
-- NH2 _ H0CH2CH, cooH 3
~cNoH20 2 H --
Example: Methylation reactions are useful in determining the structure of polysaccharides. In such methylation reactions the carbons involved in forming the glycosidic linkages are methylated.
BIOCHEMICAL EDUCATION
April 1976 Vol. 4 No. 2
The answers to all questions are in the form of a single number (alternatives are ignored) and these are entered in numbered boxes on the answer sheet for ease of transfer to punch cards for correction and analysis by computer.
performance analysis on each occasion of use. Discarded questions are kept for reference purposes. During the period of development of this type of examination, care has been taken to collect up all question books since pre-knowledge of a question would invalidate the performance analysis described above. Once a sufficiently large bank of tried questions has been established, it is envisaged that they will be available to the students for study. New questions will always be subject to analysis before being adopted for future use.
Question Pedormance Analysis A computer programme written in Fortran 1V for use on College's IBM 370/135 computer, permits the recovery of performance of every student on each question and also performance of each question in terms of two parameters, facility value and the discriminatory power.
the the the the
1. Facility value measures the difficulty of a question and is the ratio of correct answers to the total number of answers. The most satisfactory questions have facility values within the range 0.4--0.7. 2. Discriminatory power measures the comparative difficulty of a question for the above average and below average candidates. It is the difference in the facility value of a question as determir,~d for the upper third of the class and for th, Jwer third. The greater toe difference, the more discriminating is the question. In addition, the performance of distractors in multiple choice and matching pair types questions are assessed. Where distractors are not being selected by candidates, they may be discarded in favour of more plausible ones.
Performance of various question types In general the average facility va:ues (a measure of the ease of answering the questions) are high~ for matching pair types questions while the relationship analys,~ type questions prove the most difficult (Fig. la). Having regard to the discriminatory power of the questions (a measure of how questions distinguis~ between the above and below average students), relationship analysis type que~"ms have the highest average discriminatory power and matching types have the lowest (as would be expected from the average facility values) (Fig. lb). Thus, in Cork, relationship analysis type questions tend to be more difficult and more discriminating than the other types used here. Students frequently complain of the difficulty in distinguishing between the choice of answer I or 2 in the relationship analysis type questions. Consideration of the facility values for these question indicates that these are higher for questions where the correct answer is 1 or 2 than for answers 3, 4 or 5. The problem thus appears to be a psychological one where the students apparently consider that knowing both statements were correct should be sufficient to obtain marks rather than to look further for a possible relationship between the correct statements. The questions are compiled by staff members on their particular sections of the course and are filed on cards with a record of their
29
Correlations The results from the first year class of 1974-5 have been analysed by Spearman correlation between the two objective tests held during the course and that held at the end. In addition, the final objective test has been compared with the written paper in Biochemistry (three answers from six questions in ninety minutes). The results are presented for the four groups of students of which the class is comprised (Table 1). A detailed study of these figures is largely of local interest only but some points are worthy of record. In all four groups, there is greater correlation between the second and third objective test than between the first and third which probably reflects an increasing familiarity with the form of the examination in addition to an increased knowledge of biochemistry. The medical group shows a higher correlation than the others which is no doubt due to the fact that these students have been selected on the basis of an above average ability at school. The science group, on the other hand, shows the smallest initial correlation and the greatest improvement which may reflect a greater motivation as the course progresses. The comparison of the third objective test (set on the entire course) with the essay paper which was written at the same examination shows a higher correlation for the two smaller groups, dental and dairy science students, than for the two major groups. The reason for this may be seen from a scatter diagram, Fig. 2, which shows the result from the dental and medical groups. The medical group contains more high scoring students but, because of the technique employed for assessing the essay paper, the scoring of the two types of examination diverges at higher marks in favour of the objective test. The pass mark for these students is 50% We are satisfied that the form of the examination in first year Biochemistry at U.C.C. is giving fair results and also enabling the Department to select its honours students with confidence. Table 1 Correlation o/examination results for First Year Class 1974-5
Group Dental Medical Dairy Science Science
No. 18 74 26 87
Obl/Ob 3 0.30 0.54 0.48 0.21
Obz/Ob3 0.54 0.70 0.63 0.57
Ob3/Essay 0.65 0.47 0.63 0.46
a M.¢.
O
._°¢ M . R
& O
R.A.
o.'2
0.4
o'.,
o'.a
1:0
°'.
O• O •
| "• •
•
$.
:~:..,
F a c i l i t y Value=
g @
O
M.C.
z
R.A
~ d2
01,4
Discriminatory
Figure I.
01.6
01,8
110
Power
The mean and range of (a) facility values and (b) discriminatory power for the three types of question -- multiple choice (M.C.). matchina pairs (M.P.) and relationshin analvsis (R.A.L
Objtctive
Test
Figure 2. Scatter diagram of relationship between the objective test and essay paper taken by medical students (e) and dental students (o) in the class of 1974-5.