- Email: [email protected]
The evaluation of integrated courseware: Can interactive molecular modelling help students understand three-dimensional chemistry?
ComputersEduc. Vol. 26, No. 4, pp. 233-239, 1996
Pergamon
0360-1315(95)00074-7
Copyright © 1996 Elsevier Science Ltd Printed in Great Britain. All rights reserved 0360-1315/96 $15.00 + 0.00
THE EVALUATION OF INTEGRATED COURSEWARE: CAN INTERACTIVE MOLECULAR MODELLING HELP STUDENTS UNDERSTAND THREE-DIMENSIONAL CHEMISTRY? R. T. HYDE, 1 P. N. SHAW l and D. E, JACKSON 2 ~Department of Pharmaceutical Sciences, University of Nottingham, University Park, Nottingham NG7 2RD England and 2Oxford Molecular Ltd, Magdalen Centre, Oxford Science Park, Oxford OX4 4GA England
(Received 7August 1995; accepted 5 October 1995) Abstract--Molecular visualization is a prerequisite to the understanding of a diverse range of topics in the undergraduate pharmacy curriculum, from the stereochemistry of organic compounds to receptor ligand interaction. Unfortunately, traditional teaching does not allow all students to develop the ability to visualize three-dimensional molecular structure. We have developed a computer package which integrates tutorial instruction with interactive molecular modelling algorithms. The designation of R and S to chiral molecules was chosen as the subject material for the prototype courseware. We describe the design and implementation of an integrated evaluation scheme following the incorporation of the program into the undergraduate pharmacy curriculum at several U.K. universities. The results suggest that a usable and engaging interface had been produced, that provided an intuitive front-end to the complexity of the molecular modelling tools. Possible limitations in the ability of students to form mental representations of three-dimensional molecular structures have been identified. Further work on the effects of student attributes, such as gender and handedness, on effective use of the courseware is suggested.
INTRODUCTION
Molecular visualization is a prerequisite to the understanding of a diverse range of topics in the undergraduate pharmacy curriculum. Unfortunately, spatial ability is very difficult to teach in a traditional lesson format and this leaves many students without the ability to visualize three-dimensional (3D) molecular structures [1, 2]. Recently, increasingly attractive price/performance ratios have made personal computers capable of displaying interactive molecular models in real-time available for undergraduate courses. Therefore, the integration of molecular modelling into computer-aided learning (CAL) offers a potential solution to the traditional problems of teaching spatial chemistry. Instructional courseware incorporating interactive molecular modelling must be thoroughly evaluated to justify its incorporation into the curriculum. As part of this evaluation process we need to examine whether students find the molecular rotation device intuitive to use. We also need to discover how the device is used during the learning process and whether spatial ability improves following use of the program. The traditional methodology for evaluating CAL has been to compare the new technology with the old, in so-called media comparison studies [3]. The lack of definitive results on the effectiveness of CAL resulting from such work has lead to more qualitative evaluation methodologies being used. Rather than purely seeking to justify the integration of the courseware into the curriculum, more specific questions, such as, "What features of CAL make it effective?", are addressed [4]. Integrated data collection schemes, combining qualitative and quantitative data gathering, are often used in such studies to obtain the broadest picture of courseware use [5, 6]. In addition, the evaluation of CAL in a natural environment is recommended, as this increases the ecological validity of the data obtained. To achieve this, the stlbject material must be of direct relevance to the students and the courseware should be implemented in a realistic setting [4, 7, 8]. We describe the summative evaluation of an integrated CAL package, following its incorporation into the undergraduate pharmacy curriculum at several U.K. universities. Our aim was to show that the ability to interactively manipulate molecular structures in tutorial courseware helps with the 3D visualization of molecular chemistry concepts. We have employed an integrated evaluation scheme to show how multiple data collection provides a more complete scenario of the effectiveness of the molecular modelling !tools. 233
234
R.T. HYDE et
File
Section
Molecule
Tools
al.
Help
Fig. 1. The self-assessmentsection of the packageshowing the 3D moleculedisplay and functional grouping of molecular modelling and navigationaloperations. COURSEWARE DESIGN The programming and design of the package has been described in detail elsewhere [9]. Authorware Professional 2.0 [10] was used for the tutorial component of the courseware. To provide the molecular modelling functionality, Nemesis 2.0 [11] was re-coded as a dynamic link library (DLL) and linked externally to the Authorware program. The design process involved disguising the complexity of the modelling package by providing a discrete subset of its full functionality. The courseware was divided into tutorial sections that contained prerequisite material and a selfassessment section where knowledge acquisition was tested. The molecular modelling tools were introduced in the tutorial sections, to ensure that the student would be competent in their use during the self-assessment section. Figure 1 shows the screen layout during a typical question sequence. Functional grouping of operations was used to simplify the interface. Molecular modelling tools are grouped down the left side and all navigation is achieved using the buttons at the bottom of the screen. The display of the molecule in the central window is controlled from the DLL. It is possible to rotate, scale or translate the molecule and to alter its representation from stick to ball and stick or space-fill. The virtual sphere control device [12], as employed by Nemesis, was used in the DLL to provide twodimensional (2D) rotation of molecules. This device provides an intuitive interface for the real-time manipulation of 3D objects. Chert [13] found that it enabled faster and more natural rotation than any other device in a comparative study involving novice users [13]. The ability to hide or display hydrogens and labels on a molecule was provided as an exploratory tool. Depth cueing of molecules was additionally used to help with 3D visualization. The atom coordinates and highlighting features of the modelling program allowed a student to click the mouse directly on an atom in answer to a question. This enabled immediate feedback to be given, as an atom type labelt appearing next to the selected atom. It was possible to make unlimited attempts to answer any of the questions. In tNemesis 2.0 displays atom types as the atomic symbol(e.g. C for carbon) together with the numberof the atom in the molecule. This enables an atom to be uniquely identified.
Interactive molecularchemistry
235
addition, questions could be omitted and the courseware could be quit at any point in the instructional sequence. The designation of R and S to chiral molecules was chosen as the subject material for the evaluation of the prototype courseware. The instructional sequence employed for the assignment of a designation was that taught for traditional 2D media, as follows: Step 1. Identify the chiral carbon atom; Step 2. Assign priorities 1-4 to the four groups surrounding the chiral carbon atom; Step 3. Visualize group 4 (the lowest priority group) facing away from you; Step 4. Determine the direction of rotation going from group 1 to 2 to 3. If the direction of rotation is clockwise, an R designation is given to the molecule, otherwise an S designation is given. The molecules were initially presented in one of two orientations regarding the spatial positioning of the lowest priority group• This group faced alternately out of and into the screen for successive questions, being off-set slightly from the z axis, so as not to be hidden by other atoms• Students were not prompted to manipulate the molecules before step 4 or at any other point in the instructional sequence. SUBJECTS AND IMPLEMENTATION The prototype package was distributed to U.K. universities by the Pharmacy Consortium for Computer-Aided Learning (PCCAL). The data presented here result from its integration into BPh~rmI modules covering basic stereochemistry at Nottingham and Manchester Universities• The students Used the courseware while the module was being taught. A Computers & Pharmacy module, covering basic personal computer skills, had been taken by all the students at Nottingham• The minimum requirements for running the courseware were an 80386SX microprocessor and 4 MB RAM. The program was installed on 24 networked IBM compatible PCs within the CAL laboratory at Nottingham• Since the rating scale was optional and the questions could be skipped, variable numbers of students Were used in the data analysis described below. The number of participants are indicated for each set of results• EVALUATION METHODOLOGY To assess the mastery of R and S designation, cumulative scores obtained during self-assessment were recorded for individual students• A correct answer was recorded when the correct atom was identified in response to a question. Alternatively, for an R or S designation, a correct answer resulted from a click on one of the R or S labelled buttons appearing below the molecule window• An incorrect attempt was recorded following a click on the wrong atom or on the incorrect button for a designation. The percentage mastery, for each of the question types, was calculated as: (correct answers/questions answered) x 100. An optional 5-point Likert style [14] rating scale was presented in a Windows dialogue box, at the point when a student quit the program• The scale ranged from Strongly disagree (1), through Neutral (3), to Strongly agree (5). Previous computer and subject experience were assessed before the rating scale was administered. Eight statements then assessed student attitudes towards the interface and the molecular modelling tools. Finally, two open questions allowed a student to enter text descriptions of particularly good or bad features about the program. System monitoring of student mouse operations during self-assessment was used to record the Use of the virtual sphere during the instructional sequence. The data obtained for chiral centre tdentlfic~tlon, priority assignment and designation questions were appended to separate data files. IndNidual mouse drags were accumulated, and as each correct answer was received, the total distancg the molecule had been rotated was recorded. The drag distance was calculated in screen pixels as: x/((MouseUpX - MouseDownX) 2 + (MouseUpY - MouseDownY;)2). •
.
J
.
RESULTS The average mastery scores obtained during self-assessment for the three question types are shown in Table 1. Students gave more correct first attempts at questions the further through the instrudtional
R. T. HYDE et al.
236
Table 1. Student mastery of self-assessment questions (n = 13) Question type Chiral centre identification Priority assignment R and S designation Overall
Mastery (%) 71 77 88 83
(7.7) (6.9) (4.6) (3.9)
Table 2. Percentage responses to rating scale (n = 19) Rating scale statements
Agreement
(~)
I am a novice in the use of interactive computer systems I am familiar with the subject matter in the courseware The screen displays were uncluttered and easy to understand At times, I didn't know where I was in the courseware I felt in control of my progress while I was using the courseware I would have liked more interaction in the courseware The ability to rotate molecules in 3D made the courseware more interesting Rotating the molecules didn't help me learn more 3D molecular modelling would help me understand other chemistry topics
53 79 77 43 78 78 88 40 79
(6.7) (2.9) (4.4) (5.1) (3.4) (4.0) (3.2) (4.0) (3.9)
Table 3. Categorized responses to the open question, "Is there one thing that stands out as being really good about the package?" (n = 76) Response "Manipulation of molecular models" "Helps teach R and S stereochemistry" "The screen displays" Others No response
Respondents (%) 47 4 4 14 31
Table 4. Categorized responses to the open question, "Is there one thing that stands out as being really bad about the package?" (n = 76) Response "Questions too similar" "Lack of atom colour key" Others No response
Respondents (%) 15 4 10 71
sequence they worked, from chiral centre identification (71%) to designation (88%). The overall mastery of R and S designation was high (83%). The results from the rating scale are shown in Table 2. Students had intermediate computer experience (53%) and were already familiar with R and S designation (79%). The screen displays were generally intuitive (77%) and the possibility of even more interactivity was attractive (78%). The students were occasionally disorientated in the courseware (43%), though they felt in control of their progress most of the time (78%). The facility to rotate molecules certainly made the instruction more interesting (88%) and manipulation of structures improved the learning experience (60%). The prospect of further programs incorporating interactive molecular modelling to teach chemistry was an attractive prospect (79%). The results from the two open questions appended to the rating scale are summarized in Tables 3 and 4. Table 3 shows that 47% of the students picked out manipulation of molecular models as the most attractive feature of the courseware, with 31% not responding to the question. Conversely, Table 4 shows that the similarity of the questions was the least attractive feature (15%), with 71% not responding to this question. The system monitoring of virtual sphere usage during self-assessment has been categorized by question type in Fig. 2. Clearly, molecules were dragged least during priority assignment (26 pixels) and most
Interactive molecular chemistry
237
• Chiral centres [] Priorities
gq Designations •-~ 100 so ¢1 r~ "¢::1
~
6o
=
40
0
o
~
2o
< o Fig. 2. Mean mouse drag distances for chiral centre identification (n -85), priority assignment (n- 571) and final designation (n = 144). during R or S designation (93 pixels). During the identification of chiral centres, molecules were dragged by more variable distances, with the average lying between those for priority assignment and the final designation (73 pixels). Figure 3 shows the virtual sphere usage data categorized by initial molecule orientation. The mouse drag distances for a set of questions on a single molecule have been accumulated. Clearly, molecules Were dragged more when the lowest priority group was initially presented as forward facing (111 pixel~) as compared to backward facing (44 pixels). Welch's unpaired t-test revealed that this difference was significant (t = 2.03, P < 0.05). DISCUSSION The self-assessment scores showed that students achieved a high level of mastery of R and S designation. The weakest area was the identification of chiral centres. At this stage in the sequence the students were not focused on any particular region of the molecule and were possibly clicking randomly on several atoms before they correctly identified the chiral centre. The rating scale results show that a usable and intuitive interface has been designed for the courseware. • Lowest priority backwards [] Lowest priority forwards 140 e~
"~
-~
120
1oo ~
80
"~
60
E
4O
0
~
°
-
~
2o o
lB /l
Fig. 3. Mean mouse drag distances for initially forward facing (n- 46) and backward facing (n- 30) lowest priority groups.
238
R.T. HYDE et al.
The molecular modelling tools have been transparently integrated with the instruction and most of the students found this an engaging and effective learning environment. The open questions gathered more casual observations about the package design. By far the most appealing feature was the ability to manipulate molecular models, which confirmed the finding from the rating scale. Conversely, the similarity of the questions was identified by the students as an unattractive feature. This result was predictable, since in the prototype courseware the questions were of the same type and in the same sequence by necessity, to enable a standard set of virtual sphere usage data to be collected. The virtual sphere usage data showed that students were intuitively manipulating molecules to aid their visualization of 3D molecular structure during the instructional sequence. In keeping with the traditionally taught method of assignment, molecules were rotated most during the final designation. At this stage the students had to visualize the lowest priority group facing away from them. They rotated the molecules to help them achieve this. These data also revealed that molecules were rotated significantly more when the lowest priority group was initially presented as forward facing. This result highlights a possible weakness in the students' ability to form mental images of the 3D structure of molecules in this orientation. They were either rotating the lowest priority group to face away from them or spinning the molecule to enable them to visualize the spatial arrangement of the groups surrounding the chiral centre. Further examination of this stage in the designation process is necessary. This would reveal how the students are rotating the molecules to visualize 3D structure and whether the interface or instruction need modification to account for any difficulties. The emphasis of this study was to provide a realistic environment for the evaluation of the courseware. Although this has given ecological validity to the results, there was an inherent lack of control over data collection. Consequently, insufficient data were obtained to enable 3D manipulation of molecules to be correlated with scores obtained in the self-assessment section. A study of this relationship would reveal just how effective 3D molecular modelling tools are in the learning of spatial chemistry concepts. A more quantitative, controlled study would provide insights into this important area. A further area for examination would be the effect of student attributes on the use of the courseware. For example, Barke [15] reported that boys have better spatial ability than girls of an equivalent age and ability improves in both sexes with increasing age [15]. McGee [16] found that spatial ability was also related to degree of ambidexterity, such that highly ambidextrous boys performed best in mental rotation tests [16]. Results from a study of student characteristics and performance would confirm that both high and low spatial ability students could benefit from using the program. The integrated evaluation scheme used in this study has provided substantial evidence concerning the effectiveness of the courseware. Both the qualitative and quantitative data show that 3D molecular modelling can be integrated with tutorial instruction to help students understand spatial chemistry. We have also gained an insight into how the molecular modelling tools were used during the learning process and identified weaknesses in the design of the program and the spatial abilities of the students. These findings will be used to enhance future versions of the package. Acknowledgements--This
work has been supported by Oxford Molecular Ltd and the Engineering and Physical Sciences Research
Council (EPSRC). REFERENCES 1. Henkel J. G., PC-based molecular modelling in the classroom: application to medicinal chemistry and biochemistry.J. molec. Graph. 9, No. 3, 11-17 (1991). 2. Shubber K. E., Learning the visualisation of rotations in diagrams of three dimensional structures. Res. Sci. Technol. Educ. 8, 145-154 (1990). 3. Reeves T. C., Research support for interactive multimedia: existing foundations and new directions. In Interactive Multimedia: Practice and Promise (Edited by Latchem C., Williamson J. and Henderson-Lancett L.). Kogan Page, London (1993). 4. Colorado R. J., Computer-assisted instruction research: a critical assessment.J. Res. Comput. Educ. 20, 226--233 (1988). 5. Reeves T. C., Evaluation strategies for interactive multimedia products. In lnstructionalInteractive Multimedia: A Developer's Guide (Edited by Cates W. M.) (1994). 6. Macleod M., An introduction to usability evaluation. NPL report 102/92. National Physics Laboratory, Middlesex (1992). 7. Topham P. J., Methodologies for evaluating interactive video courseware. Interact. Learn. Int. 5, 19-23 (1989). 8. Reeves T. C., Computer modelling: a research tool for computer based instruction. Interact. Learn. Int. 8, 3-13 (1992). 9. Hyde R. T., Shaw P. N., Jackson D. E. and Woods K., Integration of molecular modelling algorithms with tutorial instruction: the design of an interactive three-dimensional computer-aidedlearning environment for exploring molecular structure. J. chem. Educ. In press. 10. Macromedia Inc., Authorwarc Professional V2.0 [computer program]. San Francisco, Calif. (1993).
Interactive molecular chemistry
239
11. Oxford Molecular Ltd, Nemesis V2.0 [computer program]. Oxford Molecular Ltd, Oxford (1994). 12. Chen M., 3-D rotation using a 2-D input device. Develop (TheApple TechnicalJournal) 14, 40-53 (1993). 13. Chen M., Mountford S. J. and Sellen A., A study in interactive 3-D rotation using 2-D control devices. Comp. Graph. 22, No. 4, 121-129 (1988). 14. Likert R. A., A technique for the measurement of attitudes. Archs Psychol. 140, 55 (1932). 15. Barke H. D., Chemical education and spatial ability. J. chem. Educ. 70, 968-971 (1993). 16. McGee M. G., Handedness and mental rotation. Percept. Motor Skills 47, 641--642 (1978).
Pergamon
0360-1315(95)00074-7
Copyright © 1996 Elsevier Science Ltd Printed in Great Britain. All rights reserved 0360-1315/96 $15.00 + 0.00
THE EVALUATION OF INTEGRATED COURSEWARE: CAN INTERACTIVE MOLECULAR MODELLING HELP STUDENTS UNDERSTAND THREE-DIMENSIONAL CHEMISTRY? R. T. HYDE, 1 P. N. SHAW l and D. E, JACKSON 2 ~Department of Pharmaceutical Sciences, University of Nottingham, University Park, Nottingham NG7 2RD England and 2Oxford Molecular Ltd, Magdalen Centre, Oxford Science Park, Oxford OX4 4GA England
(Received 7August 1995; accepted 5 October 1995) Abstract--Molecular visualization is a prerequisite to the understanding of a diverse range of topics in the undergraduate pharmacy curriculum, from the stereochemistry of organic compounds to receptor ligand interaction. Unfortunately, traditional teaching does not allow all students to develop the ability to visualize three-dimensional molecular structure. We have developed a computer package which integrates tutorial instruction with interactive molecular modelling algorithms. The designation of R and S to chiral molecules was chosen as the subject material for the prototype courseware. We describe the design and implementation of an integrated evaluation scheme following the incorporation of the program into the undergraduate pharmacy curriculum at several U.K. universities. The results suggest that a usable and engaging interface had been produced, that provided an intuitive front-end to the complexity of the molecular modelling tools. Possible limitations in the ability of students to form mental representations of three-dimensional molecular structures have been identified. Further work on the effects of student attributes, such as gender and handedness, on effective use of the courseware is suggested.
INTRODUCTION
Molecular visualization is a prerequisite to the understanding of a diverse range of topics in the undergraduate pharmacy curriculum. Unfortunately, spatial ability is very difficult to teach in a traditional lesson format and this leaves many students without the ability to visualize three-dimensional (3D) molecular structures [1, 2]. Recently, increasingly attractive price/performance ratios have made personal computers capable of displaying interactive molecular models in real-time available for undergraduate courses. Therefore, the integration of molecular modelling into computer-aided learning (CAL) offers a potential solution to the traditional problems of teaching spatial chemistry. Instructional courseware incorporating interactive molecular modelling must be thoroughly evaluated to justify its incorporation into the curriculum. As part of this evaluation process we need to examine whether students find the molecular rotation device intuitive to use. We also need to discover how the device is used during the learning process and whether spatial ability improves following use of the program. The traditional methodology for evaluating CAL has been to compare the new technology with the old, in so-called media comparison studies [3]. The lack of definitive results on the effectiveness of CAL resulting from such work has lead to more qualitative evaluation methodologies being used. Rather than purely seeking to justify the integration of the courseware into the curriculum, more specific questions, such as, "What features of CAL make it effective?", are addressed [4]. Integrated data collection schemes, combining qualitative and quantitative data gathering, are often used in such studies to obtain the broadest picture of courseware use [5, 6]. In addition, the evaluation of CAL in a natural environment is recommended, as this increases the ecological validity of the data obtained. To achieve this, the stlbject material must be of direct relevance to the students and the courseware should be implemented in a realistic setting [4, 7, 8]. We describe the summative evaluation of an integrated CAL package, following its incorporation into the undergraduate pharmacy curriculum at several U.K. universities. Our aim was to show that the ability to interactively manipulate molecular structures in tutorial courseware helps with the 3D visualization of molecular chemistry concepts. We have employed an integrated evaluation scheme to show how multiple data collection provides a more complete scenario of the effectiveness of the molecular modelling !tools. 233
234
R.T. HYDE et
File
Section
Molecule
Tools
al.
Help
Fig. 1. The self-assessmentsection of the packageshowing the 3D moleculedisplay and functional grouping of molecular modelling and navigationaloperations. COURSEWARE DESIGN The programming and design of the package has been described in detail elsewhere [9]. Authorware Professional 2.0 [10] was used for the tutorial component of the courseware. To provide the molecular modelling functionality, Nemesis 2.0 [11] was re-coded as a dynamic link library (DLL) and linked externally to the Authorware program. The design process involved disguising the complexity of the modelling package by providing a discrete subset of its full functionality. The courseware was divided into tutorial sections that contained prerequisite material and a selfassessment section where knowledge acquisition was tested. The molecular modelling tools were introduced in the tutorial sections, to ensure that the student would be competent in their use during the self-assessment section. Figure 1 shows the screen layout during a typical question sequence. Functional grouping of operations was used to simplify the interface. Molecular modelling tools are grouped down the left side and all navigation is achieved using the buttons at the bottom of the screen. The display of the molecule in the central window is controlled from the DLL. It is possible to rotate, scale or translate the molecule and to alter its representation from stick to ball and stick or space-fill. The virtual sphere control device [12], as employed by Nemesis, was used in the DLL to provide twodimensional (2D) rotation of molecules. This device provides an intuitive interface for the real-time manipulation of 3D objects. Chert [13] found that it enabled faster and more natural rotation than any other device in a comparative study involving novice users [13]. The ability to hide or display hydrogens and labels on a molecule was provided as an exploratory tool. Depth cueing of molecules was additionally used to help with 3D visualization. The atom coordinates and highlighting features of the modelling program allowed a student to click the mouse directly on an atom in answer to a question. This enabled immediate feedback to be given, as an atom type labelt appearing next to the selected atom. It was possible to make unlimited attempts to answer any of the questions. In tNemesis 2.0 displays atom types as the atomic symbol(e.g. C for carbon) together with the numberof the atom in the molecule. This enables an atom to be uniquely identified.
Interactive molecularchemistry
235
addition, questions could be omitted and the courseware could be quit at any point in the instructional sequence. The designation of R and S to chiral molecules was chosen as the subject material for the evaluation of the prototype courseware. The instructional sequence employed for the assignment of a designation was that taught for traditional 2D media, as follows: Step 1. Identify the chiral carbon atom; Step 2. Assign priorities 1-4 to the four groups surrounding the chiral carbon atom; Step 3. Visualize group 4 (the lowest priority group) facing away from you; Step 4. Determine the direction of rotation going from group 1 to 2 to 3. If the direction of rotation is clockwise, an R designation is given to the molecule, otherwise an S designation is given. The molecules were initially presented in one of two orientations regarding the spatial positioning of the lowest priority group• This group faced alternately out of and into the screen for successive questions, being off-set slightly from the z axis, so as not to be hidden by other atoms• Students were not prompted to manipulate the molecules before step 4 or at any other point in the instructional sequence. SUBJECTS AND IMPLEMENTATION The prototype package was distributed to U.K. universities by the Pharmacy Consortium for Computer-Aided Learning (PCCAL). The data presented here result from its integration into BPh~rmI modules covering basic stereochemistry at Nottingham and Manchester Universities• The students Used the courseware while the module was being taught. A Computers & Pharmacy module, covering basic personal computer skills, had been taken by all the students at Nottingham• The minimum requirements for running the courseware were an 80386SX microprocessor and 4 MB RAM. The program was installed on 24 networked IBM compatible PCs within the CAL laboratory at Nottingham• Since the rating scale was optional and the questions could be skipped, variable numbers of students Were used in the data analysis described below. The number of participants are indicated for each set of results• EVALUATION METHODOLOGY To assess the mastery of R and S designation, cumulative scores obtained during self-assessment were recorded for individual students• A correct answer was recorded when the correct atom was identified in response to a question. Alternatively, for an R or S designation, a correct answer resulted from a click on one of the R or S labelled buttons appearing below the molecule window• An incorrect attempt was recorded following a click on the wrong atom or on the incorrect button for a designation. The percentage mastery, for each of the question types, was calculated as: (correct answers/questions answered) x 100. An optional 5-point Likert style [14] rating scale was presented in a Windows dialogue box, at the point when a student quit the program• The scale ranged from Strongly disagree (1), through Neutral (3), to Strongly agree (5). Previous computer and subject experience were assessed before the rating scale was administered. Eight statements then assessed student attitudes towards the interface and the molecular modelling tools. Finally, two open questions allowed a student to enter text descriptions of particularly good or bad features about the program. System monitoring of student mouse operations during self-assessment was used to record the Use of the virtual sphere during the instructional sequence. The data obtained for chiral centre tdentlfic~tlon, priority assignment and designation questions were appended to separate data files. IndNidual mouse drags were accumulated, and as each correct answer was received, the total distancg the molecule had been rotated was recorded. The drag distance was calculated in screen pixels as: x/((MouseUpX - MouseDownX) 2 + (MouseUpY - MouseDownY;)2). •
.
J
.
RESULTS The average mastery scores obtained during self-assessment for the three question types are shown in Table 1. Students gave more correct first attempts at questions the further through the instrudtional
R. T. HYDE et al.
236
Table 1. Student mastery of self-assessment questions (n = 13) Question type Chiral centre identification Priority assignment R and S designation Overall
Mastery (%) 71 77 88 83
(7.7) (6.9) (4.6) (3.9)
Table 2. Percentage responses to rating scale (n = 19) Rating scale statements
Agreement
(~)
I am a novice in the use of interactive computer systems I am familiar with the subject matter in the courseware The screen displays were uncluttered and easy to understand At times, I didn't know where I was in the courseware I felt in control of my progress while I was using the courseware I would have liked more interaction in the courseware The ability to rotate molecules in 3D made the courseware more interesting Rotating the molecules didn't help me learn more 3D molecular modelling would help me understand other chemistry topics
53 79 77 43 78 78 88 40 79
(6.7) (2.9) (4.4) (5.1) (3.4) (4.0) (3.2) (4.0) (3.9)
Table 3. Categorized responses to the open question, "Is there one thing that stands out as being really good about the package?" (n = 76) Response "Manipulation of molecular models" "Helps teach R and S stereochemistry" "The screen displays" Others No response
Respondents (%) 47 4 4 14 31
Table 4. Categorized responses to the open question, "Is there one thing that stands out as being really bad about the package?" (n = 76) Response "Questions too similar" "Lack of atom colour key" Others No response
Respondents (%) 15 4 10 71
sequence they worked, from chiral centre identification (71%) to designation (88%). The overall mastery of R and S designation was high (83%). The results from the rating scale are shown in Table 2. Students had intermediate computer experience (53%) and were already familiar with R and S designation (79%). The screen displays were generally intuitive (77%) and the possibility of even more interactivity was attractive (78%). The students were occasionally disorientated in the courseware (43%), though they felt in control of their progress most of the time (78%). The facility to rotate molecules certainly made the instruction more interesting (88%) and manipulation of structures improved the learning experience (60%). The prospect of further programs incorporating interactive molecular modelling to teach chemistry was an attractive prospect (79%). The results from the two open questions appended to the rating scale are summarized in Tables 3 and 4. Table 3 shows that 47% of the students picked out manipulation of molecular models as the most attractive feature of the courseware, with 31% not responding to the question. Conversely, Table 4 shows that the similarity of the questions was the least attractive feature (15%), with 71% not responding to this question. The system monitoring of virtual sphere usage during self-assessment has been categorized by question type in Fig. 2. Clearly, molecules were dragged least during priority assignment (26 pixels) and most
Interactive molecular chemistry
237
• Chiral centres [] Priorities
gq Designations •-~ 100 so ¢1 r~ "¢::1
~
6o
=
40
0
o
~
2o
< o Fig. 2. Mean mouse drag distances for chiral centre identification (n -85), priority assignment (n- 571) and final designation (n = 144). during R or S designation (93 pixels). During the identification of chiral centres, molecules were dragged by more variable distances, with the average lying between those for priority assignment and the final designation (73 pixels). Figure 3 shows the virtual sphere usage data categorized by initial molecule orientation. The mouse drag distances for a set of questions on a single molecule have been accumulated. Clearly, molecules Were dragged more when the lowest priority group was initially presented as forward facing (111 pixel~) as compared to backward facing (44 pixels). Welch's unpaired t-test revealed that this difference was significant (t = 2.03, P < 0.05). DISCUSSION The self-assessment scores showed that students achieved a high level of mastery of R and S designation. The weakest area was the identification of chiral centres. At this stage in the sequence the students were not focused on any particular region of the molecule and were possibly clicking randomly on several atoms before they correctly identified the chiral centre. The rating scale results show that a usable and intuitive interface has been designed for the courseware. • Lowest priority backwards [] Lowest priority forwards 140 e~
"~
-~
120
1oo ~
80
"~
60
E
4O
0
~
°
-
~
2o o
lB /l
Fig. 3. Mean mouse drag distances for initially forward facing (n- 46) and backward facing (n- 30) lowest priority groups.
238
R.T. HYDE et al.
The molecular modelling tools have been transparently integrated with the instruction and most of the students found this an engaging and effective learning environment. The open questions gathered more casual observations about the package design. By far the most appealing feature was the ability to manipulate molecular models, which confirmed the finding from the rating scale. Conversely, the similarity of the questions was identified by the students as an unattractive feature. This result was predictable, since in the prototype courseware the questions were of the same type and in the same sequence by necessity, to enable a standard set of virtual sphere usage data to be collected. The virtual sphere usage data showed that students were intuitively manipulating molecules to aid their visualization of 3D molecular structure during the instructional sequence. In keeping with the traditionally taught method of assignment, molecules were rotated most during the final designation. At this stage the students had to visualize the lowest priority group facing away from them. They rotated the molecules to help them achieve this. These data also revealed that molecules were rotated significantly more when the lowest priority group was initially presented as forward facing. This result highlights a possible weakness in the students' ability to form mental images of the 3D structure of molecules in this orientation. They were either rotating the lowest priority group to face away from them or spinning the molecule to enable them to visualize the spatial arrangement of the groups surrounding the chiral centre. Further examination of this stage in the designation process is necessary. This would reveal how the students are rotating the molecules to visualize 3D structure and whether the interface or instruction need modification to account for any difficulties. The emphasis of this study was to provide a realistic environment for the evaluation of the courseware. Although this has given ecological validity to the results, there was an inherent lack of control over data collection. Consequently, insufficient data were obtained to enable 3D manipulation of molecules to be correlated with scores obtained in the self-assessment section. A study of this relationship would reveal just how effective 3D molecular modelling tools are in the learning of spatial chemistry concepts. A more quantitative, controlled study would provide insights into this important area. A further area for examination would be the effect of student attributes on the use of the courseware. For example, Barke [15] reported that boys have better spatial ability than girls of an equivalent age and ability improves in both sexes with increasing age [15]. McGee [16] found that spatial ability was also related to degree of ambidexterity, such that highly ambidextrous boys performed best in mental rotation tests [16]. Results from a study of student characteristics and performance would confirm that both high and low spatial ability students could benefit from using the program. The integrated evaluation scheme used in this study has provided substantial evidence concerning the effectiveness of the courseware. Both the qualitative and quantitative data show that 3D molecular modelling can be integrated with tutorial instruction to help students understand spatial chemistry. We have also gained an insight into how the molecular modelling tools were used during the learning process and identified weaknesses in the design of the program and the spatial abilities of the students. These findings will be used to enhance future versions of the package. Acknowledgements--This
work has been supported by Oxford Molecular Ltd and the Engineering and Physical Sciences Research
Council (EPSRC). REFERENCES 1. Henkel J. G., PC-based molecular modelling in the classroom: application to medicinal chemistry and biochemistry.J. molec. Graph. 9, No. 3, 11-17 (1991). 2. Shubber K. E., Learning the visualisation of rotations in diagrams of three dimensional structures. Res. Sci. Technol. Educ. 8, 145-154 (1990). 3. Reeves T. C., Research support for interactive multimedia: existing foundations and new directions. In Interactive Multimedia: Practice and Promise (Edited by Latchem C., Williamson J. and Henderson-Lancett L.). Kogan Page, London (1993). 4. Colorado R. J., Computer-assisted instruction research: a critical assessment.J. Res. Comput. Educ. 20, 226--233 (1988). 5. Reeves T. C., Evaluation strategies for interactive multimedia products. In lnstructionalInteractive Multimedia: A Developer's Guide (Edited by Cates W. M.) (1994). 6. Macleod M., An introduction to usability evaluation. NPL report 102/92. National Physics Laboratory, Middlesex (1992). 7. Topham P. J., Methodologies for evaluating interactive video courseware. Interact. Learn. Int. 5, 19-23 (1989). 8. Reeves T. C., Computer modelling: a research tool for computer based instruction. Interact. Learn. Int. 8, 3-13 (1992). 9. Hyde R. T., Shaw P. N., Jackson D. E. and Woods K., Integration of molecular modelling algorithms with tutorial instruction: the design of an interactive three-dimensional computer-aidedlearning environment for exploring molecular structure. J. chem. Educ. In press. 10. Macromedia Inc., Authorwarc Professional V2.0 [computer program]. San Francisco, Calif. (1993).
Interactive molecular chemistry
239
11. Oxford Molecular Ltd, Nemesis V2.0 [computer program]. Oxford Molecular Ltd, Oxford (1994). 12. Chen M., 3-D rotation using a 2-D input device. Develop (TheApple TechnicalJournal) 14, 40-53 (1993). 13. Chen M., Mountford S. J. and Sellen A., A study in interactive 3-D rotation using 2-D control devices. Comp. Graph. 22, No. 4, 121-129 (1988). 14. Likert R. A., A technique for the measurement of attitudes. Archs Psychol. 140, 55 (1932). 15. Barke H. D., Chemical education and spatial ability. J. chem. Educ. 70, 968-971 (1993). 16. McGee M. G., Handedness and mental rotation. Percept. Motor Skills 47, 641--642 (1978).