- Email: [email protected]
Evaluating a CALL software on the learning of English prepositions
Computers & Education 47 (2006) 41–55 www.elsevier.com/locate/compedu
Evaluating a CALL software on the learning of English prepositions Bing Hiong Ngu a
a,*
, Soubakeavathi Rethinasamy
b
Faculty of Cognitive Sciences and Human Development, Universiti Malaysia Sarawak, Kota Samarahan, 94300 Kota Samarahan, Sarawak, Malaysia b English Language Centre, Universiti Malaysia Sarawak, 94300 Kota Samarahan, Sarawak, Malaysia Received 15 August 2002; accepted 18 August 2004
Abstract This study assessed the effectiveness of using a CALL lesson (Computer Assisted Language Learning) over a conventional lesson to facilitate learning of English prepositions at Bario, Malaysia. CALL was developed by the Ministry of Education, Malaysia as support material to enhance learning of English prepositions. Both the conventional and the CALL lessons were matched with the same content except for the medium in which the lesson was being delivered. Students were provided with computers to go through the CALL lesson in a self-regulated manner; while a teacher taught the conventional lesson in a classroom. Test results indicate that students who received the conventional lesson outperformed those who went through the CALL lesson. The Relative Condition Efficiency measurement also showed that the conventional group learned more efficiently than the CALL group. The findings are interpreted from the perspective of cognitive load required in processing the presentation mode of the learning materials. 2004 Published by Elsevier Ltd. Keywords: Evaluation methodologies; Evaluation of CALL systems; Human–computer interface; Cognitive load; Secondary education
*
Corresponding author. E-mail address: [email protected] (B.H. Ngu).
0360-1315/$ - see front matter 2004 Published by Elsevier Ltd. doi:10.1016/j.compedu.2004.08.016
42
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
1. Introduction MalaysiaÕs Vision 2020 aims to achieve the status of a developed country by the year 2020 (Mahathir, 1991). To attain Vision 2020, there is an urgency to see that Malaysia does not lag behind in the development of Information Technology (IT). Accordingly, the Malaysian government has launched the Multimedia Super Corridor (MSC) project to propel seven flagship applications of IT that include networking Malaysians for the purposes of e-learning, e-government, e-commerce, and e-communities. One of the flagships of the MSC is the Smart School project within which a main objective is to increase IT literacy and penetration especially in rural regions of Malaysia. Present efforts and mechanisms to promote IT have, however, been concentrated mostly amongst MalaysiaÕs urban dwellers, raising a concern for those who are sidelined in the Digital Divide, i.e. marginalised populations living in telecommunication-deprived rural areas (Harris, 1999). 1.1. Bario Government Secondary School (SMK Bario) Nestled in the remote mountainous northern central part of Borneo at an altitude of approximately 1000 m above sea level, BarioÕs isolation is bridged by the only practical means of transport – a daily flight by the Malaysian Airline System plying the Highlands and the nearby towns of Marudi and Miri (Harris, 1999). SMK Bario, the only secondary school in Bario was established in 1967. It caters mostly for studentsÕ from BarioÕs 12 villages and offers lower secondary school education (students in year 7–10). The school provides boarding for its entire 158 students (as of July 1999) and has amenities such as piped water, medical services and two diesel operated generators to provide electricity (Trang, 1999). However, the only means of communication available in the school is via very high frequency (VHF) radio services. 1.2. Information Technology Literacy Program With a research grant from the Canadian International Development Research Centre, the e-bario project was initiated in early 1999 to introduce IT to the remote community of Bario. A local computer company, Conserve Computer, was invited to participate as a technology partner in this e-bario project. Apart from supplying hardwares and softwares, Conserve computer also initiated the Information Technology Literacy Program (ITLP). In January 2000, SMK Bario had benefited from a new computer lab set-up equipped with 10 personal computers, a printer, and a scanner. Each student received a manual of ITLP (written in English language) introducing them to basic software applications such as typing, word processing, Excel, etc. The ITLP was conducted once every week for 10 weeks (with an hour per session) after school hours, and occasionally at night. Included in the e-bario project was internet connectivity, but the internet had not been connected at the time of this study. Thus the ITLP training program was unable to train students how to surf the internet. 1.3. Computer Assisted Language Learning The Computer Assisted Language Learning (CALL) software was developed by the Educational Technology Division of the Ministry of Education, Malaysia as part of the Smart School
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
43
program to aid students in learning English prepositions. Some teachers, who had several years of teaching English language in secondary schools, volunteered to collaborate with the staff in the Educational Technology Division to develop CALL during school vacations. Learning English prepositions poses a great challenge for students who learn English as a second language (Jabbour-Lagocki, 1990; Jin, 1989). Inappropriate use of prepositions results in changing the meaning of English sentences; and this indirectly contributes toward the difficulty of learning the English language. In particular, the use of prepositional phrases in which prepositions are used to indicate a relationship under a particular circumstance (Collerson, 1995). As most of the prepositions express relationships between things and events, their basic sense is spatial (Borjars & Burridge, 2001). For example, one can throw the ball over/under/at/through/near the wall. To express this spatial relationship between the ball and the wall, the use of video tapes or animated pictures with computer software to illustrate the use of prepositions in various contexts should enhance learning of prepositions (Jabbour-Lagocki, 1990). This approach was incorporated in the development of the CALL lesson. Macromedia Director 7 was used to design this CALL. Teachers who assisted in the development of CALL selected the prepositions in accordance with the English language curriculum designed by the Ministry of Education, Malaysia, for students who are 13 years old. The content of CALL includes a quiz, an instruction phase, an exercise phase and a final quiz. It commences with light music and statement of the aim of the lesson. Students were given the options to proceed in any order, to repeat a particular preposition, or to quit. A quiz given before the learning phase was intended to assess studentsÕ initial knowledge of prepositions. It consisted of five multiple choice questions related to prepositions of time (on, after, at and before); and five multiple choice questions about prepositions of place (in, beside, on, and under). In either the prepositions of time or place, students had to select the correct preposition out of four to fill a blank in a sentence. The computer provided feedback with regard to the scores achieved by students in completing the quiz. The final quiz (multiple choice questions) at the end of the lesson aimed to examine the extent to which students had learned and understood the prepositions. In the instruction phase of CALL, for example, a video picture was used to illustrate the preposition of ÔatÕ (see Fig. 1(a)). This aims to capture the relationship between an event such as catching a bus and the time (Borjars & Burridge, 2001; Jabbour-Lagocki, 1990). Its sound effects included narration to explain the meaning of ÔatÕ, light music to highlight the mood of the background and the sound caused by the movement of an object (e.g., when a bus was coming). Since the preposition ÔatÕ is used to specify a particular point in time, the inclusion of a clock and the routine real life circumstance (Tsou, Wang, & Li, 2002) of school students catching a school bus at a particular time represents a powerful way to illustrate the preposition of ÔatÕ because it provides an appropriate context. Similarly, the movement of a ball in relation to a static chair represents a type of animation, which intends to demonstrate the preposition of place effectively (e.g., in front of a chair). Other video pictures capture routine meal time to illustrate the prepositions of ÔbeforeÕ and ÔafterÕ. In short, the incorporation of relevant real life settings provide contexts for students to grasp the meaning of prepositions in various situations. Some of the exercises provided were interactive in that students could move the objects to learn a preposition (see Fig. 1(b)). This type of interaction with learning materials presumably requires a high level of engagement of the learning task that should benefit students learning of prepositions (Tsou et al., 2002). The exercises were structured into three levels of difficulty. For Level
44
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
Fig. 1. (a) A screen shot of preposition of time in CALL. (b) A screen shot of an exercise.
1, students dragged and dropped (chose a correct preposition from an array of prepositions) to complete a sentence. Level 2 had multiple choice questions. Immediate feedback was provided for Levels 1 and 2 exercises. The use of multiple choice exercises (learner–self interaction) aimed to help students in reflecting (try to understand or make sense) the topic content (Soo & Bonk, 1998). For Level 3, students had to use several prepositions to complete a paragraph, and feedback was only provided when the whole task was completed. Thus, whether a student used one or several preposition(s) incorrectly, he or she had to complete the whole paragraph again. In other words, students may make multiple attempts in order to complete the paragraph accurately. This type of exercise aims to assist students to acquire skills in using appropriate prepositions based on the contexts of a whole paragraph. 1.4. Software evaluation Evaluation of early softwares is usually conducted by teachers who used a rating form to evaluate each feature of a software (MicroSIFT, 1982; Salvas & Thomas, 1984; Tolhurst, 1992).
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
45
McDougall and Squires (1994) regard these evaluation approaches as inadequate because they include only teachersÕ views of the software. Reiser and Kegelmann (1994) share a similar view to McDougall and Squires. They emphasize the importance of inviting students to participate in the evaluation process rather than relying on teachers alone to evaluate the educational effects of a given software. The main issue is whether end-users are able to acquire the skills a software program is designed to teach. Borg and Gall (1989) support this view, and they recommend learners to participate in the process of evaluating educational softwares. Jolicoeur and Berger (1988a, 1988b) proposed that educators evaluated the software; and at the same time, involved the students in the evaluation process. Thus, when students serve as participants in the evaluation process, other evaluators (e.g., teachers) observe the students as they use the software and draw conclusions about the quality of the software. The evaluators may ask students to indicate their opinions about the software and may even test what students have learned as a result of using the software. This means students will be pretested, observed as they work through the software program, and post-tested. We incorporated this approach in this study. In addition, we used a comparative study design in which the efficiency of CALL was compared with a conventional teacher taught lesson. Lockee, Moore, and Burton (2001) argued against the use of media comparison studies. They identified a range of variables (such as learner characteristics, media attributes, instructional strategy choices, and psychological theories) in the comparison studies which made the comparative design inadequate to justify the learning effects of the two instructions. That is, it is difficult to establish the cause and effect of a comparative design because it is almost impossible to match the variables for the participants in a comparison study. In contrast, several researchers (Catenazzi & Sommaruga, 1999a, 1999b; Collin, 2000; Lambert et al., 1996; Tsou et al., 2002) compared the effectiveness of a computer aided instructional presentation with a conventional mode of presenting the same learning materials. One of the aims of designing a comparative study was to replace the computer mode of presenting content with the traditional one should the former supersedes the latter.
1.5. Learning efficiency Cognitive load theory (CLT; Sweller, 1988; Sweller, van Merrienboer, & Paas, 1998) has identified three aspects of cognitive load that learners might experience in acquiring schemas: extraneous load, intrinsic load, and germane load. Extraneous load is related to an instructional design that is detrimental to learning because the design itself requires learners to process activities (e.g., search, integrate, relate pieces of information to become intelligible) unrelated to skill acquisition. The load is intrinsic if the nature of the learning materials imposes different amount of cognitive load upon the learner. For instance, a complex task will require the learner to interact with many elements simultaneously that might be taxing on the working memory load. Germane cognitive load arises when instructions direct the learner to engage in cognitive activities that profit schemas acquisition. This germane cognitive load increases mental effort to process information, but is particularly beneficial when the learning materials have low intrinsic load thus leaving sufficient cognitive resources available for skill acquisition. Taken together, the total cognitive load (intrinsic, extraneous, and germane load) should stay within the learnerÕs cognitive resources if learning is to occur (see Pass, Tuovinen, Tabbers, & Van Gerven, 2003).
46
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
The perceived mental effort required to learn a task refers to the amount of capacity that is allocated to the demand of an instructional presentation (Paas, 1992). Thus, the mental effort exerted to learn a task represents an index of cognitive load involved in learning. Research within the CLT framework has suggested that the learning efficiency of an instructional technique (Marcus, Cooper, & Sweller, 1996; Paas, 1992; Paas & van Merrienboer, 1993; Tuovinen & Paas, 2004; Tuovinen & Sweller, 1999) is a measure of the relationship between mental effort invested and performance. An effective instructional presentation mode decreases mental effort (cognitive load) but increases transfer performance scores. Having completed the CALL lesson, students in this study were required to indicate their mental effort invested in learning prepositions on a seven-point Likert scale (with 1 = extremely low and 7 = extremely high mental effort). This rating scale to assess mental effort is considered as a simple and non-invasive technique (Paas, 1992; Pass & Van Merrienboer, 1994; Tuovinen & Paas, 2004). Test performance was the scores of the correct answers. As the units of mental effort and test scores differ, their respective scores were converted to standardized scores (z scores). Then the standardized mean of mental effort (M), and the standarized mean of performance (P) were combined to indicate the efficiency (E) of learning condition that can be expressed as P M E ¼ pffiffiffi : 2 E can be either positive or negative. The perpendicular distance from the neutral efficiency condition, E = 0, is where performance = effort. The higher the absolute value of E, the more efficient (or inefficient) is the instruction. Note that the square root of 2 in the denominator is used to make the interpretation of the graph easier. This efficiency measurement attributes learning outcomes to the amount of cognitive load required in processing an instructional presentation. For instance, a high E value means an instructional presentation required low cognitive load but results in a high test performance. Indirectly, this efficiency measurement also ensures that the measurement of cognitive load is directly linked to performance outcome rather than other subjective variables (e.g., self-confidence in processing the instructional materials). 1.6. Present investigation The Malaysian Education Ministry (1997) has targeted the conversion of all primary and secondary schools to become Smart Schools by the year 2010, and hence there is a need to design a comparative study to test whether CALL would be better than conventional lesson in terms of learning outcomes. Since students in SMK Bario undertook an ITLP training program under the e-bario project, and this was coupled with the fact that they were targeted end-users of the Smart School program, we invited them to participate in the evaluation of CALL software. Previous researchers reported beneficial effects of multimedia instruction (Catenazzi & Sommaruga, 1999a, 1999b; Philips, Jenkin, Fyle, & Fyle, 1997; Preece & David, 1992; Reynolds & Ehrlich, 1992). However, the assessment of multimedia instruction normally did not include mental effort exerted in the learning process. In this study, the mental effort required to go through the CALL lesson will include the effort invested in attending to the medium as well as its content
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
47
(Lowe, 1999; Schnotz, Bockheler, & Grzondziel, 1999). Students in the conventional group exerted mental effort in attending to a conventional classroom teaching. In sum, the experimental manipulation of this study was the instructional presentation of English prepositions. Transfer performance and mental effort invested in learning the CALL and a conventional lesson were collected to form a basis of comparison. Additionally, we collected qualitative data (post-questionnaire) to examine how students perceived the two alternative modes of instructional presentations.
2. Method 2.1. Participants Participants were 30 students (with mean age 13 years old) from a Form Two class (equivalent to year 7 students) at SMK Bario. Students had 6 years primary and 1 year secondary education. Malay language was the medium of instruction in the school. English language was a compulsory subject in the primary and secondary school curriculum. Prior to the experiment, all participants attended a weekly ITLP for 10 weeks of an hour per session (see Section 1). They had basic knowledge of typing and word processing. Participants were not trained to surf internet. However, some teachers in the school had taught them to use some Smart School softwares to enhance learning of a variety of subjects. None of the participant in this study had used the CALL software prior to the experiment. 2.2. Materials The English teacher in SMK Bario studied the CALL lesson and prepared the conventional lesson and learning activities as closely as possible to that illustrated in the CALL software (see Section 1 on a detailed discussion of CALL). Every effort was made to create a learning environment similar to the one described in the CALL software. For example, a physical chair and a ball were used to illustrate the prepositions: in front of, beside, under, and over. This was similar to one of the exercises inside CALL in which a ball moved around the chair and/or a box. 2.3. Pre-test and post-test The pre-test had the same type and number of questions as the post-test (see Appendix A). All questions were related to the prepositions of time and place. The second author of this paper (who had several years of teaching English language in schools) designed these questions. Part I of this pretest and post-test had 10 multiple choice questions in which students had to choose one preposition out of four to complete each sentence. There were 15 questions in Part II. Students were required to underline the correct preposition (chose one out of two) to complete a sentence. Pretest was conducted to control studentsÕ initial knowledge of English prepositions. In addition, the anticipated differential performance of the pre-test and post-test would shed light on the relative learning efficiency of the two instruction methods.
48
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
2.4. Post-questionnaire In the post-questionnaire (Appendix B), students were required to evaluate the usability (e.g., user-friendliness, the layout of the screen, etc.) of the software. Also, a seven-point subjective scale was provided for students to rate the mental effort invested in learning the prepositions (see Section 1). In the conventional group, the post-questionnaire required students to answer questions related to the manner in which the teacher delivered the lesson, learning activities, and exercises. The aim was to know how students rated a conventional lesson in assisting them to learn English prepositions. Similar to the CALL group, a seven-point subjective scale was provided for student to rate mental effort invested in learning. In sum, the CALL and the conventional lessons were matched with the same content of English prepositions of time (by, on, before, at and after) and place (in, on, near, over, under, between, behind, in front of, and beside). The experimental manipulation was the conditions under which the content was being delivered.
2.5. Procedure The experiment was conducted during studentsÕ normal class lesson. First, a researcher informed the students concerning the aim and procedure of the experiment. Second, students were allowed 15 min to complete a pre-test. Third, students were randomly assigned to either a CALL group or a conventional group with 15 in each group. Fifteen students were requested to attend a CALL lesson in a computer laboratory in school. Because the number of computers was insufficient, students were tested in three separate group sessions. Students completed the CALL lesson at their own pace. The researchers observed that students normally took about an hour to complete the CALL lesson. Throughout the computer session, students were free to ask questions. After the CALL training session, students were given 15 min to complete a post-test; and then they filled in a post-questionnaire at their own pace. For the conventional group, after students completed the pre-test, a subject teacher took about 35 min to deliver the conventional lesson as closely as it was possible in the CALL lesson (see also Section 2.2). After this, students were allowed 15 min to complete the post-test; and they filled in a post-questionnaire at their own pace.
3. Results This study investigated whether the CALL lesson would be superior to the conventionally taught lesson. Out of the data collected from 30 students, two studentsÕ results had to be eliminated due to inadequate literacy level (they could not write or read). Also, one student in CALL group did not complete Part II of the post-test, and his scores were not analyzed. The questions in pre-test and post-test were objective types, and one mark was awarded to each correct answer. A maximum score was 25 marks.
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
49
Table 1 Means (standard deviations) of correct answers for pretest and posttest Instruction
n
Pre-test
Post-test
CALL Conventional
13 14
14.69 (5.02) 15.07 (6.31)
15.00 (5.08) 19.14 (4.26)
Graph 1. Mean scores of CALL and conventional groups in pre and post-tests.
3.1. Group comparison of scores Table 1 presents the means and standard deviations for the pre-test and post-test. A 2 (instruction) · 2 (test) ANOVA with instruction as a between-group factor and test as a within-subject repeated measure showed a significant main effect for the test, F = (1, 25) = 6.60, p = 0.02, a nonsignificant instruction main effect, F = (1, 25) = 1.54, p = 0.23, and a significant interaction effect between the test and instructions, F = (1, 25) = 4.87, p = 0.04. The significant interaction effect was caused by the conventional group outperformed the CALL on the post-test, t(25) = 2.30, SE = 1.80, p = 0.03. Thus, these results revealed that both groups improved their performance from pre-test to post-test but the conventional group showed a greater improvement than the CALL group (see Graph 1). 3.2. Relative condition efficiency In addition to scores, a relative condition efficiency graph (see Graph 2) was produced to examine the relative efficiency of the two methods of instruction. As noted previously, this takes into account of two dimensions: the perceived mental effort invested in learning the prepositions, and the performance scores on posttest. Graph 2 displays the E value and its standard deviation for
50
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
Graph 2. Relative condition efficiency.
the CALL group as well as the conventional group. A t test indicated that students in the conventional group learned more efficiently than the CALL group, t(25) = 2.77, SE = 0.92, p = 0.01. The former is located in the top left corner of the coordinate that represents a high efficiency region; whereas the latter is found in the lower right corner of the coordinate that represents a low efficiency region. 3.3. Post-questionnaire In the post-questionnaire, about 77% of students commented that it was difficult to return to the main menu, while about 85% students indicated that they could not complete the exercises without the teacherÕs assistance. In fact, most students (about 77%) needed frequent help from a teacher while doing the CALL lesson. However, half of the students (about 54%) found the screen easy to follow; and about 92% of the students found the layout of the exercise easy to read. Further, the majority of students (85%) found the CALL motivating, and they would recommend CALL to their friends. Lastly, students indicated that the best aspect of CALL was the exercises. The post-questionnaire for the conventional group showed that all 14 students could follow the teacherÕs explanation. They also found that the organization of the lesson was systematic; and the
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
51
exercises given by the teacher assisted them to understand the lesson. However, only some students (about 57%) found that the teacher actually motivated them to learn and asked challenging questions. The majority of the students (71%) indicated that the best aspect of this lesson was teacherÕs demonstration and explanation; and they would recommend the lesson to a friend.
4. Discussion This study evaluated the effectiveness of a CALL lesson when compared with a conventional lesson in assisting students to learn English prepositions. The results indicate that students who received conventional teaching performed better than those who went through the CALL lesson. Greater improvement from a pre-test to a post-test was observed for the conventional group rather than the CALL group. Further, relative condition efficiency graph also showed that the convention group learned more efficiently than the CALL group with the latter exerting more mental effort and achieving a lower transfer performance than the former. The mental effort required to learn the prepositions may include learnersÕ activities spent in navigating the system as well as in comprehending the multimedia presentation of the materials (Lowe, 1999; Schnotz et al., 1999). Analysis of the post-questionnaire reveals that students experienced difficulty in navigating the CALL lesson (see Q3, Q7, and Q8). In addition, most students were unable to complete the exercises without assistance (Q2); and students were distracted by some features of the content (Q10). Therefore, this finding indicates that neither the system nor the presentation of the content was easy to follow meaning students had to invest a high mental effort to learn the CALL lesson. Consequently, students in the CALL group not only exerted a high mental effort in learning but they learned less efficiently than the control group. Test results contradict a number of research findings investigating the effects of computer assisted multimedia instruction (Catenazzi & Sommaruga, 1999a, 1999b; Phelps & Reynolds, 1999; Philips et al., 1997). It seems that the user interface design of the CALL did not support a self-regulated learning mechanism that has normally been highlighted in a computer assisted multimedia learning environment. In contrast, the post-questionnaire shows that the conventional group could follow the lesson easily. Presumably, students were familiar with their teacher; and the learning activities were easy to understand and do. As a consequence, students in the conventional group used less mental effort but achieved a better transfer performance than students in the CALL group. The introduction of IT in SMK Bario under the e-bario project has equipped students with some basic computer skills to use CALL in enhancing their knowledge of English prepositions. Evidence was provided by the gain in learning from pre-test to post-test (but this result needs to be interpreted with caution because of the sample size. We were unable to use a bigger sample size due to the constraint of the studentsÕ population in SMK Bario). Despite the inferior test performance of the CALL group as compared to the conventional group, most students who received CALL lessons would like to recommend CALL to their friends. Therefore, students from rural schools should be encouraged to undertake ITLP program so that they can participate in the Smart School Program. However, to facilitate the transfer of learning, the design of an educational software package needs to minimize the mental effort required to handle the medium and the content. The difficulty
52
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
in navigating the CALL represents an extraneous cognitive load as it interfered with learning. The use of hierarchical concept map to direct the learner to navigate the learning materials proved to be effective (Lo, Wang, & Yeh, 2004). Hence future research should consider using this approach to lessen the possibility of the studentÕs disorientation in navigating the CALL. Recent cognitive load research in the domain of multimedia instructional presentations (Brunken, Steinbacher, Plass, & Leutner, 2000) has recommended the use of the modality effect with auditory/visual presentations to optimize the working memory capacity. In the instruction phase of the CALL, for example, the written text together with the auditory narration can be presented by auditory mode alone thus freeing an overloaded visual channel to attend to the visual picture. In addition, in the exercise phase of the CALL, the computer feedback showing the right or wrong answer could be replaced with the confidence scores to act as a learning diagnosis and to provide remedial instruction according to diagnosis (Lo et al., 2004). By requiring students to rate their confidence level in selecting their answers while they work through the exercises may lead to in-depth understanding of the appropriate use of prepositions in various contexts. To conclude, the evaluation of software can provide information concerning the suitability of the softwares for end-users. This research collected both qualitative and quantitative data to assess the CALL; and this coupled with the learning efficiency measurement based on cognitive research serves as an appropriate model of software evaluation for educators, and commercial software developers.
Acknowledgement This study was supported by a Grant from the Canadian International Development Research Centre. We thank Lucy Bala, the Headmaster of Bario Government Secondary School in assisting the collection of data, Elaine Guat Lien Khoo and Roger Harris for helpful comments on earlier versions of this paper. Appendix A. Pre-test and post-test questions Part I: Circle a correct answer his parents 1. The child is sitting A. at B. over C. in D. near the bank 2. You must save your money A. on B. in C. by D. over the wall 3. He hung his motherÕs photograph A. at B. in C. on D. by 8 oÕclock on Monday 4. There will be an assembly A. by B. on C. at D. in
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
53
Part II: Underline the correct answer 1. You can do your work (after, on) lunch 2. I will come again (on, at) Saturday 3. LetÕs finish our work (before, by) the bell rings 4. The ball went (in front of, under) the table and I had to kneel down to get it
Appendix B. Post-questionnaire: CALL group
Questions
Yes
No
1. Did you find the introductory page useful? 2. Easy to return to the main menu? 3. Were you able to complete the exercise independently (without the help of a teacher)? 4. Menus, icons and buttons were easy to understand and pleasing to look at? 5. Did you find the screen easy to read (appropriate matching of background and text color)? 6. Did you find the layout of this computerized learning exercise attractive to ÔlookÕ at? 7. Was page or screen layout easy to follow? 8. Was page or screen layout consistent throughout? 9. Did you notice any grammatical, numbering or typographic errors? 10. Were you distracted and irritaded by one or more features of the course? 11. Were graphics clear, not distracting, and donÕt complete with content? 12. This computerized learning was extremely motivating and made it likely that you would learn the content? 13. Would you recommend this computerized learning to your friends? 14. What was the best thing you like about this computerized learning? 15. What things did you dislike about this computerized learning?
13 3 2
0 10 11
10
3
12
1
12
1
7 2 6
6 11 7
7
6
8
5
11
2
10
3
Comments
Exercises, quiz None
References Borjars, K., & Burridge, K. (2001). Introducing English grammer. Published in Great Britain by Arnold, a member of the Hodder Headline Group, London NW1 3BH.
54
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
Brunken, R., Steinbacher, S., Plass, J. L., & Leutner, D. (2000). Assessment of cognitive load within multimedia learning by the dual task methodology. Experimental Psychology, 49, 109–119. Catenazzi, N., & Sommaruga, L. (1999a). The evaluation of the Hyper Apuntes interactive learning environment. Computers & Education, 32, 35–49. Catenazzi, N., & Sommaruga, L. (1999b). The evaluation of the Hyper Apuntes interactive learning environment. Computers & Education, 32, 35–49. Collerson, J. (1995). English grammar: A functional approach. Australia: Primary English Teaching Association. Collin, M. (2000). Comparing web, correspondence and lecture versions of a second-year non-major Biology course. British Journal of Educational Technology, 31(1), 21–27. Harris, R. (1999). Rural information technology for SarawakÕs development. Sarawak Development Journal, 2(1), 72–84. Jabbour-Lagocki, J. (1990). Prepositions of position: an analysis for practical application in the classroom. ERIC Document Reproduction Service No. ED 427 547. Jin, Z. Y. (1989). Difficulties in learning English for Chinese students. Test Talk, 13(3), 87–91. Jolicoeur, K., & Berger, D. E. (1988a). Implementing educational software and evaluating its academic effectiveness: Part I. Educational Technology, 28(9), 7–13. Jolicoeur, K., & Berger, D. E. (1988b). Implementing educational software and evaluating its academic effectiveness: Part II. Educational Technology, 28(10), 13–19. Lambert, T., Shepherd, J., Ngu, A., Ho, P., Whale, G., & Geissinger, H. (1996). Bridging the gap: Computer science meets distance education at UNSW. Australia: School of Computer Science and Engineering, University of New South Wales. Lo, J., Wang, H., & Yeh, S. (2004). Effects of confidence scores and remedial instruction on prepositions learning in adaptive hypermedia. Computer & Education, 42, 45–63. Lockee, B., Moore, M., & Burton, K. (2001). Old concerns with new distance education research. Educause Quarterly, 24(2), 60–62. Lowe, R. K. (1999). Extracting information from an animation during complex visual learning. European Journal of Psychology of Education, 14, 225–244. Mahathir, M. (1991). Malaysia: The way forward. In Proceedings of the paper presented at the Malaysian business council, Center for Economic Research and Services, Kuala Lumpur. Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of Educational Psychology, 88(1), 49–63. McDougall, A., & Squires, D. (1994). Perspectives interactions: a new approach in the selection of educational software. In Proceedings of the paper presented at the APITITEÕ94, Brisbane. MicroSIFT. (1982). EvaluatorÕs guide for microcomputer-based instructional packages, The International Council for Computers in Education, Wugene, Oregon. Ministry of Education, Malaysia. (1997). A vision of the Malaysian Smart School [On-line]. Available from: http:// eprd.kpm.my/exesum.html/. Paas, F. G. W. C. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach. Journal of Educational Psychology, 84, 429–434. Paas, F. G. W. C., & van Merrienboer, J. J. G. (1993). The efficiency of instructional conditions: an approach to combine mental effort and performance measures. Human Factors, 35(4), 737–743. Pass, F., & Van Merrienboer, J. J. G. (1994). Variability of worked example and transfer of geometrical problem solving skills. A cognitive approach. Journal of Educational Psychology, 86, 122–133. Pass, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38, 63–71. Phelps, H., & Reynolds, R. (1999). Formative evaluation of a web-based course in meteorology. Computers & Education, 32, 181–193. Philips, R., Jenkin, N., Fyle, G., & Fyle, S. (1997). The user interface design of learner-centred interactive multimedia programs. Retrieved January 15, 2000 from World Wide Web: http://cleo.murdoch.edu.au/tlc/Philips/Ed-Media97/ Paper294.html.
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
55
Preece, D., & David, G. (1992). Multimedia: some promises, some problems and some issues in human–systems interaction. In Proceedings of the international interactive multimedia symposium perth (pp. 259–268), Western Australia. Reiser, R. A., & Kegelmann, H. W. (1994). Evaluating instructional software: a review and critique of current methods. Educational Technology, Research and Development, 42(3), 63–69. Reynolds, L., & Ehrlich, D. (1992). Multimedia in industry and education: a decision model for design. In Proceedings of the international interactive multimedia symposium perth (pp. 117–126), Western Australia. Salvas, A., & Thomas, G. (1984). Evaluation of software. Melbourne: Education Department of Victoria. Schnotz, W., Bockheler, J., & Grzondziel, H. (1999). Individual and co-operative learning with interactive animated pictures. European Journal of Psychology of Education, 14, 245–265. Soo, S.-K., & Bonk, C. J. (1998). Interaction: what does it mean in online distance education? In Proceedings of the paper presented at the EDDMEDIA & ED-TELECOM 98, Freiburg, Germany. Sweller, J. (1988). Cognitive load during problem solving. Cognitive Science, 12, 257–285. Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296. Tolhurst, D. (1992). A checklist for evaluating content-based hypertext computer software. Educational Technology, 32(3), 17–21. Trang, T. (1999). The perception of teachers in SMK Bario towards the use of the computer. Unpublished MasterÕs Thesis, Universiti Malaysia Sarawak, Kota Samarahan, Kuching, Sarawak, Malaysia. Tsou, W., Wang, W., & Li, H. (2002). How computers facilitate English foreign language learners acquire English abstract words. Computer & Education, 39, 415–428. Tuovinen, J. E., & Paas, F. (2004). Exploring multidimensional approaches to the efficiency of instructional conditions. Instructional Science, 32, 133–152. Tuovinen, J. E., & Sweller, J. (1999). A comparison of cognitive load associated with discovery learning and worked examples. Journal of Educational Psychology, 91, 334–341.
Evaluating a CALL software on the learning of English prepositions Bing Hiong Ngu a
a,*
, Soubakeavathi Rethinasamy
b
Faculty of Cognitive Sciences and Human Development, Universiti Malaysia Sarawak, Kota Samarahan, 94300 Kota Samarahan, Sarawak, Malaysia b English Language Centre, Universiti Malaysia Sarawak, 94300 Kota Samarahan, Sarawak, Malaysia Received 15 August 2002; accepted 18 August 2004
Abstract This study assessed the effectiveness of using a CALL lesson (Computer Assisted Language Learning) over a conventional lesson to facilitate learning of English prepositions at Bario, Malaysia. CALL was developed by the Ministry of Education, Malaysia as support material to enhance learning of English prepositions. Both the conventional and the CALL lessons were matched with the same content except for the medium in which the lesson was being delivered. Students were provided with computers to go through the CALL lesson in a self-regulated manner; while a teacher taught the conventional lesson in a classroom. Test results indicate that students who received the conventional lesson outperformed those who went through the CALL lesson. The Relative Condition Efficiency measurement also showed that the conventional group learned more efficiently than the CALL group. The findings are interpreted from the perspective of cognitive load required in processing the presentation mode of the learning materials. 2004 Published by Elsevier Ltd. Keywords: Evaluation methodologies; Evaluation of CALL systems; Human–computer interface; Cognitive load; Secondary education
*
Corresponding author. E-mail address: [email protected] (B.H. Ngu).
0360-1315/$ - see front matter 2004 Published by Elsevier Ltd. doi:10.1016/j.compedu.2004.08.016
42
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
1. Introduction MalaysiaÕs Vision 2020 aims to achieve the status of a developed country by the year 2020 (Mahathir, 1991). To attain Vision 2020, there is an urgency to see that Malaysia does not lag behind in the development of Information Technology (IT). Accordingly, the Malaysian government has launched the Multimedia Super Corridor (MSC) project to propel seven flagship applications of IT that include networking Malaysians for the purposes of e-learning, e-government, e-commerce, and e-communities. One of the flagships of the MSC is the Smart School project within which a main objective is to increase IT literacy and penetration especially in rural regions of Malaysia. Present efforts and mechanisms to promote IT have, however, been concentrated mostly amongst MalaysiaÕs urban dwellers, raising a concern for those who are sidelined in the Digital Divide, i.e. marginalised populations living in telecommunication-deprived rural areas (Harris, 1999). 1.1. Bario Government Secondary School (SMK Bario) Nestled in the remote mountainous northern central part of Borneo at an altitude of approximately 1000 m above sea level, BarioÕs isolation is bridged by the only practical means of transport – a daily flight by the Malaysian Airline System plying the Highlands and the nearby towns of Marudi and Miri (Harris, 1999). SMK Bario, the only secondary school in Bario was established in 1967. It caters mostly for studentsÕ from BarioÕs 12 villages and offers lower secondary school education (students in year 7–10). The school provides boarding for its entire 158 students (as of July 1999) and has amenities such as piped water, medical services and two diesel operated generators to provide electricity (Trang, 1999). However, the only means of communication available in the school is via very high frequency (VHF) radio services. 1.2. Information Technology Literacy Program With a research grant from the Canadian International Development Research Centre, the e-bario project was initiated in early 1999 to introduce IT to the remote community of Bario. A local computer company, Conserve Computer, was invited to participate as a technology partner in this e-bario project. Apart from supplying hardwares and softwares, Conserve computer also initiated the Information Technology Literacy Program (ITLP). In January 2000, SMK Bario had benefited from a new computer lab set-up equipped with 10 personal computers, a printer, and a scanner. Each student received a manual of ITLP (written in English language) introducing them to basic software applications such as typing, word processing, Excel, etc. The ITLP was conducted once every week for 10 weeks (with an hour per session) after school hours, and occasionally at night. Included in the e-bario project was internet connectivity, but the internet had not been connected at the time of this study. Thus the ITLP training program was unable to train students how to surf the internet. 1.3. Computer Assisted Language Learning The Computer Assisted Language Learning (CALL) software was developed by the Educational Technology Division of the Ministry of Education, Malaysia as part of the Smart School
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
43
program to aid students in learning English prepositions. Some teachers, who had several years of teaching English language in secondary schools, volunteered to collaborate with the staff in the Educational Technology Division to develop CALL during school vacations. Learning English prepositions poses a great challenge for students who learn English as a second language (Jabbour-Lagocki, 1990; Jin, 1989). Inappropriate use of prepositions results in changing the meaning of English sentences; and this indirectly contributes toward the difficulty of learning the English language. In particular, the use of prepositional phrases in which prepositions are used to indicate a relationship under a particular circumstance (Collerson, 1995). As most of the prepositions express relationships between things and events, their basic sense is spatial (Borjars & Burridge, 2001). For example, one can throw the ball over/under/at/through/near the wall. To express this spatial relationship between the ball and the wall, the use of video tapes or animated pictures with computer software to illustrate the use of prepositions in various contexts should enhance learning of prepositions (Jabbour-Lagocki, 1990). This approach was incorporated in the development of the CALL lesson. Macromedia Director 7 was used to design this CALL. Teachers who assisted in the development of CALL selected the prepositions in accordance with the English language curriculum designed by the Ministry of Education, Malaysia, for students who are 13 years old. The content of CALL includes a quiz, an instruction phase, an exercise phase and a final quiz. It commences with light music and statement of the aim of the lesson. Students were given the options to proceed in any order, to repeat a particular preposition, or to quit. A quiz given before the learning phase was intended to assess studentsÕ initial knowledge of prepositions. It consisted of five multiple choice questions related to prepositions of time (on, after, at and before); and five multiple choice questions about prepositions of place (in, beside, on, and under). In either the prepositions of time or place, students had to select the correct preposition out of four to fill a blank in a sentence. The computer provided feedback with regard to the scores achieved by students in completing the quiz. The final quiz (multiple choice questions) at the end of the lesson aimed to examine the extent to which students had learned and understood the prepositions. In the instruction phase of CALL, for example, a video picture was used to illustrate the preposition of ÔatÕ (see Fig. 1(a)). This aims to capture the relationship between an event such as catching a bus and the time (Borjars & Burridge, 2001; Jabbour-Lagocki, 1990). Its sound effects included narration to explain the meaning of ÔatÕ, light music to highlight the mood of the background and the sound caused by the movement of an object (e.g., when a bus was coming). Since the preposition ÔatÕ is used to specify a particular point in time, the inclusion of a clock and the routine real life circumstance (Tsou, Wang, & Li, 2002) of school students catching a school bus at a particular time represents a powerful way to illustrate the preposition of ÔatÕ because it provides an appropriate context. Similarly, the movement of a ball in relation to a static chair represents a type of animation, which intends to demonstrate the preposition of place effectively (e.g., in front of a chair). Other video pictures capture routine meal time to illustrate the prepositions of ÔbeforeÕ and ÔafterÕ. In short, the incorporation of relevant real life settings provide contexts for students to grasp the meaning of prepositions in various situations. Some of the exercises provided were interactive in that students could move the objects to learn a preposition (see Fig. 1(b)). This type of interaction with learning materials presumably requires a high level of engagement of the learning task that should benefit students learning of prepositions (Tsou et al., 2002). The exercises were structured into three levels of difficulty. For Level
44
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
Fig. 1. (a) A screen shot of preposition of time in CALL. (b) A screen shot of an exercise.
1, students dragged and dropped (chose a correct preposition from an array of prepositions) to complete a sentence. Level 2 had multiple choice questions. Immediate feedback was provided for Levels 1 and 2 exercises. The use of multiple choice exercises (learner–self interaction) aimed to help students in reflecting (try to understand or make sense) the topic content (Soo & Bonk, 1998). For Level 3, students had to use several prepositions to complete a paragraph, and feedback was only provided when the whole task was completed. Thus, whether a student used one or several preposition(s) incorrectly, he or she had to complete the whole paragraph again. In other words, students may make multiple attempts in order to complete the paragraph accurately. This type of exercise aims to assist students to acquire skills in using appropriate prepositions based on the contexts of a whole paragraph. 1.4. Software evaluation Evaluation of early softwares is usually conducted by teachers who used a rating form to evaluate each feature of a software (MicroSIFT, 1982; Salvas & Thomas, 1984; Tolhurst, 1992).
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
45
McDougall and Squires (1994) regard these evaluation approaches as inadequate because they include only teachersÕ views of the software. Reiser and Kegelmann (1994) share a similar view to McDougall and Squires. They emphasize the importance of inviting students to participate in the evaluation process rather than relying on teachers alone to evaluate the educational effects of a given software. The main issue is whether end-users are able to acquire the skills a software program is designed to teach. Borg and Gall (1989) support this view, and they recommend learners to participate in the process of evaluating educational softwares. Jolicoeur and Berger (1988a, 1988b) proposed that educators evaluated the software; and at the same time, involved the students in the evaluation process. Thus, when students serve as participants in the evaluation process, other evaluators (e.g., teachers) observe the students as they use the software and draw conclusions about the quality of the software. The evaluators may ask students to indicate their opinions about the software and may even test what students have learned as a result of using the software. This means students will be pretested, observed as they work through the software program, and post-tested. We incorporated this approach in this study. In addition, we used a comparative study design in which the efficiency of CALL was compared with a conventional teacher taught lesson. Lockee, Moore, and Burton (2001) argued against the use of media comparison studies. They identified a range of variables (such as learner characteristics, media attributes, instructional strategy choices, and psychological theories) in the comparison studies which made the comparative design inadequate to justify the learning effects of the two instructions. That is, it is difficult to establish the cause and effect of a comparative design because it is almost impossible to match the variables for the participants in a comparison study. In contrast, several researchers (Catenazzi & Sommaruga, 1999a, 1999b; Collin, 2000; Lambert et al., 1996; Tsou et al., 2002) compared the effectiveness of a computer aided instructional presentation with a conventional mode of presenting the same learning materials. One of the aims of designing a comparative study was to replace the computer mode of presenting content with the traditional one should the former supersedes the latter.
1.5. Learning efficiency Cognitive load theory (CLT; Sweller, 1988; Sweller, van Merrienboer, & Paas, 1998) has identified three aspects of cognitive load that learners might experience in acquiring schemas: extraneous load, intrinsic load, and germane load. Extraneous load is related to an instructional design that is detrimental to learning because the design itself requires learners to process activities (e.g., search, integrate, relate pieces of information to become intelligible) unrelated to skill acquisition. The load is intrinsic if the nature of the learning materials imposes different amount of cognitive load upon the learner. For instance, a complex task will require the learner to interact with many elements simultaneously that might be taxing on the working memory load. Germane cognitive load arises when instructions direct the learner to engage in cognitive activities that profit schemas acquisition. This germane cognitive load increases mental effort to process information, but is particularly beneficial when the learning materials have low intrinsic load thus leaving sufficient cognitive resources available for skill acquisition. Taken together, the total cognitive load (intrinsic, extraneous, and germane load) should stay within the learnerÕs cognitive resources if learning is to occur (see Pass, Tuovinen, Tabbers, & Van Gerven, 2003).
46
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
The perceived mental effort required to learn a task refers to the amount of capacity that is allocated to the demand of an instructional presentation (Paas, 1992). Thus, the mental effort exerted to learn a task represents an index of cognitive load involved in learning. Research within the CLT framework has suggested that the learning efficiency of an instructional technique (Marcus, Cooper, & Sweller, 1996; Paas, 1992; Paas & van Merrienboer, 1993; Tuovinen & Paas, 2004; Tuovinen & Sweller, 1999) is a measure of the relationship between mental effort invested and performance. An effective instructional presentation mode decreases mental effort (cognitive load) but increases transfer performance scores. Having completed the CALL lesson, students in this study were required to indicate their mental effort invested in learning prepositions on a seven-point Likert scale (with 1 = extremely low and 7 = extremely high mental effort). This rating scale to assess mental effort is considered as a simple and non-invasive technique (Paas, 1992; Pass & Van Merrienboer, 1994; Tuovinen & Paas, 2004). Test performance was the scores of the correct answers. As the units of mental effort and test scores differ, their respective scores were converted to standardized scores (z scores). Then the standardized mean of mental effort (M), and the standarized mean of performance (P) were combined to indicate the efficiency (E) of learning condition that can be expressed as P M E ¼ pffiffiffi : 2 E can be either positive or negative. The perpendicular distance from the neutral efficiency condition, E = 0, is where performance = effort. The higher the absolute value of E, the more efficient (or inefficient) is the instruction. Note that the square root of 2 in the denominator is used to make the interpretation of the graph easier. This efficiency measurement attributes learning outcomes to the amount of cognitive load required in processing an instructional presentation. For instance, a high E value means an instructional presentation required low cognitive load but results in a high test performance. Indirectly, this efficiency measurement also ensures that the measurement of cognitive load is directly linked to performance outcome rather than other subjective variables (e.g., self-confidence in processing the instructional materials). 1.6. Present investigation The Malaysian Education Ministry (1997) has targeted the conversion of all primary and secondary schools to become Smart Schools by the year 2010, and hence there is a need to design a comparative study to test whether CALL would be better than conventional lesson in terms of learning outcomes. Since students in SMK Bario undertook an ITLP training program under the e-bario project, and this was coupled with the fact that they were targeted end-users of the Smart School program, we invited them to participate in the evaluation of CALL software. Previous researchers reported beneficial effects of multimedia instruction (Catenazzi & Sommaruga, 1999a, 1999b; Philips, Jenkin, Fyle, & Fyle, 1997; Preece & David, 1992; Reynolds & Ehrlich, 1992). However, the assessment of multimedia instruction normally did not include mental effort exerted in the learning process. In this study, the mental effort required to go through the CALL lesson will include the effort invested in attending to the medium as well as its content
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
47
(Lowe, 1999; Schnotz, Bockheler, & Grzondziel, 1999). Students in the conventional group exerted mental effort in attending to a conventional classroom teaching. In sum, the experimental manipulation of this study was the instructional presentation of English prepositions. Transfer performance and mental effort invested in learning the CALL and a conventional lesson were collected to form a basis of comparison. Additionally, we collected qualitative data (post-questionnaire) to examine how students perceived the two alternative modes of instructional presentations.
2. Method 2.1. Participants Participants were 30 students (with mean age 13 years old) from a Form Two class (equivalent to year 7 students) at SMK Bario. Students had 6 years primary and 1 year secondary education. Malay language was the medium of instruction in the school. English language was a compulsory subject in the primary and secondary school curriculum. Prior to the experiment, all participants attended a weekly ITLP for 10 weeks of an hour per session (see Section 1). They had basic knowledge of typing and word processing. Participants were not trained to surf internet. However, some teachers in the school had taught them to use some Smart School softwares to enhance learning of a variety of subjects. None of the participant in this study had used the CALL software prior to the experiment. 2.2. Materials The English teacher in SMK Bario studied the CALL lesson and prepared the conventional lesson and learning activities as closely as possible to that illustrated in the CALL software (see Section 1 on a detailed discussion of CALL). Every effort was made to create a learning environment similar to the one described in the CALL software. For example, a physical chair and a ball were used to illustrate the prepositions: in front of, beside, under, and over. This was similar to one of the exercises inside CALL in which a ball moved around the chair and/or a box. 2.3. Pre-test and post-test The pre-test had the same type and number of questions as the post-test (see Appendix A). All questions were related to the prepositions of time and place. The second author of this paper (who had several years of teaching English language in schools) designed these questions. Part I of this pretest and post-test had 10 multiple choice questions in which students had to choose one preposition out of four to complete each sentence. There were 15 questions in Part II. Students were required to underline the correct preposition (chose one out of two) to complete a sentence. Pretest was conducted to control studentsÕ initial knowledge of English prepositions. In addition, the anticipated differential performance of the pre-test and post-test would shed light on the relative learning efficiency of the two instruction methods.
48
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
2.4. Post-questionnaire In the post-questionnaire (Appendix B), students were required to evaluate the usability (e.g., user-friendliness, the layout of the screen, etc.) of the software. Also, a seven-point subjective scale was provided for students to rate the mental effort invested in learning the prepositions (see Section 1). In the conventional group, the post-questionnaire required students to answer questions related to the manner in which the teacher delivered the lesson, learning activities, and exercises. The aim was to know how students rated a conventional lesson in assisting them to learn English prepositions. Similar to the CALL group, a seven-point subjective scale was provided for student to rate mental effort invested in learning. In sum, the CALL and the conventional lessons were matched with the same content of English prepositions of time (by, on, before, at and after) and place (in, on, near, over, under, between, behind, in front of, and beside). The experimental manipulation was the conditions under which the content was being delivered.
2.5. Procedure The experiment was conducted during studentsÕ normal class lesson. First, a researcher informed the students concerning the aim and procedure of the experiment. Second, students were allowed 15 min to complete a pre-test. Third, students were randomly assigned to either a CALL group or a conventional group with 15 in each group. Fifteen students were requested to attend a CALL lesson in a computer laboratory in school. Because the number of computers was insufficient, students were tested in three separate group sessions. Students completed the CALL lesson at their own pace. The researchers observed that students normally took about an hour to complete the CALL lesson. Throughout the computer session, students were free to ask questions. After the CALL training session, students were given 15 min to complete a post-test; and then they filled in a post-questionnaire at their own pace. For the conventional group, after students completed the pre-test, a subject teacher took about 35 min to deliver the conventional lesson as closely as it was possible in the CALL lesson (see also Section 2.2). After this, students were allowed 15 min to complete the post-test; and they filled in a post-questionnaire at their own pace.
3. Results This study investigated whether the CALL lesson would be superior to the conventionally taught lesson. Out of the data collected from 30 students, two studentsÕ results had to be eliminated due to inadequate literacy level (they could not write or read). Also, one student in CALL group did not complete Part II of the post-test, and his scores were not analyzed. The questions in pre-test and post-test were objective types, and one mark was awarded to each correct answer. A maximum score was 25 marks.
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
49
Table 1 Means (standard deviations) of correct answers for pretest and posttest Instruction
n
Pre-test
Post-test
CALL Conventional
13 14
14.69 (5.02) 15.07 (6.31)
15.00 (5.08) 19.14 (4.26)
Graph 1. Mean scores of CALL and conventional groups in pre and post-tests.
3.1. Group comparison of scores Table 1 presents the means and standard deviations for the pre-test and post-test. A 2 (instruction) · 2 (test) ANOVA with instruction as a between-group factor and test as a within-subject repeated measure showed a significant main effect for the test, F = (1, 25) = 6.60, p = 0.02, a nonsignificant instruction main effect, F = (1, 25) = 1.54, p = 0.23, and a significant interaction effect between the test and instructions, F = (1, 25) = 4.87, p = 0.04. The significant interaction effect was caused by the conventional group outperformed the CALL on the post-test, t(25) = 2.30, SE = 1.80, p = 0.03. Thus, these results revealed that both groups improved their performance from pre-test to post-test but the conventional group showed a greater improvement than the CALL group (see Graph 1). 3.2. Relative condition efficiency In addition to scores, a relative condition efficiency graph (see Graph 2) was produced to examine the relative efficiency of the two methods of instruction. As noted previously, this takes into account of two dimensions: the perceived mental effort invested in learning the prepositions, and the performance scores on posttest. Graph 2 displays the E value and its standard deviation for
50
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
Graph 2. Relative condition efficiency.
the CALL group as well as the conventional group. A t test indicated that students in the conventional group learned more efficiently than the CALL group, t(25) = 2.77, SE = 0.92, p = 0.01. The former is located in the top left corner of the coordinate that represents a high efficiency region; whereas the latter is found in the lower right corner of the coordinate that represents a low efficiency region. 3.3. Post-questionnaire In the post-questionnaire, about 77% of students commented that it was difficult to return to the main menu, while about 85% students indicated that they could not complete the exercises without the teacherÕs assistance. In fact, most students (about 77%) needed frequent help from a teacher while doing the CALL lesson. However, half of the students (about 54%) found the screen easy to follow; and about 92% of the students found the layout of the exercise easy to read. Further, the majority of students (85%) found the CALL motivating, and they would recommend CALL to their friends. Lastly, students indicated that the best aspect of CALL was the exercises. The post-questionnaire for the conventional group showed that all 14 students could follow the teacherÕs explanation. They also found that the organization of the lesson was systematic; and the
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
51
exercises given by the teacher assisted them to understand the lesson. However, only some students (about 57%) found that the teacher actually motivated them to learn and asked challenging questions. The majority of the students (71%) indicated that the best aspect of this lesson was teacherÕs demonstration and explanation; and they would recommend the lesson to a friend.
4. Discussion This study evaluated the effectiveness of a CALL lesson when compared with a conventional lesson in assisting students to learn English prepositions. The results indicate that students who received conventional teaching performed better than those who went through the CALL lesson. Greater improvement from a pre-test to a post-test was observed for the conventional group rather than the CALL group. Further, relative condition efficiency graph also showed that the convention group learned more efficiently than the CALL group with the latter exerting more mental effort and achieving a lower transfer performance than the former. The mental effort required to learn the prepositions may include learnersÕ activities spent in navigating the system as well as in comprehending the multimedia presentation of the materials (Lowe, 1999; Schnotz et al., 1999). Analysis of the post-questionnaire reveals that students experienced difficulty in navigating the CALL lesson (see Q3, Q7, and Q8). In addition, most students were unable to complete the exercises without assistance (Q2); and students were distracted by some features of the content (Q10). Therefore, this finding indicates that neither the system nor the presentation of the content was easy to follow meaning students had to invest a high mental effort to learn the CALL lesson. Consequently, students in the CALL group not only exerted a high mental effort in learning but they learned less efficiently than the control group. Test results contradict a number of research findings investigating the effects of computer assisted multimedia instruction (Catenazzi & Sommaruga, 1999a, 1999b; Phelps & Reynolds, 1999; Philips et al., 1997). It seems that the user interface design of the CALL did not support a self-regulated learning mechanism that has normally been highlighted in a computer assisted multimedia learning environment. In contrast, the post-questionnaire shows that the conventional group could follow the lesson easily. Presumably, students were familiar with their teacher; and the learning activities were easy to understand and do. As a consequence, students in the conventional group used less mental effort but achieved a better transfer performance than students in the CALL group. The introduction of IT in SMK Bario under the e-bario project has equipped students with some basic computer skills to use CALL in enhancing their knowledge of English prepositions. Evidence was provided by the gain in learning from pre-test to post-test (but this result needs to be interpreted with caution because of the sample size. We were unable to use a bigger sample size due to the constraint of the studentsÕ population in SMK Bario). Despite the inferior test performance of the CALL group as compared to the conventional group, most students who received CALL lessons would like to recommend CALL to their friends. Therefore, students from rural schools should be encouraged to undertake ITLP program so that they can participate in the Smart School Program. However, to facilitate the transfer of learning, the design of an educational software package needs to minimize the mental effort required to handle the medium and the content. The difficulty
52
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
in navigating the CALL represents an extraneous cognitive load as it interfered with learning. The use of hierarchical concept map to direct the learner to navigate the learning materials proved to be effective (Lo, Wang, & Yeh, 2004). Hence future research should consider using this approach to lessen the possibility of the studentÕs disorientation in navigating the CALL. Recent cognitive load research in the domain of multimedia instructional presentations (Brunken, Steinbacher, Plass, & Leutner, 2000) has recommended the use of the modality effect with auditory/visual presentations to optimize the working memory capacity. In the instruction phase of the CALL, for example, the written text together with the auditory narration can be presented by auditory mode alone thus freeing an overloaded visual channel to attend to the visual picture. In addition, in the exercise phase of the CALL, the computer feedback showing the right or wrong answer could be replaced with the confidence scores to act as a learning diagnosis and to provide remedial instruction according to diagnosis (Lo et al., 2004). By requiring students to rate their confidence level in selecting their answers while they work through the exercises may lead to in-depth understanding of the appropriate use of prepositions in various contexts. To conclude, the evaluation of software can provide information concerning the suitability of the softwares for end-users. This research collected both qualitative and quantitative data to assess the CALL; and this coupled with the learning efficiency measurement based on cognitive research serves as an appropriate model of software evaluation for educators, and commercial software developers.
Acknowledgement This study was supported by a Grant from the Canadian International Development Research Centre. We thank Lucy Bala, the Headmaster of Bario Government Secondary School in assisting the collection of data, Elaine Guat Lien Khoo and Roger Harris for helpful comments on earlier versions of this paper. Appendix A. Pre-test and post-test questions Part I: Circle a correct answer his parents 1. The child is sitting A. at B. over C. in D. near the bank 2. You must save your money A. on B. in C. by D. over the wall 3. He hung his motherÕs photograph A. at B. in C. on D. by 8 oÕclock on Monday 4. There will be an assembly A. by B. on C. at D. in
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
53
Part II: Underline the correct answer 1. You can do your work (after, on) lunch 2. I will come again (on, at) Saturday 3. LetÕs finish our work (before, by) the bell rings 4. The ball went (in front of, under) the table and I had to kneel down to get it
Appendix B. Post-questionnaire: CALL group
Questions
Yes
No
1. Did you find the introductory page useful? 2. Easy to return to the main menu? 3. Were you able to complete the exercise independently (without the help of a teacher)? 4. Menus, icons and buttons were easy to understand and pleasing to look at? 5. Did you find the screen easy to read (appropriate matching of background and text color)? 6. Did you find the layout of this computerized learning exercise attractive to ÔlookÕ at? 7. Was page or screen layout easy to follow? 8. Was page or screen layout consistent throughout? 9. Did you notice any grammatical, numbering or typographic errors? 10. Were you distracted and irritaded by one or more features of the course? 11. Were graphics clear, not distracting, and donÕt complete with content? 12. This computerized learning was extremely motivating and made it likely that you would learn the content? 13. Would you recommend this computerized learning to your friends? 14. What was the best thing you like about this computerized learning? 15. What things did you dislike about this computerized learning?
13 3 2
0 10 11
10
3
12
1
12
1
7 2 6
6 11 7
7
6
8
5
11
2
10
3
Comments
Exercises, quiz None
References Borjars, K., & Burridge, K. (2001). Introducing English grammer. Published in Great Britain by Arnold, a member of the Hodder Headline Group, London NW1 3BH.
54
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
Brunken, R., Steinbacher, S., Plass, J. L., & Leutner, D. (2000). Assessment of cognitive load within multimedia learning by the dual task methodology. Experimental Psychology, 49, 109–119. Catenazzi, N., & Sommaruga, L. (1999a). The evaluation of the Hyper Apuntes interactive learning environment. Computers & Education, 32, 35–49. Catenazzi, N., & Sommaruga, L. (1999b). The evaluation of the Hyper Apuntes interactive learning environment. Computers & Education, 32, 35–49. Collerson, J. (1995). English grammar: A functional approach. Australia: Primary English Teaching Association. Collin, M. (2000). Comparing web, correspondence and lecture versions of a second-year non-major Biology course. British Journal of Educational Technology, 31(1), 21–27. Harris, R. (1999). Rural information technology for SarawakÕs development. Sarawak Development Journal, 2(1), 72–84. Jabbour-Lagocki, J. (1990). Prepositions of position: an analysis for practical application in the classroom. ERIC Document Reproduction Service No. ED 427 547. Jin, Z. Y. (1989). Difficulties in learning English for Chinese students. Test Talk, 13(3), 87–91. Jolicoeur, K., & Berger, D. E. (1988a). Implementing educational software and evaluating its academic effectiveness: Part I. Educational Technology, 28(9), 7–13. Jolicoeur, K., & Berger, D. E. (1988b). Implementing educational software and evaluating its academic effectiveness: Part II. Educational Technology, 28(10), 13–19. Lambert, T., Shepherd, J., Ngu, A., Ho, P., Whale, G., & Geissinger, H. (1996). Bridging the gap: Computer science meets distance education at UNSW. Australia: School of Computer Science and Engineering, University of New South Wales. Lo, J., Wang, H., & Yeh, S. (2004). Effects of confidence scores and remedial instruction on prepositions learning in adaptive hypermedia. Computer & Education, 42, 45–63. Lockee, B., Moore, M., & Burton, K. (2001). Old concerns with new distance education research. Educause Quarterly, 24(2), 60–62. Lowe, R. K. (1999). Extracting information from an animation during complex visual learning. European Journal of Psychology of Education, 14, 225–244. Mahathir, M. (1991). Malaysia: The way forward. In Proceedings of the paper presented at the Malaysian business council, Center for Economic Research and Services, Kuala Lumpur. Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of Educational Psychology, 88(1), 49–63. McDougall, A., & Squires, D. (1994). Perspectives interactions: a new approach in the selection of educational software. In Proceedings of the paper presented at the APITITEÕ94, Brisbane. MicroSIFT. (1982). EvaluatorÕs guide for microcomputer-based instructional packages, The International Council for Computers in Education, Wugene, Oregon. Ministry of Education, Malaysia. (1997). A vision of the Malaysian Smart School [On-line]. Available from: http:// eprd.kpm.my/exesum.html/. Paas, F. G. W. C. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach. Journal of Educational Psychology, 84, 429–434. Paas, F. G. W. C., & van Merrienboer, J. J. G. (1993). The efficiency of instructional conditions: an approach to combine mental effort and performance measures. Human Factors, 35(4), 737–743. Pass, F., & Van Merrienboer, J. J. G. (1994). Variability of worked example and transfer of geometrical problem solving skills. A cognitive approach. Journal of Educational Psychology, 86, 122–133. Pass, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38, 63–71. Phelps, H., & Reynolds, R. (1999). Formative evaluation of a web-based course in meteorology. Computers & Education, 32, 181–193. Philips, R., Jenkin, N., Fyle, G., & Fyle, S. (1997). The user interface design of learner-centred interactive multimedia programs. Retrieved January 15, 2000 from World Wide Web: http://cleo.murdoch.edu.au/tlc/Philips/Ed-Media97/ Paper294.html.
B.H. Ngu, S. Rethinasamy / Computers & Education 47 (2006) 41–55
55
Preece, D., & David, G. (1992). Multimedia: some promises, some problems and some issues in human–systems interaction. In Proceedings of the international interactive multimedia symposium perth (pp. 259–268), Western Australia. Reiser, R. A., & Kegelmann, H. W. (1994). Evaluating instructional software: a review and critique of current methods. Educational Technology, Research and Development, 42(3), 63–69. Reynolds, L., & Ehrlich, D. (1992). Multimedia in industry and education: a decision model for design. In Proceedings of the international interactive multimedia symposium perth (pp. 117–126), Western Australia. Salvas, A., & Thomas, G. (1984). Evaluation of software. Melbourne: Education Department of Victoria. Schnotz, W., Bockheler, J., & Grzondziel, H. (1999). Individual and co-operative learning with interactive animated pictures. European Journal of Psychology of Education, 14, 245–265. Soo, S.-K., & Bonk, C. J. (1998). Interaction: what does it mean in online distance education? In Proceedings of the paper presented at the EDDMEDIA & ED-TELECOM 98, Freiburg, Germany. Sweller, J. (1988). Cognitive load during problem solving. Cognitive Science, 12, 257–285. Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296. Tolhurst, D. (1992). A checklist for evaluating content-based hypertext computer software. Educational Technology, 32(3), 17–21. Trang, T. (1999). The perception of teachers in SMK Bario towards the use of the computer. Unpublished MasterÕs Thesis, Universiti Malaysia Sarawak, Kota Samarahan, Kuching, Sarawak, Malaysia. Tsou, W., Wang, W., & Li, H. (2002). How computers facilitate English foreign language learners acquire English abstract words. Computer & Education, 39, 415–428. Tuovinen, J. E., & Paas, F. (2004). Exploring multidimensional approaches to the efficiency of instructional conditions. Instructional Science, 32, 133–152. Tuovinen, J. E., & Sweller, J. (1999). A comparison of cognitive load associated with discovery learning and worked examples. Journal of Educational Psychology, 91, 334–341.