- Email: [email protected]
Assessment and feedback as student support devices
Pergamon
Studies in EducationalEvaluation, Vol. 23, No. 3, pp. 257-273, 1997 © 1997 Elsevier Science Ltd Printed in Great Britain. All rights reserved SOI91-491X(97)O0016-3 0191--491X/97 $17.00 + o.oo
A S S E S S M E N T A N D F E E D B A C K AS S T U D E N T S U P P O R T DEVICES Rob Martens and Filip Dochy Open University, Heerlen, The Netherlands
In this article we will address the effects of prior knowledge and progress assessment upon learning behaviours and motivation. This means that not the relation between for example scores on prior knowledge tests and examination results is investigated, but the effects (on the study process) of making prior knowledge or progress tests and reading the accompanying external feedback. Thus, prior knowledge tests and progress tests are methods of student support. Prior knowledge is defined as follows: 'prior knowledge is dynamic in nature, is available before a certain learning task, is structured, can exist in multiple stages (i.e., declarative, procedural, and conditional knowledge), is both explicit and tacit in nature, contains conceptual and metacognitive knowledge components' (Dochy & Alexander, 1995, p.228). What is measured by a progress test bears a similarity to the concept of prior knowledge, but of course in the present case the knowledge is built up in the course of a certain task or study. The context of this research is self study. The Open University (OU) of the Netherlands develops - to foster an independent learning process of its adult students sets of self study learning materials for distance education. De Wolf (1994, p. 1557) defined distance education as follows: "Distance education is described by a variety of labels such as 'correspondence education', 'home study', 'independent study', 'external studies', 'distance teaching' and 'open learning'". We define open learning systems as "flexible learning systems which support a learning process in which there is a delicate balance between a maximal student freedom of choices and the optimal adaptation to characteristics of the learner, in order to guarantee a maximal degree of study success" (Valcke, Dochy, & Daal, 1991, p. 3). These forms of learning have a growing influence on traditional education (e.g., Sewart, 1995; Sorensen, 1995). 257
258
Rob Martens and Filip Dochy
In situations where there is less contact between students and teachers/tutors, the material has to be sufficiently supportive. Therefore an essential part of self study materials consists of embedded support devices (ESD). ESD represent a set of formal and contentrelated add-ons, extensions and elaborations of the study materials, such as pre- and postquestions, schemes, illustrations, tasks, summaries, advance organisers, objectives and study guidelines. ESD aim at scaffolding the learning process. Course developers assume ESD have certain functions and effects, i.e., facilitation of access to the content (e.g., overviews), facilitation of the processing of the content (e.g., by using advance organisers, activities, pre-questions) and facilitation of the assessment of mastery of the content/ objectives, for example by embedding questions and test-items with feedback. "Questions with feedback" is one of the most important supportive tools for students (e.g., Butler & Winne, 1995; Lockwood, 1995). With the emergence of computers, assessment is becoming more and more of an integral part of educational software (e.g., Dochy, 1992). Computers can also be used to supply fast and adequate feedback to students on the basis of printed tests (e.g., prior knowledge tests and progress tests). ESD are designed to evoke certain learning behaviours. Even though developers' self study materials make intensive use of ESD, implicitly accepting the functionality and effectivity of ESD, there is not very much empirical evidence to ground these functions and effects in an open learning context (e.g., Lockwood, 1992; 1995; Lowyck & Simons, 1991; Martens & Valcke, 1995). Researchers have raised questions about the usefulness of some ESD (e.g., Schuemer, 1993; Marland, Patching, Putt & Putt, 1990; 1992). At the Dutch OU a series of research studies was conducted, and the following conclusions were reached with regard to questions and feedback in the printed study materials (Martens, Valcke, & Portier, in press; Martens, Valcke, Poelmans, & Daal, in press): 1. Questions and feedback are used very frequently (about 90% of the students uses this type of embedded support). 2. On a scale from 1 to 5 students indicate an average of 3.6 for the amount of support they get from questions in printed study materials. 3. The use of a combination of ESD has a demonstrably positive effect on learning outcomes. To date we have no insight in the learning effect of questions and feedback in open learning. In the research on learning effects that we did so far we only used a combination of ESD. Questions and feedback were a part of this complex of embedded support and could not be analysed separately. These results, in line with Dochy, Moerkerke, and Alexander (1995), show high student appreciation of prior knowledge tests and progress tests in open learning. A survey of research findings on effects of questions and feedback (e.g., Helgeson & Kumar, 1993; Rieber, 1994; Wiggins, 1993) shows that effects are not always consistent. Sometimes effects of assessment are positive, sometimes there are no effects, and they may even be negative. An important explanation for this might be interaction between student characteristics and assessment. This means that a certain feedback mode is not optimal for all students but that the optimal mode depends on learner characteristics (Boekaerts &
Student Support Devices
259
Simons, 1993; McGinn & Winne, 1994). Another explanation for contradicting research results can be found in the poor ecological validity of research concerning questions in self study materials, according to Lockwood (1995). Butler and Winne (1995) in their survey on feedback in self-regulated learning, criticize the narrow focus on the effects of feedback on achievement, thus neglecting interacting factors in self-regulated learning processes. In this article we will focus on the functions and effects of assessment by means of progress tests and prior knowledge tests. We consider tests with feedback as a form of embedded support, but with one major difference: the tests can be more or less embedded in the material, but the feedback should not be built in. This is to prevent what Kulhavy described as presearch availability. Feedback can sometimes have a detrimental effect on learning processes. Students tend to simply copy the correct answers when feedback is presented in written instruction. In the research design we opted for delayed feedback. Students have to generate their own answers first before they get the correct answers. In a meta-analysis a positive effect was found .for delayed feedback in test acquisition tasks (Bangert-Drowns, Kulik, Kulik & Morgan, 1991). Definition and Types of Feedback The concept of feedback is crucial in educational psychology (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991; Wiggins, 1993): "It is an accepted psychological principle that one of the essential elements needed for effective learning to take place is feedback. Knowledge of results is required to assess progress, correct errors and improve performance" (Fletcher, 1991, p. 103). The number of possible feedback modes is almost infinite. Butler and Winne (1995) make a distinction between internal (self generated) feedback and external feedback. Some examples of dimensions that can be used to characterise external feedback are: immediate or delayed feedback, feedback with or without the correct answer and with or without study guidelines. Feedback was found to be most effective when (1) it was unavailable to students prior to the construction of their own responses; (2) it provided more than verification information (e.g., 'That's right'), that is, it provided subject matter information with the correct answer and possibly further elaboration; (3) it was not preceded by a pretest, and (4) it was associated with a complex task providing few cues or instructional supports (McGinn & Winne, 1994). Tests and feedback have three main functions: 1. Feedback offers information about the study process and the mastery of the learning goals. This information is of particular interest in learning situations were there is no direct contact between learners and teachers. This important role of tests and feedback is illustrated for instance in the Keller-plan or personalised system of instruction (PSI) (Crombag, 1979) which is clearly related to more recent systems of open learning (e.g., de Wolf, 1994). Roossink (1990) describes such provision of information as the cognitive function of feedback. Feedback can confirm students' conceptual understandings or beliefs, it can add information, it can overwrite incorrect elements, tune understandings and finally it can restructure schemata if
260
.
.
Rob Martens and Filip Dochy
students hold false theories (Butler & Winne, 1995). Besides these effects on knowledge and beliefs, feedback can also influence cognitive processing (tactics and strategies). Feedback can be interpreted as a positive reinforcer or as a negative reaction (Boekaerts & Simons, 1993). The impact of feedback as reinforcement is in part determined by the interpretation that the individual student makes (e.g., his or her attribution style) (Alexander, 1993; Boekaerts & Simons, 1993). Prior knowledge assessment or progress assessment can also directly influence the study process by means of activation of prior knowledge (Dochy, 1992; Spires, Donley, & Penrose, 1990). Often also the tests used are similar to the final exam and offer training in solving a certain type of items.
The first two functions mentioned above are functions of feedback, the third is a function of testing. These functions of feedback indicate the possible impact that feedback may have on learning outcomes. Although the time used to fill in tests and to read the feedback is often relatively short when comparing other learning activities, these hypothetical functions and effects suggest an important impact on study outcomes. Thus, feedback may lead to changes in study behaviour like for instance rereading, studying with more effort, studying different subjects. From this point of view feedback primarily influences the metacognitive processes of students. Possible effects of feedback are: 1. Effects on (the sequence) of the subjects studied. Prior knowledge or progress assessment may cause students to study some parts of the study material again, skip topics or study subjects in a different order. This intermediate information about study progress is of crucial importance in modular, open learning situations (Dochy, 1992). 2. Effects on the way students study. One effect of feedback might be that students alter their study method. This is especially to be expected when the feedback contains advice or study guidelines (e.g., Wagemans, Portier & van Buuren, 1994). Positive feedback might encourage students and negative feedback may incite students to change their study method. 3. Effects on the effort, learner engagement or study time. Positive feedback can motivate students positively and stimulate them to persist. Negative feedback can stimulate students but can also discourage them, depending on student's attribution style. Problem statements such as used in prior knowledge and progress tests can enlarge student's motivation (Boekaerts & Simons, 1993). The assumptions we made above have three important implications (see Figure 1): 1. We assume that prior knowledge and progress assessment have a measurable effect on variables concerning study behaviour. 2. It is assumed that these effects are not equal for all students. The effects depend on type of feedback (positive or negative). 3. Tests can have a direct effect on study outcomes by means of activation of prior knowledge or test training.
261
StudentSupportDevices
priorknowledgeor progresstests
studyoutcomes ~
p°sit~;dbnkgati~
Figure 1: A Model for the Effects of Tests and Feedback Research Questions In this research we will focus on the effects of feedback and try to validate the theoretical functions and effects that are ascribed to feedback. Concerning the functions and effects there are five research questions: 1. Does feedback from progress- and prior-knowledge tests provide relevant information for students? 2. Does feedback lead to alterations in the subjects or the sequence of the subjects studied by students? 3. Does feedback influence the way students study (study method)? 4. Does feedback influence learner engagement or study time? 5. Do students report test training effects? The direct effects of testing on learning outcomes are not fully measured in this research, although we will investigate if students report them. We will not investigate the activating effect on prior knowledge as a result of the testing procedure. The effects on study results and study speed is reported on in a related study. Moerkerke, Dochy, and Martens (1995) and Moerkerke (1995) found that there is no effect of prior knowledge or progress assessment on study results. However, a significant effect is found on study speed/attending examinations: after seven months of study significantly more students in courses for beginners attended the exam. This effect was not found for students studying more advanced courses. In the present study we try to explain these results by taking into account variables related to the study process and students' motivation. This study is an attempt to understand the processes that occur in students that may have caused the effects we mentioned above. This research is based on the same four courses as used by Moerkerke, Dochy, and Martens (1995) (Information science, Marketing, Business, Management). We assume that all the above research questions can be answered positively and thus can be interpreted as expectancies. A distinction will be made between students who receive "positive" feedback and students who receive "negative" feedback. It is expected that the first group will be positively reinforced by feedback and will not change their study method, while the second group will change study method since the feedback signals a problem to them.
262
Rob Martens and Filip Dochy
Research Method, Materials and Subjects Two questionnaires were distributed to students in the experimental group and in the control group: one at the beginning of the study (questions about student's opinion about assessment) and one after completing the final examination (the same questions and also, depending on the conditions, questions about the tests, the feedback, motivation, study process, etc.). The prior knowledge tests and the progress tests usually consist of 40 multiple choice items each. Every question is classified along two categories: behaviour (knowledge, insight, application questions) and knowledge (facts, concepts relations, structures, methods). Feedback consists of indications of the percentage of correct answers per category and an indication of the place in the course where the main subject of a question is treated. In the feedback a norm of 70% is presented to the students, meaning that when more than 70% of the answers is correct this is sufficient while topics with less than 70% correct answers should be studied more profoundly. Less than 70% is interpreted as negative feedback. Note that the distinction between positive and negative is not controlled experimentally but depends only on students' performance. Experimental variation in positive or negative feedback, when not based on real scores, is considered nonethical with students studying nonexperimental, regular courses. Besides the correct answers the feedback also provided short explanations and separate scores on the (behaviour and knowledge) dimensions we distinguished, together with some explanation and guidelines. This distinction between dimensions is a knowledge profile. The behavioural and knowledge dimensions were proven adequate in determining prior knowledge profiles in higher education (Dochy, 1994; Dochy, Valcke, & Wagemans, 1991). So the feedback not only provided content information but also advice. Feedback was sent back within two weeks after receipt of the completed test forms. Table 1: Test Information Content domain Prior k. marketing Progress 1 marketing Progress 2 marketing Progress 3 marketing Prior k. information scoemce Progress 1 information science Progress 2 information science Progress 3 information science Prior k. business Progress 1 business Progress 2 business Progress 3 business Prior k. management
Number of Items 40 40 40 40 50 40 3O 30 40 40 40 40 31
n 55 106 74 53 53 95 6O 37 26 44 40 30 58
Cronbach's alpha .60 .63 .69 .72 .73 .72 .84 .81 .48 .69 .58 .80 .75
Student Support Devices
263
In Table 1 basic information about the prior knowledge and progress tests is presented. Our first aim was to guarantee content validity (e.g., representation of all knowledge levels) in the tests. Since the test length had to be restricted our main aim was not to make alpha as high as possible. The subjects were regular students, taking a regular course. They could either be in the control group or volunteer to fill in the prior knowledge or progress tests. There were four research groups: • Control group: These students do not fill in prior knowledge or prior knowledge tests. They only complete questionnaires concerning motivation, study planning, study method and study satisfaction. • Prior knowledge test: Students in this group receive at the beginning of their studies, a prior knowledge test and some questionnaires. After filling in the prior knowledge test students receive feedback with comment on their results. • Self paced progress tests: Students in this group receive at three points during their studies, a progress test and some questionnaires. When a student reaches a certain point in the study material he or she can apply for a progress test (self paced). After filling in the progress test they receive feedback with comment on their results. • Progress tests paced by the institute: Students in this group receive at three fixed points in time (with a fixed intermediate period) in their study a progress test and some questionnaires. After filling in the progress test they receive feedback with comments on their results. Two courses are "starter" courses, i.e., these courses are the first in a sequence that eventually leads to a diploma; the two other courses are advanced courses. Students w h o study these latter courses can be assumed to have more experience with self study in higher education. Starters courses are information science and marketing, and the two advanced courses are business and management. After filling in the questionnaires and/or the prior knowledge test and/or the progress tests students take a regular final exam. Only after this final exam they received the final questionnaire. From the total research sample of 1482 students who registered for certain courses, 404 students were in the control group, 360 in the prior knowledge group, 358 in the progress group paced by the institute and 360 in the self paced progress group. About 52% of the subjects in our sample had not taken an exam. The average percentage of students not taking the exam for the entire university is 53% after one year (Boon et al., 1995). After one year this percentage hardly increases. This means that we can assume that in the present data set almost all subjects that can be expected to take an exam are present. Differences between groups were analysed by means of univariate tests such as ttests and univariate analysis of variance. When looking at effects of correlated dependent variables jointly, MANOVA was used. Basic assumptions underlying these analyses such as normal distribution of variables were checked for. In cases of doubt non-parametric measures were used.
264
Rob Martens and Filip Dochy
Results The response rates were based upon returning the (first) questionnaire. Students who were registered for a series of courses were not included in this analysis because they might not start with the course that our tests addressed. In the experimental groups returning the questionnaire often was accompanied by returning the first test (prior knowledge test). For the complete group of subjects the response percentage was 47%. The response rate in the no-treatment group was 54%, in the prior knowledge test group 57%. This difference is not significant. The percentages for the progress groups were different because students had to study 25% of the course before sending back the first progress test and questionnaire. Taking into account the regular drop out, response rates in the progress test groups can be expected to be lower: for the group paced by the institute this was 42% and for the student paced condition 32%. These percentages are based upon a seven-month enrolment period. First students' perceptions of the tests and the feedback were explored. The following results are therefore limited to students in the experimental conditions. They answered several questions after doing their exam. The questions were Likert-type questions with 5-point scales, ranging from "completely disagree" (1) to "completely agree" (5). Scores 1 to 3 on a statement were interpreted as a negative response to the statement, scores 4-5 were interpreted as a positive (agreement) response to that statement. The three experimental groups differed significantly (MANOVA, Wilks's Lambda = .69, p<0.0001). Analysis showed that for all variables in Table 2 the progress test group differed significantly from the prior knowledge test groups.
Table 2: Perception of Tests with Feedback as Reported by Students (N=195) Variable/group Finds that doing the test(s) was useful Agrees that the feedback provides useful information about own knowledge level Finds that the test was a useful exercise for the information Find the feedback was understandable Finds that the feedback provides substantial information Agrees that the feedback was given at the fight time
Prior knowledge
Progress self
Progress OU
44.9% 51.4%
82.3% 63.1%
81.3% 60.2%
53.6% 51.5% 44.8% 68.2%
85.3% 63.6% 60.6% 58.8%
87.8% 74.7% 63.0% 54.9%
Generally the feedback is not judged as positively as the tests. Apparently not all students find the feedback that was provided sufficiently informative or clear. The results show that feedback has to be detailed and informative. The feedback that was provided in this experiment can be improved. High percentages of students report training effects with regard to the final exam (statement 3). The progress tests with feedback are judged more positively than the prior knowledge tests.
Student Support Devices
265
Table 3: Perception of Tests with Feedback, Split Up for Positive and Negative Feedback (N=185) Variable
Positive feedback Mean and SD
Test useful Information about knowledge level Useful exercise Feedback understandable Substantial information In time
4.5 3.7 4.5 3.9 3.8 3.4
(.81) (.91) (.89) (.87) ( 1.0) (1.4)
Negative feedback Mean and SD 3.9 3.7 3.9 3.6 3.5 3.9
(1.2) (1.1) (1.1) (1.0) ( 1.0) (1.2)
t-value t(167.08)=4.34" t (186)=-.33 t (187)=-3.17 t (139.59)=-2.10 t ( 187)=- 1.44 t (99,22)=2.29
* = p < .05
Table 3 is based on the same 5-point scales as in Table 2. From this table it can be concluded that perceptions of tests differ for students who receive positive or negative feedback (MANOVA, Wilks's Lambda = .86, p<0.0001). Students with insufficient results are significantly more negative about the usefulness of the tests. In the next sections the results in terms of study behaviour and motivation will be analysed. To do so the results of questionnaires taken from students in the experimental conditions will be compared to students receiving no experimental treatment. In this analysis we will compare the results on eleven multiple choice questions, with 10 point scales and two open questions concerning perceived study time in hours. We summarise the 13 variables in Figure 2. Use of study plan : Alteration of study plan : Change of order: Change of study speed: Study hours per week: Study hours for first exam: Difficulties with planning: Perception of learning material: Insight in final goals: Results unexpected: Positive about study: Change of study method: Decrease of motivation: Figure 2:
I used a study plan for this course I changed my study plan during the course I changed the order of the subjects I studied I changed the sPeed of my studies Average study hours per week (open question) Total amount of hours before the first exam (open question) I found it difficult to plan my studies Well structured I knew what I had to do to pass my exam The results of my examination were different from what I expected I feel positive about studying the course While studying the course I altered my study method While studying my motivation has decreased
Variables about Study Experiences in Questionnaire After Exam
In Table 4 the results are presented. The rows represent one question each. The variables in column one are presented in Figure 2. The second column presents the mean scores for the entire population in this research. The third column shows the results of the subjects who scored on average less than 70% on the test(s). This is interpreted as subjects receiving negative feedback. The fourth column represents the subjects with positive feedback, the fifth contains subjects in the experimental groups who did not respond to
266
RobMartensand FilipDochy
the tests and the last column contains the scores of the subjects who only received the questionnaires and had no further treatment. Table 4:
Effects of Assessment (n=333)
Variable
Complete sample Mean (SD)
Use ofstudyplan 6.2 (3.1) Alteration of study plan 5.1 (3.3) Change of order 2.2 (2.3) Change of study speed 4.5 (3.2) Study hoursperweek 12.7 (8.3) Study hours for first exam 136.8 (75.5) Difficulties with planning 3.0 (2.5) Leaming material: well 8.2 (1.7) structured Insight in final goals 7.8 (1.9) Results unexpected 4.3 (3.0) Positive aboutstudy 7.9 (1.9) Change of study method 2.7 (2.4) Decrease of motivation 3.0 (2.5)
Test: insufficient (negative feedback) Mean (SD)
Test: sufficient (positive feedback) Mean (SD)
Mean (SD) Mean (SD)
5.9 (3.2) 4.9 (3.4) 2.2 (2.3) 4.5 (3.2) 11.8 (8.8) 135.9 (74.2) 3.1 (2.6) 8.3 (1.6)
6.3 (3.2) 5.1 (3.4) 1.9 (1.9) 4.5 (3.3) 13.6 (7.7) 161.0 (64.5) 2.5 (1.9) 8.4 (1.5)
6.1 (3.2) 6.4 (3.1) 5.2 (3.2) 5.2 (3.2) 2.7 (2.7) 2.3 (3.2) 5.0 (3.3) 4.2 (3.2) 1 3 . 7 (8.5) 12.8 (8.1) 127.6 (88.9) 126.5 (75.6) 3.6 (3.2) 3.1 (2.5) 8.1 (1.7) 7.9 (2.0)
7.8 4.3 7.8 2.7 3.2
(2.0) (3.0) (1.9) (2.3) (2.6)
8.1 3.3 8.2 2.6 3.0
(1.6) (2.5) (1.7) (2.4) (2.6)
No test send back
7.8 5.2 7.7 3.2 3.0
Controlgroup
(1.6) (3.0) (2.1) (2.6) (2.6)
7.7 4.4 7.9 2.6 2.9
(2.1) (3.2) (1.9) (2.3) (2.3)
The effects of receiving tests and feedback, and the effects of the type of feedback (negative or positive) were examined jointly by means of a MANOVA-model. No effect was found (Wilks's Lambda = .84; p=.248). Nevertheless, the large differences in study hours are remarkable. Inspection of the univariate F-tests showed one significant effect: study hours per week. The amount of study hours in the experimental groups who had positive feedback was higher than that of the other groups (F(3;261)= 2.8; p<.05). Instead of concentrating on the effects of type of feedback, the influence of the conditions can be analysed. We could not examine both effects together since the analysis would then contain more than ten groups, leading to empty cells. We repeated the MANOVA with the four conditions as grouping variable. The multivariate tests for significance showed a significant effect (Hotteling's Trace .224; p<.05; Wilks's Lambda = .80; p=.05). This means that there are no strongly significant differences between students receiving prior knowledge and progress test and students who received no such tests with feedback. Univariate analysis showed there was one effect. Students in the progress tests groups study significantly longer before they take their first exam (see Table 5). Table 5: Study Hours as a Dependent Variable of Condition (n=318)
Study hours
Control group
Prior knowledge test
Progress test paced Progress test self by the institute paced
Mean 126.5
Mean 113.1
Mean 156.1
SD (75.6)
SD (70.9)
(SD) (67.9)
Mean 156.0
(SD) (78.9)
Student SupportDevices
267
I n s t e a d o f c o m p a r i n g groups and l o o k i n g for significant differences w e can also l o o k at students' o w n p e r c e p t i o n s o f the effects o f prior k n o w l e d g e or progress assessment. The f o l l o w i n g results are thus limited to students in the e x p e r i m e n t a l conditions. T h e y a n s w e r e d s o m e questions after taking their exam. The questions were m u l t i p l e c h o i c e w i t h 5 p o i n t scales, ranging from completely disagree (1) to completely agree (5). Scores 1 - 3 on a statement are interpreted as a n e g a t i v e response to the statement, scores 4-5 are interpreted as a p o s i t i v e (agreement) response to that statement. Table 6: Effects of Tests with Feedback as Reported by Students in Experimental Groups (n=200) 5.5% report to have changed study method as a result of feedback received * 23.5 % have studied parts of the course longer or again as a result of the feedback * 4.5 % of the students report to have skipped some parts of the course as a result of the tests with feedback * 15 % report to have studied longer as a result of the feedback.* • = proportion differs significantly from .50 (p<0.01, binomial test) Again, w e can split these results into positive and n e g a t i v e feedback. N o n e o f these f o u r questions s h o w e d a significant difference b e t w e e n students receiving positive f e e d b a c k and students w h o got n e g a t i v e feedback. T h e f o l l o w i n g analysis shows students' opinions about functionality o f assessment. O n l y the data o f students in the e x p e r i m e n t a l groups are used. T h e results are d i s p l a y e d in Table 7. Students' o p i n i o n at the b e g i n n i n g o f their studies is reported. The same questions were p o s e d at the end of the course, after c o m p l e t i n g the final exam. The questions were on 5-points scales, ranging form 1 (completely disagree) to 5 ( c o m p l e t e l y agree). T h e opinion significantly c h a n g e d for one variable. This difference is i n d i c a t e d in Table 7.
Table 7: Students' Opinions about Assessment (n=177) Variable
Average score
Wants prior knowledge tests Function: To assess the required prior knowledge Function: To assess to what extent a student already masters the course material Function: To know which parts are to be studied at a deep level and which at a surface level Function: To support a study plan Function: To determine a learing plan Function: To support the choice for a certain course Function: To prepare faster for the exam Function: To better organise the study Function: To be able to address study counsellors more adequately Function: To be able to motivate demands for extra study support Progress tests are a useful tool for OU students to gain insight in their study progress
3.3 1.4 1.4; after exam: 1.3" 1.6
* T-value (179) = 3.76; p<0.001
1.6 1.6 1.6 1.6 1.7 1.8 1.8 4.3
268
Rob Martens and Filip Dochy
From Table 7 it can be concluded that students find prior knowledge and progress tests useful. About the functions of prior knowledge tests there is less agreement. Students' opinions hardly change during the course of their studies and are hardly influenced by the tests with feedback, or by the fact that some respondents did their final exam and others not. Comparisons between students in the starter courses and students in the advanced courses were carded out. It was found that these two groups differ on the set of questions we used earlier (see Table 4; Wilks's Lambda = .77, p<0.001). Table 8 presents these differences. Table 8: Effects of Assessment, Split Up for Starters and Advanced Students (n=265) Variable
Starters Mean (SD)
Use of study plan Alteration of study plan Change of order Change of study speed Study hours per week Study hours for first exam Difficulties with planning Learning material: well structured Insight in final goals Results unexpected Positive about study Change of study method Decrease of motivation
6.1 5.2 2.4 4.7 12.6 158.6 3.2 8.4 8.0 4.0 8.1 3.0 2.9
(3.2) (3.4) (2.5) (3.2) (8.8) (70.0) (2.7) (1.6) (1.8) (2.9) (1.8) (2.6) (2.4)
Advanced Mean (SD) 6.3 4.9 2.0 4.1 12.7 107.9 2.8 8.0 7.6 4.5 7.6 2.3 3.2
(3.1) (3.2) (2.0)* (3.2)* (7.8) (73.1)* (2.3) (1.9)* (2.0) (3.1) (2.0)* (2.0)* (2.6)
We repeated the MANOVA in which the effects of the condition are analysed, split up for starters and students taking the advanced courses. Table 9: Effects of Assessment for Advanced Courses Only (n=265) Variable
Use of study plan Alteration of study plan* Change of order Change of study speed Study hours per week* Study hours for first exam* Difficulties with planning Learning material: well structured Insight in final goals Results unexpected Positive about study Change of study method Decrease of motivation *--p <.05
Complete sample
Control group Mean (SD)
Prior knowledge test Mean (SD)
Progress test paced by the institute Mean /SD)
Mean (SD) 6.3 4.9 2.0 4.1 12.7 107.8 2.8 8.0 7.6 4.4 7.6 2.3 3.2
(3.1) (3.2) (2.0) (3.2) (7.8) (73.1) (2.3) (1.9)
6.5 5,6 2,3 4.5 12.2 100.2 3.2 7.7
(3,1) (3.2) (2.3) (3.4) (7.0) (60.4) (2,6) (2.1)
6.0 4.0 1.9 4.2 11.6 84.8 2.8 7.8
(3.2) (Z8) 0.9) (3.1) (9.0) (60.4) (2.1) (1.0)
6.0 (3.4) 6.5 (3.1) 4.1 (3.4) 5.0 (3.6) 1.7 (1.7) 1.8 (1.3) 3.1 (2.7) 3.9 (3.3) 11.3 (4.0) 17.3 (8.7) 165,5 (100.3) 126.3 (80.7) 2.1 (1.9) 2.2 (1.8) 8.8 (1.0) 8.5 (1.7)
(2.0) (3.1) (2.0) (2.0) (2.6)
7.3 4.3 7.6 2.4 3.2
(2.2) (3.1) (1.9) (2.1) (2.4)
7.7 4.6 7.2 2.5 3.6
(1.7) (3.0) (2.1) (2.1) (3.0)
7.5 4.4 7.7 2.0 3.1
(1.8) (2.6) (1.8) (1.6) (2.8)
Progress test self paced Mean (SD)
8.l 4.7 8.0 2.0 2.7
(2.2) (3.6) (2.1) (1.8) (2.6)
Student Support Devices
269
For the students in the starter courses the MANOVA showed no significant differences between the four conditions (Wilks's Lambda = .74; p=.583). For the advanced courses some significant differences between the groups were revealed through MANOVA (Wilks's Lambda = .62; p<0.05). The univariate test results are presented in Table 9. The results in Table 9 indicate that the control group shows more alterations of study plan and that the subjects in the progress test conditions studied significantly longer for their first exam..
Discussion and Conclusions It can be concluded that the subjects have a positive appreciation of the tests. They also confirm the assumed training effect of the tests. The feedback students received later on by mail is judged less positively. This means that this feedback can still be improved, for instance by making it longer and more detailed. The effects on study behaviour are limited. A substantial effect is that 24% of the students indicated to have studied certain parts of the course longer or again as a result of the feedback they received. Students who received progress tests studied longer. Progress tests can be used as an instrument to make students study longer in a self study context. When we compare perceived study time in the control group with that in the experimental groups, no other significant differences can be found. In general there are no significant differences concerning study method or motivation. However, students from the advanced courses showed somewhat less alterations of their study plan when compared to the control group in advanced courses. To recapitulate the research hypotheses: 1. Feedback from progress- and prior-knowledge tests provides relevant information for students. 2. Feedback leads to alterations in the subjects or the sequence of the subjects studied by students. 3. Feedback influences the way students study (study method). 4. Feedback influences learner engagement or perceived study time. 5. Students report test training effects. Of these, the first, the fourth and the fifth hypothesis can be accepted. Although it was not the focus of this study, the results seem to indicate the impact of learner engagement: students who did notreturn the test also showed shorter perceived total study times. It is also notable that students who received negative feedback are more negative about the usefulness of assessment. It appears that mature students are not easily influenced by one or more tests with feedback. Negative feedback does not seem to have dramatically negative consequences for motivation, although there is a tendency for interaction with perceived study time. Taking into account the results presented above and the results of Moerkerke (1995), which demonstrate that there is no substantial effect on final examination results and a significant effect on drop-out for students studying in starter courses, several explanations can be given:
270
1.
.
.
Rob Martens and Filip Dochy
A substantial group of students reported that feedback was not well timed. A period of about two weeks delay of feedback might have been too long for this proportion of students. The learning material of the OU can be characterised has highly supportive. Up to 40% of the material consists of ESD. McGinn and Winne (1994) found that this type of learning material is less likely to produce effects in the case of tests with feedback, when compared to material that involves complex tasks with few cues or instructional support. Besides, this printed material already contains questions with feedback. It is possible that the feedback we presented contained insufficiently detailed information.
The high participation in the research was remarkable: almost 60% participation in one experimental group is very high taking into account the usual study drop out rates which are about 50%. This shows a high use of this type of embedded support, which is in line with the high use of ESD that is generally found (e.g., Martens & Valcke, 1995). The main conclusions and implications of this research are: 1. Tests with delayed feedback are appreciated by students, but do not seem to change dramatically the study process of adult students. 2. Progress tests increase the learning time students invest; this holds true especially for students who study in advanced courses. 3. The period of delay should be adapted to student demands. A standard period of about two weeks is not suitable for all students. 4. Negative feedback seems not to discourage adult students. 5. Feedback should be detailed (cf. Bangert-Drowns, Kulik, Kulik, & Morgan, 1991) and honest. When results are negative this can be reported to students. Negative feedback allows students to correct themselves. 6. Progress tests have more demonstrable effects (on study hours) than prior knowledge tests. Students also judge them more positively. 7. Type of pacing (self or institute paced) makes no difference whatever with respect to students' perception or learning process. The results we presented here are based on comparing means and group results. In the future this research could be complemented by research that is more aimed at selfregulated learning processes in individual students. A model for such a fine-grained analysis of the activities in which single students engage as they learn and the patterns of these activities that lead to successively updated, improved performance, is presented by Butler and Winne (1995). References Alexander, P.A. (1993). The interplay between domain and general strategy knowledge: Exploring the influence of motivation and situation. Texas: A&M University. Bangert-Drowns, R.L., Kulik,C.C., Kulik, J., & Morgan, M. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213-238.
StudentSupportDevices
271
Boekaerts, M., & Simons, P.R.J. (1993). Leren en instructie. Psychologie van de leerling en her leerproces [Learning and instruction: psychology of learners and the learning process]. Assen: Dekker & van de Vegt. Boon, J., Janssen, L, Joosten, G., Liefhebber, J., & Poelmans, P. (1995). Looking back to the future. In M. Valcke & K. Schlusmans (Eds.), Inside out. An introduction to the Dutch Open University. Heerlen: Open University. Butler, D.L., & Winne, P.H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65, 245-281. Crombag, H.F.M. (1979). Onderwijsresearch, 4, 176-183.
ati:
Perhaps not such a good idea after all.
Tjjdschrift voor
De Wolf, H.C. (1994). Distance education. The international encyclopaedia of education. Second edition (pp. 1557-1562). Oxford: Elsevier. Dochy, F.J.R.C. (1992). Assessment of prior knowledge as a determinant for future learning. Utrecht/London: Lemma/Jessica Kingsley. Dochy, F.J.R.C. (1994). Investigating the use of knowledge profiels in a flexible learning environment: analyzing students' prior knowledge states. In S. Vosniadou (Ed.), Psychological and educational foundations of technology-based learning (pp. 235-243). Berlin/New York: Springer Verlag. Dochy F.J.R.C., & Alexander, P.A. (1995). Mapping prior knowledge: a framework for discussion among researchers. European Journal of Psychology of Education, 10, 225-242. Dochy, F., Moerkerke, G., & Alexander, P.A. (1995). The acceptability of prior knowledge state tests andprogress tests as tools for study support. Paper presented at the 6th European Conference for Research on Learning and Instruction, Nijmegen, 1995. Dochy, F.J.R.C., Valcke, M.M.A., & Wagemans, L.J.J.M. (1991). Learning econmics in higher education: An investigation concerning quality and impact of expertise. Higher Education in Europe, 16, 123-136. Fletcher, C. (1991). Assessment feedback counselling and individual exploratory study. Counselling Psychology Quarterly, 4, 103-107.
differences: An
Helgeson, S.L., & Kumar, D.D. (1993). A review of educational technology in science assessment. Journal of Computers in Mathematics and Science Teaching, 12, 227-243. Lockwood, F. (1992). Activities in self-instructional texts. London, Philadelphia: Kogan Page. Lockwood, F. (1995). A cost beneft model to describe the perception and use of activities in self-instructional texts. European Journal of Psychology of Education, 10, 145-152. Lowyck, J., & Simons, P.R.J. (1991). Inleiding themanummer Zelfinstructie: een cognitiefpsychologisch perspectief [Introduction to special issue on self instruction]. Pedagogische Studien, 68 (7), 281-284.
272
Rob Martensand FilipDochy
Marland, P., Patching, W., Putt, I., & Putt, R. (1990). Distance learners' interactions with text while studying. Distance Education, 1, 71-91. Marland, P., Patching, W., Putt, I., & Putt, R. (1992). Thinking while studying: A process tracing study of distance learners. Distance Education, 13, 193-217. Martens, R.L., & Valcke, M.M.A. (1995). Validation of a theory about functions and effects of embedded support devices in distance education learning materials. European Journal of Psychology of Education, 10, 181-196. Martens, R., Valcke, M., Poelmans, P., & Daal, M. (in press). Functions, use and effects of embedded support devices in printed distance learning materials. Learning and Instruction. Martens, R., Valcke, M., & Portier, S. (in press). Interactive learning environments to support independent learning: The impact of discernability of embedded support devices. Computers & Education. McGinn, M.K., & Winne, P.H. (1994). Moving from products to processes: A review of feedback in mathematics education. Paper presented at annual conference of the American Educational Research Association, New Orleans, April 1994. Moerkerke, G. (1995). The effect of prior knowledge state assessment and progress assessment on study results. Unpublished manuscript. Moekerke, G., Dochy, F., & Martens, R. (1995). Effects of self-assessment by means of entry tests and progress tests on study results. Paper presented at the 6th European Conference for Research on Learning and Instruction, Nijmegen, 1995. Rieber, L.P. (1994 April). Animation as real time feedback during a computer-based simulation. Paper presented at the annual conference of the American Educational Research Association, New Orleans. Roossink, H. (1990). Terugkoppelen in het natuurwetenschappelijk onderwijs, een model voor de docent [Feedback in science education: A teacher model]. Enschede: Universiteit Twente. Schuemer, R. (1993). Some psychological aspects of distance education. Hagen: ZIFF. Sewart, D. (Ed.) (1995). One world, many voices. ICDE June 1995. London: Eyre & Spottiswoode. Spires, H.A., Donley, J., & Penrose, A.M. (1990). Prior knowledge activation: Inducing text engagement in reading to learn. Paper presented at the annual meeting of the American Educational Research Association, Boston. Sorensen, K. (1995) Evaluation of interactive televison instruction: assessing attitudes of community college students. DEOSNEWS (Distance education online journal, 5 (9)), [email protected]. Valcke, M.M.A., Dochy, F.J.R.C., & Daal, M.M. (1991). Functions and effects of support in open learning systems. Outlining current empirical findings. Heerlen: Open University, OTIC.
StudentSupportDevices
273
Valcke, M., & Thorpe, M. (1995). Distance education: a particular context for teaching and learning. European Journal of Psychology of Education, 10, 111-121. Wagemans, L.J.J.M., Portier, S.J., & van Buuren, J.A. (1994). Het ontwikkelen en gebruiken van voorkennistoetsen en -profielen in het hoger onderwijs [The development and use of prior knowlegde tests and -profiles in higher education]. Heerlen: OTIC research rapport 62. Wiggins, G.P. (1993). Assessing student performance. Exploring the purpose and limits of testing. San Francisco: Jossey-Bass. The Authors ROB MARTENS is educational researcher at the Centre for Educational Technology of the Open University of Heerlen, the Netherlands. His main research subject concerns electronic support of students in self study.
FILIP D O C H Y is research manager at the Centre for Educational Technology of the Open University of Heerlen, the Netherlands. His main field of research is assessment and evaluation. He is a coordinator of the special interest group on assessment and evaluation of the European Association for Research on Learning and Instruction (EARLI).
Studies in EducationalEvaluation, Vol. 23, No. 3, pp. 257-273, 1997 © 1997 Elsevier Science Ltd Printed in Great Britain. All rights reserved SOI91-491X(97)O0016-3 0191--491X/97 $17.00 + o.oo
A S S E S S M E N T A N D F E E D B A C K AS S T U D E N T S U P P O R T DEVICES Rob Martens and Filip Dochy Open University, Heerlen, The Netherlands
In this article we will address the effects of prior knowledge and progress assessment upon learning behaviours and motivation. This means that not the relation between for example scores on prior knowledge tests and examination results is investigated, but the effects (on the study process) of making prior knowledge or progress tests and reading the accompanying external feedback. Thus, prior knowledge tests and progress tests are methods of student support. Prior knowledge is defined as follows: 'prior knowledge is dynamic in nature, is available before a certain learning task, is structured, can exist in multiple stages (i.e., declarative, procedural, and conditional knowledge), is both explicit and tacit in nature, contains conceptual and metacognitive knowledge components' (Dochy & Alexander, 1995, p.228). What is measured by a progress test bears a similarity to the concept of prior knowledge, but of course in the present case the knowledge is built up in the course of a certain task or study. The context of this research is self study. The Open University (OU) of the Netherlands develops - to foster an independent learning process of its adult students sets of self study learning materials for distance education. De Wolf (1994, p. 1557) defined distance education as follows: "Distance education is described by a variety of labels such as 'correspondence education', 'home study', 'independent study', 'external studies', 'distance teaching' and 'open learning'". We define open learning systems as "flexible learning systems which support a learning process in which there is a delicate balance between a maximal student freedom of choices and the optimal adaptation to characteristics of the learner, in order to guarantee a maximal degree of study success" (Valcke, Dochy, & Daal, 1991, p. 3). These forms of learning have a growing influence on traditional education (e.g., Sewart, 1995; Sorensen, 1995). 257
258
Rob Martens and Filip Dochy
In situations where there is less contact between students and teachers/tutors, the material has to be sufficiently supportive. Therefore an essential part of self study materials consists of embedded support devices (ESD). ESD represent a set of formal and contentrelated add-ons, extensions and elaborations of the study materials, such as pre- and postquestions, schemes, illustrations, tasks, summaries, advance organisers, objectives and study guidelines. ESD aim at scaffolding the learning process. Course developers assume ESD have certain functions and effects, i.e., facilitation of access to the content (e.g., overviews), facilitation of the processing of the content (e.g., by using advance organisers, activities, pre-questions) and facilitation of the assessment of mastery of the content/ objectives, for example by embedding questions and test-items with feedback. "Questions with feedback" is one of the most important supportive tools for students (e.g., Butler & Winne, 1995; Lockwood, 1995). With the emergence of computers, assessment is becoming more and more of an integral part of educational software (e.g., Dochy, 1992). Computers can also be used to supply fast and adequate feedback to students on the basis of printed tests (e.g., prior knowledge tests and progress tests). ESD are designed to evoke certain learning behaviours. Even though developers' self study materials make intensive use of ESD, implicitly accepting the functionality and effectivity of ESD, there is not very much empirical evidence to ground these functions and effects in an open learning context (e.g., Lockwood, 1992; 1995; Lowyck & Simons, 1991; Martens & Valcke, 1995). Researchers have raised questions about the usefulness of some ESD (e.g., Schuemer, 1993; Marland, Patching, Putt & Putt, 1990; 1992). At the Dutch OU a series of research studies was conducted, and the following conclusions were reached with regard to questions and feedback in the printed study materials (Martens, Valcke, & Portier, in press; Martens, Valcke, Poelmans, & Daal, in press): 1. Questions and feedback are used very frequently (about 90% of the students uses this type of embedded support). 2. On a scale from 1 to 5 students indicate an average of 3.6 for the amount of support they get from questions in printed study materials. 3. The use of a combination of ESD has a demonstrably positive effect on learning outcomes. To date we have no insight in the learning effect of questions and feedback in open learning. In the research on learning effects that we did so far we only used a combination of ESD. Questions and feedback were a part of this complex of embedded support and could not be analysed separately. These results, in line with Dochy, Moerkerke, and Alexander (1995), show high student appreciation of prior knowledge tests and progress tests in open learning. A survey of research findings on effects of questions and feedback (e.g., Helgeson & Kumar, 1993; Rieber, 1994; Wiggins, 1993) shows that effects are not always consistent. Sometimes effects of assessment are positive, sometimes there are no effects, and they may even be negative. An important explanation for this might be interaction between student characteristics and assessment. This means that a certain feedback mode is not optimal for all students but that the optimal mode depends on learner characteristics (Boekaerts &
Student Support Devices
259
Simons, 1993; McGinn & Winne, 1994). Another explanation for contradicting research results can be found in the poor ecological validity of research concerning questions in self study materials, according to Lockwood (1995). Butler and Winne (1995) in their survey on feedback in self-regulated learning, criticize the narrow focus on the effects of feedback on achievement, thus neglecting interacting factors in self-regulated learning processes. In this article we will focus on the functions and effects of assessment by means of progress tests and prior knowledge tests. We consider tests with feedback as a form of embedded support, but with one major difference: the tests can be more or less embedded in the material, but the feedback should not be built in. This is to prevent what Kulhavy described as presearch availability. Feedback can sometimes have a detrimental effect on learning processes. Students tend to simply copy the correct answers when feedback is presented in written instruction. In the research design we opted for delayed feedback. Students have to generate their own answers first before they get the correct answers. In a meta-analysis a positive effect was found .for delayed feedback in test acquisition tasks (Bangert-Drowns, Kulik, Kulik & Morgan, 1991). Definition and Types of Feedback The concept of feedback is crucial in educational psychology (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991; Wiggins, 1993): "It is an accepted psychological principle that one of the essential elements needed for effective learning to take place is feedback. Knowledge of results is required to assess progress, correct errors and improve performance" (Fletcher, 1991, p. 103). The number of possible feedback modes is almost infinite. Butler and Winne (1995) make a distinction between internal (self generated) feedback and external feedback. Some examples of dimensions that can be used to characterise external feedback are: immediate or delayed feedback, feedback with or without the correct answer and with or without study guidelines. Feedback was found to be most effective when (1) it was unavailable to students prior to the construction of their own responses; (2) it provided more than verification information (e.g., 'That's right'), that is, it provided subject matter information with the correct answer and possibly further elaboration; (3) it was not preceded by a pretest, and (4) it was associated with a complex task providing few cues or instructional supports (McGinn & Winne, 1994). Tests and feedback have three main functions: 1. Feedback offers information about the study process and the mastery of the learning goals. This information is of particular interest in learning situations were there is no direct contact between learners and teachers. This important role of tests and feedback is illustrated for instance in the Keller-plan or personalised system of instruction (PSI) (Crombag, 1979) which is clearly related to more recent systems of open learning (e.g., de Wolf, 1994). Roossink (1990) describes such provision of information as the cognitive function of feedback. Feedback can confirm students' conceptual understandings or beliefs, it can add information, it can overwrite incorrect elements, tune understandings and finally it can restructure schemata if
260
.
.
Rob Martens and Filip Dochy
students hold false theories (Butler & Winne, 1995). Besides these effects on knowledge and beliefs, feedback can also influence cognitive processing (tactics and strategies). Feedback can be interpreted as a positive reinforcer or as a negative reaction (Boekaerts & Simons, 1993). The impact of feedback as reinforcement is in part determined by the interpretation that the individual student makes (e.g., his or her attribution style) (Alexander, 1993; Boekaerts & Simons, 1993). Prior knowledge assessment or progress assessment can also directly influence the study process by means of activation of prior knowledge (Dochy, 1992; Spires, Donley, & Penrose, 1990). Often also the tests used are similar to the final exam and offer training in solving a certain type of items.
The first two functions mentioned above are functions of feedback, the third is a function of testing. These functions of feedback indicate the possible impact that feedback may have on learning outcomes. Although the time used to fill in tests and to read the feedback is often relatively short when comparing other learning activities, these hypothetical functions and effects suggest an important impact on study outcomes. Thus, feedback may lead to changes in study behaviour like for instance rereading, studying with more effort, studying different subjects. From this point of view feedback primarily influences the metacognitive processes of students. Possible effects of feedback are: 1. Effects on (the sequence) of the subjects studied. Prior knowledge or progress assessment may cause students to study some parts of the study material again, skip topics or study subjects in a different order. This intermediate information about study progress is of crucial importance in modular, open learning situations (Dochy, 1992). 2. Effects on the way students study. One effect of feedback might be that students alter their study method. This is especially to be expected when the feedback contains advice or study guidelines (e.g., Wagemans, Portier & van Buuren, 1994). Positive feedback might encourage students and negative feedback may incite students to change their study method. 3. Effects on the effort, learner engagement or study time. Positive feedback can motivate students positively and stimulate them to persist. Negative feedback can stimulate students but can also discourage them, depending on student's attribution style. Problem statements such as used in prior knowledge and progress tests can enlarge student's motivation (Boekaerts & Simons, 1993). The assumptions we made above have three important implications (see Figure 1): 1. We assume that prior knowledge and progress assessment have a measurable effect on variables concerning study behaviour. 2. It is assumed that these effects are not equal for all students. The effects depend on type of feedback (positive or negative). 3. Tests can have a direct effect on study outcomes by means of activation of prior knowledge or test training.
261
StudentSupportDevices
priorknowledgeor progresstests
studyoutcomes ~
p°sit~;dbnkgati~
Figure 1: A Model for the Effects of Tests and Feedback Research Questions In this research we will focus on the effects of feedback and try to validate the theoretical functions and effects that are ascribed to feedback. Concerning the functions and effects there are five research questions: 1. Does feedback from progress- and prior-knowledge tests provide relevant information for students? 2. Does feedback lead to alterations in the subjects or the sequence of the subjects studied by students? 3. Does feedback influence the way students study (study method)? 4. Does feedback influence learner engagement or study time? 5. Do students report test training effects? The direct effects of testing on learning outcomes are not fully measured in this research, although we will investigate if students report them. We will not investigate the activating effect on prior knowledge as a result of the testing procedure. The effects on study results and study speed is reported on in a related study. Moerkerke, Dochy, and Martens (1995) and Moerkerke (1995) found that there is no effect of prior knowledge or progress assessment on study results. However, a significant effect is found on study speed/attending examinations: after seven months of study significantly more students in courses for beginners attended the exam. This effect was not found for students studying more advanced courses. In the present study we try to explain these results by taking into account variables related to the study process and students' motivation. This study is an attempt to understand the processes that occur in students that may have caused the effects we mentioned above. This research is based on the same four courses as used by Moerkerke, Dochy, and Martens (1995) (Information science, Marketing, Business, Management). We assume that all the above research questions can be answered positively and thus can be interpreted as expectancies. A distinction will be made between students who receive "positive" feedback and students who receive "negative" feedback. It is expected that the first group will be positively reinforced by feedback and will not change their study method, while the second group will change study method since the feedback signals a problem to them.
262
Rob Martens and Filip Dochy
Research Method, Materials and Subjects Two questionnaires were distributed to students in the experimental group and in the control group: one at the beginning of the study (questions about student's opinion about assessment) and one after completing the final examination (the same questions and also, depending on the conditions, questions about the tests, the feedback, motivation, study process, etc.). The prior knowledge tests and the progress tests usually consist of 40 multiple choice items each. Every question is classified along two categories: behaviour (knowledge, insight, application questions) and knowledge (facts, concepts relations, structures, methods). Feedback consists of indications of the percentage of correct answers per category and an indication of the place in the course where the main subject of a question is treated. In the feedback a norm of 70% is presented to the students, meaning that when more than 70% of the answers is correct this is sufficient while topics with less than 70% correct answers should be studied more profoundly. Less than 70% is interpreted as negative feedback. Note that the distinction between positive and negative is not controlled experimentally but depends only on students' performance. Experimental variation in positive or negative feedback, when not based on real scores, is considered nonethical with students studying nonexperimental, regular courses. Besides the correct answers the feedback also provided short explanations and separate scores on the (behaviour and knowledge) dimensions we distinguished, together with some explanation and guidelines. This distinction between dimensions is a knowledge profile. The behavioural and knowledge dimensions were proven adequate in determining prior knowledge profiles in higher education (Dochy, 1994; Dochy, Valcke, & Wagemans, 1991). So the feedback not only provided content information but also advice. Feedback was sent back within two weeks after receipt of the completed test forms. Table 1: Test Information Content domain Prior k. marketing Progress 1 marketing Progress 2 marketing Progress 3 marketing Prior k. information scoemce Progress 1 information science Progress 2 information science Progress 3 information science Prior k. business Progress 1 business Progress 2 business Progress 3 business Prior k. management
Number of Items 40 40 40 40 50 40 3O 30 40 40 40 40 31
n 55 106 74 53 53 95 6O 37 26 44 40 30 58
Cronbach's alpha .60 .63 .69 .72 .73 .72 .84 .81 .48 .69 .58 .80 .75
Student Support Devices
263
In Table 1 basic information about the prior knowledge and progress tests is presented. Our first aim was to guarantee content validity (e.g., representation of all knowledge levels) in the tests. Since the test length had to be restricted our main aim was not to make alpha as high as possible. The subjects were regular students, taking a regular course. They could either be in the control group or volunteer to fill in the prior knowledge or progress tests. There were four research groups: • Control group: These students do not fill in prior knowledge or prior knowledge tests. They only complete questionnaires concerning motivation, study planning, study method and study satisfaction. • Prior knowledge test: Students in this group receive at the beginning of their studies, a prior knowledge test and some questionnaires. After filling in the prior knowledge test students receive feedback with comment on their results. • Self paced progress tests: Students in this group receive at three points during their studies, a progress test and some questionnaires. When a student reaches a certain point in the study material he or she can apply for a progress test (self paced). After filling in the progress test they receive feedback with comment on their results. • Progress tests paced by the institute: Students in this group receive at three fixed points in time (with a fixed intermediate period) in their study a progress test and some questionnaires. After filling in the progress test they receive feedback with comments on their results. Two courses are "starter" courses, i.e., these courses are the first in a sequence that eventually leads to a diploma; the two other courses are advanced courses. Students w h o study these latter courses can be assumed to have more experience with self study in higher education. Starters courses are information science and marketing, and the two advanced courses are business and management. After filling in the questionnaires and/or the prior knowledge test and/or the progress tests students take a regular final exam. Only after this final exam they received the final questionnaire. From the total research sample of 1482 students who registered for certain courses, 404 students were in the control group, 360 in the prior knowledge group, 358 in the progress group paced by the institute and 360 in the self paced progress group. About 52% of the subjects in our sample had not taken an exam. The average percentage of students not taking the exam for the entire university is 53% after one year (Boon et al., 1995). After one year this percentage hardly increases. This means that we can assume that in the present data set almost all subjects that can be expected to take an exam are present. Differences between groups were analysed by means of univariate tests such as ttests and univariate analysis of variance. When looking at effects of correlated dependent variables jointly, MANOVA was used. Basic assumptions underlying these analyses such as normal distribution of variables were checked for. In cases of doubt non-parametric measures were used.
264
Rob Martens and Filip Dochy
Results The response rates were based upon returning the (first) questionnaire. Students who were registered for a series of courses were not included in this analysis because they might not start with the course that our tests addressed. In the experimental groups returning the questionnaire often was accompanied by returning the first test (prior knowledge test). For the complete group of subjects the response percentage was 47%. The response rate in the no-treatment group was 54%, in the prior knowledge test group 57%. This difference is not significant. The percentages for the progress groups were different because students had to study 25% of the course before sending back the first progress test and questionnaire. Taking into account the regular drop out, response rates in the progress test groups can be expected to be lower: for the group paced by the institute this was 42% and for the student paced condition 32%. These percentages are based upon a seven-month enrolment period. First students' perceptions of the tests and the feedback were explored. The following results are therefore limited to students in the experimental conditions. They answered several questions after doing their exam. The questions were Likert-type questions with 5-point scales, ranging from "completely disagree" (1) to "completely agree" (5). Scores 1 to 3 on a statement were interpreted as a negative response to the statement, scores 4-5 were interpreted as a positive (agreement) response to that statement. The three experimental groups differed significantly (MANOVA, Wilks's Lambda = .69, p<0.0001). Analysis showed that for all variables in Table 2 the progress test group differed significantly from the prior knowledge test groups.
Table 2: Perception of Tests with Feedback as Reported by Students (N=195) Variable/group Finds that doing the test(s) was useful Agrees that the feedback provides useful information about own knowledge level Finds that the test was a useful exercise for the information Find the feedback was understandable Finds that the feedback provides substantial information Agrees that the feedback was given at the fight time
Prior knowledge
Progress self
Progress OU
44.9% 51.4%
82.3% 63.1%
81.3% 60.2%
53.6% 51.5% 44.8% 68.2%
85.3% 63.6% 60.6% 58.8%
87.8% 74.7% 63.0% 54.9%
Generally the feedback is not judged as positively as the tests. Apparently not all students find the feedback that was provided sufficiently informative or clear. The results show that feedback has to be detailed and informative. The feedback that was provided in this experiment can be improved. High percentages of students report training effects with regard to the final exam (statement 3). The progress tests with feedback are judged more positively than the prior knowledge tests.
Student Support Devices
265
Table 3: Perception of Tests with Feedback, Split Up for Positive and Negative Feedback (N=185) Variable
Positive feedback Mean and SD
Test useful Information about knowledge level Useful exercise Feedback understandable Substantial information In time
4.5 3.7 4.5 3.9 3.8 3.4
(.81) (.91) (.89) (.87) ( 1.0) (1.4)
Negative feedback Mean and SD 3.9 3.7 3.9 3.6 3.5 3.9
(1.2) (1.1) (1.1) (1.0) ( 1.0) (1.2)
t-value t(167.08)=4.34" t (186)=-.33 t (187)=-3.17 t (139.59)=-2.10 t ( 187)=- 1.44 t (99,22)=2.29
* = p < .05
Table 3 is based on the same 5-point scales as in Table 2. From this table it can be concluded that perceptions of tests differ for students who receive positive or negative feedback (MANOVA, Wilks's Lambda = .86, p<0.0001). Students with insufficient results are significantly more negative about the usefulness of the tests. In the next sections the results in terms of study behaviour and motivation will be analysed. To do so the results of questionnaires taken from students in the experimental conditions will be compared to students receiving no experimental treatment. In this analysis we will compare the results on eleven multiple choice questions, with 10 point scales and two open questions concerning perceived study time in hours. We summarise the 13 variables in Figure 2. Use of study plan : Alteration of study plan : Change of order: Change of study speed: Study hours per week: Study hours for first exam: Difficulties with planning: Perception of learning material: Insight in final goals: Results unexpected: Positive about study: Change of study method: Decrease of motivation: Figure 2:
I used a study plan for this course I changed my study plan during the course I changed the order of the subjects I studied I changed the sPeed of my studies Average study hours per week (open question) Total amount of hours before the first exam (open question) I found it difficult to plan my studies Well structured I knew what I had to do to pass my exam The results of my examination were different from what I expected I feel positive about studying the course While studying the course I altered my study method While studying my motivation has decreased
Variables about Study Experiences in Questionnaire After Exam
In Table 4 the results are presented. The rows represent one question each. The variables in column one are presented in Figure 2. The second column presents the mean scores for the entire population in this research. The third column shows the results of the subjects who scored on average less than 70% on the test(s). This is interpreted as subjects receiving negative feedback. The fourth column represents the subjects with positive feedback, the fifth contains subjects in the experimental groups who did not respond to
266
RobMartensand FilipDochy
the tests and the last column contains the scores of the subjects who only received the questionnaires and had no further treatment. Table 4:
Effects of Assessment (n=333)
Variable
Complete sample Mean (SD)
Use ofstudyplan 6.2 (3.1) Alteration of study plan 5.1 (3.3) Change of order 2.2 (2.3) Change of study speed 4.5 (3.2) Study hoursperweek 12.7 (8.3) Study hours for first exam 136.8 (75.5) Difficulties with planning 3.0 (2.5) Leaming material: well 8.2 (1.7) structured Insight in final goals 7.8 (1.9) Results unexpected 4.3 (3.0) Positive aboutstudy 7.9 (1.9) Change of study method 2.7 (2.4) Decrease of motivation 3.0 (2.5)
Test: insufficient (negative feedback) Mean (SD)
Test: sufficient (positive feedback) Mean (SD)
Mean (SD) Mean (SD)
5.9 (3.2) 4.9 (3.4) 2.2 (2.3) 4.5 (3.2) 11.8 (8.8) 135.9 (74.2) 3.1 (2.6) 8.3 (1.6)
6.3 (3.2) 5.1 (3.4) 1.9 (1.9) 4.5 (3.3) 13.6 (7.7) 161.0 (64.5) 2.5 (1.9) 8.4 (1.5)
6.1 (3.2) 6.4 (3.1) 5.2 (3.2) 5.2 (3.2) 2.7 (2.7) 2.3 (3.2) 5.0 (3.3) 4.2 (3.2) 1 3 . 7 (8.5) 12.8 (8.1) 127.6 (88.9) 126.5 (75.6) 3.6 (3.2) 3.1 (2.5) 8.1 (1.7) 7.9 (2.0)
7.8 4.3 7.8 2.7 3.2
(2.0) (3.0) (1.9) (2.3) (2.6)
8.1 3.3 8.2 2.6 3.0
(1.6) (2.5) (1.7) (2.4) (2.6)
No test send back
7.8 5.2 7.7 3.2 3.0
Controlgroup
(1.6) (3.0) (2.1) (2.6) (2.6)
7.7 4.4 7.9 2.6 2.9
(2.1) (3.2) (1.9) (2.3) (2.3)
The effects of receiving tests and feedback, and the effects of the type of feedback (negative or positive) were examined jointly by means of a MANOVA-model. No effect was found (Wilks's Lambda = .84; p=.248). Nevertheless, the large differences in study hours are remarkable. Inspection of the univariate F-tests showed one significant effect: study hours per week. The amount of study hours in the experimental groups who had positive feedback was higher than that of the other groups (F(3;261)= 2.8; p<.05). Instead of concentrating on the effects of type of feedback, the influence of the conditions can be analysed. We could not examine both effects together since the analysis would then contain more than ten groups, leading to empty cells. We repeated the MANOVA with the four conditions as grouping variable. The multivariate tests for significance showed a significant effect (Hotteling's Trace .224; p<.05; Wilks's Lambda = .80; p=.05). This means that there are no strongly significant differences between students receiving prior knowledge and progress test and students who received no such tests with feedback. Univariate analysis showed there was one effect. Students in the progress tests groups study significantly longer before they take their first exam (see Table 5). Table 5: Study Hours as a Dependent Variable of Condition (n=318)
Study hours
Control group
Prior knowledge test
Progress test paced Progress test self by the institute paced
Mean 126.5
Mean 113.1
Mean 156.1
SD (75.6)
SD (70.9)
(SD) (67.9)
Mean 156.0
(SD) (78.9)
Student SupportDevices
267
I n s t e a d o f c o m p a r i n g groups and l o o k i n g for significant differences w e can also l o o k at students' o w n p e r c e p t i o n s o f the effects o f prior k n o w l e d g e or progress assessment. The f o l l o w i n g results are thus limited to students in the e x p e r i m e n t a l conditions. T h e y a n s w e r e d s o m e questions after taking their exam. The questions were m u l t i p l e c h o i c e w i t h 5 p o i n t scales, ranging from completely disagree (1) to completely agree (5). Scores 1 - 3 on a statement are interpreted as a n e g a t i v e response to the statement, scores 4-5 are interpreted as a p o s i t i v e (agreement) response to that statement. Table 6: Effects of Tests with Feedback as Reported by Students in Experimental Groups (n=200) 5.5% report to have changed study method as a result of feedback received * 23.5 % have studied parts of the course longer or again as a result of the feedback * 4.5 % of the students report to have skipped some parts of the course as a result of the tests with feedback * 15 % report to have studied longer as a result of the feedback.* • = proportion differs significantly from .50 (p<0.01, binomial test) Again, w e can split these results into positive and n e g a t i v e feedback. N o n e o f these f o u r questions s h o w e d a significant difference b e t w e e n students receiving positive f e e d b a c k and students w h o got n e g a t i v e feedback. T h e f o l l o w i n g analysis shows students' opinions about functionality o f assessment. O n l y the data o f students in the e x p e r i m e n t a l groups are used. T h e results are d i s p l a y e d in Table 7. Students' o p i n i o n at the b e g i n n i n g o f their studies is reported. The same questions were p o s e d at the end of the course, after c o m p l e t i n g the final exam. The questions were on 5-points scales, ranging form 1 (completely disagree) to 5 ( c o m p l e t e l y agree). T h e opinion significantly c h a n g e d for one variable. This difference is i n d i c a t e d in Table 7.
Table 7: Students' Opinions about Assessment (n=177) Variable
Average score
Wants prior knowledge tests Function: To assess the required prior knowledge Function: To assess to what extent a student already masters the course material Function: To know which parts are to be studied at a deep level and which at a surface level Function: To support a study plan Function: To determine a learing plan Function: To support the choice for a certain course Function: To prepare faster for the exam Function: To better organise the study Function: To be able to address study counsellors more adequately Function: To be able to motivate demands for extra study support Progress tests are a useful tool for OU students to gain insight in their study progress
3.3 1.4 1.4; after exam: 1.3" 1.6
* T-value (179) = 3.76; p<0.001
1.6 1.6 1.6 1.6 1.7 1.8 1.8 4.3
268
Rob Martens and Filip Dochy
From Table 7 it can be concluded that students find prior knowledge and progress tests useful. About the functions of prior knowledge tests there is less agreement. Students' opinions hardly change during the course of their studies and are hardly influenced by the tests with feedback, or by the fact that some respondents did their final exam and others not. Comparisons between students in the starter courses and students in the advanced courses were carded out. It was found that these two groups differ on the set of questions we used earlier (see Table 4; Wilks's Lambda = .77, p<0.001). Table 8 presents these differences. Table 8: Effects of Assessment, Split Up for Starters and Advanced Students (n=265) Variable
Starters Mean (SD)
Use of study plan Alteration of study plan Change of order Change of study speed Study hours per week Study hours for first exam Difficulties with planning Learning material: well structured Insight in final goals Results unexpected Positive about study Change of study method Decrease of motivation
6.1 5.2 2.4 4.7 12.6 158.6 3.2 8.4 8.0 4.0 8.1 3.0 2.9
(3.2) (3.4) (2.5) (3.2) (8.8) (70.0) (2.7) (1.6) (1.8) (2.9) (1.8) (2.6) (2.4)
Advanced Mean (SD) 6.3 4.9 2.0 4.1 12.7 107.9 2.8 8.0 7.6 4.5 7.6 2.3 3.2
(3.1) (3.2) (2.0)* (3.2)* (7.8) (73.1)* (2.3) (1.9)* (2.0) (3.1) (2.0)* (2.0)* (2.6)
We repeated the MANOVA in which the effects of the condition are analysed, split up for starters and students taking the advanced courses. Table 9: Effects of Assessment for Advanced Courses Only (n=265) Variable
Use of study plan Alteration of study plan* Change of order Change of study speed Study hours per week* Study hours for first exam* Difficulties with planning Learning material: well structured Insight in final goals Results unexpected Positive about study Change of study method Decrease of motivation *--p <.05
Complete sample
Control group Mean (SD)
Prior knowledge test Mean (SD)
Progress test paced by the institute Mean /SD)
Mean (SD) 6.3 4.9 2.0 4.1 12.7 107.8 2.8 8.0 7.6 4.4 7.6 2.3 3.2
(3.1) (3.2) (2.0) (3.2) (7.8) (73.1) (2.3) (1.9)
6.5 5,6 2,3 4.5 12.2 100.2 3.2 7.7
(3,1) (3.2) (2.3) (3.4) (7.0) (60.4) (2,6) (2.1)
6.0 4.0 1.9 4.2 11.6 84.8 2.8 7.8
(3.2) (Z8) 0.9) (3.1) (9.0) (60.4) (2.1) (1.0)
6.0 (3.4) 6.5 (3.1) 4.1 (3.4) 5.0 (3.6) 1.7 (1.7) 1.8 (1.3) 3.1 (2.7) 3.9 (3.3) 11.3 (4.0) 17.3 (8.7) 165,5 (100.3) 126.3 (80.7) 2.1 (1.9) 2.2 (1.8) 8.8 (1.0) 8.5 (1.7)
(2.0) (3.1) (2.0) (2.0) (2.6)
7.3 4.3 7.6 2.4 3.2
(2.2) (3.1) (1.9) (2.1) (2.4)
7.7 4.6 7.2 2.5 3.6
(1.7) (3.0) (2.1) (2.1) (3.0)
7.5 4.4 7.7 2.0 3.1
(1.8) (2.6) (1.8) (1.6) (2.8)
Progress test self paced Mean (SD)
8.l 4.7 8.0 2.0 2.7
(2.2) (3.6) (2.1) (1.8) (2.6)
Student Support Devices
269
For the students in the starter courses the MANOVA showed no significant differences between the four conditions (Wilks's Lambda = .74; p=.583). For the advanced courses some significant differences between the groups were revealed through MANOVA (Wilks's Lambda = .62; p<0.05). The univariate test results are presented in Table 9. The results in Table 9 indicate that the control group shows more alterations of study plan and that the subjects in the progress test conditions studied significantly longer for their first exam..
Discussion and Conclusions It can be concluded that the subjects have a positive appreciation of the tests. They also confirm the assumed training effect of the tests. The feedback students received later on by mail is judged less positively. This means that this feedback can still be improved, for instance by making it longer and more detailed. The effects on study behaviour are limited. A substantial effect is that 24% of the students indicated to have studied certain parts of the course longer or again as a result of the feedback they received. Students who received progress tests studied longer. Progress tests can be used as an instrument to make students study longer in a self study context. When we compare perceived study time in the control group with that in the experimental groups, no other significant differences can be found. In general there are no significant differences concerning study method or motivation. However, students from the advanced courses showed somewhat less alterations of their study plan when compared to the control group in advanced courses. To recapitulate the research hypotheses: 1. Feedback from progress- and prior-knowledge tests provides relevant information for students. 2. Feedback leads to alterations in the subjects or the sequence of the subjects studied by students. 3. Feedback influences the way students study (study method). 4. Feedback influences learner engagement or perceived study time. 5. Students report test training effects. Of these, the first, the fourth and the fifth hypothesis can be accepted. Although it was not the focus of this study, the results seem to indicate the impact of learner engagement: students who did notreturn the test also showed shorter perceived total study times. It is also notable that students who received negative feedback are more negative about the usefulness of assessment. It appears that mature students are not easily influenced by one or more tests with feedback. Negative feedback does not seem to have dramatically negative consequences for motivation, although there is a tendency for interaction with perceived study time. Taking into account the results presented above and the results of Moerkerke (1995), which demonstrate that there is no substantial effect on final examination results and a significant effect on drop-out for students studying in starter courses, several explanations can be given:
270
1.
.
.
Rob Martens and Filip Dochy
A substantial group of students reported that feedback was not well timed. A period of about two weeks delay of feedback might have been too long for this proportion of students. The learning material of the OU can be characterised has highly supportive. Up to 40% of the material consists of ESD. McGinn and Winne (1994) found that this type of learning material is less likely to produce effects in the case of tests with feedback, when compared to material that involves complex tasks with few cues or instructional support. Besides, this printed material already contains questions with feedback. It is possible that the feedback we presented contained insufficiently detailed information.
The high participation in the research was remarkable: almost 60% participation in one experimental group is very high taking into account the usual study drop out rates which are about 50%. This shows a high use of this type of embedded support, which is in line with the high use of ESD that is generally found (e.g., Martens & Valcke, 1995). The main conclusions and implications of this research are: 1. Tests with delayed feedback are appreciated by students, but do not seem to change dramatically the study process of adult students. 2. Progress tests increase the learning time students invest; this holds true especially for students who study in advanced courses. 3. The period of delay should be adapted to student demands. A standard period of about two weeks is not suitable for all students. 4. Negative feedback seems not to discourage adult students. 5. Feedback should be detailed (cf. Bangert-Drowns, Kulik, Kulik, & Morgan, 1991) and honest. When results are negative this can be reported to students. Negative feedback allows students to correct themselves. 6. Progress tests have more demonstrable effects (on study hours) than prior knowledge tests. Students also judge them more positively. 7. Type of pacing (self or institute paced) makes no difference whatever with respect to students' perception or learning process. The results we presented here are based on comparing means and group results. In the future this research could be complemented by research that is more aimed at selfregulated learning processes in individual students. A model for such a fine-grained analysis of the activities in which single students engage as they learn and the patterns of these activities that lead to successively updated, improved performance, is presented by Butler and Winne (1995). References Alexander, P.A. (1993). The interplay between domain and general strategy knowledge: Exploring the influence of motivation and situation. Texas: A&M University. Bangert-Drowns, R.L., Kulik,C.C., Kulik, J., & Morgan, M. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213-238.
StudentSupportDevices
271
Boekaerts, M., & Simons, P.R.J. (1993). Leren en instructie. Psychologie van de leerling en her leerproces [Learning and instruction: psychology of learners and the learning process]. Assen: Dekker & van de Vegt. Boon, J., Janssen, L, Joosten, G., Liefhebber, J., & Poelmans, P. (1995). Looking back to the future. In M. Valcke & K. Schlusmans (Eds.), Inside out. An introduction to the Dutch Open University. Heerlen: Open University. Butler, D.L., & Winne, P.H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65, 245-281. Crombag, H.F.M. (1979). Onderwijsresearch, 4, 176-183.
ati:
Perhaps not such a good idea after all.
Tjjdschrift voor
De Wolf, H.C. (1994). Distance education. The international encyclopaedia of education. Second edition (pp. 1557-1562). Oxford: Elsevier. Dochy, F.J.R.C. (1992). Assessment of prior knowledge as a determinant for future learning. Utrecht/London: Lemma/Jessica Kingsley. Dochy, F.J.R.C. (1994). Investigating the use of knowledge profiels in a flexible learning environment: analyzing students' prior knowledge states. In S. Vosniadou (Ed.), Psychological and educational foundations of technology-based learning (pp. 235-243). Berlin/New York: Springer Verlag. Dochy F.J.R.C., & Alexander, P.A. (1995). Mapping prior knowledge: a framework for discussion among researchers. European Journal of Psychology of Education, 10, 225-242. Dochy, F., Moerkerke, G., & Alexander, P.A. (1995). The acceptability of prior knowledge state tests andprogress tests as tools for study support. Paper presented at the 6th European Conference for Research on Learning and Instruction, Nijmegen, 1995. Dochy, F.J.R.C., Valcke, M.M.A., & Wagemans, L.J.J.M. (1991). Learning econmics in higher education: An investigation concerning quality and impact of expertise. Higher Education in Europe, 16, 123-136. Fletcher, C. (1991). Assessment feedback counselling and individual exploratory study. Counselling Psychology Quarterly, 4, 103-107.
differences: An
Helgeson, S.L., & Kumar, D.D. (1993). A review of educational technology in science assessment. Journal of Computers in Mathematics and Science Teaching, 12, 227-243. Lockwood, F. (1992). Activities in self-instructional texts. London, Philadelphia: Kogan Page. Lockwood, F. (1995). A cost beneft model to describe the perception and use of activities in self-instructional texts. European Journal of Psychology of Education, 10, 145-152. Lowyck, J., & Simons, P.R.J. (1991). Inleiding themanummer Zelfinstructie: een cognitiefpsychologisch perspectief [Introduction to special issue on self instruction]. Pedagogische Studien, 68 (7), 281-284.
272
Rob Martensand FilipDochy
Marland, P., Patching, W., Putt, I., & Putt, R. (1990). Distance learners' interactions with text while studying. Distance Education, 1, 71-91. Marland, P., Patching, W., Putt, I., & Putt, R. (1992). Thinking while studying: A process tracing study of distance learners. Distance Education, 13, 193-217. Martens, R.L., & Valcke, M.M.A. (1995). Validation of a theory about functions and effects of embedded support devices in distance education learning materials. European Journal of Psychology of Education, 10, 181-196. Martens, R., Valcke, M., Poelmans, P., & Daal, M. (in press). Functions, use and effects of embedded support devices in printed distance learning materials. Learning and Instruction. Martens, R., Valcke, M., & Portier, S. (in press). Interactive learning environments to support independent learning: The impact of discernability of embedded support devices. Computers & Education. McGinn, M.K., & Winne, P.H. (1994). Moving from products to processes: A review of feedback in mathematics education. Paper presented at annual conference of the American Educational Research Association, New Orleans, April 1994. Moerkerke, G. (1995). The effect of prior knowledge state assessment and progress assessment on study results. Unpublished manuscript. Moekerke, G., Dochy, F., & Martens, R. (1995). Effects of self-assessment by means of entry tests and progress tests on study results. Paper presented at the 6th European Conference for Research on Learning and Instruction, Nijmegen, 1995. Rieber, L.P. (1994 April). Animation as real time feedback during a computer-based simulation. Paper presented at the annual conference of the American Educational Research Association, New Orleans. Roossink, H. (1990). Terugkoppelen in het natuurwetenschappelijk onderwijs, een model voor de docent [Feedback in science education: A teacher model]. Enschede: Universiteit Twente. Schuemer, R. (1993). Some psychological aspects of distance education. Hagen: ZIFF. Sewart, D. (Ed.) (1995). One world, many voices. ICDE June 1995. London: Eyre & Spottiswoode. Spires, H.A., Donley, J., & Penrose, A.M. (1990). Prior knowledge activation: Inducing text engagement in reading to learn. Paper presented at the annual meeting of the American Educational Research Association, Boston. Sorensen, K. (1995) Evaluation of interactive televison instruction: assessing attitudes of community college students. DEOSNEWS (Distance education online journal, 5 (9)), [email protected]. Valcke, M.M.A., Dochy, F.J.R.C., & Daal, M.M. (1991). Functions and effects of support in open learning systems. Outlining current empirical findings. Heerlen: Open University, OTIC.
StudentSupportDevices
273
Valcke, M., & Thorpe, M. (1995). Distance education: a particular context for teaching and learning. European Journal of Psychology of Education, 10, 111-121. Wagemans, L.J.J.M., Portier, S.J., & van Buuren, J.A. (1994). Het ontwikkelen en gebruiken van voorkennistoetsen en -profielen in het hoger onderwijs [The development and use of prior knowlegde tests and -profiles in higher education]. Heerlen: OTIC research rapport 62. Wiggins, G.P. (1993). Assessing student performance. Exploring the purpose and limits of testing. San Francisco: Jossey-Bass. The Authors ROB MARTENS is educational researcher at the Centre for Educational Technology of the Open University of Heerlen, the Netherlands. His main research subject concerns electronic support of students in self study.
FILIP D O C H Y is research manager at the Centre for Educational Technology of the Open University of Heerlen, the Netherlands. His main field of research is assessment and evaluation. He is a coordinator of the special interest group on assessment and evaluation of the European Association for Research on Learning and Instruction (EARLI).