- Email: [email protected]
Computer-based practical work at a distance: A case study
0360- 1315/94 $6.00 + 0.00 Copyright Q 1994 Pergamon Press Ltd
ComparrersE&c. Vol. 22, No. l/2, pp.27-37, 1994 Printed in Great Britain. All rights reserved
COMPUTER-BASED
PRACTICAL WORK AT A DISTANCE: A CASE STUDY ANN JONES and MARIAN F%TRE
Institute
of Educational
Technology,
The Open University, Walton Hall, Milton Keynes MK7 6AA, England
Abstract-How do students engage with distance teaching materials for carrying out practical computing work? How can we design teaching materials to make the situation better? Questions about students’ use of instructional materials are particularly important in the context of distance learning, when students have little or no teacher support. This paper reports the results of a case study of home computing use in the Open University course: ‘Computers and Learning’: which is typical of many tXJ courses in its use of different media: texts, video, audio-tapes and of course the computer practical work. We describe how students go about learning to use the computer for their work; the kinds of problems they have and how they overcome them. Nearly all the students successfully completed their practical work: and did so within the allocated time. Nevertheless the process of carrying out the practical work, was not without its problems, and the students’ reports about their practical work indicate more general issues in instructional design. A major issue is the design of instructional materials that are geared to students with different prior knowledge and that can be readily used for reference ufer the student’s initial exposure to it. The student needs to be able to structure the material in such a way that it can be reviewed, restructured and rearranged. Students are active in their learning and instructional materials need to facilitate this by providing an environment designed for exploration, but which supports learners and helps them when they make mistakes.
INTRODUCTION
Personal computers are being used increasingly in distance education, and they have the potential to change the nature of that education radically. For the student learning at a distance or using self-instruction material, computers can be powerful study tools, whether providing general ‘clerical’ support, e.g. word processing facilities, spreadsheets, databases, or contributing to the subject area, e.g. via simulations in Physics, calculation and statistical packages in Mathematics, programming environments in Computer Science. Open University distance learning courses have used computers as tools in all of the above forms. Questions about students’ use of instructional materials are particularly important in the contexts of distance learning and independent study, when students are isolated from peers and teachers. Practical issues about self-instruction materials are amplified when the work is computer based and students must learn about using the computer as a tool as well as about the information it contains. Where do students look for information? Do they read and study in the ways we expect? What kinds of problems do they encounter and how do they cope with them? What are the implications if they use materials in unexpected ways? How do students cope with the unexpected? Imagine that a student studying independently receives a study pack that includes a screwdriver and instructions to use this tool to open a box of materials. The course designers might assume that the student will use the screwdriver to remove the screws fixing the lid on the box; what are the implications when the student uses the screwdriver as a lever to prize the lid off, or as a poker to pierce the box? Will unscrewers, leverers, and piercers all achieve an appropriate understanding? This paper examines the assumption that students use instructional materials effectively, appropriately, and predictably. It reports preliminary results of a case study of home computing use in the Open University course EH232: Computers and Learning. The study used student questionnaires and journals to identify practical issues for the design of computer-oriented instructional materials for independent study. This paper first sets the case study in the context of the problem of computer access in distance education and in the context of related research on independent study of different sorts, Then, having described the set-up of the study, it enumerates as general phenomena the sorts of obstacles, both conceptual and practical.
28
ANN JONESand MARIANPETRE
BACKGROUND:
The Home
Computing
ASSESSING
ACCESS
UNDER
THE
HOME
COMPUTING
POLICY
Policy
Until 1988, the main method of access to computers for Open University students was through terminals at regional study centres which were connected to one of the University’s mainframe computers. By the mid-1980s this kind of access was becoming increasingly problematic for the following reasons: (1) The mainframe system was unable to cope with increasing student demand. (2) More students were buying microcomputers for use on their courses and wanted to buy one that would be suitable for whichever computer-based course they followed. (3) It had been acknowledged that using a computer at home was far preferable to the problems of using one at a study centre[l]. A ‘Home Computing Policy’ was introduced in 1988 which specified a minimum computer system: an MS-DOS machine with 510 Kbytes of memory, a floppy disk drive, another storage device, and a mouse. The specification was modest, mainly to minimize the cost to students. By 1992, 11 courses required students to have access to a personal computer with this specification; this meant that about 17,000 Open University students were studying courses requiring the use of a computer at home. Evaluation
of the Policy
With so many students expected to obtain, set up and use computers, without on-hand help or supervision, there was great potential for things to go wrong. It was imperative, therefore, to monitor the effects of the Policy. An evaluation was undertaken through large-scale student surveys, student journals, and interviews with staff and students. The overall success of the Policy was quickly established, and information was obtained on issues such as the nature and quality of access; the extent to which it differed for different groups; the extent to which students sought help and what routes they took; and so on. The Home Computing Policy largely solved the problem of access to the computer itself and so provided a common medium for learning. The case study However, having a computer at hand is only the outer layer of access. Students also need access to the inside of the ‘box’. For the computer to be an effective learning tool, students need to be able to use it to run the necessary applications. The large-scale, general evaluation of the Home Computing Policy could not properly address this level of access. There is some evidence[2] that learning to use a computer with the aid of self-instructional material can be problematic, and so the case study reported here was undertaken to investigate this process, to ascertain the extent to which students were experiencing difficulties, and to discover something about the problems that occurred, under what conditions they occurred, and how the students coped with them. The case study focussed on two important issues in the design of instructional materials: the selection and structuring of key information, and students’ actual use of resources, whether provided as part of the course or available as part of the student’s own environment. A number of questions were prominent: l
l l l
What are the implications for the sorts of information structures required by and being formed by students? How do people actually use the resources, e.g. tutorials, documents, tapes, that are provided? When does the time it takes to become minimally competent with a system become prohibitive? What is the impact of a student’s previous experience?
With no teacher present to interpret the information and help the student incorporate it into an appropriate framework and usable form, the instructional materials are the keys (or the barriers) to access. Related
research
Many of the same issues arise in designing self-instructional materials for practical computing as in producing distance education teaching materials: in both cases the learner is almost completely
Practical work
29
reliant on the packaged teaching material. Research findings from two related areas are relevant to teaching computer related activities at a distance: first, studies about learning to use applications such as word processors [3-51, and second, research on learning to program using distance learning materials[68]. Mack et a1.[9] studied learners using word processors. The learners in these studies were particularly active in their learning: they did not wait for instructions, nor did they read the manuals properly but jumped ahead and tried things out for themselves; they used their prior knowledge (which often led to misunderstandings); and they had great difficulty in recovering from errors. Another study, of learner programmers using highly-structured distance learning materials suggested that the students operated at too low a level: they were able to follow the steps of the instruction but lacked an overview or a cognizance of the goal towards which they were working[8]. Such an approach to learning is similar to what Marton and Saljo [lo] refer to as surface level processing: they comment that “students adopt and approach determined by their expectations of what is required of them” (p. 125). Lewis and Mack[ll] observe that complex learning such as self-instruction in practical computing is often characterized by incomplete and ambiguous information. They suggest that, if learners are to try to understand the process that might lie behind what they experience, then they must use abductive reasoning: generating a hypothesis to account for one or more observations. One implication of abduction is that the consequences of errors are explained away, and learners do not realize they have made a mistake. Influenced by superficial resemblances between what they think they need to know and what they see or do, learners make wrong abductions which they fail to test. Taylor’s study of beginner programmers[l2] supports this view: she notes that because beginners do not have specialized programming skills, they have to make use of any existing information which seems relevant. Yet, although error-prone, abduction is not without value. One view of the abduction process is that learners are building up mental models, which, although often wrong, based as they are on the fragmented, ambiguous information perceived by the learner, may provide a basis for predicting or interpreting future events. Because learners may be active (even pre-emptive), and because they interpret what they learn in the light of their existing knowledge, the designer can never assume that instructional material studied by an independent learner is being used or interpreted in the way that the designer intended. This is true for all self-instructional or distance teaching material, but it has particular consequences for practical computing activities. The only way to find out how learners will engage with specific instructional material is to study them: there is no design process or method that can predict the outcomes or intercept all of the learner’s problems. CASE
STUDY:
COURSE
EH232,
COMPUTERS
AND
LEARNING
This study focussed on students enrolled in Computers and Learning (EH232), an Open University course which concerns the use of computers as educational tools, addressing issues such as whether computers facilitate learning, which pedagogical features are important in analysing and classifying programs, how to integrate computer-based learning with other teaching methods, and how to evaluate a program’s effectiveness. The course provides a variety of instructional materials: tapes, texts tutorials, example files, and exercises. The course assumes some experience using MS-DOS and using computers as educational tools, and it introduces Framework, a generalpurpose applications suite including word processor, spreadsheet, and outliner, as the vehicle for its practical exercises. Framework is reckoned to be accessible to novices without requiring a comprehensive computing mastery, but it is nevertheless reasonably complex, and its interface is not ideal. In the first year of the course, some of the students expressed difficulties and dissatisfactions with using the software provided by the course, particularly the Framework package. The Framework applications are driven by a mixture of commands and menus, and consistency of syntax and command structure is patchy across applications. The design of the Framework command structure is not evident from the interface, and the mixture of keystrokes and mouse gestures makes it difficult to ‘poke around’. In providing obstacles to the learner, Framework offers opportunities to the researcher for observing how students overcome gaps in their knowledge. The course provided an ideal test bed,
ANN JONES and MARIANF%TRE
30
therefore, for examining the difficulties that learners may have in using such software activities which are integrated with other forms of self-instructional materials.
for practical
The students All of the Computers and Learning students were asked if they were willing to take part in the study. Those who volunteered to participate were asked to keep a journal of ‘significant events’ in their practical work for the first ‘block’ of the course, and to fill in a questionnaire about their background and previous experience. Of the 140 students enrolled at the start of the course, 24 students completed both the questionnaire and the journal, and 37 other students completed the questionnaire alone, i.e. 44% of the total course population (61/140) completed the questionnaire, and 17% (24/140) completed both the questionnaire and the journal and are quoted in this study. Based on the questionnaire and the previous studies of Computers and Learning students, the 24 students on whom this study focussed appear reasonably representative and are certainly no less qualified and motivated overall than the typical student. Table 1 summarizes the profiles gleaned from the questionnaires. As can be seen from Table 1, the general profile of those 24 in this study is of students with high educational qualifications: three-quarters of them have either a degree or a professional qualification requiring two or more years of study. The lowest educational qualifications were ‘0 level’ achievement tests normally taken at around age 16: three of the students were at this level. All of the others had further qualifications and/or training, usually professional training: three had ‘A level’, ‘advanced level’ achievement tests normally taken at around age 18; a further 10 had teaching certificates; 18 of the students had degrees or post-graduate diplomas, one of them an OU degree. For those starting with only ‘0 levels’, this was not their first OU course. Table 1 shows that the
Table I. Student
profile
Percent of 24 in studv
Percent of all 61 ouestionnaires
Educational background
Degree or professional equivalent A-levels only O-levels only Teaching certificate Previous OU courses Lists maths or computing as interest or area of expertise Employed Help
75
72
13
13 41 83 38
13 IO 43 67 57
100
98
2s 50 92 92
23 4 82 84
21 8 17 54
30 II 5 52
58 42 0
51
4 33 63
5 31 64
21 63 17
IO 61 2s
58 8 21 46 17
49 26 20 46 26
orurlabie
Home Colleagues Use computer Use computer
at work at home
Programming experience
Lots SOIIK ou c0urse only None MS-DOS
e.rprriencr
In years In
hours NOW FW
39 IO
e.rpperimcr
Lots From OU courses only NOW GEM
urperience
Lots From OU courses only NOW Other
computing
experience 5 years
or more
l-5 years NOW A hobbyist Have taken courses
Practicalwork
31
students in the study have similar profiles to the total group of respondents: i.e. the 61 students who filled in the questionnaire, although rather more of the students in the study have taken previous OU courses (83% compared with 67% of respondents overall) and fewer listed maths or computing as an area of interest or expertise (38% compared with 57% of total respondents). Thirteen of the students were teachers or in education or training, e.g. as consultants. Others worked in a variety of jobs: administrators, oil refiner, naval lieutenant, policeman, and mechanical fitter. Only one was not in paid employment. The majority (92%) used a computer at work, although it was not a major part of their work. Many of the students were expecting to make increased use of computers at work and perhaps to develop their careers in ways that involved more computer use. Half of the students reported that they could seek help from colleagues. Typically, students had 5 years or more of experience of using computers at home, usually having started as a hobby. Few had extensive experience of programming. 54% had no programming experience; the rest had only limited experience, but they had taken other OU courses in the Home Computing Policy, notably the Technology Faculty foundation course and ‘Introduction to Info~ation Technology’. They all used word processors, and all except two used other applications. Their experience diverged when it came to MS-DOS: 14 (58%) had extensive experience, and the other 10 just a few hours. Similarly, there were two rough groups in terms of Framework experience: 15 of the students (63%) had no previous experience; eight (33%) had only used Framework on previous OU courses and just one student claimed extensive experience. Again the percentages are roughly similar for all respondents. A caveat regarding ‘previous experience’ There is no overall, easy measure of previous experience. We have summarized the students’ experience using a common shorthand: exposure time. Yet this gives no indication of particular competence. Different activities require different skills and knowledge; hence, without a skills assessment, one has little basis for predicting proficiencies or pitfalls from a student profile. Students enrolling in Computers and Learning are required to have some familiarity with computers-and in particular to have experience using MS-DOS-but the nature and extent of that familiarity varies considerably. What little prior experience students had using Framework came mainly from other OU courses. The jo~rnffls Students were asked to keep a journal of their computing activities. Each activity was allocated a page which was divided into two sections. The first section asked about the time of day they were working, how long the work session was, whether it took more or less time than anticipated, and what resources were at hand. The second section asked them to comment on significant events, both positive and negative. This format worked well, although, not surprisingly, the comments varied considerably in their length and detail. PATTERNS
OF PROBLEMS
Part of the philosophy of conducting a case study rather than, say, a controlled study, is that individual cases can illuminate and focus discussion of issues by providing examples and evidence even when statistical significance is not easily achievabfe. We made efforts to secure a reasonable level of participation in the study to detect whether observations were representative, because our interest was in general phenomena, but we considered this to be primarily an information gathering exercise, and so we treated all observations as having significance. The philosophy that no problem should be dismissed lightly is especially important in the context of distance learning, when interactions between students and course designers are limited. Even small problems can add up and seem ove~helming to the student working inde~ndently. Experience tells us that, for every one student who reveals a problem, there are many others who ‘suffer in silence’. The students’ reports about Framework indicate more general issues, both conceptual and practical, in the design of instructional materials. Students were asked in general to report ‘significant events’, i.e. insights, annoyances, or difficulties; the issues discussed below were identified from patterns across their reports. We illustrate our discussion of each with quotations
ANN JONFSand MARIANF’ETRE
32
from the students’ journals, which appear in italics. Counts are given of the students who reported instances of the issues in their journals (these appear in parentheses as n/24). These numbers indicate students who noted a difficulty, rather than those who experienced one; we have no record of problems omitted from the journals or just unnoticed. The numbers of students experiencing difficulties may be higher. Relating
new information
to existing
knowledge
Clearly, it is crucial for learners to be able to use and build on the knowledge they already have and to assimilate new knowledge into their existing models. However, this was a problem for many students. The job of sifting partially-familiar material, recognizing the information already known, and identifying the novel, seemed to be a demanding one. One experienced student reported difficulty in sorting out which parts of the material to pay attention to because it was already familiar: as someone acquainted with MS-DOS, could he safely ignore the re-introduction to it, or did he have to plough through in case there was something new? Although familiarity with computers and some knowledge of MS-DOS are prerequisite, the course includes one practical session re-introducing MS-DOS in order to ensure a common foundation. Another practical session introduces Framework. The main introductions to MS-DOS and Framework are via audio-tape. This allows the learner to follow instructions while keeping attention on the screen and keyboard. It has proved useful for novices in other courses, but its use in Computers and Learning raises the issue of how best it can be used by non-novices. How can the learner know in advance what will be new information? Most students (14/24) worked painstakingly through the tapes and accepted that the first part was revision, but some were too impatient to do this: “Found the tape slow in explaining the basics and the book was slightly sketchy . I found I had to wander around doing other things until the information I required was given. [Decided] I would be better to intuitively use Framework with the limited help of the book”. “The cassette explained the activity very well, although I found it a little hard going because it was familiar territory.” Nevertheless, there were many positive comments about the tapes. means of filling in gaps and consolidating their knowledge:
For some,
they provided
a
“I liked the tape. I had some knowledge of copy, DIR and DEL commands but this improved my understanding. I have pieced together quite a lot about MS-DOS on my own-have had no instruction-the book and the tape both helped”. Too busy to think; the trouble with assimilation
from
tutorial sessions
The main problem with tape-driven introductions and other tutorials, though, is that the information is presented in a linear form through which it is difficult to search, whether to review information or to look up a command. The problem is exacerbated by the audio-tape medium, which makes it difficult even to simply reverse in order to recover from a lapse in attention: “
. got lost a few times following the tape and then I pressed again as I couldn’t find my way back to where I was.”
Students (10/24) found tutorials frustrating, too preoccupied to assimilate the material: “Although understand
because the business
it states in activity 2 you will be learning what I was doing, I was just following
a wrong
of following
key and had to start
instructions
by doing I was too preoccupied instructions.”
left them
to really
Many students (1 l/24) found that they were left in a position where, having worked through the tape and followed instructions, they could not later easily apply what they had learned. Students had difficulty in identifying pertinent information and finding it again: “Needed to go through the tape twice more to remember is good but never tells you all you need!”
all the relevant
bits. The help facility
Practical
work
33
Such reports are consistent with a well-known problem with step-by-step instruction: learners are able to follow the sequence successfully but only operate at this procedural level; thus, they fail to assimilate such knowledge and are unable to adapt it to subsequent goals. Retention
Materials must take into account how regular the student’s use of the system will be and whether the student will have time and opportunity to reinforce learning. Students (1 l/24) often complained about forgetting commands or details between sessions: “Still wish we had a Framework mini-manual I could refer to when I forget the commandswhich I often do.” Transfer
of knowledge
Experience did not guarantee easy acquisition of Framework skills; rather, mild familiarity with other systems sometimes (for 7/24 students) led to confusion, suggesting that ‘a little knowledge is a dangerous thing’. Particular difficulty arose with familiar terms, because variations in use led to misunderstandings, especially if the particular relevance and interpretation of a new usage were not established clearly enough to supercede the student’s pre-existing interpretation. This was especially true of syntax: “It was a disadvantage knowing other wp packages because I would get the keys mixed up.” “I have a delete key on my keyboard and keep forgetting it won’t work with Framework.” [Even by the end of the activities, when writing up assignments this student was still having problems with Framework.]. Students also transferred other assumptions about how the package worked: “The spreadsheet that is being worked on is not the one which will necessarily be saved. Other software I’ve used of this type saves what is being worked on. It took some time to work out a reliable method of saving the work we had done for future use.” Students made unfavourable
comparisons:
“FW seems so old fashioned and clumsy [compared] with modern packages like Symphony.” Paradoxically, although at surface level previous experience caused some confusion, wellknowledge was helpful and enabled students to cope and to work out ways around problems more easily than less experienced students could: assimilated
“Previous experience of computer courses/understanding of computer languages etc. helped a lot to facilitate understanding of the program and such things as relative and absolute assigning.” Relevance
Students need to be convinced of the relevance of the work they are asked to do. Practicals designed to develop computer skills were viewed as gratuitous when they were not related clearly to course material. Some tasks were designed specifically to give students fundamental information or to provide enough practice with Framework to use it effectively; for example, students were asked to use the outlining facility at various points. But such persuasion was resented (by lo/24 studentstand often rejected-as impractical: “To use a computer to take notes is cumbersome and in any case notes are less accessible afterwards so I used pen and paper.” “Irritated I have to ‘save my findings on disk’. Why, when disk is not necessarily the most immediately accessible medium? Notes on paper.”
ANN JONESand MARIANF’ETRE
34
The need for reference materials
Students (14/24) had difficulty in gaining overviews of the system and in organizing information within it. They (1 l/24) found it hard to identify the information they wanted--or even to identify what was pertinent-and to find it: “I only discovered control CE by randomly looking through Intro to Computing Book 2.” “A card with all the commands printed on it and language syntax would be useful.” The demand for focussed reference materials was universal, and it increased with experience. Although few students (6/24) compiled their own reference materials, those who did benefitted markedly. “The POSNEG file was very hard to find. It would have been useful to have a printed directory of all the disks to make finding files easier and I did this.” and “Again, I needed my printed list of files to locate the FW file quickly.” Not everything can be anticipated, and the possibility of novel exploration amplifies the need for good reference materials, so that students can confirm insights or rectify mis-apprehensions (i.e. easy reference means that students need not rely solely on abduction). Students who create their own reference representations are providing both a record of the structure they’re evolving and a means for verifying their insights systematically, i.e. a mechanism for making their abductions explicit and for testing them. In the absence of monitoring, self-verification becomes essential. Practical d#icuities
Practical matters-transferring files, command syntax, recognizing file names, locating files on disks and within applications, system configuration, etc.-are of great importance. Such difficulties can easily swamp the concepts being presented. Although all of the students achieved some early success with Framework, they were distracted by practical issues well into the practical work. Such practical difficulties can be sub-divided into:
lfinding files, both on disks and within applications; manipulating files, e.g. saving and retrieving; disk difficulties; 0 printing; l running applications~ l command syntax. l
l
These will be considered in turn. Finding files. Locating files, both on disks and within applications, was a problem for half of the students (l2/24). The course uses a large number of files spread among several disks. Some are run by the desktop application GEM, and others via Framework, while some are started up by a quite different procedure. “I had a problem finding a particular program but found it in the end by using the menu bar.” “Couldn’t load GEM initially-missing batch file. Couldn’t find batch file on original disks . . . Resorted to manual and found correct batch file by reading small print closely.” ~~nipulating~~es e.g. saving and retr~e~~~g,IIaving located the files they needed, many students fl0/24) had problems saving and retrieving material, especially the Framework spreadsheets:
“I saved the notes on reading 1 on the hard disk but couldn’t make a back up copy without exiting from FW. When I typed ‘TYPE’ gobbledygook came up from both disks. Eventually I deleted both programs then in the middle of the night I realised I probably needed to be in FW to read them. So I did it all over again!” “Saving spreadsheets was not easy-we tried saving them to B drive and got several files from drive A (fwk 2) disk saved across as well but they seemed OK. When we tried to recall them for the children to use we found they contained garbage and it took a good half hour to create a dummy spreadsheet and to establish exactly how to save the files and how relatively complex that was.”
Practical work
35
Disk d@culties. Managing the floppy disks was another recurrent problem (reported by 1 l/24 students), and it was compounded by a foul-up: students were accidentally sent write-protected disks. An erratum slip was sent, but by that time many students had struggled with the problem. Some had given up, although many had solved it in different, ingenious ways. The sheer number of disks posed some organizational difficulty. Although disk juggling may seem trivial, it can have significant effect in a course of this sort, where the multiple media and the number of different resources and bits of equipment all contribute to a high organizational load. Some students also struggled to cope with multiple disk drives. Printing. Printing was a problem for several students (8/24). Sometimes this was a hardware problem, sometimes a problem with the printer software. Some students were using machines which were not strictly ‘compatible’: “Major snags with printing. Blank pages skipped between printed pages. OU computing help service advice sought and problem resolved eventually by using different paper length.” Running applications. Half of the students (12/24) had difficulty running applications. Some never got particular applications to run at all; others did so through trial, error and experimentation; and a few read the instructions carefully after they had had problems. The following comment is from a student who had a mixture of many practical problems: “Running LP chart was not straightforward. First I couldn’t find it, and then couldn’t get it started. Then it wouldn’t let me exit the program (it turned out I’d write protected the disk). I’d started the program via GEM and couldn’t return to it-1 had to reboot. This meant reading the software info on Linkway so I had to look at the info for Linkway and it also took me time to realise that Learning Processes chart was LP chart that I’d been asked to run . . . It didn’t take me that long but I imagine someone less experienced would have been quite frustrated. Using the LP chart was also confusing: you had to click on crosses to get the analysis. I hadn’t understood this from the help and it took me a while to find out.” Command syntax. Command syntax in MS-DOS and Framework can appear arcane. Spaces and punctuation have significance, and numerous ‘meta’ keys change meanings of other keys. Although there is some consistency, the command structure and command names may not be amenable to novice intuition. It can be difficult for the student to identify the significant features of a command string. For example, files names appear in different forms in command lines and directory listings. Students (1 l/24) struggled with various elements of command syntax and had particular difficulty with the outliner commands that make substantial use of function keys. “Got a little outliner-tried on!”
bogged down on trying to move to next section i.e. from 1.l to 1.2 in to use level down key and thus deleted heading just typed in-had NUM lock
Time to competence In spite of such practical problems, some familiarity and confidence were achieved within first few sessions, and students tended to complete their practical work successfully in roughly time estimated by the course designers, except when significant practical problems arose:
the the
“Before starting this session, I had an electricity cut whilst I was installing a new 3” drive . . . this resulted in major problems including loss of system files from the hard disc. I had to transfer all my existing hd drive files to floppy disks, reformat the hard disk, then replace all the files. This took approx. 18 hours . . .” Cussedness,
or the need to take control sometimes
For some students, reading the manual was clearly a last resort. “ . . . Got some way with these before giving up! (Thought I ought to be able to sort out this problem!) Gave up. Resorted to manual . . .” Although this may be mere cussedness, it is more likely to reflect a fundamental need in the independent student to take an active role. Such behaviour would not be inappropriate in the CAE 22,1-*--D
ANN JONESand MARIANPETKE
36
context of a friendlier system, one that gives the explorer hints about how it is structured and where things can be found, one designed to facilitate learning by exploration. Such experimenting (reported by IO/24 students) is consistent with the notion that the student is building up a mental model of the material, assimilating it into an internal knowledge structure that relates it to the student’s previous knowledge. That assimilation may create tension-between the student’s growing mental model and the structure of the instructional materials. This tension may be manifest as impatience, anticipating information, jumping ahead, bypassing the prepared material, and so on, even if that behaviour is inefficient or counter-productive. “I made . . . mistakes by anticipating instructions and then doing them incorrectly.” “I didn’t revise the tape, but preferred to experiment and explore on my own.” In needing to work things out for themselves, these students (who were not complete novices) may be developing what Petre has observed in experts: operational reasoning, which is often manifest as a need to understand and explore how something works even if such knowledge is not strictly necessary for the task in hand[l3]. If the student is indeed building up a mental model of the material, then the instructional material must both provide useful guidance in palatable form, whether as an overview in the text or embedded in the interface as perceptual cueing or obvious structuring. There must also be readily accessible means (such as reference materials) of verifying uncertain discoveries and correcting misconceptions. “It is worth noting . . . that often new programming successes are achieved almost in the end-hours of adjusting minor details and ultimately not fully understanding achieved the result.”
by chance how you
Given that students are likely to approach instructional materials in unexpected, even haphazard ways, the instructional materials must provide opportunities for the student to re-align his or her experience with the course.
CONCLUSIONS
AND
1MPLICATIONS
The students’ experiences emphasize the importance of assessing the overheads involved in using a particular computer-based tool. When the application was perceived to be inappropriate, cumbersome, or uncomfortable, e.g. for note-taking or reading lengthy text, students avoided using it. Avoidance could provoke extreme behaviours: some students (5124) found Framework sufficiently unpalatable to invest instead in transferring the files into another format in order to work within a different application. The designer of self-instruction material cannot control independent study. Not everything can be anticipated, and the notion of the actizle student suggests that unexpected use of instructional materials is normal, even desirable. A student’s unruliness may contribute to independent learning and to the development of a mental model. The challenge is to facilitate exploration and experiment while ensuring provision of the necessary foundation and protecting students from the pitfalls of their own discoveries. The means for doing so would have to include: provision of a framework on which to build, e.g. the skeleton of a model, a suggested overview, a representation of conceptual structure l provision of an interface amenable to exploration and discovery, one made friendly by clarity. consistency, and the reflection of conceptual structure in perceptual cues, and 0 provision of a means for self-verification, for easy access, e.g. via appropriate reference materials, to the information that can confirm or correct a student’s discoveries. l
The key issue appears to be enabling the student to structure the information, that is, to take control of it. Much attention is given to teaching-more needs to be given to what happens qfler the initial exposure, when the student must make use of the information, must recall it, reconsider it, refer to it, re-arrange it, review it. Only by understanding how the learner goes about using materials and owning information, will we be able to design instructional materials to facilitate the transfer of control from instructional material to learner.
Practical work
31
REFERENCES 1. Jones A. and O’Shea T., Barriers to the use of computer assisted learning. Br. J. Educl Technol. 13, 207-217 (1982). 2. Carroll J., The Nurnberg Funnel: Designing Minimalist Instruction for Practical Computing Skill. The MIT Press, London (1990). 3. Carroll J. M., Mack R. L., Lewis C., Grischkowski N. and Robertson S., Exploring a word processor. HumanComputer Interaction 1, 283-307 (1985). 4. Czaja S. J., Hammond K., Blascovich J. J. and Swede H., Learning to use a word-processing system as a function of training strategy. Behau. Information Technol. 5, 203-216 (1986). 5. Polson P. G. and Kieras D. E., A quantitative model of the learning and performance of text editing knowledge. CHI ‘85, Association for Computing Machinery, New York, pp. 207-212 (1983).
6. Jones A. C.. Bmnirical studies of novices learning programming. Ph.D. thesis No. 10, Centre for Information Technology in Education, Institute of Educational Technology, The Open University (1990). 7. du Boulay B., O’Shea T. and Monk J., The glass box inside the black box: presenting computing concepts to novices. Int. J. Man-Machine Stud. 14, 237-249 (1981). 8. Eisenstadt M., A friendly software environment for psychology students. AISB Q. (1979). 9. Mack R., Lewis C. and Carroll J., Learning to use word processors: problems and prospects. ACM Trans. Ofice Systems 3, 185-204 (1983).
10. Marton F. and Saljo R., On qualitative differences in learning-II. Outcome as a function of the learner’s conception of the task. Br. J. Educl Psychol. 46, 115-127 (1976). Il. Lewis C. and Mack R., The role of abduction in learning to use a computer system. IBM Watson Research Centre Report RC9433, No. 41620, Yorktown Heights, New York (1982). 12. Taylor J., Analysing novices analysing Prolog: what stories do novices tell themselves about Prolog? Insfructional Sci. 19, 283-309 (1990). 13. Petre M., Shifts in reasoning about software and hardware systems: must operational models underpin declarative ones? Invited paper presented at The Third Workshop of the Psychology of Programming Interest Group, Hatfield (January 1991).
ComparrersE&c. Vol. 22, No. l/2, pp.27-37, 1994 Printed in Great Britain. All rights reserved
COMPUTER-BASED
PRACTICAL WORK AT A DISTANCE: A CASE STUDY ANN JONES and MARIAN F%TRE
Institute
of Educational
Technology,
The Open University, Walton Hall, Milton Keynes MK7 6AA, England
Abstract-How do students engage with distance teaching materials for carrying out practical computing work? How can we design teaching materials to make the situation better? Questions about students’ use of instructional materials are particularly important in the context of distance learning, when students have little or no teacher support. This paper reports the results of a case study of home computing use in the Open University course: ‘Computers and Learning’: which is typical of many tXJ courses in its use of different media: texts, video, audio-tapes and of course the computer practical work. We describe how students go about learning to use the computer for their work; the kinds of problems they have and how they overcome them. Nearly all the students successfully completed their practical work: and did so within the allocated time. Nevertheless the process of carrying out the practical work, was not without its problems, and the students’ reports about their practical work indicate more general issues in instructional design. A major issue is the design of instructional materials that are geared to students with different prior knowledge and that can be readily used for reference ufer the student’s initial exposure to it. The student needs to be able to structure the material in such a way that it can be reviewed, restructured and rearranged. Students are active in their learning and instructional materials need to facilitate this by providing an environment designed for exploration, but which supports learners and helps them when they make mistakes.
INTRODUCTION
Personal computers are being used increasingly in distance education, and they have the potential to change the nature of that education radically. For the student learning at a distance or using self-instruction material, computers can be powerful study tools, whether providing general ‘clerical’ support, e.g. word processing facilities, spreadsheets, databases, or contributing to the subject area, e.g. via simulations in Physics, calculation and statistical packages in Mathematics, programming environments in Computer Science. Open University distance learning courses have used computers as tools in all of the above forms. Questions about students’ use of instructional materials are particularly important in the contexts of distance learning and independent study, when students are isolated from peers and teachers. Practical issues about self-instruction materials are amplified when the work is computer based and students must learn about using the computer as a tool as well as about the information it contains. Where do students look for information? Do they read and study in the ways we expect? What kinds of problems do they encounter and how do they cope with them? What are the implications if they use materials in unexpected ways? How do students cope with the unexpected? Imagine that a student studying independently receives a study pack that includes a screwdriver and instructions to use this tool to open a box of materials. The course designers might assume that the student will use the screwdriver to remove the screws fixing the lid on the box; what are the implications when the student uses the screwdriver as a lever to prize the lid off, or as a poker to pierce the box? Will unscrewers, leverers, and piercers all achieve an appropriate understanding? This paper examines the assumption that students use instructional materials effectively, appropriately, and predictably. It reports preliminary results of a case study of home computing use in the Open University course EH232: Computers and Learning. The study used student questionnaires and journals to identify practical issues for the design of computer-oriented instructional materials for independent study. This paper first sets the case study in the context of the problem of computer access in distance education and in the context of related research on independent study of different sorts, Then, having described the set-up of the study, it enumerates as general phenomena the sorts of obstacles, both conceptual and practical.
28
ANN JONESand MARIANPETRE
BACKGROUND:
The Home
Computing
ASSESSING
ACCESS
UNDER
THE
HOME
COMPUTING
POLICY
Policy
Until 1988, the main method of access to computers for Open University students was through terminals at regional study centres which were connected to one of the University’s mainframe computers. By the mid-1980s this kind of access was becoming increasingly problematic for the following reasons: (1) The mainframe system was unable to cope with increasing student demand. (2) More students were buying microcomputers for use on their courses and wanted to buy one that would be suitable for whichever computer-based course they followed. (3) It had been acknowledged that using a computer at home was far preferable to the problems of using one at a study centre[l]. A ‘Home Computing Policy’ was introduced in 1988 which specified a minimum computer system: an MS-DOS machine with 510 Kbytes of memory, a floppy disk drive, another storage device, and a mouse. The specification was modest, mainly to minimize the cost to students. By 1992, 11 courses required students to have access to a personal computer with this specification; this meant that about 17,000 Open University students were studying courses requiring the use of a computer at home. Evaluation
of the Policy
With so many students expected to obtain, set up and use computers, without on-hand help or supervision, there was great potential for things to go wrong. It was imperative, therefore, to monitor the effects of the Policy. An evaluation was undertaken through large-scale student surveys, student journals, and interviews with staff and students. The overall success of the Policy was quickly established, and information was obtained on issues such as the nature and quality of access; the extent to which it differed for different groups; the extent to which students sought help and what routes they took; and so on. The Home Computing Policy largely solved the problem of access to the computer itself and so provided a common medium for learning. The case study However, having a computer at hand is only the outer layer of access. Students also need access to the inside of the ‘box’. For the computer to be an effective learning tool, students need to be able to use it to run the necessary applications. The large-scale, general evaluation of the Home Computing Policy could not properly address this level of access. There is some evidence[2] that learning to use a computer with the aid of self-instructional material can be problematic, and so the case study reported here was undertaken to investigate this process, to ascertain the extent to which students were experiencing difficulties, and to discover something about the problems that occurred, under what conditions they occurred, and how the students coped with them. The case study focussed on two important issues in the design of instructional materials: the selection and structuring of key information, and students’ actual use of resources, whether provided as part of the course or available as part of the student’s own environment. A number of questions were prominent: l
l l l
What are the implications for the sorts of information structures required by and being formed by students? How do people actually use the resources, e.g. tutorials, documents, tapes, that are provided? When does the time it takes to become minimally competent with a system become prohibitive? What is the impact of a student’s previous experience?
With no teacher present to interpret the information and help the student incorporate it into an appropriate framework and usable form, the instructional materials are the keys (or the barriers) to access. Related
research
Many of the same issues arise in designing self-instructional materials for practical computing as in producing distance education teaching materials: in both cases the learner is almost completely
Practical work
29
reliant on the packaged teaching material. Research findings from two related areas are relevant to teaching computer related activities at a distance: first, studies about learning to use applications such as word processors [3-51, and second, research on learning to program using distance learning materials[68]. Mack et a1.[9] studied learners using word processors. The learners in these studies were particularly active in their learning: they did not wait for instructions, nor did they read the manuals properly but jumped ahead and tried things out for themselves; they used their prior knowledge (which often led to misunderstandings); and they had great difficulty in recovering from errors. Another study, of learner programmers using highly-structured distance learning materials suggested that the students operated at too low a level: they were able to follow the steps of the instruction but lacked an overview or a cognizance of the goal towards which they were working[8]. Such an approach to learning is similar to what Marton and Saljo [lo] refer to as surface level processing: they comment that “students adopt and approach determined by their expectations of what is required of them” (p. 125). Lewis and Mack[ll] observe that complex learning such as self-instruction in practical computing is often characterized by incomplete and ambiguous information. They suggest that, if learners are to try to understand the process that might lie behind what they experience, then they must use abductive reasoning: generating a hypothesis to account for one or more observations. One implication of abduction is that the consequences of errors are explained away, and learners do not realize they have made a mistake. Influenced by superficial resemblances between what they think they need to know and what they see or do, learners make wrong abductions which they fail to test. Taylor’s study of beginner programmers[l2] supports this view: she notes that because beginners do not have specialized programming skills, they have to make use of any existing information which seems relevant. Yet, although error-prone, abduction is not without value. One view of the abduction process is that learners are building up mental models, which, although often wrong, based as they are on the fragmented, ambiguous information perceived by the learner, may provide a basis for predicting or interpreting future events. Because learners may be active (even pre-emptive), and because they interpret what they learn in the light of their existing knowledge, the designer can never assume that instructional material studied by an independent learner is being used or interpreted in the way that the designer intended. This is true for all self-instructional or distance teaching material, but it has particular consequences for practical computing activities. The only way to find out how learners will engage with specific instructional material is to study them: there is no design process or method that can predict the outcomes or intercept all of the learner’s problems. CASE
STUDY:
COURSE
EH232,
COMPUTERS
AND
LEARNING
This study focussed on students enrolled in Computers and Learning (EH232), an Open University course which concerns the use of computers as educational tools, addressing issues such as whether computers facilitate learning, which pedagogical features are important in analysing and classifying programs, how to integrate computer-based learning with other teaching methods, and how to evaluate a program’s effectiveness. The course provides a variety of instructional materials: tapes, texts tutorials, example files, and exercises. The course assumes some experience using MS-DOS and using computers as educational tools, and it introduces Framework, a generalpurpose applications suite including word processor, spreadsheet, and outliner, as the vehicle for its practical exercises. Framework is reckoned to be accessible to novices without requiring a comprehensive computing mastery, but it is nevertheless reasonably complex, and its interface is not ideal. In the first year of the course, some of the students expressed difficulties and dissatisfactions with using the software provided by the course, particularly the Framework package. The Framework applications are driven by a mixture of commands and menus, and consistency of syntax and command structure is patchy across applications. The design of the Framework command structure is not evident from the interface, and the mixture of keystrokes and mouse gestures makes it difficult to ‘poke around’. In providing obstacles to the learner, Framework offers opportunities to the researcher for observing how students overcome gaps in their knowledge. The course provided an ideal test bed,
ANN JONES and MARIANF%TRE
30
therefore, for examining the difficulties that learners may have in using such software activities which are integrated with other forms of self-instructional materials.
for practical
The students All of the Computers and Learning students were asked if they were willing to take part in the study. Those who volunteered to participate were asked to keep a journal of ‘significant events’ in their practical work for the first ‘block’ of the course, and to fill in a questionnaire about their background and previous experience. Of the 140 students enrolled at the start of the course, 24 students completed both the questionnaire and the journal, and 37 other students completed the questionnaire alone, i.e. 44% of the total course population (61/140) completed the questionnaire, and 17% (24/140) completed both the questionnaire and the journal and are quoted in this study. Based on the questionnaire and the previous studies of Computers and Learning students, the 24 students on whom this study focussed appear reasonably representative and are certainly no less qualified and motivated overall than the typical student. Table 1 summarizes the profiles gleaned from the questionnaires. As can be seen from Table 1, the general profile of those 24 in this study is of students with high educational qualifications: three-quarters of them have either a degree or a professional qualification requiring two or more years of study. The lowest educational qualifications were ‘0 level’ achievement tests normally taken at around age 16: three of the students were at this level. All of the others had further qualifications and/or training, usually professional training: three had ‘A level’, ‘advanced level’ achievement tests normally taken at around age 18; a further 10 had teaching certificates; 18 of the students had degrees or post-graduate diplomas, one of them an OU degree. For those starting with only ‘0 levels’, this was not their first OU course. Table 1 shows that the
Table I. Student
profile
Percent of 24 in studv
Percent of all 61 ouestionnaires
Educational background
Degree or professional equivalent A-levels only O-levels only Teaching certificate Previous OU courses Lists maths or computing as interest or area of expertise Employed Help
75
72
13
13 41 83 38
13 IO 43 67 57
100
98
2s 50 92 92
23 4 82 84
21 8 17 54
30 II 5 52
58 42 0
51
4 33 63
5 31 64
21 63 17
IO 61 2s
58 8 21 46 17
49 26 20 46 26
orurlabie
Home Colleagues Use computer Use computer
at work at home
Programming experience
Lots SOIIK ou c0urse only None MS-DOS
e.rprriencr
In years In
hours NOW FW
39 IO
e.rpperimcr
Lots From OU courses only NOW GEM
urperience
Lots From OU courses only NOW Other
computing
experience 5 years
or more
l-5 years NOW A hobbyist Have taken courses
Practicalwork
31
students in the study have similar profiles to the total group of respondents: i.e. the 61 students who filled in the questionnaire, although rather more of the students in the study have taken previous OU courses (83% compared with 67% of respondents overall) and fewer listed maths or computing as an area of interest or expertise (38% compared with 57% of total respondents). Thirteen of the students were teachers or in education or training, e.g. as consultants. Others worked in a variety of jobs: administrators, oil refiner, naval lieutenant, policeman, and mechanical fitter. Only one was not in paid employment. The majority (92%) used a computer at work, although it was not a major part of their work. Many of the students were expecting to make increased use of computers at work and perhaps to develop their careers in ways that involved more computer use. Half of the students reported that they could seek help from colleagues. Typically, students had 5 years or more of experience of using computers at home, usually having started as a hobby. Few had extensive experience of programming. 54% had no programming experience; the rest had only limited experience, but they had taken other OU courses in the Home Computing Policy, notably the Technology Faculty foundation course and ‘Introduction to Info~ation Technology’. They all used word processors, and all except two used other applications. Their experience diverged when it came to MS-DOS: 14 (58%) had extensive experience, and the other 10 just a few hours. Similarly, there were two rough groups in terms of Framework experience: 15 of the students (63%) had no previous experience; eight (33%) had only used Framework on previous OU courses and just one student claimed extensive experience. Again the percentages are roughly similar for all respondents. A caveat regarding ‘previous experience’ There is no overall, easy measure of previous experience. We have summarized the students’ experience using a common shorthand: exposure time. Yet this gives no indication of particular competence. Different activities require different skills and knowledge; hence, without a skills assessment, one has little basis for predicting proficiencies or pitfalls from a student profile. Students enrolling in Computers and Learning are required to have some familiarity with computers-and in particular to have experience using MS-DOS-but the nature and extent of that familiarity varies considerably. What little prior experience students had using Framework came mainly from other OU courses. The jo~rnffls Students were asked to keep a journal of their computing activities. Each activity was allocated a page which was divided into two sections. The first section asked about the time of day they were working, how long the work session was, whether it took more or less time than anticipated, and what resources were at hand. The second section asked them to comment on significant events, both positive and negative. This format worked well, although, not surprisingly, the comments varied considerably in their length and detail. PATTERNS
OF PROBLEMS
Part of the philosophy of conducting a case study rather than, say, a controlled study, is that individual cases can illuminate and focus discussion of issues by providing examples and evidence even when statistical significance is not easily achievabfe. We made efforts to secure a reasonable level of participation in the study to detect whether observations were representative, because our interest was in general phenomena, but we considered this to be primarily an information gathering exercise, and so we treated all observations as having significance. The philosophy that no problem should be dismissed lightly is especially important in the context of distance learning, when interactions between students and course designers are limited. Even small problems can add up and seem ove~helming to the student working inde~ndently. Experience tells us that, for every one student who reveals a problem, there are many others who ‘suffer in silence’. The students’ reports about Framework indicate more general issues, both conceptual and practical, in the design of instructional materials. Students were asked in general to report ‘significant events’, i.e. insights, annoyances, or difficulties; the issues discussed below were identified from patterns across their reports. We illustrate our discussion of each with quotations
ANN JONFSand MARIANF’ETRE
32
from the students’ journals, which appear in italics. Counts are given of the students who reported instances of the issues in their journals (these appear in parentheses as n/24). These numbers indicate students who noted a difficulty, rather than those who experienced one; we have no record of problems omitted from the journals or just unnoticed. The numbers of students experiencing difficulties may be higher. Relating
new information
to existing
knowledge
Clearly, it is crucial for learners to be able to use and build on the knowledge they already have and to assimilate new knowledge into their existing models. However, this was a problem for many students. The job of sifting partially-familiar material, recognizing the information already known, and identifying the novel, seemed to be a demanding one. One experienced student reported difficulty in sorting out which parts of the material to pay attention to because it was already familiar: as someone acquainted with MS-DOS, could he safely ignore the re-introduction to it, or did he have to plough through in case there was something new? Although familiarity with computers and some knowledge of MS-DOS are prerequisite, the course includes one practical session re-introducing MS-DOS in order to ensure a common foundation. Another practical session introduces Framework. The main introductions to MS-DOS and Framework are via audio-tape. This allows the learner to follow instructions while keeping attention on the screen and keyboard. It has proved useful for novices in other courses, but its use in Computers and Learning raises the issue of how best it can be used by non-novices. How can the learner know in advance what will be new information? Most students (14/24) worked painstakingly through the tapes and accepted that the first part was revision, but some were too impatient to do this: “Found the tape slow in explaining the basics and the book was slightly sketchy . I found I had to wander around doing other things until the information I required was given. [Decided] I would be better to intuitively use Framework with the limited help of the book”. “The cassette explained the activity very well, although I found it a little hard going because it was familiar territory.” Nevertheless, there were many positive comments about the tapes. means of filling in gaps and consolidating their knowledge:
For some,
they provided
a
“I liked the tape. I had some knowledge of copy, DIR and DEL commands but this improved my understanding. I have pieced together quite a lot about MS-DOS on my own-have had no instruction-the book and the tape both helped”. Too busy to think; the trouble with assimilation
from
tutorial sessions
The main problem with tape-driven introductions and other tutorials, though, is that the information is presented in a linear form through which it is difficult to search, whether to review information or to look up a command. The problem is exacerbated by the audio-tape medium, which makes it difficult even to simply reverse in order to recover from a lapse in attention: “
. got lost a few times following the tape and then I pressed again as I couldn’t find my way back to where I was.”
Students (10/24) found tutorials frustrating, too preoccupied to assimilate the material: “Although understand
because the business
it states in activity 2 you will be learning what I was doing, I was just following
a wrong
of following
key and had to start
instructions
by doing I was too preoccupied instructions.”
left them
to really
Many students (1 l/24) found that they were left in a position where, having worked through the tape and followed instructions, they could not later easily apply what they had learned. Students had difficulty in identifying pertinent information and finding it again: “Needed to go through the tape twice more to remember is good but never tells you all you need!”
all the relevant
bits. The help facility
Practical
work
33
Such reports are consistent with a well-known problem with step-by-step instruction: learners are able to follow the sequence successfully but only operate at this procedural level; thus, they fail to assimilate such knowledge and are unable to adapt it to subsequent goals. Retention
Materials must take into account how regular the student’s use of the system will be and whether the student will have time and opportunity to reinforce learning. Students (1 l/24) often complained about forgetting commands or details between sessions: “Still wish we had a Framework mini-manual I could refer to when I forget the commandswhich I often do.” Transfer
of knowledge
Experience did not guarantee easy acquisition of Framework skills; rather, mild familiarity with other systems sometimes (for 7/24 students) led to confusion, suggesting that ‘a little knowledge is a dangerous thing’. Particular difficulty arose with familiar terms, because variations in use led to misunderstandings, especially if the particular relevance and interpretation of a new usage were not established clearly enough to supercede the student’s pre-existing interpretation. This was especially true of syntax: “It was a disadvantage knowing other wp packages because I would get the keys mixed up.” “I have a delete key on my keyboard and keep forgetting it won’t work with Framework.” [Even by the end of the activities, when writing up assignments this student was still having problems with Framework.]. Students also transferred other assumptions about how the package worked: “The spreadsheet that is being worked on is not the one which will necessarily be saved. Other software I’ve used of this type saves what is being worked on. It took some time to work out a reliable method of saving the work we had done for future use.” Students made unfavourable
comparisons:
“FW seems so old fashioned and clumsy [compared] with modern packages like Symphony.” Paradoxically, although at surface level previous experience caused some confusion, wellknowledge was helpful and enabled students to cope and to work out ways around problems more easily than less experienced students could: assimilated
“Previous experience of computer courses/understanding of computer languages etc. helped a lot to facilitate understanding of the program and such things as relative and absolute assigning.” Relevance
Students need to be convinced of the relevance of the work they are asked to do. Practicals designed to develop computer skills were viewed as gratuitous when they were not related clearly to course material. Some tasks were designed specifically to give students fundamental information or to provide enough practice with Framework to use it effectively; for example, students were asked to use the outlining facility at various points. But such persuasion was resented (by lo/24 studentstand often rejected-as impractical: “To use a computer to take notes is cumbersome and in any case notes are less accessible afterwards so I used pen and paper.” “Irritated I have to ‘save my findings on disk’. Why, when disk is not necessarily the most immediately accessible medium? Notes on paper.”
ANN JONESand MARIANF’ETRE
34
The need for reference materials
Students (14/24) had difficulty in gaining overviews of the system and in organizing information within it. They (1 l/24) found it hard to identify the information they wanted--or even to identify what was pertinent-and to find it: “I only discovered control CE by randomly looking through Intro to Computing Book 2.” “A card with all the commands printed on it and language syntax would be useful.” The demand for focussed reference materials was universal, and it increased with experience. Although few students (6/24) compiled their own reference materials, those who did benefitted markedly. “The POSNEG file was very hard to find. It would have been useful to have a printed directory of all the disks to make finding files easier and I did this.” and “Again, I needed my printed list of files to locate the FW file quickly.” Not everything can be anticipated, and the possibility of novel exploration amplifies the need for good reference materials, so that students can confirm insights or rectify mis-apprehensions (i.e. easy reference means that students need not rely solely on abduction). Students who create their own reference representations are providing both a record of the structure they’re evolving and a means for verifying their insights systematically, i.e. a mechanism for making their abductions explicit and for testing them. In the absence of monitoring, self-verification becomes essential. Practical d#icuities
Practical matters-transferring files, command syntax, recognizing file names, locating files on disks and within applications, system configuration, etc.-are of great importance. Such difficulties can easily swamp the concepts being presented. Although all of the students achieved some early success with Framework, they were distracted by practical issues well into the practical work. Such practical difficulties can be sub-divided into:
lfinding files, both on disks and within applications; manipulating files, e.g. saving and retrieving; disk difficulties; 0 printing; l running applications~ l command syntax. l
l
These will be considered in turn. Finding files. Locating files, both on disks and within applications, was a problem for half of the students (l2/24). The course uses a large number of files spread among several disks. Some are run by the desktop application GEM, and others via Framework, while some are started up by a quite different procedure. “I had a problem finding a particular program but found it in the end by using the menu bar.” “Couldn’t load GEM initially-missing batch file. Couldn’t find batch file on original disks . . . Resorted to manual and found correct batch file by reading small print closely.” ~~nipulating~~es e.g. saving and retr~e~~~g,IIaving located the files they needed, many students fl0/24) had problems saving and retrieving material, especially the Framework spreadsheets:
“I saved the notes on reading 1 on the hard disk but couldn’t make a back up copy without exiting from FW. When I typed ‘TYPE’ gobbledygook came up from both disks. Eventually I deleted both programs then in the middle of the night I realised I probably needed to be in FW to read them. So I did it all over again!” “Saving spreadsheets was not easy-we tried saving them to B drive and got several files from drive A (fwk 2) disk saved across as well but they seemed OK. When we tried to recall them for the children to use we found they contained garbage and it took a good half hour to create a dummy spreadsheet and to establish exactly how to save the files and how relatively complex that was.”
Practical work
35
Disk d@culties. Managing the floppy disks was another recurrent problem (reported by 1 l/24 students), and it was compounded by a foul-up: students were accidentally sent write-protected disks. An erratum slip was sent, but by that time many students had struggled with the problem. Some had given up, although many had solved it in different, ingenious ways. The sheer number of disks posed some organizational difficulty. Although disk juggling may seem trivial, it can have significant effect in a course of this sort, where the multiple media and the number of different resources and bits of equipment all contribute to a high organizational load. Some students also struggled to cope with multiple disk drives. Printing. Printing was a problem for several students (8/24). Sometimes this was a hardware problem, sometimes a problem with the printer software. Some students were using machines which were not strictly ‘compatible’: “Major snags with printing. Blank pages skipped between printed pages. OU computing help service advice sought and problem resolved eventually by using different paper length.” Running applications. Half of the students (12/24) had difficulty running applications. Some never got particular applications to run at all; others did so through trial, error and experimentation; and a few read the instructions carefully after they had had problems. The following comment is from a student who had a mixture of many practical problems: “Running LP chart was not straightforward. First I couldn’t find it, and then couldn’t get it started. Then it wouldn’t let me exit the program (it turned out I’d write protected the disk). I’d started the program via GEM and couldn’t return to it-1 had to reboot. This meant reading the software info on Linkway so I had to look at the info for Linkway and it also took me time to realise that Learning Processes chart was LP chart that I’d been asked to run . . . It didn’t take me that long but I imagine someone less experienced would have been quite frustrated. Using the LP chart was also confusing: you had to click on crosses to get the analysis. I hadn’t understood this from the help and it took me a while to find out.” Command syntax. Command syntax in MS-DOS and Framework can appear arcane. Spaces and punctuation have significance, and numerous ‘meta’ keys change meanings of other keys. Although there is some consistency, the command structure and command names may not be amenable to novice intuition. It can be difficult for the student to identify the significant features of a command string. For example, files names appear in different forms in command lines and directory listings. Students (1 l/24) struggled with various elements of command syntax and had particular difficulty with the outliner commands that make substantial use of function keys. “Got a little outliner-tried on!”
bogged down on trying to move to next section i.e. from 1.l to 1.2 in to use level down key and thus deleted heading just typed in-had NUM lock
Time to competence In spite of such practical problems, some familiarity and confidence were achieved within first few sessions, and students tended to complete their practical work successfully in roughly time estimated by the course designers, except when significant practical problems arose:
the the
“Before starting this session, I had an electricity cut whilst I was installing a new 3” drive . . . this resulted in major problems including loss of system files from the hard disc. I had to transfer all my existing hd drive files to floppy disks, reformat the hard disk, then replace all the files. This took approx. 18 hours . . .” Cussedness,
or the need to take control sometimes
For some students, reading the manual was clearly a last resort. “ . . . Got some way with these before giving up! (Thought I ought to be able to sort out this problem!) Gave up. Resorted to manual . . .” Although this may be mere cussedness, it is more likely to reflect a fundamental need in the independent student to take an active role. Such behaviour would not be inappropriate in the CAE 22,1-*--D
ANN JONESand MARIANPETKE
36
context of a friendlier system, one that gives the explorer hints about how it is structured and where things can be found, one designed to facilitate learning by exploration. Such experimenting (reported by IO/24 students) is consistent with the notion that the student is building up a mental model of the material, assimilating it into an internal knowledge structure that relates it to the student’s previous knowledge. That assimilation may create tension-between the student’s growing mental model and the structure of the instructional materials. This tension may be manifest as impatience, anticipating information, jumping ahead, bypassing the prepared material, and so on, even if that behaviour is inefficient or counter-productive. “I made . . . mistakes by anticipating instructions and then doing them incorrectly.” “I didn’t revise the tape, but preferred to experiment and explore on my own.” In needing to work things out for themselves, these students (who were not complete novices) may be developing what Petre has observed in experts: operational reasoning, which is often manifest as a need to understand and explore how something works even if such knowledge is not strictly necessary for the task in hand[l3]. If the student is indeed building up a mental model of the material, then the instructional material must both provide useful guidance in palatable form, whether as an overview in the text or embedded in the interface as perceptual cueing or obvious structuring. There must also be readily accessible means (such as reference materials) of verifying uncertain discoveries and correcting misconceptions. “It is worth noting . . . that often new programming successes are achieved almost in the end-hours of adjusting minor details and ultimately not fully understanding achieved the result.”
by chance how you
Given that students are likely to approach instructional materials in unexpected, even haphazard ways, the instructional materials must provide opportunities for the student to re-align his or her experience with the course.
CONCLUSIONS
AND
1MPLICATIONS
The students’ experiences emphasize the importance of assessing the overheads involved in using a particular computer-based tool. When the application was perceived to be inappropriate, cumbersome, or uncomfortable, e.g. for note-taking or reading lengthy text, students avoided using it. Avoidance could provoke extreme behaviours: some students (5124) found Framework sufficiently unpalatable to invest instead in transferring the files into another format in order to work within a different application. The designer of self-instruction material cannot control independent study. Not everything can be anticipated, and the notion of the actizle student suggests that unexpected use of instructional materials is normal, even desirable. A student’s unruliness may contribute to independent learning and to the development of a mental model. The challenge is to facilitate exploration and experiment while ensuring provision of the necessary foundation and protecting students from the pitfalls of their own discoveries. The means for doing so would have to include: provision of a framework on which to build, e.g. the skeleton of a model, a suggested overview, a representation of conceptual structure l provision of an interface amenable to exploration and discovery, one made friendly by clarity. consistency, and the reflection of conceptual structure in perceptual cues, and 0 provision of a means for self-verification, for easy access, e.g. via appropriate reference materials, to the information that can confirm or correct a student’s discoveries. l
The key issue appears to be enabling the student to structure the information, that is, to take control of it. Much attention is given to teaching-more needs to be given to what happens qfler the initial exposure, when the student must make use of the information, must recall it, reconsider it, refer to it, re-arrange it, review it. Only by understanding how the learner goes about using materials and owning information, will we be able to design instructional materials to facilitate the transfer of control from instructional material to learner.
Practical work
31
REFERENCES 1. Jones A. and O’Shea T., Barriers to the use of computer assisted learning. Br. J. Educl Technol. 13, 207-217 (1982). 2. Carroll J., The Nurnberg Funnel: Designing Minimalist Instruction for Practical Computing Skill. The MIT Press, London (1990). 3. Carroll J. M., Mack R. L., Lewis C., Grischkowski N. and Robertson S., Exploring a word processor. HumanComputer Interaction 1, 283-307 (1985). 4. Czaja S. J., Hammond K., Blascovich J. J. and Swede H., Learning to use a word-processing system as a function of training strategy. Behau. Information Technol. 5, 203-216 (1986). 5. Polson P. G. and Kieras D. E., A quantitative model of the learning and performance of text editing knowledge. CHI ‘85, Association for Computing Machinery, New York, pp. 207-212 (1983).
6. Jones A. C.. Bmnirical studies of novices learning programming. Ph.D. thesis No. 10, Centre for Information Technology in Education, Institute of Educational Technology, The Open University (1990). 7. du Boulay B., O’Shea T. and Monk J., The glass box inside the black box: presenting computing concepts to novices. Int. J. Man-Machine Stud. 14, 237-249 (1981). 8. Eisenstadt M., A friendly software environment for psychology students. AISB Q. (1979). 9. Mack R., Lewis C. and Carroll J., Learning to use word processors: problems and prospects. ACM Trans. Ofice Systems 3, 185-204 (1983).
10. Marton F. and Saljo R., On qualitative differences in learning-II. Outcome as a function of the learner’s conception of the task. Br. J. Educl Psychol. 46, 115-127 (1976). Il. Lewis C. and Mack R., The role of abduction in learning to use a computer system. IBM Watson Research Centre Report RC9433, No. 41620, Yorktown Heights, New York (1982). 12. Taylor J., Analysing novices analysing Prolog: what stories do novices tell themselves about Prolog? Insfructional Sci. 19, 283-309 (1990). 13. Petre M., Shifts in reasoning about software and hardware systems: must operational models underpin declarative ones? Invited paper presented at The Third Workshop of the Psychology of Programming Interest Group, Hatfield (January 1991).