- Email: [email protected]
An evaluation format for “open” software tools
Pergamon
Computers in Human Behavior, Vol. 11, No. 3-4, pp. 619~31, 1995 Copyright © 1995 Elsevier Science Ltd
Printed in the USA. All rights reserved 0747-5632/95 $9.50 + .00
0747-5632(95)00046-1
An Evaluation Format for "Open" Software Tools Cheryl A. Murphy West Virginia University
Because higher education is increasingly adopting computer technology, the selection of effective software is imperative for making these investments sound (Bangert-Drowns & Kozma, 1989). In recognition of this, various evaluation techniques and guidelines have been implemented in recent years, in an attempt to sift out poorly designed programs and recognize exemplary computer-based instruction (CBI), computer-aided instruction (CAI), and hypermedia-aided instruction (HA1). Factors such as content, screen layout, use of feedback, length, and appropriate testing methods are but a few of the many characteristics used to evaluate computer programs during this process. However, these evaluative characteristics are inadequate and inappropriate when trying to evaluate software such as authoring programs, instructional tools, or "open" software, all of which are being used more and more in higher education. Authoring programs, allow individuals to create screen layouts, content, and feedback. However, they do not contain these features which are familiar in other types of program; they are only tools for the creation of such features. Therefore, authoring programs cannot be evaluated based on these characteristics. Likewise, instructional tools provide opportunities for students to instruct themselves and to form their own content and links (Bangert-Drowns & Kozma, 1989). Evaluation of these programs based upon the length of the program and program testing methods is impossible and senseless. "Open" software, which is essentially empty of content and is customized by users to fit their own context (McMahon, 1992), cannot be evaluated using the aforementioned characterRequests for reprints should be addressed to Cheryl A. Murphy, West Virginia University, 405-A Allen Hall, Morgantown, W V 26506. E-mail: [email protected] 619
620
Murphy
istics. Therefore, evaluative characteristics that can be applied to these specific and distinct forms of software are needed. To accomplish this form of evaluation, it is necessary to identify which characteristics are the most basic essentials of a software program. These characteristics should encompass nearly all computer software in general and should be based on research and theory. The following characteristics have been identified through the literature as being the most basic essentials of a "good" software program. In support of this reasoning, each characteristic is described and explained in the subsequent sections. 1. GoodDocumentation(Bangert-Drowns&Kozma, 1989;Bergeron, 1990;Cook, 1989; Dudley-Marling, Owston, & Searle, 1988; Godfrey & Sterling, 1982; Hazen, 1987; Hunka, 1989; Schaefermeyer, 1990; Schwier & Misanchuk, 1993; Whiting, 1989) 2. Learner Control (Bangert-Drowns & Kozma, 1989; Freyd, 1989; Hannafin & Peck, 1988; Hazen, 1987; Hunka, 1989; Jonassen & Hannum, 1987; Liu, 1993; Schaefermeyer, 1990; Schwier & Misanchuk, 1993) 3. Branching Capabilities (Bangert-Drowns & Kozma, 1989; Cook, 1989; DudleyMarling, Owston, & Searle, 1988; Hannafin & Peck, 1988; Hunka, 1989; Schwier & Misanchuk, 1993) 4. Portability (Cook, 1989; Freyd, 1989; Hazen, 1987; Hunka, 1989; Liu, 1993; Schwier & Misanchuk, 1993) 5. Ease of Use (Bangert-Drowns & Kozma, 1989; Bergeron, 1990; Cook, 1989; Freyd, 1989; Hazen, 1987; Liu, 1993; MacKnight & Balagopalan, 1989; Schaefermeyer, 1990; Schwier & Misanchuk, 1993; Whiting, 1989) 6. Cost Effectiveness (Cook, 1989; Freyd, 1989; Hannafin & Peck, 1988; Hazen, 1987; Schwier & Misanchuk, 1993) To examine the validity of these characteristics as evaluative tools, six "open" software programs, consisting of authoring languages, instructional tools, and presentation software were evaluated and rated using these characteristics. These same programs were openly evaluated during interviews with four students who have worked with the six programs. These interviews rated the quality of the programs from the student perspective and allowed comparisons between student-identified characteristics used for the assessment of program quality and the six literature-identified characteristics that were used for program assessment. The programs evaluated were Authorware, HyperCard, Action!, SemNet, Inspiration, and Learning Tool. Brief descriptions of each program's documentation, learner control, branching capabilities, portability, ease of use, and cost effectiveness are provided in the following sections on the six literature-based characteristics.
DOCUMENTATION
As Schwier and Misanchuk (1993) have stated, "Most of us have had negative experiences with reference manuals and tutorials received with programs" (p. 139). It is certain that most of us can recall leafing through shoddy manuals or completing worthless tutorials in a fruitless effort to clarify a term or gain an understanding of the program being used. When inadequate documentation is
Evaluating "open" software tools
621
provided, these instances can become agitating and frustrating, often distracting the user from the initial task at hand. To promote the possibility of an enjoyable interaction, documentation should be as user friendly as possible and should allow users to maintain their focus on the task in hand. Documentation should be written in a usable format, be easily understood, and have enough technical information to allow problems to be solved (Schaefermeyer, 1990). Good documentation should also accommodate the various styles of learners by providing sample programs, on-line help, tutorials, and reference and technical manuals (Hazen, 1987). All of the six programs evaluated provide some sort of sample programs and manuals. However, on-line help is available for Authorware, Action!, HyperCard, and Learning Tool only. Inspiration and SemNet do not contain on-line help. Likewise, SemNet does not offer a telephone consultant. Therefore, if the manual is insufficient, the user is potentially helpless when working with these last two programs. Organization of documentation must also be addressed with respect to indices and tables of contents (Bangert-Drowns & Kozma, 1989). All of the evaluated programs contain reference manuals. However, the organization of these differs. Authorware, Action!, and Learning Tool have well-organized manuals. Specific information and commands are easy to find owing to the organization of the table of contents and indices. SemNet, Hypercard, and Inspiration, on the other hand, tend to contain keystrokes for commands throughout the manuals in a haphazard manner with no reference to these commands in the table of contents. This makes searching for specific information or commands more difficult. To make consultation easy, Schwier and Misanchuk (1993) have also identified the use of reference sections and concrete examples as indicators of appropriate documentation. All of the programs evaluated contain examples or sample programs. Some, such as Authorware, are more elaborate and user friendly than others, but all are adequate. In summary, various researchers have argued for the necessity of adequate documentation (Bangert-Drowns & Kozma, 1989; Bergeron, 1990; Cook, 1989; Dudley-Marling, Owston, & Searle, 1988; Godfrey & Sterling, 1982; Hazen, 1987; Hunka, 1989; Schaefermeyer, 1990; Schwier & Misanchuk, 1993; Whiting, 1989). Software programs which contain adequate documentation are far easier for the user in the long run than those programs which lack appropriate documentation.
LEARNER CONTROL The issue of allowing students to control their learning has been heavily debated for years. However, when dealing with the issue of learner control and computer programs, most researchers agree that allowing learner control is a desirable element of a computer program (Bangert-Drowns & Kozma, 1989; Freyd, 1989; Hannafin & Peck, 1988; Hazen, 1987; Hunka, 1989; Jonassen & Hannum, 1987; Liu, 1993; Schaefermeyer, 1990; Schwier & Misanchuk, 1993). Jonassen and Hannum (1987) have argued that "It is both philosophically and pedagogically satisfying to allow the learner to make decisions about the content, method, and style of instruction with which she/he would like to interact" (p. 8). Likewise, Schaefermeyer (1990) has stated that the notion of interactivity is realized when software allows the student to weave an individual educational environment.
622
Murphy
Each of the programs evaluated allows for user control in some way and promotes interactivity. They enable users to produce a personalized educational environment by choosing the content, methods, and style. They also allow users to branch from section to section at their own discretion. Authorware and HyperCard are programs tailored toward learner control. Users can have as much or as little control as they desire, and they may produce programs that allow varying amounts of learner control. A software program should provide resources that allow self-teaching and self-monitoring (Bangert-Drowns & Kozma, 1989) and offer the user the flexibility to create according to personal style (Hunka, 1989; Schwier & Misanchuk, 1993). According to Jonassen and Hannum (1987), giving the learners control is effective. Learning Tool promotes learner control by allowing the user to self-teach and self-monitor progress. Using concept maps created by the users in their various styles, the program creates quizzes for the users which enable them to evaluate progress in a manner that is compatible with their style. SemNet and Inspiration also allow for the user to self-teach through concept mapping. Maps are created by the user, thus allowing the users to produce conceptual images of content in a manner and style with which they are comfortable. To accomplish learner control, computer programs must create a democratic environment or an environment where control of instruction is shifted to the users, permitting them to influence what is learned, how it is learned, or at least the order in which it is learned (Schwier & Misanchuk, 1993). A good computer program should allow for at least one of these instances to occur. Action! permits learner control by letting the user create a presentation of content in the form of slides. These slides may be arranged in any order, and the content contained in the slides is determined by the user. Thus, Action! allows the user to create a presentation that contains the content to be learned and the order in which it is to be learned.
BRANCHING CAPABILITIES With the increasing popularity of hypermedia, the idea of branching has also become an area of increased attention. More and more researchers are proposing that most software should include branching capabilities (Cook, 1989; Schaefermeyer, 1990). All of the programs evaluated contain some form of branching, but the levels of branching differ from program to program. Branching is the single most prominent element that makes the computer a valuable tool for individualized instruction (Schaefermeyer, 1990), and most researchers agree that individualization is good (Bangert-Drowns & Kozma, 1989; Dudley-Marling, Owston, & Searle, 1988; Hannafin & Peck, 1988; Hunka, 1989; Schwier & Misanchuk, 1993). HyperCard is a program constructed to allow for ease in the branching of content by users. Authorware, although it allows for branching in user programming, limits the use of branching capabilities to more proficient users of the program. The program tends to follow a linear format which is somewhat incompatible with branching, but it is possible. Accessing various parts of a program at any given time is an important and necessary feature of software to allow for the individualized development of both
Evaluating "open" software tools
623
the program and the user's knowledge. Learning Tool allows for branching from one portion of the program to another while showing both portions simultaneously. Therefore, the user can see different portions of the program simultaneously at any point throughout the program. SemNet, Action!, and Inspiration also allow the user access to various parts of the program at any given time, but only one part at a time. Although not as extensive as Learning Tool, they still contain efficient branching capabilities.
PORTABILITY It has been stated that software use depends on the availability of hardware (Freyd, 1989). During software evaluation, especially for the purposes of purchasing, one of the first issues that should be addressed is portability. According to Schwier and Misanchuk (1993), portability addresses the following questions: 1. How mobile is the program? 2. What equipment is required? 3. Do compatible systems for the program exist and are they easily accessible? 4. Does the program require a run-time disk? 5. Do you need the program in order to run a program developed on it? Of the six programs evaluated, the only one that can generate a packaged product which can be used on a computer without the program present is Authorware. HyperCard stacks can be run on HyperCard Player. All of the other programs require a copy of the program to be on the machine in order to run a product of that program. This hinders the program's portability and mobility. Liu (1993) has stated it is preferable to obtain a computer program that will work on various platforms (IBM, Mac, etc.), and Cook (1989) has stated it is important to identify whether a program can translate between platforms. Authorware translates easily between Macintosh and IBM platforms, but HyperCard is restricted to Macintosh use only which limits its portability. Learning Tool, SemNet, Action! and Inspiration were only viewed on Macintosh machines. Although IBM versions of these programs may be available, the programs will not translate between platforms. Likewise, portability is affected by the peripherals that are available. Interaction with peripherals is desired (Hazen, 1987; Hunka, 1989; Liu, 1993). However, it should not hinder the portability of the program. All of the programs evaluated work with the printer peripheral, but only HyperCard, Action!, and Authorware work with peripherals such as videodisk players, CDROMs, soundboards, and audio-recorders. Because of the extensive peripherals that may be used, the portability of Authorware, HyperCard, and Action! may become difficult. The memory requirements for the Authorware and Action! programs are quite extensive, thus limiting the portability of these programs to machines with large amounts of RAM and a large hard disk. HyperCard, SemNet, Inspiration, and Learning Tool are all fairly small programs that will fit on most computers, making them somewhat more mobile. The issues of portability essentially center
624
Murphy
on how well a computer program is supported by hardware, whether it runs on various platforms, and whether its support peripherals are also portable.
EASE OF USE Despite the powers and capabilities of a program, if it is too difficult to use, the program is useless to the learner. A minimum amount of time and attention should be required to learn to operate the program (Bangert-Drowns & Kozma, 1989; Bergeron, 1990; Hazen, 1987; Whiting, 1989) and the time spent must be reasonable for the learner (Schaefermeyer, 1990). Learning Tool and Action! contain commands that are intuitive for the user. They use buttons, icons, or commands that are easy to understand, find, and use. Authorware and HyperCard also contain icons and commands, but they are not as intuitive and it may take a little longer for the user to learn how to utilize them. Some of the aids that promote ease of use in a program are the inclusion of templates which provide the user with a preformatted design area (MacKnight & Balagopalan, 1989) and pull-down menus which allow easy access to commands and various sections of the program. All of the programs evaluated contain pulldown menus for easy access to commands. However, Action!, Authorware, and Inspiration are the only programs that contain templates. Other features may include easy access to an exit (Freyd, 1989) or easily accessible peripherals (Cook, 1989; Hazen, 1987; Liu, 1993; Schwier & Misanchuk, 1993). Each of the programs contains an exit that is always accessible in the menu bar, preventing the user from being "trapped" in any portion in the program. As far as peripherals are concerned, Authorware and Action! incorporate the use of peripherals with little effort. HyperCard incorporates peripherals, but it takes some time to learn to do and is not as intuitive as it is in Action! or Authorware. No matter what the aid may be, it is important to recognize that the program's ease of use depends on the quality and amount of program aids that it possesses.
COST EFFECTIVENESS Regardless of the quality of a program, the bottom line when considering purchasing a program is always the cost. The budget restrictions imposed on higher education have created the need to evaluate software for cost effectiveness as well as quality. As far as cost is concerned, Authorware and Action! are expensive programs. However, they are also high-quality programs. Learning Tool was the least expensive program, but its capabilities are not nearly as extensive as SemNet or Inspiration, which cost a bit more. A good rule to follow when purchasing software is for the users to make sure that they are getting what they pay for. Cost effectiveness has been identified as an important dimension in software evaluation (Hazen, 1987; Hannafin & Peck, 1988; Cook, 1989; Freyd, 1989; Schwier & Misanchuk, 1993). Something to consider when evaluating the cost is the hardware that will be required to support the software. Inspiration, SemNet, Learning Tool, and HyperCard have very little requirements of hardware and
Evaluating "open" software tools
625
will run effectively on low-memory equipment. Authorware and Action! require much more memory and higher grades of hardware in order to run properly. Determining the program's suitability for use on a network (if one is available) so that numerous users can have access at the same time may also be important. It is necessary to calculate the projected amount of time that the program will be in use and how many users will need access. Primarily, a potential purchaser should determine the specific needs to be fulfilled and then buy the least expensive program to meets these needs, making sure it supports all of these needs before purchasing.
SOFTWARE RATINGS The individual characteristic ratings and overall ratings of the six programs evaluated can be seen in Table 1 and Table 2, respectively. Authorware received an overall rating of "fairly good". The price of the program (cost effectiveness) is the most negative aspect of the program. The flexibility for the user (learner control) and the manuals and tutorials (documentation) are the most positive aspects. The program Action! received a rating of "very good". The price (cost effectiveness) and memory requirements (portability) are negatives for this program, whereas the flexibility (learner control) and ease of use are the positive aspects. Both Inspiration and SemNet were rated as "good" programs. Compared with Authorware and Action!, these programs received lower ratings because each lacks on-line help (documentation) and contains secret commands (ease of use) which hinder the use of the programs. On the other hand, both programs are flexible (learner control) and allow the user to go from one portion of the program to another easily (branching). Learning Tool received the highest rating of "good to excellent". The drawbacks of the program include the hidden commands that are not intuitive (ease of use) and the inability to translate from one platform to another (portability). The biggest positive of the program is the price (cost effectiveness). A cheaper program with similar scope would be difficult to find. HyperCard received a rating of "good" and is also priced very well for its capabilities (cost effectiveness). It contains extensive branching capabilities and allows flexibility and learner control. Negative features of HyperCard are that the creation of cards is not intuitive (ease of use) and the user must look elsewhere for adequate documentation. It should be noted that, although these programs are rated, they are not compared with one another during this rating process. Rather, each program is rated using the six literature-based characteristics in relation to the specific purposes and abilities it is meant to fulfill. Therefore, a program with a rating of "good" may actually perform more difficult tasks and functions than a program with a rating of "very good". Thus, comparisons between the programs rated in this document should not be made using the ratings listed in Table 2. However, the six literature-based characteristics could be used to compare software programs if the programs were evaluated based on the same purposes and abilities. To conclude this portion of the paper, it should be added that the six software packages were evaluated during the summer of 1993. These programs may have
"0
6 .> t~
g~
.~®
.~®
-~'=~
~
8 ~ ' - - :3 I.L "* .C~
Ck
o.
WlJ.
8
.0
t',O
=~ >~
--"
"-
E
.~,~O
~'~
•- ~
mE
~o~
.- =~1~ 8.
U.~0
~
m r"
~=~
O
O
g ~.-=~
W
E e
oR~,
O
=.-~,
O
~,r"
R,~
.~..~ R o >
> .o .c. oo-_o
"~ ~
~
Ft0
N C ore .O m
O) -~
r-
O;
t-
~ --
t0
_
~_.~
,~ ®
"
~.8
e-"
_
_o:
II1
"e"
E~ ~=~
w<"6._E
~'6E
~o~
oo®
~o
~
~
._
wxo~_ < o.~_
, , q ~ ~>'~, w ~m'm - =
oo®
oo
(,) O
tO
E --I
-1 ~m r-
.9 ~
I'--
e " ,,_r
E
8
~
ooo m
o._~o-~
.~.
~
x ="-
"6 °
E
~
o r=
= 0 .~
~ "-~
z E
"~ ~
,~
<
<
_c
o~
.~
-r
626
. '-"~ ~
~>~
> O ~
5
,~>~>
O .¢- -O
o®
~
"0
~_~ ~o.-=.~
~8
~o
m m
._~
U
~.
-~.-
.I=
~
>,oN
z
=OE
~ ~"o ~
(DE .c: E'~
0-~
~ . . ~ . ".~
o- E ~
o
"~
f~ po
u
t~
~,._o~ .~ O~
~o
o
"0 •
*-~
o
~, e=
•
i:~.
~ ,',.o .c_
"T
~.-
~.~®
0~-~
~ ~~
~.~_
~
I-o
._x~
~
~-
~mm~
~o
~®
~o
~x®
.~ " ~E
.=
Ns '*" ~
E_O=
i,- ~@
g
E o-
~--o ~ ~ o~,~-~ ~> ~ E ~ ~ .~_ ~ o
.~z
E
--
~ c
,.~ O ~
_ ~,~ e- x
~. o _ ~
$ I"- ~
I-'-
O
O ,r-
ID
O
e-
N
8
~
X LU
O O
X
o O
"~
U.
a-
8
>
O
8
8
0
o
"~
2~
8
o
0
>
O e~
~-
0
_.
=
..O m
627
2
628
Murphy
changed extensively since then and the evaluations may not be as relevant to the newer versions.
INTERVIEWS The previous sections of this paper have focused on the evaluation of various software packages with the six literature-based characteristics. In an attempt to confirm the legitimacy of the six characteristics as effective evaluative features, four audio-taped interviews of approximately 30 minutes each were performed with computer-knowledgeable individuals to ascertain the characteristics they use to evaluate "open" software. The individuals were doctoral students majoring in instructional computing. The individuals were not informed of the purpose of the interviews, other than to evaluate specific software programs. The list of the six characteristics found in the literature was not provided or mentioned to the participants in order to avoid influencing the interviews. Three questions were answered for each of the six pieces of software by each individual. These questions were as follows: 1. As a student user of(x) program, what do you like/dislike about it? 2. As a potential purchaser of (x) program, what do you like/dislike about it? 3. Overall, how would you rate (x) program and why? Once the interviews were completed, the audio-tapes were reviewed in search of predominant themes and characteristics identified and used during the evaluation of the six software programs. The findings revealed that all six of the literature-based basic characteristics were identified at one time or another, but some of these characteristics were more predominant than others in the individual evaluations (Table 3). Learner control, ease of use, and cost effectiveness were mentioned by all of the individuals and were the most dominant themes that emerged from the interviews. Positive comments supporting learner control as an evaluative characteristic included statements such as "I liked the fact that it allowed me to create", "I liked to be able to click on an area and be launched to that idea", and "The fact that I am able to program i t . . . is important". Negative comments based on the lack of learner control in a program included "It's not active and it's linear" and "I didn't like the fact that I couldn't manipulate it". Statements that advocated the use of cost effectiveness as an evaluative characteristic could be found in comments such as "It's inexpensive and easy to use. Those would be things I like about it", "In a cost analysis you look at the capabilities as well as the price", "It seems like it could be cheaper", and "The fact that it costs a lot m o r e . . , would be a dislike". Further statements provided by the interviewees that established ease of use as an evaluative characteristic included "I like the fact that it's fairly intuitive", "It's easy enough to use that a lot of people can use it. That's a good quality", "I didn't like that the commands didn't come easy to me", and "I don't like a program if it's not real user friendly". Other comments were "I like it because it's so easy. It uses words you can understand", and "A good thing was that it was pretty easy for me to learn". Branching and portability were mentioned by three of the four individuals, and documentation was mentioned by two individuals during the interviews. In
.-
t-
-~.~
O
~.00
_m . ' ~ - o
_~ ®~.~
e~ m --, ~ . 0 0 ~ 0 ~ ~.
~.. ~ - t'~
n 0 (.5 00 D <
D
C 0
> e~
~
E
W
.- a" m
0
o
~'~
~~
no0
~-Ow
"r
¢J
~o
I~ 0
,', "¢0 c
0 if) 0 0
>,.=_.o ~ , :
W
.-g}
c ~
:3
¢"--
0
~
~.~_ ~ ~-~
W_cOwm Z~
~
E
0
0-~
e-
e-
0
._~
0 D..
~=~o
0~'~-
~<
ED
r-
t-
0
~ ®x~
e-
pE U.
0
E @
.~
._
¢~0
o
~"
N
. -
o ~
'-
~._~
~ o. _>( .~9 _m e n
..a
"0
[
p
r-
~E
e-
"7Z~3
o_
m :~0_~-7
¢0
ot -~
o
t'~ r -
E
~8
-R ~ - ~ o
__1
ou_:E o
~ ~,.~
~wE~S
¢0
c
eO
E
omm
o 0 0
o~ "o"~
ID t-
0 0 e-
0
0
8 t-
0-
O.
n
629
0
c 0 0 OJ
630
Murphy
trying to explain the absence of the documentation characteristic by two of the interviewees, a postinterview discussion with one of the individuals shed some light. He indicated that a "competent" computer user rarely relies upon, or considers, using documentation. It is used only as a last resort. Other interviewees indicated the importance of documentation in statements such as "It had some examples with the package that I saw and I was able to learn from the examples", "I did not like the manual . . . . To be honest I have read it cover to cover and it tells you things you can figure out on your own", and "I wanted to learn more about the program so I had to go to the library and get a book". Portability was an evaluative characteristic mentioned by the interviewees. Comments such as "I liked the fact that you could bundle a program and give it to someone who doesn't have the program to run it", "I dislike the fact that it takes up a lot of memory", and "It's really hard to install and I didn't like that" related to portability. Branching was also supported as an evaluative characteristic during the interviews. Favorable statements were "It was possible to branch to many different screens", "I like it because you can branch off to many different areas from one central point", "I liked it because you can branch out and you can come back to the place you were before", and "There is a way that you can branch out to other p r o g r a m s . . , and I like that a lot". These were all supportive of the characteristic of branching. Other themes which emerged from the interviews that were not a part of the six literature-based basic characteristics were a program's use of color, graphics, and sound as well as the overall usefulness and applicability of the program (Table 3). These characteristics were mentioned by all of the individuals interviewed and appeared to be dominant themes. "Being able to bring in color, graphic, and sound added a lot", "It's good because you can import color and sound", "It was a pretty good program in that you can use color and different graphics", and "The ability to import and export with the program is great and easy because the sound and pictures were right there" were typical comments. Statements such as " Y o u can use it to learn almost anything", "It lends itself to educational courseware design", "I think the program is great because there's an endless list of applications for it", and "I think it would be a wonderful program for the classroom" endorse the themes of applicability and usefulness as evaluative characteristics. The emergence and predominance of these themes may indicate that the characteristics color, graphics, sound, usefulness, and applicability should be added to the list of six basic characteristics identified by the literature.
CONCLUSION The characteristics for the evaluation of computer programs have been described. It should be noted that this list of characteristics is not exhaustive and there are other characteristics not mentioned that may influence various individuals differently in decisions concerning software purchasing. It is proposed, however, that the six characteristics listed (with the possible inclusion of the five identified by the interviewees) are essential parts of any program and an evaluation may be accomplished using these characteristics. After performing these software evaluations and examining the responses of the interviewees, it has become apparent that one should always remember that,
Evaluating "open" software tools
631
n o m a t t e r h o w valuable a software p r o g r a m appears to be w h e n c o m p a r e d w i t h the characteristics above, the true evaluation occurs within the c o m p u t e r lab or c l a s s r o o m where the p r o g r a m is actually used.
REFERENCES Bangert-Drowns, R., & Kozma, R. (1989). Assessing the design of instructional software. Journal of Research on Computing in Education, 21, 241-262. Bergeron, B. (1990). Program instrumentation: A technique for evaluating educational software. Collegiate Microcomputer, 8, 34-46. Cook, E. (1989). The use of Macintosh authoring languages in effective computer-assisted instruction. Journal of Educational Technology, 18, 109-122. Dudley-Marling, C., Owston. R., & Searle, D. (1988). A field-testing approach to software evaluation. Computers in the Schools, 5, 241-249. Freyd, P. (1989). What educators really want in software design. Media and Methods, 25(4), 44; 46-47. Godfrey, D. & Sterling, S. (1982). The elements of CAL. Victoria, BC: Press Porcepic. Hannafin, M., & Peck, K. (1988). The design, development, and education of instructional software. New York: MacMillan. Hazen, M. (1987). Criteria for choosing among instructional software authoring tools. Journal of Research on Computing in Education, 20, 156-164. Hunka, S. (1989). Design guidelines for CAI authoring systems. Educational Technology, 29, 12-17. Jonassen, D. & Hannum, W. (1987). Research-based principles for designing computer software. Educational Technology, 27, 7-14. Liu, M. (1993). Notes taken on lectures. Curriculum and Instruction 380. Summer Session I, West Virginia University. Unpublished manuscript. MacKnight, C. & Balagopalan, S. (1989). Authoring systems: Some instructional implications. Journal of Educational Technology Systems, 17, 123-134. McMahon, H. (1992). "Open" software design: A case study. Educational Technology, 32, 43-55. Schaefermeyer, S. (1990). Standards for instructional computing software design and development. Educational Technology, 30, 9-15. Schwier, R. & Misanchuk, E. (1993). Interactive multimedia instruction. Englewood Cliffs, NJ: Educational Technology Publications. Whiting, J. (1989). An evaluation of some common CAL and CBT authoring styles. Educational and Training Technology International, 26, 186-200.
Computers in Human Behavior, Vol. 11, No. 3-4, pp. 619~31, 1995 Copyright © 1995 Elsevier Science Ltd
Printed in the USA. All rights reserved 0747-5632/95 $9.50 + .00
0747-5632(95)00046-1
An Evaluation Format for "Open" Software Tools Cheryl A. Murphy West Virginia University
Because higher education is increasingly adopting computer technology, the selection of effective software is imperative for making these investments sound (Bangert-Drowns & Kozma, 1989). In recognition of this, various evaluation techniques and guidelines have been implemented in recent years, in an attempt to sift out poorly designed programs and recognize exemplary computer-based instruction (CBI), computer-aided instruction (CAI), and hypermedia-aided instruction (HA1). Factors such as content, screen layout, use of feedback, length, and appropriate testing methods are but a few of the many characteristics used to evaluate computer programs during this process. However, these evaluative characteristics are inadequate and inappropriate when trying to evaluate software such as authoring programs, instructional tools, or "open" software, all of which are being used more and more in higher education. Authoring programs, allow individuals to create screen layouts, content, and feedback. However, they do not contain these features which are familiar in other types of program; they are only tools for the creation of such features. Therefore, authoring programs cannot be evaluated based on these characteristics. Likewise, instructional tools provide opportunities for students to instruct themselves and to form their own content and links (Bangert-Drowns & Kozma, 1989). Evaluation of these programs based upon the length of the program and program testing methods is impossible and senseless. "Open" software, which is essentially empty of content and is customized by users to fit their own context (McMahon, 1992), cannot be evaluated using the aforementioned characterRequests for reprints should be addressed to Cheryl A. Murphy, West Virginia University, 405-A Allen Hall, Morgantown, W V 26506. E-mail: [email protected] 619
620
Murphy
istics. Therefore, evaluative characteristics that can be applied to these specific and distinct forms of software are needed. To accomplish this form of evaluation, it is necessary to identify which characteristics are the most basic essentials of a software program. These characteristics should encompass nearly all computer software in general and should be based on research and theory. The following characteristics have been identified through the literature as being the most basic essentials of a "good" software program. In support of this reasoning, each characteristic is described and explained in the subsequent sections. 1. GoodDocumentation(Bangert-Drowns&Kozma, 1989;Bergeron, 1990;Cook, 1989; Dudley-Marling, Owston, & Searle, 1988; Godfrey & Sterling, 1982; Hazen, 1987; Hunka, 1989; Schaefermeyer, 1990; Schwier & Misanchuk, 1993; Whiting, 1989) 2. Learner Control (Bangert-Drowns & Kozma, 1989; Freyd, 1989; Hannafin & Peck, 1988; Hazen, 1987; Hunka, 1989; Jonassen & Hannum, 1987; Liu, 1993; Schaefermeyer, 1990; Schwier & Misanchuk, 1993) 3. Branching Capabilities (Bangert-Drowns & Kozma, 1989; Cook, 1989; DudleyMarling, Owston, & Searle, 1988; Hannafin & Peck, 1988; Hunka, 1989; Schwier & Misanchuk, 1993) 4. Portability (Cook, 1989; Freyd, 1989; Hazen, 1987; Hunka, 1989; Liu, 1993; Schwier & Misanchuk, 1993) 5. Ease of Use (Bangert-Drowns & Kozma, 1989; Bergeron, 1990; Cook, 1989; Freyd, 1989; Hazen, 1987; Liu, 1993; MacKnight & Balagopalan, 1989; Schaefermeyer, 1990; Schwier & Misanchuk, 1993; Whiting, 1989) 6. Cost Effectiveness (Cook, 1989; Freyd, 1989; Hannafin & Peck, 1988; Hazen, 1987; Schwier & Misanchuk, 1993) To examine the validity of these characteristics as evaluative tools, six "open" software programs, consisting of authoring languages, instructional tools, and presentation software were evaluated and rated using these characteristics. These same programs were openly evaluated during interviews with four students who have worked with the six programs. These interviews rated the quality of the programs from the student perspective and allowed comparisons between student-identified characteristics used for the assessment of program quality and the six literature-identified characteristics that were used for program assessment. The programs evaluated were Authorware, HyperCard, Action!, SemNet, Inspiration, and Learning Tool. Brief descriptions of each program's documentation, learner control, branching capabilities, portability, ease of use, and cost effectiveness are provided in the following sections on the six literature-based characteristics.
DOCUMENTATION
As Schwier and Misanchuk (1993) have stated, "Most of us have had negative experiences with reference manuals and tutorials received with programs" (p. 139). It is certain that most of us can recall leafing through shoddy manuals or completing worthless tutorials in a fruitless effort to clarify a term or gain an understanding of the program being used. When inadequate documentation is
Evaluating "open" software tools
621
provided, these instances can become agitating and frustrating, often distracting the user from the initial task at hand. To promote the possibility of an enjoyable interaction, documentation should be as user friendly as possible and should allow users to maintain their focus on the task in hand. Documentation should be written in a usable format, be easily understood, and have enough technical information to allow problems to be solved (Schaefermeyer, 1990). Good documentation should also accommodate the various styles of learners by providing sample programs, on-line help, tutorials, and reference and technical manuals (Hazen, 1987). All of the six programs evaluated provide some sort of sample programs and manuals. However, on-line help is available for Authorware, Action!, HyperCard, and Learning Tool only. Inspiration and SemNet do not contain on-line help. Likewise, SemNet does not offer a telephone consultant. Therefore, if the manual is insufficient, the user is potentially helpless when working with these last two programs. Organization of documentation must also be addressed with respect to indices and tables of contents (Bangert-Drowns & Kozma, 1989). All of the evaluated programs contain reference manuals. However, the organization of these differs. Authorware, Action!, and Learning Tool have well-organized manuals. Specific information and commands are easy to find owing to the organization of the table of contents and indices. SemNet, Hypercard, and Inspiration, on the other hand, tend to contain keystrokes for commands throughout the manuals in a haphazard manner with no reference to these commands in the table of contents. This makes searching for specific information or commands more difficult. To make consultation easy, Schwier and Misanchuk (1993) have also identified the use of reference sections and concrete examples as indicators of appropriate documentation. All of the programs evaluated contain examples or sample programs. Some, such as Authorware, are more elaborate and user friendly than others, but all are adequate. In summary, various researchers have argued for the necessity of adequate documentation (Bangert-Drowns & Kozma, 1989; Bergeron, 1990; Cook, 1989; Dudley-Marling, Owston, & Searle, 1988; Godfrey & Sterling, 1982; Hazen, 1987; Hunka, 1989; Schaefermeyer, 1990; Schwier & Misanchuk, 1993; Whiting, 1989). Software programs which contain adequate documentation are far easier for the user in the long run than those programs which lack appropriate documentation.
LEARNER CONTROL The issue of allowing students to control their learning has been heavily debated for years. However, when dealing with the issue of learner control and computer programs, most researchers agree that allowing learner control is a desirable element of a computer program (Bangert-Drowns & Kozma, 1989; Freyd, 1989; Hannafin & Peck, 1988; Hazen, 1987; Hunka, 1989; Jonassen & Hannum, 1987; Liu, 1993; Schaefermeyer, 1990; Schwier & Misanchuk, 1993). Jonassen and Hannum (1987) have argued that "It is both philosophically and pedagogically satisfying to allow the learner to make decisions about the content, method, and style of instruction with which she/he would like to interact" (p. 8). Likewise, Schaefermeyer (1990) has stated that the notion of interactivity is realized when software allows the student to weave an individual educational environment.
622
Murphy
Each of the programs evaluated allows for user control in some way and promotes interactivity. They enable users to produce a personalized educational environment by choosing the content, methods, and style. They also allow users to branch from section to section at their own discretion. Authorware and HyperCard are programs tailored toward learner control. Users can have as much or as little control as they desire, and they may produce programs that allow varying amounts of learner control. A software program should provide resources that allow self-teaching and self-monitoring (Bangert-Drowns & Kozma, 1989) and offer the user the flexibility to create according to personal style (Hunka, 1989; Schwier & Misanchuk, 1993). According to Jonassen and Hannum (1987), giving the learners control is effective. Learning Tool promotes learner control by allowing the user to self-teach and self-monitor progress. Using concept maps created by the users in their various styles, the program creates quizzes for the users which enable them to evaluate progress in a manner that is compatible with their style. SemNet and Inspiration also allow for the user to self-teach through concept mapping. Maps are created by the user, thus allowing the users to produce conceptual images of content in a manner and style with which they are comfortable. To accomplish learner control, computer programs must create a democratic environment or an environment where control of instruction is shifted to the users, permitting them to influence what is learned, how it is learned, or at least the order in which it is learned (Schwier & Misanchuk, 1993). A good computer program should allow for at least one of these instances to occur. Action! permits learner control by letting the user create a presentation of content in the form of slides. These slides may be arranged in any order, and the content contained in the slides is determined by the user. Thus, Action! allows the user to create a presentation that contains the content to be learned and the order in which it is to be learned.
BRANCHING CAPABILITIES With the increasing popularity of hypermedia, the idea of branching has also become an area of increased attention. More and more researchers are proposing that most software should include branching capabilities (Cook, 1989; Schaefermeyer, 1990). All of the programs evaluated contain some form of branching, but the levels of branching differ from program to program. Branching is the single most prominent element that makes the computer a valuable tool for individualized instruction (Schaefermeyer, 1990), and most researchers agree that individualization is good (Bangert-Drowns & Kozma, 1989; Dudley-Marling, Owston, & Searle, 1988; Hannafin & Peck, 1988; Hunka, 1989; Schwier & Misanchuk, 1993). HyperCard is a program constructed to allow for ease in the branching of content by users. Authorware, although it allows for branching in user programming, limits the use of branching capabilities to more proficient users of the program. The program tends to follow a linear format which is somewhat incompatible with branching, but it is possible. Accessing various parts of a program at any given time is an important and necessary feature of software to allow for the individualized development of both
Evaluating "open" software tools
623
the program and the user's knowledge. Learning Tool allows for branching from one portion of the program to another while showing both portions simultaneously. Therefore, the user can see different portions of the program simultaneously at any point throughout the program. SemNet, Action!, and Inspiration also allow the user access to various parts of the program at any given time, but only one part at a time. Although not as extensive as Learning Tool, they still contain efficient branching capabilities.
PORTABILITY It has been stated that software use depends on the availability of hardware (Freyd, 1989). During software evaluation, especially for the purposes of purchasing, one of the first issues that should be addressed is portability. According to Schwier and Misanchuk (1993), portability addresses the following questions: 1. How mobile is the program? 2. What equipment is required? 3. Do compatible systems for the program exist and are they easily accessible? 4. Does the program require a run-time disk? 5. Do you need the program in order to run a program developed on it? Of the six programs evaluated, the only one that can generate a packaged product which can be used on a computer without the program present is Authorware. HyperCard stacks can be run on HyperCard Player. All of the other programs require a copy of the program to be on the machine in order to run a product of that program. This hinders the program's portability and mobility. Liu (1993) has stated it is preferable to obtain a computer program that will work on various platforms (IBM, Mac, etc.), and Cook (1989) has stated it is important to identify whether a program can translate between platforms. Authorware translates easily between Macintosh and IBM platforms, but HyperCard is restricted to Macintosh use only which limits its portability. Learning Tool, SemNet, Action! and Inspiration were only viewed on Macintosh machines. Although IBM versions of these programs may be available, the programs will not translate between platforms. Likewise, portability is affected by the peripherals that are available. Interaction with peripherals is desired (Hazen, 1987; Hunka, 1989; Liu, 1993). However, it should not hinder the portability of the program. All of the programs evaluated work with the printer peripheral, but only HyperCard, Action!, and Authorware work with peripherals such as videodisk players, CDROMs, soundboards, and audio-recorders. Because of the extensive peripherals that may be used, the portability of Authorware, HyperCard, and Action! may become difficult. The memory requirements for the Authorware and Action! programs are quite extensive, thus limiting the portability of these programs to machines with large amounts of RAM and a large hard disk. HyperCard, SemNet, Inspiration, and Learning Tool are all fairly small programs that will fit on most computers, making them somewhat more mobile. The issues of portability essentially center
624
Murphy
on how well a computer program is supported by hardware, whether it runs on various platforms, and whether its support peripherals are also portable.
EASE OF USE Despite the powers and capabilities of a program, if it is too difficult to use, the program is useless to the learner. A minimum amount of time and attention should be required to learn to operate the program (Bangert-Drowns & Kozma, 1989; Bergeron, 1990; Hazen, 1987; Whiting, 1989) and the time spent must be reasonable for the learner (Schaefermeyer, 1990). Learning Tool and Action! contain commands that are intuitive for the user. They use buttons, icons, or commands that are easy to understand, find, and use. Authorware and HyperCard also contain icons and commands, but they are not as intuitive and it may take a little longer for the user to learn how to utilize them. Some of the aids that promote ease of use in a program are the inclusion of templates which provide the user with a preformatted design area (MacKnight & Balagopalan, 1989) and pull-down menus which allow easy access to commands and various sections of the program. All of the programs evaluated contain pulldown menus for easy access to commands. However, Action!, Authorware, and Inspiration are the only programs that contain templates. Other features may include easy access to an exit (Freyd, 1989) or easily accessible peripherals (Cook, 1989; Hazen, 1987; Liu, 1993; Schwier & Misanchuk, 1993). Each of the programs contains an exit that is always accessible in the menu bar, preventing the user from being "trapped" in any portion in the program. As far as peripherals are concerned, Authorware and Action! incorporate the use of peripherals with little effort. HyperCard incorporates peripherals, but it takes some time to learn to do and is not as intuitive as it is in Action! or Authorware. No matter what the aid may be, it is important to recognize that the program's ease of use depends on the quality and amount of program aids that it possesses.
COST EFFECTIVENESS Regardless of the quality of a program, the bottom line when considering purchasing a program is always the cost. The budget restrictions imposed on higher education have created the need to evaluate software for cost effectiveness as well as quality. As far as cost is concerned, Authorware and Action! are expensive programs. However, they are also high-quality programs. Learning Tool was the least expensive program, but its capabilities are not nearly as extensive as SemNet or Inspiration, which cost a bit more. A good rule to follow when purchasing software is for the users to make sure that they are getting what they pay for. Cost effectiveness has been identified as an important dimension in software evaluation (Hazen, 1987; Hannafin & Peck, 1988; Cook, 1989; Freyd, 1989; Schwier & Misanchuk, 1993). Something to consider when evaluating the cost is the hardware that will be required to support the software. Inspiration, SemNet, Learning Tool, and HyperCard have very little requirements of hardware and
Evaluating "open" software tools
625
will run effectively on low-memory equipment. Authorware and Action! require much more memory and higher grades of hardware in order to run properly. Determining the program's suitability for use on a network (if one is available) so that numerous users can have access at the same time may also be important. It is necessary to calculate the projected amount of time that the program will be in use and how many users will need access. Primarily, a potential purchaser should determine the specific needs to be fulfilled and then buy the least expensive program to meets these needs, making sure it supports all of these needs before purchasing.
SOFTWARE RATINGS The individual characteristic ratings and overall ratings of the six programs evaluated can be seen in Table 1 and Table 2, respectively. Authorware received an overall rating of "fairly good". The price of the program (cost effectiveness) is the most negative aspect of the program. The flexibility for the user (learner control) and the manuals and tutorials (documentation) are the most positive aspects. The program Action! received a rating of "very good". The price (cost effectiveness) and memory requirements (portability) are negatives for this program, whereas the flexibility (learner control) and ease of use are the positive aspects. Both Inspiration and SemNet were rated as "good" programs. Compared with Authorware and Action!, these programs received lower ratings because each lacks on-line help (documentation) and contains secret commands (ease of use) which hinder the use of the programs. On the other hand, both programs are flexible (learner control) and allow the user to go from one portion of the program to another easily (branching). Learning Tool received the highest rating of "good to excellent". The drawbacks of the program include the hidden commands that are not intuitive (ease of use) and the inability to translate from one platform to another (portability). The biggest positive of the program is the price (cost effectiveness). A cheaper program with similar scope would be difficult to find. HyperCard received a rating of "good" and is also priced very well for its capabilities (cost effectiveness). It contains extensive branching capabilities and allows flexibility and learner control. Negative features of HyperCard are that the creation of cards is not intuitive (ease of use) and the user must look elsewhere for adequate documentation. It should be noted that, although these programs are rated, they are not compared with one another during this rating process. Rather, each program is rated using the six literature-based characteristics in relation to the specific purposes and abilities it is meant to fulfill. Therefore, a program with a rating of "good" may actually perform more difficult tasks and functions than a program with a rating of "very good". Thus, comparisons between the programs rated in this document should not be made using the ratings listed in Table 2. However, the six literature-based characteristics could be used to compare software programs if the programs were evaluated based on the same purposes and abilities. To conclude this portion of the paper, it should be added that the six software packages were evaluated during the summer of 1993. These programs may have
"0
6 .> t~
g~
.~®
.~®
-~'=~
~
8 ~ ' - - :3 I.L "* .C~
Ck
o.
WlJ.
8
.0
t',O
=~ >~
--"
"-
E
.~,~O
~'~
•- ~
mE
~o~
.- =~1~ 8.
U.~0
~
m r"
~=~
O
O
g ~.-=~
W
E e
oR~,
O
=.-~,
O
~,r"
R,~
.~..~ R o >
> .o .c. oo-_o
"~ ~
~
Ft0
N C ore .O m
O) -~
r-
O;
t-
~ --
t0
_
~_.~
,~ ®
"
~.8
e-"
_
_o:
II1
"e"
E~ ~=~
w<"6._E
~'6E
~o~
oo®
~o
~
~
._
wxo~_ < o.~_
, , q ~ ~>'~, w ~m'm - =
oo®
oo
(,) O
tO
E --I
-1 ~m r-
.9 ~
I'--
e " ,,_r
E
8
~
ooo m
o._~o-~
.~.
~
x ="-
"6 °
E
~
o r=
= 0 .~
~ "-~
z E
"~ ~
,~
<
<
_c
o~
.~
-r
626
. '-"~ ~
~>~
> O ~
5
,~>~>
O .¢- -O
o®
~
"0
~_~ ~o.-=.~
~8
~o
m m
._~
U
~.
-~.-
.I=
~
>,oN
z
=OE
~ ~"o ~
(DE .c: E'~
0-~
~ . . ~ . ".~
o- E ~
o
"~
f~ po
u
t~
~,._o~ .~ O~
~o
o
"0 •
*-~
o
~, e=
•
i:~.
~ ,',.o .c_
"T
~.-
~.~®
0~-~
~ ~~
~.~_
~
I-o
._x~
~
~-
~mm~
~o
~®
~o
~x®
.~ " ~E
.=
Ns '*" ~
E_O=
i,- ~@
g
E o-
~--o ~ ~ o~,~-~ ~> ~ E ~ ~ .~_ ~ o
.~z
E
--
~ c
,.~ O ~
_ ~,~ e- x
~. o _ ~
$ I"- ~
I-'-
O
O ,r-
ID
O
e-
N
8
~
X LU
O O
X
o O
"~
U.
a-
8
>
O
8
8
0
o
"~
2~
8
o
0
>
O e~
~-
0
_.
=
..O m
627
2
628
Murphy
changed extensively since then and the evaluations may not be as relevant to the newer versions.
INTERVIEWS The previous sections of this paper have focused on the evaluation of various software packages with the six literature-based characteristics. In an attempt to confirm the legitimacy of the six characteristics as effective evaluative features, four audio-taped interviews of approximately 30 minutes each were performed with computer-knowledgeable individuals to ascertain the characteristics they use to evaluate "open" software. The individuals were doctoral students majoring in instructional computing. The individuals were not informed of the purpose of the interviews, other than to evaluate specific software programs. The list of the six characteristics found in the literature was not provided or mentioned to the participants in order to avoid influencing the interviews. Three questions were answered for each of the six pieces of software by each individual. These questions were as follows: 1. As a student user of(x) program, what do you like/dislike about it? 2. As a potential purchaser of (x) program, what do you like/dislike about it? 3. Overall, how would you rate (x) program and why? Once the interviews were completed, the audio-tapes were reviewed in search of predominant themes and characteristics identified and used during the evaluation of the six software programs. The findings revealed that all six of the literature-based basic characteristics were identified at one time or another, but some of these characteristics were more predominant than others in the individual evaluations (Table 3). Learner control, ease of use, and cost effectiveness were mentioned by all of the individuals and were the most dominant themes that emerged from the interviews. Positive comments supporting learner control as an evaluative characteristic included statements such as "I liked the fact that it allowed me to create", "I liked to be able to click on an area and be launched to that idea", and "The fact that I am able to program i t . . . is important". Negative comments based on the lack of learner control in a program included "It's not active and it's linear" and "I didn't like the fact that I couldn't manipulate it". Statements that advocated the use of cost effectiveness as an evaluative characteristic could be found in comments such as "It's inexpensive and easy to use. Those would be things I like about it", "In a cost analysis you look at the capabilities as well as the price", "It seems like it could be cheaper", and "The fact that it costs a lot m o r e . . , would be a dislike". Further statements provided by the interviewees that established ease of use as an evaluative characteristic included "I like the fact that it's fairly intuitive", "It's easy enough to use that a lot of people can use it. That's a good quality", "I didn't like that the commands didn't come easy to me", and "I don't like a program if it's not real user friendly". Other comments were "I like it because it's so easy. It uses words you can understand", and "A good thing was that it was pretty easy for me to learn". Branching and portability were mentioned by three of the four individuals, and documentation was mentioned by two individuals during the interviews. In
.-
t-
-~.~
O
~.00
_m . ' ~ - o
_~ ®~.~
e~ m --, ~ . 0 0 ~ 0 ~ ~.
~.. ~ - t'~
n 0 (.5 00 D <
D
C 0
> e~
~
E
W
.- a" m
0
o
~'~
~~
no0
~-Ow
"r
¢J
~o
I~ 0
,', "¢0 c
0 if) 0 0
>,.=_.o ~ , :
W
.-g}
c ~
:3
¢"--
0
~
~.~_ ~ ~-~
W_cOwm Z~
~
E
0
0-~
e-
e-
0
._~
0 D..
~=~o
0~'~-
~<
ED
r-
t-
0
~ ®x~
e-
pE U.
0
E @
.~
._
¢~0
o
~"
N
. -
o ~
'-
~._~
~ o. _>( .~9 _m e n
..a
"0
[
p
r-
~E
e-
"7Z~3
o_
m :~0_~-7
¢0
ot -~
o
t'~ r -
E
~8
-R ~ - ~ o
__1
ou_:E o
~ ~,.~
~wE~S
¢0
c
eO
E
omm
o 0 0
o~ "o"~
ID t-
0 0 e-
0
0
8 t-
0-
O.
n
629
0
c 0 0 OJ
630
Murphy
trying to explain the absence of the documentation characteristic by two of the interviewees, a postinterview discussion with one of the individuals shed some light. He indicated that a "competent" computer user rarely relies upon, or considers, using documentation. It is used only as a last resort. Other interviewees indicated the importance of documentation in statements such as "It had some examples with the package that I saw and I was able to learn from the examples", "I did not like the manual . . . . To be honest I have read it cover to cover and it tells you things you can figure out on your own", and "I wanted to learn more about the program so I had to go to the library and get a book". Portability was an evaluative characteristic mentioned by the interviewees. Comments such as "I liked the fact that you could bundle a program and give it to someone who doesn't have the program to run it", "I dislike the fact that it takes up a lot of memory", and "It's really hard to install and I didn't like that" related to portability. Branching was also supported as an evaluative characteristic during the interviews. Favorable statements were "It was possible to branch to many different screens", "I like it because you can branch off to many different areas from one central point", "I liked it because you can branch out and you can come back to the place you were before", and "There is a way that you can branch out to other p r o g r a m s . . , and I like that a lot". These were all supportive of the characteristic of branching. Other themes which emerged from the interviews that were not a part of the six literature-based basic characteristics were a program's use of color, graphics, and sound as well as the overall usefulness and applicability of the program (Table 3). These characteristics were mentioned by all of the individuals interviewed and appeared to be dominant themes. "Being able to bring in color, graphic, and sound added a lot", "It's good because you can import color and sound", "It was a pretty good program in that you can use color and different graphics", and "The ability to import and export with the program is great and easy because the sound and pictures were right there" were typical comments. Statements such as " Y o u can use it to learn almost anything", "It lends itself to educational courseware design", "I think the program is great because there's an endless list of applications for it", and "I think it would be a wonderful program for the classroom" endorse the themes of applicability and usefulness as evaluative characteristics. The emergence and predominance of these themes may indicate that the characteristics color, graphics, sound, usefulness, and applicability should be added to the list of six basic characteristics identified by the literature.
CONCLUSION The characteristics for the evaluation of computer programs have been described. It should be noted that this list of characteristics is not exhaustive and there are other characteristics not mentioned that may influence various individuals differently in decisions concerning software purchasing. It is proposed, however, that the six characteristics listed (with the possible inclusion of the five identified by the interviewees) are essential parts of any program and an evaluation may be accomplished using these characteristics. After performing these software evaluations and examining the responses of the interviewees, it has become apparent that one should always remember that,
Evaluating "open" software tools
631
n o m a t t e r h o w valuable a software p r o g r a m appears to be w h e n c o m p a r e d w i t h the characteristics above, the true evaluation occurs within the c o m p u t e r lab or c l a s s r o o m where the p r o g r a m is actually used.
REFERENCES Bangert-Drowns, R., & Kozma, R. (1989). Assessing the design of instructional software. Journal of Research on Computing in Education, 21, 241-262. Bergeron, B. (1990). Program instrumentation: A technique for evaluating educational software. Collegiate Microcomputer, 8, 34-46. Cook, E. (1989). The use of Macintosh authoring languages in effective computer-assisted instruction. Journal of Educational Technology, 18, 109-122. Dudley-Marling, C., Owston. R., & Searle, D. (1988). A field-testing approach to software evaluation. Computers in the Schools, 5, 241-249. Freyd, P. (1989). What educators really want in software design. Media and Methods, 25(4), 44; 46-47. Godfrey, D. & Sterling, S. (1982). The elements of CAL. Victoria, BC: Press Porcepic. Hannafin, M., & Peck, K. (1988). The design, development, and education of instructional software. New York: MacMillan. Hazen, M. (1987). Criteria for choosing among instructional software authoring tools. Journal of Research on Computing in Education, 20, 156-164. Hunka, S. (1989). Design guidelines for CAI authoring systems. Educational Technology, 29, 12-17. Jonassen, D. & Hannum, W. (1987). Research-based principles for designing computer software. Educational Technology, 27, 7-14. Liu, M. (1993). Notes taken on lectures. Curriculum and Instruction 380. Summer Session I, West Virginia University. Unpublished manuscript. MacKnight, C. & Balagopalan, S. (1989). Authoring systems: Some instructional implications. Journal of Educational Technology Systems, 17, 123-134. McMahon, H. (1992). "Open" software design: A case study. Educational Technology, 32, 43-55. Schaefermeyer, S. (1990). Standards for instructional computing software design and development. Educational Technology, 30, 9-15. Schwier, R. & Misanchuk, E. (1993). Interactive multimedia instruction. Englewood Cliffs, NJ: Educational Technology Publications. Whiting, J. (1989). An evaluation of some common CAL and CBT authoring styles. Educational and Training Technology International, 26, 186-200.