Development of diagnostic tests for computer literacy

0360- I3 I s/90 s3.00 + 0.00 Copyright 8 1990 Pergamon Press plc

Compurers E~UC.Vol. 14. No. 3. pp. 213-216, 1990 Primed in Great Britam. All rights reserved

DEVELOPMENT OF DIAGNOSTIC TESTS FOR COMPUTER LITERACY FRED PYFKZAK School of Education, Department of Educational Foundations, California State University, Los Angeles, 5151 State University Drive, Los Angeles, CA 90032, U.S.A. (Received 2 June 1989; revision receiced 11 July 1989)

Abstract-A set of 16 diagnostic tests for computer literacy was developed based upon a content analysis of 22 computer-literacy curriculum guides. Objectives for the tests and the test items were reviewed for accuracy and appropriateness by 35 educators.

Numerous definitions of “computer literacy” have been offered and often include reference to (1) using application programs, (2) the social context of computer usage, (3) understanding what a computer is and how it works, (4) the history of computers, and (5) a working knowledge of at least one high-level programming language. Whether a school district is teaching computer literacy in a separate course or is integrating computer literacy instruction throughout its curricula, it needs tests in order to determine not only the general level of students’ computer literacy but also to diagnose areas of student strength and weakness, which will enable appropriate assignment to learning activities, courses, etc. An early effort to develop computer literacy tests was undertaken by the Northwest Regional Educational Laboratory for the Department of Defense Dependents Schools[l]. They developed three tests, one each for grades 4, 7 and 11. Each is a survey test containing multiple-choice items that cover “computer literacy” in a broad sense. These are useful for obtaining a general overview of the levels of students’ skills, but are not designed to provide diagnostic information. The purpose of the present study was to develop a set of diagnostic tests that would pinpoint areas of strength and weakness in minimal computer literacy skills and would be appropriate for administration to students in any grade from five through college. PROCEDURE

FOR IDENTIFYING

THE

OBJECTIVES

The first problem was to identify the objectives to be covered by the tests. These objectives would serve as the definition of computer literacy for this project. A random sample of 125 California school districts was identified using the schools listed in the California Public Schools Directory as the population. The random process was restricted so that at least one district was selected from each county in the state. A questionnaire on local computer literacy curricula was mailed to each of the 125 chief district administrations with the request that it be forwarded to the appropriate person in his or her district. Fifty-six (44.8%) responses were obtained. Of these, 32 (57.1%) indicated that their districts use locally developed computer literacy curriculum guides. Of those who do not, 5 (8.9%) reported using externally developed guides; the remainder did not answer the question. To determine the need for the project, the questionnaire included a description of the proposed tests followed by a question asking whether the respondent wished to receive a copy of the tests for use in his or her district. Fifty (89.3%) answered “yes”. Since the first draft of the tests and the objectives underlying them would need to be reviewed, subjects were asked whether they or someone in their district would be willing to review the preliminary draft. Forty-one (73.2%) answered “yes”. Follow-up letters soliciting copies of both the locally and externally developed curriculum guides mentioned above resulted in the author receiving 22 guides. Since there were many hundreds of objectives in the guides and since they varied in wording. specificity and organization, considerable subjectivity was required in order to produce a synthesis 213

FRED PYRCZAK

214

on which to base the tests. In producing the synthesis, the author kept in mind the need to have each objective be broad enough to permit the development of a test with at least ten test items on it since a test with fewer items would be of questionable reliability. Furthermore, each objective needed to be fairly broad so that the final list of objectives would be reasonably short. Otherwise, since there was to be one test for each objective, more tests would be developed than one could use with students. The procedure for producing the synthesis was to start with the curriculum guide that, in this writer’s opinion, had the largest number of objectives that were sufficiently broad for the purposes of this project. These objectives were typed on a master list. Each of the other guides was examined for objectives that were not included on this list; each such objective was added to the list. Finally the frequency with which each objecttive was included in the entire set of guides was tallied. Objectives that were included in at least five guides were retained for use in this project. The only exception to this procedure involved the objectives on programming. If they were included, an entire set of diagnostic tests just on this topic would need to be written; as a result these objectives were deleted with the intention of developing such a set of tests at a later date. An objective on knowledge abour programming languages was retained, however.

PROCEDURE

FOR ITEM DEVELOPMENT

AND REFINEMENT

OF OBJECTIVES

Multiple-choice items were written by the author. Since some of the curriculum guides provided listings of content as well as objectives, all guides were reviewed again for content. For each item of content, a multiple-choice item was written. Some of the resulting test items did not clearly belong under any of the objectives in this author’s master list of objectives. For those items that did not fit but seemed important, some objectives were reworded to make them somewhat broader so that these items would have homes. The process described above resulted in less than ten items per objective. This problem was resolved, in part, by consulting computer literacy textbooks and other reference materials for additional content to be covered under each objective. Additional items were written, but it was still found that the content underlying some objectives was too “thin” to yield ten test items each. As a result, some related objectives were merged with each other. Table 1 shows the resulting set of I6 objectives arranged into seven groups.

I.

Table

I

GENERAL

PROCEDURES

FOR

USING

Computer

(A)

Identifies

correct

(B)

Identifies

the meamngs

of symbols

on the keyboard

(C) HOW

Identifies

simple

commands

and commands

(A)

Identifies

functions

of indiwdual

Identifies (B) 111 COMPUTER

functions

of major

II.

(A)

and shut-down

system

COMPUTERS

objectives

procedures

as well and

EQUIPMENT

as ptcoedures

functions

in BASIC

and

for handling

of spectal

and

purpose

storage

of disks

keys

LOGO

WORK pieces of hardware

components

inside

in a computer

a computer,

system,

including

mcluding

their

roles rn input,

the roles of the binary

number

proccssmg, system

and output

and

software

APPLICATIONS

ldenttfies applied

(B)

start-up

literacy

AND RELATED

COMPUTERS

the purposes arts.

Identifies

and charactcrtsrtcs

of major

types of software

as well

as specific

applications

in business,

education,

the

etc.

problems

that are approprime

and mappropriatc

for solution

by computer

as well

as advantages

and disadvantages

of

usmg computers IV

KNOWLEDGE

OF

PROGRAMMING

Identrfies charactertstics (A) V COMPUTER HISTORY (A)

ldentrfics

early

(B)

Identifies

gencrattons

calculatmg

0

Identrhes

names of investors

dates and countries

VI. COMPUTERS (A) (B) VII.

AND

Identifies Identifies

The students

will

and the work

place

dewces.

of computers,

programming

languages

and distinctions

between

then

functions

and approximate

data

thetr

features,

ditTerences,

who contributed

names

to the development

identify computer

(A)

computer

(B)

software

0

computer-related

of modern

languages

and computer

languages

of introduction and

computers;

approximate identifies

dates of introduction

their contributions,

approxtmate

SOCIETY

ethical and legal issues mvolving computer use computer-related careers, including job requirements,

with

of specific computers

human

of origin

in the work place COMPUTER VOCABULARY

press, mass-market

LANGUAGES

of popular

FOR

the meanings magazine

respect

COMMUNICATING of technical

articles,

terms

computer

to each of the following:

hardware and programmmg acronyms

compensation

ABOUT frequently

and typical

job duties,

as well

as effect of computers

COMPUTERS

used in communicating

hardware and software

advertisements,

about

computers

in articles

in the popular

as well as in oral communications

in school

Development

of diagnostic tests for computer

literacy

215

Due to limited resources and lime constraints, it was not possible to have others review the synthesis of objectives in order to establish the interrater reliability of the process for selecting objectives. The objectives were reviewed, however, for appropriateness as described below. PROCEDURE

FOR

REVIEWING

OBJECTIVES

AND

TEST

ITEMS

The next step was to send out the preliminary draft of the tests for review by respondents to the questionnaire who had indicated a willingness to review them. The tests for each of the seven groups of objectives were sent to five reviewers. Different reviewers were used for each group. Thus, a total of 35 reviewers were sent tests. Each reviewer was also sent a complete list of the objectives and asked to comment on them. It would have been desirable to have had the master list of objectives reviewed before the test items were written. However, limited resources did not permit a separate mailing for this activity. Twenty reviews were received. At least one review for each of the seven sets of tests was received. The reviews of the objectives were quite favorable. Only minor editorial changes were made based upon the reviewers’ comments. Reviews of the tests were also highly favorable. However, the reviewers pointed out a number of technical flaws in specific items. On the basis of these comments, the items were revised. The result was a set of 16 tests with ten multiple-choice items each. Table 2 shows one sample item for each of the 16 tests. DISCUSSION Because the tests are based upon a survey of a number of computer-literacy curriculum guides, the tests are likely to be applicable in a wide variety of settings. It is important to note that the set of tests may be easily customized for use in particular settings since there is a separate test on each of the 16 objectives. In a setting where computer history, for example, is not emphasized, the test on this topic may simply be deleted from the set. Table I-A

When

writing

(A)

a ball

(B)

a pencil

on the label

point

pen to make

so that

a felt-tip

0

any of the above If only

pen to avold

WIII appear

the computer‘s

(C)

a listmg

of the program

@I

nothing

will

memory

the program

(B)

runs over a program

(C)

presents

the instructions

0)

operates

the program

cathode

(B)

modem

w

mouse

(D)

light

II-B.

The

m UK wll

in a program a~ twcc

translates

speed

computer

signals

into

telephone

tones and

back?

ray rube

pen

central

processing

umt contains unit and external

(B)

unit and control

(Cl

the disk drive

and mam

memory unit

memory

all of the above Data

base software

would

k

likely

to bt used by a large

busrnrss to do which

of the following?

type, edit and print form ictters to clients file a list of names and addrcsvs of customers

CC)

perform

(D)

all of the above

III-B.

on the screen

its normal

the arithmetic/logic

(A) (B)

the computer

and erases it

the arithmetic/logic

XII-A.

happpcn?

IIS instructions

(A)

0

will

be produced

IS gwcn.

and follows

of the following

(A)

whar

WIII bc cleared

to run a program

loads

Which

using a program.

on the screen

happen

(A)

II-A.

copy

iT necessary

key 1s pressed while

message

If the command

for objcctwes

use

damage

(B)

14

a permanent

test items

one should

arc O.K.

the CONTROL

a control

(A)

2. Sample

IO a diskette,

it can be erased

(C)

I-B.

attached

A person

of a computer

a large

plans

number

of calculations

to buy a microcomputer

on data

in order

to store a collection

of twenty

recipes.

Which

of the following

is a disadvantage

in this situation?

(A)

the cost would

(B)

it would

(Q

the computer

(D)

all of the above

take

be high in relation more and

time

to learn

peripherals

to the benefits how

would

to use the computer take

up more

than

space than

to file the recipes

a card

by

hand

file

continued oculeqf

216

FRED F’YRCZAK Table 2-conrinwd

IV-A. Which of the followmg is most sunilar to human languages? high-level languages (A) low-level languages (B) assembly languages (C) machine language (D) V-A. Which machine earned its mventor the tillc of “Father of Computers7” Napier’s bones (A) punched card loom (B) tabulating machine ((2 analytic engine (D) V-B Mxrocomputers were first offered for sale to the pubhc m the 1940s (A) 1950s (B) 1960s (Cl 1970s (D) V-C. Who invented rhc punched card loom? Charles Babbage (A) Herman Hollerith (B) Joseph Jacquard 0 B&e Pascal (D) VI-A. Which of the followmg presents the fewesl ethical/legal problems when a user copies copyrighted software wIthour first obtammg permissIon? To give copies to coworkers to use only m a business (A) (8~

To

let students

(C)

To

make

(D) VI-B

III a school

a backup

copy

use the copies

to be stored

To give copies IO indiwduals The

main

LO use onI4

loh of a svstems analvst

only

away

only

for educar~onal

in Ihc prwacy

IS to

(A) (8) CC)

develop-a gencral’plan for u&g compuers m a busmess write advanced programs for computers to use do financial analyses usmg computers

CD)

design

peripherals

that

purposes

if the original is damaged of rheir homes

LO be used

can be used wllh

computers

VII-A

Dot-matrix refers to a type of printer (A) monllor (8) CC) disk personal computer CD)

VII-B (A) (B) 0 (D)

A flowchart IS a

VII-C (A) (B) (Cl (D)

RAM

diagram

for

star~st~al program graph

that

showmg srands

ready

produces Row

program

by computer

financial

of informarlon

program

charts mslde

a computer

memor)

all-purpose

all-program

random

a computer

produced

for

arithmetic

revolbing read

plannmg

graph

memory

mamframe

access memory

In addition to diagnosing the strengths and weaknesses of individual students, the tests may also be used in the overall evaluation of computer-literacy programs. A low average for students on a test will indicate an area of weakness in a program; a high average will indicate a program’s success in the area covered by the test. Additional developmental work on the tests is needed. Specifically, the tests need to be tried out with students are various stages of exposure to computer literacy instruction and the differences among the groups need to be tested statistically. In addition, statistical analysis of individual test items may provide insight into the nature of the items and lead to further refinements of items. Individuals who are willing to help with this phase are encouraged to contact this author to obtain a copy of the tests, to use them, and to share the results with the author. Although this will not provide adequate data for establishing norms for a standardized test, it will provide a base of data that will be used for a preliminary item analysis and other preliminary statistical work on the tests. A single copy of the tests on microfiche may be obtained by writing to Educational Testing Service, Princeton, NJ 08541, U.S.A. REFERENCE 1.

Gabriel R. M.. Cornpurer Literacy ond.Science Tests. Examiner’s Manuel. Northwest Portland, Ore. (1984).

Regional EducatIonal

Laboratory.