Conducting social science laboratory experiments on the world wide web

Conducting Social Science Laboratory Experiments on the World Wide Web Alison I. Piper

Simmons College Researchers in the behavioral and social sciences, particularly psychology, are beginning to conduct laboratory experiments on the World Wide Web. The Web venue offers conveniences that are apparent to market researchers and academic investigators, who have used the Web extensively for survey research. Like traditional experimental researchers, Web experimenters must assure the reliability and inferential validity of their experiments to determine whether a causal relationship exists between independent and dependent variables. In research design and implementation, therefore, Web experimenters must avoid the many artifacts of the experimental situation that threaten internal validity, construct validity, and external validity. This article discusses the application of methodological techniques to social science Web experiments and suggests topics in library and information science most conducive to research on the Web. At the close of the 19th century, Norman Triplett conducted the first actual laboratory experiment in the then nascent field of social psychology. A century later, researchers are conducting the first virtual laboratory experiments in social psychology and other behavioral and social sciences, including library and information science. Triplett’s work is noted not so much because of his finding (i.e., that subjects who worked in groups performed faster than those who were alone, which was later termed social facilitation) as for his innovative decision to employ experimental methods to collect data about social interactions in a face-to-face, laboratory setting (Triplett, 1898). At the close of the 20th century, researchers must similarly decide whether, and if so, how, to pioneer the use of experimental methods to gather data about interpersonal interactions in simulated, World Wide Web (Web) settings (see also Browndyke, 1996d; Hewson, Laurent, & Vogel, 1996; Reips, 1997). This article considers the benefits, detriments, and appropriateness of the Web as a venue for experimental research in the social sciences, including library and

Direct all correspondence .

to:

Alison

I. Piper,

17 Rocky

Brook

Road,

Dover,

Massachusetts

02030

Library & Information Science Research, Volume 20, Number 1, pages 5-21 Copyright 0 1998 Ablex Publishing Corporation All rights of reproduction in any form reserved. ISSN: 0740-8188

5

6

Piper

information science. The first section reviews the philosophical antecedents and methodological requirements of experimental design and implementation. This review is followed by the application of these research standards to experiments conducted on the Web. Finally, conclusions and caveats about Web experiments are used to suggest topics in library and information science that would be most appropriate for experimental research using the Web and other Internet modalities. As yet, only a few researchers have reported use of the Web in behavioral and social science experiments, primarily in psychology (e.g., Henss, n.d.; Krantz, Ballard, & Scher, 1997; Piper, 1997a; Reips, 1996b; Schubert & Waldzus, 1997; Steinhart, Fowler, & Brown, 1995; Welch & Krantz, 1996). This statement may seem preposterous to Web users who are constantly badgered by requests for participation in suwey research in listserv groups (i.e., automatically administered discussion services) and on Web sites. The Web has certainly been used extensively by academic and market researchers to conduct surveys about products, services, and concepts. In many cases, the researchers divide respondents into groups, according to the criteria of interest, including gender, age range, and income. In addition, if a survey involves evaluation of a particular item or service, the participants will be divided into groups determined by amount of use or experience with the product or service. Although a valid and useful form of research, surveys do not represent controlled experiments, which are designed to explain, rather than describe, behavior. Such experiments have been conducted by social scientists since Triplett’s first examination of social facilitation.

CAUSAL INFERENCE

AND SOCIAL SCIENCE RESEARCH

The study of cause and effect in the behavioral and social sciences is philosophically grounded in the writings of 18th century Scottish philosopher David Hume, as qualified by 19th century English philosopher John Stuart Mill. According to Hume, causality is defined by contiguity of the cause and effect in time and space, prior existence of the cause, and necessity of co-occurrence of the cause and effect (Boring, 1950). Mill reconsidered contiguity as a mental relationship, rather than a physical and temporal proximity, and required “frequency” (and later the more rigorous “inseparability”) of co-ocurrence, as well as temporal primacy of the cause. Both Hume and Mill advocated examining causal relationships using not deduction, but logical methods that prefigure later experimental techniques (Boring, 1950). Cook and Campbell (1979) applied the philosophical work of Hume and Mill to experimental research. They state that causality is determined by three conditions: l

l

l

Correlation of the independent and dependent variables; Dependent variable is preceded by the independent variable; and Nothing else explains the relationships between the dependent and independent variables.

Experiments on the World Wide Web

7

Independent variables may be any variables that are manipulated by the experimenter to determine whether they cause differences in the other, dependent variables that are being examined. As part of the research design, the experimenter must operationalize the dependent and independent variables, so they can be observed and measured. Many potential independent variables that may be of interest to researchers cannot be manipulated and therefore are not appropriate for study using experiments (e.g., gender, age, and marital status). Nonexperimental methods allow social science researchers to note positive or negative correlations between variables, but do not allow researchers to draw conclusions about causality. Quasi-experiments are often used in situations such as evaluations of treatments or programs in which researchers do not have complete control over the independent variables and cannot randomly assign subjects to conditions. Experiments, conducted either in the laboratory or in the field (i.e., real-life settings), are intended to determine whether a causal relationship exists between two or more constructs. An experimenter should only infer causality if the experiment has reliability (i.e., the experiment can be replicated at another time and location, with different subjects) and inferential validity (i.e., researchers may legitimately draw causal inferences from the results) (Rosenthal & Rosnow, 1984). Inferential validity implies that the experimenter has avoided the many threats to construct validity, internal validity, and external validity in both research design and implementation (Rosnow & Aiken, 1973). Internal validity indicates the degree to which the manipulation of the independent variables actually causes the changes in the dependent variables. Construct validity reflects why the independent variables cause the observed changes in the dependent variables. External validity refers to the degree of generalizability of the results of the experiment. There are various threats to internal validity, construct validity, and external validity. These threats must be avoided or minimized to assure the inferential validity of an experiment.’

THE WEB AS A VENUE FOR RESEARCH Despite numerous threats to validity, the laboratory has been a fruitful venue for experimental research in the social sciences. Why then consider the use of the Web as an alternative location for experiments? The Web offers pragmatic benefits that are apparent to both market researchers and academic investigators, who have conducted many surveys, in a variety of fields, in an amazingly brief period of time. Web surveys are less expensive and more efficient to design and implement than off-line market research (Hewson, Laurent, & Vogel, 1996; Schmidt, 1997). Also, because Web questionnaire respondents are more geographically diverse than

1 Validity and research design of experiments and quasi-experiments Campbell and Stanley (1966) and Cook and Campbell (1979).

are discussed at length in

8

Piper

traditional participants (Marable, 1997; Pitkow & Kehoe, 1997) researchers have ready access to an abundance of information about the many small, specialized markets that are represented on the Internet (Dvorak, 1996). Many market research companies have an online division or specialize in Internet research, which includes interactive focus groups as well as Web questionnaires (Marable, 1997). In addition, companies or professionals with limited market research budgets can use a variety of new survey construction products and statistical packages to create and analyze surveys “in-house” (Marable, 1997). Web surveys, which encompass both user site feedback forms and questionnaires about off-line products and services, have been conducted on topics as diverse as: hair loss products (Regrowth, 1997), antihypertensive medications (Priory Lodge Education, Ltd., 1996), evaluations of Internet access (Multiverse, 1997), Christian beliefs (Stand to Reason, 1997), computer games (Strategic Simulations, Inc., 1997) and radio programs (WJIE, 1997). Fewer researchers have conducted Internet surveys about attitudes or behaviors that are not directly connected to a product or service. Like market research, however, academic surveys conducted via the Internet probe a broad range of topics, using both existing and newly created instruments. Recent questionnaires have investigated: anger (Navas, 1997) brain injury (Browndyke, Santa Maria, Pinkston, & Gouvier, 1996), birth order (Jackson, 1997), irrational food beliefs (Osberg, 1997), religious beliefs (Snell, 1997) marriage (Schaefer, 1996) student worries (McDaniel & Eison, 1997), Internet usage (Barzilay & Itzhaki, 1997; Roberts, n.d.), test anxiety (Mueller & Jacobsen, 1997), hand washing (Lehman, 1997) and needs and coping mechanisms of cancer survivors (Fawcett & Buhle, 1995). Academic researchers have noted the benefits of Web surveys, which include: ease of solicitation of subjects (Michalak, 1996); convenience of completion for respondents (Fawcett & Buhle, 1995), which may produce longer, more thoughtful answers; ease of data collection (Michalak, 1996); perceived anonymity of interaction, which may be uninhibiting for subjects (Fawcett & Buhle, 1995); geographic diversity of participants (Krantz, Ballard, & Scher, 1997); and relative ease and accuracy of data entry and analysis (Michalak, 1996). To create and implement valid, reliable surveys, Web researchers must carefully adapt and apply rigorous methodological standards of survey research, including: instrument construction, sampling techniques, administration, and analysis (Babbie, 1992). Despite these and other methodological and pragmatic concerns, which are addressed at length elsewhere (see Browndyke, 1996~; Schmidt, 1997; Pitkow & Kehoe, 1997; Piper, 1997b), the Web has been used extensively by market researchers and academic investigators for survey research. Web Experiments

vs. Laboratory

Experiments

It is not surprising that use of the Web would also appeal to researchers conducting behavioral and social science experiments or quasi-experiments. The Web offers

Experiments on the World Wide Web 9

convenient, expeditious, and inexpensive opportunities to find and run subjects, create and duplicate experimental materials, and analyze and publish results. Experimenters working in laboratory settings must seek funding or other departmental or organizational resources to advertise for and then compensate subjects and design and duplicate materials. They must take the time to advertise for subjects, in written and oral pleas. Next, experimenters must contact (often repeatedly) subjects and confederates or assistant investigators who agree to participate. Then they must administer the experiment, often on an individual or small group basis. After all subjects have been run, experimenters must code and analyze data. Finally, they write up and, if appropriate, submit their findings for publication. By contrast, experimenters working on the Internet need little or no funding to advertise for and compensate subjects, create and copy materials, and analyze, write up, and publish results. They can prepare a brief advertisement and broadcast it to many mailgroups simultaneously, with little effort. They can contact interested subjects by e-mail, taking advantage of the benefits of the medium: they can duplicate and modify a single response, contact many subjects simultaneously, send and receive at any time, from many locations, and record and/or print messages (Miller & Owens, 1994). Similarly, experimenters can prepare written, aural, or visual stimulus and response materials once, with no need to duplicate items for each subject or group of subjects. Experimenters who have access to the Internet through educational or research institutions can usually provide these materials to their subjects via Web sites created and maintained for little or no cost. Subjects can provide data easily and at convenient times and places (Hewson, Laurent, & Vogel, 1996; Krantz, Ballard, & Scher, 1997). Results can be copied from e-mail responses or ported directly from Web forms into databases or spreadsheets on the experimenters’ institutional or personal computers, and can be statistically analyzed with minimal effort and time (Krantz, Ballard, & Scher, 1997). Finally, results can be easily published, either directly on the Web or in electronic journals or Internet conferences, using hyperlinks to stimulus and response materials, as well as in traditional print journals. Although authors should be cautious of publishing entire manuscripts on the Internet, due to copyright concerns and journal restrictions (American Psychological Association, 1997) distribution of experimental materials on the Internet will facilitate replication by other experimenters and, thus, assessments of reliability of the findings. Recent Web Experiments As experimenters recognize and discuss the pragmatic benefits of conducting research on the Web, many will undoubtedly design and implement quasi-experiments and experiments. To date, researchers have not reported the use of the Web as a venue for evaluation research utilizing quasi-experimental designs. This situation is likely to change, however, because the Web is conducive to implementation of research designs that increase the internal validity of quasi-experiments.

10

Piper

For example, due to the convenience of participation by subjects, Web researchers can readily use multiple time-series designs, in which experimental groups and nonequivalent control groups are compared several times before and after the introduction of the treatment or stimulus (Babbie, 1992). The Web has been used as a venue for experimental research, although few researchers have published results of Web experiments, either in print or on the Web. These experimenters have, however, used the Web to investigate diverse issues in the behavioral and social sciences. Topics of Web experiments in psychology include: cognitive processes (Reips, 1996b), judgments of musicality (Steinhart, Fowler, & Brown, 1995), estimates of age, height, and weight from facial photographs (Henss, n.d.), judgments of female attractiveness (Krantz, Ballard, & Scher, 1997), and estimates of shapes and distribution of money (Schubert & Waldzus, 1997). In a recent Web experiment in library and information science, Piper (1997a) examined the influence of word choice (i.e., Germanic roots v. Latinate roots) in text e-mail reference interactions on perceptions of librarians and found that, when deprived of nonverbal expression, librarians can communicate interpersonal warmth through word selection. To be certain that their findings of causality have inferential validity, these and other researchers must apply rigorous standards of research design and implementation to their Web experiments. As previously noted, the structure of social science experiments has evolved as researchers have tried to avoid artifacts of the experimental situation that threaten reliability and validity. Can experiments conducted on the Web avoid the many threats to internal validity, construct validity, and external validity? Internal Validity of Web Experiments Among the threats to internal validity, often called confounding variables, are history, maturation, instrumentation, selection biases, experimental mortality, and contamination of control groups (Campbell & Stanley, 1966; Cook & Campbell, 1979). History refers to extraneous events that may occur during experimentation, ranging from world events to telephone calls in the laboratory. Maturation includes ordinary development of or changes in individuals participating in experiments, from general aging in long-term research to fatigue or hunger in brief experiments. Instrumentation refers to actual changes in the instrument or in its use by experimenters over time. Selection biases occur when subjects with particular relevant traits, beliefs, aptitudes, or attitudes are presented with the independent variable. Experimental mortality refers to subjects’ failure to participate in or complete the experiment, which may be elicited by the requirements of the experimental treatment, qualities of the subjects, or an interaction between the two. Contamination of control groups occurs when members of the experimental groups communicate with members of the control groups about the treatment. In Web experiments, many threats to internal validity, including history, maturation, instrumentation, and selection biases, can be avoided through use of random

Experiments

on the World Wide Web

11

assignment to control or experimental groups. Experimenters working on the Web can readily employ random assignment of subjects. Piper (1997a) advertised for subjects in various listservs and received e-mails from subjects who agreed to participate. Each subject was randomly assigned to a condition and sent the appropriate URL (i.e., universal resource locator) by e-mail. Random assignment may, however, have been negatively impacted by accidental or deliberate multiple participation by some subjects. Multiple submissions within condition can be easily noted and eliminated if subjects are required to use a unique password (Rieps, 1997b) or nickname (see Navas, 1997). In a technologically more sophissubjects could be randomly assigned to a control or ticated experiment, experimental group by a dynamic script embedded in an initial Web site. In this way, multiple participation by specific subjects would not affect random assignment, although it would still impact internal validity through contamination of data (see below). As is demonstrated by the e-mail reference experiment (Piper, 1997a), control groups can be used to ascertain that changes in the dependent variables were caused by the independent variables, rather than by history, maturation, instrumentation, or selection biases. Experimenters must, as in laboratory experiments, be sure that groups are exposed to stimulus materials that are comparable in every way except for the manipulated variable. Piper (1997a) noted that many subjects were annoyed by the lack of bibliographic citations in the stimulus materials; because this material was omitted from both Web sites, however, it did not provide an alternative, confounding explanation for the differences in the dependent variables. Other threats to internal validity produce more of a challenge for Web experiments. In the e-mail reference experiment, for example, Piper experienced almost 50% experimental mortality (in this case, defined as failure to participate after assignment to condition) due, in part, to lack of an e-mail feature on some subjects’ browsers. Subjects in one condition were, however, no more likely to drop out than those in the other condition, because subjects were randomly assigned to conditions. As in laboratory experiments, the threat to internal validity posed by experimental mortality in Web experiments can be minimized by random assignment and comparability between conditions provided by appropriate control groups. Contamination of control groups is a particularly serious threat to the internal validity of experiments conducted on the Web. In laboratory experiments, it is usually not difficult for experimenters to separate subjects in different conditions. Subjects are either run together, with supervision, to ensure no discussion or perusal of other stimulus materials or sequentially, with strong injunctions against discussion of the details of the experiment. Although the possibility of contamination exists, subject role motivations generally encourage subjects to honor the experimenters’ requests for confidentiality. In a Web experiment, however, this threat is more difficult to minimize, because more than one person may be viewing the experimental Web sites simultaneously

12

Piper

(Reips, 1997). In addition, subjects may be less likely to honor requests for confidentiality from an experimenter they have not seen and are unlikely to ever meet in person. Piper (1997a) found that a few subjects offered to forward the e-mail message that contained the randomly assigned condition URL to other, likely participants. Although these subjects did not do so when the experimenter (politely) objected, it is not known how many other subjects simply passed along or bookmarked the URL, allowed others to sit at their computers, or participated another time, after they completed the experiment, without informing the experimenter. The threat of contamination could be addressed by experiments using dynamic Web sites, in which subjects would receive (and could potentially forward) only the URL of the initial, welcome site. Subjects could, however, use the “back” button on their browsers to move from the dependent variable site to the condition stimulus site and allow another subject to begin from that point in the experiment (or complete the experiment a second time). In the future, emerging technology may eliminate this possibility (Michalak, 1996). At present, as noted above, researchers can issue unique passwords to each subject, which, if used more than once, would alert experimenters to data that are potentially contaminated by prior knowledge of the experiment. It is, of course, possible that subjects determined to participate more than once would request the URL and password from an e-mail address that is anonymous (Reips, 1996a) or generic (e.g., a library or computer laboratory location). Therefore, experimenters should clearly and strongly request that subjects not view or discuss the Web sites with other potential subjects or complete the experiment more than once. Construct Validity of Web Experiments Construct validity can be threatened by artifacts of the experimental situation, including experimenter expectancy effects and subject role effects (i.e., evaluation apprehension and demand characteristics). Experimenters’ expectations of subject performance, as influenced by hypotheses, instructions from the principal investigator, and/or initial results, can be subtly communicated to subjects. Different nonverbal cues can be given to the experimental and control group subjects, which would then elicit different responses, rather than manipulation of the independent variables (Rosenthal, 1976). According to Rosenberg (1969), subjects are susceptible to subtle cues from experimenters because they are fearful of being judged during the experiment. This evaluation apprehension may be more extreme in one or more experimental condition and may therefore effect differences in dependent variables. In addition, according to Orne (1969), many participants want to be “good subjects” and further scientific knowledge. Other, more negativistic subjects may try to disprove the experimenters’ hypotheses. Subjects’ responses to the demand characteristics of the experiment may prompt them to respond to the experimenters’ subtly communicated expectations about differences in responses by the experimental and control groups.

Experiments

on the World Wide Web

13

Artifacts of the experimental situation, including experimenter expectancy effects and subject role effects that threaten construct validity, may be less powerful in experiments conducted on the Web than in most laboratory experiments. The lack of nonverbal communication has been noted as a major detriment of e-mail communication (Abels, 1996; Miller & Owens, 1994; Porter, 1984; Sproull & Kiesler, 199 1) and the same is certainly true of information provided on Web sites. This otherwise negative feature does, however, enable experimenters to reduce the impact of expectancy effects on dependent variables in Web experiments. According to Rosenthal (1976) if nonverbal communication can be limited or controlled, then experimenter expectancy effects can be minimized during the experiment or assessed during analysis of the results. Although, as noted above, subjects may respond to the demand characteristics of the experiment, by trying to be “good subjects” (or oppositional subjects), it will be difficult for them to determine the experimenter’s goals without nonverbal cues. Web experiments are examples of automated experiments, which have been suggested as a strategy to minimize contact between experimenters and subjects (Rosenthal, 1976). Also, instructions and other communications via e-mail and/or Web sites can be standardized in a way that is desirable, but rarely achieved in face-to-face interactions or individual written communication. Even while using written scripts, researchers in laboratory experiments often subtly change their communications in response to the nonverbal cues of their subjects. Also, laboratory experimenters are more likely than Web experimenters to answer clarifying questions during the course of the experiment, even when written media are used. The e-mail reference experiment (Piper, 1997a) was conducted on the Web, rather than by individual e-mail exchanges, to increase uniformity of presentation and discussion of stimulus materials. Most Web experiments do include an e-mail address or a mailto link to the experimenter (e.g., Krantz, Ballard, & Scher, 1997; Piper, 1997a; Steinhart, Fowler, & Brown, 1997) but subjects may be less likely to make the effort involved to contact the experimenters than subjects in a face-toface situation. Experimenters still concerned about the biases introduced by experimenter expectancy effects in Web experiments can use expectancy control groups, for which the expectancy effect is manipulated separately from the independent variable (Rosenthal, 1976). Experiments conducted on the Web may also tend to reduce subjects’ evaluation apprehension. Due to their relative anonymity and lack of individual contact, subjects may be less concerned about the power and/or persuasiveness of experimenters than in laboratory experiments (see Rosenberg, 1969). Demand characteristics can also be minimized in experiments conducted on the Web. At present, experimental research on the Web is so rare that many subjects are not fully aware that they are participating in an experiment. Piper (1997a) noted that many subjects referred to the experiment as a survey, even after completion of the experiment. This lack of awareness will surely drop as researchers begin to use the medium more extensively.

14

Piper

Presentation of false, but plausible, hypotheses and areas of interest may divert subjects’ attention from and therefore ability to accept or reject the actual hypotheses being tested in the experiment (Orne, 1969). Researchers must consider carefully the necessity, potential effects, and alternatives to deception in Web experiments before implementation. As in other types of research, Web experimenters must also obtain informed consent and thoroughly debrief subjects as soon as possible after completion of the research (American Psychological Association, 1997). Debriefing may be particularly challenging in Web experiments, because many subjects will opt not to view a debriefing site, read a debriefing e-mail, or participate in an interactive debriefing session. External Validity of Web Experiments External validity is at risk when subjects, situations, and variables are not comparable to those outside the laboratory (Cook & Campbell, 1979). Most behavioral and social science experiments use college students as subjects, because they are generally convenient, inexpensive, and cooperative. They are not, however, similar to the population as a whole or to many target subgroups, due, in part, to relative homogeneity of age, culture, nationality, and experience. In addition, although random sampling of subjects from the larger population is the methodologically ideal selection technique, many experimenters must rely on students who are volunteers. According to Rosenthal and Rosnow (1969) the personality traits of volunteers differ significantly from those of nonvolunteers. Experimental situations also usually differ from “real life,” in that they do not often involve competing influences, long-term relationships with the experimenters or confederates, or perceived significance of the consequences of any decisions or actions. Variables in laboratory experiments are also often unlike their real world counterparts, because they are less complex and are time-limited. The manipulation of the independent variable may therefore only affect a particular type of person, in a particular type of situation, when operationalized in a particular way. Experiments conducted on the Web are not threatened by some of the limitations that affect the external validity of laboratory experiments. Subjects are less likely to be college students from a particular country because the Web attracts intemational users of a broad range of ages, cultures, and experiences. In addition, although random sampling from the entire target population is again not feasible, subjects may be less likely to fit the traditional profile of volunteers (see Rosenthal & Rosnow, 1969) because experiments conducted on the Web are very convenient and relatively novel. This is not to claim, however, that subjects who participate in Web experiments are any more representative of the larger society than their counterparts in laboratory experiments. Although use of the Internet and the Web in particular are growing dramatically, world-wide saturation has not and will not be reached anytime soon, if ever. A recent survey of 6,600 North American Internet users by CommerceNet and Nielsen Media Research indicates that just 23% of people ages

Experiments on the World Wide Web

15

16 and older have access to the Internet. Moreover, although demographic figures have changed dramatically during the past few years, this survey revealed that Internet users are still disproportionately male (58%) (Nielsen Media Research, 1997). Results of another recent survey of 15,000 international Internet users depict them as even less representative of the general population-almost 69% male in the United States, and 80% male in Europe (Pitkow & Kehoe, 1997). Findings from Web experiments can be generalized to Web users generally or a target sub-population, if subjects are randomly sampled from the population in question. Also, experiments conducted on the Web can be designed to replicate experiments previously conducted in laboratories with signi~~~tly different subject populations (Krantz, Ballard, & Scher, 1997). In either case, experimenters can increase external validity by recruiting subjects appropriately (i.e., widely for study of the general population and narrowly for study of a target subgroup) and reducing technological or task requirements that would limit the number and type of subjects (Piper, 1997a). The ability to generalize from subjects in Web experiments to any other population is, however, severely limited by the ease with which subjects can deceive experimenters about traits such as gender, age, race, many of which are observable and, thus, verifiable in face-to-face experiments. Although this critical difference does not automatically disqualify the Web as a venue for expe~mental research, it is a threat to external validity that is often reported (Chapman, 1995; McRae, 1996; Pitkow & Kehoe, 1997; Shade, 1993; Van Gelder, 1996) and difficult to eliminate. Use of social security numbers or other means of identification would violate subjects’ privacy (American Psychological Association, 1997) and greatly reduce the number and types of subjects willing to participate in Web experiments (Browndyke, 1996b). Technological innovations, especially high-speed network access, are likely to be implemented in the near future (Sun Microsystems, 1997). Such advances would allow interactive audio-visual communication between subjects and experimenters, either prior to random assignment to conditions or during the course of the experiment. Transmission of audio-visual info~ation about subjects could, like other forms of identification, violate subjects’ privacy (Browndyke, 1996a). At present, and perhaps indefinitely, experimenters must resist the temptation to generalize findings about a particular subgroup of subjects to others in the population with apparently the same traits or characteristics. External validity of Web experiments is, like that of laboratory experiments, also threatened by lack of realism of situations and variables. This may be less critical to some types of research, including that which examines responses to written materials (e.g., Piper, 1997a), still visual images (e.g., Henss, nd.; Krantz, Ballard, & Scher, 1997), or noninteractive auditory stimuli (e.g., Steinhart, Fowler, & Brown, 199’7). It is unlikely that Web experiments that attempt to examine dynamic processes will ever attain the realism of face-to-face interaction. Use of existing tools to create and integrate animated graphics {Bury, 1997) or audio-

16

Piper

visual communication (Lee, 1995) may, however, increase the realism of both variables and situations in experiments conducted over the Web.

CONCLUSION It is evident that the Web will be used increasingly by social science researchers to conduct experiments because it offers so many pragmatic conveniences, notably of time and money. It is less clear, however, whether these researchers will conduct methodologically rigorous experiments from which they can legitimately draw causal inferences. Some of the artifacts that often threaten the validity of laboratory experiments are inherently less problematic (e.g., experimenter expectancy effects, subject role effects, and use of college students as subjects) or can be addressed easily in Web experiments (e.g., history, maturation, selection biases, and experimental mortality). Other threats to validity are much more challenging or even impossible for researchers to eliminate (e.g., lack of realism of variables and situations, use of nonrepresentative, volunteer subjects, and deception by subjects). If researchers cannot avoid these artifacts in Web experiments, then, as in laboratory or field experiments, they must both acknowledge the limitations of the results and replicate these experiments in diverse settings, with different types of subjects, and a variety of operationalizations of constructs. The Web can also be used, with fewer concerns about validity, to conduct pilot studies of planned experiments, so experimenters can receive feedback about materials, procedures, or subject role effects, or collect preliminary data, to calculate effect size or demonstrate feasibility for funding sources (Fawcett & Buhle, 1995). The Web also allows for replication of experiments previously conducted with more limited subject populations (e.g., students) (Rrantz, Ballard, & Scher, 1997). To date, most Web expe~ments have been conducted by psychologists, primarily in the areas of perception and cognition, in part because Web technology is most conducive to presentation of noninteractive sets of stimuli. This does not preclude the investigation of dyadic or group interaction, but necessitates articulation of focused hypotheses about specific areas of interaction (e.g., Piper, 1997a). Web experimenters can also combine visual materials and role playing scenarios, which have been used successfully by experimenters in situations in which excessive deception or other harm to subjects precludes actual manipulation of independent variables (Rosenthal & Rosnow, 1984). As library and information science researchers contemplate the Web as a venue for both survey research (Piper, 1997b) and experiments, they must consider the implications of the Web’s particular benefits and challenges for choice of research topics. The Web is especially useful for investigation of issues that require a subject population that is very narrow (e.g., interactions in one-person libraries), geographically expansive (e.g., international comparisons of services or materials), or computer literate (e.g., innovations in e-mail systems, catalogs, or Web

Experiments

on the World Wide Web

17

sites). The Web is also particularly effective for study of of written or visual materials (e.g., user instructions, signs, and training manuals). the Web is to consideration of and exacerbate experimenter expectancy and subject in face-to-face experimental situations of characteristics as race, gender, disability, or sexual orientation on In addition to experiments, library and information science researchers design and implement of internal validity and construct validity to

of variables specially

in

or visual settings,

as e-mail exchanges, or MUDS (i.e.,

Dungeons/Dimensions/Domains-programs in which participants interact with and the environment using text commands). in methodologically rigorous experiments, may seem unlikely for serious researchers, but pioneering requires innovaIn 1897, out fishing to children and asked them to the reels alone or in groups, it is unlikely of his fellow psychologists he was initiating a legitimate research method. As the second century of experimental research in social sciences commences, the opportunity to and methods and venues to the understanding of in library and information

G. (1996). The e-mail RQ, 35 (3) 145-59. American Psychological Association. (1997, October 8-last update). Ethical principles of psychologists and code of ethics. [Online]. Available: http:// www.apa.org/ethics/code.html [ 1997, October 191. American Psychological Association. (1997, August 28-last update). Posting information on the Internet. [Online]. Available: http://www.apa.org/journals/ posting.html [ 1997, October 191. Babbie, Earl. (1992). The practice of social research. Belmont, California: Wadsworth. Barzilay, Arik, & Itzhaki, Neta. (1997, March 26-last update). Znternet su@ng motivations. [Online]. Available: http:Nwww.biu.ac.il/PS/Projects/Arik/ intro.htm [1997, October 191. Boring, Edwin G. (1950). A history of experimental psychology. New York: Appleton-Century-Crofts.

18

Piper

Browndyke, Jeffrey N. (1996a, April 14). Digest of sci.psyc.research Net discussion. [Discussion]. Psycresearch-online [Online]. Available: ftp:// ftp.premier.net/pub/lists/psycresearch-online/psycresearchonline.archive.9606 [1997, October 191. Browndyke, Jeffrey N. (1996b, April 16). E-mail grabbing and subject verification. [Discussion]. Psycresearch-online [Online]. Available: ftp:// ftp.premier.net/pub/lists/psycresearch-online/psycresearchonline.archive.9606 [1997, October 191. Browndyke, Jeffrey N. (1996c, May 20). Re:Online Neuropsychology Project (fwd). [Discussion]. Psycresearch-online [Online]. Available: ftp:// ftp.premier.net/pub/lists/psycresearch-online/psycresearchonline.archive.9605 [1997, October 191. Browndyke, Jeffrey N. (1996d). Psycresearch-online. Available at http:// www.premier.net/-cogito/psycresearch-online/intro.html [ 1997, October 191. Browndyke, Jeffrey N., Santa Maria, Mike, Pinkston, Jim, & Gouvier, W. D. (1996). Brain injury and safety: Attitudes and biases. [Online]. Available: http://www.premier.net/-cogito/project/onp.html [ 1997, October 191. Bury, Scott. (1997). Make your graphics take off. Macworld, 14(5), 132-l 35. Campbell, Donald T., & Stanley, Julian C. (1966). Experimental and quasiexperimental designs for research. Chicago, IL: Rand-McNally. Chapman, Gary. (1995). Flamers: Cranks, fetishists and monomaniacs. The New Republic, 212(15), 13-15. Cook, Thomas D., & Campbell, Donald T. (1979). Quasi-experimentation: Design and anaEysisforfieEd settings. Chicago, IL: Rand-McNally. Dvorak, John. (1996). Web wake-up call. PC/Computing, 9(8), 59. Fawcett, Jacqueline, & Buhle, E. Loren. (1995). Using the Internet for data collection: An innovative electronic strategy. Computers in Nursing, 13, 273279. Henss, Ronald. (n.d.). Estimate age, height, and weight. [Online]. Available: http://www.cops.uni-sb.de/ronald/experim/estimate/est_mla.html [1997, October 191. Hewson, Claire M., Laurent, Dianna, & Vogel, Carl M. (1996). Proper methodologies for psychological and sociological studies conducted via the Internet. Behavior Research Methods, Instruments, & Computers, 28, 186191. Jackson, Betsy. (1997, March 5-last update). Birth order survey. [Online]. Available: http://www.psych.ucsb.edu/-jackson/survey.html [1997, October 191. Krantz, John H., Ballard, Jody, & Scher, Jody. (1997). Comparing the results of laboratory and World-Wide Web samples on the determinants of female attractiveness. Behavior Research Methods, Instruments, & Computers, 29, 264-269. Lee, David. (1995). QuickCam. Mucworld, 12(3), 73.

Experiments

on the World Wide Web

19

Lehman, Galen. (1997, March 1 l-last update). Hand washing questionnaire. http://beta109.emu.edu/lehmangr/hands.htm [1997, [Online]. Available: October 191. Marable, Leslie. (1997, April 17-last update). Online market research begins to catch on. WebWeek, 3(S). [Online]. Available: http://www.webweek.corn/ 97Mar3 l/markcomm/market.html [1997, October 191. McDaniel, Patricia Smoot, & Eison, James. (1997, April 30-last update). Assessing student worries. [Online]. Available: http://www.imsa.edu/-grung/ psych/index.html [1997, October 191. McRae, Shannon. (1996). Coming apart at the seams: Sex, text and the virtual body. In L. Cherny & E. R. Weise (Eds.), Wired women: Gender and new realities in cyberspace (pp. 242-263). Seattle, Washington: Seal Press. Michalak, Erin. (1996, April 18). Advantages and disadvantages of using the Internet. [Discussion]. Psycresearch-online [Online]. Available: ftp:// ftp.premier.net/pub/lists/psycresearch-online/psycresearchonline.archive.9606 [1997, October 191. Miller, Jerry P., & Owens, A. (1994). Should you get wired? Library Journal, 119 (2) 47-49. Mueller, John, & Jacobsen, Michele. (1997, October 6-last update). Test anxiety and computer experience. [Online]. Available: http://www.ucalgary.ca/ -mueller/tai-forrnhtml [ 1997, October 191. Multiverse. (1997, June 1 l-last update.). Multiverse customer service survey. [Online] Available: http://www.multiverse.com/service.html [ 1997, October 191. Navas, Sharonne. (1997, March 19-last update). Anger disorder survey. [Online]. Available: http://www.liii.com/-fantine/consent.html [ 1997, October 191. Nielsen Media Research. (1997, September 15-last update). “Startling increase” in Internet shopping reported in new Commercenet/Nielsen Media Research survey. [Online] Available: http://www.nielsenmedia.corn/news/ shopping.html [ 1997, October 191. Ome, Martin T. (1969). Demand characteristics and the concept of quasi-controls. In R. Rosenthal & R. L. Rosnow (Eds.), Artifact in behavioral research (pp. 143-179). New York: Academic Press. Osberg, Timothy. (1997, September 15-last update). Irrational food belief survey. [Online]. Available: http://www.niagara.edu/psychology/osberg/ foodsurv.html [1997, October 191. Piper, Alison I. (1997a). The influence of word choice in e-mail reference interactions. Unpublished manuscript, Simmons College. Piper, Alison I. (1997b). Conducting survey research on the Internet. Unpublished manuscript, Simmons College. Pitkow, Jim, & Kehoe, Colleen. (1997, June 9-last update). GVU’s 6th WWW survey [Online]. Available: http://www.cc.gatech.edulgvuJ page. user_surveys/survey-lo-1996/ [ 1997, October 191.

20

Piper

Porter, Peggy. (1984). “Dear PP05”: Interpersonal communication and the computer: Entering a brave new world. Library Hi Tech, 2(l), 23-27. Priory Lodge Education, Ltd. (1996, May 23-last update). Antihypertensive survey. [Online]. Available: http://www.priory.co.uk/survhype.htm [1997, October 191. Reips, Ulf-Dietrich. (1996a, July 1). Re: Psycresearch-online: April posting digest. [Discussion]. Psycresearch-online [Online]. Available: ftp:// ftp.premier.net/pub/lists/psycresearch-online/psycresearchonline.archive.9606 [1997, October 191. Reips, Ulf-Dietrich. (1996b, September 16-last update). AAAbacus. [Online]. Available: http://134.2.133.25/WWWExpE.html [1997, October 191. Reips, Ulf-Dietrich. (1997, July 22-last update). Web Experimental Methods. [Online]. Available: http://www.uni-tuebingen.de/uni/sii/Ulf/Lab/ WebExpMethod.html [1997, October 191. Regrowth. ( 1997, October 15-last update). Reactive! Surveys. [Online]. Available: http://www.regrowth.com/reactive/surveys [1997, October 191. Roberts, Lynne. (n.d.). The newbie survey: A survey of personality and virtual environments. [Online]. Available: http://psych.curtin.edu.au/people/robertsl/ newbiein.htm [ 1997, October 191. Rosenberg, Milton J. (1969). The conditions and consequences of evaluation apprehension. In R. Rosenthal & R. L. Rosnow (Eds.), Artifact in behavioral research (pp. 280-348). New York: Academic Press. Rosenthal, Robert, (1976). Experimenter efsects in behavioral research. New York: Irvington Publishers. Rosenthal, Robert, & Rosnow, Ralph L. (1969). The volunteer subject. In R. Rosenthal & R. L. Rosnow (Eds.), Artifact in behavioral research (pp. 61118). New York: Academic Press. Rosenthal, Robert, & Rosnow, Ralph L. (1984). Essentials of behavioral research: Methods and data analysis. New York: McGraw-Hill. Rosnow, Ralph L., & Aiken, L.S. (1973). Mediation of artifacts in behavioral research. Journal of Experimental Social Psychology, 9, 181-201. Schaefer, Earl. (1996, November 21-last update). Marriage inventory---MARI. http://www.unc.edu/-schaefer/mariint.htm [1997, [Online]. Available: October 191. Schmidt, William C. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments, & Computers, 29,274-279. Schubert, Thomas, & Waldzus, Sven. (1997, September 30-last update). Estimating shapes and distributing money. [Online]. Available: http:// www.uni-jena.de/-ssw/labor.htm [1997, October 191. Shade, Leslie Regan. (1993). Gender issues in computer networking. [Talk given at Community Networking: The International Free-Net Conference. Carleton University, Ottawa, Canada], [Online]. Available: http://www.mit.edu:8001/ people/sorokin/women/lrs.html [ 1997, October 191.

Experiments on the World Wide Web

21

William E. (1997, April 21-last update). Religion and the 10 Commandments: A survey of people’s belief in religion and the 10 Available: http://www2.semo.edulsnell/ Commandments. [Online]. study 1.html [ 1997, October 191. Sproull, Lee, & Kiesler, Sara. (1991). Computers, networks, and work. Scientific American, 26.5(3), 116- 123. Stand to Reason. (1997, August 16-last update). Suwey. [Online]. Available: http://www.str.org/formnews.htm [1997, October 191. Steinhart, Dave, Fowler, Rick, & Brown, Jeremy. (1995, April 27-last update). What makes music musical? An Internet study in musicality judgements. [Online]. Available: http://psych.colorado.edu/-angst/musicality.html [ 1997, October 191. Strategic Simulations, Inc. (1997, October 17). Survey. [Online]. Available: http:// www.ssionline.com/forms/mail_list.html [1997, October 191. Sun Microsystems. (1997, October 14- last update). Sun Microsystems supports latest IEEE connectivity. [Online]. Available:http://www.eu.sun.com/smi/ Press/sunflash/9603/sunflash.960325.12280.html [1997, October 191. Triplett, Norman. (1898). The dynamogenic factors in pacemaking and competition. American Journal of Psychology, 9, 507-533. Van Gelder, Lindsy. (1996). The strange case of the electronic lover. In R. Kling, (Ed.), Computerization and controversy: Value conflicts and social choices (pp. 533-546). San Diego, CA: Academic Press. Welch, Norma, & Krantz, John H. (1996). The World-Wide Web as a medium for psychoacoustical demonstrations and experiments: Experience and results. Behavior Research Methods, Instruments, & Computers, 28(2), 192- 196. WJIE. (1997, October 18-last update). Listener survey. [Online]. Available: http://www.wjie.org/survey.html [1997, October 191.