- Email: [email protected]
Self-service computerized bibliographic retrieval: A comparison of colleague and PaperChase, programs that search the MEDLINE data base
COMPUTERS
AND
BIOMEDICAL
RESEARCH
21, 488-501 ( 1988)
Self-Service Computerized Bibliographic Retrieval: Comparison of Colleague and PaperChase, Programs Search the MEDLlNE Data Base DOUGLAS
A that
PORTER,* ROBERT S. WIGTON,? MARIE A. R~IDELBACH,$ HOWARD L. BLEICH,* AND WARNER V. SLACK*
*Churles A. Dunu Resetrrch Institute and Hurvurd Thorndike Lahorutory. Beth Isrue Hospitul. Department of Medicine. Hurvard Medical School nt the Beth lsruel and Brigham und Women’s Hospituls, nnd the Center for Clinical Computing. Harvard Medical School. Boston. University of NLJhrrrska College oj Massachusetts 02115: tDepartment of Internal Medicine, Medicine. ONIU~LI, Nehruskr 68101; rend $McGoo,qarr Lihrtrry of’ Medicine. University I:/ Nchrcrsku Mcdicrrl Center. Omuhcr. NehnrsXtr 6KIOI
Received December 29, 1987
Colleague and PaperChase are the two most widely used computer systems designed specifically for clinicians and scientists who wish to search the National Library of Medicine’s MEDLINE data base of references to the biomedical literature. The present study compares the performance of these two systems. Two matched groups of second-year medical students each received 3 hr of instruction. one group in Colleague. the other in PaperChase. Each student then attempted IO test searches. The next day the groups were reversed, and each student attempted 5 additional searches. During 31 hr allocated for searching, users of Colleague attempted 64 test searches and retrieved 326 target references: users of PaperChase attempted 78 searches and retrieved 496. Users of Colleague took a mean of 2.2 min and spent a mean of $1.20 to find each target reference; users of PaperChase took I .6 min and spent $0.92. We conclude that after limited training. medical students find more references faster and at lower cost with PaperChase than with Colleague. I ILIXX Academic
Press. Inc.
In 1966, the National Library of Medicine (NLM) introduced MEDLINE, the computerized data base of references to the biomedical literature from which In&x Medicus is derived (I). Three years later, the National Library of Medicine developed ELHILL, a computer system that now provides interactive access to the MEDLINE data base by means of a computer terminal and telephone line. ELHILL, and its alternatives-computer systems provided by BRS Information Technologies (BRS) and DIALOG-are used principally by specially trained reference librarians who serve as intermediaries between the person who wants to search the literature and the MEDLINE data base. Mastery of these computer systems, and of companion volumes that describe how the medical literature is indexed, requires special training and experience 488 0010-4809188 $3.00 Copyright All rights
0 1988 by Academic Press. Inc. of reproduction in any form reserved.
COMPUTERIZED BIBLIOGRAPHIC
RETRIEVAL
489
(2). Efforts to promote the use of these computer systems by students, clinicians, and scientists without the assistance of a trained librarian have, for the most part, been unsuccessful (3-5). In recent years, programs for personal computers have been developed to help the inexperienced user perform searches. These programs, such as Sci-Mate, Pro-Search, FIRSTUSER, SEARCH HELPER, CANSearch, SearchMaster, Searchware, OASIS, GRATEFUL MED, and Med-Base, translate the computer commands of NLM, BRS, or DIALOG into choices from which the user can pick (6). Some of these programs also help the user find Medical Subject Headings, capture references in a personal computer for subsequent printing or searching, and reformat references in a variety of ways. Finally, three computer systems, easier to use than NLM, BRS, or DIALOG and designed specifically with the clinician or scientist in mind, permit end users to search the MEDLINE data base directly. DIALOG provides Medical Connection, BRS provides Colleague (7), and Boston’s Beth Israel Hospital provides PaperChase, which was developed in our laboratory (8, 9). Medical Connection was announced only recently; at present, Colleague and PaperChase provide most of the end-user searching of the MEDLINE data base. Previous studies of computer systems that search the MEDLINE data base have generally provided descriptive accounts of the nature and extent of users’ searches (2, 5, 10-12). In one study, however, data from searches of experienced librarians were used to advise clinicians on the choice of a system (13). To our knowledge there have been no controlled, prospective studies in which clinicians and scientists have formulated and conducted their searches. In the present study, we employed second-year medical students to compare the performance of Colleague with that of PaperChase in searching the MEDLINE data base. METHODS
The study was designed by the Center for Clinical Computing at Boston’s Beth Israel Hospital and conducted with cooperation from BRSlSaunders Colleague. PaperChase is a service of Boston’s Beth Israel Hospital, and Colleague is now a service of BRS Information Technologies, Inc. The study was conducted at the McGoogan Library of Medicine at the University of Nebraska Medical Center, Omaha, Nebraska, in collaboration with the Department of Internal Medicine. One of us (D.P.) was responsible for the conduct of the experiment in collaboration with representatives from Colleague and PaperChase. Because the study was sponsored by the same organization that provides PaperChase, special precautions were taken to guard against methods of procedure or accidents of execution that could have biased the results in favor of PaperChase (Appendix). The study was conducted in two adjoining rooms, each equipped with personal computers, modems, telephone lines, and communications software (Smartcorn II) programmed to access Colleague in Latham, New York, and PaperChase in Boston, Massachusetts.
490
PORTER
ET AL.
Twelve second-year medical students at the University of Nebraska were each paid $100 to participate. Before the study, each student answered a questionnaire about education and experience with computers. and on the basis of their answers, the students were divided into two equivalent groups of six students each. No student reported prior experience with computerized bibliographic retrieval. Five additional second-year medical students were each paid $150 to work as research assistants: they administered the test searches, observed and clocked the behavior of the participants, and helped with the computer equipment. During the morning of the first day, one group of students was instructed in Colleague by the representative of Colleague. the other group in PaperChase by the representative of PaperChase. That afternoon, each student was asked to use the system in which he or she had been instructed to conduct 10 test searches. The next day, students experienced with Colleague were instructed in PaperChase, and students experienced with PaperChase were instructed in Colleague. That afternoon, five new test searches were administered. Time available to perform searches was limited to 2 hr on the first day and to 1Bhr on the second day. Selection
of Test Semches
Six of the test searches were adapted from a previous study (IS). and the remainder were chosen by the investigators to be representative of those that a physician or scientist would require. The goal of the first two searches was to retrieve a specific reference from fragmentary information: the goal of the next two searches was to produce, to the extent possible, an exhaustive bibliography about a topic-one suitable for use in writing a review article; and the goal of the final six searches was to answer a specific clinical question. On the second day. the goal of the first search was to find a specific reference from fragmentary information, the next to perform an exhaustive retrieval. and the final three to answer a specific clinical question. For searches to answer a clinical question, the students were asked to print the first two references that. on the basis of their titles and abstracts, seemed relevant and that were published in one of the journals on a core list that we obtained from the National Library of Medicine. Preliminary testing indicated that with sufficient expertise each search could be done on Colleague or PaperChase within a few minutes. On the other hand, the task for the students was difficult; there were medical school librarians with years of experience with the MEDLINE data base who could not perform all 10 searches of the first day in the time allocated to the students. With each search, the participant had to formulate the strategy, conduct the search, signal when the first target reference was found, and print out the references found. The performance of each participant was timed with a stopwatch: additional data concerning time and cost were obtained from printouts produced by the computer programs. All searches were conducted over the entire MEDLINE
COMPUTERIZED
BIBLIOGRAPHIC
RETRIEVAL
491
file, 1966 to the present; a description of each search is provided in the Appendix. After each teaching session, students answered a brief questionnaire about the instruction they had just received; immediately after each searching session, they answered a questionnaire about their reactions to the searches; and at the conclusion of the study, they completed a final questionnaire about their participation in the study and reactions to Colleague and PaperChase. Evaluation
of References
Retrieved
We examined each reference printed to see if it was a target reference. Searches to find a specific reference on the basis of fragmentary information had but one possible target reference. References printed during an exhaustive search were deemed on target if they included the appropriate Medical Subject Headings or text words (title and abstract words). References retrieved to answer a specific chnical question were deemed on target if their titles or abstracts pertained directly to the clinical question posed, and if they were published in one of the high quality journals on the list that we provided. A panel of three blinded physicians used identically formatted printouts to arbitrate questionable calls. To compare Colleague and PaperChase, we considered three correlated measures of performanceon-line time, on-line cost, and number of target references printed. If no target references were printed, time and cost were still included. To compare on-line time and on-line cost across searches that retrieved different numbers of target references, we converted these measures into minutes per reference and dollars per reference. As a measure of the time required to formulate and enter the logic of the searches, we computed the total time-including search preparation time- to display the first target reference. Unless otherwise specified, we averaged the performance of each student for each type of search and then submitted these mean values to the MannWhitney U test, with nl = 6, n2 = 6, to compute two-tailed P values. RESULTS
Both for Colleague and for PaperChase, we compared results of the searches on the first day with results on the second day and found no statistically significant differences between the 2 days on any measure of performance for either program. Mean two-tailed P values were 0.51 for Colleague, 0.31 for PaperChase (Mann-Whitney U tests with IZI = 6, 122 = 6). Accordingly, unless otherwise specified, the data from the 2 days were pooled. Searches for a Single Reference
Data from searches for a single reference are summarized in Table 1. When only a single reference was to be retrieved, differences in performance were measured in time and cost, not in the number of references found. On-line time
492
PORTER ET AL TABLE
1
SINGLE REFERENCE S~ARC.HES
Searches attempted” On-line time (minutes/search) Search preparation (minutes/search) Total search time (minutes/search) On-line cost (dollars/search) Number of target references found Minutes per reference Dollars per reference Mean time to display first target reference
Colleague
PaperChase
I8 12.1 I .s Ii.6 h.S? I5 16.3 7.82 14.8
18 8.Y 0.0 8.Y I.Ci I? Y.3 4.79 6.3
---
__Mann-Whitney’ P values
/’ Six participants each attempted all three searches. * Mann-Whitney U test: two-tailed P values with n, = 6. II? = h
averaged 14.6 min for users of Colleague, 9.4 min for users of PaperChase (P <0.05. Mann-LWhitney U test). On-line cost averaged $7.82 for users of Colleague and $4.79 for users of PaperChase (P < 0.02). In 15 out of 18 searches users of Colleague found the required reference, compared with 17 out of 18 for users of PaperChase. Exhnustivr
Searches
Data from exhaustive searches (Table 2) show no significant difference in time spent. Users of Colleague incurred a mean on-line cost per search of $4.72 compared with $6.97 for users of PaperChase (P < 0.004). Out of a possible 95 target references for these three searches, users of Colleague found a mean of TABLE
2
EXHAUSTIVE SEARCHES
Searches attempted“ On-line time (minutes/search) Search preparation (minutes/search) Total search time (minutes/search) On-line cost (dollars/search) Number of target references found Minutes per reference Dollars per reference Mean time to display first target reference
Colleague
PaperChase
18 9.4 0.1 9,s 4.71 276 0.6 0.31 7.9
IX 9.7 0.0 9.7 6.97 413 0.4 0.30 8.2
o Six participants each attempted all three searches. * Mann-Whitney I/ test: two-tailed P values with n, = 6, nZ = 6.
Mann-Whitney” P values n.c. n.5. n.a. n.5. 10.004 10.004 n.s. n.s n.s.
COMPUTERIZED
BIBLIOGRAPHIC TABLE
493
RETRIEVAL
3
CLINICAL SEARCHES Colleague
PaperChase
28 12.3 0.3 12.6 6.76 35 9.8 5.40 11.2
42 10.3 0.0 10.3 5.96 66 6.6 3.79 6.8
Searches attempted0 On-line time (minutes/search) Search preparation (minutes/search) Total search time (minutes/search) On-line cost (dollars/search) Number of target references found Minutes per reference Dollars per reference Mean time to display first target reference
Mann-Whitney* P values co.02 co.05 n.a. -co.05 co.03 co.003 <0.008 co.02 n.s.
u Colleague and PaperChase participants each attempted as many searches as they could within the time permitted. * Mann-Whitney U test: two-tailed P values with n, = 6, n2 = 6.
46 references, users of PaperChase a mean of 69 (P < 0.004). Thus, users of Colleague spent one-third less money while users of PaperChase produced more complete searches and obtained one-third more references. Mean on-line cost per target reference was $0.31 for users of Colleague, $0.30 for users of PaperChase. Mean on-line time to find each target reference was 0.6 min with Colleague compared with 0.4 min with PaperChase, a difference that was not statistically significant. Searches to Answer a Speci$c Clinical
Question
Data from searches to address a specific clinical question are summarized in Table 3. Users of Colleage took a mean of 9.8 min compared with 6.6 min for TABLE
4
SUMMARYOFALLSEARCHES
Searches attempted On-line time (minutes/search) Search preparation (minutes/search) Total search time (minutes/search) On-line cost (dollars/search) Number of target references found Minutes per reference Dollars per reference Mean time to display first target reference * Mann-Whitney
Colleague
PaperChase
64 11.4 0.6 12.0 6.12 326 2.2 1.20 11.1
78 9.8 0.0 9.8 5.86 496 1.6 0.92 7.0
I/ test: two-tailed P values with ni = 6,
n2
= 6.
Mann-Whitney* P values co.03 n.s. na. co.02 co.02 co.002 co.002 co.02 <0.004
494
PORTERETAL
PaperChase (P < 0.008), and spent a mean of $5.40 to find each target reference, whereas users of PaperChase spent $3.79 (P < 0.02). On average, users of PaperChase found their first target reference in 65% less total time (on-line time plus search preparation time) than did users of Colleague (6.8 versus Il.2 min). Users of PaperChase took less time to complete each clinical search and. because of shorter times on previous searches, they had more time available. As a result. they were able to attempt 42 of the clinical searches, compared with 28 for users of Colleague (P < 0.02).
Data for the entire study (Table 4) reveal that users of Colleague spent a total of $391.39 while attempting 64 searches and finding 326 target references. Each search attempted required a mean of 1 I .4 min on-line and cost $6.12. To find and print each target reference required a mean of 2.2 min on-line and cost $1.20. Users of PaperChase spent a total of $457.30 while attempting 78 searches and finding 496 target references. Each search attempted required a mean of 9.8 min on-line and cost $5.86. To find and print each target reference required a mean of I .6 min on-line and cost $0.92. Table 5 shows the number of Colleague and PaperChase participants who had time to attempt each of the test searches. On the first day, time ran out for one Colleague participant after search 4, for four more Colleague participants after search 6, and for the remaining Colleague participant after search 8. Time ran out for one PaperChase participant after search 7, for three more after search 8, and for the remaining two after search 9. None of the participants had sufficient time to attempt search 10. On the second day, four Colleague and five PaperChase participants were able to get to search 5. In every instance, searches were omitted only because time ran out. Table 6 shows, on a search-by-search basis, which system performed better as measured by the number of target references retrieved. minutes per target reference, dollars per target reference. and total time (on-line time plus search preparation time) to find the first target reference. For each measure, “c” indicates superior performance by users of Colleague, “p” TABLE NUMBER
OF COLLEAGUE
Search Day I 2
Colleague PaperChase Colleague PaperChase
5
AND PAPERCHASE PARTICIPANTS TEST SEARCH
WHO
ATTEMPTED
EACH
number
I
2
3
4
5
6
7
8
9
10
6 6 6 6
6 6 6 6
6 6 6 6
6 6 6 6
5 6 4 5
5 6
1 6
1 5
0 1
0 0
-
-
-
-
-
Total attempted 36 49 28 29
COMPUTERIZED
BIBLIOGRAPHIC TABLE
495
RETRIEVAL
6
SEARCH-BY-SEARCH DISPLAY OF PROGRAM WITH THE BETTER PERFORMANCE ON EACH OF FOURMEASUREMENTS(C = COLLEAGUE, p = PAPERCHASE) Searches Measures Day 1 Number d references Minut-m per reference DoLIars per reference Time to first reference Colleague totals PaperChase totals Day 2 Number of references Minutes per reference Dollars per reference Time to first reference Colleague totals PaperChase totals
Totals
123456789~~ Tie
P
P
P
P P 0 3
Tie C
C P
P P
P P C c
P P P P
0 4
2 2
0 4
p c c
-
p p p P P P p P P PPPP----OS
P P P P
P P P c
c c c p
0
1
3
0
-
4
3
I
4
-
-
-
_ -
p
-
1
6
p p p
-
I 2 2
1 6 6
0 2 2
4 3 3
2 0002---14442----
superior performance by users of PaperChase. On Day 1, PaperChase provided the better overall performance on six out of the eight searches that were undertaken by both Colleague and Paper-Chase participants. The exceptions were Search 7, for which Colleague showed a better performance on three out of four measures, and Search 3, for which the two systems were tied. No comparisons are possible for Search 9 because it was not attempted by any of the Colleague participants. On Day 2, PaperChase outperformed Colleague on three searches; Colleague outperformed PaperChase on one, and performance on the remaining search was tied. When performance of the two systems is compared by the type of search undertaken-single reference (Day 1, searches 1 and 2, and Day 2, search 1); exhaustive (Day 1, searches 3 and 4, and Day 2, search 2); and clinical question (Day 1, searches 5-8, and Day 2, searches 3-5)-PaperChase outperformed Colleague for single reference searches 8 measures to 2, for exhaustive searches 10 measures to 2, and for clinical problems, 22 measures to 6. For the entire study, PaperChase outperformed Colleague, 40 measures to 10 (Table 6). PaperChase and Colleague use different strategies to print results. With PaperChase, the most convenient method is to scan references and flag good ones to be printed at the end of the search. Colleague cannot flag references; users either scan and print alternately, keep track of references manually and print them later, or leave the printer on throughout the search. Many leave the printer on. The result is a mixture of relevant and extraneous references which
496
PORTER ET AL
makes the search harder to refine and the printout harder to use. In this study, users of PaperChase printed a mean of 6.9 references per search, 6.5 (94%) of which were on target; users of Colleague printed a mean of lS.O references, 5.2 (35%) of which were on target 0’ < 0.001). By questionnaire after each teaching session, the students reported equal satisfaction with the preparation offered by the Colleague and PaperChase representatives. At the end of their final search on Day 2. participants responded to a questionnaire that included: Have you enjoyed the experience? Given further access to Colleagues or PaperChase. would you conduct additional searches on topics of your own choice? Would you have a preference for one of the systems? All participants reported that they enjoyed learning how to search the medical literature. Ten of the 12 participants anticipated conducting their own searches in the near future. Nine expressed a preference for PaperChase (strong = 4, moderate = 3, slight = 3). two participants had a preference for Colleague (strong = 1. moderate = I), and one had no preference. DISCUSSION
The data from this study indicate that after 3 hr of concentrated instruction and practice, second-year medical students search MEDLINE more effectively with PaperChase than they do with Colleague. Our data can be compared with those of a previous study in which PaperChase was judged the easiest system to use, but also the most expensive (/3). In that study. research and reference librarians who had had years of prior experience with the computer programs provided by NLM, Dialog, and BRS, but less than 4 hr of prior experience with PaperChase, designed, formulated, and conducted all searches. The computer programs provided by NLM, Dialog, and BRS are similar to one another and to Colleague: PaperChase is markedly different. With the advantage of their training and expertise, the librarians searched quickly and efficiently with the programs provided by NLM, Dialog, and BRS. With PaperChase, by contrast. the librarians lacked the experience needed to set up their equipment properly for PaperChase. to print references all at once instead of one at a time, to type a few characters rather than entire words or phrases, to bypass the menus, and to employ the other techniques that would be expected with experience. In the present study, in which experience with Colleague and PaperChase was virtually identical, students who used PaperChase found more target references (496 vs 3261, and their on-line time per target reference (1.6 min vs 2.2 min) and on-line cost per target reference ($0.92 vs $1.20) were lower as well (Table 4). Among experienced searchers there is a widely held belief that a commanddriven interface, such as Colleague, permits more rapid and flexible searches than does a menu-driven interface, such as PaperChase. A command-driven interface saves time, since the user need not wait for menus to be painted on the screen. Both Colleague and PaperChase permit multiple commands to be “stacked” instead of being entered one at a time, but Colleague permits this to a greater extent than does PaperChase.
COMPUTERIZED
BIBLIOGRAPHIC
RETRIEVAL
497
On the other hand, computer techniques built into PaperChase help users conduct searches in ways that seem to overshadow benefits that may obtain from the command-driven interface of Colleague (8, 9). With Colleague, search questions and answers scroll off the screen; to refine a search, the user must either remember previous steps or enter a command to redisplay them. PaperChase keeps previous steps in continual view. To display references with Colleague requires entry of a series of commands to specify the search question, the format of the display, and the number of references to be displayed-all entered in the correct syntax. If the format excludes abstracts, the user who spots a promising title and wants to view the abstract has to halt the display and enter the entire series of commands again. Similarly, if the format includes abstracts, the user has to wait while irrelevant abstracts pass by in order to locate pertinent ones. With PaperChase, by contrast, the user displays references by typing three characters and displays abstracts selectively by typing one more. Colleague requires that each search statement be complete, accurate, and correctly spelled. Incomplete entries or inappropriate search terms retrieve no references. PaperChase accepts partial entries, offers alternative, often more appropriate search terms, and tries to deal with misspellings. When asked for their comments, students repeatedly mentioned that PaperChase helped them find better search terms. The data from this study were collected after 3 hr of concentrated instruction and practice. Had the period of instruction been shorter, the outcome would probably have favored PaperChase by a wider margin. Had it been longer, all users would have become more skillful1 and thus less dependent on the help that PaperChase provides. In these circumstances, differences in the performance of the two programs could be expected to narrow. Our analysis of cost was limited to on-line charges for daytime use in effect at the time of the study. Excluded were the costs of computer terminal or personal computer, printer, modem, telephone, paper, and ribbons. Both services provide lower rates for volume users and lower rates for students; both services, particularly Colleague, are less expensive at night. Had the study been conducted at night, there would have been no statistically significant difference in cost per target reference for the two services. At the time of the study, Colleague had a one-time subscription charge of $95, a monthly minimum payment of $15; PaperChase had neither a subscription charge nor a monthly minimum. At the time of the study, Colleague offered, in addition to the MEDLINE data base with searchable abstracts, the full text of 25 textbooks and recent issues of 80 journals. The scientist who needs data bases other than MEDLINE will not be satisfied with PaperChase alone. APPENDIX
Since this study was sponsored by Boston’s Beth Israel Hospital, the provider of PaperChase, extraordinary precautions were needed to guard against bias. To help readers make their own determination about the possibility of
498
PORTER
bias, we report here a description taken. Description
ET AL
of the searches and the precautions
that were
of Searches
On Day 1, the goal of the first two searches was to find the 1982 reference in disease of the pancreas and congenital adrenal hyperplasia, and to find the reference in Annals of Orology. Rhinology, and Luryngology that deals with gas chromatography of the middle ear. The goal of the next two searches was to prepare a complete bibliography of references that discuss the effects of Title 18 and Title 19 Programs on medical education, and on the role of interleukin 2 in patients who have Kaposi’s sarcoma due to AIDS. For the remaining searches, each student was presented with the following six clinical questions: Should Timolol be given to a patient who has had a myocardial infarction‘? Can sputum cytology be used to screen for lung cancer‘? Is CEA useful in detecting a recurrence of colon cancer? Can a patient with cold agglutinins also have warm antibodies‘? What are the hepatic manifestations of Osler-Rendu Disease? What is the effect, if any. of Tylenol on platelets? On Day 2, the goal of the first search was to find the reference in the Journal of Psychiatric Research that deals with eye movements as a possible marker for inherited susceptibility to schizophrenia. The goal of the next search was to prepare a complete bibliography of references that discuss whether therapy with psoralen and ultraviolet light can cause basal cell carcinoma. squamous cell carcinoma, or melanoma of the skin. The goal of the remaining three searches was to answer the following questions: Do smoking mothers have babies that are small for their gestational age’? Does the hepatitis B virus cause liver cancer? Which method of natural family planning works best? To help guard against searches that would play to the strength or weakness of either program, searches used in this study included a variety of topics and required, to the extent possible, a broad variety of strategies and techniques. Nine of the 15 searches were provided by us (single reference and exhaustive searches on both days. clinical questions 8-10 on Day I). and the remaining six were taken from a previous study of Haynes and his colleagues (13) (clinical questions 5-7 on Day 1 and 3-S on Day 2). Our search questions were developed from personal experience and from searches conducted by users of PaperChase. The search questions of Haynes and his colleagues were developed by research and reference librarians who had had years of experience with the programs provided by NLM, Dialog, and BRS-programs that resemble Colleague. Each search was pretested by us with the systems provided by Colleague. PaperChase, NLM, Dialog, and BRS. Any search that could not be done with each of these five systems was discarded. Searches used on the first day of the study were then tested by 10 experts-two experts for each system. Each expert was then asked if in his or her opinion any of the searches was biased Acta Paediatrica Scundinauica that deals with fibrocystic
COMPUTERIZED
BIBLIOGRAPHIC
RETRIEVAL
499
either for or against their system. No bias was reported. At the conclusion of each day’s session, the student participants were asked, “Do you think that the searches were biased against the search system you have just used?” There were no “yes” responses, 19 “no” responses, 1 “maybe,” and 3 “not sure” out of the 23 responses given on Days 1 and 2. To further check for bias in the searches, we compared the performance of the programs on searches developed by us with performance on searches developed by Haynes et al. When data from our searches are pooled, users of Colleague required 1.4 min per target reference, users of PaperChase required 0.96-a 31% reduction. Users of Colleague spent $0.72 to find each target reference, users of PaperChase spent $0.57-a 21% reduction. When data from Haynes’s searches are pooled, users of Colleague required 9.4 min per target reference, users of PaperChase required 5.8-a 38% reduction. Users of Colleague spent $5.17 per target reference, users of PaperChase spent $3.40-a 34% reduction. Thus, users of PaperChase outperformed users of Colleague by a wider margin with searches developed by Haynes et al. than they did with searches developed by us. Experimenter
Bias
The individual responsible for conduct of the experiment (D.P.) is an employee of Beth Israel Hospital, the provider of PaperChase. He is a research psychologist experienced in the design and execution of evaluation studies, with regular duties that do not include any responsibility for PaperChase. The study was conducted in collaboration with the Associate Dean of Academic Affairs and Graduate Medical Education at the University of Nebraska Medical Center (R.S.W.) and the Head of the Reference Department at the McGoogan Library (M.A.R.), neither of whom have any connection with either Colleague or PaperChase. The representatives of Colleague (Judy Consales) and PaperChase (Diane Winand) were consulted during the design of the study; both approved the experimental design and procedures, the suitability of the facilities, the equipment and communications software, the logistics of the experiment, and the questionnaire that was used to form equivalent participant groups. On-site decisions were made with the advice and consent of both representatives, including the assignment of students to study groups, schedules for teaching and testing, choice and setup of the experimental rooms, and the kinds of assistance that would be permitted during the test searches. Both Ms. Consales and Ms. Winand knew that their students would be asked to conduct a wide variety of MEDLINE searches, but neither had information about the search questions to be used for testing. The search questions were known to Ms. Anne Saunier, then Director of Marketing for Colleague, who showed them to at least one other person at Colleague, but not to Ms. Consales; the searches were known to the experimenter and to two other people at Beth Israel Hospital, but not to Ms. Winand.
500
PORTERETAL
EXPERIMENTAL
SUBJECTS
Experimental subjects and research assistants were recruited by one of us (R.S.W.) from the second-year class at the University of Nebraska College of Medicine. The students were given a detailed explanation of how the study was to be conducted and were consulted in setting the testing schedules and time limits. They were told that the study came from and was supported by Beth Israel Hospital, the organization that provides PaperChase, and they were asked to watch closely for bias or unfairness. At the end of each day’s session the students were asked by questionnaire if, in their opinion, the day’s activities had been biased or unfair. All 12 responded with “no” except for one “not sure” on each day. One “not sure” respondent had no comment; the other wrote, “I have little experience with running an experimental study. but 1 feel this one was well done with little bias and appropriate methods.” At the conclusion of the study, the students were paid by personal check from the experimenter. During administration of the test searches, Colleague and PaperChase students were distributed evenly throughout the afternoon time slots, and research assistants were assigned in symmetrical rotation to supervise the searches of both groups. In other matters. too, we tried to treat Colleague and PaperChase groups equally. Equal treatment was not possible in the training of research assistants, assignment of students to Colleague and PaperChase groups, space and equipment. and assistance provided during test searches: in each of these instances, we made the procedural decision that we believed to favor Colleague. Research assistants were trained by the experimenter on a Sunday afternoon. After being stepped through the testing protocol, they participated in a simulated testing session in which the experimenter posed dilemmas and asked questions that were typical of what they would face when the experiment began on the next day. Training began with PaperChase, using a printed protocol that was identical for both programs, except for a slight difference in the print commands. A similar training session with Colleague was planned, but because the experimenter had not checked the schedule, he did not realize that Colleague is unavailable on Sunday evenings. As a result, further simulations were run with PaperChase that emphasized the print commands of Colleague. Then, to ensure that the research assistants received at least as much exposure to Colleague as they had to PaperChase, the next morning all research assistants attended the complete training session of Colleague. Prior to the beginning of the study, each student was given an eight-item questionnaire that asked about previous education and experience with computers and with bibliographic retrieval. On the basis of their scores the students were paired as closely as possible, and the student with the higher score in each pair was assigned alternately to one group or the other. The mean scores of the two groups were 7.33 and 9.33 (U > 0.30, two-tailed. paired I test); the group with the higher score (the group that may have been more computer literate) was then assigned to Colleague on the first day.
COMPUTERIZED
BIBLIOGRAPHIC
RETRIEVAL
501
Two rooms were available for training. The six Colleague trainees had a room to themselves with seven Epson personal computers and printers on individual tables arranged in a curve that faced their instructor. The six PaperChase trainees had one end of a room with four IBM personal computers and printers on Day 1, five on Day 2, arranged on long tables facing a wall. Persons not connected with the study worked on other computers in the same room and conversed quietly. Also, because all testing took place in the room in which the Colleague students were trained, PaperChase students were tested in a different location and on different computers than were used during their instruction. During administration of the test searches, the experimenter circulated among the participants to answer questions that the research assistants could not handle. The Colleague representative was present to assist Colleague searchers with technical problems, but on the first day of the study the PaperChase representative had to be absent. As a result, PaperChase students may have received less technical assistance on the first day than did their Colleague counterparts. ACKNOWLEDGMENTS We thank Dr. Michael Feldstein of the Harvard School of Public Health for statistical consultation, Ms. Lisa Underhill for editorial assistance, and Dr. Clifford M. Goldsmith for his many helpful suggestions for improvement of the manuscript. REFERENCES 1. ROGERS, F. B. MEDLARS operating experience at the University of Colorado. Bull. Med. Libr. Assoc. 54(l), 1 (1966). 2. FEINGLOS, S. J. MEDLINE at BRS, DIALOG, and NLM: Is there a choice? Bull. Med. Libr. Assoc. 71(l), 6 (1983). 3. DOSZKOCS, T. E., RAPP, B. A., AND SCHOOLMAN, H. M. Automated information retrieval in science and technology. Science 208,25 (1980). 4. SEWELL, W., AND BEVAN, A. Nonmediated use of MEDLINE and TOXLINE by pathologists and pharmacists. BUN. Med. Libr. Assoc. 64(4), 382 (1976). 5. OLSON, P. E. Mechanization of library procedures in the medium-sized medical library. XV. A study of the interaction of nonlibrarian searches with the MEDLINE retrieval system. Bull. Med.
Libr.
Assoc.
63, 35 (1975).
6. WOOD, M. S., HORAK, E. B., AND SNOW, S. “End User Searching in the Health Sciences.” Haworth Press, New York, 1986. 7. ELIA, J. J. Another medium for the Journal (Editorial). N. Engl. J. Med. 311, 1631 (1984). 8. HOROWITZ. G. L., AND BLEICH, H. L. PaperChase: A computer program to search the medical literature. N. Engf. J. Med. 305, 924 (1981). 9. HOROWITZ, G. L., JACKSON, J. D., AND BLEICH, H. L. PaperChase: Self-service bibliographic retrieval. J. Amer. Med. Assoc. 250, 2494 (1983). 10. SEWELL, W., AND BEVAN, A. Nonmediated use of MEDLINE and TOXLINE by pathologists and pharmacists. 8~11. Med. Libr. Assoc. 64(4), 382 (1976). Il. BURROWS, S., AND KYLE. S. Searching the MEDLARS file on NLM and BRS. Bull. Med. Libr. Assoc. 67(l), 1.5 (1979). 12. FENICHEL, C. H. The process of searching online bibliographic data bases: A review of research. Libr. Res. 2, 107 (1980). 13. HAYNES, R. B., MCKIBBON, K. A., WALKER, C. J., ef a[. Computer searching of the medical literature. Ann. Infern. Med. 103(5), 812 (1985).
AND
BIOMEDICAL
RESEARCH
21, 488-501 ( 1988)
Self-Service Computerized Bibliographic Retrieval: Comparison of Colleague and PaperChase, Programs Search the MEDLlNE Data Base DOUGLAS
A that
PORTER,* ROBERT S. WIGTON,? MARIE A. R~IDELBACH,$ HOWARD L. BLEICH,* AND WARNER V. SLACK*
*Churles A. Dunu Resetrrch Institute and Hurvurd Thorndike Lahorutory. Beth Isrue Hospitul. Department of Medicine. Hurvard Medical School nt the Beth lsruel and Brigham und Women’s Hospituls, nnd the Center for Clinical Computing. Harvard Medical School. Boston. University of NLJhrrrska College oj Massachusetts 02115: tDepartment of Internal Medicine, Medicine. ONIU~LI, Nehruskr 68101; rend $McGoo,qarr Lihrtrry of’ Medicine. University I:/ Nchrcrsku Mcdicrrl Center. Omuhcr. NehnrsXtr 6KIOI
Received December 29, 1987
Colleague and PaperChase are the two most widely used computer systems designed specifically for clinicians and scientists who wish to search the National Library of Medicine’s MEDLINE data base of references to the biomedical literature. The present study compares the performance of these two systems. Two matched groups of second-year medical students each received 3 hr of instruction. one group in Colleague. the other in PaperChase. Each student then attempted IO test searches. The next day the groups were reversed, and each student attempted 5 additional searches. During 31 hr allocated for searching, users of Colleague attempted 64 test searches and retrieved 326 target references: users of PaperChase attempted 78 searches and retrieved 496. Users of Colleague took a mean of 2.2 min and spent a mean of $1.20 to find each target reference; users of PaperChase took I .6 min and spent $0.92. We conclude that after limited training. medical students find more references faster and at lower cost with PaperChase than with Colleague. I ILIXX Academic
Press. Inc.
In 1966, the National Library of Medicine (NLM) introduced MEDLINE, the computerized data base of references to the biomedical literature from which In&x Medicus is derived (I). Three years later, the National Library of Medicine developed ELHILL, a computer system that now provides interactive access to the MEDLINE data base by means of a computer terminal and telephone line. ELHILL, and its alternatives-computer systems provided by BRS Information Technologies (BRS) and DIALOG-are used principally by specially trained reference librarians who serve as intermediaries between the person who wants to search the literature and the MEDLINE data base. Mastery of these computer systems, and of companion volumes that describe how the medical literature is indexed, requires special training and experience 488 0010-4809188 $3.00 Copyright All rights
0 1988 by Academic Press. Inc. of reproduction in any form reserved.
COMPUTERIZED BIBLIOGRAPHIC
RETRIEVAL
489
(2). Efforts to promote the use of these computer systems by students, clinicians, and scientists without the assistance of a trained librarian have, for the most part, been unsuccessful (3-5). In recent years, programs for personal computers have been developed to help the inexperienced user perform searches. These programs, such as Sci-Mate, Pro-Search, FIRSTUSER, SEARCH HELPER, CANSearch, SearchMaster, Searchware, OASIS, GRATEFUL MED, and Med-Base, translate the computer commands of NLM, BRS, or DIALOG into choices from which the user can pick (6). Some of these programs also help the user find Medical Subject Headings, capture references in a personal computer for subsequent printing or searching, and reformat references in a variety of ways. Finally, three computer systems, easier to use than NLM, BRS, or DIALOG and designed specifically with the clinician or scientist in mind, permit end users to search the MEDLINE data base directly. DIALOG provides Medical Connection, BRS provides Colleague (7), and Boston’s Beth Israel Hospital provides PaperChase, which was developed in our laboratory (8, 9). Medical Connection was announced only recently; at present, Colleague and PaperChase provide most of the end-user searching of the MEDLINE data base. Previous studies of computer systems that search the MEDLINE data base have generally provided descriptive accounts of the nature and extent of users’ searches (2, 5, 10-12). In one study, however, data from searches of experienced librarians were used to advise clinicians on the choice of a system (13). To our knowledge there have been no controlled, prospective studies in which clinicians and scientists have formulated and conducted their searches. In the present study, we employed second-year medical students to compare the performance of Colleague with that of PaperChase in searching the MEDLINE data base. METHODS
The study was designed by the Center for Clinical Computing at Boston’s Beth Israel Hospital and conducted with cooperation from BRSlSaunders Colleague. PaperChase is a service of Boston’s Beth Israel Hospital, and Colleague is now a service of BRS Information Technologies, Inc. The study was conducted at the McGoogan Library of Medicine at the University of Nebraska Medical Center, Omaha, Nebraska, in collaboration with the Department of Internal Medicine. One of us (D.P.) was responsible for the conduct of the experiment in collaboration with representatives from Colleague and PaperChase. Because the study was sponsored by the same organization that provides PaperChase, special precautions were taken to guard against methods of procedure or accidents of execution that could have biased the results in favor of PaperChase (Appendix). The study was conducted in two adjoining rooms, each equipped with personal computers, modems, telephone lines, and communications software (Smartcorn II) programmed to access Colleague in Latham, New York, and PaperChase in Boston, Massachusetts.
490
PORTER
ET AL.
Twelve second-year medical students at the University of Nebraska were each paid $100 to participate. Before the study, each student answered a questionnaire about education and experience with computers. and on the basis of their answers, the students were divided into two equivalent groups of six students each. No student reported prior experience with computerized bibliographic retrieval. Five additional second-year medical students were each paid $150 to work as research assistants: they administered the test searches, observed and clocked the behavior of the participants, and helped with the computer equipment. During the morning of the first day, one group of students was instructed in Colleague by the representative of Colleague. the other group in PaperChase by the representative of PaperChase. That afternoon, each student was asked to use the system in which he or she had been instructed to conduct 10 test searches. The next day, students experienced with Colleague were instructed in PaperChase, and students experienced with PaperChase were instructed in Colleague. That afternoon, five new test searches were administered. Time available to perform searches was limited to 2 hr on the first day and to 1Bhr on the second day. Selection
of Test Semches
Six of the test searches were adapted from a previous study (IS). and the remainder were chosen by the investigators to be representative of those that a physician or scientist would require. The goal of the first two searches was to retrieve a specific reference from fragmentary information: the goal of the next two searches was to produce, to the extent possible, an exhaustive bibliography about a topic-one suitable for use in writing a review article; and the goal of the final six searches was to answer a specific clinical question. On the second day. the goal of the first search was to find a specific reference from fragmentary information, the next to perform an exhaustive retrieval. and the final three to answer a specific clinical question. For searches to answer a clinical question, the students were asked to print the first two references that. on the basis of their titles and abstracts, seemed relevant and that were published in one of the journals on a core list that we obtained from the National Library of Medicine. Preliminary testing indicated that with sufficient expertise each search could be done on Colleague or PaperChase within a few minutes. On the other hand, the task for the students was difficult; there were medical school librarians with years of experience with the MEDLINE data base who could not perform all 10 searches of the first day in the time allocated to the students. With each search, the participant had to formulate the strategy, conduct the search, signal when the first target reference was found, and print out the references found. The performance of each participant was timed with a stopwatch: additional data concerning time and cost were obtained from printouts produced by the computer programs. All searches were conducted over the entire MEDLINE
COMPUTERIZED
BIBLIOGRAPHIC
RETRIEVAL
491
file, 1966 to the present; a description of each search is provided in the Appendix. After each teaching session, students answered a brief questionnaire about the instruction they had just received; immediately after each searching session, they answered a questionnaire about their reactions to the searches; and at the conclusion of the study, they completed a final questionnaire about their participation in the study and reactions to Colleague and PaperChase. Evaluation
of References
Retrieved
We examined each reference printed to see if it was a target reference. Searches to find a specific reference on the basis of fragmentary information had but one possible target reference. References printed during an exhaustive search were deemed on target if they included the appropriate Medical Subject Headings or text words (title and abstract words). References retrieved to answer a specific chnical question were deemed on target if their titles or abstracts pertained directly to the clinical question posed, and if they were published in one of the high quality journals on the list that we provided. A panel of three blinded physicians used identically formatted printouts to arbitrate questionable calls. To compare Colleague and PaperChase, we considered three correlated measures of performanceon-line time, on-line cost, and number of target references printed. If no target references were printed, time and cost were still included. To compare on-line time and on-line cost across searches that retrieved different numbers of target references, we converted these measures into minutes per reference and dollars per reference. As a measure of the time required to formulate and enter the logic of the searches, we computed the total time-including search preparation time- to display the first target reference. Unless otherwise specified, we averaged the performance of each student for each type of search and then submitted these mean values to the MannWhitney U test, with nl = 6, n2 = 6, to compute two-tailed P values. RESULTS
Both for Colleague and for PaperChase, we compared results of the searches on the first day with results on the second day and found no statistically significant differences between the 2 days on any measure of performance for either program. Mean two-tailed P values were 0.51 for Colleague, 0.31 for PaperChase (Mann-Whitney U tests with IZI = 6, 122 = 6). Accordingly, unless otherwise specified, the data from the 2 days were pooled. Searches for a Single Reference
Data from searches for a single reference are summarized in Table 1. When only a single reference was to be retrieved, differences in performance were measured in time and cost, not in the number of references found. On-line time
492
PORTER ET AL TABLE
1
SINGLE REFERENCE S~ARC.HES
Searches attempted” On-line time (minutes/search) Search preparation (minutes/search) Total search time (minutes/search) On-line cost (dollars/search) Number of target references found Minutes per reference Dollars per reference Mean time to display first target reference
Colleague
PaperChase
I8 12.1 I .s Ii.6 h.S? I5 16.3 7.82 14.8
18 8.Y 0.0 8.Y I.Ci I? Y.3 4.79 6.3
---
__Mann-Whitney’ P values
/’ Six participants each attempted all three searches. * Mann-Whitney U test: two-tailed P values with n, = 6. II? = h
averaged 14.6 min for users of Colleague, 9.4 min for users of PaperChase (P <0.05. Mann-LWhitney U test). On-line cost averaged $7.82 for users of Colleague and $4.79 for users of PaperChase (P < 0.02). In 15 out of 18 searches users of Colleague found the required reference, compared with 17 out of 18 for users of PaperChase. Exhnustivr
Searches
Data from exhaustive searches (Table 2) show no significant difference in time spent. Users of Colleague incurred a mean on-line cost per search of $4.72 compared with $6.97 for users of PaperChase (P < 0.004). Out of a possible 95 target references for these three searches, users of Colleague found a mean of TABLE
2
EXHAUSTIVE SEARCHES
Searches attempted“ On-line time (minutes/search) Search preparation (minutes/search) Total search time (minutes/search) On-line cost (dollars/search) Number of target references found Minutes per reference Dollars per reference Mean time to display first target reference
Colleague
PaperChase
18 9.4 0.1 9,s 4.71 276 0.6 0.31 7.9
IX 9.7 0.0 9.7 6.97 413 0.4 0.30 8.2
o Six participants each attempted all three searches. * Mann-Whitney I/ test: two-tailed P values with n, = 6, nZ = 6.
Mann-Whitney” P values n.c. n.5. n.a. n.5. 10.004 10.004 n.s. n.s n.s.
COMPUTERIZED
BIBLIOGRAPHIC TABLE
493
RETRIEVAL
3
CLINICAL SEARCHES Colleague
PaperChase
28 12.3 0.3 12.6 6.76 35 9.8 5.40 11.2
42 10.3 0.0 10.3 5.96 66 6.6 3.79 6.8
Searches attempted0 On-line time (minutes/search) Search preparation (minutes/search) Total search time (minutes/search) On-line cost (dollars/search) Number of target references found Minutes per reference Dollars per reference Mean time to display first target reference
Mann-Whitney* P values co.02 co.05 n.a. -co.05 co.03 co.003 <0.008 co.02 n.s.
u Colleague and PaperChase participants each attempted as many searches as they could within the time permitted. * Mann-Whitney U test: two-tailed P values with n, = 6, n2 = 6.
46 references, users of PaperChase a mean of 69 (P < 0.004). Thus, users of Colleague spent one-third less money while users of PaperChase produced more complete searches and obtained one-third more references. Mean on-line cost per target reference was $0.31 for users of Colleague, $0.30 for users of PaperChase. Mean on-line time to find each target reference was 0.6 min with Colleague compared with 0.4 min with PaperChase, a difference that was not statistically significant. Searches to Answer a Speci$c Clinical
Question
Data from searches to address a specific clinical question are summarized in Table 3. Users of Colleage took a mean of 9.8 min compared with 6.6 min for TABLE
4
SUMMARYOFALLSEARCHES
Searches attempted On-line time (minutes/search) Search preparation (minutes/search) Total search time (minutes/search) On-line cost (dollars/search) Number of target references found Minutes per reference Dollars per reference Mean time to display first target reference * Mann-Whitney
Colleague
PaperChase
64 11.4 0.6 12.0 6.12 326 2.2 1.20 11.1
78 9.8 0.0 9.8 5.86 496 1.6 0.92 7.0
I/ test: two-tailed P values with ni = 6,
n2
= 6.
Mann-Whitney* P values co.03 n.s. na. co.02 co.02 co.002 co.002 co.02 <0.004
494
PORTERETAL
PaperChase (P < 0.008), and spent a mean of $5.40 to find each target reference, whereas users of PaperChase spent $3.79 (P < 0.02). On average, users of PaperChase found their first target reference in 65% less total time (on-line time plus search preparation time) than did users of Colleague (6.8 versus Il.2 min). Users of PaperChase took less time to complete each clinical search and. because of shorter times on previous searches, they had more time available. As a result. they were able to attempt 42 of the clinical searches, compared with 28 for users of Colleague (P < 0.02).
Data for the entire study (Table 4) reveal that users of Colleague spent a total of $391.39 while attempting 64 searches and finding 326 target references. Each search attempted required a mean of 1 I .4 min on-line and cost $6.12. To find and print each target reference required a mean of 2.2 min on-line and cost $1.20. Users of PaperChase spent a total of $457.30 while attempting 78 searches and finding 496 target references. Each search attempted required a mean of 9.8 min on-line and cost $5.86. To find and print each target reference required a mean of I .6 min on-line and cost $0.92. Table 5 shows the number of Colleague and PaperChase participants who had time to attempt each of the test searches. On the first day, time ran out for one Colleague participant after search 4, for four more Colleague participants after search 6, and for the remaining Colleague participant after search 8. Time ran out for one PaperChase participant after search 7, for three more after search 8, and for the remaining two after search 9. None of the participants had sufficient time to attempt search 10. On the second day, four Colleague and five PaperChase participants were able to get to search 5. In every instance, searches were omitted only because time ran out. Table 6 shows, on a search-by-search basis, which system performed better as measured by the number of target references retrieved. minutes per target reference, dollars per target reference. and total time (on-line time plus search preparation time) to find the first target reference. For each measure, “c” indicates superior performance by users of Colleague, “p” TABLE NUMBER
OF COLLEAGUE
Search Day I 2
Colleague PaperChase Colleague PaperChase
5
AND PAPERCHASE PARTICIPANTS TEST SEARCH
WHO
ATTEMPTED
EACH
number
I
2
3
4
5
6
7
8
9
10
6 6 6 6
6 6 6 6
6 6 6 6
6 6 6 6
5 6 4 5
5 6
1 6
1 5
0 1
0 0
-
-
-
-
-
Total attempted 36 49 28 29
COMPUTERIZED
BIBLIOGRAPHIC TABLE
495
RETRIEVAL
6
SEARCH-BY-SEARCH DISPLAY OF PROGRAM WITH THE BETTER PERFORMANCE ON EACH OF FOURMEASUREMENTS(C = COLLEAGUE, p = PAPERCHASE) Searches Measures Day 1 Number d references Minut-m per reference DoLIars per reference Time to first reference Colleague totals PaperChase totals Day 2 Number of references Minutes per reference Dollars per reference Time to first reference Colleague totals PaperChase totals
Totals
123456789~~ Tie
P
P
P
P P 0 3
Tie C
C P
P P
P P C c
P P P P
0 4
2 2
0 4
p c c
-
p p p P P P p P P PPPP----OS
P P P P
P P P c
c c c p
0
1
3
0
-
4
3
I
4
-
-
-
_ -
p
-
1
6
p p p
-
I 2 2
1 6 6
0 2 2
4 3 3
2 0002---14442----
superior performance by users of PaperChase. On Day 1, PaperChase provided the better overall performance on six out of the eight searches that were undertaken by both Colleague and Paper-Chase participants. The exceptions were Search 7, for which Colleague showed a better performance on three out of four measures, and Search 3, for which the two systems were tied. No comparisons are possible for Search 9 because it was not attempted by any of the Colleague participants. On Day 2, PaperChase outperformed Colleague on three searches; Colleague outperformed PaperChase on one, and performance on the remaining search was tied. When performance of the two systems is compared by the type of search undertaken-single reference (Day 1, searches 1 and 2, and Day 2, search 1); exhaustive (Day 1, searches 3 and 4, and Day 2, search 2); and clinical question (Day 1, searches 5-8, and Day 2, searches 3-5)-PaperChase outperformed Colleague for single reference searches 8 measures to 2, for exhaustive searches 10 measures to 2, and for clinical problems, 22 measures to 6. For the entire study, PaperChase outperformed Colleague, 40 measures to 10 (Table 6). PaperChase and Colleague use different strategies to print results. With PaperChase, the most convenient method is to scan references and flag good ones to be printed at the end of the search. Colleague cannot flag references; users either scan and print alternately, keep track of references manually and print them later, or leave the printer on throughout the search. Many leave the printer on. The result is a mixture of relevant and extraneous references which
496
PORTER ET AL
makes the search harder to refine and the printout harder to use. In this study, users of PaperChase printed a mean of 6.9 references per search, 6.5 (94%) of which were on target; users of Colleague printed a mean of lS.O references, 5.2 (35%) of which were on target 0’ < 0.001). By questionnaire after each teaching session, the students reported equal satisfaction with the preparation offered by the Colleague and PaperChase representatives. At the end of their final search on Day 2. participants responded to a questionnaire that included: Have you enjoyed the experience? Given further access to Colleagues or PaperChase. would you conduct additional searches on topics of your own choice? Would you have a preference for one of the systems? All participants reported that they enjoyed learning how to search the medical literature. Ten of the 12 participants anticipated conducting their own searches in the near future. Nine expressed a preference for PaperChase (strong = 4, moderate = 3, slight = 3). two participants had a preference for Colleague (strong = 1. moderate = I), and one had no preference. DISCUSSION
The data from this study indicate that after 3 hr of concentrated instruction and practice, second-year medical students search MEDLINE more effectively with PaperChase than they do with Colleague. Our data can be compared with those of a previous study in which PaperChase was judged the easiest system to use, but also the most expensive (/3). In that study. research and reference librarians who had had years of prior experience with the computer programs provided by NLM, Dialog, and BRS, but less than 4 hr of prior experience with PaperChase, designed, formulated, and conducted all searches. The computer programs provided by NLM, Dialog, and BRS are similar to one another and to Colleague: PaperChase is markedly different. With the advantage of their training and expertise, the librarians searched quickly and efficiently with the programs provided by NLM, Dialog, and BRS. With PaperChase, by contrast. the librarians lacked the experience needed to set up their equipment properly for PaperChase. to print references all at once instead of one at a time, to type a few characters rather than entire words or phrases, to bypass the menus, and to employ the other techniques that would be expected with experience. In the present study, in which experience with Colleague and PaperChase was virtually identical, students who used PaperChase found more target references (496 vs 3261, and their on-line time per target reference (1.6 min vs 2.2 min) and on-line cost per target reference ($0.92 vs $1.20) were lower as well (Table 4). Among experienced searchers there is a widely held belief that a commanddriven interface, such as Colleague, permits more rapid and flexible searches than does a menu-driven interface, such as PaperChase. A command-driven interface saves time, since the user need not wait for menus to be painted on the screen. Both Colleague and PaperChase permit multiple commands to be “stacked” instead of being entered one at a time, but Colleague permits this to a greater extent than does PaperChase.
COMPUTERIZED
BIBLIOGRAPHIC
RETRIEVAL
497
On the other hand, computer techniques built into PaperChase help users conduct searches in ways that seem to overshadow benefits that may obtain from the command-driven interface of Colleague (8, 9). With Colleague, search questions and answers scroll off the screen; to refine a search, the user must either remember previous steps or enter a command to redisplay them. PaperChase keeps previous steps in continual view. To display references with Colleague requires entry of a series of commands to specify the search question, the format of the display, and the number of references to be displayed-all entered in the correct syntax. If the format excludes abstracts, the user who spots a promising title and wants to view the abstract has to halt the display and enter the entire series of commands again. Similarly, if the format includes abstracts, the user has to wait while irrelevant abstracts pass by in order to locate pertinent ones. With PaperChase, by contrast, the user displays references by typing three characters and displays abstracts selectively by typing one more. Colleague requires that each search statement be complete, accurate, and correctly spelled. Incomplete entries or inappropriate search terms retrieve no references. PaperChase accepts partial entries, offers alternative, often more appropriate search terms, and tries to deal with misspellings. When asked for their comments, students repeatedly mentioned that PaperChase helped them find better search terms. The data from this study were collected after 3 hr of concentrated instruction and practice. Had the period of instruction been shorter, the outcome would probably have favored PaperChase by a wider margin. Had it been longer, all users would have become more skillful1 and thus less dependent on the help that PaperChase provides. In these circumstances, differences in the performance of the two programs could be expected to narrow. Our analysis of cost was limited to on-line charges for daytime use in effect at the time of the study. Excluded were the costs of computer terminal or personal computer, printer, modem, telephone, paper, and ribbons. Both services provide lower rates for volume users and lower rates for students; both services, particularly Colleague, are less expensive at night. Had the study been conducted at night, there would have been no statistically significant difference in cost per target reference for the two services. At the time of the study, Colleague had a one-time subscription charge of $95, a monthly minimum payment of $15; PaperChase had neither a subscription charge nor a monthly minimum. At the time of the study, Colleague offered, in addition to the MEDLINE data base with searchable abstracts, the full text of 25 textbooks and recent issues of 80 journals. The scientist who needs data bases other than MEDLINE will not be satisfied with PaperChase alone. APPENDIX
Since this study was sponsored by Boston’s Beth Israel Hospital, the provider of PaperChase, extraordinary precautions were needed to guard against bias. To help readers make their own determination about the possibility of
498
PORTER
bias, we report here a description taken. Description
ET AL
of the searches and the precautions
that were
of Searches
On Day 1, the goal of the first two searches was to find the 1982 reference in disease of the pancreas and congenital adrenal hyperplasia, and to find the reference in Annals of Orology. Rhinology, and Luryngology that deals with gas chromatography of the middle ear. The goal of the next two searches was to prepare a complete bibliography of references that discuss the effects of Title 18 and Title 19 Programs on medical education, and on the role of interleukin 2 in patients who have Kaposi’s sarcoma due to AIDS. For the remaining searches, each student was presented with the following six clinical questions: Should Timolol be given to a patient who has had a myocardial infarction‘? Can sputum cytology be used to screen for lung cancer‘? Is CEA useful in detecting a recurrence of colon cancer? Can a patient with cold agglutinins also have warm antibodies‘? What are the hepatic manifestations of Osler-Rendu Disease? What is the effect, if any. of Tylenol on platelets? On Day 2, the goal of the first search was to find the reference in the Journal of Psychiatric Research that deals with eye movements as a possible marker for inherited susceptibility to schizophrenia. The goal of the next search was to prepare a complete bibliography of references that discuss whether therapy with psoralen and ultraviolet light can cause basal cell carcinoma. squamous cell carcinoma, or melanoma of the skin. The goal of the remaining three searches was to answer the following questions: Do smoking mothers have babies that are small for their gestational age’? Does the hepatitis B virus cause liver cancer? Which method of natural family planning works best? To help guard against searches that would play to the strength or weakness of either program, searches used in this study included a variety of topics and required, to the extent possible, a broad variety of strategies and techniques. Nine of the 15 searches were provided by us (single reference and exhaustive searches on both days. clinical questions 8-10 on Day I). and the remaining six were taken from a previous study of Haynes and his colleagues (13) (clinical questions 5-7 on Day 1 and 3-S on Day 2). Our search questions were developed from personal experience and from searches conducted by users of PaperChase. The search questions of Haynes and his colleagues were developed by research and reference librarians who had had years of experience with the programs provided by NLM, Dialog, and BRS-programs that resemble Colleague. Each search was pretested by us with the systems provided by Colleague. PaperChase, NLM, Dialog, and BRS. Any search that could not be done with each of these five systems was discarded. Searches used on the first day of the study were then tested by 10 experts-two experts for each system. Each expert was then asked if in his or her opinion any of the searches was biased Acta Paediatrica Scundinauica that deals with fibrocystic
COMPUTERIZED
BIBLIOGRAPHIC
RETRIEVAL
499
either for or against their system. No bias was reported. At the conclusion of each day’s session, the student participants were asked, “Do you think that the searches were biased against the search system you have just used?” There were no “yes” responses, 19 “no” responses, 1 “maybe,” and 3 “not sure” out of the 23 responses given on Days 1 and 2. To further check for bias in the searches, we compared the performance of the programs on searches developed by us with performance on searches developed by Haynes et al. When data from our searches are pooled, users of Colleague required 1.4 min per target reference, users of PaperChase required 0.96-a 31% reduction. Users of Colleague spent $0.72 to find each target reference, users of PaperChase spent $0.57-a 21% reduction. When data from Haynes’s searches are pooled, users of Colleague required 9.4 min per target reference, users of PaperChase required 5.8-a 38% reduction. Users of Colleague spent $5.17 per target reference, users of PaperChase spent $3.40-a 34% reduction. Thus, users of PaperChase outperformed users of Colleague by a wider margin with searches developed by Haynes et al. than they did with searches developed by us. Experimenter
Bias
The individual responsible for conduct of the experiment (D.P.) is an employee of Beth Israel Hospital, the provider of PaperChase. He is a research psychologist experienced in the design and execution of evaluation studies, with regular duties that do not include any responsibility for PaperChase. The study was conducted in collaboration with the Associate Dean of Academic Affairs and Graduate Medical Education at the University of Nebraska Medical Center (R.S.W.) and the Head of the Reference Department at the McGoogan Library (M.A.R.), neither of whom have any connection with either Colleague or PaperChase. The representatives of Colleague (Judy Consales) and PaperChase (Diane Winand) were consulted during the design of the study; both approved the experimental design and procedures, the suitability of the facilities, the equipment and communications software, the logistics of the experiment, and the questionnaire that was used to form equivalent participant groups. On-site decisions were made with the advice and consent of both representatives, including the assignment of students to study groups, schedules for teaching and testing, choice and setup of the experimental rooms, and the kinds of assistance that would be permitted during the test searches. Both Ms. Consales and Ms. Winand knew that their students would be asked to conduct a wide variety of MEDLINE searches, but neither had information about the search questions to be used for testing. The search questions were known to Ms. Anne Saunier, then Director of Marketing for Colleague, who showed them to at least one other person at Colleague, but not to Ms. Consales; the searches were known to the experimenter and to two other people at Beth Israel Hospital, but not to Ms. Winand.
500
PORTERETAL
EXPERIMENTAL
SUBJECTS
Experimental subjects and research assistants were recruited by one of us (R.S.W.) from the second-year class at the University of Nebraska College of Medicine. The students were given a detailed explanation of how the study was to be conducted and were consulted in setting the testing schedules and time limits. They were told that the study came from and was supported by Beth Israel Hospital, the organization that provides PaperChase, and they were asked to watch closely for bias or unfairness. At the end of each day’s session the students were asked by questionnaire if, in their opinion, the day’s activities had been biased or unfair. All 12 responded with “no” except for one “not sure” on each day. One “not sure” respondent had no comment; the other wrote, “I have little experience with running an experimental study. but 1 feel this one was well done with little bias and appropriate methods.” At the conclusion of the study, the students were paid by personal check from the experimenter. During administration of the test searches, Colleague and PaperChase students were distributed evenly throughout the afternoon time slots, and research assistants were assigned in symmetrical rotation to supervise the searches of both groups. In other matters. too, we tried to treat Colleague and PaperChase groups equally. Equal treatment was not possible in the training of research assistants, assignment of students to Colleague and PaperChase groups, space and equipment. and assistance provided during test searches: in each of these instances, we made the procedural decision that we believed to favor Colleague. Research assistants were trained by the experimenter on a Sunday afternoon. After being stepped through the testing protocol, they participated in a simulated testing session in which the experimenter posed dilemmas and asked questions that were typical of what they would face when the experiment began on the next day. Training began with PaperChase, using a printed protocol that was identical for both programs, except for a slight difference in the print commands. A similar training session with Colleague was planned, but because the experimenter had not checked the schedule, he did not realize that Colleague is unavailable on Sunday evenings. As a result, further simulations were run with PaperChase that emphasized the print commands of Colleague. Then, to ensure that the research assistants received at least as much exposure to Colleague as they had to PaperChase, the next morning all research assistants attended the complete training session of Colleague. Prior to the beginning of the study, each student was given an eight-item questionnaire that asked about previous education and experience with computers and with bibliographic retrieval. On the basis of their scores the students were paired as closely as possible, and the student with the higher score in each pair was assigned alternately to one group or the other. The mean scores of the two groups were 7.33 and 9.33 (U > 0.30, two-tailed. paired I test); the group with the higher score (the group that may have been more computer literate) was then assigned to Colleague on the first day.
COMPUTERIZED
BIBLIOGRAPHIC
RETRIEVAL
501
Two rooms were available for training. The six Colleague trainees had a room to themselves with seven Epson personal computers and printers on individual tables arranged in a curve that faced their instructor. The six PaperChase trainees had one end of a room with four IBM personal computers and printers on Day 1, five on Day 2, arranged on long tables facing a wall. Persons not connected with the study worked on other computers in the same room and conversed quietly. Also, because all testing took place in the room in which the Colleague students were trained, PaperChase students were tested in a different location and on different computers than were used during their instruction. During administration of the test searches, the experimenter circulated among the participants to answer questions that the research assistants could not handle. The Colleague representative was present to assist Colleague searchers with technical problems, but on the first day of the study the PaperChase representative had to be absent. As a result, PaperChase students may have received less technical assistance on the first day than did their Colleague counterparts. ACKNOWLEDGMENTS We thank Dr. Michael Feldstein of the Harvard School of Public Health for statistical consultation, Ms. Lisa Underhill for editorial assistance, and Dr. Clifford M. Goldsmith for his many helpful suggestions for improvement of the manuscript. REFERENCES 1. ROGERS, F. B. MEDLARS operating experience at the University of Colorado. Bull. Med. Libr. Assoc. 54(l), 1 (1966). 2. FEINGLOS, S. J. MEDLINE at BRS, DIALOG, and NLM: Is there a choice? Bull. Med. Libr. Assoc. 71(l), 6 (1983). 3. DOSZKOCS, T. E., RAPP, B. A., AND SCHOOLMAN, H. M. Automated information retrieval in science and technology. Science 208,25 (1980). 4. SEWELL, W., AND BEVAN, A. Nonmediated use of MEDLINE and TOXLINE by pathologists and pharmacists. BUN. Med. Libr. Assoc. 64(4), 382 (1976). 5. OLSON, P. E. Mechanization of library procedures in the medium-sized medical library. XV. A study of the interaction of nonlibrarian searches with the MEDLINE retrieval system. Bull. Med.
Libr.
Assoc.
63, 35 (1975).
6. WOOD, M. S., HORAK, E. B., AND SNOW, S. “End User Searching in the Health Sciences.” Haworth Press, New York, 1986. 7. ELIA, J. J. Another medium for the Journal (Editorial). N. Engl. J. Med. 311, 1631 (1984). 8. HOROWITZ. G. L., AND BLEICH, H. L. PaperChase: A computer program to search the medical literature. N. Engf. J. Med. 305, 924 (1981). 9. HOROWITZ, G. L., JACKSON, J. D., AND BLEICH, H. L. PaperChase: Self-service bibliographic retrieval. J. Amer. Med. Assoc. 250, 2494 (1983). 10. SEWELL, W., AND BEVAN, A. Nonmediated use of MEDLINE and TOXLINE by pathologists and pharmacists. 8~11. Med. Libr. Assoc. 64(4), 382 (1976). Il. BURROWS, S., AND KYLE. S. Searching the MEDLARS file on NLM and BRS. Bull. Med. Libr. Assoc. 67(l), 1.5 (1979). 12. FENICHEL, C. H. The process of searching online bibliographic data bases: A review of research. Libr. Res. 2, 107 (1980). 13. HAYNES, R. B., MCKIBBON, K. A., WALKER, C. J., ef a[. Computer searching of the medical literature. Ann. Infern. Med. 103(5), 812 (1985).