- Email: [email protected]
What difference does it make? One study of student background and the evaluation of library instruction
What Difference Does It Make? One Study of Student Background and the Evaluation of Library Instruction by Cathy Moore-Jansen
The results of a six-pr study indicate little relationship between students’ demographics, previous libray instruction, or prior use of libray resources and how they emluated libray instruction forone undergraduate anthropology wmse. There is a more positive relationship betuxen subject interest and evaluation of the instruction. These results suggest that librarians should be less wmxrned about students’ prior experience and more concerned with ensuring that course-related libray instruction focuses on specific goals and reinforces student interest in the subject.
A
s long as librarians provide library instruction, they will be concerned with its effectiveness. For students evaluating its effectiveness, however, the quality or focus of the instruction might matter less than the wide variety of experiences and interests that they bring to an instructional setting. Upper-level undergraduates and graduate students might, for example, find library instruction less useful than students just beginning an academic career. Students who have had previous experience with library instruction or resources might find additional instruction repetitive and less useful than Cathy Moore-Jansen is Social Sciences Librarian, Wichita State University, Wichita, KS. E-mail: [email protected] Research Strategies, vol. 15, no. 1, pp. 26-38 Q1997 by Research Strategies. All rights reserved.
students receiving formal instruction or using the library for the first time. Students with more interest in the subject matter of the course might rate the instruction more useful than those with less. On the other hand, how students evaluate library instruction related to a required assignment for a particular course may have less to do with individual background or interests and more with how well the instruction helped them complete the assignment. To explore the impact of students’ backgrounds and interests on their evaluation of library instruction, a six-year study was conducted to examine possible relationships between variables related to student demographics, library experience, and subject interests and variables related to student evaluation of library instruction in one undergraduate anthropology course. The intent of this study was to document, if possible, statistically significant relationships among variables-relationships that may or may not seem “obvious,” depending on one’s perspective of library instruction. The method involved collecting data on students taking one course over an extended period of time and seeking statistically significant influences on how useful they perceived the library instruction to be. Such influences could then be taken into account when planning library instruction for this and similar courses. SAMPLE POPULATION AND DESCRIPTION OF ASSIGNMENT The students surveyed in this study were enrolled in the anthropology course Biological Anthropology, lOlQ, at Wichita State University during the summers of 1990 through 1995. They take this course as a science or anthropology requirement, as an elective, or for a combination of reasons. The subject content of the course, which is taught each semester by the same professor, includes human and nonprimate evolution, fossil evidence, primates, osteology, forensics, human variability, and genetics. In this course students are exposed to the scholarly periodical literature and reference sources in biological anthropology. All students are required to write reviews of scholarly journal articles related to some aspect of biological anthropology. Since 1993, students have also been asked to locate article citations on specific topics using periodical indexes. This assignment gives the students needed practice in locating journals and articles on topics that interest them and discourages them from simply browsing journals for articles to review. From the beginning of the study, the library instruction stressed how to locate specific journal titles and journals on a general topic such as “primates” in the online catalog or periodical listing printout as well as journal articles on a specific topic such as “blood genetic markers in human populations” in periodical indexes.
28
RESEARCHSTRATEGIES
Winter1997
The students enrolled in the undergraduate anthropology course during this study represented all academic levels, numerous different majors, and several nationalities. Additional assumptions or expectations about the group are that: they had diverse backgrounds and interests; * some had had previous library instruction, in various disciplines; and l they had varying degrees of interest in the subject content of the course.
l
To explore how these diverse elements might influence the usefulness of library instruction in helping students do their assignments, preand post-library instruction surveys were developed that posed questions about demographic attributes such as major and academic status, prior experience with library instruction and resources, and general evaluation of the library instruction. The primary objectives of the study were: . to determine if the library instruction was useful in helping students do their assignment; l to describe the students in terms of demographics, previous library experience, and subject interests; and l to look for significant relationships between all of those areas and how students evaluated the library instruction. LITERATURERRVIEW The literature was reviewed for studies that used a surveying, rather than testing, methodology for evaluating library instruction. Many such efforts distributed questionnaires to evaluate how “effective” a particular library class or program was in teaching library skills and to gather student opinions about certain aspects of that library instruction.’ Opinion surveys have collected data on variables related to age, gender, language, academic status, major, previous instruction and experience with library research, library skill levels, attitudes towards libraries or librarians, student confidence in using the library, resources used before or during the instruction, usefulness of different parts of the instruction, level of satisfaction with the library instruction, and how easy or difficult the library is to use. Studies that explore relationships between these variables are harder to find. Piette and Dance concluded that graduate students with no or some library experience were more satisfied with an information workshop than were students with more experience. On the other hand, students with greater experience were more satisfied
Maore-Jamen 29
with a database searching workshop than were less experienced students. Whether or not the students were native English speakers made no difference in their satisfaction with either workshop.2 Ackerson, Howard, and Young examined the relationship between the amount of previous library instruction students had had and the quality of their bibliographies, and found conflicting preliminary resu1ts.3 Trail and Gutierrez studied several descriptive variables, including gender, academic status and division, and previous library instruction, to see if these factors influenced students’ ratings of their own skills in library research. Chi-square analysis showed no significant relationships between self-rated skill level and the demographic variables.’ Although evaluation studies are reported that have considered a number of variables related to demographics, resource use, previous instruction, and evaluation of the library instruction, no studies were located that examined student interest in course content and how that interest might be related to evaluation of the instruction. METHODOLOGY
To collect data for this study, two survey questionnaires were developed; one was administered to students at the beginning of the anthropology course, the other at the end of the course. The preinstruction questionnaire concentrated on prior library experience, and the post-instruction questionnaire focused on evaluation of the library session. Of the 680 students who participated in the study, 533 (78 percent) completed the pre-questionnaire, 554 (81 percent) the post-questionnaire, and 403 (59 percent) completed both. Because of scheduling problems, pre-questionnaires were not given to students who took the course in 1995. Those students did, however, complete the post-questionnaire as well as a supplement that asked some of the pre-questionnaire questions. Questionnaires were distributed during class periods to ensure a good response rate. Students were asked to put an identifying number on both pre- and post-questionnaires so that they could be matched and entered as individual records. The data from the questionnaires was encoded in a dBASE IV structure, exported to an ASCII file, and then analyzed using procedures of the Statistical Analysis System (SAS) available on the Wichita State University (WSU) mainframe. Frequencies were calculated for each of 23 variables, organized into five groups: Evaluation, Demographics, Previous Library Instruction, Previous Use of Resources, and Subject Interest. Chi-square analysis was used to determine the statistical significance of the relationships between the variables in the Evaluation group and all other variables in the other four groups. Relationships were considered significant when the probability value was less than .05; the significance applies to entire tables, not individual cells within the tables.
30 RESEARCHSTRATEGIES
Winter 1997
FINDINGS
Evaluation Students were asked to rate the usefulness of the library instruction in helping them complete the assignment and to indicate any change in their confidence in using the library after instruction. Of the 545
Table 1 Academic Status and Change in Confidence After Library Instruction (N-532) Change
Academic
Status
Freshman
(n=96)
Sophomore
More Confident RowlCol. No.
(n=129)
in Confidence
%
No Change or Less Confident* RowKol. % No.
76
79%/18%
20
21%/17%
108
84%/26%
21
16%/17%
Junior
(n=llS)
91
79%/22%
24
21%/20%
Senior
(n=175)
120
69%/29%
55
31%/45%
Graduate
16
(n=17)
411
Total
94%/
4%
77%/100%
1 121
6%/
1%
23%/100%
Chi-square=13.760; p=O.O08; df=4 *As only three students in this table were “Less Confident,” the numbers for “Less Confident” and “No Change” were combined.
Table 2 Discipline of Previous Library Instruction and Rated Usefulness of Library Instruction (N=244) Rating of Library Discipline of Previous Instruction
No.
Very Useful RowlCol.
%
Instruction Somewhat Useful or Not Useful* RowKol. % No.
92
68%/63%
43
32%/43%
English and other discipline (n=60)
22
37%/15%
38
63%/38%
Other discipline (n=49)
31
63%/‘21%
18
37%/W%
99
41%/100%
English
Total
only (n=135)
only 145
59%/100%
Chi-square=17.449; p=O.OOO;df-2 *As only seven students in this table found the library instruction “Not Useful at All,” the numbers for “Not Useful at All” and “Somewhat Useful” were combined.
Moore-Jansen
31
students who rated the usefulness of the general library instruction, 59 percent found it very useful, 36 percent somewhat useful, and 5 percent not at all useful. Concerning change in confidence after library instruction, 77 percent of these students were more confident in using the library, 22 percent indicated no change in confidence level, and only 1 percent felt less confident. Demographics Students were asked to indicate their academic sfaf~s, major, and whether or not English was their native language. Of the 545‘students who noted their academic status, 18 percent were freshmen, 24 percent sophomores, 22 percent juniors, 32 percent seniors, 3 percent graduate students, and 1 percent “other.” Of the 516 students who listed their major on the post-questionnaires, 26 percent majored in a social science other than anthropology, 15 percent in education or communications, 11 percent in anthropology, 10 percent in business, 9 percent in the sciences, 7 percent in the humanities, 7 percent in a general major, 4 percent in fine arts, 3 percent in health, and 2 percent in engineering; 6 percent were undecided. Of the 529 students who responded to the language question, 90 percent were native English speakers. Only one of the six pairings of description and evaluation variables indicated a statistically significant relationship: academic status and change in confidence after library instruction (see Table 1). Although a high percentage of students at all academic levels felt more confident after the library session, graduate students were particularly likely to see a positive change in confidence level. Nearly half of the graduate students at WSU are over 45 years old; many are returning to academia with little confidence in how to use recently introduced electronic library resources. Formal library instruction may be a way to boost their confidence in using the library. Otherwise, evaluation of the usefulness of the library instruction appeared little related to demographic factors such as academic status, major, and language. Previous Library Instruction Students were asked whether they had hadprevious library instruction by a librarian and, if so, the discipline of the previous instruction. Of the students who answered this question, 46 percent had had previous library instruction. Of the 307 students who reported the discipline of the previous instruction, 55 percent listed English, 22 percent another discipline, and 18 percent library instruction in both English and some other discipline; another 5 percent had either taken a special class or did not identify the specific course.
32 RESEARCHSTRATEGIES
Winter1997
Only one of the four pairings of previous instruction and evaluation variables indicated a significant relationship: discipline of the previous library instruction and perceived usefulness of the library instruction. Table 2 shows the results for the 244 students who both responded to the question about discipline of previous instruction and rated the usefulness of the library instruction. Students who had had previous library instruction for either an English class or for a class in another discipline tended to rate the usefulness of the instruction for the anthropology course higher than students who had had library instruction in both English and another discipline. The simple experience of previous library instruction in any course did not affect the rating of current instruction, but prior experience with more than one type of library session significantly lowered the rating of usefulness, possibly because students with that much experience had had more exposure to library resources and searching techniques. Previous Use of Library Resources Students were asked to indicate whether or not they had prior experience with six resources or searching tools, all of which figured prominently in the library assignment and instruction: current periodicals, LUIS (the online catalog); periodical indexes in general, Social Sciences Index in particular, online keyword searching, and online truncation. Questions about the prior use of the Social Sciences Index, keyword searching, and truncation were first included in 1995, whereas data on the prior use of current periodicals and LUIS was not collected in 1995. Table 3 Plans to Take More Anthropology Courses and Rated Usefulness of Instruction fN=5331 Rating of Instruction
Plan
Take more
Very Useful No. RowlCol. %
Somewhat Useful No. RowlCol. %
Not Useful No. RowlCol. %
anthropology (n-222) No more anthropology (n=293) Undecided (n=18)
162
73%/51%
54
24%/28%
6
3%/24%
140
48%/44%
134
46%/69%
19
6%/76%
Total
315
13
72%/
4%
59%/100%
Chi-square= 35.184; p=O.OOO; df=4
5 193
28%/
3%
36%/100%
0 25
O%/ 0% 5%/100%
There is considerable variation in the percentages of students with prior experience using these library resources and tools: 70 percent had previously used current periodicals, 76 percent LUIS, 51 percent a periodical index, 53 percent the Social Sciences Index, 77 percent keyword searching, and 38 percent truncation. For three of these, the percentage of students with previous experience increased dramatically during the study: from 52 percent in 1990 to 79 percent in 1994 for current periodicals; from 54 percent in 1990 to 86 percent in 1994 for LUIS; and from 38 percent in 1990 to 82 percent in 1995 for periodical indexes. These increases are probably attributable to the greater accessibility of online resources, the expansion of library instruction at WSU, and the larger number of course assignments requiring library research, particularly at the undergraduate level. Regardless of the percentage of students with prior experience in using these resources, there were no significant relationships indicated between such experience and the evaluation of the library instruction. Whether or not the students were introduced to new resources apparently had little influence on their ratings of this particular library instruction. Subject Interest To gauge interest in the subject content of the anthropology course, the post-instruction survey asked students whether they had plans to take mure anthropology courses. Beginning in 1992, students were asked to rate from 1 to 5 (lowest to highest), their interest in eight subject areas of the course: human evolution, genetics, primates, skeletal biology, osteology/forensics, non-human primate evolution, fossil evidence, and human variation. In 1995 they were also asked to rate their overall interest in course content. Of the 543 students who responded to the question about taking additional anthropology courses, 41 percent planned to do so, 55 percent had no such plans, and 3 percent were undecided. The eight subject areas can be ranked by percentages of students who assigned them an interest rating of 4 or 5: primate studies (71 percent), fossil evidence (67 percent), evolutionary history and theory (64 percent), osteology and forensic anthropology (60 percent), human variability (also 60 percent), skeletal biology (52 percent), non-human primate evolution (49 percent), and genetics and cell biology (34 percent). In 1995, when 97 students rated their overall interest in the course on a scale of 1 to 5, the results were as follows: 5 (44 percent), 4 (33 percent), 3 (17 percent), and 2 and 1 (3 percent each). Of the 20 pairings of variables related to subject interest with variables related to evaluation of library instruction, 19 had chisquare probability values indicating significant relationships. Whether or not a student planned to take additional anthropology courses was
34 RESEARCHSTRATEGIES
Winter1997
significantly related to both evaluation variables: the library instruction was considered “very useful” by 73 percent of the students planning to take additional anthropology courses, but by only 48 percent of the students not planning to take more anthropology (see Table 3); and confidence increased for 86 percent of students planning to take additional anthropology courses but for only 69 percent of those not planning to take more anthropology. The eight variables representing the students’ interest in specific subject areas of the course were all significantly related to the perceived usefulness of the library instruction. In general, the higher the interest rating in any particular subject (regardless of which subject), the more useful the student considered the instruction. The relationship between the interest rating for non-human primate evolution and perceived usefulness of the library instruction is representative of the pattern (see Table 4). Similar patterns occur between subject interest variables and the variable related to change in confidence after instruction. For all but one of the eight variables rating a student’s interest in a particular subject, higher interest in the subject (no matter which subject) was significantly related to greater confidence in using the library after instruction. The relationship between the interest rating for human variability and any change in confidence is representative of that pattern (see Table 5). Overall interest in this anthropology course was also significantly related to both evaluation variables. Over 80 percent of the students who gave the highest interest rating (5) to the course considered the library instruction “very useful,” and 90 percent of students who found the instruction very useful rated their interest in the course at 4 or 5. On the other hand, only 27 percent of those who rated course interest at 3 or lower found the course very useful, and only 55 percent of those who found the instruction somewhat or not useful rated course interest at 4 or 5 (see Table 6). Interest in the course was also positively related to greater confidence in using the library after instruction (see Table 7). CONCLUSIONS
The first objective of this study was to determine how effective the library instruction was in helping students complete their library assignment. A high 95 percent evaluated the session as at least somewhat useful for their library research, and 59 percent rated it very useful. A clear majority of students felt more confident in their ability to use the library after the instruction. A second objective of the study was to describe the students taking this course through several variables related to demographics, expe-
Mwre-Jansen
35
rience with library instruction and resource use, and interest in the course. The study indicated that the diversity of backgrounds and interests of students was profound, with over 40 majors and all academic levels represented. Nearly half the students had had previous library instruction, in a variety of disciplines. Most students had Table 4 Interest in Non-Human Primate Evolution and Rated Usefulness of Library Instruction (N-437) Rating of Instruction in Non-Human Primate Evolution* Interest
Somewhat Useful No. Row/Cal. %
Very Useful RowKol. No. %
Not Useful No. RowlCol. %
5 (n=90) 4 (n=123)
65
72%/26%
24
27%/15%
1
l%/
4%
69
56%/28%
53
43%/32%
1
l%/
4%
3 (n=149)
79
53%/32%
60
40%/36%
10
7%/42%
2 (n=53)
24
45%/100/o
22
42X/13%
7
13%/29%
1 (n=22)
11
50%/
27%/
5
23%/21%
24
5%/100%
Total
248
4%
57%/100%
6 165
4%
38%/100%
(X-square= 36.617; p=O.OOO;df=8 *5=highest rating, l=lowest
Interest
Table 5 in Human Variability and Change in Confidence After Library Instruction (N=440) Change
Interest Human
in Variability*
More Confident Row/Cal. No.
in Confidence
%
No Change or Less Confident** RowKol. % No.
5 (n=123)
105
85%/31%
18
15%/18%
4 (n=144)
114
79%/33%
30
21%/30%
3 (n=llO)
82
75%/24%
28
25%/28%
2 (n=46)
31
67%/
9%
15
33%/15%
1 (n=17)
9
53%/
3%
8
Total
341
78%/100%
99
47%/
8%
22%/100%
Chi-square=13.720; p=O.O08; df=4 *5=highest rating, l=lowest **AS only two students in this table were “Less Confident,” the numbers for “Less Confident” and “No Change” were combined.
36 RESEARCHSTRA7’EGIES
Winter1997
had some experience with the resources explored in this assignment. Nearly half planned to take additional anthropology courses, and interest in course content varied widely, both overall and within specific subject areas. The final objective was to seek significant relationships between the evaluation of the library instruction and the background and interests of the students. The results showed little evidence that the demographic variables of major, academic status, or language significantly influenced the evaluation of the instruction except that graduate students experienced a marked increase in confidence in using the library. Despite the great variety in the number and content of previous library sessions that students had attended, those sessions seem not to have significantly influenced the evaluation of this instruction, except for the negative impact of multiple previous sessions. This study also indicated surprisingly little relationship between students’ prior experience with certain resources and how useful they considered the library instruction. The notable exception to the lack of significant relationships between evaluation of library instruction and student characteristics comes with variables representing student interest in the course: in general, the stronger the interest, the more positive the evaluation and the more confidence students gained in using the library. Regardless of their background, past experience, or interests, students found the library instruction beneficial: 95 percent rated it somewhat useful, and 77 percent gained confidence as a result of it. Although subject interest did have a clear positive impact on evaluation, a relatively high percentage of students with little interest in course content nonetheless rated the instruction very useful or felt more confident after it. The relatively small impact of students’ backgrounds and interests on the positive evaluation suggests that the instruction was indeed effective. Perhaps the emphasis on learning resources and tools specific to this assignment was a more important influence on perception of the instruction than was any background variable of the students. The strong tie between the instruction and the assignment may simply have outweighed other potential influences. Future research with these students needs to move beyond a general evaluation of the instruction and explore specific elements of the assignment and the instruction. Do students relate the “usefulness” of library instruction primarily to successfully completing a specific assignment or do they perceive a more general usefulness? Is interest in the assignment itself important to students? Are specific resources consulted because of the instruction or because of prior experience with them? How successful were the students in complet-
ing their assignments? Can the success be tied to the library instruction or are there other factors? To explore some of these questions, librarians must collaborate with course instructors in evaluating
completed assignments and collecting data on specific resources used in each assignment. Table 6 Overall Interest in Ant~opology Course and Rated Usefulness of Library Instruction (N-97) Rating of Instruction Somewhat Interest Course*
Very
in
No.
Useful
or Not Useful**
Useful
RowlCol.
%
No.
RowlCol.
5 (n-43)
35
81%/57%
8
19%/22%
4 (n=32)
20
63%/33%
12
38%/33%
3 (n-16)
4
25%/
7%
12
75%/33%
2 or 1 (n=6f
2
33%/
3%
4
67%/11%
61
Total
63%/100%
36
%
37% /lOO%
Chi-square=18.399; p=O.OOO; df=3 *5=highest rating, l=lowest; ratings 1 and 2 were combined because of small cell size. **AS only one student in this table found the instruction “Not Useful at All,” the numbers for “Not Useful at All” and “Somewhat Useful” were combined.
Table 7 Overall Interest in Anthropology Course and Change in Confidence After Library Instruction (N=97) Change Interest Course*
in
in Confidence No Change or Less Confident**
More Confident No.
RowlCol.
%
No.
Row/Cal.
5 (n=43)
41
95%/53%
2
5%/11%
4 (n=32)
25
78%/32%
7
22%/37%
3 (n=16)
8
50%/10%
8
50%/42%
2 or 1 (n=6)
4
67%/
2
33%/11%
19
20%/100%
Total
78
Chi-square=16.312; p=O.OOl;df=3
5%
80%,‘100%
%
*5=highest rating, lnlowest; ratings 1 and 2 were combined because of small cell size. **As only one student in this table was “Less Confident,“ the numbers for “Less Confident” and “No Change” were combined.
38 RESEARCHSTRATEGIES
Winter1997
This study indicates that, when planning and teaching courserelated library instruction, librarians might do better by gauging student interest in the subject than by considering demographics or previous library experience. At the very least, librarians should not underestimate the influence that subject interest has on students’ appreciation of the instruction. Although librarians cannot control student interest in a subject, they can develop assignments and instruction that build on that interest. For example, if primate studies and fossil evidence are known to be of high interest to students in this anthropology course, the examples and problems covered in instruction and library assignments can concentrate on these areas. Similarly, assignments can be developed that allow students as much latitude as possible in pursuing their own interests within the context of the course. To achieve this goal, librarians must work closely with course instructors. By coordinating subject and library instruction, the academic ideal of achieving substantive results in teaching both content related research skills can be strengthened. REFERENCES
‘Recent examples of evaluation studies that used opinion surveys include: Tom Diamond and Joan Espey McGee, “Bibliographic Instruction for Business Writing Students: Implementation of a Conceptual Framework,” RQ 34 (Spring 1995): 340-360; Rae Haws, “An Attitudinal Study of Students Toward a Required Library Instruction Course,” Research Strategies 5 (Fall 1987): 172179; Gordon B. Leighton and Marsha C. Markman, “Attitudes of College Freshmen Toward Bibliographic Instruction,” College & Research Libraries News 52 (January 1991): 36-38; Marsha Markman and Gordon B. Leighton, “Exploring Freshman Composition Student Attitudes about Library Instruction Sessions and Workbooks: Two Studies,” Research Strategies 5 (Summer 1987): 126-134. Evaluation studies based on both survey questionnaires and paper-and-pencil tests include: Donald Barclay, “Evaluating Library Instruction: Doing the Best You Can with What You Have,” RQ 32 (Winter 1993): 195-202; Joan Kaplowitz, “A Pre- and Post-Test Evaluation of the English 3Library Instruction Program at UCLA,” Research Strategies 4 (Winter 1986): 11-17; Virginia Tiefel, “Evaluating a Library User Education Program: A Decade of Experience,” College & Research Libraries 50 (March 1989): 249-259. *Mary I. Piette and Betty Dance, “A Statistical Evaluation of a Library Orientation Program for Graduate Students,” Research Strategies 11 (Summer 1993): 164-173. 3Linda G. Ackerson, Jeanne G. Howard, and Virginia E. Young, “Assessing the Relationship Between Library Instruction Methods and the Quality of Undergraduate Research,” Research Strategies 9 (Summer 1991): 139-141. ‘Mary Ann Trail and Carolyn Gutierrez, “Evaluating a Bibliographic Instruction Program,” Research Strategies 9 (Summer 1991): 124-129.
The results of a six-pr study indicate little relationship between students’ demographics, previous libray instruction, or prior use of libray resources and how they emluated libray instruction forone undergraduate anthropology wmse. There is a more positive relationship betuxen subject interest and evaluation of the instruction. These results suggest that librarians should be less wmxrned about students’ prior experience and more concerned with ensuring that course-related libray instruction focuses on specific goals and reinforces student interest in the subject.
A
s long as librarians provide library instruction, they will be concerned with its effectiveness. For students evaluating its effectiveness, however, the quality or focus of the instruction might matter less than the wide variety of experiences and interests that they bring to an instructional setting. Upper-level undergraduates and graduate students might, for example, find library instruction less useful than students just beginning an academic career. Students who have had previous experience with library instruction or resources might find additional instruction repetitive and less useful than Cathy Moore-Jansen is Social Sciences Librarian, Wichita State University, Wichita, KS. E-mail: [email protected] Research Strategies, vol. 15, no. 1, pp. 26-38 Q1997 by Research Strategies. All rights reserved.
students receiving formal instruction or using the library for the first time. Students with more interest in the subject matter of the course might rate the instruction more useful than those with less. On the other hand, how students evaluate library instruction related to a required assignment for a particular course may have less to do with individual background or interests and more with how well the instruction helped them complete the assignment. To explore the impact of students’ backgrounds and interests on their evaluation of library instruction, a six-year study was conducted to examine possible relationships between variables related to student demographics, library experience, and subject interests and variables related to student evaluation of library instruction in one undergraduate anthropology course. The intent of this study was to document, if possible, statistically significant relationships among variables-relationships that may or may not seem “obvious,” depending on one’s perspective of library instruction. The method involved collecting data on students taking one course over an extended period of time and seeking statistically significant influences on how useful they perceived the library instruction to be. Such influences could then be taken into account when planning library instruction for this and similar courses. SAMPLE POPULATION AND DESCRIPTION OF ASSIGNMENT The students surveyed in this study were enrolled in the anthropology course Biological Anthropology, lOlQ, at Wichita State University during the summers of 1990 through 1995. They take this course as a science or anthropology requirement, as an elective, or for a combination of reasons. The subject content of the course, which is taught each semester by the same professor, includes human and nonprimate evolution, fossil evidence, primates, osteology, forensics, human variability, and genetics. In this course students are exposed to the scholarly periodical literature and reference sources in biological anthropology. All students are required to write reviews of scholarly journal articles related to some aspect of biological anthropology. Since 1993, students have also been asked to locate article citations on specific topics using periodical indexes. This assignment gives the students needed practice in locating journals and articles on topics that interest them and discourages them from simply browsing journals for articles to review. From the beginning of the study, the library instruction stressed how to locate specific journal titles and journals on a general topic such as “primates” in the online catalog or periodical listing printout as well as journal articles on a specific topic such as “blood genetic markers in human populations” in periodical indexes.
28
RESEARCHSTRATEGIES
Winter1997
The students enrolled in the undergraduate anthropology course during this study represented all academic levels, numerous different majors, and several nationalities. Additional assumptions or expectations about the group are that: they had diverse backgrounds and interests; * some had had previous library instruction, in various disciplines; and l they had varying degrees of interest in the subject content of the course.
l
To explore how these diverse elements might influence the usefulness of library instruction in helping students do their assignments, preand post-library instruction surveys were developed that posed questions about demographic attributes such as major and academic status, prior experience with library instruction and resources, and general evaluation of the library instruction. The primary objectives of the study were: . to determine if the library instruction was useful in helping students do their assignment; l to describe the students in terms of demographics, previous library experience, and subject interests; and l to look for significant relationships between all of those areas and how students evaluated the library instruction. LITERATURERRVIEW The literature was reviewed for studies that used a surveying, rather than testing, methodology for evaluating library instruction. Many such efforts distributed questionnaires to evaluate how “effective” a particular library class or program was in teaching library skills and to gather student opinions about certain aspects of that library instruction.’ Opinion surveys have collected data on variables related to age, gender, language, academic status, major, previous instruction and experience with library research, library skill levels, attitudes towards libraries or librarians, student confidence in using the library, resources used before or during the instruction, usefulness of different parts of the instruction, level of satisfaction with the library instruction, and how easy or difficult the library is to use. Studies that explore relationships between these variables are harder to find. Piette and Dance concluded that graduate students with no or some library experience were more satisfied with an information workshop than were students with more experience. On the other hand, students with greater experience were more satisfied
Maore-Jamen 29
with a database searching workshop than were less experienced students. Whether or not the students were native English speakers made no difference in their satisfaction with either workshop.2 Ackerson, Howard, and Young examined the relationship between the amount of previous library instruction students had had and the quality of their bibliographies, and found conflicting preliminary resu1ts.3 Trail and Gutierrez studied several descriptive variables, including gender, academic status and division, and previous library instruction, to see if these factors influenced students’ ratings of their own skills in library research. Chi-square analysis showed no significant relationships between self-rated skill level and the demographic variables.’ Although evaluation studies are reported that have considered a number of variables related to demographics, resource use, previous instruction, and evaluation of the library instruction, no studies were located that examined student interest in course content and how that interest might be related to evaluation of the instruction. METHODOLOGY
To collect data for this study, two survey questionnaires were developed; one was administered to students at the beginning of the anthropology course, the other at the end of the course. The preinstruction questionnaire concentrated on prior library experience, and the post-instruction questionnaire focused on evaluation of the library session. Of the 680 students who participated in the study, 533 (78 percent) completed the pre-questionnaire, 554 (81 percent) the post-questionnaire, and 403 (59 percent) completed both. Because of scheduling problems, pre-questionnaires were not given to students who took the course in 1995. Those students did, however, complete the post-questionnaire as well as a supplement that asked some of the pre-questionnaire questions. Questionnaires were distributed during class periods to ensure a good response rate. Students were asked to put an identifying number on both pre- and post-questionnaires so that they could be matched and entered as individual records. The data from the questionnaires was encoded in a dBASE IV structure, exported to an ASCII file, and then analyzed using procedures of the Statistical Analysis System (SAS) available on the Wichita State University (WSU) mainframe. Frequencies were calculated for each of 23 variables, organized into five groups: Evaluation, Demographics, Previous Library Instruction, Previous Use of Resources, and Subject Interest. Chi-square analysis was used to determine the statistical significance of the relationships between the variables in the Evaluation group and all other variables in the other four groups. Relationships were considered significant when the probability value was less than .05; the significance applies to entire tables, not individual cells within the tables.
30 RESEARCHSTRATEGIES
Winter 1997
FINDINGS
Evaluation Students were asked to rate the usefulness of the library instruction in helping them complete the assignment and to indicate any change in their confidence in using the library after instruction. Of the 545
Table 1 Academic Status and Change in Confidence After Library Instruction (N-532) Change
Academic
Status
Freshman
(n=96)
Sophomore
More Confident RowlCol. No.
(n=129)
in Confidence
%
No Change or Less Confident* RowKol. % No.
76
79%/18%
20
21%/17%
108
84%/26%
21
16%/17%
Junior
(n=llS)
91
79%/22%
24
21%/20%
Senior
(n=175)
120
69%/29%
55
31%/45%
Graduate
16
(n=17)
411
Total
94%/
4%
77%/100%
1 121
6%/
1%
23%/100%
Chi-square=13.760; p=O.O08; df=4 *As only three students in this table were “Less Confident,” the numbers for “Less Confident” and “No Change” were combined.
Table 2 Discipline of Previous Library Instruction and Rated Usefulness of Library Instruction (N=244) Rating of Library Discipline of Previous Instruction
No.
Very Useful RowlCol.
%
Instruction Somewhat Useful or Not Useful* RowKol. % No.
92
68%/63%
43
32%/43%
English and other discipline (n=60)
22
37%/15%
38
63%/38%
Other discipline (n=49)
31
63%/‘21%
18
37%/W%
99
41%/100%
English
Total
only (n=135)
only 145
59%/100%
Chi-square=17.449; p=O.OOO;df-2 *As only seven students in this table found the library instruction “Not Useful at All,” the numbers for “Not Useful at All” and “Somewhat Useful” were combined.
Moore-Jansen
31
students who rated the usefulness of the general library instruction, 59 percent found it very useful, 36 percent somewhat useful, and 5 percent not at all useful. Concerning change in confidence after library instruction, 77 percent of these students were more confident in using the library, 22 percent indicated no change in confidence level, and only 1 percent felt less confident. Demographics Students were asked to indicate their academic sfaf~s, major, and whether or not English was their native language. Of the 545‘students who noted their academic status, 18 percent were freshmen, 24 percent sophomores, 22 percent juniors, 32 percent seniors, 3 percent graduate students, and 1 percent “other.” Of the 516 students who listed their major on the post-questionnaires, 26 percent majored in a social science other than anthropology, 15 percent in education or communications, 11 percent in anthropology, 10 percent in business, 9 percent in the sciences, 7 percent in the humanities, 7 percent in a general major, 4 percent in fine arts, 3 percent in health, and 2 percent in engineering; 6 percent were undecided. Of the 529 students who responded to the language question, 90 percent were native English speakers. Only one of the six pairings of description and evaluation variables indicated a statistically significant relationship: academic status and change in confidence after library instruction (see Table 1). Although a high percentage of students at all academic levels felt more confident after the library session, graduate students were particularly likely to see a positive change in confidence level. Nearly half of the graduate students at WSU are over 45 years old; many are returning to academia with little confidence in how to use recently introduced electronic library resources. Formal library instruction may be a way to boost their confidence in using the library. Otherwise, evaluation of the usefulness of the library instruction appeared little related to demographic factors such as academic status, major, and language. Previous Library Instruction Students were asked whether they had hadprevious library instruction by a librarian and, if so, the discipline of the previous instruction. Of the students who answered this question, 46 percent had had previous library instruction. Of the 307 students who reported the discipline of the previous instruction, 55 percent listed English, 22 percent another discipline, and 18 percent library instruction in both English and some other discipline; another 5 percent had either taken a special class or did not identify the specific course.
32 RESEARCHSTRATEGIES
Winter1997
Only one of the four pairings of previous instruction and evaluation variables indicated a significant relationship: discipline of the previous library instruction and perceived usefulness of the library instruction. Table 2 shows the results for the 244 students who both responded to the question about discipline of previous instruction and rated the usefulness of the library instruction. Students who had had previous library instruction for either an English class or for a class in another discipline tended to rate the usefulness of the instruction for the anthropology course higher than students who had had library instruction in both English and another discipline. The simple experience of previous library instruction in any course did not affect the rating of current instruction, but prior experience with more than one type of library session significantly lowered the rating of usefulness, possibly because students with that much experience had had more exposure to library resources and searching techniques. Previous Use of Library Resources Students were asked to indicate whether or not they had prior experience with six resources or searching tools, all of which figured prominently in the library assignment and instruction: current periodicals, LUIS (the online catalog); periodical indexes in general, Social Sciences Index in particular, online keyword searching, and online truncation. Questions about the prior use of the Social Sciences Index, keyword searching, and truncation were first included in 1995, whereas data on the prior use of current periodicals and LUIS was not collected in 1995. Table 3 Plans to Take More Anthropology Courses and Rated Usefulness of Instruction fN=5331 Rating of Instruction
Plan
Take more
Very Useful No. RowlCol. %
Somewhat Useful No. RowlCol. %
Not Useful No. RowlCol. %
anthropology (n-222) No more anthropology (n=293) Undecided (n=18)
162
73%/51%
54
24%/28%
6
3%/24%
140
48%/44%
134
46%/69%
19
6%/76%
Total
315
13
72%/
4%
59%/100%
Chi-square= 35.184; p=O.OOO; df=4
5 193
28%/
3%
36%/100%
0 25
O%/ 0% 5%/100%
There is considerable variation in the percentages of students with prior experience using these library resources and tools: 70 percent had previously used current periodicals, 76 percent LUIS, 51 percent a periodical index, 53 percent the Social Sciences Index, 77 percent keyword searching, and 38 percent truncation. For three of these, the percentage of students with previous experience increased dramatically during the study: from 52 percent in 1990 to 79 percent in 1994 for current periodicals; from 54 percent in 1990 to 86 percent in 1994 for LUIS; and from 38 percent in 1990 to 82 percent in 1995 for periodical indexes. These increases are probably attributable to the greater accessibility of online resources, the expansion of library instruction at WSU, and the larger number of course assignments requiring library research, particularly at the undergraduate level. Regardless of the percentage of students with prior experience in using these resources, there were no significant relationships indicated between such experience and the evaluation of the library instruction. Whether or not the students were introduced to new resources apparently had little influence on their ratings of this particular library instruction. Subject Interest To gauge interest in the subject content of the anthropology course, the post-instruction survey asked students whether they had plans to take mure anthropology courses. Beginning in 1992, students were asked to rate from 1 to 5 (lowest to highest), their interest in eight subject areas of the course: human evolution, genetics, primates, skeletal biology, osteology/forensics, non-human primate evolution, fossil evidence, and human variation. In 1995 they were also asked to rate their overall interest in course content. Of the 543 students who responded to the question about taking additional anthropology courses, 41 percent planned to do so, 55 percent had no such plans, and 3 percent were undecided. The eight subject areas can be ranked by percentages of students who assigned them an interest rating of 4 or 5: primate studies (71 percent), fossil evidence (67 percent), evolutionary history and theory (64 percent), osteology and forensic anthropology (60 percent), human variability (also 60 percent), skeletal biology (52 percent), non-human primate evolution (49 percent), and genetics and cell biology (34 percent). In 1995, when 97 students rated their overall interest in the course on a scale of 1 to 5, the results were as follows: 5 (44 percent), 4 (33 percent), 3 (17 percent), and 2 and 1 (3 percent each). Of the 20 pairings of variables related to subject interest with variables related to evaluation of library instruction, 19 had chisquare probability values indicating significant relationships. Whether or not a student planned to take additional anthropology courses was
34 RESEARCHSTRATEGIES
Winter1997
significantly related to both evaluation variables: the library instruction was considered “very useful” by 73 percent of the students planning to take additional anthropology courses, but by only 48 percent of the students not planning to take more anthropology (see Table 3); and confidence increased for 86 percent of students planning to take additional anthropology courses but for only 69 percent of those not planning to take more anthropology. The eight variables representing the students’ interest in specific subject areas of the course were all significantly related to the perceived usefulness of the library instruction. In general, the higher the interest rating in any particular subject (regardless of which subject), the more useful the student considered the instruction. The relationship between the interest rating for non-human primate evolution and perceived usefulness of the library instruction is representative of the pattern (see Table 4). Similar patterns occur between subject interest variables and the variable related to change in confidence after instruction. For all but one of the eight variables rating a student’s interest in a particular subject, higher interest in the subject (no matter which subject) was significantly related to greater confidence in using the library after instruction. The relationship between the interest rating for human variability and any change in confidence is representative of that pattern (see Table 5). Overall interest in this anthropology course was also significantly related to both evaluation variables. Over 80 percent of the students who gave the highest interest rating (5) to the course considered the library instruction “very useful,” and 90 percent of students who found the instruction very useful rated their interest in the course at 4 or 5. On the other hand, only 27 percent of those who rated course interest at 3 or lower found the course very useful, and only 55 percent of those who found the instruction somewhat or not useful rated course interest at 4 or 5 (see Table 6). Interest in the course was also positively related to greater confidence in using the library after instruction (see Table 7). CONCLUSIONS
The first objective of this study was to determine how effective the library instruction was in helping students complete their library assignment. A high 95 percent evaluated the session as at least somewhat useful for their library research, and 59 percent rated it very useful. A clear majority of students felt more confident in their ability to use the library after the instruction. A second objective of the study was to describe the students taking this course through several variables related to demographics, expe-
Mwre-Jansen
35
rience with library instruction and resource use, and interest in the course. The study indicated that the diversity of backgrounds and interests of students was profound, with over 40 majors and all academic levels represented. Nearly half the students had had previous library instruction, in a variety of disciplines. Most students had Table 4 Interest in Non-Human Primate Evolution and Rated Usefulness of Library Instruction (N-437) Rating of Instruction in Non-Human Primate Evolution* Interest
Somewhat Useful No. Row/Cal. %
Very Useful RowKol. No. %
Not Useful No. RowlCol. %
5 (n=90) 4 (n=123)
65
72%/26%
24
27%/15%
1
l%/
4%
69
56%/28%
53
43%/32%
1
l%/
4%
3 (n=149)
79
53%/32%
60
40%/36%
10
7%/42%
2 (n=53)
24
45%/100/o
22
42X/13%
7
13%/29%
1 (n=22)
11
50%/
27%/
5
23%/21%
24
5%/100%
Total
248
4%
57%/100%
6 165
4%
38%/100%
(X-square= 36.617; p=O.OOO;df=8 *5=highest rating, l=lowest
Interest
Table 5 in Human Variability and Change in Confidence After Library Instruction (N=440) Change
Interest Human
in Variability*
More Confident Row/Cal. No.
in Confidence
%
No Change or Less Confident** RowKol. % No.
5 (n=123)
105
85%/31%
18
15%/18%
4 (n=144)
114
79%/33%
30
21%/30%
3 (n=llO)
82
75%/24%
28
25%/28%
2 (n=46)
31
67%/
9%
15
33%/15%
1 (n=17)
9
53%/
3%
8
Total
341
78%/100%
99
47%/
8%
22%/100%
Chi-square=13.720; p=O.O08; df=4 *5=highest rating, l=lowest **AS only two students in this table were “Less Confident,” the numbers for “Less Confident” and “No Change” were combined.
36 RESEARCHSTRA7’EGIES
Winter1997
had some experience with the resources explored in this assignment. Nearly half planned to take additional anthropology courses, and interest in course content varied widely, both overall and within specific subject areas. The final objective was to seek significant relationships between the evaluation of the library instruction and the background and interests of the students. The results showed little evidence that the demographic variables of major, academic status, or language significantly influenced the evaluation of the instruction except that graduate students experienced a marked increase in confidence in using the library. Despite the great variety in the number and content of previous library sessions that students had attended, those sessions seem not to have significantly influenced the evaluation of this instruction, except for the negative impact of multiple previous sessions. This study also indicated surprisingly little relationship between students’ prior experience with certain resources and how useful they considered the library instruction. The notable exception to the lack of significant relationships between evaluation of library instruction and student characteristics comes with variables representing student interest in the course: in general, the stronger the interest, the more positive the evaluation and the more confidence students gained in using the library. Regardless of their background, past experience, or interests, students found the library instruction beneficial: 95 percent rated it somewhat useful, and 77 percent gained confidence as a result of it. Although subject interest did have a clear positive impact on evaluation, a relatively high percentage of students with little interest in course content nonetheless rated the instruction very useful or felt more confident after it. The relatively small impact of students’ backgrounds and interests on the positive evaluation suggests that the instruction was indeed effective. Perhaps the emphasis on learning resources and tools specific to this assignment was a more important influence on perception of the instruction than was any background variable of the students. The strong tie between the instruction and the assignment may simply have outweighed other potential influences. Future research with these students needs to move beyond a general evaluation of the instruction and explore specific elements of the assignment and the instruction. Do students relate the “usefulness” of library instruction primarily to successfully completing a specific assignment or do they perceive a more general usefulness? Is interest in the assignment itself important to students? Are specific resources consulted because of the instruction or because of prior experience with them? How successful were the students in complet-
ing their assignments? Can the success be tied to the library instruction or are there other factors? To explore some of these questions, librarians must collaborate with course instructors in evaluating
completed assignments and collecting data on specific resources used in each assignment. Table 6 Overall Interest in Ant~opology Course and Rated Usefulness of Library Instruction (N-97) Rating of Instruction Somewhat Interest Course*
Very
in
No.
Useful
or Not Useful**
Useful
RowlCol.
%
No.
RowlCol.
5 (n-43)
35
81%/57%
8
19%/22%
4 (n=32)
20
63%/33%
12
38%/33%
3 (n-16)
4
25%/
7%
12
75%/33%
2 or 1 (n=6f
2
33%/
3%
4
67%/11%
61
Total
63%/100%
36
%
37% /lOO%
Chi-square=18.399; p=O.OOO; df=3 *5=highest rating, l=lowest; ratings 1 and 2 were combined because of small cell size. **AS only one student in this table found the instruction “Not Useful at All,” the numbers for “Not Useful at All” and “Somewhat Useful” were combined.
Table 7 Overall Interest in Anthropology Course and Change in Confidence After Library Instruction (N=97) Change Interest Course*
in
in Confidence No Change or Less Confident**
More Confident No.
RowlCol.
%
No.
Row/Cal.
5 (n=43)
41
95%/53%
2
5%/11%
4 (n=32)
25
78%/32%
7
22%/37%
3 (n=16)
8
50%/10%
8
50%/42%
2 or 1 (n=6)
4
67%/
2
33%/11%
19
20%/100%
Total
78
Chi-square=16.312; p=O.OOl;df=3
5%
80%,‘100%
%
*5=highest rating, lnlowest; ratings 1 and 2 were combined because of small cell size. **As only one student in this table was “Less Confident,“ the numbers for “Less Confident” and “No Change” were combined.
38 RESEARCHSTRATEGIES
Winter1997
This study indicates that, when planning and teaching courserelated library instruction, librarians might do better by gauging student interest in the subject than by considering demographics or previous library experience. At the very least, librarians should not underestimate the influence that subject interest has on students’ appreciation of the instruction. Although librarians cannot control student interest in a subject, they can develop assignments and instruction that build on that interest. For example, if primate studies and fossil evidence are known to be of high interest to students in this anthropology course, the examples and problems covered in instruction and library assignments can concentrate on these areas. Similarly, assignments can be developed that allow students as much latitude as possible in pursuing their own interests within the context of the course. To achieve this goal, librarians must work closely with course instructors. By coordinating subject and library instruction, the academic ideal of achieving substantive results in teaching both content related research skills can be strengthened. REFERENCES
‘Recent examples of evaluation studies that used opinion surveys include: Tom Diamond and Joan Espey McGee, “Bibliographic Instruction for Business Writing Students: Implementation of a Conceptual Framework,” RQ 34 (Spring 1995): 340-360; Rae Haws, “An Attitudinal Study of Students Toward a Required Library Instruction Course,” Research Strategies 5 (Fall 1987): 172179; Gordon B. Leighton and Marsha C. Markman, “Attitudes of College Freshmen Toward Bibliographic Instruction,” College & Research Libraries News 52 (January 1991): 36-38; Marsha Markman and Gordon B. Leighton, “Exploring Freshman Composition Student Attitudes about Library Instruction Sessions and Workbooks: Two Studies,” Research Strategies 5 (Summer 1987): 126-134. Evaluation studies based on both survey questionnaires and paper-and-pencil tests include: Donald Barclay, “Evaluating Library Instruction: Doing the Best You Can with What You Have,” RQ 32 (Winter 1993): 195-202; Joan Kaplowitz, “A Pre- and Post-Test Evaluation of the English 3Library Instruction Program at UCLA,” Research Strategies 4 (Winter 1986): 11-17; Virginia Tiefel, “Evaluating a Library User Education Program: A Decade of Experience,” College & Research Libraries 50 (March 1989): 249-259. *Mary I. Piette and Betty Dance, “A Statistical Evaluation of a Library Orientation Program for Graduate Students,” Research Strategies 11 (Summer 1993): 164-173. 3Linda G. Ackerson, Jeanne G. Howard, and Virginia E. Young, “Assessing the Relationship Between Library Instruction Methods and the Quality of Undergraduate Research,” Research Strategies 9 (Summer 1991): 139-141. ‘Mary Ann Trail and Carolyn Gutierrez, “Evaluating a Bibliographic Instruction Program,” Research Strategies 9 (Summer 1991): 124-129.