Proving the efficacy of library instruction evaluation

PROVING THE EFFICACY OF LIBRARY INSTRUCTION EVALUATION

RACHEL Eastern

FENSKE

Washing ton Umversl ty

ANN ROSELLE Eastern

Washmgton

Umverslty

ABSTRACT Thts article demonctrutes the uyejulnerr of evaluatcon surveyr m mprovmg a one-hour library mstructton session Result7 from the surveysfor an Englzsh composltlon clasc were used to modljj the mstructlon session A companson of the results from the student evaluation surveys from the orcgmal hbran, mstructlon session and the modljied one showed szgmjicant improvement m several ureas It IF concluded that evaluation surveys do have an integral role to pla) m lmprovrng library mstructlon

How many times have mstructlon librarians taught a one-hour sessionand wonderedif the mstructlon was really effective? In sucha brief period of time, librarians may be askedto teach search processesand methodologies, conduct demonstrations, employ evaluation techniques,and engagestudentsm hands-onexerclres or other actlvltles that really require more instructional time Evaluation surveys can help hbrarlans identify those elements of a sessionthat are effective and thosethat are not Caseslllustratmg the strengthsand weaknessesof various library mstructlon evaluation methodsat the umverslty level have been presentedthroughout the library literature I A recent article by Patrick Ragamspresents results from a national survey of librarians at 44 colleges and umversltles, all of whom were asked questlons about how library mstructlon was evaluated at their mstltutlon * The article states that the one-hour session1sproblematic becauseof the limited mstruction time and that It may be difficult to evaluate such a sessionusing common evaluation methods such as satisfaction surveys *Du-ect all correspondence to Rachel Fenske Eastern Wa~hmgton Umversrt-, Marl Stop 84, 816 F Street, Chene>, WA 99004-2423, USA E-mall rachelfenske@mad ewu edu RESEARCH STRATEGIES, Volume 16, Number CopyrIght 0 1999 by Elsevler Science Inc All rights of reproduction m any form reserved ISSN 0734-3310

3, pages 175-185

176

RESEARCH

Vol

STRATEGIES

16/No

3/1998

At Eastern Washington Umverslty, d team of hbrananc developed an evaludtlon form that IS used m all m\tructlon sessions, mcludmg the one-hour session The evaluation forms have been used since fall 1997, and the mformatlon gamed from the surveys has been invaluable The followmg article deTcrlbe5 the one-hour library session wlthm the English Composition Program, the evaluation methods employed, dnd the impact the evaluation results have hdd on the mstructlon program In particular, the article illustrates the efficacy of student and faculty evaluation of the one-hour Instruction \esslon and how the Ea\tem Washington University Llbranes used this mformatlon to enhance and expand the mstluctlonal time for a large cohort of faculty and students who participate m hbrary mstructlon every quarter

OVERVIEW

OF INSTRUCTION

SESSION

Enghsh 201, a component of the English Composlhon Program at Eastern Washmgton Umverqlty, 1s a required course for dll students The composltlon of the classe\ 1sprimarily freshman and transfer students Eastern Wayhmgton Umver\lty attracts many transfer students from local community college\ and has a significant nontradltlonal student populdtlon Enghsh 201 engages students m a multitude of writing exercises and provides them with their first formal mtroductlon to library resources Six papers are wrltten throughout the quarter, including summary papers, papers that construct and analyze arguments, dnd those that objectively de\crlbe an ls\ue The last two papersof the quarter, d fdct-finding paper and a persuajlon paper, require that students use at least four hbrary resources Theseresourcescan be monographs,however, emphasis1$placed on perlodlcal literature The library Instruction for English 201, a rmgle one-hour session 1sdeclgnedto help the students learn how to engage m the researchplocesq Inctructlon m the orgamzatlon and structure of literature, ds well db in how to identify dndnarrow d topic, 15given Ddtabace searchjkllls are introduced with an explanation of Booledn operators,dlstmLtlonsbetween keyword dnd subject searchingtechmque5,and the use of controlled vocabuldry Criteria usedin critically evaluating resourcesfor their value and relevancedre dlsoemphd\ized Ten sectionsof English 201, consl\tmg of dpproxlmdtely 25 studentr each, were tdught during the fall quarter of 1997 Each cectlon was taught by a second-yearEnghshgrdduate student, commonly referred to as d teaching asslstdnt The hbrary instruction scsslons were taught by a team of hbrarldns (two hbraridns to d 5esslon)during the rlxth week of the quarter Instruction was conducted during this week becauce it wd\ only by this time that studentc were begmnmg to formulate their research topics The team tedching dpproachprovided the opportunity for librarians to learn from one another, dqslstedhbrarianc who had not been engaged m mstructlon for some time dnd gave the studentsthe opportunity to expellence various teaching ctyles Prior to the library mstructlon, the Library In,tructlon Coordmator met with the teaching asslatantsto explain and dl\cu\\ the \esclon content In addltlon, to better prepale the qtudentj for the mstructlon session,a pre-library as\lgnment (Appendix A) was developed by the coordinator dnd was dlstrlbuted to the studentsby their teaching aFslstant\ four to five days before receiving library mstructlon The purposeof the a\\lgnment was to hdve the studentsformulate a resedrchtopic and begm andlyzmg thdt topic m prepdratlon for conductmg library research The teaching as$lrtantcwere to work with their studentsdulmg claqs time to narrow their topic\ dnd to develop a ll\t of keywords that would dde-

Proving

the Effmcy

of Lb-ary

Instruct/on

fva/ut/on

177

quately describe each concept of their topic The assignmentwould then be dlscussed durmg the hbrary mstruction session,hlghhghtmg the importance of fine tumng the focus of a topic and demonstratmgthat different databasesoften use different terminology to describea concept Techniquesm combmmg various terms with Boolean operatorswould al?obe demonstratedusmg conceptsfrom the students’pre-library assignments After the mstructlon session,a library exercise was distributed to the students This exercise served as a tool to reinforce what studentshad learned about using the library’s electronic databases,mterpretmg cltatlons, developmg search strategies,doing Boolean searches,and evaluatmg mformatlon 3

EVALUATION

METHOD

The pnmary purpose of evaluating the English 201 library mstructlon sessionwas to improve the mstructlon program However, teaching accountablhty 1smcreasmglybecoming important throughout the entire University, and evaluation resultsdo serve asone component, amongmany others, of hbrary faculty reviews It should be pointed out that library faculty include their own commentsabout their mstructlon sessionswhich often shedhght on evaluation results Similar to Ragams’ survey results, a combmatlon of methods was used to evaluate the English 201 library mstructlon program First, students were asked to complete a 15question closed-endedsurvey about the content of the mstructlon session,the structure of the session,and the presentation of the material by the hbranan (Appendix B) Two open-endedquestions,“What did you or your classdo to preparefor the library session?” and “What else would you like to know about locating information m the hbrary’” were also included The student evaluations, which take approximately one to two minutes to complete, were dlstnbuted to the studentsby the teaching assistantsthe next day m class Second, the teaching assistantswere askedto complete a four-question evaluation form on preplanning activities, the content of the session,and the mstructor’s teaching effectlveness(Appendix C) The teaching assistantswere encouragedto make further commentsm writing or m person According to Ragams, student and faculty evaluations were two of the techniquesmost frequently used by the 44 collegesand umversltles m his survey4 Of course, Ragamsaccurately cautionsagainstdrawmg too broad of a conclusion from his resultsbecausethey were gatheredfrom a self-selectedsample Third, the studentswere askedto complete a post-assignmentsurvey, which wasdlstnbuted to the studentsby their teaching assistantsat the end of the quarter after they had completed their final writing proJect This occurred approximately two weeks after the library mstructlon sessionThe survey consistedof five closed-endedquestions The openendedquestion “What elsewould you have liked presentedm the hbrary mstructlonal session?” 1salso repeated The idea behind the post-assignmentsurvey was to obtain feedback from students after they had the opportunity to apply the skills and knowledge obtained during the library mstructlon session Fourth, on a mutually agreeablebasis, English 201 hbrary instruction sessionswere observedby peersand by the Library Instruction Coordmator It was believed that observers could gather ideasthrough attending sessionsthat could be usedto enhancetheir own classes In addition, colleglal feedback from the observer was used by the hbrman to

RESEARCH

178

STRATEGIES

Vol 1 ~/NO 3/1998

make Improved adjustmentsto their classesA peer review processamong hbrary faculty takes place once d yea and, It 1sbelieved, that observation of mstructlonal Se\slonScould assistm obtaining mformatlon to be Incorporated mto this process

RESULTS

FROM

STUDENT AND TEACHING EVALUATION FORMS

ASSISTANT

The remammg dlrcusslon ~111concentrate primarily on the effectlvene\s of, and the impact of results from, the mltlal studentevaluation form and the faculty evaluation form These forms were developed by an Instruction Evaludtlon Task Force within the Edstern Washington University Libraries dnd were approved by the library faculty asa whole The evaluation forms were developed u\mg Teleform@software which enables mtormatlon from the forms to be scdnnedmto a computer and then saved ac a ddtd file The ddtd file\ can then be handled m SPSS for Wmdows to calculate frequencies dnd other evaluative summary data Each librarian who instructed m the program received d copy of the trequenclesfor each survey question, a list of commentsfrom the \tudentc, and the faculty evaluation for their sections The Library Instruction Coordmator revlewed all of the evdluatlon forms Resultsfrom the evaluation forms were then discussedat dn mformdl meetmg At the meeting, the Librdry Instruction Coordmator was able to ,ummarlze certain points, the hbranans were dble to give their own interpretations of the evaluation result\, and many anecdotalexperienceswhile instructing the English 201 studentswere shared

Time

Constraints

From the studentevaluation forms, it wasascertamedthat the one-hoursessiondid not provide an adequateamount of time for mstructlon It was found that 34 9% of the students felt that the amount of materId covered m the cessionwas not suited to the time Jlotted In addition, 33 1% of the studentswere not comfortable with the pace of the mstructlondl presentation The hbranans’ own “rushed feelings” while instructing concurred with the\e finding\ Indeed, this supports Ragams’own reservdtlon\ about the one-hour \ecFlon It becameimportant. nevertheless,that the students’perceptlonsmatchedthoseof the hbrarians Thor evaluation question was integral m negotiating with the Dlrector of the Engh\h Compoation Program for more m\tructlondl time As d result, the Enghsh 201 hbrdv instruction componentnow con3lrtsof two one-hourm\tructlonal sessions, one week dpart

Hands-on

Experience

Due to the time llmltatlons of the one-hour \e\slon dnd becduselndlvldual student workstations had not yet been installed m the library m\tructlon room no hands-oncomponent wds provided for the students According to the studentevaluation forms, 60 3% of the studentsfelt that they would hdve Irked a hands-onexperience Comment\ trom the teaching assistants’evdluatlon forms concurred that the studentsneeded to be engdged m the useof the databasesActive leammgwas lmperdtlve for then 5ucces5As a result, the English 201 library mstructlon program was modlhed to mclude a 20-mmute hdnds-on component durmg the second instructional sessionon termmdls throughout the library Without the evaluation forms the hbrarlanc would only be mdkmg an educdtedguessthat

Provmg the Efficacy of Library Instructron

Evalution

179

the studentsneededhands-onexperiences It 1sexpected that the hands-oncomponentwill greatly improve once the library mstructlon room 1sproperly equipped with workstations

Tour From both the teaching asslstants’evaluations and from the open-endedcommentsby the students,it was discovered that the studentswould have liked more mformatlon on the location of the resourcesbemg dlscussedand/or a tour of the hbrary This came as a surpnse to the hbranans becausemtroductory tours were provided for most of these same studentsm English 101 classesGeneral tours for any EasternWashington Umverslty student were also provided during the first 2 weeks of the fall semesterWIthout the evaluation forms, the librarians would not have ldentlfied this information need The requestby studentsfor a tour also assistedm negotiating for additional mstructlonal time

Content In an effort to visually depict Boolean searching,a quick exercise using all of the studentsm the classwasconducted Studentswere requestedto standif they met a certamcntena, e g , wearmg blue Jeans Additional crltena were then usedto demonstratethe Boolean operators“AND” and “OR ” Studentsremainedstandmgor took their seatsdependingon the cntena 5 This exercise was reconsideredby severalof the hbrarlansfor future English 201 library instruction sessionsbased on some of the negative comments received from the students This exercisemay not have beenas effective at EasternWashingtonUniversity as it may be at other mstltutlons due to a high number of nontradltlonal students With an additional secondsession,the librarians madeseveralother changesto the content of the mstructlon sessionsThese decisionswere made m part basedon the results from the student evaluation forms, which found that 48 0 % felt the material presentedm the mstructlonal sessionwas not new to them Only 27 2% felt the material presented m the sessionwas new to them Without the evaluation forms, hbranans would only be making assumptions,basedon then interactions with studentsat the reference desk and m other instructional sessions, about mformatlon and searchskills that would be new to the students Although many of the studentsdid not feel the material was new to them, according to the evaluation forms, they were able to conceive the apphcablhty of the library mstructlon session Many, 48 8%, felt that the mformatlon m the sessioncould help them m English 201, 58 1% felt that they could apply the mformatlon to their studiesm general

Performance

of the Librarians

The Library Instruction Coordmator was able to review all of the resultsfrom the evaluatlon form questionsthat were meantto measurethe hbrman’s mstructlonal effectiveness This was particularly Important to the successof the program becauseso many different librarians were partlclpatmg as mstructors Both the student and faculty evaluation forms reflected that all of the hbranans were performmg equally well This was encouraging because,although the content of the mstructlon sessionwas similar, teachmg styles do vary Without these evaluation results, it would have been more difficult to support the concept of mvolvmg many different librarians, both from public and technical services,in the English 201 hbrary mstructlon program

180

RESEARCH

Library

Vol 16/No

STRATEGIES

3/l 998

Preparedness

It becameevident from the student evaluation forms that not all the teaching afslstdnts were adequately preparing their classesfor the library m,tructlon sessionOnly 36 8% of the studentscompleted the pre-library assignmentbefore coming to the hbrary secslonAs previously described, the library component of the English LompoWlon progrdm Ldlied for the teaching assistantsto provide an overvlew of the researchprocessto their studenta and to engagethem in the pre-hbrdry assignment This exercl\e was meant to engagethe studentsin the thought processesof researchsothdt when they attendedthe librdry lecture the camecomponents/processes would be emphdslzed Becduse there wds not 100% pdrticipation in the pre-library dc\lgnment, the Library Instruction Coordindtor wac dbk to discussthe lack of preparednesswith the Director of the English Compo\ltlon Program From this discussion,\ugge\tlons were developedon how to restructure the class\ylldbus so that the tedching dsslstdnt\ had adequatetime m their Clds\ periods to participate in there library pre-planning dctlvltles These modlficdtlons would then be implementedduring winter quarter

COMPARISON

OF FALL AND WINTER

INSTRUCTIONAL

SESSIONS

Modifications to the library mctructlon component of English 201, basedon evaluation results, were implemented m the winter quarter of 1998 One mechanismusedto determme the successof theseLhangeswas to compare the results from the student evaluation forms m fall quarter (137 survey\) to those m wmter quarter ( I9 1 survey\) Five que$tlons from the evaluation forms were selected for comparisonpurposes Those question\ dedlmg with the performanceof the librarians and thosethdt could not be accurately compdred becauseof drdmatic differences m the ses\lonu,I e , opinions &out handc-onexperiences, were not selected Resultsfrom the evaluation forms \how a stgmficant overall nnprovement m the hbrdry mstructlondl sessionsm winter quarter, as perceived by the studentson four of the five questions(Table I) These results show no slgmficant change m the percentage of students who felt that the material presented In the In\tructlonal 4es51onwas new to them It 15speculdted

Comparison

of Fall Quarter

TABLE 1 and Winter Quarter

Student

Evaluation

% Agreed Survey quest/on I wds comfortable wth the pace ot the presentation The amount ot materA covered wa\ wltcd to the tmx allotted The mformdtmn I learned m thl\ w\\mn will help me m this course I thmk I cdn apply the mformatlon I lexned to my \tudlei The materul pre,ented m the mstructlonal \ewon wa new to me

Forms

to Strongly Agreed

Fall 1997

Wm ter 1998

18 1

62 9’

14 3

56 6‘

48 8 58 I

63 ‘)k 6X 9*

212

7X 8

Provmg the Efficacy of Library lnstruct~on

Evalutlon

181

that perhaps by winter quarter many of the students had been exposed to library resourcescompared to their fall counterparts Nevertheless, results from the other four questions clearly demonstratean improvement It appearsthat the additional mstructlon time m winter quarter had a positive impact on students’ opmlons about pace and allocated time Content changes, including a hands-on component, may have posltlvely affected the students’ opmlons about the apphcablhty of hbrary mstructlon to English 20 1 and to then overall studies

CONCLUSION Use of student and teaching assistantevaluation forms unequivocally provided valuable information which was usedto improve the library mstructlon component of the English CompositionProgram at EasternWashingtonUniversity A successfulexpansionof mstructlon time, an additional one-hour session,wasone suchimprovement The extra classsession enabled the hbranans m subsequentquarters to teach more adequately the elements of the researchprocessand databasesearching Most importantly, this extra time was used to incorporate a hands-on component It would have been difficult to negotiate for more library mstructlon time without hard evidence from the evaluation methods Another area of improvement, basedon mformatlon gathered from the evaluation forms, was the content of the mstructlon Feedback from the studentsand the teaching assistantsenabledthe hbranans to make modlficatlons to better addressand accommodatethe mformatlon needs of the students Similar to Ragams,the authors wholeheartedly support the notlon of continuously evaluating and makmg adjustmentsto the evaluation methodsthat are applied After reviewing the results from the question on the student evaluation form about whether or not the mclass assignmentwas useful, it was discovered that many of the studentswere mlsmterpretmg the wording of the question In responseto this, the survey question 1sbeing reworded In addition, the student survey question about appropnatenessof the amount of time allocated 1sbeing reworded to provide studentsthe opportunity to reflect whether they felt there was too little or too much time allocated to the library instruction topic Constantrefinement of evaluation instruments1shighly recommended Survey forms are, of course,Just one way to assessinstructional sessionsIt is recommendedthat hbranans explore alternative evaluation methodsand employ a combmatlon of evaluation methods to obtain as complete a picture as possibleabout the instructional session6 For example, at Eastern WashmgtonUniversity, the use of studentpre-and posttestshasbeenusedto demonstratethe extent of successof a library mstructlon component incorporated within a computer literacy course7 Good overviews of various evaluation methodsused by librarians are presentedby Richard Hume Werkmgs and, more recently by Chnstopher Bober, SomaPoulm, and Lmgma Vlleno 9 At Eastern Washington University, library mstructlon 1sincorporated into the CWTKUlum m various ways This article portrayed how basic library skills have beenincorporated mto a required composltlon course The current complexity of retrlevmg and evaluating mformatlon m a highly automated and expansive mformatlon environment has necessltated more dlsclplme-specific mstructlon and the development of a one-credit library course at EasternWashington Umverslty Llbranes Nevertheless,until one-hour sessions

RESEARCH

182

Vol 16lNo

STRATEGIES

3/1998

are no longer bemg employed, they will be continued to be evdludted Wlthout someform of evaluation, attempts at lmprovmg the one-hour cession,especially becauseof it\ time llmltatlon~, will not be as successful

APPENDIX Library

Worksheet

A

for Fact Finding

Paper

NAME Complete this ds\lgnment BEFORE you come to the library m\tluctlon session 1 Describeyour resedrchtopic ac completely a\ possiblem two or three centencec

2 Now rephrdseyour topic asd questlon

3 Analyze this topic carefully and hst the important concepts or keywords (Your topic may only include two concept\ ) For each concept, create a list of synonym, Thlj will be important when you begin your searchfor materials The more keywords you have the ea$lerit wdl become to locate relevant mater14 on your topic

Concept

1

Concept

2

Concept

3

I choose not to answer A

Not Pronded B

0 0 0 0 0 0 0 The

purpose

of the

0 0 0 0 0 0 0 The

matenal

presented

0 0 0 0 0 0 Cl The information 0 0 Q 0 0 0 0 I thtnk

I can

0 0 0 0 0 0 (3 The amount 0 0 0 0 0 0 0 The

use

covered

presented

voice

were

0 0 0 0 0 0 0 I was

comfortable

0 0 0 0 0 0 0 The m-class

What

else

would

you

was

experience

more

to know

hands-on about

and

to ask pace with was

and and

me in this

to the time well

allotted

organrzed

dunng

the

session

presentation

resources

was

helpful

useful experience

locatrng

information

tn the Irbrary7

Stronelv

to me by my course new

to understand

questrons

the

clear was

rllustratrons

easy

of the

Dmume 4

to my studres

suited

clear

clear

wrth the

assrgnment

like like

was

made

will help

I learned

was

Neutral 3

session

session

of examples

opportunrbes

hands-on

0 0 0 0 0 0 B I would

made

was

instructional

In this

of matenal

0 0 0 0 0 0 B There

Aeree 2

session

in the

tnformation

0 0 0 0 0 0 CI The libranan’s

0 0 0 0 0 0 GI The

mstructronal

the

was

Aeree 1

I learned apply

information

0 0 0 0 0 0 @I A good

Stronelv

to me course

Dlsaeree 5 lnstrbctor

184

RESEARCH

STRATEGIES

APPENDIX

Course

Section

Number

Number

Please

Vol 1 ~/NO 3/l 998

C

fill m vertically

0123456769

0000000000 0000000000

I choose not to answer A 0 0 0 0 0 0 0

Not Prowded I3

Stronglv

I feel more comfortable

0 0 0 0 0 0 0 The handouts 0 0 0 0 0 0 0 The Instructional

Aeree

h

Neutral

1

2

using the library

3

now as a result of the instructional

session

were helpful session

Improved

0 0 0 0 0 0 0 I feel the quallty of my research library Instructtonal session 0 0 0 0 0 0 0 I feel more comfortable since the tnstructlonal

asking

my library

paper/course

a reference

skills assignment

libranan

Improved

as a result of the

for help when usmg the library

session

What else would you have Ilked presented

NOTES

In the library

InstructIonal

session?

AND REFERENCES

I See for example, Mdrla R Sugrdnes and Jame\ A NedI, “Evaluahon of a Self-Paced BIbhographic InstructIon Course” College LUI~ Rvwanh Llhrarfe~ 44 (Nobember 1983) 444-457, Demur Isbell dnd Lisa Kdmmerlocher A Formative, Collegldl Approach to Evaluatmg Course-Integrated Instrwllon Reraurth Strcltq’glec I2 (Wmter 1994) 24-32, and Ann Roscllc, “Usmg the ALA’s ‘Evdhdtlng Llbrdry In?tructlon ( lYY6) ’ Jo~~ntcrl of Atudemcc Lhr~~rrand~y~ 23 (September 1997) 390-397 2 Patrick Ragam<, “Evdluatlon 01 ALddemrc Libralidn~’ hxtructional Perrtormance Report of d Ndtlonai Survey,” Rcwcrrck Tlrnfegle, 15 (Summer 1997) 159-l 75

Provmg the Efficacy of Library instruct/on

Evalutlon

185

3 The library exercise 1s returned to the teaching assistants m the Enghsh Department, and they are responsible for correcting the assignment In the future, results from this exercise ~111 be analyzed by the Library Instruction Coordmator and ~111 serve as another evaluation method used to assess student learning based on the library mstructlon session 4 Ragams, p 163 5 This Boolean logic exercise was taken from Trudl E Jacobson and Beth L Mark, “Teaching m the Information Age Active Learning Techniques to Empower Students,” Reference Ltbrarum 51-52 (1995) 105-120 6 See Ann Roselle, “Usmg the ALA’s ‘Evaluatmg Library In%tructlon’ (1996),” Journal oJ Acudemlc Llbrurumshp 23 (September 1997) 390-397 for discussion of alternative evaluation methods 7 Rachel Fenske, “Computer Literacy and the Library A New ConnectIon,” Reference Servzces Review 26 (Summer 1998) 67-72,78 8 Richard Hume Werkmg, “Evaluatmg Blbhographlc Education A Review and Cntlque,” hbrary Trends 29 (Summer 1980) 153-72 9 Chnstopher Bober, Soma Pouhn, and Lmgma Vileno, “Evaluating Library Instruction m Academic Llbrarles A Critical Review of the Literature, 1980-1993,” m ObruQ Instructron Revlsrted Blbltographzc Instructzon Comes of Age, edited by Lynne M Martin (New York Haworth Press, 1995), pp 53-71